Thread by @c_d_hewson

In a massive shock to literally no-one, the #REF2021 draft impact case study template is almost exactly the same as in 2014, with the exception of a few additional boxes at the outset and *drum roll* it can be 5 pages in length.

So.. for everyone who decided to create their own REF2021 planning template rather than adapting the 2014 one, a large 👎

Also a very narrow definition of a 'continued' case study in the #REF2021 guidance. My feeling is that many social science cases we assumed would be continuations are nothing of the sort under these rules.

Interesting paragraph... potential for some interesting mapping (regional/national) and indeed competitor analysis

Research outputs will NOT listed by author name, although I'm sure the power of #bigdata could automatically re-attribute?? #REF2021

Sensible move re: impact case study database, with CC licenced content #REF2021

Interesting to note that after all the post-Stern rumblings around mapping HESA data onto UoAs for #REF2021, we return to this model of elegant flexibility... with a sting in the tail!

The issue is that if you undertake research in a particular field, you have to either submit to that UoA, or create a good argument as to why researchers are being submitted elsewhere. If you have a 'low performing' dept with no natural allies, there's nowhere to hide. #REF2021

Implied permission for HEIs to 'hedge' around #REF2021 multiple submissions to a single UoA.

Excellent, but please don't commission the outfit who created any of HMRC's online portals. 😏 #REF2021

An extra 63 days after submission to get our corroboration for #REF2021 case studies uploaded. What could possibly go wrong?

Quite a lot. You could potentially engage in brinkmanship if you know another HEI requires similar corroboration (e.g. get a doc published on 29/1/2021).

I like this. There's almost a sense of 'we know you probably made some stuff up last time', please try to play a little less 'fast and loose' with your 'facts' this time around #REF2021

'Knock Knock, it's the Sweeney' #REF2021 #nichejoke

A 'model privacy notice' will be provided by @ResEngland - should be useful when collecting letters of corroboration, albeit onus still on HEIs to ensure processes are compliant #REF2021

Potential reductions in outputs requirements in relation to ECRs. I suspect this will be one of the more 'rigorously debated' aspects of the #REF2021 consultation,

A useful paragraph here. Summary: you can either ask for reductions in output requirements (in line with various criteria) or not, but EITHER way, there are ethical requirements that must not be brushed over #REF2021

One thing that is immediately obvious is that Stern's desire to remove gaming from #REF2021 is... not going well.

And there we have it... the #REF2021 calculus

combination of circumstances
--------------------------------- = submission to the inevitable
additional circumstances

Your circumstances can't be interdisciplinary or co-produced 😏😂 #REF2021

Requirements for research outputs for #REF2021... 'effectively shared' doing a fair bit of heavy lifting here.

What would academia be without the 'tolerance of non-compliance'? A generous 5% of outputs allowed to be non-compliant with #REF2021 OA requirements.

Bonfire of the Quangos (the sequel) #REF2021

Very clear, and yet in an ontological sense, not... #REF2021

Continuity of impact definitions from 2014 to #REF2021 (with addition of 'impact on students, teaching or other activities...)

Useful clarification of the knotty issue of 'world-leading' vs geographical range of a #REf2021 impact case study. I dare say this won't shut-down the debate, but the hope is that the level of 'appropriate' reach/significance will happen at UoA level.

Additional guidance on use of quantitative indicators in #REF2021 impact case studies, in line with funder (etc) requirements identified post REF2014.

It'll be very interesting to see whether more #REF2021 case studies/evidence packages will have conditions attached to them (redaction/non-publication) in response to the rather (and perhaps unintentionally) 'open' approach in REF2014

This is a VERY IMPORTANT paragraph, and demonstrates continuity with REF2014. It was always possible this position would be reversed for #REF2021; or at least the 'strongest examples of impact' admonition replaced with a requirement to provide 'representative impact case studies'

Again, continuity with REF2014 - nothing here about how these cases are assessed in the round (possibly in the panel criteria?) #REF2021

As noted earlier, a very soft definition of what a #REF2021 'Continued Case Study' comprises of... this seems a case of preventing HEIs from benefiting from past glories. Areas where academics still consult/manage projects based on 'old' research potentially 'in scope' here.

A helpful paragraph on Public Engagement in #REF2021, albeit not adding anything we didn't already know. This and the paragraph on 'expert testimony' more about further encouraging impact case study authors to widen their scope, as long as underpinning research exists.

Sensible that #REF2021 now requires these fields for impact case study submissions, rather than suggesting they are a list of optional indicators. Will make the post-REF analysis more robust (or in some cases actually robust)

If memory serves, this is a change in #REF2021, as I distinctly remember having to procure translations for several pieces of REF2014 corroboration. Possibly linked to improvements in machine learning?

Institutional Environment is a welcome example of a #REF2021 rule change that saves a good deal of HEI administrators' time. Also helps internal co-ordination processes, as each UoA lead has to 'sing from the same hymn sheet' at the outset.

Good link from use of quantitative data in UoA Environment Templates (REF5b) to @ResMetrics guidance #REF2021

... and as suspected, research and impact strategies to be outlined in the same section of #REF2021 impact template - albeit their appears to be no separate section for 'Open Research' as might have been expected.

Further clarification that starred levels (for outputs and impact) don't equate to geographical scales. Important as this ALWAYS crops up at #REF2021 public meetings.

And once again, by juxtaposing these two paragraphs it is made clear that we shouldn't throw the baby (b) out with the bathwater (a) in #REF2021 - something that occurred (albeit not systematically) in REF2014.

I regret to inform the sector that the #REF2021 template parameters remain the same - no Garamond or Cambria typefaces allowed. 😔

My favourite paragraph. Similar to REF2014 where I engaged in pitched battles with folks who insisted that indicative section limits weren't... erm... indicative.

Pro-tip, you want your impact section to take up as much space as possible in your #REF2021 impact case study,

I like the inclusion of impact in section 3 of the #REF2021 unit-level template, as well as section 1. 👍👍👍

Right... I'll take a breather, and will hopefully have a chance to look at the draft panel criteria in due course. 😃

So. I'm not seeing a whole lot new/of note in the panel criteria as yet, but it's my overall impression that the documents are far better constructed (and phrased) than for REF2014. @ResEngland deserve credit for this.

Case in point, this para. summarising #REF2021 in the round.

Useful recognition that some #REF2021 UoAs are inherently interdisciplinary in nature (in this case UoA2).

Periodic reminder that gaming #REF2021 - by submitting outputs to UoAs that generate more QR money, and then cross-referring them en masse to where they should have been submitted all along - was 'noted' in REF2014. 😉

#REF2021 UoA16, keeping it short and sweet, as we've come to expect. 😃😂

Also notable that UoA19 (Politics and International Studies) DOESN'T mention overlaps with UOA25 (Area Studies), whereas UoAs 20/21 (Social Work & Social Policy /Sociology) are all over the overlaps. 🤔🤔

A welcome occurrence of my favourite 'passive aggressive' HEI phrase/ #REF2021

Welcome recognition that #REF2021 UoA23 (Education), will accept pedagogical work from a range of disciplinary areas (i.e. HEIs have real options here, but probably need to set their institutional strategies in accordance - further guidance on page 39).

Policing interdisciplinary assessment will clearly be one of the pillars upon which #REF2021 will 'succeed' or otherwise. A huge, and vitally important task.

Liking the colour coding of the main panel pop-out boxes btw. #REF2021

This #REF2021 paragraph should be printed in font size 72, laminated, and nailed to every faculty building (inside and out) in the UK. @ResMetrics

Useful clarifications on double-weighting, along with an 'expectation' for monographs (panels C & D) #REF2021

Submitting to #REF2021 Panel C?

Have always wanted to submit an interpretative dance to bring your paper alive?

This *may* be your chance. 👍

Again, some #REF2021 panels (mostly A&B) will be allowed to use citation data, but it's role will be clearly demarcated and cannot eclipse other concerns (for Panel C, ONLY Economics & Econometrics, for Panel D, no UoAs)

This sums up #REF2021 impact perfectly. These lists of definitions/pathways/models are not exhaustive. Every case study is, to some extent, its own little island on a sea of potential relevance.

More could be said about this section, but most of it has been said before. What it does, again, highlight is that impact case studies are iterative rather than formulaic, and require building from the ground up. #REF2021

Interesting note here on panels (B/C/D) NOT wanting an explicit narrative connecting REF2014 to #REF2021 impact case studies. Presumably to discourage cut-and-paste approaches?

Now we enter the 'voluminous lists' section of the impact guidance, albeit there's a very useful paragraph on how Public Engagement will be viewed in #REF2021 (which actually differs little from REF2014) and some material on pedagogy.

Caveat for #REF2021 Panel C (without really explaining how these impact cases will be judged).

Great paragraph on narrative, which will hopefully form the basis for many an impact manager's presentation once the #REF2021 guidance has been finalised.

Again, we knew this but a useful reiteration on what underpinning means in practice (i.e. not constitutative of the quality of a case study, or necessarily sitting in a linear alignment to activities emerging) #REF2021

A seamless transition from the impact template (REF2014) to impact as part of the UoA's research environment (also requirement to explain connection to submitted #REF2021 case studies)

Para. on 'Open Research', that doesn't differ substantively from what we expected out of last year's #REF2021 guidance (albeit some additional material re: Panel D).

I'll skip the Panel processes, and note that Annex A (Examples of Impact and Indicators) is useful, inasmuch as it is all presented together (unlike REF2014 where indicators/types noted separately).

In terms of Appendix E, I'd probably like to see more on Information Security around #REF2021 corroborative evidence, but I assume this will come later, alongside 'User Guidance' documents (as per REF2014)

So... that's it in a nutshell. Two very well written documents that seem to address most of the UK HE sector's concerns, whilst opening up a few more areas for ongoing debate.

Ain't going to make anyone a REF fan who wasn't already...

... but has set off the starter's pistol on a process that will end in just over 28 months' time.

*ENDS*