Impact and the REF: Get informed!

It ain’t exciting but it sure is helpful…

Nicola Sainsbury, the REF’s liaison manager at King’s College London, put together this super-helpful fact sheet about the REF, and how it might process and quantify the work you do outside of specialist academia. We’re not advocating a box-ticking approach to public engagement – because we think there’s more to dialogue than the ‘impact’ rhetoric. But it’s important to stay informed.


The Research Excellence Framework (REF) 2014: a summary


The REF is a single framework for assessment across all disciplines, with a common set of data required in all submissions, standard definitions and procedures, and assessment by expert panels against broad generic criteria.


The REF is a process of discipline-based expert review founded upon expert judgement. Panels draw on appropriate quantitative indicators to support their professional judgement, although expert review remains paramount. Sub-panels for each of 36 units of assessment (UOAs) carry out the assessment, working under the leadership and guidance of four main panels. Further information on the units of assessment and main panels can be found in Appendix 1.


All HEIs are required to prepare and submit a Code of Practice on the fair and transparent selection of staff for their REF submissions.



The primary purpose of REF 2014 is to assess the quality of research in each submission made by institutions:

  1. to use the assessment outcomes to inform the selective allocation of the grant for research to HEIs, with effect from 2015-16.
  2. To provide accountability for public investment in research and produce evidence of the benefits of this investment.
  3. To provide benchmarking information and establish reputational yardsticks, for use within the higher education (HE) sector and for public information.


The Framework states that the following principles govern the conduct of the REF. They set the framework in which the REF team co-ordinates the exercise and to which the panels should adhere.

  1. Equity
  2. Equality
  3. Transparency



In October 2012 the four UK higher education funding bodies invited UK higher education institutions to make submissions to the 2014 REF.


The deadline for submissions was 29 November 2013. Submissions will be assessed by the REF panels during the course of 2014 and results will be published in December 2014. Submission was via an online database as for 2008.


Assessment process

UoAs will form an overall view about each submission, assessing three distinct elements of each submission, against the following generic criteria:

  1. Outputs: The sub-panels will assess the quality of submitted research outputs in terms of their ‘originality, significance and rigour’, with reference to international research quality standards. This element will carry a weighting of 65 per cent in the overall outcome awarded to each submission.

  2. Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit’s approach to enabling impact from its research. This element will carry a weighting of 20 per cent.

  3. Environment: The sub-panels will assess the research environment in terms of its ‘vitality and sustainability’, including its contribution to the vitality and sustainability of the wider discipline or research base. This element will carry a weighting of 15 per cent.


For each submission the sub-panels will develop a ‘sub-profile’ for each of the three elements of the assessment (outputs, impact and environment) that will be combined into an overall quality profile. The profile will show the proportion of the submission at each of the starred levels given below:


Four star Quality that is world-leading in terms of originality, significance and rigour.
Three star Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence.
Two star Quality that is recognised internationally in terms of originality, significance and rigour.
One star Quality that is recognised nationally in terms of originality, significance and rigour.
Unclassified Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.



Content of submissions

a)         REF1a/b/c: Information on staff in post on the census date, 31 October 2013, selected by the institution to be included in the submission.


b)         REF2: Details of publications and other forms of assessable output which they have produced during the publication period (1 January 2008 to 31 December 2013). Up to four outputs must be listed against each member of staff included in the submission.


Each research output must be the product of research, and must be first brought into the public domain, ie made publicly available during the publication period, 1 January 2008 to 31 December 2013 or, if a confidential report, lodged with the body to whom it is confidential during this same period (see paragraphs 111-113); and must be produced or authored solely, or co-produced or co-authored, by the member of staff against whom the output is listed, regardless of where the member of staff was employed at the time they produced that output.


c)         REF3a/b: A completed template describing the submitted unit’s approach during the assessment period (1 January 2008 to 31 July 2013) to enabling impact from its research, and case studies describing specific examples of impacts achieved during the assessment period, underpinned by excellent research in the period 1 January 1993 to 31 December 2013.


d)         REF4a/b/c: Data about research doctoral degrees awarded and research income related to the period 1 August 2008 to 31 July 2013.


Research doctoral degrees awarded (REF4a): each submission must include the number of research doctoral degrees awarded in each academic year (1 August to 31 July) 2008-09 to 2012-13 to students supervised within the submitted unit. The REF team will provide to institutions data collected by HESA on the numbers of research doctoral degrees awarded, to help in preparing submissions and institutions should allocate these data to the relevant REF UoAs in which they are submitting; or they may prepare their data from internal systems using HESA definitions, but the extent to which the total number of doctoral degrees awarded that is submitted by the institution as a whole can exceed the total reported by the institution to HESA in each academic year will be limited


Research income (REF4b): Each submission must include data on the submitted unit’s external research income for each academic year 2008-09 to 2012-13, according to HESA definitions of research income in the Finance Statistics Return (FSR) table 5b. The REF team will provide to institutions research income data collected by HESA and in preparing their submissions, institutions should allocate these data to the relevant UOAs they are submitting in; or they may prepare their data from internal systems, but the REF submission system will limit the extent to which the total income submitted by the institution to the REF can exceed the totals reported by the institution to HESA in each year.

Income-in-kind (REF 4c): The estimated value of Research Council facility time allocated through peer review and used by researchers at submitted units will be provided to institutions for use in preparing submissions and returned in REF 4c.


e)         REF5: A completed template describing the research environment, related to the period 1 January 2008 to 31 July 2013.


Information is required about the research environment (REF 5) for each submitting unit relating to the period 1 January 2008 to 31 July 2013. Each submission must include a single completed REF5 form, consisting of the following sections:


  1. Overview.
  2. Research strategy.
  3. People, including:

–           Staffing strategy and staff development

–           Research students.

  1. Income, infrastructure and facilities.
  2. Collaboration and contribution to the discipline.


Where a submission includes staff from distinct ‘departments’ or other organisational units, the submission should explain this and any distinctive aspects of the research environments of these organisational units, within each section of the environment template. There is no expectation that the environment element of a submission will relate to a single department or coherent organisational unit.


The environment sub-profile will be obtained by assessing the information submitted in REF5 (the environment template), informed by the data submitted in REF4a/b/c.


Key definitions

For the purposes of the REF, research is defined as a process of investigation leading to new insights, effectively shared. It includes work of direct relevance to the needs of commerce, industry, and to the public and voluntary sectors; scholarship; the invention and generation of ideas, images, performances, artefacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and routineanalysis of materials, components and processes such as for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research.


It includes research that is published, disseminated or made publicly available in the form of assessable research outputs, and confidential reports.


Category A staff are defined as academic staff with a contract of employment of 0.2 FTE or greater and on the payroll of the submitting HEI on the census date (31 October 2013), and whose primary employment function is to undertake either ‘research only’ or ‘teaching and research’. Regardless of their job title, all staff who satisfy the definition at paragraph 78, along with the supplementary criteria in paragraphs 79-81, are eligible as Category A staff. Please see the paragraphs for further information.


Research assistants are not normally eligible to be returned to the REF unless, exceptionally, they are named as principal investigator or equivalent on a research grant or significant piece of research work on the census date and satisfy the definition of Category A staff in paragraph 78. Research assistants must not be listed as Category A staff purely on the basis that they are named on one or more research outputs.


Category Cstaff are defined as individuals employed by an organisation other than an HEI, whose contract or job role (as documented by their employer) includes the undertaking of research, and whose research is primarily focused in the submitting unit on the census date. They may be employed by the NHS, a Research Council unit, a charity or other organisation except for an HEI. Submitted outputs by Category C staff will inform the quality profiles awarded to submissions, but these staff will not contribute to the volume measure for funding purposes. Definitions for Category C have been tightened considerably from those used in RAE 2008.


Early career researchers are defined as members of staff who meet the criteria to be selected as Category A or Category C staff on the census date, and who started their careers as independent researchers on or after 1 August 2009. For the purposes of the REF, an individual is deemed to have started their career as an independent researcher from the point at which they held a contract of employment of 0.2 FTE or greater, which included a primary employment function of undertaking ‘research’ or ‘teaching and research’, with any HEI or other organisation, whether in the UK or overseas, and they undertook independent research, leading or acting as principal investigator or equivalent on a research grant or significant piece of research work.


Individual staff circumstances: Individuals may be returned with fewer than four outputs without penalty in the assessment, where their circumstances have significantly constrained their ability to produce four outputs or to work productively throughout the assessment period.


For clearly defined circumstances, the panel criteria statements will provide tariffs to determine the number of outputs that may be reduced without penalty in the assessment, depending on the duration of the circumstance (or combination thereof). For more complex circumstances, the institution will need make a judgement on the appropriate reduction in the number of outputs submitted, and a single panel constituted by the Funding Council for the REF, the Equalities and Diversity Panel will consider these cases on a consistent basis across all UoAs.


Impact: for the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. It includes, but is not limited to, an effect on, change or benefit to:

  • the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

  • of an audience, beneficiary, community, constituency, organisation or individuals

  • in any geographic location whether locally, regionally, nationally or internationally.

Impact includes the reduction or prevention of harm, risk, cost or other negative effects.

Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK or internationally) are excluded. Impacts on students, teaching or other activities within the submitting HEI are excluded. Other impacts within the higher education sector, including on teaching or students, are included where they extend significantly beyond the submitting HEI.


The number of case studies required in each submission will be determined by the number (FTE) of Category A staff returned in the submission. Submissions will not be expected to provide impact case studies that are representative of the spread of research activity across the whole submitted unit.



Nicola Sainsbury

Updated November 2013



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s