Pandemic and beyond: tracking student outcomes and programme impact over time

Our Data Analysis and Insights Manager, James Yarde, analyses the outcomes for our programmes alumni using data from the Higher Education Access Tracker (HEAT).

Since 1997, the Sutton Trust’s programmes have helped to support over 50,000 young people from less advantaged backgrounds to access leading universities and careers. Over the years, many Sutton Trust alumni have shared their stories with us – extending from their programme experiences, all the way to what this has meant in terms of their long-term educational and career trajectories. 

But, given the large number of young people that the Trust has worked with over the years, this selection of alumni stories just scratches the surface when it comes to overall programmatic impact. At the Sutton Trust, we use a variety of evaluation tools to help ensure that we can get an accurate indication of how impactful our programmes are. Indeed, one such way we can get a better sense of their overarching impact is to look at longer term education and career outcomes data collected through the Higher Education Access Tracker (HEAT). This annual data return allows us to analyse a wide range of metrics over time, including who goes on to university (and, if so, where they go and what they study), as well as degree attainment and graduate destinations. 

The latest analysis of student data collected through HEAT – and conducted by one of our external evaluation partners, the Institute for Employment Studies (IES) – considers outcomes across 73k+ student records. Of these, around 32k participated in a Sutton Trust programme, while 41k applied for a programme but did not get a place. This latter group – composed of young people who applied for a programme running from summer 2018 onwards – provides a benchmark that we can use to help quantify programme impact.

The eligibility criteria for our programmes are informed by our research and help us to target support for those from the most disadvantaged backgrounds. Over time, we have seen a marked increase in the proportion of programme participants who meet the geographic criteria that are used – namely POLAR, IMD and ACORN – and we would therefore expect this to be reflected in the outcomes data, all else being equal. By definition, those who applied for a programme, but were unable to be offered a place, meet fewer of the eligibility criteria on average. To help ensure we get an accurate measure of the impact of our programmes the analysis therefore controls for these underlying factors – including measures such as eligibility for free school meals.

For the first time, the HEAT analysis includes young people who participated in their programme online due to restrictions to delivery made necessary in response to the COVID-19 pandemic. These students (the “2019/20 cohort”) experienced their programme’s summer delivery in 2020, before completing their A levels in the summer of 2021 and, for the most part, enrolling at university in September 2021. By looking at this particular cohort, we can begin to get a sense of the outcomes for young people participating in one of the Sutton Trust programmes solely online. This, along with insights from previous cohorts, serve to develop our understanding of how data related to university access and graduate outcomes have been impacted by the pandemic more broadly.

Patterns of university access for programme participants 

When young people apply for a Sutton Trust programme, run in collaboration with our university and employer partners, they tend to already have a strong sense that they see themselves studying at university. When asked at the point of their programme application to rate on a 9-point scale how likely it is that they will apply to university (with a higher score indicating a greater likelihood), UK Summer Schools (UKSS) participants from the 2019/20 cohort gave an average score of 8.66. This is comparable to those who applied to the UKSS programme but did not get a place, with an average score of 8.64, even though these students tend to be from less disadvantaged backgrounds. 

University access rates for UKSS participants remain high, with almost three quarters (74%) of participants in the 2019/20 programme cohort accessing university in their first HE-ready year – although this is slightly below the access rate for unsuccessful programme applicants (76%). All else being equal, we would expect unsuccessful applicants to be more likely to end up at university as they typically meet fewer markers of disadvantage (such as eligibility for free school meals) than programme participants. When we factor this into statistical modelling, there is no significant difference between the two groups in terms of the odds of enrolling at university.

Only when we look at access rates over the longer term (enrolling within two-years of completing A-levels) does a difference emerge. Looking at the 2018/19 programme cohort, we find that nine in ten (90%) participants had enrolled at university within two years. And when we take into account the background characteristics of participants, this translates to an estimated +60% increase in the odds of enrolling at university compared to those who did not get a UKSS programme place.

Where the real value of the Trust’s programmes reveals itself, though, is with respect to the types of universities that programme participants go on to attend. And, similarly, the degree subject they enrol to study – especially for the Trust’s employability programmes (such as Pathways to Medicine and Pathways to Law). Looking again at the UK Summer Schools programme, around three quarters (74%) of those from the 2019/20 cohort went on to study at a research-intensive university – enrolling at a Sutton Trust partner, or a member of either the Russell or 1994 groups (the former including Oxbridge). When we control for their socio-economic backgrounds, programme participants from this cohort were an estimated +46% more likely to enrol at a research-intensive university than those who applied for UKSS but did not get a place. 

For employability programmes where there is a clear trajectory from degree programme into the workplace (such as Law and Medicine), there is also evidence of programme impact in terms of the subjects studied. Participants in the Pathways to Law (PtL) 2019/20 cohort were significantly more likely to go on to study law at university compared to those who applied for the programme but did not get a place. Around three in five (62%) went on to do so, 7 percentage points higher than unsuccessful applicants. The gap between programme participants and other applicants is even greater for Pathways to Medicine, where just shy of three in five (59%) went on to enrol in either a medicine or dentistry degree course (14 percentage points higher than unsuccessful applicants). This speaks to how these programmes have been designed to empower young people to make informed choices about their future education and career paths, as well as prepare them with the sector knowledge and work experience opportunities to set them up to succeed. 

University outcomes in context 

The reverberations of the pandemic for university access, more broadly, have been visible in the data for some time – not just for those whose programme delivery was directly affected. A blog series published by the Higher Education Statistical Agency (HESA) explores a range of ways in which responses to the pandemic have bled through to outcomes measured in the HEAT data. This ranges from how the move to Centre Assessed Grades (CAGs) to determine A-level grades compounded long-term increases in rates of HE enrolment, to how no-detriment policies – introduced by universities to mitigate the worst impacts of the pandemic on learning and exams – were associated with uplifts in continuation rates and the proportion of students receiving top degree classifications.  

Likewise, evidence from the COVID Social Mobility and Opportunities (COSMO) study – a research collaboration between UCL and the Sutton Trust, tracking young people over time – suggests that the pandemic has had a profound effect on educational outcomes for young people, as well as their aspirations. Among the study’s key findings, almost two thirds (64%) reported that their educational plans had changed (at least to some extent) and attributed this to the pandemic. Furthermore, contextual factors, such as socio-economic background, were found to be strongly associated with decision-making around where to study. Those from disadvantaged backgrounds, for instance, are more likely to plan on living at home while studying at university than their peers. 

Enrolments to research intensive universities were down slightly for the UKSS 2019/20 cohort (74%) compared to the previous year (77%) of students – even though it is broadly in line with the average over the last 10 years of the programme. This slight decrease may in part be explained by the kinds of effects of the pandemic on young people’s educational outcomes that have been evidenced in the COSMO study. Over the longer term, as we collect data for university enrolments within two years of completing A-levels – a measure which tends to be more stable over time than 1-year admission rates – we will be able to get a better sense of the role that the pandemic may have played in terms of university access. 

The pandemic has had clearer cut ramifications for other longer term outcomes that we measure through HEAT, such as continuation rates and degree classifications. Over the past six years, the proportion of programme participants not continuing into the second year on their degree course has typically been around half the level seen in the HE sector as a whole , at 3-4% compared to 6-7%. In the most recent year of data, from the 2019/20 academic year, this gap has narrowed slightly, with 3% of UKSS participants not continuing into their second year, compared to a sector average of just over 5%. The introduction of no-detriment policies to examinations across many UK universities is likely to have contributed to the narrowing of this gap. 

We are proud that, as with previous years, our HEAT analysis demonstrates the strong positive impact of the Sutton Trust’s programmes on outcomes for young people – especially when it comes to access to leading universities and top degree programmes. But there is now also a growing body of evidence regarding the ways in which young people have been impacted by the pandemic. We can now see some of this emerging in terms of outcome measures used in the Sutton Trust’s impact evaluation work via the analysis of HEAT data. As more data is collected for the 2019/20 cohort (and those which follow) this should crystallise further still – allowing us to better understand the impacts of the pandemic and online delivery on the young people we work with.