Measuring impact and engagement in online programmes

Our Head of University Access and Digital, Jonny Tyndall, takes a closer look at the impact of our online programme, Sutton Trust Online, since its inception in 2020.

Sutton Trust Online was launched deep in the lockdown of summer of 2020, at a time when school teaching had been severely disrupted and much access and outreach work was forced online. Created with the aim of providing students with the information, advice, guidance and support that they need to access a leading university or apprenticeship through an 18-month journey, we have since onboarded four cohorts of students to the platform, developing our content and technological offering as well as improving our ability to measure the platform’s engagement and impact. As we pass over the three-year mark since the first lockdown, we reflect on our journey measuring online impact and engagement. 

Sutton Trust Online 

In the late 2010s it was becoming apparent that in-person capacity constraints were limiting the students we could reach, meaning we were having to turn away an increasingly high number of highly eligible students from our university access programmes. It became clear to us that we needed to provide a source of support for these students, even if we couldn’t create space for them on an in-person programme. From this, we decided to create a digital platform that would distil elements of our Summer Schools programme through an online, light-touch intervention- Sutton Trust Online.  

We originally planned to launch STO in 2021, but when COVID-19 hit the rapid development of the pandemic significantly shifted the needs of our students and our programmes. We reacted by expediting the platform’s development and welcomed our first cohort of students in summer 2020, building STO in less than four months! This was not the STO that had been originally envisioned- we were not simply offering light-touch support, but helping over 20 universities and hundreds of employers to deliver our existing suite of university, careers and apprenticeship programmes.  

STO was what, during this period, kept the Trust able to deliver our programmes and reach thousands of students across the UK. However, as the world has reopened and activities have resumed, we have returned to our initial aims. Post-pandemic, the vision for STO is simple: to make the digital-only experience as impactful as possible for the 10,000+ students on the platform. 

Measuring engagement and impact 

Delivering programmes online opens up new possibilities for measuring engagement and impact. Whereas on our face-to-face Summer Schools the main engagement metric was whether a student attended the programme or not, with a digital platform like Sutton Trust Online we can access much more granular detail on which elements of the programme students are engaging in and when they are doing so. This poses an interesting question: should we expect engagement and impact from an online programme to be the same as in-person? Our digital intervention was designed to support students at scale in a light–touch way and, given that STO costs significantly less per student than the amount it takes to send a student on a week-long residential summer school, we wouldn’t expect to see the same impact. With this in mind, we have shifted our thinking and are focusing on using technology and data to make the programme more bespoke for each student and tailoring our communications to highlight the content and opportunities most relevant to each young person

What have we found so far? 

Students on STO have high ambitions in relation to their educational journey but need support narrowing down options and navigating the routes to their preferred destination. The majority of students know what subject they’d like to study but are less sure on their preferred universities, with 10% of students selecting “don’t know” as their first choice. Around 2/3 of students intend on applying to a course or university that requires the submission of their application by the early UCAS deadline. Only 15% know someone studying the course they are interested in, and 18% know someone studying at the university that they are interested in, highlighting the growing need to connect students with undergraduates and support them to understand what student life is really like. 

Almost 23,000 students have used STO in the first three years. Nearly a third of these students have attended a webinar, getting vital information and insights from our alumni, university partners and experts. 65,000 messages have been sent to alumni advisors and student ambassadors across 3,700 conversations. 6,000 students have used our personal statement builder, with 1,800 receiving subject specific mentors. 

Students typically engage with just over two of the seven core elements of the programme on STO. We had initially been pushing students to use all elements of the platform but, as we investigated this more fully, we found that, whilst a significant minority of users didn’t know about specific elements of the platform or programme, between 1/3 and 2/3 of users chose not to engage with parts of the programme due to access support elsewhere, whether from schools, friends and family or other organisations. Further research with our students has shown that they are clear about what support and information they need, and when they want to access this. Trusting the intelligent and ambitious young people we work with to access what they need (with regular reminders through emails and texts!) can give them a sense of autonomy and empower them to take control of their own educational journey.  

We have begun to evaluate the efficacy of our online programmes by looking at student destinations. Analysis of UCAS/STROBE data of our first cohort has shown that greater engagement with the online programme was correlated with slightly higher access rates to high tariff institutions (two percentage points) and Russell Group universities (four percentage points). At this early stage of evaluation, it is important to note that the students included in this analysis had all participated in a Sutton Trust Summer School digitally, alongside using STO. As subsequent cohorts, which include students receiving just the STO intervention, reach university age, we will be better able to understand STO’s impact on student destinations. 

“STO broke down university applications so easily. When applying to Oxbridge, it helped make it clear and easy to understand deadlines and what universities expect from us. The whole thing just wasn’t as nerve wracking as it would have been without it.” – STO student

What else do we want to know? 

We have been able to build a solid foundation for engagement and impact measurement on STO over the last three years, incorporating learnings as we have grown and developed the platform. However, as with any innovation, we are continually looking for ways to improve this and better understand what works in the world of online programmes! 

As time goes on, we will be able to gather more destination data as cohorts finish school. Linking this to student engagement is one way in which we will deepen our understanding of the impact of STO.  

Another key focus over the next two years will be joining up the myriad data sets (demographic, attitudinal, engagement and impact) that we have from students on STO to improve our understanding of the impact of the platform and tell compelling stories. Using data science and evaluation techniques will hopefully allow us to pinpoint which elements of the STO programme will most help students with certain characteristics to achieve specific outcomes. If successful, this would allow us to further personalise students’ programme experience and drive engagement in targeted areas. 

We are incredibly proud that over its three years STO has allowed us to support as many as 6,000 more students annually. As we look to the future of STO, it is crucial to us that we continue to ensure that programme impact, quality and evaluation remain at its core. 

“Sutton Trust Online has been the confidence and motivation boost that I needed. I felt rather discouraged and isolated, but being surrounded by likeminded individuals completely changed my outlook.” – STO student