Briefing No. 7 - Attainment and Assessment

Context

The COVID-19 pandemic and the public health restrictions that followed changed the structure and experiences of education for young people in the UK. School closures, home schooling, online learning and exam cancellations were some of the consequences of the public health measures taken. School closures were intermittent between March 2020 and March 2021 across early years, primary and secondary education settings in the UK.

Home schooling and online learning were direct consequences of school closures that placed unprecedented pressure on schools, teachers, pupils and parents, with the latter becoming the main facilitators of learning during the initial school closure periods. Online learning placed technological expectations on pupils and parents, with socio-economic disparities and inequalities emerging according to technology access, internet access and place to study in the home.¹ Such expectations contributed to widening attainment disparities between pupils from different socio-economic backgrounds during the COVID-19 pandemic.

Rigorous evidence from around the world has highlighted the negative effects of the COVID-19 pandemic on young people’s learning, and the fact that these effects have been particularly large for those from lower socio-economic backgrounds.² We are not able to directly assess lost learning of members of the COVID Social Mobility and Opportunities (COSMO) Study cohort at this time, because pupils in the COSMO cohort did not sit examinations at age 16 in the usual way and we do not have a separate attainment measure. However, the use of Teacher Assessed Grades (TAGs) as alternative methods of assessment raises important issues in its own right.

Exam cancellations were experienced for Years 11, 12 and 13 pupils who were due to sit their GCSE and A-level examinations in 2020 and 2021 (specifically, the COSMO cohort’s GCSEs were affected in 2021). The performance of members of the COSMO cohort was assessed by their teachers based on their knowledge of pupils’ performance during the course, with internally set and marked assessments to support this. However, this method was met with criticism due to the potential for disparities according to pupil characteristics such as socio-economic background.³

Previous academic literature is clear that this is likely to lead to gender⁴ and ethnic⁵ biases in such assessments, and it seems likely this would be true along other dimensions of inequality, too. In the case of A-levels, Anders et al.⁶ highlighted a graduate parent advantage in TAGs.

There is less evidence examining pupils’ attitudes to school closures and the consequences of exam cancellations compared to other educational consequences of the COVID-19 pandemic. Exam cancellations have also had wider implications for young people’s educational experiences. Studies have highlighted pupils’ feelings of helplessness, uncertainty, confusion and anxiety about exam cancellations, as well as a desire for more information in advance about how the system would work and how it would be ensured this would be fair.⁷ Pupils also highlighted concerns about the legitimacy of the awarded grades.⁸ Mylona and Jenkins⁹ also highlighted that pupils expressed concern in relation to exam cancellations due to delays in receiving course qualifications which affected their ability to earn.

In response to the impact of the COVID-19 pandemic on the education of young people, the UK government announced additional funding for education initiatives (February 2021). These included recovery premium payments to schools to support education catch-up, expansion of the National Tutoring Programme, funding extensions for the 16-19 tuition fund and early language development programmes, and summer school provisions in post-primary settings.¹⁰ The National Tutoring Programme particularly focussed around small group and one-to-one tutoring, among other initiatives. More recently, the National Tutoring Programme has been overhauled to direct more of the funding directly to schools.¹¹

Given that context, in this paper we use data from the COSMO Study) – a longitudinal cohort study of over 13,000 young people who were in Year 11 in the 2020/21 academic year – to explore young people’s perceptions of the assessment process they went through in place of GCSE examinations in the academic year 2020/21, including how these differ depending upon young people’s experiences. We then look directly at differences in young people’s performance in their TAGs to examine the extent of inequalities in their performance. We also look at the performance of those who received small group or one-to-one tutoring as part of efforts to support their post-COVID education recovery, particularly compared to those who were offered but did not take up tutoring of this kind.

Pupils’ perceptions of Teacher Assessed Grades

Almost half of pupils overall (48%) reported that they had performed as expected in their teacher assessed grades (Figure 1). As this is not information asked of pupils when receiving grades in a ‘normal’ year, we lack a clear baseline. In addition, disruption to ‘mock’ assessments is likely to have affected young people’s ability to form expectations in the way that they would have in the usual course of education.

We are, however, able to compare these perceptions of performance compared to expectations between different groups in our sample. Across different school types, similar proportions said their grades were ‘as expected’, but the proportions for whom they were better or worse differ markedly. Pupils from independent schools were much more likely to think they had done better than expected than those in state schools (43% compared to 34% for those in state grammars and 29% in state comprehensives). And, on the other side of the coin, independent school-educated pupils were also much less likely to report doing worse than expected than their state-educated peers (at 7%, compared to 15% of those in state grammars and 23% in state comprehensives). This, to some extent, mirrors the finding from COSMO Briefing 2: Education Recovery and Catch-Up, which shows that state school students were much more likely to report that they had fallen behind their classmates than their independent school peers.¹²

Within the state sector, there are also inequalities, with those in grammar schools being more likely to perform better than they had expected. However, within the state sector we can adjust this analysis by prior attainment (not possible with independent schools because many do not do end of Key Stage 2 tests at age 11), since we can see that those with higher prior attainment are more likely to report that they performed better than expected and less likely to report that they performed worse than expected. This means that we are effectively comparing differences in young people’s perceptions among those who have the same level of prior attainment. When doing so, the picture reverses and grammar school pupils are, if anything, slightly less likely to report that their TAGs are better than expected compared to peers in state comprehensive schools with the same level of prior attainment.

Young people reported a range of other concerns regarding the process of being assessed for their GCSEs in 2021 (Figure 2). Over half felt there were too many assessments carried out by teachers instead of the exams (57%), which were too close together (67%) and that there was too much grading uncertainty (78%). Some of these are somewhat in tension with high proportions reporting that they feel underprepared for future studying (60%) as a result, although we do not have a clear baseline against which these percentages should be compared.

A significant minority (around a third) of pupils reported feeling that teachers were biased against certain groups in their teacher assessments (further detail below), with a similar proportion disagreeing with this statement, and the remaining third not expressing a view one way or the other. However, perhaps surprisingly given some of these concerns, only a third said that they would have preferred to have taken normal exams, and just under half (48%) explicitly disagreed with this statement.

Perceptions that teachers’ grading was biased against certain groups varied somewhat by ethnic background and socio-economic status (Figure 3), but there were no notable differences by gender. Black and, especially, Asian young people were more likely to agree with this statement (at 37% and 43% respectively), while those with more advantaged backgrounds (parents with higher occupational status jobs) were less likely to report a perception of bias in teachers’ grading (28% compared to 38% for those whose parents have lower occupational status jobs). Perhaps unsurprisingly, young people were also considerably more likely to see teachers’ grading as biased against certain groups if their performance was worse than expected (Figure 4).

62% agreed with this statement if they reported their performance was worse than expected, whereas only 21% agreed if they felt they had done better than expected. As well as the wider concerns about biases in teacher assessed grades, this finding highlights the ill-feeling that can be engendered by a teacher assessment system such as this.

Young people were also more likely to report that they planned to resit their qualifications at some point in the future if they felt that they had performed worse than expected (Table 1). As data are not routinely collected on young people’s plans to resit GCSEs, we cannot say if these are higher or lower than usual, but it does seem that they are being driven by how well they feel they did compared to expectations.

Differences in Teacher Assessed Grades

Throughout these analyses, we focus on young people’s Teacher Assessed Grades (TAGs) relative to those we would predict a young person would receive based on their performance in Key Stage 2 tests taken at age 11.

Scores above zero represent an average grade for this group that is higher than would be predicted based on age 11 performance alone, while scores below zero represent an average grade below that prediction.¹³ These are commonly used as ‘value-added’ or ‘progress’ measures of attainment.

As a result, our analysis of Teacher Assessed Grades (TAGs) focuses only on pupils in state schools. This is because many pupils in independent schools do not take end of Key Stage 2 national curriculum tests (‘SATs’) at age 11. Given the importance of a measure of prior attainment to understanding the context of pupils’ performance in exams at age 16, it is also necessary to exclude pupils at independent schools when we do not consistently have this baseline information. In addition, a significant proportion of pupils in independent schools are entered for International GCSE (IGCSE) qualifications, which are not included in the Department for Education (DfE)’s National Pupil Database. This further complicates comparisons between state and independent school pupils: the performance of private school pupils appears artificially lower due to absence of some or all of their qualifications.

Young people’s performance in their GCSE TAGs varied substantially by their demographic characteristics. Girls made substantially more progress than boys compared to expected performance based on their performance in Key Stage 2 tests taken when they were 11 (Figure 5). The difference in performance is almost half a grade per subject difference across those subjects included in pupils’ best eight, while differences of more than half a grade are evident between those living in the most and least deprived neighbourhoods (Figure 6).

Considering differences by ethnicity (Figure 7 below), those from White and Mixed backgrounds received lower TAGs than their Asian and Black peers. However, with all of these differences, we stress that we have no comparison for how these patterns would have differed had traditional exams been used, rather than teacher assessment. Analysis by the Sutton Trust and Education Datalab suggests minimal changes in sociodemographic differences in performance at age 16 in the years when teacher assessment was used compared to the years leading up to it.¹⁴

Differences in young people’s performance depending upon experiences of COVID-19, particularly reporting having experienced long-COVID and having to shield, are discussed in more depth in COSMO Briefing 5: Health Impacts and Behaviours.¹⁵ These mirror the findings here that young people with more intense direct experiences of COVID-19 received worse GCSE Teacher Assessed Grades, even after we take into account prior performance and demographic characteristics.

Figure 8 demonstrates that young people who reported having to shield¹⁶ during the pandemic were judged as having made over half a grade less progress compared to expectations based on age 11 attainment than their peers without this experience.

Did catch-up tutoring help?

Significant efforts were made to support pupils to catch up in the aftermath of the pandemic’s disruption. The highest profile of this was through the government-funded National Tutoring Programme, which among other activities, organised the subsidised offer of one-to-one or small group tutoring for state schools to buy in to support young people most in need of this. 34% of the sample reported that they had been offered one-to-one or small group tutoring by their school as part of efforts to help them catch up. Not all young people took up this offer, with 28% of young people reporting that they had actually received this tutoring (just over 80% of those who were offered it).

Since this programme was intended to be targeted at those who were most in need of support, and given the socio-economic inequalities in the impacts of COVID-19 disruption in terms of pupils’ lockdown learning¹⁷ and education catch up¹⁸ we have documented elsewhere, it is reassuring to see (Figure 9) that those from lower socio-economic backgrounds were most likely to be offered and to receive tutoring. 39% of those with parents with routine/manual occupations or who have never worked were offered tutoring (33% received it), compared to 29% of those with parents with higher/managerial occupations being offered tutoring (24% received it). Consistently, COSMO Briefing No. 2: Education Recovery and Catch-Up¹⁹ also reports that those from the most deprived schools were the most likely to report being offered tuition and taking part.

Similarly, as shown in Figure 10, those with lower levels of prior attainment were the most likely to be offered (41%) and receive (34%) tutoring. And, while there is no gender difference (Figure 11) in receipt of tutoring (28%), there was a small gender difference in the probability of being offered tutoring, with boys more likely to be offered (36% cf. 33% for girls), but less likely to take up that offer.

But did receiving tutoring help young people’s performance, as measured using their Teacher Assessed Grades? At first glance, as in the top section of Figure 12, it might appear tutoring was counterproductive: those who received tutoring made less progress between their national attainment tests at age 11 and their Teacher Assessed Grades at age 16. However, this ignores that pupils who received tutoring were likely to be those most badly affected by disruption, so we would expect their performance to be worse.

To address this, we first focus only on those who were offered tutoring (bottom section of Figure 12). Having done this, there is now more progress between age 11 and 16 among those who received tutoring than those who were offered tutoring but did not take it up. Of course, we must acknowledge that young people’s decision of whether or not to take up the tutoring offered will be predicted by other factors, including their demographic characteristics and, importantly, the perception of the value of such tutoring to them.

As such, we use the statistical method of regression modelling to try to compare differences in TAGs among individuals with the same set of characteristics. Our results are reported in Table 2 and tell us the difference in young people’s best eight GCSE TAGs between those who received tutoring and those who did not, among those with the same level of characteristics indicated. Unlike the analyses above, we are using unadjusted attainment 8 scores, so performance is not adjusted for baseline attainment at age 11 until stated otherwise.

The difference reported in the first column (Raw Diff.) simply compares those who did receive tutoring with those who were offered it but did not take it up. Those who received tutoring performed worse than their peers who did not. However, we know that prior attainment predicts uptake of tutoring, so when we compare those with the same level of prior attainment (column 2; + Prior Attainment), we see that young people who received tutoring appear to perform slightly better than those who were offered but did not receive it (although this is not statistically significant).

When we compare those with similar background demographic characteristics, as well as prior attainment (column 3 of Table 2; + Demographics), this slightly reduces the size of the difference in performance between those who received tutoring and those who did not.

Finally, we also compare young people with the same reported perception of need for catch-up support using their self-reported concerns about having fallen behind during the pandemic, specifically agreement with the statements “I have fallen behind my classmates as a result of the COVID-19 pandemic” and “My progress during Year 11 suffered due to disruption caused by the COVID-19 pandemic”. We do acknowledge risks that their response to these statements may have been affected by their performance in their GCSE Teacher Assessed Grades,²⁰ which could exaggerate our estimated differences.

However, taking into account the difference reported when also comparing those with the same responses to these questions (column 4 of Table 2; + Catch Up Concerns), which is a difference equivalent to a one-sixth of a grade improvement in each of their best eight GCSEs, as well as the preceding results, we think that this is encouraging evidence that there is an improvement in pupils’ Teacher Assessed Grades associated with receiving tutoring, compared to similar pupils who chose not to take up this offer.

Conclusions and policy implications

In this briefing, we highlight a number of important findings about young people’s experiences of assessment during the COVID-19 pandemic, particularly for our cohort who received teacher assessed GCSEs, rather than traditional externally-set and –marked examinations.

Almost half of pupils reported that they had performed as expected in their teacher assessed grade, but pupils from independent schools were much more likely to think they had done better than expected (and more less likely to think they had done worse than expected) than those in state schools. A quarter of young people reported that they planned to retake GCSEs, and were especially likely to report this if they felt they had done worse than expected in their teacher assessed grades. Around a third of pupils reported feeling that teachers were biased against certain groups in their teacher assessments, and this was most likely among those who felt they had done worse than expected (even after adjusting for the actual differences in their performance). Nevertheless, perhaps surprisingly, given the proportion of pupils agreeing with negative statements about the TAGs process, only a third said they would have preferred to have taken exams in the normal way.

Without further context, which may yet come in the shape of A-level results this year, we are unable to provide direct evidence of whether teacher assessment relatively advantaged or disadvantaged groups compared to what they would have received from traditional exams. Inequalities between groups based on demographic characteristics and socio-economic status are evident, but further work will be needed to understand to what extent these represent pre-pandemic inequalities, differences attributable to variation in pandemic experiences, and directly to the teacher assessed grades.

One-to-one or small group tutoring as catch up provision was most likely to be offered to those from less advantaged backgrounds and those who had lower prior attainment. Boys were more likely to be offered tutoring but, as they were less likely to take it up, there was no gender difference in reported receipt of tutoring. Those who received one-to-one and small group tutoring received slightly higher teacher assessed GCSE grades (after adjusting for attainment at age 11) than their peers who were offered this tutoring but did not take it up, providing encouraging evidence as to the effectiveness of this catch up provision to those who received it. However, this should be tempered by the fact that only just over a quarter of the sample reported that they have received one-to-one or small group tutoring, making it unlikely that this will have made a big dent in addressing learning lost at the cohort level.

As with many findings relating to the COVID-19 pandemic,²¹ our results highlight the substantial differences in experience between those from different backgrounds and the large implications for educational and wider opportunities stemming from these. Educational providers and employers should be mindful of young people’s experiences during the pandemic – and in their wider contexts more generally – when considering qualifications in the coming years, seeking to understand if relative differences in performance are explained by differences in these experiences rather than underlying potential when considering applications.

Efforts to support young people to catch up potential lost learning through one-to-one and small group tutoring do appear to be associated with better performance in GCSE TAGs, compared to the performance of similar individuals who were offered tutoring but did not take it up. This is encouraging for continued use of this method to support those placed at a disadvantage by their circumstances, whether or not this is related to the COVID-19 pandemic.

About The COVID Social Mobility and Opportunities (COSMO) study

The COVID Social Mobility & Opportunities (COSMO) study is a new national cohort study generating high-quality evidence about how the COVID-19 pandemic has affected socio-economic inequalities in life chances, both in terms of short- and long-term effects on education, wellbeing, and career outcomes. A representative sample of young people in England who were in Year 11 in the 2020/2021 academic year were invited to take part in the survey, with the aim of following them as they progress through the final stages of education and into the labour market. A sample of more than 13,000 cohort members was recruited in Wave 1.

This work was supported by UK Research and Innovation Economic and Social Research Council as part of their COVID-19 response fund [grant number ES/W001756/1]. COSMO is a collaboration between the UCL Centre for Education Policy & Equalising Opportunities (CEPEO), the Sutton Trust, and the UCL Centre for Longitudinal Studies (CLS). Our principal fieldwork partner is Kantar Public.

Researchers can access data from Wave 1 of the study through the UK Data Service.²²

Citing this briefing

Anders, J., Cullinane, C., De Gennaro, A., Early, E., Holt-White, E., Montacute, R., Shao, X., & Yarde, J., (2023). Wave 1 Initial Findings – Attainment and Assessment. COVID Social Mobility & Opportunities study (COSMO) Briefing No. 7. London: UCL Centre for Education Policy & Equalising Opportunities & Sutton Trust. Available at: https://cosmostudy.uk/publications/attainment-and-assessment

Sample and methods

The data for this briefing come from Wave 1 of the COVID Social Mobility & Opportunities (COSMO) study. COSMO is based on a probability sample drawn from the Department for Education’s National Pupil Database (plus additional recruitment from pupils at private schools), with clustering within schools (for practicality reasons) and oversampling of certain groups using stratification.

Our analysis in this briefing is primarily based on descriptive statistics reporting averages, distributions and differences between groups. Analyses use weights to take into account the over-sampling inherent in the study design, as well as initial non-response by young people and, where relevant, their parents. Differences are only highlighted where these are found to be statistically significant at the p<0.05 level. Any statistical inference testing reported and/or used in such decisions account for clustering and stratification in the study design.

While our full sample of young people has N=12,828 the parents of participants were not as likely to respond, reducing analyses involving parents to at most N=9,330. As noted above, young person and parental non-response have been modelled separately, with different weights to ensure (insofar as is possible) representativeness of our analysis sample to the intended population. Item-level non-response also results in some further variation to the analysis sample, which is minimised within analyses to ensure consistency. Analyses of some groups, for example those who attended special schools or who identify as non-binary/in another way, have not been able to reported due to small sample sizes. Some questions were only asked to half of the sample (the other half were asked alternative questions); as this allocation was performed at random the resulting missing data should be missing completely at random.

Aspects of the analysis use administrative data from the Department for Education (DfE)’s National Pupil Database (NPD), where consent was gained for this linkage (73% of young people), with additional weighting carried out to ensure (insofar as is possible) representativeness of analysis using linked administrative data. This work was produced using statistical data from the DfE processed in the Office for National Statistics’ (ONS) Secure Research Service (SRS).

The use of the DfE statistical data in this work does not imply the endorsement of the DfE or ONS in relation to the interpretation or analysis of the statistical data. This work uses research datasets, which may not exactly reproduce National Statistics aggregates. The DfE did not produce Progress 8 scores for pupils during the pandemic, since these were not judged to provide a reliable guide to pupil progress for school accountability purposes, given that the outcomes are based on Teacher Assessed Grades (TAGs). As such, while noting these reliability concerns, we estimated our own progress score for pupils in our sample by estimating the residual from a linear regression model of a pupil’s Attainment 8 score (non-response and linking consent weights applied) on their reading; grammar, punctuation and spelling; and maths point scores from age 11/end of Key Stage 2 National Curriculum tests. Although Progress 8 scores were not provided, predicted Attainment 8 scores based on performance at age 11 were provided, so we also constructed progress scores based on a simple difference between these predictions and the realised Attainment 8 scores provided. This correlated 0.73 with the difference between the predicted and realised Attainment 8 scores provided (unweighted count for analysis of 8,157).