Big data and algorithms: lessons from class of 2020

Potential discriminatory effects of big data and algorithms

But the combination of big data and algorithms, and their potential discriminatory effects on individuals, gave us a powerful insight into one possible (dystopian) future. Algorithms are increasingly becoming part of our everyday lives, employed by many businesses and increasingly by governments. Used appropriately, they can improve decision-making and increase efficiency. But when they go wrong, they can have a profound adverse effect on individuals, as the class of 2020 has found out.

The A-Level and GCSE results problems affected hundreds of thousands of young people across the UK. When the coronavirus pandemic forced the closure of schools and the cancellation of exams, a new system was needed to allow students who would have been sitting their A-Levels or GCSEs to be graded. The authorities proposed collecting teacher assessments, which would then be moderated centrally to ensure a consistent approach and to prevent so-called ‘grade inflation’. An algorithm was developed which would amend the teacher assessments to ensure that the 2020 grades were broadly comparable with those of previous years, using information including the past performance of schools and colleges.

The algorithm appeared to work perfectly at this macro level, ensuring that broadly the same percentage of students received the top grades as in previous years. But it proved catastrophic for individual students, as around 40% of grades were lowered, and some individuals received grades substantially below their teacher assessments. This seemed to particularly affect high-achieving students in schools which had traditionally performed less well, heightening the appearance of unfairness.

In the face of overwhelming political pressure, the four governments across the UK all decided to revert to teacher assessments. Some of these problems were obvious with hindsight. Because schools had been shut since March, no one had been able to drop out or underperform against expectations, so the algorithm was always going to have to downgrade some students to compensate. And whilst this downgrading rightly reflected the fact that some students would underperform, it felt cruel and unfair to the actual individuals whose grades were lowered.

Automated decision making in business

In a business context, automated decision-making is beginning to be used more widely, especially in recruitment and selection. This creates enormous opportunities for business to improve their efficiency, make better hiring decisions and ultimately increase their profitability. But it comes with risks. Algorithms are not magic. They can only ever be as good as their design and the data that goes into them. Errors in either can lead to unexpected biases being exaggerated and result in more flawed decisions. A considerable amount of work went into getting the exams algorithm right. Still, ultimately it suffered from both a design bias, in that the goal of ensuring fairness at a cohort level led to unfairness at an individual level, and from a lack of robust data, which meant that schools with smaller class sizes appeared to benefit at the expense of larger centres.

Automated decision-making is undoubtedly here to stay, and algorithms are only likely to get more sophisticated. The 2020 exam results scandal doesn’t mean we should give up entirely on automated decision-making. But it should make all businesses pause to consider their fairness and the potential impact on individuals. Otherwise, they could face not only legal challenges but also significant reputational damage.

Read the article in full here.