Algorithms Evolve – How human error, not Artificial Intelligence failed our A Level students
In a previous blog I waxed lyrical about how one benefit of COVID-19 has been the very clear use of data (not always the right data) and analytics/modelling to help drive decision making (not always the right decision!). Then the dominating UK news this week has been the right royal mess up over the A level results and the use of the ‘Algorithm’ to change some people’s final gradings, resulting in around 40% of results coming out lower than predicted.
Exam results time is always stressful and there will always be winners, losers, and surprises in between. The marking of most subjects is certainly going to be more art than science and in a ‘normal’ year there will always be a challenge around consistency of marking. Allowing each student’s teacher to grade them is intrinsically going to impart bias that wouldn’t be there if it was handled by an external examiner. This latest fiasco, with a last minute change in methodology, will mean that there is even more stress and uncertainly as a result. There are students who have initially missed out on places who may now retrospectively get the grades they need and the place has been offered to someone else. I personally think that the Universities with oversubscribed courses should scrap any offers to date and start the process all over again if it’s not too late to do so.
Algorithms are becoming increasingly pervasive in our lives, and many people are not aware of their uses. From insurance quotes based on home postcodes, to selective advertising on social media – our clicks, likes and location are being used by a number of Artificial Intelligence (AI) algorithms to, in theory, serve us appropriate content. It doesn’t always work – the number of times I am promoted hair care products is testimony to that! There are plenty of examples where there are good applications of AI, from insurance costs based on how you drive through data collected via a black box, to uses in healthcare where speedier diagnosis and access to the most appropriate treatment can vastly improve outcomes. The algorithms are only as good as the data they utilise in order to develop their ‘intelligence’ and in a lot of cases require use of inferences that are wide of the mark. Timandra Harkness (@TimandraHarkness) summed it all up quite nicely: “the data used by an algorithm to make a decision about you is largely about other people, rather than you personally”.
I stumbled upon an excellent piece in the Guardian by Dan Davies who succinctly summarised the A level issue by saying: “The problem was fundamentally insoluble, from a mathematical point of view. If the system is dependent on exams to allocate the grades, but it can’t have the exams, then it can’t allocate the grades. No statistical method in the world is going to be able to give you good results if the information you’re looking for is fundamentally not there in the dataset that you’re trying to extract it from.” It’s worth looking into his article in more detail:
Read: This year’s A-level results are a fiasco – but the system was already broken by Dan Davies
Ofqual’s Research and Analysis Findings
Ofqual, the Examinations watchdog have published their findings on their whole approach (be warned, it is a 318-page extravaganza).
I’d love to hear from anyone who has read the report from cover to cover!
Ofqual have to be held accountable for this and I’m sure Gavin Williamson will be getting some tips from Dominic Cummings on self-preservation.
Life is a Roller Coaster
It’s been a roller coaster in our household, with my step-son being one of the Covid-affected students who did not manage to sit his A level finals this year. At the start of it all, when he realised he wouldn’t be sitting exams and they announced his grades would largely be based on his mocks, he was all for re-doing the whole year again. Like many students he didn’t apply himself in the run up to his mocks and was aiming to knuckle down and achieve at least one grade above his predicted grades in his finals. I advised him that as long as he got to a decent University, studying a good course, his A levels wouldn’t really matter in the long run. Granted, employers still look at A level subjects and grades to get an insight into the person (e.g. are they Arts/Maths/Science focused) but grades and subjects aren’t necessarily a guarantee of success in a role – strong marks merely indicate that the candidate is good at study and retaining information for test in exam conditions.
Fast forward to last week’s results day and my step-son was much more philosophical – he was a little disappointed with his final grades but happy in the knowledge that he had been accepted into his first choice University on his first-choice course. I was also happy as he had achieved his aim whilst coming in below what I’d budgeted for when I offered him a financial, grades based, incentive to encourage him to put the effort in.
These are some of my tips for the weeks ahead:
- Don’t Panic – September is only around the corner but there are a lot of people in the same boat. I know that Universities will be bombarded with enquiries so please be patient but persistent in your enquiries. Remember that the admissions folk will be working flat out to ensure they deal with it as quickly and fairly as possible.
- You are not Alone – Every individual is different in terms of what they got, compared to what they expected, and where they will end up. But you are united in the fact that you are the class of COVID-19 and there will be outpourings of empathy for what you have had to experience.
- Universities of the UK Unite – This is easier said than done and I know there has been a lot of work to date, but I do think that most Universities will be overly accommodating to the current situation and do their best to help students who have been unjustly failed by the system. If they aren’t, then imagine what they may be like when they have already collected your fee income?!
- Employers’ Empathy – As someone working in recruitment, I do believe that the class of 2020 will have a special place in peoples’ (HR and hiring managers) hearts. There will be a degree of leniency towards those who went through that year and the grades achieved. If necessary, make sure you stand out in other ways – your passion, knowledge, skills, experience and drive will get you much further than your A level grades.
What next for this class of 2020? I personally think Ofqual should not be seeking to fit a ‘bell shaped curve’ or attainment quota to the distribution of grades and just go all out to award people the grades that their teachers felt they could have achieved. So what if this year more people than ever got higher grades compared to previous years – they deserve it for the disruptions that COVID-19 and this marking fiasco have provided to them.
For me personally, I wait with baited breath as my step-son is already anticipating an improvement on his grades, which will mean additional ‘incentive payments’ due from me. The impact of COVID-19 could have further financial implications for me personally I’m afraid!
I’d love to hear your thoughts… let’s have a heated debate!