https://www.bbc.co.uk/news/amp/uk-scotland-53740588

A squealing U-turn… Nicola’s comments last week suggest a proper cock up at SQA & Scot Gov, and she isn’t happy. (Rightly)

This rather calls into question the current examination system.

I get why they’ve done that, but huge can of worms. If Ofqual don’t do the same then there’s massive potential inequality between Scottish and English /Welsh/Northern Irish students ‘competing’ for places for the autumn. If Ofqual follow suit (I don’t actually think they will because Scotland… but you never know), then students from any schools that did try to game the system get a massive advantage.

Best solution I’ve seen is the suggestion that all 2020 UK certificates show two grades – moderated grade (school awarded grade), and institutions/employers can then make their own mind up.

I am no fan of the SQA, but I thought they did a reasonable job, given an impossible situation. I fear this U-turn, will render this year’s results near worthless.

Nicola’s comments last week suggest a proper cock up at SQA & Scot Gov, and she isn’t happy. (Rightly)

It would seem to me that the responsibility lies with Swinney and ultimately Sturgeon herself.

but huge can of worms

Just effectively devalued my daughters five A’s.

I’d imagine that the Universities and colleges will just up their entrance requirements to compensate for the increased grades.

I think questionable rather than worthless.
If those with the highers can get into the universities they’ve chosen and those with Nat 5s can progress how they want to, then they have done their jobs.

Edit Maybe Junior’s Computing D will be upgraded to go with the 6 As he got 🤞😆

but huge can of worms

Just effectively devalued my daughters five A’s.

I’d imagine that the Universities and colleges will just up their entrance requirements to compensate for the increased grades.

I did not congratulate your daughter before – but her results were very impressive! They were for Higher iirc, perhaps the Unis will look to Advanced Highers now, more than giving unconditionals based on Highers in S5?

I think questionable rather than worthless.

Yes, sorry – worthless was a rather large exaggeration. However, I bet for a long time children will get a “but that was only because of COVID”. Especially from other children just a little older or younger.

Perchypanther increased grades are not being rescinded, so your daughters 5 A’s still stand. Extra places are also being provided by colleges and universities to compensate

It would seem to me that the responsibility lies with Swinney and ultimately Sturgeon herself.

Indeed. I know a few of the SQA, Scot.Gov. Education dept and Education Scotland people. There isn’t as many as you might think, and they all work pretty directly with the minster for Education. They’re all being very tight lipped having asked one today about it…

I agree with Perchy – this upping of grades devalues those who were awarded grades. Our lad got two C’s at Higher and A at Advanced Higher – and lives in a good postcode. He’s now going to have 125000 others with similar grades which may or may not have been awarded fairly – a reverse but similar situation to before…

There still is anomalies here – schools where this year’s grades at Prelim and put forward to SQA were way above average for the past 5 years for a similar cohort in that school. Will SQA reinstate that anomaly of no fails and much higher grades? Then next year the school is back to ‘normal’.

The relative value of the teacher vs revised grades relies on the revision method being fair and trustworthy. From what I can gather, the SQA’s process was designed to favour a notion of ‘robustness’ over fairness and I think there’s a good case to say that it didn’t provide an accurate picture of the results. Can we trust that a ‘revised’ grade A is any more valid than the teacher’s grade for a randomly selected student? I don’t think we can.

FWIW, I’m reasonably impressed with how it’s been handled. ScotGov tried their best with a really tricky situation and when they realised they’d messed up they held their hands up and did their best to fix it. I’ll reserve full judgement on this until they’ve reviewed the exam system.

Can we now appeal every previous year where exam results did not match the teachers assessment?

(Asking for a friend daughter.)

Very interesting to see what happens in GCSE’s in England now. They have a week to review their algorithm, so i wonder if they will come out with a closer match from the start or do another shitstorm and backtrack.

I can’t help feeling results that are >10% higher then previous years can’t be correct, that doesn’t happen.

But my daughter is almost the poster child for the disadvantaged in this scenario. She has had good assessments throughout, good mocks, and we assume good predictions. However the school’s overall results have been deteriorating since she joined, but it has just in the last 18mo started to quickly reverse that trend with a massive change to SLT, teachers and curriculum. OK, 18mo is a short time to have an effect but for those kids that got on board and responded it has. She was a kid who always worked hard, supported by us, her parents to close gaps that opened as a result of poor teaching, and now we add good teaching to the mix the standard of work, and understanding and marking has improved beyond recognition.

So to assume that her and classmates like her have had their grades boosted and now need pulling back because the school under a previous regime wasn’t very good is very unfair. Particularly if you also assume ‘good’ schools wouldn’t have boosted their grades so to reduce the pass rates you pull the lower performing schools down further.

But who goes to good schools…. and who are they likely to vote for. So will Gav GAS about the good pupils from less well off families, as much as the SNP did?

Can we trust that a ‘revised’ grade A is any more valid than the teacher’s grade for a randomly selected student? I don’t think we can.

Well what we do know is that significantly more people will have got higher grades than previous years. I mean it’s a terrible coincidence that this year’s students were so much better than other years just when there weren’t any exams to prove it.
Just give everyone a prize for taking part.

There’s been a lot of outrage in the media over this and a U-turn. But I’m not sure what any other solution should have been. It’s a very nice statistical problem: You have multiple predictors (subject, gender, age, past performance, past school performance, location, eye colour…) and you have outcome – grade. You also have staggering amounts of data to build a model for prediction. The law of large numbers is really working in your favour here. Any algorithm will have been rigorously tested for performance on previous years data, including a training set and a validation set. One doesn’t just make these things up on the fly! Only then is it used to make predictions for the new data. The issue is what happens when this year’s predicted grades look so different to past years (and they will have been compared). Of course there will be a discrepancy from the past predictions because statistical algorithms predict within the range of observed data. It wold be nice to see the validation work done to demonstrate that the model was fit for purpose. But then nobody cares about details.

What significant event may have led teachers to increase their predicted grades in 2020? It’s more likely to be a failure of teachers to be consistent than algorithms to make poor predictions.

Well done to those who worked hard and got what they needed. Mine are both past this stage but one is off to a self-funded course in Ireland in a month. It is likely there will also be more places due to a collapse in foreign students. Roll on England and Wales next week.

Well what we do know is that significantly more people will have got higher grades than previous years. I mean it’s a terrible coincidence that this year’s students were so much better than other years just when there weren’t any exams to prove it.

Which actually calls into question the competence and/or honesty of the teachers.

I’m going to take a punt and bet that the First Minister and her Education Secretary knew about the revised grades before they were published to the public. They were good enough then. Published them and suddenly panicked and sadly will lead to the perception that all grades have been inflated – regardless. From an academic point of view, can someone in the Scottish Government please explain to me why this years student body is so much better than last years.

I feel sorry for the students and their parents who have been used as a political football (although given the First Minsters current disgust of footballers this may not be an apt phrase). Perhaps a better solution would be to invite students who were unhappy with their grades to sit the exams later and then take the best of two. I suspect that logistically may not be possible.

From an academic point of view, can someone in the Scottish Government please explain to me why this years student body is so much better than last years.

You mean better than every year?

Like excess mortality data, what matters is the distribution. It is fair to decide that this year’s results should match a recent distribution. At the population level, nobody would think that inequitable. But the sheer difference in this year’s predicted grades really left them no choice.

The plots on that BBC page are awful – based on the algorithm with teacher predicted grades (assume most other covariates are unchanged from previous years), the Highers had a prediction of 88.8% compared with a recent historic mean of 76.5% and a 10 year range of 74-79%. If this was a health outcome there would be an inquiry. As it is, they settled on 78.9%, which is still at the peak of the ten year range.

Everyone is already disadvantaged because of Covid due to less teaching. If kids got ‘A’s that didn’t ‘deserve’ it we will never know who they are but there will be enough uni spots for everyone so i don’t think it matters at all.

Once they get to uni they either pass or fail and will be judged based on that.

The kids who would have had straight A’s regardless will be against wee timmy who would have failed if he had to actually take an exam. If timmy gets a good grade at uni then why would that be a bad thing?

Of course this is just looking at the uni side of things rather than other oppertunities but i think the same thing applies

Don’t think there was ever any short term solution to this. I’m glad that my kids are old enough to be unaffected.. almost unaffected one of them is a teacher. Maybe it would have been better to let the results stand, after the appeal process has run its course and to address the inequalities in education at their root. Address the effect of poverty in our communities and scrap the tax breaks for private schools

Which actually calls into question the competence and/or honesty of the teachers.

Not so sure (but then I am admittedly bound to say that)…

Submitted grades won’t have been what teachers suggested. They will have gone through the lens of Heads of Department and Senior Leadership Teams. I have no doubt that in some schools the grades submitted would have borne precious little resemblance to what individual teachers submitted.

No matter how honest and conscientious teachers were in their grades (I’m pretty confident that very few, if any, of my colleagues in our school – incidentally with a catchment in the biggest deprevation index – tried to game the system or were anything but honest in their grading), I suspect there has been a subconscious element of ‘these grades are going to be brought down and that’s not fair on the kids’ thinking. I know that school leaderships will then have also factored this and whatever glimpses of the exalted algorithm they could get into their revisions.

TLDR. Too little transparency and trust in the system for it ever to have worked smoothly. Questioning the teaching profession is exactly what DomCum wants out of all this – look at his time at the DfE with Gove to see his gameplan.

Best solution I’ve seen is the suggestion that all 2020 UK certificates show two grades – moderated grade (school awarded grade), and institutions/employers can then make their own mind up

The trouble with that is that the moderated grade doesn’t tell you much about the individual’s abilities just whether or not they fell foul of the statistical process.

Which actually calls into question the competence and/or honesty of the teachers

They were estimated grades based on a variety of different info, is it really a surprise that they didn’t match the long term averages? And is it really a surprise that where there was doubt teachers gave pupils the benefit of that doubt? No playing the system, not incompetence or dishonesty but if anything more likely a slight bias towards wanting the best for pupils.

And is it really a surprise that where there was doubt teachers gave pupils the benefit of that doubt?

Which is why it was correct to moderate the results

Which actually calls into question the competence and/or honesty of the teachers.

Please excuse what may sound as bigheadedness:
I’m not sure how good a teacher I am, but I have an uncanny skill for predicting pupils percentages in exams. Also one of my many flaws is that I am too honest…
There are a number of issues with teacher predicted grades. Some are dishonest and will bump everybody up a grade. Others lack the experience to accurately predict the grade – it is much more difficult than one might presume. But many I fear were just ‘too nice’ and gave their pupils too much benefit of the doubt, if in any doubt over a pupil who was borderline between two grades they inevitably went for the higher grade.

Robust evidence is also a minefield. Many teachers give hints to their pupils before prelims, some blatantly so. Class tests are often poorly supervised. At the end of the day, every other year pupils have had the level playing field of a final exam. I’m sitting here this evening feeling that I have failed my pupils somewhat by accurately predicting their results, whilst others I fear (know) inflated theirs.

Which is why it was correct to moderate the results

Which was also inequitable because it was not based on pupil ability.

I don’t think there was a solution to this that could have kept everyone happy. Postponing exams would have been fair (or as fair as exams ever are) but would have thrown all sorts of other things out of kilter.

Personally I think the pitchforks should have been reserved until after the appeals process which would likely have yielded a set of results somewhere between teacher estimates and moderated grades.

And of course one person’s embarrassing climbdown is another’s having the humility to admit you got it wrong.

Which is why it was correct to moderate the results

I agree.

The severity of the moderation and (more importantly) the disadvantaging of a pupil by postcode is the issue for me.

The severity of the moderation and (more importantly) the disadvantaging of a pupil by postcode is the issue for me.

I agree, I’m still unsure how this happened.

I don’t think there was a solution to this that could have kept everyone happy.

Indeed, it was an impossible task. There were pupils who were a ‘guaranteed’ A grade whatever the exam paper had looked like. But many pupils would have managed an A on some days, but a B on perhaps a more difficult or even easier paper. Similarly with other grades, the idea that each pupil was destined to get a certain grade as comes across in the media, is silly. Every year some pupils are disappointed in their results.

This rather calls into question the current examination system.

The current examination system is utter crap and the SQA should be junked

I agree, I’m still unsure how this happened

I think it was an artefact of the process rather than anything more sinister. Amongst other things pupils from disadvantaged areas are more likely to be borderline at a grade and therefore more likely to get downgraded.

Which is why it was correct to moderate the results

The severity of the moderation and (more importantly) the disadvantaging of a pupil by postcode is the issue for me.

This exactly. It seems unfair that ‘lower performing schools’ ie, from worse catchment areas have been assumed to have done most of the grade inflation so instead of lowering the distribution they tilted it to achieve the same average.

The current examination system is utter crap and the SQA should be junked

This.

The SQA are very good at some things but they have grown far beyond their original role as an exam board and now exert an unhealthy influence on Scottish education.

One problem of appeals would be the time it would take to do them all. Check prelim and then paper for suitability. Then any unit assessments, any other pieces of work such as essays/ folio/ assignment if it was done before lockdown. Repeat the process X many thousand times. Been the strangest year in education,hasn’t it?

Amongst other things pupils from disadvantaged areas are more likely to be borderline at a grade and therefore more likely to get downgraded.

Genuine question – how come? Is that because less disadvantaged schools have more A & B grades rather than across the whole range? When we do our first prelims we often get results from 0% through to 97 or 100% – the modal mark often being 0%!

The SQA are very good at some things but they have grown far beyond their original role as an exam board and now exert an unhealthy influence on Scottish education.

Difficult to agree more with this.

Darren McGarvey made a good point about the equity/attainment gap issue, to paraphrase; the inflation of grades that would have resulted from awarding teacher estimates was seen as threatening the integrity of the awards and many disadvantaged kids took the hit on that. But the inflation of grades by the things wealth and privilege can provide does not threaten the integrity of the awards and is in fact seen as normal.

Hmmmm

Not feeling very comfortable with this – and my last act in a English school before starting in a Scottish school is to work the phones and ‘triage’ pupils and parents on A level results day so awaiting a shit storm on Thursday. If Boris does not do the same for A levels there will be holly hell to pay for the tories.

There is always moderation of results – it’s just not as blatant or in the news. My subject is Design Technology. 60% of the grade came from centre assessed coursework. I would assess it and give it a grade. A moderator would a assess it, congratulate me on my assessment and agree with my marking. UMS would then swing in and downgrade about 40% of the results. Every. Single. Year. This is nothing new.

Teachers are optimists!I know – hard to believe isn’t it! On average the forecast grades for a UCAS application are 1.5 grades (over 3 A level- so half a grade per subject for every student) higher than the actual results. Every. Single. Year. If you had a kid’s future in your hands as a subject HoD last May and you were in two minds between two grades – and that will be pretty much every HoD with every student in every subject – only a complete arse would have plumped for the worse of the two options. As I said – teachers are optimists. They also tend to like the kids they work with.

If you are at a successful school where kids get high grades you had nowhere to go with your grade inflation. You could not invent another higher grade above A*. If you worked at a less successful school with traditionally lower results and the kids in front of you were destined for B-Ds, there was plenty of ‘headroom’ to whack a grade or two on top and think sunny thoughts. Sadly less successful schools tend to be in less affluent areas. I very much suspect these school have had their results ‘massaged’ more because teachers had the ability to over estimate more than the high flying school could. Because maths.

Another factor I have not heard in training sessions, webinars, or the eduction press is the concept of bad luck and cockup. In my old job I was looking after one hundred 18yr olds and would hear the tails of woe as the students returned from their exams. Every morning and afternoon there was some kid who would come back from the exam hall saying they had forgotten to turn over the paper and read the last question, had misread a question and answered the wrong way, had taken a punt and not revised a topic that turned out to be a huge part of the paper, that their girlfriend broke up with them the night before and they could not concentrate etc etc. Back luck and cock up massively outweighs good luck and fortune in exam sitting and plays a part in the final grades when looked at holistically. This year no one got that bad luck as they didn’t sit the exams – so if you want the grades to look broadly like results every other year you have to engineer the bad luck back in. As the schools have been asked to rank order the students in their school per subject it stands to reason the bad luck gets given to the student at the bottom of the ranking in each grade boundary.

Whatever – having a system and then dicking about with it when the inevitable downgrading came is good for no one in having any confidence in the system.

But the inflation of grades by the things wealth and privilege can provide does not threaten the integrity of the awards and is in fact seen as normal.

Amen.

Can we now appeal every previous year where exam results did not match the teachers assessment?

(Asking for a friend daughter.)

Yes, but only if we can retrospectively cause a global pandemic that causes her exams to be cancelled.

Darren McGarvey made a good point about the equity/attainment gap issue, to paraphrase; the inflation of grades that would have resulted from awarding teacher estimates was seen as threatening the integrity of the awards and many disadvantaged kids took the hit on that. But the inflation of grades by the things wealth and privilege can provide does not threaten the integrity of the awards and is in fact seen as normal.

Surely they are different things though.

It was pointed out on radio 4 that increasing the numbers who receive good grades is unhelpful for disadvantaged children, because employers etc then rely on ‘soft skills’ like hobbies and clubs etc to determine who should get a job.

NEW: I understand current UK government position is that England will NOT be following Scotland's decision to accept teacher estimates without moderation @theipaper

— Will Hazell (@whazell) August 11, 2020

Nice post Convert.