small cohort of any subject

What’s the definition of small and do you know if there are levels of interference relative to size?

AA, so you got an inside track on what’s going on or you just being unpleasent and abusive for fun. It’s not my opinion, it’s what I’ve been told by people who are actually involved, and I don’t mean some random chip on their shoulder teacher.

I was agreeing with you, the method sounds like a load of old bollocks. Maybe its you that needs to examine their shoulder for potato based food options.

Major study using UCAS data showed for 2013-15 grades were over predicted in 75% of cases!

UCAS predicted grades are very different, firstly they are done at the start of year 13 only halfway through the course, and secondly no one takes them seriously, Uni’s or schools, they are done by negotiation with pupils.

Fair enough anagallis, didn’t fully appreciate that difference. Discussing with a maths teacher I’m not sure that there is so much difference, but I think this just shows the different approaches taken by schools and even departments. Overall very tough to do anything other than moderate as a distribution which will always leave some individuals adversely affected ☹️

Discussing with a maths teacher I’m not sure that there is so much difference, but I think this just shows the different approaches taken by schools and even departments.

Every school I’ve worked in, every year science departments get moaned at for not being good at predicting grades. It would appear subjects like maths are easier, if you can do A,B and C you can do it in the exam. Science is different, you need the skills AND you need to have learnt the content, you cannot predict who will learn the content well and which bit will be tested in the exam.

Best outcome so far is that these are all voters next time around. As are all the GCSE’ers coming up. And their parents.

The stats say 60% of grades are unmoderated, so I assume there are plenty who have been awarded their expectations and we do only hear the anomalies. Some of these will be people who will have actually exceeded their true ability, if their teachers were ‘optimistic’ in their chances particularly at grade boundaries. Which I get, I’m not a teacher but if I had a kid who was an upper 6 but on a good day and with the right questions coming up could be a 7 then I think you would say so, in the same way as you wouldn’t say that on a bad day / wrong questions they could easily be a 5. Is anyone prepared to say they did get their fair, or even better than fair result to counter the headlines? (I know that wrong is wrong in either direction, I also know life is distinctly unfair at times so on the rare day you’re the pigeon rather than the statue, just keep cooing)

I wonder now on the GCSE’s. More exams, more numbers to base the stats on, will that lead to greater accuracy in moderation? Are Gav and co burning the candle at both ends to rewrite the algorithm to avoid another clusterf*ck or do they not GAS? I’ve got a chip in next week’s game, the poster child of a high achiever in a below average school (although not ‘poor’ in comparison to the country as a whole) – will they be next week’s headlines?

The one report I read that really alarmed me was about poor schools with high achieving pupils. Which does and can happen. I read somewhere that they work out how often that school has a high achiever and then if another is forecasted to come along sooner than that then they have been downgraded, in some cases severely, because ‘on average’ they only come along once every X years and you had one less than X years ago! I’m going to make a whatiffery here but this could well be a future genius from an inner city school that now doesn’t get to go to a top University – exactly who the system should be nurturing.

I also read that the equality commission wants a look at how the algorithm has been applied. That could be quite an eye opener too. In fact, maybe only ‘even better if’ it turns out that one of Dom’s chums was commissioned to write it and Dom’s told him to make sure the Tory voter kids all get what they want and screw the poor….. but that would be a dream! But, makes you think….how did we get to a world where that thought even crosses my mind?

Teachers were asked to predict what students could get and so will always over predict. As what you could get and what you do get are different things.

Looking at the system they’ve apparently used, it seems even more brutal.

Asking teachers to rank all their pupils from first to last, then applying a grade along a predetermined curve based on historical grade achievement in that school. So basically totally disregarding the grade offered by the teacher.

All in the name of preventing ‘grade inflation’, simultaneously trashing the dreams of high-achieving pupils while less able pupils from selected schools who have been uplifted are on the phone to clearing taking their places on degree courses which they wouldn’t get near on a normal year.

Absolute cluster, and those who signed off on such a flawed, dehumanising system should be out of a job.

So basically totally disregarding the grade offered by the teacher

This. All they really wanted from us was the rank order, but only asking for that would have exposed the (flaws in the) system too early so we had to provide the (meaningless in most cases) grades too.

In other news, LOOK! MORE COUNTRIES ON THE QUARANTINE LIST!

The stories coming out of schools across the country are heartbreaking.

But LOOK! SOFT PLAY CENTRES, CASINOS AND BEAUTY SALONS ARE REOPENING!

This is interesting: https://news.sky.com/story/a-levels-exam-regulator-ignored-expert-help-after-statisticians-wouldnt-sign-non-disclosure-agreements-12049289

I mean why would you want to take advantage of expert advice?

Why would you not sign an NDA?

I’ve read the UCAS paper on grade predictions and it’s truly eye-opening. The typical (mean) difference between predicted grade and achieved grade is 1.5. The distribution shows that pupils predicted A/A* are much more likely to achieve this than those predicted C/D. Hence if you were predicted lower grades, then you were more likely to be downgraded based on past achievement patterns. Which schools typically achieve lower grades? Independent?

Even if the algorithm is 95% predictive (and teachers are only 40% btw), that leaves 5% of 740k exams not predicted well, so possibly 37,000 disgruntled pupils taking to Twitter/FB. The laws of big numbers mean that even a well-performing algorithm will generate a lot of outliers. Those are the ones that have a reason for appeal. Some schools seem to be disproportionately affected. But again the numbers are huge here, so outliers will occur.

I’d like to see how well the algorithm performed on the 2019 training and test datasets before being used on the 2020 data. If this has done anything, it has unveiled to the public some stark realities in education, in the same way that COVID19 has done for the NHS.

I’d like to see how well the algorithm performed on the 2019 training and test datasets before being used on the 2020 data.

Pretty sure that’s available on the DfE site/blog. I’ve not looked myself but seen a couple of whispers on eduTwitter that it didn’t work too well in ‘predicting’ 2019 actuals…

Making no claims about the accuracy or significance of these comments (my data skills are OK for teaching/Head of Year purposes but nothing compared to TiRed et al).

There’s a really curious and slightly under-explained table in the technical notes, where they discuss testing the model by back-casting the 2019 results. And they find that, even in history, the easiest subject to forecast, they get one in three grades wrong. pic.twitter.com/IKNVfyQNMG

— Chris Cook (@xtophercook) August 13, 2020

Ouch – -I don’t think 67% is particularly high-performing. I’ll see if I can give it a read – not found it yet. Normally one would take 2/3 of the 2019 data to create the algorithm and then test it on the remainder. There is an issue I can see, which is if the algorithm needs within-school ranking then you only have a system that works at school level.

Prediction within one grade is pretty good – acceptance by Universities would do well to follow that paradigm too.

Prediction within one grade is pretty good – acceptance by Universities would do well to follow that paradigm too.

Seems that the relatively small number of grades available at A level (6) wouldn’t help this either? When we get to GCSEs next week dropping from an 8 to a 7 for example will be disappointing but not critical (although those students dropping from 4 to 3 will be a huge issue IMO), whereas going from a B to a C at A Level can have massive implications.

It’s the real world effect rather than the algorithm itself that’s the issue.

martinhutch
Subscriber

Asking teachers to rank all their pupils from first to last, then applying a grade along a predetermined curve based on historical grade achievement in that school. So basically totally disregarding the grade offered by the teacher.

See, that approach is basically sound, if you apply it to the entire cohort against an averaged curve from the previous few years- with enough students. But you can’t apply it to small groups.

(it reminds me so much of working for bank of scotland, when I was told “You should have received the top grading and the top bonus this year, but the calibration process means that for every grade above “average” we have to give someone else a grade below, to make it fair. Yes, even in your 6 person team. So you’re getting an average grade and so is everyone else, that’s fair because it’s average, and you’re not getting a bonus and that’s fair too because neither is anyone else”)

Well no students from Eton were downgraded so I don’t see what anyone’s complaining about – system worked as intended. Job done.

Didn’t read the whole thread but…
https://www.theguardian.com/education/2020/aug/14/punishment-by-statistics-the-father-who-foresaw-a-level-algorithm-flaws?CMP=Share_AndroidApp_News_Feed&fbclid=IwAR3nOD7pUnm6oDX3E6oA1PKoww0QZehCHSzobMtGXw3pspCOijEvY_RXGrA

Was thinking about the algorithm more last night and can see where it failed. Using teacher predicted grades is a reasonable starting point. The UCAS data shows that high achieving students are often under predicted, but low achieving are over predicted. Also predicting a sum of three numbers has better precision that predicting one number (as over and under will tend to cancel). To account for school differences I would have made one change.

Rather than match previous school achievements and their grade distribution, I would use previous predicted grade accuracy instead. Schools that historically achieved high precision would have their grades upheld. Schools that Have been optimistic would have their grades downgraded. Schools with no past history in a subject can be adjusted by the mean across all other subjects and new schools would take the grades as is.

Then I would just give the number of UCAS points 6=A* and 1=E. the data is unequivocal. Look at the difference between achieved and predicted points in the graph below.

Source. https://www.ucu.org.uk/media/8409/Predicted-grades-accuracy-and-impact-Dec-16/pdf/Predicted_grades_report_Dec2016.pdf

Much better solution. Most teachers would be wary of overpredicting because you would look such a lemon. In terms of appeals against results, the league table is the like status table, Eton with the most complaints and so on down. Plus they do Cambridge International exams (easier) which are not allowed in the state sector.

Seems obvious now someone else has suggested it.

My daughter was downgraded in one subject, as were all her contemporaries she spoke to (some in multiple subjects). She has her place at Uni confirmed, her grades were always likely to be much higher than her offer, so we weren’t worried, but most others have missed out. One of the things they showed us 2 years ago when we were checking out sixth forms was how accurate their predictions were… as an indicator that their staff were fully involved with all the kids, and get to understand what they are capable of very quickly and completely. At the last parents evening they made it clear that they were expecting big things from the upcoming exams, and the year group as a whole were flying. It’s a very successful, and very large, urban sixth form college, taking pupils from a very large geographic area, which seem to be exactly the kind of institution to have fallen foul of the “moderation” across the country.

The UCAS paper is a very good analysis. I’ve also read the Ofqual 300 page methodology report, which has basically tried to maintain comparability to past years rather than provide a scoring for university choice. And obvious is only so after it’s been suggested 🙂 . I have nothing in this. Mine finished their a levels two years ago.

You have to have some grading of school performance, but the less obvious one May be more accurate. As is prediction of a sum rather than three individual values.

She has her place at Uni confirmed,

Pleased to hear it. The algorithm used is really only good to within one grade, so universities should be more flexible in admission I expect there are a few phone calls between UCAS and govt and unis.

From her contemporaries… it seems the Unis are not being flexible at all… which surprised me. But then, they have a cap on admissions this year, yet have no idea what’s coming up next… (that’s what happens when government makes up policy on the hoof). If they give lower offers to those that have been moderated down, but then lots of other applicants get their grades adjusted under whatever the appeal process becomes, they could be stuck in a tight position… over subscribed yet capped by law.

“Brilliant pupils from economically disadvantaged schools have seen their dreams dashed – while others from wealthy backgrounds saw their predicted grades confirmed,” reads the petition from Oxford alumni.“It cannot be right that bright, hard-working young people from poorer backgrounds have been denied their chance to overcome odds that were already stacked against them.”

amen, pretty disappointed it’s not the default position for all universities

It’s the right move, but at the moment it’s only one Oxford College. Would be the easiest universal way to bypass the mess, and send a message of distrust (that they’ll obviously ignore) to the DfE – ‘your algorithm was misbegotten, and we don’t believe your appeals system will work either’

A lot of research has suggested there’s not a great correlation between A level grades and class of degree, the best indicator used to be the grade in General Studies. In cases where they’ve set no criteria for entry (in the US) it was found students benefited pro-rata in terms of origins and destinations, so what was the point apart from the desire to exclude?
Problem with algorithms is that it can’t cope with a bright student from a comprehensive school. I taught working class students who became lawyers, a judge, an MP, all could well have been marked down and ended up as deskjockeys.
Universities don’t seem to be showing enormous flexibility despite the loss of O/S students because they are worried about the graduate destinations league table. Tail wagging dog.
The system is not ‘robust’, it robbed us.

You simply could not make this up > Ofqual suspends A-level appeal criteria hours after announcing them | Education | The Guardian https://t.co/1lIDyKC3Gl

— Kate Green (@KateGreenSU) August 16, 2020

(EDIT cross posted with kelvin) Shouldn’t be surprised, but Ofqual have now withdrawn their published appeals procedure for “review” just hours after making it public.

I need to stop trying to second guess this…

Of course if Gove hadn’t taken out the course work element of GCSEs, relying instead on a single day’s exam such as his golden youth, then we would be in a much better position to grade this years students.

Private schools do different GCSE’s, which still include course work, but state schools aren’t allowed to do them..

Who killed AS levels?

Having proper end of year 12 grades would have been handy, and not just as regards this mess.

The Oxford example (a single college of) is admirable, but not practical for all universities. Oxbridge are much more selective in who they make an offer to – anyone holding an offer should do well there. Other unis, much less so.

Rumour is that predicted grades are making their way into the OFQUAL appeal process

Ghill in what respect? And which predicted grades, UCAS grades or the centre assessed grades? At the moment there’s no right to appeal the centre assessed grades (UCAS grades tend to be inflated at the best of times), only if there is discrepancy between the awarded grade and the CAG. Even then the school has to provide evidence why the award is wrong, which most already have when they submitted the grades in the first place which sems tk have been generally ignored.

Northwind nailed it in an earlier post, to try and keep things ‘fair’ they tried to issue the same number of grades at each level as last year, trouble was they issued the high ones to schools with small cohorts where they didnt have enough data to properly statistically model the usual attainment, small schools with small classes, a lot of which were independants. They just used the CAGs (which were inflated across the board) and then for big schools they did the proper modelling for the school but reduced the number of high grades to keel the national totals looking right. They used a prior attainment value to fudge this which they dont sem to be able to explain the workjngs behind. Should have done it the other way round, assessed the schools with good data first, they looked at the small schools. The bigger schools are already pulling together data (because they have it too) to mathematically show the process was flawed.

It seems like the institutions involved are taking a step back and allowing the standing ovation to go to Pike Williamson. How he still has a job is pure modern magic.

Suspending the criteria for appeals seems to confirm that they really are just making this up as they go along.

Given that they’ve known this was coming since March, the obvious question it raises is WTF have this lot actually been doing for the last five months? The level of incompetence is staggering, but then you just have to take one look at who’s presently winging it as education secretary…

Chances of them getting this all sorted in time for next weeks GCSE results?

Zero.

Stumpy, see here: https://twitter.com/hensforhire/status/1294761985884725259

No idea if this is real or a mock-up.

I can understand how an algorithm can grade down a very bright student from a poor performing school but how is it that ALL private schools are getting much better results than last year?

They aren’t having the algorithm applied to them.

Oh, ignore the education security… none of this has anything to do with him… look into who developed and applied the algorithm and surrounding rules, and how they are linked to Gove& Cummings… our Gav was just pushed forward to deliver confusing cover noise for their mess.

look into who developed and applied the algorithm and surrounding rules, and how they are linked to Gove& Cummings

Which makes this ‘Leopards Ate My Face’ op-ed all the more remarkable. (Apologies for Mail Link)

https://www.dailymail.co.uk/debate/article-8629107/SARAH-VINE-says-chaos-incompetence-weeks-level-results-final-betrayal.html