[Closed] Who can do data visualisation - Exam Grade Demotion vs Average Earnings per area

Assuming you have postal codes for each school. And a relative increase/decrease value.

Excel Power Maps might do it for you and possibly wouldn’t require any fancy visualisation tools.

Haven't seen anything like that yet, but I've asked eduTwitter and will pass on anything that comes up.

Honestly not a troll question below:

I understand that this year's exams and the results are a S**T storm and I sympathise with all those caught up
I also know that there is a narrative that economically disadvantaged areas have been hit harder than the more affluent areas

The thing I am struggling with is how would the examiners know where a paper comes from? I guess it could happen at a level where someone has all the papers and knows which schools (and the area that they come from)

A bit confused if someone could help!

Cheers
Alastair

The social economic status (SES) for every centre is already in the model. The results and cross checking show that those with lowest SES had better performance but this was very close to historic data.

The noise you are hearing is due to the following facts. Those predicted to have lower grades are most likely to underperform. Often by some margin. Such pupils are more likely to be in poorer performing schools in areas of greater deprivation. But the SES is not the driver of the downgrading, it is a consequence of the methodology of matching to past achievements. The converse is that higher grades are better predicted, and private schools achieve higher grades so their marks are matched to a higher distribution. That will tend to mark some up more than state schools.

There ARE no papers this year...

A crude overview (STW's resident dataists could explain better EDIT - I hope my cross-posted layman's description doesn't cut across his tech explanation)...

Students were ranked by their teachers in each subject in each school. This rank was applied to a formula (the mythical 'algorithm') based on each school's prior results and students' historical attainment. Teachers were also asked to provide an actual grade, but in many cases this is not factored in to the rest of the process for reasons...

So if you are an exceptional (or even just hard working) student in a school with historically weak results you are likely to be downgraded. If you are a mediocre student in a school with historically good results you are likely to benefit. There is a correlation between deprivation and historic results...

All compounded by a 'quirk' of the process where small courses (in terms of student numbers) do not have the algorithm fully applied (or even applied at all). Guess whether poor state schools or wealthy fee-paying schools are more likely to have small 'niche' courses running?

I've been listening to the TV/Radio and reading the papers about all the students who have been deprived their choice of Uni or failed to get on some high demand course because their grades where 'wrong' or 'unjustified' or 'illegally downgraded'. I can sympathise with every single individual, but not one person suggested a better way of doing it.

If we took teachers assessments then there would have been a grade inflation of more than 10% (just see the Scottish system) which is unrealistic. As it is the 'model' has a 2.3% grade inflation, in line with previous years. There has been unsubstantiated comments that some schools predicted that the entire year group would get least As across the board - when that have never happened before. Is there any reason to suggest that this years exam candidates are that much better than last years?

There will be individual cases where the model does not work. In particular an exceptional scholar in a poor performing school (and equally a poor performer in a over achieving school). And that is what is being reported about.

The model does not and cannot deal with the fact that the difference between a good school and a poor performing school can often be a two or three grade difference. There is a link between poor performing schools and social deprivation. All the model does is to try to map the standard achievements from a school onto this year cohort.

It may not be the best way - but so far no one has come up with a better method. Going through every student individually won't solve it. Award the teachers assessment of what a student could achieve would lead to a massive grade inflation. I feel sorry for next's years Scottish exam candidates since they are not going to match this year.

So what is a better way of dealing with the issue this year? Changing the entire exam system is not an option - but needs to happen long term. Fixing social deprivation would be the best fix, but I am not sure that this can be accomplished in our lifetime. So how would you award the grades.

(Of course for every one who does not get into the University of choice, many do - we don't hear about those. I also suspect that Uni's may need to increase the numbers and will take lower grades. And after Uni how many businesses use A level results?)

Mischon de Reya are on the case

My sister in law is head of maths in a small comprehensive in the North West. Her A-level group was 8 students and she had a realistic spread from A* to E of her students (the A* ended with an A, but she said that this one was an exceptional student, amongst the best agreed ever taught). She was devastated on Thursday as she had 70% downgraded, a couple by 3 grades. Her predictions over more than 20 years teaching for expected grades have been within 10%, usually less, so she is feeling absolutely rotten for her students.
I see that ourGav is already started to row back on things, so who knows what will happen.

Obviously those bright/talented children who feel they've been assigned the wrong grades have the option to actually take the exam in the autumn - that's always been an option.

I also suspect that Uni’s may need to increase the numbers and will take lower grades.

That doesn't really seem to be the case at the moment.

As things stand, the least worst solution is probably to allow the CAGs to stand (grade inflation is a bit of a red herring as this year's results will not be used for official league tables etc.), and to (temporarily?) remove the cap on university places that this government introduced. I understand this is what at least France have done so there is precedent if we need it.

but not one person suggested a better way of doing it.

I did in the other thread 🙂

Use past school prediction accuracy rather than past pupil performance as the means of modulating the current 2020 predictions. Schools that year-on-year make accurate pupil predictions are likely to do the same again. Schools that are optimistic will be downgraded accordingly. Also forget individual grades and give a "three best A'Level" score, which will have greater accuracy than three individual grades.

The only issue there is the lack of any consistent, centralised way of collating prior predicted grades. Predictions for UCAS are just as problematic as CAGs, and for the GCSEs this week there is no recording of predicted grades which are used internally in schools/trusts but not 'published'.

If we took teachers assessments then there would have been a grade inflation of more than 10% (just see the Scottish system) which is unrealistic

But why does it matter? If pupils got their predicted or mock results they'd get into their preferred uni and courses, they'd then pass of fail these courses.

But why does it matter? If pupils got their predicted or mock results they’d get into their preferred uni and courses, they’d then pass of fail these courses.

So why does it matter if they got a lower grade than THEY thought they would get. The universities know this has happened and can act accordingly.

It is ultimately all just a guess and I think the model used was okay given what options there were. I wish the system was around when I did my a levels as I was a poor student in a very good school!

Obviously those bright/talented children who feel they’ve been assigned the wrong grades have the option to actually take the exam in the autumn – that’s always been an option.

So why does it matter if they got a lower grade than THEY thought they would get. The universities know this has happened and can act accordingly.

They were predicted/told these grades, they didnt make them up.

But the universities aren't acting accordingly, from everything I have seen students had offers conditional on grades and those that didn't get those weren't accepted. Just give them the predicted grades, it has no knock on impact IMO

Depends what you think is the bigger problem - that kids this year get higher grades than they should, or that they get lower grades than they should. I'd say that the bigger unfairness is to downgrade kids, so I repectfully disagree with TiRed 🙂 that statistics is the answer to this problem

So if you are an exceptional (or even just hard working) student in a school with historically weak results you are likely to be downgraded.

There are other stochastic effects. I know of one college that only enters the very best maths students for further maths. This means that the further maths students all get A or A*. I haven't looked at the algorithm but it appears that it has spread the grades over the same curve as other subjects so some of the very best maths students in the college have been awarded Es, which is clearly bobbins. So talking about overall statistics and trends regarding inflation masks this kind of event.

If the Further Maths cohort was more than 15 students, they will have had the algorithm applied and produce the impact you describe.

It's nonsense.

There seems to be a non linearity here. The headteacher of Marlow Grammar said that the results were 10% lower than their previous lowest results. Which is poor but not exceptional. However she also said that the proportion of pupils who failed to get their first choice university place had gone from about 5% to more that 30% (she gave numbers for the 225 total). That’s likely to be the issue. A small change in prediction leads to a large consequence. Welcome to non linearity

It is ultimately all just a guess

Technically, it's an estimate, not a guess. All grades are an estimate of some sort. You're trying to make a general prediction about ability from a limited sample of observations. If you have a longer test, you will get a smaller measurement error and a better prediction. If your test is a better representation of the real world, you will get a better prediction of real world performance. If you have multiple human judges instead of a single judge, you will get better predictions.

One problem is that people don't realize how large the error component of most assessments are, they assume that the test score is somehow real rather than being an estimate with confidence intervals. If you retest a bunch of humans using a parallel test written to the same specifications, you'll find that there is a surprising amount of variation in the scores. This is literally one way to determine the confidence interval of a test. Changing up or down one grade level is pretty normal (i.e. the confidence interval is of the same order of magnitude as the intervals between grade levels), so you will also get a lot of students changing two grade levels if you have a large sample. That's in a normal testing (or public opinion survey) situation.

In this case, the model seems to be based on some assumptions about the stability of average grades across large groups. The problem is that this will never give very precise estimates of individuals, even though it should give very good predictions of large groups. That's not to say there aren't better or worse ways of modeling it, but people are very unrealistic about how precise assessments like these can be.

For example, if you want to know the weight of a person and the only information you have about them is that they are a British male, using the modal weight (i.e. the most commonly reported weight) would be defensible, even though you would have very little confidence in the precision of the estimate. That's because you really have very little information on which to build a model. If you knew that the person was a male firefighter aged 35 years old who ran 20 miles per week, you would be able to make a better estimate, but you still would not be surprised if your estimate was off by a huge amount.

If we took teachers assessments then there would have been a grade inflation of more than 10% (just see the Scottish system) which is unrealistic.

Like the justice system and the saying "better a guilty man go free than an innocent man be condemmed".

I think that would have been a fairer outcome. What damage is really done by someone getting a higher grade than perhapse they would have. The teacher and mock exam probably backed it up anyway. It's perhapse not fair on a kid that got a B last year when they were predicted an A, but in later life no one will really care.

Whereas the kid that got a B this year after being predicted an A has now been denied their place at uni.

Obviously those bright/talented children who feel they’ve been assigned the wrong grades have the option to actually take the exam in the autumn – that’s always been an option.

Whilst most kids have safety nets in the form of stable families at 18. Not all do. Some might well have been told they're out. And the rest, thats a years working life they've lost, hardly fair. And if they've already been told they havent got their uni place this year regardless of the appeal. That'll be in competition next year with that cohort, plus all those who have defered a year due to COVID, so next year may be even more competitive.

The gobermint missed a real trick here by not accepting the teachers predictions for grades. Then they could have blamed any problems on the teachers! Oh wait, they do that anyway.

What damage is really done by someone getting a higher grade than perhapse they would have

They go to uni after all, saddle themselves with a couple of years of student debt before finally throwing in the towel and dropping out with nothing?

Not saying being downgraded is better, but unrealistic upgrading has its consequences too.

What damage is really done by someone getting a higher grade than perhapse they would have

They get a university seat or job that someone else would have got. Problem is, you are assigning scarce resources on the basis of supposed merit, but the measurement error is actually very large. This means that a lot of the decisions are actually just coin tosses. The very best students are far enough above the cut points that they aren't affected, the very lowest students are far enough below that they aren't affected, but acceptance decisions for average students are actually as much a matter of luck as merit. And that's for normal tests. This debacle just makes it obvious how much a matter of luck it is.

Welsh Government have folded under the pressure.

They go to uni after all, saddle themselves with a couple of years of student debt before finally throwing in the towel and dropping out with nothing?

I'd argue thats not true.

Is someone who got an A in their mock, their teacher assessed them as an A and then had a bad day in the final exam going to be a better Doctor, Dentist, Engineer, Vet or Geography graduate than someone who fluked the exam?

TBH I was the one who had a nightmare in my Chemistry final exam at that age which meant I ended up with AAAB and my second choice uni course, which in retrospect was a mistake, I should have taken the AAA course at a different uni. So my oppinion that exams are the worst way to examine anyway is a bit biased.

The very best students are far enough above the cut points that they aren’t affected

I'd dispute that too, if you have a Physics class of 8-10 kids at A level, the bell curve might only allow for 1 A. How is that fair on the two equally bright kids that both want to study Engineering?

I’d dispute that too, if you have a Physics class of 8-10 kids at A level, the bell curve might only allow for 1 A

The Bell curve isn't what I'm talking about. I'm talking about confidence intervals / measurement error. Completely different concepts.