Can o’ worms

Following Louise Tickle’s piece in the Guardian, there has been widespread speculation regarding my motivation for wanting 11+ test marks to be made public.  Contrary to some speculation, I am not trying to get my 16 year old son into year 7 at the local grammar. That would be silly.

This is purely about whether information should or should not be disclosed.

Why is this important?

To understand why releasing the information is important requires an understanding of standardisation although ironically, if the information was released everything would be crystal clear.

As any statistician will explain, a standardised score measures the individual relative to the population.  What 11+ providers don’t tell you is, behind the scenes, they interpret population as just one year’s cohort.  Given a set of test marks, a desired pass mark and the proportion needed “pass” in order to fill the school, it takes less than half an hour to ensure your grammar school standard selects the correct number of applicants to the exact child.  To me it is deceptive to refer to something as standardised when it is anything but.  However, given the amount of trouble I’ve had explaining this I made a five-minute film explaining it the issues.

Click on Snow White, all will be revealed and you’ll only be five minutes older.

Public Interest

The Freedom of Information Act operates on the principle that information should be disclosed unless there is a reason not to.  In this case Durham have said that release of this information would be commercially damaging.  The court now need to decided whether overall, it is in the public interest for the information to be released.

There is some confusion about what constitutes public interest.  Whilst the public are probably quite interested in how much Manchester United paid for Cristiano Ronaldo it’s not in the public interest in the same way that it would be useful to understand what exiting the EU actually means.  In the same way, an individual parent knowing why their child did/n’t get offered a place at a given school isn’t in the public interest in the same way that knowing how schools admissions operate in general.  There follows a few reasons why disclosure of this information is in the public interest.

Reasons for Disclosure

If there is a downturn in overall admissions numbers then, like the self-levelling suspension on those funny old Citroëns, grammar school standard is automatically lowered. That’s good for the grammars as they remain full and therefore fully funded but no school is an island. In Kent, which is fully selective, this must come from taking pupils who in other years would have attended the neighbouring secondary moderns.  Those schools are faced with dwindling numbers and loss of funding.  The heads can’t complain about moving goal posts because they don’t even get to see the goal posts (raw marks).

The law states, “Admission authorities must ensure that their arrangements will not disadvantage unfairly, either directly or indirectly, a child from a particular social or racial group.”  GLA have told the Schools Adjudicator that Verbal Reasoning tests are biased against EFL children so it would be in the public interest to understand what weighting each authority gives to each discipline (Maths, Verbal Reasoning, Non-Verbal Reasoning).  That way any particular social or racial group who may feel they are unfairly prejudiced can challenge these arrangements but these groups don’t get to see the goal posts either.

Reading School’s 2015 entry set a cut-off mark of 110.  The standardised results show that one boy scored 110.00 whilst another only 109.99.  To quote the school’s published admissions criteria the second was, “deemed not to benefit from the style of education provided at Reading School.”  It continues, “The following criterion will be used to allocate places as between borderline candidates who cannot be distinguished in terms of academic ability.  With regard to the historic links with the Borough of Reading, eligible boys living nearer to the School will be accorded a higher priority in the allocation of day-boy places.”  This tie-breaker has never been used!!  If a child in Newbury scores 0.01 higher than one in Newtown he’s given the place.  Data from North York, where children sit two tests a week apart show the residual variance (‘noise’) is about 20 – 30% so a 95% confidence limit would put the true scores of these test takers somewhere in a band of about 20 marks, but the results are recorded to two decimal places!  It’s in the public interest to understand that this ‘distinction’ is entirely synthetic so they can question the decisions made by what is an increasing number of unaccountable governing boards of Academies who have been given the power to act as their own admissions authorities.  Admissions law says, ‘Parents should be able to look at a set of arrangements and understand easily how places for that school will be allocated’ but the basis on which schools are currently ‘distinguishing’ between candidates is completely hidden.

When the location of the itinerant goal posts is a closely guarded secret, there isn’t much incentive for the referee to pay attention.  One school ‘standardised’ six candidates, who sat a late test, as an entirely separate population.  As standardisation selects a proportion of the population regardless of absolute attainment, this blunder guaranteed two of these candidates a place.  The candidate originally ranked 100th (for 100 places) in the main test scored 113/150 but was pushed down two places by applicants with raw scores of 87/150 and 76/150.  Standardisation might come across as a bit of a black art but if you check the raw marks, this gaffe stands out like a sore thumb.  This information was not released until the First Tier Tribunal ordered it but if it had been routinely released the parents of the two children who missed out on a place as a result would have had no trouble whatsoever picking up on the mistake.

The 2016 standardised scores for Bucks county included some negative scores.  Durham explained this was the result of, “… adjusting the standard deviation to account for a fixed qualifying score of 121 and a year-on-year increase in the number of candidates.”  If that’s not moving goal posts I don’t know what is.  The scores for the 2016 Verbal Reasoning test consisted of over six thousand different standardised scores.  I’d like to understand how slightly over a half an hour’s multiple choice questions can differentiate candidates to such a degree.  I can conceive some particularly ingenious mathematical ways to generate such arbitrary precision but it’s about as smart as making a sundial so big you are able to read the time to the nearest second.  (Who needs a space elevator when you have a Durham ™ sundial?)

There is also that claim floating around that Durham have somehow managed to invent a test which can’t be tutored.  In Coombs vs ICO (2016)  Judge Hamilton considered, “… the appellant had provided evidence that the claimed USP of tutorproofing was highly questionable and that the public interest warranted close examination of this claim which could only be achieved through the disclosure of the disputed material.”  So at least one Tribunal judge thinks that it would be in the greater interest to be able to challenge such a claim.

There is general public interest in public authorities being transparent in their decision making.  Durham’s income from the 11+ in 2016 was £815,801 but this is only part of the picture.  These tests are used to decide which children benefit by attending one of 163 grammar schools.  Estimating their the annual budget as £5m per school, the best part of a billion pounds of public funding each year depends on the outcome of 11+ tests.  Transparency specifically over whether the allocation of school places is based on sound decision making is in the public interest.

Reasons for non-disclosure

Having listed a few reasons why it would be beneficial to release the information it’s important to put the other side of the arguments.  The reasons the Information Commissioner gave for non-disclosure are:

  1. It is in the public interest to develop tutor proof tests.
  2. The Information Commissioner says that CEM’s rival is not subject to Freedom of Information thus placing CEM at a commercial disadvantage.
  3. The revenue from these tests helps fund Durham Vice Chancellor’s £470,000 basic salary.

Regarding the first of these points, the Commons Select Committee of Education found, “the Government has yet to demonstrate how an admissions system could be designed in a manner which would be immune to gaming, or down to the ability to pay.Giving evidence, the Chief Scientific Advisor for Education Tim Leunig said, “Buckinghamshire had attempted to come up with one [a tutor-proof test] but it had not worked.” and schools minister Nick Gibb has likened this objective to the quest for the holy grail.

Regarding the second, GLA are indeed a private company, but the information itself is held by schools.  If, for example, I thought GLA were selling tutor proof tests or snake oil I’d ask the schools for information.

With the last point the Information Commissioner’s exact words put a bit more lipstick on this particular pig but still boils down to not releasing information because Durham are making money from something.  Exactly what, has me confused.  Each time I mentioned “tutor proof tests” during the hearing I was reprimanded by the judge who tells me Durham’s unique selling point is that their tests are seen to be more resistant to tutoring and that it works hard to design tests that are as resistant to tutoring as possible.  C’mon that’s one hell of a mouthful!

The hearing lasted about six hours without even managing to establish that commercial damage would result from the release of the information.  As we seemed to be in total deadlock over this point I thought it worth conceding so we can get on to the question of which is most important; Durham’s profits or opening up the can of worms.

This entry was posted in Uncategorised. Bookmark the permalink.

2 Responses to Can o’ worms

  1. Jo says:

    That’s a good list or reasons why raw scores should be open. To my mind transparency with raw scores means parents understand the test and get a feel for what it’s doing. My Kent perspective is all about the opaqueness of short papers where many get the same score.

    Kent Test papers have 25, 26 and 80 questions, with a pass needed in each. 15,000 children taking a 25 question test means loads getting the same mark, so age weighting is everything. But hardly any parents know how short the papers are, or get to see that their son/daughter missed out just because of date of birth standardisation.

    I also believe if raw scores are generally available there’ll be no benefit for tutors, because the many 11-plus information websites will get the data, and publish raw score and pass marks to help their web hits. They’d be bonkers not to, just add a load of tutoring ads and it’s easy money. 🙂

    So I think grammar schools, primary teachers, parents, will get easy online access to the same information as any 11-plus tutor. This would almost certainly lead to councils and admission authorities deciding to publish clear and open information themselves, probably in a table, or plain english like: ‘Last year to pass this test the score before standardisation was approximately 12/25 maths 14/26 english and 55/80 reasoning.’ With a load of notes explaining that the test varies each year, and age weighting can mean that higher or lower scores can give a pass.

    I think this sort of openess is excellent, everyone knows what the test is doing. And they can make up their own minds about whether it is fair that 1,700 Kent children got around the same mark in maths but only half of them were ‘grammar standard’… 🙂

    I think the judge will be worried that tutored children would start using the test tactic of going for their 12 best maths questions in this case. But so many would be doing it that I don’t think it would be a tutor advantage. In Kent primary schools care about pass marks, and might pass this info on. Though I’m not sure that it is such a deal breaking tactic, so maybe they wouldn’t bother?

    Keep fighting the good fight. Good luck!

  2. Pingback: News round-up : 11-plus scores, 'annexe' grammar schools, and good news in Guernsey – Comprehensive Future

Leave a Reply

Your email address will not be published. Required fields are marked *