Coombs vs ICO

Why are 11+ test results shrouded in such secrecy?  For any other comparable test the student knows exactly what to expect.  They’re given specimen papers, marking schemes, past examples.  The whole process is open and transparent.  The way grammar schools select who they admit is more covert than a Kremlin extraterritorial execution.

I’m intrigued as to how 90 minutes of multiple choice questions can be used to generate three individual scores for Maths, Verbal Reasoning and Non-Verbal Reasoning given to two decimal places, so back in 2014 asked Durham University’s Centre for Evaluation and Monitoring (CEM) to ‘show their workings out’.  My request hit a raw nerve.  From their reaction you’d think I’d asked for a complete list of questions and answers for the next 10 years’ tests.

Durham say their Unique Selling Point (USP) is the provision of tests “more resistant to coaching influences” and that disclosing how they process the results to get the ‘standardised’ scores given to parents would prejudice their commercial interests.  Touchy!

The chronology of my attempts to simply find out how 11+ tests are marked is as follows.

  • October 2014 asked Durham University for raw and “standardised” test marks from their 11+ tests
  • The next 18 months were spent exhausting all the available avenues of arbitration before finally challenging the Information Commissioner in court
  • March 2016 the court ruling went against me but was not unanimous. The case was heard ‘on the papers’ in which both sides submit written arguments which are reviewed by a judge and two lay members.  The judge, in the minority, thought my case was “compelling” and “provided evidence that the claimed USP of tutorproofing was highly questionable and that the public interest warranted close examination of this claim which could only be achieved through the disclosure of the disputed material.”  The two lay members argued it made no difference whether Durham’s tests were tutor-proof or not; they just needed to demonstrate that they were profiting from claiming their tests were tutor proof.  Durham’s 2011 income from 11+ testing was just £329,695.  They told the Information Commissioner it had grown to £1m as a direct consequence of their ‘USP’.  The court recorded, “CEM asserts that one of the benefits of its 11+ testing is that it is ‘tutor proof’, or more ‘tutor proof’ than the alternatives. … the test is a better predictor of natural ability and the advantage gained by children in families that can afford private tuition is reduced.” and concluded that disclosure would indeed prejudice their commercial interests.
  • Sept 2016 although the court came to no decision at all over whether Durham’s tests were tutor proof it appears their ruling was taken out of context by the government as evidence to support this. Lord Nash told his peers that the Grammar Schools Heads Association GSHA were working on developing tests, “much less susceptible to coaching”.  Theresa May followed that up by telling the whole nation, “many selective schools are already employing much smarter tests that assess the true potential of every child.  New grammars will be able to select in a fair and meritocratic way, not on the ability of parents to pay.

  • Nov 2016 Nick Gibb, being grilled by the House of Commons Select Committee for Education, was forced to admit that the prospect of developing a tutor-proof test was as realistic as finding the Holy Grail.
  • May 2017. Buckinghamshire’s 13 selective schools, switched back to using their previous 11+ test provider.  This is particularly significant because they specifically selected Durham for their tutor proof tests.  The contract had accounted for 20% of Durham’s revenue which explains why my innocent question had hit such a raw nerve.  Schools Week reported that Bucks had sacked Durham prompting a rebuffal that they ‘chose’ not to re-tender, presumably in much the same way that Jeffrey Archer ‘chose’ not to continue standing for London Mayor after it emerged he had lied during his 1987 libel case.

Although I was terrified at the prospect of appearing in court to challenge the Information Commissioner, Durham University, barristers, solicitors and the whole legal machinations I felt it was essential if my earlier appeal ruling was being used by the government and GSHA to perpetuate a false claim about tutor-proof tests.

One bizarre aspect of the case is that Durham convinced both the Information Commissioner and the court that I shouldn’t even see their reasons for not disclosing this information as they claimed that would undermine their claimed ‘USP’.

When someone is being this evasive it’s clear they’re hiding something.  Judge Hamilton, from my first appeal, described Durham’s case as, ‘highly technical and incomprehensible to anyone without a qualification in statistics’ which is ironic as I do have a qualification in statistics (linear regression taken as my BSc third level optional module).

Durham didn’t attend the hearing, leaving it to the Information Commissioner’s barrister to present their case which turned out to be an anticlimax.  Durham say that if tutors knew how ‘standardised’ marks were calculated they could work out the inverse function and use this to calculate raw marks from ‘standardised’ ones then train children to focus on answering the minimum number of questions needed to “pass”.

There are a couple of fundamental flaws with this line of reasoning.  Firstly, the 11+ test is a ranking test.  The objective is to score more than the other competing candidates.  A strategy not answering questions has to be one of the most inanely stupid ideas ever.  Secondly, if tutors were given the raw test marks they’d hardly go to great lengths to work this out.

After all these years it turns out that Durham’s refusal to release information nothing to do with thwarting tutoring.  They’re trying to hide the way they mark these tests.  Otto von Bismark famously likened laws to sausages, “Gesetze sind wie würste” essentially saying that confidence in both is dependent on ignorance of how they’re created.  The Bismark principle extends to the 11+; public confidence is dependent upon remaining ignorant of how 90 minutes of multiple choice questions are used to conjure up three separate measures given to two decimal places.  Durham could hardly tell the Information Commissioner the real reason they didn’t want to show their workings was because that would force them to start being open and transparent so instead made up a story that hiding the way they mark tests somehow reduces the effects of tutoring.

I’ll finish by quoting Professor Rebecca Allen, Director of UCLs Centre for Education Improvement Science. “The myth that the 11-plus separates children effectively into those who can and cannot benefit from the grammar school education is only sustained because we are not transparent about the extent to which the system must be misclassifying some children through this relatively short test.”  If society is to debate selective education in an objective and rational way then the first thing we need to do is peel away the layers of secrecy surrounding the test upon which the whole system is founded.

The judge adjourned the hearing to give Durham yet another opportunity to present their case.