Coombs vs ICO

Why are 11+ test results shrouded in such secrecy?  For any other comparable test the student knows exactly what to expect.  They’re given specimen papers, marking schemes, past examples.  The whole process is open and transparent, but the way grammar schools select who they admit is more covert than a Kremlin extraterritorial execution.

I wanted to know how 90 minutes of multiple choice questions could be used to generate three individual scores for Maths, Verbal Reasoning and Non-Verbal Reasoning given to two decimal places, so back in 2014 asked Durham University’s Centre for Evaluation and Monitoring (CEM) to ‘show their workings out’.  My request hit a raw nerve.  From their reaction you’d think I’d asked them to provide a complete list of questions and answers for the next 10 years’ tests.

Durham say their Unique Selling Point (USP) is the provision of tests “more resistant to coaching influences” and that disclosing how they process the results to get the ‘standardised’ scores given to parents would prejudice their commercial interests.  A summary of my attempts to simply find out how 11+ tests are marked is as follows.

  • October 2014 I asked Durham University for raw and “standardised” test marks from their 11+ tests
  • The next 18 months were spent exhausting all the available avenues of arbitration before finally challenging the Information Commissioner in court
  • March 2016  The case was heard ‘on the papers’ whereby both sides submit written arguments which are reviewed by a judge and two lay members.  The judge thought my case was “compelling” and “provided evidence that the claimed USP of tutorproofing was highly questionable and that the public interest warranted close examination of this claim which could only be achieved through the disclosure of the disputed material.”  He was in the minority.  The two lay members argued it made no difference whether Durham’s tests were tutor-proof or not; they just needed to demonstrate that they were profiting by making this claim.  The court recorded, “CEM asserts that one of the benefits of its 11+ testing is that it is ‘tutor proof’, or more ‘tutor proof’ than the alternatives. … the test is a better predictor of natural ability and the advantage gained by children in families that can afford private tuition is reduced.” and concluded that disclosure would effect Durham’s ability to make money.  The central question, whether or not Durham’s tests are less susceptible to tutoring, was not answered.
  • Sept 2016  I only became aware of this after the event, but appears my ruling was taken out of context as “evidence” that a tutor proof 11+ test was now a reality by a government dogmatically fantasising over a return to selective education.  Durham University told the Grammar Schools Heads Association GSHA who told Lord Nash who told Theresa May who told the entire nation, “many selective schools are already employing much smarter tests that assess the true potential of every child.  New grammars will be able to select in a fair and meritocratic way, not on the ability of parents to pay.
  • Nov 2016  The House of Commons Select Committee for Education were not convinced.  Nick Gibb was forced to admit to them that the prospect of developing a tutor-proof test was as realistic as finding the Holy Grail
  • May 2017. Buckinghamshire’s 13 selective schools, switched back to using their previous 11+ test provider.  This should have been a key turning point because the Bucks grammars are on record as specifically chosing Durham for their tutor proof tests and they accounted for 20% of Durham’s 11+ testing revenue.  Buck’s U-turn was so widely reported in the national press it seemed inevitable to kill forever the “tutor-proof” myth but this was not to be because by then, this “evidence” was propping up the government’s entire education policy and the myth had acquired a life of its own. 
  • Jan 2018.  By now, I’d gone through the whole process of requesting similar information from Durham and reached the point of appealing against the Information Commissioner’s ruling in court but this time requested an oral hearing so I could properly explain things to the Judge.  I was quite frankly terrified at the prospect of facing teams of barristers, solicitors and the whole legal machinations but once we got started realised the professionals were having a far harder time of Judge Kennedy’s intense grilling than I was.  Remarkably, Durham didn’t show up to justify their claim, central to the entire case, that simply revealing how they mark their tests would benefit tutors so the hearing was adjourned.
  • Feb 2018.  Reflecting on points made by Judge Kennedy during the hearing, I conceded, in a very qualified way, that Durham had profited from false claims that their tests were less susceptible to tutoring.  This meant the legal exemption under which they’d refused the information was properly “engaged” but the court would need establish whether it was in the public interest to disclose the information.  If tutors really could benefit from the information then that clearly would not be in the public interest but Durham had failed to demonstrate that to the satisfaction of the court. 
  • Aug 2018.  After a lengthy wait during which I was expecting to be told when the case hearing would continue, I was handed down a judgement dismissing my appeal.
  • Sept 2018.  I asked the First Tier Tribunal for permission to take my appeal to the Upper Tribunal on a number of grounds mainly that they failed to establish that tutors would benefit from disclosure of the marks.  They refused.
  • Nov 2018.  I appealed to the Upper Tribunal asking their permission and Judge Nicholas Wikeley agreed that a number of my grounds requesting to appeal to the higher court were “arguable in law” and granted me permission to take my case to the higher court.
  • Dec 2018.  The Information Commissioner, who is my formal opponent, responded by agreeing that the lower court had erred in law and that my appeal should be re-heard by them.

If my appeal to the Upper Tribunal is successful, then it will be sent back to the lower court to hear the case afresh. 

Throughout, Durham have managed to persuade both the Information Commissioner and the court that I should not even see their explanation of how the claim tutors could benefit from knowing raw test marks.  I was sent a page of blacked out text and had to leave the hearing whilst the Information Commissioner tried her best to explain what Durham had written to the panel. 

The parts of Durham’s argument I have seen are if tutors understood how they marked their tests they could work out the “pass mark” for each section and then coach their students to answer the minimum number of questions correctly.  There are a number of fatal flaws in this argument; firstly the 11+ is a ranking test so ‘passing’ entails scoring more than others.  There is no concept of a “pass mark”.  Secondly, it’s common knowledge (sixty thousand children sit their tests each year) that Durham’s tests are broken into short separately timed sections for the express purpose of preventing candidates from concentrating on one section.

Judge Hamilton who I assume saw the unredacted version of this during my first appeal described Durham’s explanation as, ‘highly technical and incomprehensible to anyone without a qualification in statistics’ which is ironic as my BSc Hons degree included linear regression, single and multiple regression models, binary regression, GLMs, diagnostic checking and log-linear modelling. 

Durham’s refusal to release information has nothing to do with minimising the effects of tutoring and bears more in common with Hans Christian Andersen’s tale, the Emperor’s New Clothes.  Their “tutor-proof” test are one big bluff which would melt away faster than an ice lolly on hot tarmac if they were instructed to release the information.  Otto von Bismark famously likened laws to sausages, “Gesetze sind wie würste” saying that confidence in both is predicated upon ignorance of how they’re created.  Public confidence in the eleven plus test largely depends on them remaining ignorant of how 90 minutes of multiple-choice questions are used to conjure up three separate measures given to two decimal places enabling grammar schools to precisely, if not particularly accurately, determine who they admit.  

I’ll finish by quoting Professor Rebecca Allen, Director of UCLs Centre for Education Improvement Science. “The myth that the 11-plus separates children effectively into those who can and cannot benefit from the grammar school education is only sustained because we are not transparent about the extent to which the system must be misclassifying some children through this relatively short test.”  If society is to debate selective education in an objective and rational way then the first thing we need to do is peel away the layers of secrecy surrounding the test upon which the whole system is founded.

Leave a Reply

Your email address will not be published. Required fields are marked *