I originally wrote this blog in 2019 shortly after the BBC reported the Bucks 11+ test had contained “impossible-to-answer” questions. I have rewritten it to correct some assumptions, but the main contention, that it is not possible to remove questions from a test without impacting reliability, remains unchanged.
In 2019 Buckingham Grammar Schools (TBGS) and their test provider Granada Learning Associates (GLA) removed eight questions from the Verbal Reasoning (VR) assessment. They told parents this did not impact reliability. A group calling itself The Parental Alliance for Choice in Education described those claims as, “… a deliberate act of misdirection.”
Number of VR questions
TBGS say the number of questions in their test is a Trade Secret. This is despite the fact that ten thousand children sit the test every year. They have argued in court that if they disclosed the number of VR questions tutors would use that to game the test. Despite this refusal to be open and transparent, it is possible to estimate this from other sources. GLA’s guidance shows their standard VR assessment has 80 questions set over 60 minutes. Children have 45 seconds to complete each question. TBGS allow just 20 minutes for VR, limiting the number of questions to about 27 or 28. With 8 questions removed the 2019 VR results would have further reduced that to about 20 questions.
Reliability
Each time a candidate takes a test they get a different score. Retest reliability is a measure of how consistent the score is between tests. This theoretically ranges between
- 100%: Scores are exactly the same each time.
- 0%: Scores are completely random.
Prof Dylan Wiliam explains (2001) “There are only two really effective ways of increasing the reliability of a test: make the scope of the test narrower, so you can ask more questions on the same topic, or, what amounts to the same thing, make the test longer.” This confirms common sense. If it was possible to remove questions from the test without reducing reliabiity we could just remove them all and still have a reliable assessment!
The starting point to working out the actual reliability is the technical information published by GLA. When their Verbal Reason has 75 questions its reliability is 81%. Wiliam explains, “In general if we have a test of reliability r and we want a reliability of R, then we need to lengthen the test by a factor of n given by”:
(1) ![]()
By rearranging† for r we can turn the question around and ask what effect altering the number of questions has on reliability.
(2) ![]()
n is the ratio of questions between different length tests. The calibration test had 75 questions and initial reliability is R=0.81. For the normal Bucks VR test n=75/28. Plugging these values into equation (2)
(3) ![]()
When 8 questions were removed in 2019 n=75/20. Plugging these values into equation (2)
(4) ![]()
Inconsistencies
In 2020, the TBGS website FAQ page said, “In relation to the reliability of this year’s Secondary Transfer Test … our analysis has found that the overall reliability of the test has not been compromised. The results indicate that both the Verbal Reasoning and English sections score well above 0.80 reliability, which is considered to be very good.”
TBGS need to explain how a test which has a reliability of 0.81 (81%) when it has 75 questions can still have a reliability of “well above 0.80” when three quarters of the questions have been removed.
†Rearranging Wiliam’s formula
The steps in getting from equation (1) above to (2) are:
(5) ![]()
(Multiply out RHS denominator)
(6) ![]()
(Multiply out brackets)
(7) ![]()
(Add rR both sides)
(8) ![]()
Factor out r LHS
(9) ![]()
Divide by n -nR +R
(10) ![]()
Of course not all all test centres handled the error discovery in the same way which introduces a whole new level of unfairness to this test. Some knew in advance so instructed children to miss out the incorrect questions while others stopped the test part way through when children began querying the mistake and impossible to answer question, then restarted it. Some centres continued without stopping the test leaving the children to decide how to handle the impossible questions themselves. So the whole point of giving everybody a level playing field on which to play has been destroyed.
What nobody seems to consider with this dreadfully inadequate solution proposed by GL is that the removal of a whole block of 8 verbal reasoning (VR) answers from the test will now confer an unfair disadvantage to those children whose strongest subject is verbal reasoning as it weights the overall test differently so that it is no longer 50% verbal reasoning (because 8 VR questions have been removed). Unless the proposal is to alter the weighting to correct for the disadvantage to children strong in VR? But they don’t say that they will do this. There is an unnerving lack if transparency about their proposed fix to this broken test that gives no comfort that we will end up with a decision that is fair to everyone. GL’s responses so far raise far more questions than they give answers. Unfortunately GL’s mistake has been so monumental on this occasion that they cannot now correct this in a way which is fair to everybody nor in a way which will satisfy everybody. I would like to see them deliver to everybody two sets of marks for the test. The first that gives the results for the whole test, without the removal of the 8 questions, and the second that gives the mark following removal of the 8 questions.
Surely then, the fairest way to handle this would be for each child to have their highest mark of the two carried forward for selection purposes so as not to disadvantage those who could not finish the test on time because of the errors and also to correct the disadvantage conferred to children strong in verbal reasoning.
Any controlled test needs to minimise residual variance or ‘noise’. Testing conditions should be identical. That’s common sense. I would (finger in the air) guess there are a couple hundred primary schools in Bucks but obviously no consistency in the way they react when test takers start questioning the test itself. I can’t see any way of squaring that particular circle.
The problem with issuing two sets of results is parents will argue that the one which ranks their child highest is the definitive. Best of two doesn’t work for those whose children get 120 via both methods when they compare against children who scored 121 via one method but only 118 via the other.
I take issue with GLA’s assertion that a statistician appointed by TBGS is ‘independent’ and it should be easier to understand the impact of all this when TBGS/GLA publish the report they have done so that this is open to independent public scrutiny. Public authorities need to be open and transparent.
Standardisation doesn’t work quite as you think. The VR element will still make up 50% of the overall mark. With standardisation the mean (average) for all test takers is calculated then each individual’s score is compared relative to that point. (Initially this is measured in standard deviations (denoted by Greek sigma σ) which range from about -2.5σ to + 2.5σ but because it would seem odd to tell a test taker they have a negative score these are scaled, conventionally by multiplying by 15 and adding 100. TBGS carry out unconventional scaling to match a totally arbitrary pass mark of 121 which one year did result in some children getting negative marks!)
The following is a simplification, but imagine a test has 40 questions the average raw score is 20 and a child getting 30 ending up scoring +1σ.
If all the questions were equal difficulty and half removed at random, the average raw score would be 10 and that same individual would score 15 (although maybe they’d score 14 or 16 because the removed questions happened to favour/disfavour them). They’d still end up with a standardised score of +1σ (or maybe not!)
Removing questions reduces overall reliability. I suspect that TBGS/GLA’s reasoning is that it is not particularly reliable in the first place, we have to decide somehow which children are admitted and the impact of this error doesn’t make much difference to their cohorts. Parents might see things differently.
James, just wanted to say thanks for posting this and putting in the effort to provide a deeper explanation of the statistical analysis. Much appreciated.
GL’s letter calls the paper ‘Verbal Skills’ of which ‘Verbal reasoning’ is a section, as you say. GL have always regarded their question banks as ‘purer’ in terms of validity so VR and English questions (which form the other part of the paper) come from separate parts of their question banks. That would be my understanding. GL used to try and dissuade schools from these combined tests but commercial pressure proved too great.
Thanks for this. IF all 8 questions were removed from a ‘subtest’ containing only 19 questions then the reliability for that subtest would decrease by a factor of roughly 24% [Expressed as an Excel formula =(1/sqrt(19))/(1/sqrt(11)) ]. If the reliability was 85% to begin with it would be ~65% although it should be borne in mind that this is only one of four ‘subtests’ used to determine the overall scores.
Ideally though, we should not have to speculate about this. If TBGS, a public authority, is using these tests to determine admissions they should be disclosing this type of information as a matter of *routine* openness and transparency.* I have asked them to provide a copy of the “detailed statistical analysis” mentioned in the Oct 1 letter via https://www.whatdotheyknow.com/request/statistical_analysis_carried_out. There are four other requests for information so clearly a lot of public interest.
*I have discovered an FAQ on the TBGS website https://www.thebucksgrammarschools.org/faqs which says, “both the Verbal Reasoning and English sections score well above 0.80 reliability, which is considered to be very good.” This falls a long way short of providing actual reliability values for each ‘subtest’ and presumably relates to the tests without any errors. Professor Rebecca Allen described https://ffteducationdatalab.org.uk/2017/05/what-does-north-yorkshire-tell-us-about-how-reliable-the-11-plus-is/ the reliability of North Yorks tests as being “only 0.89” suggesting that she didn’t think that was reliable enough to decide a child’s future. Those tests had 80 questions and, all things being equal, would be twice as reliable as a test with only 20 questions.
James, my son has been awarded 118 and we strongly think the removal of the last 6 points from verbal reasoning, which was his strongest subject, made all the difference. How would you advise we challenge their testing method/statistics as I do not have a statistics background. Any advice would be greatly appreciated.
Hi M,
You have my sympathy, although there should be absolutely no reason for you to *need* any knowledge of statistics. That’s not just common sense, it’s written into the Schools Admissions Code, the legislative framework under which schools must operate. (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/389388/School_Admissions_Code_2014_-_19_Dec.pdf)
Section 14 states, “In drawing up their admission arrangements, admission authorities must ensure that the practices and the criteria used to decide the allocation of school places are fair, clear and objective. Parents should be able to look at a set of arrangements and understand easily how places for that school will be allocated.”
Public authorities, which tax payers all contribute to, should be open in transparent. The schools are in possession of all the facts and there is nothing preventing them from explaining exactly what impact this has had on the ability to determine who is admitted in 2020.
I can only comment on the effect of reducing the number of questions in the test, but really it’s far more complex than that. This type of testing is predicated on it being is conducted under controlled conditions but I understand that some children didn’t find out about the problem until the test had finished, others were interrupted part way through and some were told before they started. Regardless of the removal of the last six questions, the latter group effectively had 2/19 additional time to complete this test because they didn’t need to attempt two questions.
On a personal level you could ask your son when his group found out but it is incumbent on the schools to explain exactly what the impact of this error had on him in a way which you can understand. I would keep pressing them for an answer.
Due to corona virus the grammar school appeals may not take place. GL assessment should help Bucks CC to ensure these do take place. Please can you help put pressure on? Please see message from GL on twitter “To all our customers – we just wanted to let you know that we’re here to help”. Bucks CC is a customer. Please see Bucks message “The independent Appeal Panel members are unpaid trained volunteers; there may may unavoidable cancellations / delays due to the virus”.