I originally wrote this blog in 2019 shortly after the BBC reported the Bucks 11+ test had contained “impossible-to-answer” questions. I have rewritten it to correct some assumptions, but the main contention, that it is not possible to remove questions from a test without impacting reliability, remains unchanged.
In 2019 Buckingham Grammar Schools (TBGS) and their test provider Granada Learning Associates (GLA) removed eight questions from the Verbal Reasoning (VR) assessment. They told parents this did not impact reliability. A group calling itself The Parental Alliance for Choice in Education described those claims as, “… a deliberate act of misdirection.”
Number of VR questions
TBGS say the number of questions in their test is a Trade Secret. This is despite the fact that ten thousand children sit the test every year. They have argued in court that if they disclosed the number of VR questions tutors would use that to game the test. Despite this refusal to be open and transparent, it is possible to estimate this from other sources. GLA’s guidance shows their standard VR assessment has 80 questions set over 60 minutes. Children have 45 seconds to complete each question. TBGS allow just 20 minutes for VR, limiting the number of questions to about 27 or 28. With 8 questions removed the 2019 VR results would have further reduced that to about 20 questions.
Reliability
Each time a candidate takes a test they get a different score. Retest reliability is a measure of how consistent the score is between tests. This theoretically ranges between
- 100%: Scores are exactly the same each time.
- 0%: Scores are completely random.
Prof Dylan Wiliam explains (2001) “There are only two really effective ways of increasing the reliability of a test: make the scope of the test narrower, so you can ask more questions on the same topic, or, what amounts to the same thing, make the test longer.” This confirms common sense. If it was possible to remove questions from the test without reducing reliabiity we could just remove them all and still have a reliable assessment!
The starting point to working out the actual reliability is the technical information published by GLA. When their Verbal Reason has 75 questions its reliability is 81%. Wiliam explains, “In general if we have a test of reliability r and we want a reliability of R, then we need to lengthen the test by a factor of n given by”:
(1) ![]()
By rearranging† for r we can turn the question around and ask what effect altering the number of questions has on reliability.
(2) ![]()
n is the ratio of questions between different length tests. The calibration test had 75 questions and initial reliability is R=0.81. For the normal Bucks VR test n=75/28. Plugging these values into equation (2)
(3) ![]()
When 8 questions were removed in 2019 n=75/20. Plugging these values into equation (2)
(4) ![]()
Inconsistencies
In 2020, the TBGS website FAQ page said, “In relation to the reliability of this year’s Secondary Transfer Test … our analysis has found that the overall reliability of the test has not been compromised. The results indicate that both the Verbal Reasoning and English sections score well above 0.80 reliability, which is considered to be very good.”
TBGS need to explain how a test which has a reliability of 0.81 (81%) when it has 75 questions can still have a reliability of “well above 0.80” when three quarters of the questions have been removed.
†Rearranging Wiliam’s formula
The steps in getting from equation (1) above to (2) are:
(5) ![]()
(Multiply out RHS denominator)
(6) ![]()
(Multiply out brackets)
(7) ![]()
(Add rR both sides)
(8) ![]()
Factor out r LHS
(9) ![]()
Divide by n -nR +R
(10) ![]()
Recent Comments