All students will have seen the first full statement from the IFoA on their exam marking failures in today's student newsletter - some thoughts below:
“Examination marking
In recent months we have become aware of some social media activity which is alleging that there are systemic failures in the quality of the IFoA's examiners' marking, and in our examiners' ability to follow our own examination procedures. These issues have also been raised directly with us by a very small number of student members.
We take all feedback seriously, and would like to reassure the wider student membership that there are no systemic failures within our examination system.”
The IFoA have said to students in one-to-one correspondence that they don’t have to third mark papers with a material discrepancy between the two markers. However their own published policy says that they will third mark under these circumstances. Failing to apply a rule across the board is a systematic failure.
“As part of our programme of continuous improvement, we have visibility of the performance of our examiners and how examination teams adhere to our procedures.
“Visibility” is an interesting choice of words - suggests that evidence of how the process is ran is there but they don’t particularly interrogate what they see.
"We have carried out a thorough assessment of the quality of examination marking. We have looked at almost 15,000 scripts from across all the April 2016 examinations. The outcome is that, for each examination, over 90% of scripts have a variance between first and second independent markers which is within the normal academic expectation of independent marking (0-10 marks on a 100 mark total paper). However, as a leading professional body, we are working to increase this percentage further in collaboration with our examiners to ensure that all examining lies within our target range.
So, let’s be clear here, up to 10% of scripts (so almost a huge 1,500 exam entires) have a difference of more than 10 marks in the final score from each marker (I wonder what the highest difference is?), and therefore definitely should have been third marked. This is more than anyone had reason to expect before today.
I notice that the IFoA don’t comment on whether these papers were third marked, which is disappointing as this is one of the key complaints with the IFoA. The fact that the IFoA are “working to increase this percentage” is little comfort for the almost 1,500 exam scripts submitted in April 2016, which likely accounts for around 1,000 students who have not had their paper marked in accordance with the Institute’s published exam marking policy. At around £200-£400 a time, this must represent broadly £500,000 of exam fees backing an exam entry that has not been marked fairly. Particularly since October 2015, where a service contract is not carried out with “reasonable care and skill”, under the Consumer Rights Act 2015 students are entitled to the service being carried out again without charge and within a reasonable time period. All that students are asking for is for the papers to be marked in accordance with the published policy.
Regarding this new 10 mark difference standard, the Institute’s published policy to date has also only referred to “Where the first two markers disagree a script will also be third marked by an examiner.” so it has been very ambiguous what constitutes disagreement between the two markers, and certainly the threshold has not been set at a 10 mark difference until today. However, any reasonable person would say that students who have sat papers prior to today who see material differences either in the final mark, or material differences in the marks for individual questions, following a SAR surely fall with the remit of a “disagreement” between the two markers and are therefore eligible for third marking.
The IFoA also refer to a “target range” but don’t say if this is no more than a 10 mark difference - again more transparency is needed here.
"We have also checked the third independent marking of pass/fail borderline scripts and those with higher than normal variance. Appropriate further review is taking place and there is no systemic failure in the operation of this process. However, we have recognised that there is a need to improve the audit trail of this 'third marking'. We have put in place a programme of such improvement under the leadership of the Chair of the Board of Examiners. Improvements have already been made for marking of the September examinations. From initial marking, it is clear that appropriate operation of third independent review is taking place. We will be updating the Student Handbook to better communicate these third marking procedures."
Firstly, I was surprised that the (equally important) borderline script marking issue received little focus on this statement as a whole - given this is an area where the failings are more obvious I would hope the IFoA would have quickly identified those students negatively affected and put things right by marking the paper a third or fourth time. However there is no sign this has happened.
Again, I have seen multiple examples myself of students getting a borderline final mark but not being marked a third or fourth time. To repeat the IFoA’s published policy:
“Once the initial pass mark has been decided then scripts which are around the borderline will be marked a third or possibly fourth time to ensure that the examiners are happy with the proposed pass mark.”
This has not been happening and is again a systematic failure, so the claim that there has been “no systematic failure” smacks of denial by the IFoA.
"It is extremely important to us that you have confidence in the robustness of our examinations and examination process. By sharing with you the steps we are taking to monitor and improve our policies and processes, we hope this gives you confidence in your professional body and the professional qualifications that you are studying. Finally, we encourage you to use the Student Consultative Forum to raise any matters of general concern so that these can be discussed in a considered and transparent way.”
There are a lot of words here, but there is still no direct answer to why the IFoA have not been following their own published process (and seemingly not thinking this is a problem).
The unacceptable way students who have raised this directly with the IFoA have been treated by the IFoA executive (e.g. management refusing to address complaints escalated to them, instead just responding with “Communication with the IFoA on this matter is closed”) has also not been addressed in this statement, so there is still a long way to go before the IFoA have fixed the hole they are digging for themselves.
Given the way complaints have been treated to date, I was hoping for a change in tone by now, with an admission that things have gone wrong and an undertaking to fix things from the IFoA. Instead the IFoA are continuing to dig their head in the sand and maintain that there has been no systematic failure, despite the growing body of evidence from students who have done a SAR and found marking process failures.
Any thoughts on this?
Last edited by a moderator: Dec 14, 2016