Exam marking problems

[ActuaryStudent9123116]

I've been sent an email chain originating from a student who has uncovered (in their own words) "failures in the exam marking process" across three seperate exams they sat this year.

Issue being around the exam papers not beig marked in accordance with the published exam marking rules the institute publish. In one case, they got 59 out of 100 for an exam (the pass mark was 60), but their paper was not marked a third time, which is what the Institute say they will do with a borderline script in the current student handbook (see page 66):

https://www.actuaries.org.uk/documents/student-handbook-20152016

"Once the initial pass mark has been decided then scripts which are around the borderline will be marked a third or possibly fourth time to ensure that the examiners are happy with the proposed pass mark.

Where the first two markers disagree a script will also be third marked by an examiner."

They only found out about this by doing a Subject Access Request for the marks and all supporting information under the Data Protection Act, which gave a free-of-charge (and what looks like very detailed) breakdown of their scores from the two markers including the markers comments. I'm now worried whether my borderline paper really was marked a third/fourth time in April as I thought!

My colleague who forwarded the email to me said lots of students have been sending a Subject Access Request to the institute because of their concerns, but has anyone here done this and also uncovered any marking irregularities? Given this happened on three seperate exams for an individual, it's clearly not a one-off. The exams in question were:

CA2 sat in May 2016
SA4 sat in April 2016, and
CA3 sat at some point in 2016.

but may affect all exams that have been sat!

Given the amount of hours we put in to study, the least the Institute can do is follow their own marking policy, so would be interested to know if anyone else is aware of this issue?

 

[StudentActuary_02]

I have also found a failure in the marking process for an exam I sat earlier this year, so this is not a one-off!!

In my case, the two markers gave wildly different marks for my script. I only found out about this by doing a Subject Access Request (SAR) too, which the IFoA have to process under the Data Protection Act as exam marks, minutes from meeting discussing your mitigating circumstances form etc. are all personal data. I found the following template from the ICO website helpful (about halfway down this page) for how to word my request, but just make sure you ask for everything they could hold:

https://ico.org.uk/for-the-public/personal-information/

Also see here:

https://ico.org.uk/for-the-public/schools/exam-results/

Lots of my colleagues have done a SAR as well, and are just waiting for a response now! Apparently the IFoA only keep records for a year, so you have to get your SAR in ASAP if you want a mark breakdown for the September 2015 exam session. Send it by email to Clifford.Campbell@actuaries.org.uk or education.services@actuaries.org.uk for speed and to avoid any issues with the post.

I agree that the mark breakdowns are excellent for revision purposes too! (far better than reading the examiners report as the information is specific to your own paper)

 

[almost_there]

I also found contradictory marking going on when I did a SAR a few years ago. Once I paid for an exam counselling report, which included contradictory comments, therefore utterly unhelpful. I suggest all complaints get referred to FRC. enquiries@frc.org.uk

 

[StudentActuary_02]

I understand that the FRC require you to raise the complaint with the IFoA first, but agree that if the IFoA reject you then the case must be referred to the FRC (otherwise nothing will be done and you could be resitting a paper you shouldn't have failed).

 

[Anacts]

have also found a failure in the marking process for an exam I sat earlier this year, so this is not a one-off!! In my case, the two markers gave wildly different marks for my script.

On its own, I don't think this is necessarily a failure. Sometimes two markers disagree. The process would then lead to third marking. If that didn't happen then arguably a failure (although if both marks were miles off the pass mark then it's less important, but appreciate it's useful to know how close you were).

 

[bystander]

This is interesting. I'm qualified since before disclosure came to be. I know that you are now told your score on a paper rather than grades, but how is that score derived where papers are multi marked? Anacts point seems very valid that if both markers say that the score is well below the pass mark a third marker most likely wouldn't give a pass mark. I believe at the start of the marking process there is a sample batch and all examiners look the same script to try to iron out what should and shouldn't give credit so marks don't deviate violently. Clearly it can happen and the process of investigation is laid out here in an earlier post. It may be though that it takes a long time to resolve and as unpalatable as resits are, it may be quicker to resit and pass than go complaint route. Before anyone shoots me down in flames, I had to resit at my own expense various exams so I do understand. I also had counselling, which then was my only recourse to finding out where I had gone wrong and again, I paid for that. Good luck to you all in crossing the final hurdle. It's a great feeling when it comes.

 

[almost_there]

The issue for me is trust. If they publish a process, then people are uncovering it's not being followed properly: that would be completely unacceptable.

 

[StudentActuary_02]

The issue for me is trust. If they publish a process, then people are uncovering it's not being followed properly: that would be completely unacceptable.

And unfortunately that is exactly what is happening and it is completely unacceptable. If we tell a client we will do our work in line with our published processes then we as actuarial staff have to follow it, and the IFoA should too.

Really it's quite simple, when I, and anyone else who has ever sat an exam, paid the exam fee, the latest student handbook that was available at the time said they will third mark on a borderline case or where the two markers disagree significantly. This clearly forms part of the contract we entered when the exam fee is paid, so the IFoA have to do it. This is the line anyone should be using when taking their complaint to the IFoA. Once you have exhausted their internal complaint process, then you can refer it to the FRC.

Latest student handbook is attached for anyone's use.

 

[StudentActuary_02]

Student handbook from August 2015 to October 2015:

 

[StudentActuary_02]

Student handbook from November 2015 to June 2016:

 

[almost_there]

Makes me wonder if there's a breach of contract here...

 

[StudentActuary_02]

Finally (last post for today...) earlier this week I heard of another student who was a borderline fail for their April 2016 exam.

In their case, they scored 59 and 60 (the pass mark was 60). Their two scores were averaged to 59.5 and they were given a fail. However, despite what the student handbook says, their paper was not marked a third time.

Despite there being a 50/50 chance of this student already having passed this exam (and therefore being entitled to a promotion and a big salary increase), they are now facing having to devote another 150+ hours of study to sit it again.

 

[StudentActuary_02]

Given CT results are out later today, a reminder (or really a call to action) to all students to read this thread and raise a Subject Access Request if you were unsuccessful in this sitting.

 

[Tarbuck]

I would disagree! If everyone who fails starts doing this for feeling a bit sore from failing then it's going to damage the credibility of anyone who might have a genuine complaint! I would also guess if any positive action does come from this it'll be in the form of a tightened process around a 3rd marking with little to no change to pass rates so I doubt there would be a large number of cases it would make a difference at all! I certainly wouldn't expect any retrospective passes issued. I'd focus more on passing the next sitting than spending too much time on a perceived injustice from the last sitting. At the end of the day, everybody is subject to the same process.

 

[almost_there]

Former US president Ronald Reagan once said wisely "Trust... but verify". That's what students are doing when raising a SAR. If in doing so they spot problems with how the process was carried out then they're doing everyone a favour.

 

[Tarbuck]

I can see the sense in appealing for CA3 where marking is open to interpretation and hence susceptible to big variation. Encouraging everyone who fails a CT which is largely maths based and difficult to vary much in marking to submit a subject access request just smacks of sour grapes for failing and if it happens en masse is only going to result in higher costs and longer delays.

Whatever the process is, there is always going to be borderline students that are disappointed. I really wouldn't try and unify those people into action hoping it will help in getting a fairer CA3 system - to me it sounds the gripes against CA3 are fair and any subject specific action that could be gained for that is going to be lost/ delayed/ watered down if they start having SARs, complaints and appeals from every student close to the pass mark in every exam.

 

[almost_there]

Tarbuck, I must agree with your point about the mathematical and therefore more objective nature of CT marking. Personally I never had a complaint about CT or ST marking, nor the acted preparation for these exams, even when I failed a couple of them. SA and CA3 are more contestable. Also I think the people who will be the most disappointed with a fail are those in the FA category. I think for them it's worth doing this if they feel hard done by and to check whether they were third marked...

 

[StudentActuary_02]

I would disagree! If everyone who fails starts doing this for feeling a bit sore from failing then it's going to damage the credibility of anyone who might have a genuine complaint! I would also guess if any positive action does come from this it'll be in the form of a tightened process around a 3rd marking with little to no change to pass rates so I doubt there would be a large number of cases it would make a difference at all! I certainly wouldn't expect any retrospective passes issued. I'd focus more on passing the next sitting than spending too much time on a perceived injustice from the last sitting. At the end of the day, everybody is subject to the same process.

Tarbuck, the problem is have with this approach is not acceptable, whether it happens to an individual or the entire student population. Students should be encouraged to hold the Institute to account if they have misled about the way their paper is marked, and students sitting the mathematical CT papers are also finding surprisingly large discrepencies between the marks given.

The IFoA seem happy to reap the reputational rewards of having a robust exam system that offers third marking on borderline/significant discrepency marks, yet the evidence is growing that they aren't actually doing this. A borderline 58/100 or a paper should be third marked according to the policy, and it is only due to the Institute's failure to be open and transparent about its processes that students have to resort to SARs to know if their paper has been marked fairly. After 150/200 hours of revision per subject (i.e. essentially a month of work, carried out in the space of half a year in their own free time), and the ever growing cost of entering for the exams, students should be confident that their paper will be marked in line with the Institute's published policy, but at the moment no student can be without making a SAR.

 

[almost_there]

I agree with student actuary. We should hold the IFoA to the same standards as anyone else in society as a service provider. They have a duty to meet their obligations on the third marking. If they don't & the FRC do nothing about it then perhaps it's time to draw attention to these matters elsewhere.

 

[StudentActuary_02]

I agree with student actuary. We should hold the IFoA to the same standards as anyone else in society as a service provider. They have a duty to meet their obligations on the third marking. If they don't & the FRC do nothing about it then perhaps it's time to draw attention to these matters elsewhere.

Agreed - if the FRC are given sufficient examples of marking process failures and "fob off" correspondence from individual complaint cases with the IFoA but still refuse to act, then there should be some thought given to organising the next level of escalation.

I wouldn't bother involving the SCF with this, students are better off launching their complaint directly with the IFoA and then taking it to the FRC.

 

[StudentActuary_02]

All students will have seen the first full statement from the IFoA on their exam marking failures in today's student newsletter - some thoughts below:

“Examination marking

In recent months we have become aware of some social media activity which is alleging that there are systemic failures in the quality of the IFoA's examiners' marking, and in our examiners' ability to follow our own examination procedures. These issues have also been raised directly with us by a very small number of student members.

We take all feedback seriously, and would like to reassure the wider student membership that there are no systemic failures within our examination system.”

The IFoA have said to students in one-to-one correspondence that they don’t have to third mark papers with a material discrepancy between the two markers. However their own published policy says that they will third mark under these circumstances. Failing to apply a rule across the board is a systematic failure.

“As part of our programme of continuous improvement, we have visibility of the performance of our examiners and how examination teams adhere to our procedures.

“Visibility” is an interesting choice of words - suggests that evidence of how the process is ran is there but they don’t particularly interrogate what they see.

"We have carried out a thorough assessment of the quality of examination marking. We have looked at almost 15,000 scripts from across all the April 2016 examinations. The outcome is that, for each examination, over 90% of scripts have a variance between first and second independent markers which is within the normal academic expectation of independent marking (0-10 marks on a 100 mark total paper). However, as a leading professional body, we are working to increase this percentage further in collaboration with our examiners to ensure that all examining lies within our target range.

So, let’s be clear here, up to 10% of scripts (so almost a huge 1,500 exam entires) have a difference of more than 10 marks in the final score from each marker (I wonder what the highest difference is?), and therefore definitely should have been third marked. This is more than anyone had reason to expect before today.

I notice that the IFoA don’t comment on whether these papers were third marked, which is disappointing as this is one of the key complaints with the IFoA. The fact that the IFoA are “working to increase this percentage” is little comfort for the almost 1,500 exam scripts submitted in April 2016, which likely accounts for around 1,000 students who have not had their paper marked in accordance with the Institute’s published exam marking policy. At around £200-£400 a time, this must represent broadly £500,000 of exam fees backing an exam entry that has not been marked fairly. Particularly since October 2015, where a service contract is not carried out with “reasonable care and skill”, under the Consumer Rights Act 2015 students are entitled to the service being carried out again without charge and within a reasonable time period. All that students are asking for is for the papers to be marked in accordance with the published policy.

Regarding this new 10 mark difference standard, the Institute’s published policy to date has also only referred to “Where the first two markers disagree a script will also be third marked by an examiner.” so it has been very ambiguous what constitutes disagreement between the two markers, and certainly the threshold has not been set at a 10 mark difference until today. However, any reasonable person would say that students who have sat papers prior to today who see material differences either in the final mark, or material differences in the marks for individual questions, following a SAR surely fall with the remit of a “disagreement” between the two markers and are therefore eligible for third marking.

The IFoA also refer to a “target range” but don’t say if this is no more than a 10 mark difference - again more transparency is needed here.

"We have also checked the third independent marking of pass/fail borderline scripts and those with higher than normal variance. Appropriate further review is taking place and there is no systemic failure in the operation of this process. However, we have recognised that there is a need to improve the audit trail of this 'third marking'. We have put in place a programme of such improvement under the leadership of the Chair of the Board of Examiners. Improvements have already been made for marking of the September examinations. From initial marking, it is clear that appropriate operation of third independent review is taking place. We will be updating the Student Handbook to better communicate these third marking procedures."

Firstly, I was surprised that the (equally important) borderline script marking issue received little focus on this statement as a whole - given this is an area where the failings are more obvious I would hope the IFoA would have quickly identified those students negatively affected and put things right by marking the paper a third or fourth time. However there is no sign this has happened.

Again, I have seen multiple examples myself of students getting a borderline final mark but not being marked a third or fourth time. To repeat the IFoA’s published policy:

“Once the initial pass mark has been decided then scripts which are around the borderline will be marked a third or possibly fourth time to ensure that the examiners are happy with the proposed pass mark.”

This has not been happening and is again a systematic failure, so the claim that there has been “no systematic failure” smacks of denial by the IFoA.

"It is extremely important to us that you have confidence in the robustness of our examinations and examination process. By sharing with you the steps we are taking to monitor and improve our policies and processes, we hope this gives you confidence in your professional body and the professional qualifications that you are studying. Finally, we encourage you to use the Student Consultative Forum to raise any matters of general concern so that these can be discussed in a considered and transparent way.”

There are a lot of words here, but there is still no direct answer to why the IFoA have not been following their own published process (and seemingly not thinking this is a problem).

The unacceptable way students who have raised this directly with the IFoA have been treated by the IFoA executive (e.g. management refusing to address complaints escalated to them, instead just responding with “Communication with the IFoA on this matter is closed”) has also not been addressed in this statement, so there is still a long way to go before the IFoA have fixed the hole they are digging for themselves.

Given the way complaints have been treated to date, I was hoping for a change in tone by now, with an admission that things have gone wrong and an undertaking to fix things from the IFoA. Instead the IFoA are continuing to dig their head in the sand and maintain that there has been no systematic failure, despite the growing body of evidence from students who have done a SAR and found marking process failures.

Any thoughts on this?