Exam marking problems

[almost_there]

So, let’s be clear here, up to 10% of scripts (so almost a huge 1,500 exam entires) have a difference of more than 10 marks in the final score from each marker (I wonder what the highest difference is?), and therefore definitely should have been third marked. This is more than anyone had reason to expect before today.

That really is an astonishing revelation!! If I ran an exam system with a performance like that I'd be ashamed of myself. I congratulate the "very small number" of students for making their complaints heard - bravo!

 

[almost_there]

Given the way complaints have been treated to date, I was hoping for a change in tone by now, with an admission that things have gone wrong and an undertaking to fix things from the IFoA. Instead the IFoA are continuing to dig their head in the sand and maintain that there has been no systematic failure, despite the growing body of evidence from students who have done a SAR and found marking process failures.

Any thoughts on this?

Sadly this attitude has been there for years. I wonder just how many people need to complain sometimes in order for people to admit there's a problem and something to sort out. I think the defensive responses perhaps are partly due to their fear that admitting failures on their part could lead to individuals pursuing them in small claims court etc. If I'm right about that then it's a sad state of affairs. When insurance companies get it wrong then TCF applies and money is returned to customers etc.

 

[trevor888]

Hi guys, after reading this thread and your posts , I will proceed to file a Subject Access Request as well to verify the marks given by the 1st and 2nd marker for my CA1 exam sat in April 2016 which was extremely marginal. I am expecting to see three markers comments and the date marked. The previous post by StudentActuary_02 is raising some doubts as apparently there was no Third Marker! I am very worried about this and only realised this was an issue after reading the newsletter today.

When the newsletter came out, it was not clear whether the marginal fail scripts were marked after the publication of results or before the results. If ANY third marking was done after the publication of results for April 2016, THEN we can say that the third marking is biased or prejudiced as the third marker already knew of the fail result and has incentive not to reverse a fail decision since this will damage the reputation of the institute.

If we are not happy, The only option to escalate this is via an appeal it seems.

"
You may appeal a mark you’ve been given in an exam if you believe:

• a procedure wasn’t followed correctly;
• there was bias or prejudice in the decision-making; or
• the examiner has made a mistake.

" (from the website shared by StudentActuary_02) https://ico.org.uk/for-the-public/schools/exam-results/

 

[StudentActuary_02]

Hi guys, after reading this thread and your posts , I will proceed to file a Subject Access Request as well to verify the marks given by the 1st and 2nd marker for my CA1 exam sat in April 2016 which was extremely marginal. I am expecting to see three markers comments and the date marked. The previous post by StudentActuary_02 is raising some doubts as apparently there was no Third Marker! I am very worried about this and only realised this was an issue after reading the newsletter today.

When the newsletter came out, it was not clear whether the marginal fail scripts were marked after the publication of results or before the results. If ANY third marking was done after the publication of results for April 2016, THEN we can say that the third marking is biased or prejudiced as the third marker already knew of the fail result and has incentive not to reverse a fail decision since this will damage the reputation of the institute.

If we are not happy, The only option to escalate this is via an appeal it seems.

"
You may appeal a mark you’ve been given in an exam if you believe:

• a procedure wasn’t followed correctly;
• there was bias or prejudice in the decision-making; or
• the examiner has made a mistake.

" (from the website shared by StudentActuary_02) https://ico.org.uk/for-the-public/schools/exam-results/

Glad to hear the thread has been useful, isn't it shocking that this has been known for months and only today have the IFoA given a full comment on the situation? The SCF should have called an emergency meeting with the IFoA as soon as this came to light, followed by contacting the student body and gathered stats on how many students have been affected that they could take to the meeting.

And remember, the IFoA have up to 40 days to respond to a SAR, and you won't know if you have grounds for appeal until they do. Therefore their 15 day deadline for appeal once results are published is utterly unreasonable, and therefore invalid. Contact the Information Commissioners Office if you are having problems with the IFoA refusing to process an appeal.

I also seriously doubt they can charge hundreds of pounds for an appeal if you have evidence of a breach in the exam marking policy from your SAR, again the Information Commissioners Office should be able to help here.

I genuinely hope your paper was marked in line with the published exam marking policy, so good luck!

 

[almost_there]

And remember, the IFoA have up to 40 days to respond to a SAR, and you won't know if you have grounds for appeal until they do. Therefore their 15 day deadline for appeal once results are published is utterly unreasonable, and therefore invalid.

Excellent point. I was caught in that situation once.

 

[bannangio]

This is an interesting conversation, and I would like to thank ActuaryStudent9123116 for raising this issue and StudentActuary_02 and almost_there for their helpful points.

I sat SA4 in April 2016 and was given a mark of 55 (pass mark was 59).

I was surprised, as I thought I had done enough to pass, and decided to appeal on the basis of a procedural irregularity (this was before you needed evidence to do so - although who knows how you are supposed to get evidence these days in the 15 days needed to submit an appeal, when an SAR takes 40 days to come through).

Obviously, my appeal failed and my mark of 55 was maintained.

A few months later I came across this topic and decided to submit an SAR for my SA4 results and all internal correspondence relating to my exam sitting and appeal, primarily to find out which question I did so badly on that I failed.

This turned up some interesting results: Marker 1 had awarded me 62 marks (i.e. a pass) and Marker 2 had awarded me 60 marks (i.e. also a pass).

However, there was a third column titled ‘adopted mark’, which summed to 55.2 (i.e. a fail), and which is the mark I was given for my paper.

I went back to the Institute and asked them to double-check the information they provided, as I believed they had either provided me with the wrong person’s marks, or that I had uncovered some kind of mistake in the averaging of my marks.

The Institute have now responded and let me know that my marks were ‘normalised’ in order to ensure discrepancies between markers do not unfairly disadvantage students.

The have also confirmed that my ‘normalised mark’ was reviewed by senior examiners and deemed appropriate, although no evidence of this was provided as part of my initial SAR.

I’m not sure where to go from this to be honest (probably nowhere if I pass it on Thursday!) but thought this tale might be of interest to others.

In the interests of balance, it is worth mentioning that I also failed CA3 and included this in my SAR. One marker passed me (just) and the other gave me a fail. The two marks were averaged (to a fail), and then my script was reviewed by a third actuary who confirmed that taken as a whole, a fail was appropriate.

 

[StudentActuary_02]

This is an interesting conversation, and I would like to thank ActuaryStudent9123116 for raising this issue and StudentActuary_02 and almost_there for their helpful points.

I sat SA4 in April 2016 and was given a mark of 55 (pass mark was 59).

I was surprised, as I thought I had done enough to pass, and decided to appeal on the basis of a procedural irregularity (this was before you needed evidence to do so - although who knows how you are supposed to get evidence these days in the 15 days needed to submit an appeal, when an SAR takes 40 days to come through).

Obviously, my appeal failed and my mark of 55 was maintained.

A few months later I came across this topic and decided to submit an SAR for my SA4 results and all internal correspondence relating to my exam sitting and appeal, primarily to find out which question I did so badly on that I failed.

This turned up some interesting results: Marker 1 had awarded me 62 marks (i.e. a pass) and Marker 2 had awarded me 60 marks (i.e. also a pass).

However, there was a third column titled ‘adopted mark’, which summed to 55.2 (i.e. a fail), and which is the mark I was given for my paper.

I went back to the Institute and asked them to double-check the information they provided, as I believed they had either provided me with the wrong person’s marks, or that I had uncovered some kind of mistake in the averaging of my marks.

The Institute have now responded and let me know that my marks were ‘normalised’ in order to ensure discrepancies between markers do not unfairly disadvantage students.

The have also confirmed that my ‘normalised mark’ was reviewed by senior examiners and deemed appropriate, although no evidence of this was provided as part of my initial SAR.

I’m not sure where to go from this to be honest (probably nowhere if I pass it on Thursday!) but thought this tale might be of interest to others.

In the interests of balance, it is worth mentioning that I also failed CA3 and included this in my SAR. One marker passed me (just) and the other gave me a fail. The two marks were averaged (to a fail), and then my script was reviewed by a third actuary who confirmed that taken as a whole, a fail was appropriate.

As for next steps, I would recommend speaking to the ICO about whether you are entitled to receive information on why that level of normalisation was applied, with supporting evidence. Given an adjustment was applied to the marks awarded by both markers, I think you are entitled to know how the adjustment was calculated, but I will let them confirm formally and then you have something strong to go back to the IFoA with!

Give the IFoA a reasonable period to respond (again the ICO should be able to advise on a reasonable timeframe), and then make sure they stick to it. The ICO can intervene if they refuse to deal with your follow-up queries relating to a SAR response and may issue a "Letter of Concern" to the IFoA directly if they are being unresponsive or refusing to provide data they hold.

Finally, am genuinely glad to hear that your CA3 paper was marked in line with the policy.

 

[almost_there]

bannangio, wow, I find that jaw-dropping. If they normalise marks then set a pass mark then the pass mark 'transparency' is worthless.

Look, if the IFoA came out tomorrow and said "We are running a quota system. We normalise marks. We don't third mark. We take average of marker 1 & 2. We only let top 40% pass these exams".... you know what? I'd respect them for declaring these things. I'd then work as hard as I could to get into the top 40%. Why on earth is that so difficult? It's like anything in life where a challenge is set- declare the rules of the game, let people play the game to the best of their ability, determine the winner according to the rules of the game. Transparency is paramount for people to have confidence in the game and participate in it. There's no use being dismissive to complainants or hiding anything at all. To be quite honest with you, although it is denied that there is a quota system at play, after I failed a couple of CT exams I then went about to study to a level of detail, with the assumption in mind that there WAS a quota system in operation, so that I left the exam thinking "I definitely beat 60% of entrants with that script". Funnily enough I then passed like 10 exams in a row with that mentality. As when you score so highly these issues don't matter to you. Secretly I believe people have that belief of a quota system, which is why they'll study harder in order to pass. Such people can get qualified then declare no further interest in the process or in anyone being hard done by on the borderline. However that's not the full story since if there are large marking discrepancies in CT exams, which are mostly mathematical and not so open to opinion, then God help folks when we get to SA or CA3 stage where the opinion of markers carries huge weight.

 

[almost_there]

If I may repeat once more my thoughts that the reason they won't own up to any failings is because they have some solicitors there advising them not to, so that they may not be left open to compensations in the courts, such as the small claims court. Seriously, imagine a proper judge getting this in front of them... !

 

[trevor888]

If I may repeat once more my thoughts that the reason they won't own up to any failings is because they have some solicitors there advising them not to, so that they may not be left open to compensations in the courts, such as the small claims court. Seriously, imagine a proper judge getting this in front of them... !

Agreed! Any legal action or trial involving them would turn very ugly.

 

[Gavin Kelly]

This is an interesting conversation, and I would like to thank ActuaryStudent9123116 for raising this issue and StudentActuary_02 and almost_there for their helpful points.

I sat SA4 in April 2016 and was given a mark of 55 (pass mark was 59).

I was surprised, as I thought I had done enough to pass, and decided to appeal on the basis of a procedural irregularity (this was before you needed evidence to do so - although who knows how you are supposed to get evidence these days in the 15 days needed to submit an appeal, when an SAR takes 40 days to come through).

Obviously, my appeal failed and my mark of 55 was maintained.

A few months later I came across this topic and decided to submit an SAR for my SA4 results and all internal correspondence relating to my exam sitting and appeal, primarily to find out which question I did so badly on that I failed.

This turned up some interesting results: Marker 1 had awarded me 62 marks (i.e. a pass) and Marker 2 had awarded me 60 marks (i.e. also a pass).

However, there was a third column titled ‘adopted mark’, which summed to 55.2 (i.e. a fail), and which is the mark I was given for my paper.

I went back to the Institute and asked them to double-check the information they provided, as I believed they had either provided me with the wrong person’s marks, or that I had uncovered some kind of mistake in the averaging of my marks.

The Institute have now responded and let me know that my marks were ‘normalised’ in order to ensure discrepancies between markers do not unfairly disadvantage students.

The have also confirmed that my ‘normalised mark’ was reviewed by senior examiners and deemed appropriate, although no evidence of this was provided as part of my initial SAR.

I’m not sure where to go from this to be honest (probably nowhere if I pass it on Thursday!) but thought this tale might be of interest to others.

In the interests of balance, it is worth mentioning that I also failed CA3 and included this in my SAR. One marker passed me (just) and the other gave me a fail. The two marks were averaged (to a fail), and then my script was reviewed by a third actuary who confirmed that taken as a whole, a fail was appropriate.

That is disgusting. Honestly is. Why should you have to go through the stress and cost of resitting when the two markers gave you a pass. What is going on here?

As for CA3, was your papers third marked, or just reviewed? If third marked then surely it is reasonable to expect to get the same breakdown of marks as per the 1st two markers. I am in the same situation with CA3, and I have requested it to be third marked as per the guidance in the student handbook. I was told that it was only 'reviewed' and my final mark was deemed appropriate.

See the forum on this: https://www.acted.co.uk/forums/inde...pass-mark-technical-difficulties.13044/page-3

 

[Teacher's Pet]

This is an interesting conversation, and I would like to thank ActuaryStudent9123116 for raising this issue and StudentActuary_02 and almost_there for their helpful points.

I sat SA4 in April 2016 and was given a mark of 55 (pass mark was 59).

I was surprised, as I thought I had done enough to pass, and decided to appeal on the basis of a procedural irregularity (this was before you needed evidence to do so - although who knows how you are supposed to get evidence these days in the 15 days needed to submit an appeal, when an SAR takes 40 days to come through).

Obviously, my appeal failed and my mark of 55 was maintained.

A few months later I came across this topic and decided to submit an SAR for my SA4 results and all internal correspondence relating to my exam sitting and appeal, primarily to find out which question I did so badly on that I failed.

This turned up some interesting results: Marker 1 had awarded me 62 marks (i.e. a pass) and Marker 2 had awarded me 60 marks (i.e. also a pass).

However, there was a third column titled ‘adopted mark’, which summed to 55.2 (i.e. a fail), and which is the mark I was given for my paper.

I went back to the Institute and asked them to double-check the information they provided, as I believed they had either provided me with the wrong person’s marks, or that I had uncovered some kind of mistake in the averaging of my marks.

The Institute have now responded and let me know that my marks were ‘normalised’ in order to ensure discrepancies between markers do not unfairly disadvantage students.

The have also confirmed that my ‘normalised mark’ was reviewed by senior examiners and deemed appropriate, although no evidence of this was provided as part of my initial SAR.

I’m not sure where to go from this to be honest (probably nowhere if I pass it on Thursday!) but thought this tale might be of interest to others.

In the interests of balance, it is worth mentioning that I also failed CA3 and included this in my SAR. One marker passed me (just) and the other gave me a fail. The two marks were averaged (to a fail), and then my script was reviewed by a third actuary who confirmed that taken as a whole, a fail was appropriate.

Just an observation and not sure if anyone agrees. If the adopted mark is 55.2, it would mean that it was seen by a multiple of examiners(committee possibly?) at the same time? Otherwise, the average number would not have ended with a decimal of .2?

This is assuming each bullet point is still worth 0.25marks and material points are worth 0.5marks.

 

[StudentActuary_02]

That is disgusting. Honestly is. Why should you have to go through the stress and cost of resitting when the two markers gave you a pass. What is going on here?

As for CA3, was your papers third marked, or just reviewed? If third marked then surely it is reasonable to expect to get the same breakdown of marks as per the 1st two markers. I am in the same situation with CA3, and I have requested it to be third marked as per the guidance in the student handbook. I was told that it was only 'reviewed' and my final mark was deemed appropriate.

See the forum on this: https://www.acted.co.uk/forums/inde...pass-mark-technical-difficulties.13044/page-3

Good spot Gavin, the third mark wasn't stated in bannangio's message, so we can only assume it wasn't provided in the SAR. I withdraw my relief regarding his CA3 exam!

 

[bannangio]

Good spot Gavin, the third mark wasn't stated in bannangio's message, so we can only assume it wasn't provided in the SAR. I withdraw my relief regarding his CA3 exam!

To clarify this, my CA3 paper was not third marked, but reviewed by a third marker who added additional comments as to why he/she believed a fail was appropriate.

The comments (both from the first 2 markers and the reviewer) are very helpful – if I end up failing CA3 again I will probably do another SAR just for these comments.

Just an observation and not sure if anyone agrees. If the adopted mark is 55.2, it would mean that it was seen by a multiple of examiners(committee possibly?) at the same time? Otherwise, the average number would not have ended with a decimal of .2?


This is assuming each bullet point is still worth 0.25marks and material points are worth 0.5marks.

The ‘adopted marks’ column in my SAR gave a mark to 1d.p. for each part question, e.g. 4.1, 6.5, 2.8 etc. I can’t see a committee going through multiple papers at that level of detail, so it would be interesting to hear from anyone else who has submitted an SAR for an SA subject and whether they have this column or not.

My feeling (with no evidence to back it up) is that some kind of formula or computer programme is applied to each marker’s marks to scale them up or down depending on how generous / strict a marker is.

I have tried backsolving for these scaling factors and it appears a different one is applied to each part question.

I suspect that use of the word ‘normalisation’ in the Institute’s correspondence with me was not an accident, and that the actual method for scaling marks involves fitting given marks to a normal distribution somehow.

If I fail again on Thursday I may go back for more details on the scaling method, but I’m not sure what information they would actually give me.

 

[almost_there]

I don't see why any marks need to be adjusted. Why don't they just look at the exam paper, set the pass mark, then let average of 2 markers determine your pass/fail? If markers are more than 10 marks out from each other then (a) train your markers better (b) if one marker persists in generating low/high scores don't use again (c) 3rd/4th mark it. Simples.

 

[almost_there]

Having just read the student consultative forum notes for Nov 18, which have just been published on their website, it appears they don't really find anything wrong with anything, is the impression I get. They'll only concede and take action on minor points regarding exam timetables or minor technical issues.I think the SCF is a waste of time, just a place for them to direct complaints at to be left alone. Then they even filter out some complaints so they don't even get raised, such as the Q1 Nov 2016 CA3 issues I've raised. There appears to be no follow-up on the actions raised in previous meetings. Utterly pointless.

 

[almost_there]

Quoting from their notes Nov 18 :

A discussion on third marking followed. It was agreed that further clarity was required. There were expectations that a whole script was remarked, but it was identified that this was not always the case. Examiners could in fact review or remark only one or two questions where there were significant variances, may remark the complete script, or may review the complete script. Any reviewing or remarking of scripts was confirmed as being performed by the Principal Examiner or a similarly experienced examiner in the subject team. The Principal Examiner may also standardise the initial marking to normalise any variations to ensure that no candidate is disadvantaged prior to any third stage moderation. It was explained that at this stage the Principal Examiners have the authority to apply their academic judgement to determine who a competent candidate is. Such candidates will demonstrate they have competency in the concepts and principles of the subject.

A question was asked around the CA3 exam which was the initial examination of concern and it was explained that both CA2 and CA3 were still being worked through. However on the initial exams reviewed there were no major concerns.

It was agreed that in the light of these discussions the IFoA would revise the content of the Student Handbook on the marking process as part of its commitment to transparency.​

 

[almost_there]

Personally I find it a shambles they have no proper process on third marking and don't think it's adequate to spin it that remarking one question or 'reviewing', whatever that is - to me that's not remarking at all - is equivalent to any serious understanding of what's meant by third marking.

Once again I smell the influence of a lawyer here, expanding the definition of third marking so much that they can't fail to have done it, even if not a single question has been remarked, only 'reviewed'! This is so that they can more easily defend themselves in the event a student becomes annoyed enough to take them to a small claims court for failing to follow their stated process. I am speculating they must be quite worried about such a possibility. Seriously, imagine the potential liability!

So all they'll do is, as they've stated, amend the wording in the student handbook so that in effect they allow themselves total discretion over third marking and what that means, so that their principal examiner can look at your script for 1 second and say "looked at it, keep original decision" and then they can claim third marking has been satisfied. It has nothing to do with their commitment to transparency, that's just spin.

In my experience, this is all complaining to them ever achieves, them tightening some aspect of their wording just so they can bat away that complaint next time easier. This is not the same as actually correcting the problem and making a redress. In a funny way by complaining we are just acting as free quality control consultants.

I'll discuss the examiner amending the pass mark after 3rd/4th marking in another thread.

To conclude, I believe that third marking should mean marking the ENTIRE SCRIPT. Page 70 student handbook states "scripts marked (1st and 2nd)", then "scripts marked (3rd)". Not scripts reviewed, not some questions remarked but "scripts marked" and this description is consistent between 1st, 2nd & 3rd. Whether they change that wording now, they can't go back in time and change their wording retrospectively so don't let them fob you off with moving the goalpost here since any change in wording doesn't apply until the next exams.

Clearly from the examples given & their admissions scripts have not been fully marked a third time as they should have.

 

[S&GMummy]

Dear all

I am new to this forum. I have just got the results from last Thursday, I failed SA2 (my last exam) by one mark, I have 61 and pass mark was 62.

I have already accepted the fact that I have failed, but many people around me, some being actuaries and some aren't, most think that I should appeal. What will you do if you were me?

Having read the this thread, if I am going to appeal, do I have enough ground? Or should I just raise a Subject Access Request straight away. I am very confused and hope you guys can help me. Thanks.

 

[Gavin Kelly]

S&GMummy. I would 100% put in a SAR right away. This would provide you with some evidence to appeal against. It is clearly a borderline case for you and so I would expect you to have had your paper marked a 3rd time. If it has not been then that would at least give you grounds to get it marked again.

The IFoA have failed to follow their stated marking process before a number of times, and I have no confidence in them to prevent that from happening again in the future. For my recent CA3 result they have not followed their marking procedure and are refusing to admit to their mistake and mark my paper a 3rd time, which I am still fighting with them on almost 9 weeks after the results were published.

The appeal system is flawed, you only have 15 days to appeal, but they have 40 days to provide you with results of your SAR. It is clearly unfair. I would not appeal, as they will only just take £200 from you, and provide you with a 1 paragraph answer saying that your appeal has been rejected.

 

[S&GMummy]

Gavin, if I raise a SAR and not submit an appeal form at the same time, will they invalidate my appeal?