15 marks difference between exam markers

[Maxit]

I was unsuccessful with SA2 in the April 2018 sitting and my subject access shows marker 1 placed me 7 marks above the pass mark and marker 2 placed me 8 marks below the pass marks. Overall this comes to a difference of 15 marks between the two markers!

Someone please explain to me how this is even possible?! Is this how subjective the.marking is?

Eventually I was averaged to fail 2 marks below the pass mark. I'm devastated.

 

[Viki2010]

That's normal. I had a similar situation with CA3. You don't have any other choice than to retake the exam and hope for the best next time.

 

[almost_there]

At the very least you should write to IFoA to ask them to address the matter, despite us all knowing almost certainly they'll do nothing except expect you to come back and try it again. If you don't give them a chance to respond then no one else e.g. Financial Reporting Council will look at it unless IFoA have had a chance to address it first. Re Sitting an exam like SA is no joke and people shouldn't be subject to a marker lottery, it's simply not good enough and you have my full sympathy Maxit.

 

[Dom B]

I was unsuccessful with SA2 in the April 2018 sitting and my subject access shows marker 1 placed me 7 marks above the pass mark and marker 2 placed me 8 marks below the pass marks. Overall this comes to a difference of 15 marks between the two markers!

Someone please explain to me how this is even possible?! Is this how subjective the.marking is?

Eventually I was averaged to fail 2 marks below the pass mark. I'm devastated.

Out of interest, what was the difference between the total of the highest marks given by an examiner for each question versus the total of the lowest mark given for each question (rather than the total for examiner 1 versus total for examiner 2)? I expect the difference would be higher than 15 marks.

I recently found a difference of 4.5 marks on my CT8 paper and that is of course a largely mathematical paper. Given this happened on CT8, I expected that a marking difference of 10-15 marks was possible for the later exams where marking is more subjective. However, from your experience it seems that I underestimated this and the difference could be more like 15-20 marks!

This further reinforces my view that these exams are largely a game of roulette. I am not discrediting those who have passed them, I just think that a large number of students who fail are no less knowledgeable or less prepared than those who passed - they simply drew the wrong paper or the examiner did not interpret their answers correctly.

When marking is clearly this subjective and therefore varies to this extent, there is a high probability that you wrote a better SA2 paper than many of the students who scored 2-3% higher than you and ended up passing.

You have my full sympathy.

 

[Dom B]

That's normal. I had a similar situation with CA3. You don't have any other choice than to retake the exam and hope for the best next time.

I think you just summed up the exams perfectly when you said "hope for the best".

 

[Maxit]

Out of interest, what was the difference between the total of the highest marks given by an examiner for each question versus the total of the lowest mark given for each question (rather than the total for examiner 1 versus total for examiner 2)? I expect the difference would be higher than 15 marks.

I recently found a difference of 4.5 marks on my CT8 paper and that is of course a largely mathematical paper. Given this happened on CT8, I expected that a marking difference of 10-15 marks was possible for the later exams where marking is more subjective. However, from your experience it seems that I underestimated this and the difference could be more like 15-20 marks!

This further reinforces my view that these exams are largely a game of roulette. I am not discrediting those who have passed them, I just think that a large number of students who fail are no less knowledgeable or less prepared than those who passed - they simply drew the wrong paper or the examiner did not interpret their answers correctly.

When marking is clearly this subjective and therefore varies to this extent, there is a high probability that you wrote a better SA2 paper than many of the students who scored 2-3% higher than you and ended up passing.

You have my full sympathy.

Using the higher of the two for each question brings the difference to 19 marks which is within the range you say

 

[Dom B]

Using the higher of the two for each question brings the difference to 19 marks which is within the range you say

So based on your scores for two examiners:
According to the lowest marks awarded for each question you could have a total of 45%.
According to the highest marks awarded for each question you could have a total of 64%.

And i'm sure that there are other students who have a variance of greater than 19 marks in their SAR breakdown.

Peoples careers are being decided by this system.

Hopefully when the wheel spins again in September your number will come up. Good luck.

 

[almost_there]

Has the paper been third marked?

 

[Maxit]

Has the paper been third marked?

Yes - comes to the average of the two markers. Surprise surprise.

 

[almost_there]

Yes - comes to the average of the two markers. Surprise surprise.

=Average(M1:M2)

 

[almost_there]

Shouldn't IFoA provide an explanation for this discrepancy and why they consider an average to be acceptable in these situations. It doesn't follow at all that an average is appropriate- it could be that one marker is consistently harsh compared to average.

 

[almost_there]

Peoples careers are being decided by this system.

Indeed. For some reason they haven't disclosed exactly how they get from raw marks to final mark etc. I don't find such lack of transparency acceptable, especially given the efforts people make towards these exams and the consequences of failing on pay, study packages etc.

 

[Jamie Brown]

Yes - comes to the average of the two markers. Surprise surprise.

Their guidance, published last year (?), states that they use the average when both marks are above or below the pass mark and not too close to it. It might just be a coincidence it's ended up where it has - let's face it, there's a pretty high chance it would be close. An examiner would have looked at your script after the first two markers, and in particular the questions where the marks differed, and decided what the appropriate score was for each question. I don't know why they don't publish this bit - you could ask perhaps?
Looking at the higher of the two all the way through is pretty pointless. You could look at the lower of the two instead - but just as pointless.
Treat it like a lottery if you wish, but then a high chance you'll fail again - unless you just get lucky. Your odds are highly likely to be increased if you work hard and effectively for the resit.
Good luck!!!

https://www.actuaries.org.uk/studying/exam-results/marking-guidelines

 

[almost_there]

Treat it like a lottery if you wish, but then a high chance you'll fail again - unless you just get lucky. Your odds are highly likely to be increased if you work hard and effectively for the resit.
Good luck!!!

This does not address the issue, which is inconsistent marking and is an attempt to shift the blame for failing onto the student who is the victim of inconsistent marking.

 

[Tarbuck]

I work for a large insurer (I think the biggest in the UK) with a huge graduate scheme so I've seen a lot of people go through the exams. I have to say, there seems to be a remarkable trend where some people manage to pass this sequence of lotteries with little to no passes and others struggle, repeatedly failing the exams (especially the wordy ones).

Some would say this trend would demonstrate its not luck, but they just have untrained eyes. Luckily you can see through it to the conspiracy that's really in place. I wonder how they choose the candidates they want to fail? I'm guessing names form a hat.

 

[Maxit]

I work for a large insurer (I think the biggest in the UK) with a huge graduate scheme so I've seen a lot of people go through the exams. I have to say, there seems to be a remarkable trend where some people manage to pass this sequence of lotteries with little to no passes and others struggle, repeatedly failing the exams (especially the wordy ones).

Some would say this trend would demonstrate its not luck, but they just have untrained eyes. Luckily you can see through it to the conspiracy that's really in place. I wonder how they choose the candidates they want to fail? I'm guessing names form a hat.

Not nice to be sarcastic about inconsistent marking for such an important and expensive exam.
I've spoken with a colleague who used to be a chief examiner at one point. He said that markers do receive the same student's paper if the student was resitting an exam and the same marker happened to be around. However that's not always the case.

Anyway I've raised a complaint about the issue with the IFoA and waiting to hear back although not with high hopes.

This is my final exam to fellowship and I'm not sure how confident I'll be in the next sitting where I may possibly convince one marker but fail because the second marker wasn't convinced.

 

[Dom B]

I have to say that one or two comments on this thread seem rather insensitive and lacking in maturity.

Maxit missed the pass mark on a tough SA paper by 1% and discovered that there was a 19% variation between examiners' marking. That seems like a very legitimate concern for Maxit to raise, particularly as these exams frequently come down to fine margins (many students who pass and fail ST/SA exams do so by only a few %). This is a person who missed out on Fellowship by 1% when sitting an SA paper that only 38% of students passed - so is clearly a capable individual worthy of some respect.

To attempt to re-frame such a genuine concern and it's implications as 'conspiracy' theory is frankly childish.

I'm sure that there are a small minority of people who get through these exams with little or no fails, but these are rare (i.e. if only 38% passed SA2 and 31% passed CT8, evidently a vast majority of students are failing exams on a regular basis). The vast majority actually pass and fail these exams by a margin of a few %, so a marking variation of 19% between two examiners means that a vast majority of pass/fails are largely subjective.

That logically means for a vast majority of well prepared students, this is roulette. And if you've ever sat at a roulette table, you will have seen some people continually winning, some intermittently winning/losing and others continually losing. However, the difference at the roulette table is that the players who win generally recognise that they have been lucky rather than being 'more competent roulette players' than those who have lost.

 

[Jamie Brown]

used to be a chief examiner at one point. He said that markers do receive the same student's paper if the student was resitting an exam and the same marker happened to be around. However that's not always the case.

If two markers randomly mark, say 80 scripts - then the chances you'll be marked by the same marker will depend on marker turnover and the total number of scripts .

Not sure if you thought my first post was lacking in maturity. I would say quite the opposite. Just encouraging good revision and suggesting that hard work for an exam pays off. I think it's far less sensical to just suggest that its a lottery.

 

[Jamie Brown]

I'm not sure how confident I'll be in the next sitting where I may possibly convince one marker but fail because the second marker wasn't convinced.

If you manage to do that then you're in a good place - because your script will then be looked at by one of the more senior examiners. If they don't think that you should pass then I'm afraid that you porbably haven't oen enough. But at least you are then in consideration. If you don't get enough marks from any of your first two markers then you won't get a third shout - unless you're very close.

Inconsistency in marking can happen for very many reasons - in an ideal world it wouldn't happen but it does at all levels.

 

[Geraldine]

Yes - comes to the average of the two markers. Surprise surprise.

Yes, I've also had this - two of my SAR breakdowns had third sets of marks which were the bang-on average. It's simply not believable, especially when you realise others have suffered this same, dubiously predictable fate! Why won't they listen to us?

 

[Dom B]

Yes, I've also had this - two of my SAR breakdowns had third sets of marks which were the bang-on average. It's simply not believable, especially when you realise others have suffered this same, dubiously predictable fate! Why won't they listen to us?

The probability that the third marker gives the exact average of the previous two markers on several occasions is almost zero. If examiner 3 were truly independent of the previous 2, then it would be equally likely that they would give exactly the same mark as examiner 1 or examiner 2, as they would an exact average (there is no reason to suggest that examiner 1 & 2 lie at the extreme ends of the scale and every other examiner would be somewhere in between)

If you think about it carefully, at present, this leaves the IFoA with two undesirable options to choose from:

1) Ensure that the third marker engineers an almost exact average so we can maintain the facade that both our marking process and examiners are near 100% reliable, even though it will be obvious to the more perceptive students that this 3rd mark has been engineered;

2) Allow the 3rd examiner to mark the paper independently of the previous two examiners and run the almost certain risk that their mark will not be an average of the previous two examiners, or worse still gives a more extreme score than 1 or 2. This will be an admission that either our marking process is unreliable and highly subjective or that one of the two original examiners are likely to be unreliable (which of course implies that passing or failing exams by a few % comes largely down to luck).

Essentially, the IFoA have decided that 1) is the lesser of two evils.