You will find a link to the FIG's newly published book of results at the Olympic Games here. This year, they have broken down the judge's execution scores so you can see exactly how each judge evaluated the gymnasts' performances. It makes for interesting reading - if only I had more time to analyse each judge's marking. A skim reading already highlights multiple inconsistencies in individual judges' marks and makes you wonder why they bother with the jury at all.
I have taken the time to look at the reference judges' scores for the top four in the women's all around. The FIG explains here what their role is, and how they are selected. I even used my calculator, which is a risky thing in my hands. My, how I wish we could have seen a similar document for the Tokyo World Championships.
I wonder if anyone can explain how, if the FIG's Code of Points is so objective and fair, it is possible to come up with two different results using two different sets of judges? Presumably the reference judges are expected to be highly reliable in their evaluation? Here is what the result of the women's all around competition would have been, according to the reference judges. It casts a whole different light on the competition, one which, satisfyingly or frustratingly, is reflected by the judgement of many on the gymternet.
1 Viktoria Komova 62.1
2 Gabby Douglas 61.75
3 Aliya Mustafina 60.05
4 Alexandra Raisman 59.2
One of the key differences appears to be in the bars scores, where the e-score situation for Douglas and Komova is more or less reversed by adopting the reference judges' score over the jury's. Bars scores had seemed particularly inconsistent all week.
Valentina should have a field day with this document. I am not sure what conclusions to draw, but I hope you have fun reading it. Do comment, please.
I can agree with this. I'm American but I stand by my thoughts on ms. Komova winning the all around. The judges were inconsistent in almost every aspect of the comp.
ReplyDeletekeep going
ReplyDeleteI think there should be an exchange of medals, and a new ceremony... is this petition possible? Judgement can´t be that subjective, there is a code and sufficient technology on cameras to prove this kind of errors...
ReplyDeleteI doubt a petition would have much effect, unfortunately. Despite the fact that the system has flaws, the medals were awarded according to the rules.
DeleteIMO marking gymnastics will always be subjective. The only problem is pretending objectivity. The real question is - as far as I am concerned - how do they make the marking fair, and promote the growth and development of gymnastics that is aesthetically pleasing?
One problem is, an unstated assumption exists within the basic premises of the current Code of Points - that artistry is undesirable, mostly because it is impossible to evaluate fairly. This position is unfounded. There are methodologies that could foster fair evaluation of artistry, but the FIG seems unwilling and/or unable to consider them.
Thanks for your answer, I visit every day your blogspot because I´m an all time russian gymnastics fan, I really appreciate all the information you bring to us.
DeleteThanks again,
Barbara (from Chile!)
Thanks Barbaera O:-)
Delete"artistry is undesirable, mostly because it is impossible to evaluate fairly."
DeleteYou make a very good point! However I also agree with you when you say there are methodologies that could create a fair system of evaluation. I hope to see them over the coming years.
The FIG has a software tool to calculate the bias of every judge related to the reference judge. They know what happens at every competition very well.
ReplyDeleteMeaning? Could you elaborate please?
DeleteAll the judges'scores are analyzed related to the supervisors' scores. It's not difficult to calculte the judges who are always "harsh" towards a country's gymnast.
DeleteDoes it take into account the possibility of a collective bias?
DeleteI still feel like USA should win cause i think gabby did do the best all around. Komova took some serious steps on her vault that could have easily got more deductions and Gabby seemed to be more consistent in each routine even though she wasn't the highest score in each.
ReplyDeleteOk, Douglas didn't take steps on her Amanar, but she had other mistakes in form (bars, beam and floor). If the judges can't judge execution, there's a serious problem in gymnastics!!!
DeleteHow much time and work does it take to train a gymnast with Komova's or Grishina's execution?
With the same standard you've just put, then Maroney shouldn't have gotten the silver after she SAT her second vault not just few steps like Komova ... Double standard alert!!
DeleteIt's not a double standard because Maroney's point system for both vaults were higher than the rest. The fact that she sat on her butt and still had enough points from ONE vault to earn the silver medal tells everything you need to know. Her successful completion of her first vault already guaranteed her a medal - it was really up to her to screw things up, and not anyone else.
Delete@Anonymous What exactly where Douglas' mistakes in form on bars, beam and floor? I would like to know.
Deleteplz make komova win the gold medal!!!! she def deserved it!!! I petition for a exchange of medals!!!!
ReplyDeletepoor komova...
ReplyDeleteThis is so unfair...Komova should have won and most people know it. Kind of like Khorkina all over again..
ReplyDeleteBoth Komova and Khorkina did not deserve it on the day! I am a Russian fan myself and I know Komova could have won but you all need to stop whining! Gabrielle's performances were superior to Komova's all around, and therefore she deserved the gold. I would question Gabby's qualifying though..she should have been the third American and then would have been knocked out of the All Around instead of Jordyn Wieber.
DeleteI think that Komova´s bars and floor were proof enough for her Gold Medal... Komova´s majestic execution, awesome difficulty and beautiful dance movements are the best example of what gymnastics is about
ReplyDeleteAnd yet her four steps off the mat plus the form, height, distance, twisting, direction and landing errors on vault are the best example of how *gold medals* are lost. She may well be the better gymnast, but she didn't bring it across four apparatuses on the day and the mental side is just as important as physical. Move on.
DeleteI completely agree. If Vikuska had been an Olympic Champion, God would have made her one. Champions are role models to the rest of the field, both psychologically and physically. They don't just overcome hardships, they crush them down so as to never meet them again. 'How much time and work does it take to train a gymnast with Komova's or Grishina's execution?' well.. do you know how much time and work does it take to train a gymnast who has had a disastrous first televised competition (visas) and had been bullied in the gym? Gabrielle has come back fighting from one of the worst performances I have ever seen a gymnast do. Vik couldn't even be bothered to fight till the end of her final performance at the 2012 Olympics. Her shoulders said 'pfff' on more than one occasion. That God (and parent) given talent make her a top gymnast, but not at all an Olympic Champion.
DeleteSandra
It's called " All-around Final " not " Vault Final " and speaking of VT final Maroney sat (not few steps like Komova) her second vault but still managed to get silver!!!
DeleteI must confess it's the first time I have heard God cited as support in a sporting decision.
DeleteWho was the best gymnast on the day? There have been two different decisions about that from two different judging panels. The difference between the total scores is hardly negligible and results in different winners. Doesn't this make the judging of the whole competition rather a fiasco? Subjective or not, it should be possible for judges to agree who was the best on the day ... which, think about it, they haven't.
I do not agree that Gabby's personal circumstances have any bearing on this argument. Medals are awarded for gymnastics, not personality, and Komova has more character than most. Yes, she bursts into tears very easily and tends to sulk. I lost count of the number of grown men I saw crying at the Olympics. It isn't a criminal offence, it is an expression of emotion. People cry because something matters to them. I fail to see why teenage girls should have to be so 'righteous' when a 35 year old Azerbaijani wrestler can sob his guts out over winning a silver medal when he thought he had won the gold.
I wasn't talking about Vika's body language off the apparatus, I specifically meant her presentation on the apparatus. She did not fight as hard as she could when the medal was not at stake. Any sport is more than about just about winning and so are (especially) the Olympic Games. Gabrielle's 'personal circumstances' line was an answer to the one quoted before, not directed at the judge's decision. And I have nothing against anybody's crying, I do that a lot myself as an athlete.
DeleteSandra
Hi Sandra, I agree that Vika can sometimes appear to just give up, but I don't think anyone can say that she didn't give her all in the all around ...
DeleteShe is a technique gymnast rather than one who musckes her way through her routines. This is what makes her work appear so effortless and when it comes off produces moments of genius such as her floor.
It's always so easy to judge whether a person is making an effort or not, no?
DeleteA key point of professional performance, be it in sport or anything really, is to make it seem effortless and I think to say someone 'did not fight as hard as she should' is something subjective and not really an aspect you have the right to judge.
Only Komova knows whether she fought or gave up.
"If Vikuska had been an Olympic Champion, God would have made her one."
DeleteI think that takes the cake as possibly the silliest comment I've ever heard relating to a gymnast's performance. God doesn't make elite gymnasts, good genes and a lifetime of training, hard work and sacrifice do.
I find it interesting how it's definitely being discussed that the entire 5 judge jury panel overscored douglas and underscored komova but missing that there seemed to be large discrepancies between the two reference judges themselves. I do agree with your article that there were areas in individual scoring where there seemed too large a gap between the jury E score and ref judge E score, but it seems there are inconsistencies between the ref judges as well. So when you also add that to the mix, was there really such a huge gap in E scores from the jury to the ref judges?
ReplyDeleteFor example lets go ahead and use the all around scores. On vault komova got an E score of 9.066 from the jury, and 9.00 from the ref judges but one of those ref judges gave her an 8.8 and the other a 9.2. That's considerably different. Same on beam for Douglas who received an E score of 9.0 from the jury and 8.8 from the ref judges but one was an 8.6 the other a 9.0. Same thing...a 0.4 difference which amounts to huge scoring differentials once you start taking into account the accumulative scores at the competition.
So I'm not saying that komova should have won or that douglas deserved her win but it's nice to point out that there's not just bad scoring between the judges jury panel but also the supposed 'highly trained' reference judges. I haven't done a super thorough math crunch yet but I actually really plan to because I'd like to know what consistency (less then 0.2 difference in e-scores I'd say is consistent) would have done for many of these athletes. I have already done a few and noticed that the top 8 going into event finals would certainly have been altered.
I wonder how it would be possible to enforce a more strict juding code that would force fair judging? Probably can't be done just like you ultimately can't force a jury to decide a guilty/not guilty verdict you can only try to convince them at your best. Sigh to subjectively judged sports. haha.
IMO forcing judges to 'comply' and to award marks that agree, more or less, to those of their colleagues merely encourages a herd mentality where judging comes down to the lowest common denominator of what can easily be agreed amongst them; switching off individual judgement and thinking.
DeleteIt's no wonder we see judging that focusses on error deduction rather than true evaluation of whole routines.
I tend to agree with the viewpoint that a framework of objectivity would be more useful in encouraging judges to evaluate whole routines. A discussion of 'what makes this a good piece of work'? would highlight the strengths of a piece of work as well as allowing for deductions for obvious errors, and level the playing field for gymnasts like Komova, who possesses qualities that are clearly superior to the rest of the field, yet is punished disproportionately for her errors.
I meant to say ... a framework of *qualitative* objectivity.
DeleteSo just did that massive number crunching tonight and found some interesting results! Banking on what seems a group agreement from the comments on here that we should consider the E scores from the two reference judges over the judges jury panel it seems to have drastically altered what would have been several medal outcomes! I'm certainly not the head of the FIG but if my two 'highly trained' reference judges scoring would have caused a change in the podium on multiple events I would think they would want to look into that.
DeleteNotably, using these REFERENCE JUDGES E Scores over the JURY panel of judges this is what I found (man have to do with swaps thanks to the 2 per country rule):
EVENT FINAL QUALIFICATIONS:
-Diana Chelaru (ROU) would have qualified over Catalina Ponor (ROU) for the 2nd ROU spot in floor finals
-Kyla Ross (USA) would have qualified over Aly Raisman (USA) for the beam finals, thus stripping that bronze medal on beam from Aly!
-Anastasia Grishina (RUS) would have qualified over Ksenia Afanaseva (RUS) also for the 2nd RUS spot in beam finals
-Diana Bulimar (ROU) would have qualified over Catalina Ponor (ROU) for the 2nd ROU spot in beam finals (so no beam OR Floor finals for Ponor and no tie break between Ponor and Raisman on beam for bronze because neither would have been there)
EVENT FINALS:
-Ksenia Afanaseva (RUS) would have scored over Aliya Mustafina (RUS) on floor finals giving Afanaseva the bronze medal and knocking Mustafina to fourth (still beating out a 4th/5th place tie with Ferrari)
-here's one that will get everyone talking: McKayla Maroney (USA) would not only have been securely ahead of Maria Paseka (RUS) in that silver/bronze debate everyone's been chattering about but she would have won GOLD over Izbasa (ROU)!! wow! Definitely look at the scores on that one everyone!!
here's the link to the 'google docs' spreadsheet i made to show all this taking the info from the scores packet. I plan to do the same for team and a/a to see what would have happened for those scores as well!
https://docs.google.com/spreadsheet/ccc?key=0Amp3Xtqc1ymvdDJ2U2ZXWXppdXdaVUJ5QThXYzR6Znc#gid=0
Thank you, this is fascinating stuff ... keep up the good work!
DeleteElizabeth - can you help me understand the difference between Jury and Ref judges? Are the Ref judges - the more accomplished judges that they would go to to help solve problems?
DeleteI never use to really pay attention to judges.
Thanks
By the way, I love Afan but how could they even think of giving her 3rd with a 9.0 execution with 2 steps out of bounds and a 5.9 SV?
DeleteAnd Maroney getting 1st with her landing on her butt would have been interesting, lol.
Hi M,
DeleteAs I understand it, the reference judges are there to provide scores which can be used as a rapid resolution in the event of a large discrepancy between the execution marks of the e panel of five judges. They are specifically selected for this role on the basis of their performance in judging exams, and given the importance of their role you would expect the FIG to be exercising discrimination and choosing individuals with a proven track record.
It is really only a point of interest that the reference judges' scores favour Komova in this instance - it makes no difference of substance, as the medals were awarded according to the rules and the results will not change. But the marks recorded by the reference judges are, by and large, what the majority would consider intuitively to be the 'right' result. Elsewhere on the gymternet observers far better qualified than me have commented that judges on the e panel can judge at the Olympics with relatively little competitive experience.
I think the significance of the difference between the two scores is not so much the difference between the numbers, as the differences in the finishing orders of the gymnasts. There is something fundamentally wrong with the methodology used to evaluate execution, with the training of the judges and with the Code. One key area is that the terms of reference are very unclear. Some key definitions need to be applied in order to assist in the evaluation of artistry. There is some very good wording that appeared in the Code right up until 1997 and it would provide a good starting point to help judges move towards a collective understanding - assuming that the will to foster artistry exists.
I very much doubt that applying sticker plaster to the existing framework, as the FIG has attempted to do in its revised Code for 2013-2016, will make any significant improvement in terms of the public seeing the best winning.
Oh and I think Rodionenko is quite right to require his coaches to learn English - the Americans win the lobbying game all too easily, and Russia needs to put up more of a diplomatic fight (without getting people's backs up all the time as Valentina does). .
Sorry I have rambled here
Ah I see, thanks a lot for the explanation. I agree, with the differences in numbers (of the execution judges), there needs to be some definitions or something for the judges to understand how to evaluate artistry etc. It is obvious, everyone has a different definition based on the judging.
DeleteIf judges can have relatively little competitive experience to be judging the most important competition in gymnasts' lives (i.e. Olympics), something is definitely wrong there as well. The most experienced/accomplished should be used for Olympics, no ifs or buts about it.
Yeah the Russian coaches learning English would be good in order to put up a more diplomatic fight, just not Valentina because I think she would make things worse if she did :)
Hi Elizabeth ... according to your explanation why didn't they apply the Ref. judges' scores in the case of Grishina's UB routine in Qual.when there was one point between the lowest and highest score and in spite on counting out both of them there's still half point between lowest and highest score ... or because the gap between the Ref. judges' scores was wider (nine tenth). isn't it indicate the insufficiency of the current judging system?
DeleteI won't really get into this because it has passed and it really can't be changed. Bars judging was beyond inconsistent. I realize the same 2 judges who gave Musty and Komova the lowest scores are the ones who gave the Americans a higher one. Raisman getting 8.8 execution on bars which is higher than Musty and I think Komova is beyond laughable.
ReplyDeleteStill think Komova should have gotten a higher execution score on her floor. Oh well, can't really change it now. Hopefully she will just work harder and try her best which is all she can do. Bias happens in the judging and it will continue. So they just have to move on from this and continue doing their best.
Did you see September's International Gymnast, M? Paul Ziert, the American publisher of the pro American journal of gymnastics, said that Vika's execution score on floor in the all around final was 'the biggest joke of the Olympic games'.
DeleteNo I haven't seen it. I don't like Intl gymnast because of whatever that guy was smoking when he was talking bad about Aliya during podium training (I think it was). However, I agree It was the biggest joke, there is no way that should only get a 9.0 execution. I don't know, it's as if they wanted to see her get second again or something.
DeleteI hope she eventually gets that All around win because if she fails again, I am almost positive that she probably will just give up gymnastics.
The AA is 4 events. Thinking that someone is fabulous on two of them is not an argument that she should replace someone who competed clean.
ReplyDeleteWhile I have some problems with Gabby Douglas's form and artistry, there is no doubt that she made the fewest obvious errors and was the only woman of the top 4 or 5 to have a completely clean competition (no falls, stumbles, etc)
Do I wish that Komova had won? Yes. Do I think that she should have, given her mistakes. No.
I would just say that E score should mean execution score, and not error score. Virtuosity is about more than an absence of errors, and only virtuosity should merit an Olympic gold medal.
DeleteAs things stand, the marking focusses on error deductions and most judges do not seem to use discretion in marking whole routines and awarding credit for superior performance. Perhaps this is because the Code does not allow for it, although the degree of variance amongst the marks would suggest otherwise.
Until judges begin to judge, no Code of Points will work properly, and we'll continue to see questionable results like this one.
OK, I looked over some of the reserve judges scores vs. the 5 regular judges, not only in the WAG but event finals etc. I am not sure if there is voting bias, but it seems it may go both ways. For example Judge E1 on Floor in WAG is from Belarus so would this judge be biased in favor of the Russian gymnasts? In the WAG it seemed so as she game Komova a 9.5, the other 4 scores were 9.1,9.1,9.1,9.0. For Musty's floor she also gave her the highest score. On the flip side, this Belarus judge also was the highest judge on Douglas's floor and in the middle of the pack on Raisman's floor.
ReplyDeleteThe 2 reserve judges gave Komova 9.3 and 9.1. The 9.3 judge was from ... Kazakhstan. Just something to think about.
P.S. I do think Douglas was over-scored on bars and beam, but I am just trying to show this judging bias thing may still be happening, but it is happening both ways still.
if it happens both ways that doesn't make it right or legitimate it makes the whole thing is "fiasco" right?
DeleteI agree. The judges need better training so they are more consistent and to remove as much as humanly possible any biases.
DeleteI also noticed that the ER judges scores were sometimes applied to the final E score (which means the E score from the panel and the ER score from the two reference judges were averaged). On Komova's beam score in AA for instance this was used, but when this rule is applied is very confusing and inconsistent.
Did the Belarusian judge overscore the Russians? yes
DeleteDid she underscore the Americans? No ... she gave Gabby the highest score also and gave Raisman the second highest score ... that make them bad jugdes
But the thing here that the pro-american judges overscored their favorites and in parallel they underscored the rivals ... that make them more bad judges
This Book made me believe that:
ReplyDelete1.some judges were biased and were underscoring certain gymnasts from certain countries … for instance (in page 140 UB Qual. scores) you can easily find that E5 was underscoring the Russian gymnasts, she gave Komova the lowest score among the whole panel 8.5 and guess what she did the same to Mustafina 8.3 and finally Grishina she felt sorry for her and gave her the second lowest score with 7.4 … is that coincidence??
But guess what? She gave Douglas the highest score among the panel 9.1, and did the same to Ross with 8.7 and second highest score to Raisman 8.5 and to cover it up she gave poor Wieber 8.2
(And see E3 in the same page, she did the same but more cleverly)
2.The incompetence of some of those judges, I mean discrepancies among the judges’ scores could be up to one point!!!!!! and that is unacceptable range, you wanna look a funny one go to page 141 and look at Grishina’s scores 7.9 8.1 7.1 7.9 7.4 (one point between the highest and the lowest score and after you abolish both of them there’s still half point between the highest and the lowest score!!!) and even the reference Judges’ scores are no better 8.3 7.4 ( nine tenth discrepancy!!!)
Gabby won, get over it!! Sheesh, Komova is most likely coming back for 2013 Worlds and then she could possibly get her gold in the AA,finally, if she has a great day.
ReplyDeleteThe persons discussing here the scores all konw that Douglas won. We were just wondering why the scoring is so subjective. Why is gymnastics rewarding difficulty over artistry? Why are there judges who consider that Raisman deserves a good execution bar score, for instance?. All the system is "I give your gymnast a good score, tomorrow in the finals you give gymnast x an x score.
DeleteIt's not spontaneous!
And again it's the second time this happens to Komova. We could calculte her reference judge's score in Tokyo as well.
Jackson, I'm afraid I accidentally deleted your second post. Please could you resend? Apologies.
ReplyDeleteok, why don't we take the average execution score from all the jury and the reference judges, (not just the jury) to see who's the winner :)
ReplyDeleteUsing the simplest method of just averaging the competition scores with those using the reference judges' scores (shown in this blog post) that leads to the following result:
Delete1 Komova 62.0365
2 Douglas 61.991
3 Mustafina 59.808
4 Raisman 59.383
I wonder how the reference judges felt about the result, and whether the FIG looks at these figures as indicators.
I wonder what did Nellie Kim tell Komova on All-Around podium ceremony? "Sorry we screwed you for the second consecutive time!"
Deleteseriously, I can't believe it. I knew for a fact that the reference judges scores will be used in the event that it has a big gap from the main judges. but in sports, even 0.0001 can count for the differences in medal. If adding the reference judges scores will alter the final medal results, it should be counted towards the main judges scores as well. seriously. this is just out right unfair.
DeleteNellie Kim doesn't like the Russians. She screwed Komova and she'll screw her again.
DeleteApparently sports judgement are a true reflection of the USA political behavior and how they press the entire world to favor their convictions... And how this mister Paul Ziert refers to Komova´s execution score is, again, the perfect example of the american arrogance. What would happen if Komova had win the Gold Medal? I really can´t imagine that...
ReplyDeleteThe screams from the crowd in London during competitions remind me the ones in Atlanta. Fortunately we could listen Komova´s music, not like Galieva who couldn´t, twice.
When are the Olympics going to have place in a country that has no fear of the USA????? I hope Rio could be one... Let´s see!!
Yeah and I can say that most attending British spectators weren't neutral, they cheered the American as much as they did to their players and teams
DeleteHi Barbara ... do you know that one of judges who constantly overscored the Americans and underscored the Russians in London are from your country? go and see it youreslf
ReplyDeleteIsabel Lazo is a long time gymnastics trainer and judge in my country. Her daughter, Simona Castro who competed in London, studies and trains in the USA, and even if she wouldn´t Isabel had overscored the Americans anyway... In my case, I don´t like very much the way that the Americans stand in front of the world, but unfortunately for a south-american country is difficult to go against the USA, you know...
ReplyDeleteDespite this, I´m able to appreciate some american gymnasts like Nastia Liukin, who is a very elegant athlete. Anyway, the fact that Nastia had won the Gold in Beijing over Shawn, for me was a victory of Russia, yeah!
Hi Barbara ... Thank you for the elaboration
DeleteI can't stand people who are complaining about Komova's crying. Try to think about it: she bursted into tears just because she trained hard for years, she expressed by far the best gymnastics and the gold she deserved was robbed (and it's the second time). What would you have done in her place?
ReplyDeleteThis would be easier if judges can watch replay. Sure, it would slow the competition down, but it seems to me that a more even score can be given then, even in spite of the subjective nature between judges.
ReplyDeleteIn any case, artistry in gymnastics definitely needs to be better emphasised.
What can you do with this information? If this is true the FIG must publish a release and explain that.
ReplyDeleteElizabeth, I just wanted to say thank you for always writing such consistent, well thought out and intelligent posts. You have similar opinions and views to me and I always find it a pleasure to come and read your blog and see my own opinions expressed so much better than I could express them!
ReplyDeleteBarbara from Chile - I'm not sure I agree with the idea that there is a widespread American bias happening with the judges, but I did notice that there seemed to be 3 levels of cheers coming from the crowd. One level for everyone, another level that almost blew the roof off for the Brits (of course!), and a third level sitting in between those two... for the Americans.
I found this perplexing. I thought I was imagining it at first but it was a clear difference. I still don't understand how that managed to happen. Is it just because the American team are the "stars" of the gymnastics world, the celebrities? Or did they allow more American supporters into the arena than they did for any other country? I would be very curious to find this out.
You forgot another assumption which is many Brits spectators weren't neutral and that is the right one (IMO) for the following reasons:
Delete1. they can't let more American supporters in because it's ticketing and shares thing or otherwise it would be a preferential treatment.
2. the Americans aren't the only "stars" of gymnastics there're also Russia, China, Romania and to make sure that this hypothesis is not right watch MAG's competition where Americans weren't the "stars" of this division.
Here is a really cool interview with Aliya and her family. A lot of the content we already knew, but still cool. For some reason, I always laugh when I read about her and Alexander not talking for days.
ReplyDeletehttp://www.eg.ru/daily/sports/33805/
FIY, the link to the explanation of the role of the reference judges is broken.
ReplyDelete