League Point Calculations & Outliers - Technical and Mapping Issues

Re: League Point Calculations & Outliers

Rory — Tue, 14 Jan 2020 08:44:31 +0000

Hi Val.
Yes that\'s me - \"English Rory\".
I did notice a few events, such as Emo, where slow times did impact the points distribution. But you don\'t really see the effect of this at league level (except possibly on the brown?).

Maybe easier to think about an exaggerated example:
7 runners at 50, 67,68,69,70,71,and 100 minutes. Maybe the fast runner dropped down a course and the last runner has a bad day, whatever.
Current method, the slow guy increases the average time and the midpack get a bit more than 1000 because they are faster than the mathematical average.
IQR method, the fast guy gets a much better score, the midpack get 1000 on average (but more spread out)
Speed method, the midpack get approx 1000 but the fast person gets more points, the influence of the last finisher is diluted and the slower they finish, the less effect each extra minute has.
Formula1 method, the scores are exactly the same as if it were a photo finish.
Relative time method, in my formulation of it the winner gets 1000 with the midpack only getting 600 or so.

If there are particular events you have in mind, the data is there in the file and should be easily filterable, so let\'s have a look!

The idea of the superman method is kinda cool but would it be demoralising for the average runner?!

Re: League Point Calculations & Outliers

Val — Mon, 13 Jan 2020 21:57:14 +0000

Thanks for you work and analysis, Rory (Wallace, right?)
I was looking for discrepencies at various events to see which system might address them, but I didn\'t come to any conclusion, apart from I believe it\'s really slow runners that disrupt the current system.

Here\'s another system that occurred to me. Use the so-called superman time, that is the accumulated time of the fastest leg times on each course, allocate 1000 points to it, then calculate points for each runner in proportion to it. However it might be difficult to program.

Val

Re: League Point Calculations & Outliers

Rory — Mon, 13 Jan 2020 20:51:17 +0000

No problem - ah sure tis good to have a debate every now and again to blast off the mental cobwebs.

When Paul is back (or even before) we can argue the case for and against changing the current system.

Re: League Point Calculations & Outliers

stuart — Mon, 13 Jan 2020 12:11:20 +0000

Hi Rory,

Thanks for that analysis, that\'s really interesting! Especially how the speed is less volatile. That said, it makes understanding the calculation trickier too.

As we said before, it really comes down to what type of scoring system people prefer. I would also stick with the status quo for the reasons mentioned previously.

On a more important note, is this the first forum thread on the \'new\' forum to run to 5 pages?! We\'ve had countless debates over why people don\'t use the forum. Now we know why. We just need more maths!

Stuart

Re: League Point Calculations & Outliers

Rory — Sat, 11 Jan 2020 18:31:24 +0000

Hello.
So I have some analysis to share. Apologies if it is a bit hard to follow as it was written under the influence of lemsip-type drugs !
I\'ll explain what I did, followed by a few graphs, then some comments.

I have reproduced the league table based on the anonymised data set Stuart uploaded. So this is the 2019 Leinster league. Names, age class, gender are omitted as are OPC status. So each person\'s score is based on their best 6 points scores per course, across the league ,compared to all finishers.

There are 5 methods used for calculating the points, as follows:
First, the current method. Points = 1000 - 200 * (Time - Average Time) / Stdev (Time).
Then, the \"Current-IQR\". Instead of the Standard deviation of times in the previous formula, I use a modified estimator, ofthe Interquartile Range divided by 1.48. The idea behind this has been discussed in the thread - if there are outliers the standard deviation can be unusually high which can compress the scores close to 1000 using the current method.
Thirdly, the method suggested by Paul (more-or-less). The winner gets 1000 points with everybody elses time based on relative time, 200 points less for every 20% slower. So if you take twice as long you get zero points.
Fourthly, the \"formula 1\" method, based on finishing position, 50/46/43/40/38/36/34/32/30/29/28/27 etc.
Fifthly, the \"speed\" method. I noticed that although the finishing TIMES are skewed, the inverse of the times are more evenly distributed. So I simply took (10000/finishing time in minutes) and used this in the current method, where the average speed per course gets 1000 with every stdev(speed) above or below counting for 200 points.

Now the graphs (I\'ve included Brown, Blue and Green but they\'re all in the spreadsheet which I\'ll upload).
I have found the total for each league competitor using each of the 5 systems and then rescaled them relative to the current league winner for presentation purposes. I\'ve only included the top 25 for clarity.

Now my comments:

None of the methods cause a significant reshuffling of the first few league places. On the brown the 2nd,3rd and 4th finishers are very close so they do swap places; on the green the league winner surprisingly drops back based on \"time relative to winner\", but nothing else of note.

Generally, the methods based on time or speed relative to the average, give quite similar results. The \"current-IQR\" and \"speed\" methods seem to have a little more distinguishing power than the current method. On the brown course there are a few places further down the rankings where the IQR adjustment makes a difference, havent looked into this but it may be related to the lower number of runners on brown.

However... If we use the old \"formula 1\" method, and even more so if we base scores on time relative to the winner, there are some quite big adjustments. You could say that the league positions based on relative time are all over the place. I will leave it to others to argue if this is good or bad.

My personal view after doing this is, I don\'t think a change is required; however if people feel a change would be a good idea to remove the effect of outliers I would suggest using the \"speed\" (i.e. 10,000/time) method which should be a very simple change operationally.

File in Excel format at www.orienteering.ie/forumuploads/LeaguePointsDiscussion/AlternativePoints.xlsx
Look forward to hearing your thoughts.
Rory

Re: League Point Calculations & Outliers

Rory — Tue, 07 Jan 2020 08:50:24 +0000

Hi Marcus..
So I started on this yesterday with the blue course.
Thoughts so far:
Median vs Mean makes very little difference.
The times are highly skewed to the right and the standard devation is high in some cases (Emo Court is one I recall). The interquartile range is a lot more stable.
I\'ll continue looking at it over the next few days and include a % of winning time system and the old system. If anyone can call out outlier courses to focus on it might be helpful.

I agree with the other points about league promotion, encouraging participation, friendly competition and OPC volunteerism. I don\'t have strong views for or against the current system although i do prefer time based over position based.

Re: League Point Calculations & Outliers

formline — Mon, 06 Jan 2020 16:36:32 +0000

I think this discussion about the calculation system is fantastic and I am in awe of those who have the mathematical skills to make sense of it. I feel that a little bit of historical context is worth putting on the record:

In the noughties (up to 2006) I produced the two Leinster League tables (Autumn Series and Spring Cup), and an inter-club league, using a spreadsheet. Points were awarded as 50 for first place, 46 for 2nd, then 43,40,38,36,35,34,33,32,31,30... etc. OPC points were awarded as the highest points you scored in the other events in that league. Only one set of OPC points per competitor per league was allowed. The points system pre-dated me, I presume it was put in place in the ‘90s. The inter-club league spanned both Autumn and Spring individual leagues.

Producing the spreadsheet was extremely laborious and eventually I managed to persuade Stuart Scott to use his legendary programming skills to automate the process. He asked me for an algorithm, I did some googling and found the Scottish six-day .PDF file that we use now. So if anyone wants to blame someone for the system currently in use, please blame me.

While running the spreadsheet system I received very little feedback, but what I do remember is:

The points-based system does not reward exceptional performance
If there are only five or six events in a league then the OPC points can skew the results
Mushing the women and men together makes women’s league placing depend to some degree on the men who tend to run the same course.
Runners running-down a course or two mess up the league for everyone else (e.g. the normally Brown runner who takes it easy for the day and wins the Green).

They are the only comments I remember receiving and they have all been thoroughly analysed in this discussion.

Moving to the Scottish 6-day system was an attempt to fix #1. I think #2 is probably (?) fixed by the recent reversion to a single ten or eleven event league. #3 and #4 remain problems.

While inadequacies in the scoring system do need to be addressed, I would like to see us also address other issues with the league (with apologies to David Masterson for polluting his thread with items that are nothing to do with how the results are calculated; I love the way the discussion started with the words “Maths Warning”).

- Fixing items #3 and #4 above

- Finding a way to better highlight juniors’ performance

- Re-vitalising the inter-club league (are we the only sport in Ireland that does not have an inter-club competition?)

- Building on Val’s work to promote the league winners. It should be a very prestigious title to hold and something that you fight to retain.

I know none of the above contributes to the discussion about the scoring system, but I thought maybe the historical context would be of interest.

Trivial point: one of the reasons I did the spreadsheet in the noughties was because I wanted to become familiar with Excel’s VLookup and HLookup functions. I still hate them.

Marcus Geoghegan
Ajax

Re: League Point Calculations & Outliers

Val — Mon, 06 Jan 2020 14:27:35 +0000

I\'m neutral on the position based versus the time based system.
The advantage of the position based system proposed by Paul is that a win is a win, regardless of time difference. Runners out of category can be excluded. It is also easy to understand, and easier to allocate OPC points.
If we want a time based system no system is going to be perfect. The current system is fairly good. It is less impacted by who turns up on the day. I think it\'s better at rewarding consistency and participation. I suspect Eoghan Whelan would have won the blue if he had done more than 6 events. Also it keeps more uncertainty of who will win up to the last day. There\'s unlikely to be a tie, which is more likely in position based (separate based on head to head?). The main flaw is because of the low numbers very slow runners can pull down the overall points on the course.

On the OPC points, since I\'ve been doing league prizes I only started allocating them in the last 3 or 4 years. They rarely make a difference except when someone needs it to make up the 6 events, as with Colm and Eileen this year. When we had a position based system years ago, with 50 points for a win, the OPCs also got 50 points (correct me if I\'m wrong, Marcus)

Re: League Point Calculations & Outliers

Rory — Mon, 06 Jan 2020 11:58:40 +0000

Hi Paul
Yes I get your point, you want the results to be based on % of winning time, I can throw that in as well (someone who takes twice the time of the winner gets 500, three times 333, etc)
I just want to do a really quick comparison of the main types ofmethod with the least work possible.
If we see that there are significant differences over a full league then it can be fine tuned by excluding ineligible runners etc.
Yes one point per place will work the same as 10 points per place but it will make the graphs look nicer.
Enjoy the exam corrections I have some work to do myself for the night o season.
Rory

Re: League Point Calculations & Outliers

stuart — Mon, 06 Jan 2020 11:56:28 +0000

It would be great if this was the exam question! It\'s far harder than anything I came across in college.

The current system does take into account the relative times (albeit in an obscure way). If we based the league table on time alone, then an abnormally difficult event will unfairly benefit or penalise runners who show up.

Scaling points to have the same maximum per event has the same result. It assumes all events of equal standard which we know they are not.

Runners may indeed be \'lucky\' when finding a control. This is why it\'s better to base the points on the average runner rather than an individual.

So that leaves us with the original question...do we remove outliers or not?! If outliers are shown to be unfairly influencing the scores then we should. In my earlier post with the histograms there is some evidence of this due to the numbers involved but I didn\'t think it was significant.

I agree we could certainly split the scoring by gender and adult\\junior status but that would just amplify the effect of small sample sizes. As separate prizes are given to these categories anyway, does it really make much difference?

To sum up, Ruairí\'s example highlights perfectly the differences between the two scoring systems. Both produce very different types of results (a bit like proportional representation -v- first past the post - which system produces a better government?!). I think Paul and I both have very valid arguments in favour of each alternative. It\'s up to the community to decide which system they prefer!

Stuart