Imagine there has been an outbreak of a rare disease in your hometown. Only 1 out of every 200 people is infected and there is a test that can detect if you are infected (or not) with an accuracy of 95%. You take the test and the result is positive, indicating that you have the disease.
What is the probability that you are infected?
Intuitively, the answer to this question seems to be 95%. But if you are familiar with Bayes’ theorem (or if you read the title of this blogpost) you might know that the probability of having the disease is much lower than that. It’s only 8.7% to be exact.
How can this be?
To understand this, let’s take a look at your hypothetical home town:
This is the entire population of let’s say 200,000 people. We don’t know who is infected and who isn’t, but we know the true base rate that 1 out of 200 people are infected.
In other words: Only 1,000 people have the disease (green) and 199,000 don’t (red).
Now, there are two possibilities for you: either you have the disease, or you don’t. And there are two possibilities for the test: either it says you are infected or it says you aren’t.
This adds up to four possible results:
- True Negatives
95% of the people who are not infected will be correctly diagnosed as healthy. Since the negative test results are true in their case, these results are called true negatives.
- False Positives
5% of the people who are not infected will be incorrectly diagnosed as infected. Since the positive test results are false in their case, these results are called false positives.
- True Positives
95% of the people who are infected will be correctly diagnosed. Since the positive test result is true in their case, these results are called true positives.
- False Negatives
5% of the people who are infected will be incorrectly diagnosed as healthy. Since the negative test results are false in their case, these results are called false negatives.
The true probability of being infected
As you can see in the table a lot of healthy people will get incorrectly diagnosed as infected. Out of our hypothetical hometown a total of 10,900 people got a positive test result. Yet out of these people only 950 actually have the disease.
Because the disease is so rare, a positive test result only means that there is an 8.7% chance that you are infected.
The accuracy of a test is not necessarily the probability of the test result being correct. To calculate the probability of a test result being true, we also need the true base rate, or at least an estimate of how rare whatever we are testing for is.
Evaluating a betting record
Every betting record is a test for one question: Does the bettor have an edge over the market? However, some lucky bettors without an edge still make a profit. But when the number of bets increases, the probability of being in profit despite not having an edge gets smaller, as we can see in this table posted on Pinnacle.com
The “Probability of being in profit”-column is the percentage of bettors who don’t have an edge over the market but will still return a profit over the number of bets in the first column. Those are false positives. Even though they don’t have any true edge over the market, their bets returned a profit.
Now let’s pretend we are looking at a record of a bettor that was able to achieve a positive return over 10,000 bets at odds of 1.95. This bettor will be a recurring figure throughout this blogpost, so we’ll call him Joe.
Obviously, Joe thinks he has an advantage over the market, or else he wouldn’t have placed his bets. He has a very good understanding of how the market works and a deep knowledge of players, coaches and tactical systems. Every bet he places is based on extensive research. Joe knows that he has an edge over the market, because his betting record proves it. He has been successful over many years and the probability of him returning a profit based on chance alone is as slim as 0.51%.
Unfortunately for Joe however, this does not mean that he is 99.49% likely to have an edge over the market.
To correctly estimate the likelihood of a bettor having an edge, we need three things:
- Estimate of the true base rate: How many bettors actually hold an edge over the market?
- Likelihood of a betting record returning a profit based on chance
- Likelihood of a betting record returning a profit based on skill
At the end of this blogpost you can download an excel sheet to help you estimate the likelihood of your own betting record being a result of skill, while accounting for all three.
We will now go through how to use the excel sheet by taking Joe’s betting record as an example.
How to use the sheet
The first thing we need to do is establish a base rate. How many bettors hold an edge over the market to begin with? If you are not a sportsbook, you probably don’t know what the true base rate is. This will have to be an estimate and we should be aware of that.
However, there has been some research on this and we can make an educated guess. To quote Joseph Buchdahl, who analysed over one million bets by 6,000 tipsters here:
The data (…) fairly emphatically reveal that if any skilled forecasters in the world of sports betting really exist, they probably aren’t numbered in one-in-tens or one-in-hundreds but perhaps one-in-thousands or even one-in-ten-thousands.
When you have decided on a base rate, type it into the excel sheet like this:
In our example we will work with an estimate for the true base rate of 1 out of every 1000 bettors.
This means that if our hometown from the beginning of this blogpost consisted entirely of bettors, out of the 200,000 people, 200 will have a real edge and 199,800 won’t.
Next, we need to know the likelihood of our betting record returning a profit based on chance alone.
For this, we need to simulate the betting returns based on the implied odds of the bookmakers. If you are unsure how to do this, here is a step-by-step guide how to do multiple simulations in excel.
How many times of the simulations did you observe a negative betting record? How many times did you observe a positive record?
In our example, Pinnacle already did the math for us. The probability of Joe’s betting record returning a profit based on chance alone is 0.51%.
This is how you fill out the excel sheet:
This means that out of the 199,800 bettors that don’t have an edge, 99.49% would be correctly classified through their negative betting returns. 0.51% of these bettors will return a profit, based only on chance, not because of any edge.
The last thing we need to account for is the probability of the betting record returning a profit based on skill. If Joe was one of the few bettors with an edge, how likely is it that his betting record will return a profit?
For this, we will need to do another simulation. This time however, we won’t use the implied probabilities from the bookmakers, but our own probabilities to simulate the matches.
If you don’t have probability estimates for every bet you placed, it is okay to use the same figures from step two, since we are working with estimates anyways.
In Joe’s case, we don’t know what his bets were based on, so we will have to do with the same figures from step two:
Out of the 200 bettors with an edge, almost every bettor will have positive betting returns and only one will observe a negative betting record.
Now we have everything that we need to estimate the probability of Joe having a real edge over the market.
Out of our sample of 200,000 bettors with similar betting records to Joe’s, a total of 1,218 bettors will return a profit and only 199 of those will be true positives.
And this is what we are really after, when we evaluate a betting record. Does the bettor truly have an edge or not?
To summarise this: It is important to acknowledge that test results are not the same thing as the events they are testing for. We have a test for a disease, separate from the event of actually having the disease. And we have betting records, separate from actually having an edge.
In Joe’s case we don’t know if he has an edge or not. We only know that his betting record has a positive return, and that (only) an estimated 16% of bettors with similar positive betting records really had an edge.
These estimates however depend on the base rate we were using. Here are some probabilities based on different base rates for Joe’s betting record:
You can download the excel sheet here:
Evaluating a prediction model using scoring rules
An estimated probability of 16% to have an edge over the market after 10,000 successful bets is a frustrating result.
When you look again at the table from Pinnacle.com, you can see that betting returns just take too long to be used as a feedback for predictions.
If you get positively tested for a rare disease, the sensible thing to do is to undergo further testing to make sure. If you have accumulated a couple of thousand bets however, waiting another couple of thousand bets, doesn’t seem sensible.
If you are able to express your predictions as probabilities however, you can assess the quality of your predictions much quicker by using scoring rules.
Scoring rules measure the error in probabilistic predictions and they need far less games to accurately assess the quality of a set of predictions than betting records. If you want to get a feeling how much faster they are, you can continue to read this blogpost here.
If you have any questions you can message me on twitter @fussbALEXperte