There is a fascinating article on the BBC news website. Apparently, some people have been getting upset about the random function on Spotify.
Cue the creepy X-files music. You see, Spotify users have been spotting patterns. Favourite songs coming up more often than you would expect. Songs from the same genre of even the same band. I have even heard people say that the random function on their I-pad understands their moods.
Is it really random? Is there something out there? Are the music companies paying Spotify to make their songs come up more often? Is there a supernatural explanation?
Ah, no. Sorry, but no. Spotify have confirmed that the random function is exactly that. It really does pick tracks at random. No conspiracies. Nothing hidden. What is happening is that the human brain is very bad at understanding randomness. We are gullible little primates.
We expect random to mean “evenly spread”. But that is not what happens in practice. Randomness simply does not work that way. It is perfectly understandable to get clusters of similar results close to each other, just as it is possible for some results to be missing for quite some time.
Let’s have an experiment. I am going to roll a dice and record what happens when I roll it 100 times, 1,000 times and 10,000 times.
To be perfectly honest, I am not going to do the dice rolling myself. I am going to use a spreadsheet to generate random numbers. We can be fairly certain that the spreadsheet’s random function is sufficiently random for our purposes. There is no conspiracy inside Excel to favour 4s over 5s, say.
And the results are more interesting than you might expect. Each number 1-6 has as much chance of coming up as any other. But over 100 dice rolls, we nearly always get some numbers coming up more than you would expect.
First 100 dice rolls – the number 3 comes up 21 times instead of the 16.666 that we would expect, but we only get 13 rolls of a 1. Hmm. Try again.
Second batch of 100. 1 and 5 are rolled 20 times each. This time we only get 13 rolls of a 2.
The third batch is heavily skewed towards the bigger numbers. We have 23 rolls of a six and 21 rolls of a 5, but only 11 rolls each of 1 and 3. In other words, a six came up more than twice as often as a 1 or 3 – over 100 rolls.
Stepping up to 1,000 rolls, and the results get closer to each other, but there are still noticeable differences. The first batch of 1,000 rolls generated 199 4s, but only 155 2s.
At 10,000 rolls the total get closer to each other, but we still get hot and cold numbers. The important point is that the hot and cold numbers change every time we refresh the calculation. They are not unusual.
But let’s take a look at the numbers themselves. And we see that the series are full of unexpected little clusters. I can see a run of seven numbers composed only of 6s and 3s. The same number being repeated three, four or sometimes five times in a row. A long sequence without a particular number coming up at all.
And each time I refresh the random numbers, new clusters appear. Different clusters . New sets of hot and cold numbers.
If you want to play with the spreadsheet yourself, email me on firstname.lastname@example.org and I’ll send you a copy.
This isn’t some conspiracy theory or anything supernatural. It is simply probability doing its thing. Coincidences and clusters happen.
So it’s nothing to get excited about when you visit a foreign town and bump into someone that you know. Or if you think about Aunt Ethel and she phones the next day. These are simply natural coincidences. They only seem to be out of the ordinary because you don’t remember all the times when a coincidence doesn’t happen.
The conclusion to the Spotify story is fascinating. Spotify tried to tell its customers that the random function was truly random, but the public wouldn’t accept it. What a suspicious lot we are!
So Spotify are introducing a new “random” feature which produces results that fit more into the human perception of randomness. If you own four songs by one artists, they will be evenly spread across your random selections – 25% apart.
Which is more than slightly odd. In order to give us something that we can accept as random, they have to manipulate it so that it isn’t random at all.
This might sound trivial. It is hard to get worked up about dice rolls and random music selections. But at the heart of this is one of our biggest problems. Humans are alternately gullible and cynical.
We look for patterns that are not there. This means that we can dispute the experts who know what they are talking about, and we get hoodwinked by the snake oil salesmen who take advantage of our lack of understanding.
Spotify have taken the easy way out and provided a non-random randomiser. That is probably understandable. They are a commercial organisation and need to make a profit.
But don’t you find yourself wishing that we weren’t so stupid in the first place?