The Law of Large Numbers and Why Trial & Error Works
Image Source
Trial and error is an inevitable part of our life.
We make mistakes, readjust, and try again.
This cycle repeats over and over until we get satisfying results that may only need some internal improvement.
Trial and error and testing, in general, is also a key part of marketing. We run A/B tests to find out the best variation and then we do another A/B test where A is a winning sample from the prior test.
If this approach is so crucial and leads to great results why itâs so buried in a usual path of education?
Today, Iâd like to move the needle on two issues.
First, why trial & error works.
Second, why we donât teach that approach (and why we should).
Okay, the first one finds its answer in the Law of large numbers.
This particular law states that when you run an experiment certain number of times, the more you do it, the closer the average of all results gets to the expected value.
Thus, the more trials you perform the higher the chance youâll get the desired result.
Thatâs why testing and experimentation is so powerful, especially when we focus on one sample and test it over and over.
Even in marketing tests are not one-day trials.
Statistically, if you want to prove the increase of conversion rate by more than 6.3% you need a sample of 1000 unique views.
On the other hand, if you want to prove the increase of conversion rate by 2% your test needs a sample of at least 10000 unique views.
This shows that if you want to test macro changes you need a smaller sample compared to testing micro changes.
All in all, itâs more likely that youâll find the right answer if you run the experiment over and over again.
Now onto the second point.
If trial & error is THE WAY, why donât we incorporate it into our educational system?
Or maybe we already did?
Well, take a look at how our schools provide feedback regarding studentsâ progress.
1 - you get uniform grades and 2 - you take uniform tests.
Such a âgood or badâ approach doesnât leave room for an error and further improvement. Youâre either good now or youâll suffer later.
Tinkering is often the only way to go from 0 to 1 and yet we tend to kill the not working for the only right âit must be working from the first tryâ approach.
This obviously puts pressure on a learner and instead of trying to go from 0 to 1, they move from 1 to n only reproducing what has already been made.
Hereâs a great TED Talk by Tim Harford who talks about the so-called God Complex and why it kills the trial & error approach.
In it, Tim touches on one more aspect of why trial & error is oftentimes the only way to find the right answer.
And thatâs the fact that our world is way too complex for us to understand.
What we call obvious now, was a mystery 200 years ago.
What we call a mystery today, will be an everyday thing in 100 years.
Our brains are not good at crunching huge amounts of data and thatâs what todayâs world is pushing upon us. Hundreds of items, names, formulas, combinations, places, recipes, books, movies, things to doâŚ
Thatâs also what computers were made for. Theyâre great at what we suck. And so weâre the best buddies.
Our world is so far out of our reach that agreeing on one arbitrary solution is childish if not irrational. And thatâs really what the God Complex is about.
To conclude.
Trial & error is not taking a blind guess, no.
If we stick to the rules of the experiment and we run it for long enough weâll get closer to the expected value. Yet, it doesnât mean that it can be any experiment done in random circumstances.
Know what youâre testing, have a clear goal, monitor the progress, and use thoughtful trial & error to find the result that satisfies you.