The Law of Large Numbers and Why Trial & Error Works

Image Source

Trial and error is an inevitable part of our life.

We make mistakes, readjust, and try again.

This cycle repeats over and over until we get satisfying results that may only need some internal improvement.

Trial and error and testing, in general, is also a key part of marketing. We run A/B tests to find out the best variation and then we do another A/B test where A is a winning sample from the prior test.

If this approach is so crucial and leads to great results why it’s so buried in a usual path of education?

Today, I’d like to move the needle on two issues.

First, why trial & error works.

Second, why we don’t teach that approach (and why we should).

Okay, the first one finds its answer in the Law of large numbers.

This particular law states that when you run an experiment certain number of times, the more you do it, the closer the average of all results gets to the expected value.

Thus, the more trials you perform the higher the chance you’ll get the desired result.

That’s why testing and experimentation is so powerful, especially when we focus on one sample and test it over and over.

Even in marketing tests are not one-day trials.

Statistically, if you want to prove the increase of conversion rate by more than 6.3% you need a sample of 1000 unique views.

On the other hand, if you want to prove the increase of conversion rate by 2% your test needs a sample of at least 10000 unique views.

This shows that if you want to test macro changes you need a smaller sample compared to testing micro changes.

All in all, it’s more likely that you’ll find the right answer if you run the experiment over and over again.

Now onto the second point.

If trial & error is THE WAY, why don’t we incorporate it into our educational system?

Or maybe we already did?

Well, take a look at how our schools provide feedback regarding students’ progress.

1 - you get uniform grades and 2 - you take uniform tests.

Such a “good or bad” approach doesn’t leave room for an error and further improvement. You’re either good now or you’ll suffer later.

Tinkering is often the only way to go from 0 to 1 and yet we tend to kill the not working for the only right “it must be working from the first try” approach.

This obviously puts pressure on a learner and instead of trying to go from 0 to 1, they move from 1 to n only reproducing what has already been made.

Here’s a great TED Talk by Tim Harford who talks about the so-called God Complex and why it kills the trial & error approach.

In it, Tim touches on one more aspect of why trial & error is oftentimes the only way to find the right answer.

And that’s the fact that our world is way too complex for us to understand.

What we call obvious now, was a mystery 200 years ago.

What we call a mystery today, will be an everyday thing in 100 years.

Our brains are not good at crunching huge amounts of data and that’s what today’s world is pushing upon us. Hundreds of items, names, formulas, combinations, places, recipes, books, movies, things to do…

That’s also what computers were made for. They’re great at what we suck. And so we’re the best buddies.

Our world is so far out of our reach that agreeing on one arbitrary solution is childish if not irrational. And that’s really what the God Complex is about.

To conclude.

Trial & error is not taking a blind guess, no.

If we stick to the rules of the experiment and we run it for long enough we’ll get closer to the expected value. Yet, it doesn’t mean that it can be any experiment done in random circumstances.

Know what you’re testing, have a clear goal, monitor the progress, and use thoughtful trial & error to find the result that satisfies you.