Beware the Backtest

Backtesting is a popular tool amongst traders and marketers. Traders use it to experiment and uncover new strategies. When one looks promising, the marketers go to work! Creating fancy charts and graphs to sell you on hypothetical performance. But are strong backtests indicative of strong future results?

The world of "Smart-Beta" is filled with promising backtests. Unlike plain-vanilla ETFs that simply track an index, Smart-Beta aims to outperform by adjusting their holdings - sometimes greatly deviating from their index.

A recent study called the Smart Beta Mirage analyzed the performance of these "smarter" funds from 2000 - 2018. Sadly, it found that outperformance only lived in the data.

Smart-beta in backtests showed an above-market returns of 2.77%, but when put into action, they underperformed by 0.44%.

There are several reasons for the disappointing results. The most obvious one, no one can predict the future. The oldest adage in investing is "Past performance is no guarantee of future results". Things change, what worked in the past might not work in the future.

Another reason is backtesting is flawed. It's susceptible to bias and data mining. ETF providers are incentivized to attract as many investors as possible. A great way to do this is to show great performance would've been, even if it sacrifices the integrity of testing.

While backtests are interesting and a great tool if used correctly, like star ratings and analyst targets, they do little in predicting the future. Don't count on them.

Comments

Today's Special: Humble Pie

You champion a project, fight for an idea, and then...reality sets in. That churning in your stomach isn't butterflies, it's the realization you've missed the mark. Pride will puff up your chest, and kick in the "defend at all costs" instinct. But arguing with the umpire never changed a call. Admitting you're wrong isn't a sign of weakness. It can strengthen your professional standing. In a world obsessed with the illusion of infallibility, the courage to adjust course is a breath of fresh air. It shows you're confident enough to be wrong, and adaptable enough to learn from it. Do your research, think critically, and stand behind your decisions. But when the data whispers (or screams) otherwise, don't be afraid to swallow that slice of humble pie. Be the first to acknowledge. Don't wait for someone to point out your mistake. Be open, take responsibility, and most importantly, focus on what you're going to do to address it. Don't dwell ...

Negative Feedback, Positive Lessons

In the battle against plastic bags, a five-cent tax was shown to be much more successful at deterring usage than a five-cent credit for bringing your own bags. Carrots satisfy but sticks sting, and they sting hard. So we default to the less painful choice of avoiding loss. Loss aversion impacts the way we process information. A 2019 study invited participants to learn through a series of multiple choice questions. Each question only had two options to choose from. Whether guessing correctly or not, they would still learn the right answer. Despite the identical learning opportunity, participants were much more successful at recalling the answers they guessed correctly than those they got wrong. "You're right!" feels good. We savour the moment, analyzing every detail. "You're wrong!" stings. We want to quickly forget, dismiss, and move on. When we succumb to loss aversion, we miss opportunities to learn. Failure is part of the process. We'll experie...

Starting Really Really Small

On your desk is one of the most intimidating sights known to man. A blank page. The prospect of filling it up with anything resembling decent seems insurmountable. Staring at the long road ahead fills you with anxiety and dread. The first step is the most difficult. So we procrastinate. We " research ", we " prep ", we " plan ". We do everything except tackling the problem. We avoid the pain for as long as we can. To make a blank page less intimidating. Tear it in half. There, half as scary, twice as easy. Still too much? Do it again. And again. Keep doing it until the task is so small that it's too easy not to do. Getting starting is the hardest part. So make the hardest part as easy as possible. This doesn't guarantee amazing results, but it gets you in the game. You can't win if you don't play.

Herman Ye

Search