Be Cautions When Considering Backtests of a Proposed Investment Strategy

When evaluating new investment portfolios, the use of backtesting to justify portfolio construction and trading methods is a common marketing tool among many firms. Backtesting is a traditional way of saying that a proposed investment strategy would have worked in the past, and that it would likely be successful in the future. That assumption, however, is very contentious from several perspectives.

First, backtesting is fundamentally flawed because of its inherent reliance on historical performance or a past financial forecast. Backtests can also create overly specialized strategies that could be riddled with potential risk for future investment. And unfortunately, some advisors too often evaluate the financial strategies of outside firms based solely on backtesting, without any consideration of more modern statistical methods.

The basic flaw is that backtesting never really tells us if a proposed strategy will work in the future, even though it functioned successfully in the past. Consequently, you have to ask yourself, “Why did it work?” And did it work in a way that is consistent with what financial advisors believe is going to continue in current and future capital markets? It’s always prudent to remember that quite-familiar caveat — past performance does not guarantee future returns.

Moreover, a major drawback to backtesting is the fact that past time periods can be arbitrarily selected and data easily manipulated to suit a desired outcome. For instance, a proposed strategy could be adjusted to perform well in a particular time period, say 2008 to 2009, or conversely, a specific time period could be adjusted to show success for the strategy. The manipulated strategy can then be refined repeatedly to what the strategists call “a perfect fit” to a historical period. However, what has actually been created is a customized strategy that performed well for a particular investment period, but it in no way ensures similar performance for the future. In other words, refining a backtest until a desired result is achieved can be extremely dangerous or risky, and at the very least, is a dishonest practice.

Finally, backtesting often fails to consider historical fluctuations in trading prices. If there’s not much liquidity in the market, the prices that we think are available would not be, if we actually were in the market trying to trade. In fact, trading costs and associated fees could actually eat up whatever benefit the proposed strategy thought was viable. The best approach, ethically, is for investment advisors to remain constantly aware of potential abuses of backtesting — particularly in terms of manipulating data to manufacture a positive benefit for a strategy — and not ignore better methods for evaluating investment strategies. One of the more practical alternatives, the simulation test, will be addressed in our next discussion.