Developing an Investment Strategy? Consider a Simulation Test

In our last discussion, we pointed out some of the fundamental flaws of backtesting investment strategies. So, what’s a more viable approach toward evaluating financial strategies? We believe a more statistically rigorous and objective means is through use of simulation tests. Briefly explained, a simulation test is a way of comparing procedures for building portfolios by applying them to a variety of simulated outcomes over time.

In fact, simulation tests serve a variety of general purposes in all fields of business and finance. Without tailoring the method to perfectly match a specific set of historical data, as backtesting so often does, a simulation test rewards methods that work well for multiple scenarios. In other words, a simulation test estimates all of the possible outcomes for a particular strategy based on established statistical data and then determines which approach performs the best. The key to a simulation test is that the truth is known, but the procedures are run based on simulated data, independently from this truth. In other words, the methods are simulated as in real life, where the truth is unknown. The truth is then used to evaluate the realized performance of the methods. The mechanism of the simulation test can be imagined as a set of actors: a referee, who simulates a history based on the true parameters, and players, who run their respective methods on the simulated histories without knowing the truth. The referee then can score the methods with the truth behind the simulations. This is preferable to a backtest framework since the true capital market assumptions can never be known in a backtest.

Simulation tests can realistically evaluate which methods perform best for a particular return distribution model. In simulation tests that we have run and in independently run simulation tests, Michaud optimization has done very well.  Perhaps what is most important to note, however, is that a simulation test is less tied to a specific investment time period and how assets performed, as compared to backtesting. A simulation test uses just one simulation model to generate multiple datasets across the board for all approaches; it’s simply a matter of applying a different use of that information to determine statistically better performers, averaged over any time period.