Backtesting

Become a sponsor »

Backtesting is the process of applying a set of trading rules, or trading system, to a past sequence of actual market data, in order to simulate the results of the trading system over time.

Purpose and Scope

Backtesting is used as a way to evaluate the merits and behavior of a trading system. System developers often use backtesting as a way to gain insight and test their progress as trading systems are being created. Professional traders may also use backtesting results as a way to demonstrate the behavior of new systems that are being offered to potential investors.

Louis B. Mendelsohn was the first person to introduce strategy back testing in PC software in 1983 with the software, ProfitTaker.

Since the goal of backtesting is to create a simulation of "what would have happened", the practice is limited by nature to trading systems that are mechanical.

Trading methods that contain qualitative judgments (i.e. a human making a decision based on intuitive factors or applying other parameters that are external to the buy and sell signals being generated) cannot be guaranteed to produce the same results on the same data every time. A trader who receives a signal to buy in a market, but bypasses the signal because the perceived chances of success are too low, may react differently to an identical signal given under different emotional or market conditions. Any trading method where the trading decisions could vary depending on factors external to the mechanical rules, cannot be tested effectively through backtesting.

Any fully-quantified trading system is potentially a candidate for backtesting, regardless of whether it is simple or complex, fully numeric (e.g. comparisons of price relationships) or based on specific conditions (e.g. "short the market each day after a new contract month is listed at the exchange, and offset the trade three days later"). The key is not whether a system has a certain kind of rule, but rather that the system can be fully rendered in a mechanical way.

Methods

Prior to the emergence of cheap, universal, powerful desktop computing in the early 1990s, backtesting methods for the average trader largely required the use of paper charts and individually recorded results on ledger paper or other handcrafted tables. Welles Wilder's book New Concepts in Technical Trading Systems (ISBN 0894590278), published in 1978, contains good examples of the daunting processes required to backtest even simple technical trading systems by hand.

The arrival of the desktop computer brought a new era in technical analysis and backtesting. Software platforms to display market charts were combined with analysis tools, historical market price databases and real-time data feeds, together providing the opportunity for both average investor and professional trader to research and develop new ideas with unprecedented ease and speed.

Modern backtesting practices generally involve the creation of a trading program within a development platform (MetaStock and Tradestation are the long-term survivors in this category; RTS Realtime Systems Group offers an array of different software solutions for institutional and individual clients). The trader uses a programming language and an extensible library of analysis and trading functions provided by the development platform, to detail all the conditions for entering and exiting the target market or markets.

Once the program is ready for testing, the trader simply selects the historical market data to be used for execution. Even if the development platform doesn't support its own historical price database, it will have some mechanism for accepting and processing a range of market data. The testing software steps through the trader's program, applying the programmatic rules and recording buys, sells and trade outcomes.

Typically, testing platforms will produce a results summary at the end of execution, including information that some traders find useful, such as the percentage of winning trades, largest drawdown, worst and best outcomes, and overall profitability. The trader will often evaluate the results of the execution and adjust the system's rules or parameters based on the outcome.

With all that said, the use of a comprehensive trading platform is not a necessity or a requirement for the creation of a successful trading program. It is possible to use more generic tools such as Microsoft Excel, for those traders who are also proficient in the Visual Basic scripting language that Excel provides. By using a more manual approach, traders exchange the ease and convenience of the development platform with the ability to fully control how all decisions and functions are implemented, and can tailor their end results to include any kind of data from the mundane to the arcane. The solution can also be appealing to individual traders who lack the budget to license a fully-extensible and flexible suite of tools.

Effectiveness

Backtesting is often treated as a means of developing confidence in the future success of a trading method; it is sometimes mentioned along with paper trading as a way for new traders to become successful. There is also a fair amount of writing dismissing backtesting as wholly ineffective.

Backtesting Problems

When considered as a direct measure of a trading system's future effectiveness, backtesting fails. One of the main causes for that failure is that the exact market conditions and relationships used in testing are simply not going to be duplicated in the market data of the future. Whatever results a mechanical system returns across one set of market data will be different across all sets. The evidence of changing and unpredictable market conditions is so pervasive that regulatory bodies such as the National Futures Association have encoded rules and requirements to warn potential investors not to rely solely on past results.

System developers who backtest their methods inevitably find episodes where their system's performance is unacceptable. In order to improve their system, they make changes to the rules to correct its behavior during the failure episode. Unfortunately, at that point it becomes difficult to detect whether the developer is actually improving the system or merely optimizing its behavior to fit the specific set of market circumstances in the test data. Since backtesting with actual historical data can use only a limited set of market data, after the first round of testing the risk of optimization increases. The more rounds of testing and changing, the greater the risk that the system will become less flexible through optimization.

In addition, there are several areas of analysis that backtesting generally doesn't cover that are critically important for the success of a trading program:

• Like any other piece of software, a mechanical trading system is vulnerable to logic errors and coding defects. Any software effort should include rigorous design and implementation analysis to find problems. Backtesting only superficially addresses those needs.
• Backtesting doesn't take into account environmental risks and failures, such as network downtime, hardware failure, time off for the trader, and other things that can impact bottom-line results.
• Backtesting isn't generally able to take market liquidity factors into account when determining entries and exits. Some programmers factor in a certain amount of fill slippage, assuming actual prices to be a certain amount worse than the system intends, but a "normal" amount of slippage may not apply during frantic trading after a surprising economic report, or during an extended period of volatility that can last days or weeks.
• Some systems are capable of producing an ambiguous trade results in certain conditions. For example, if a trading system is to go either long or short on a breakout from the previous day, and the day's market activity is an outside day, producing both a higher high and a lower low, then which way would the trade have actually gone? Unless the simulator has a way to note or resolve such issues, the trading results are at risk for more inaccuracy.

Backtesting Strengths

By leaving behind the notion that backtesting is a way to say, "This is what would have happened," and even more dangerously, "...and this is what you can expect in the future," running simulations across historical data is an excellent method for identifying logical or assumptive errors in the trading program.

For example, a trading system that doesn't bother to set loss points will very likely appear in simulation to be extremely hazardous over time. As long as the developer thinks, "Oh no, I guess I can't run a system that has no risk controls" instead of, "If I adjust the entry requirement by three ticks then that painful account wipeout would not have happened," then backtesting has served a valuable purpose.

Backtesting can therefore identify errors in assumption, provide what-if answers, (e.g. "What if the system isn't right as often as we projected?"), and can aid in determining the practicality and flexibility of the system. Backtesting may also provide good information concerning what's needed for properly capitalizing the system for trading.

Nevertheless, backtesting's effectiveness is limited to the variety of circumstances contained within the test data. Testing only in a trend or only in volatile circumstances, may not yield useful information about how the system behaves during quieter times.