Backtesting Anyone?

One of the things I like to do is take a practice that I observe in one industry, and think about how it might apply to another very different sector. As I have indicated in past columns, I genuinely believe that the best insights are often revealed by looking well outside of your own organizational or industry boundaries.

To that point, I’ve had an opportunity to talk last night with a friend of mine who is a “big time” equity trader. By “big time”, I mean he trades many times over what I could ever dream of investing. And for him, the last several weeks have been downright gut wrenching. Trends have had a real difficult time staying in tact, often reversing course on what would have normally been a longer term run. While his methodology plans for a good amount of that (i.e.- he expects to “lose” on about 40% of his trades (with built in stop losses of 2-3%)), he more than compensates by winning much larger returns (say 10-20%) on the other 60% that follow the expected trend.

But as most of you know, the market has been about as unpredictible lateley as it’s ever been. Traders have, for years, banked on trends which, believe it or not, are generally pertty predictible. But the last few weeks have not seen the ‘follow through’ that they ‘should’ have. We’ve had big breakouts that have reversed course unpredictibly. We’ve also had big breakdowns (like the major drop we experienced after the London bombings) that sent many traders “shorting” the market, only to see a major buying frenzy that lasted well into today. I’ve done a little bit of trading in my past, and I can tell you that it’s week’s like this that make you want to throw in the towel. And it’s those times that you need to be extra careful.

It’s during the unexpected change in performance patterns that traders start messing with their methodology. And there is nothing inherently wrong with that, as long as you can be reasonably assured that the change would have produced better results. Traders call that “backtesting”.

What differentiates great traders from poor ones (and I mean that literally) is that they backtest rigorously just about any change in methodology BEFORE they apply real money to it. How do they do that? They literally take the change in methodology (say entry or exit parameters) and apply it to all past trades, even ones they may have skipped, and see whether or not it would have produced more favorable results. Then and only then do they actually implement the process change. You may say that breeds over-analysis. Probably so. But I know from talking to a lot of these guys that it is this mindset that truly makes or breaks a trader.

I couldn’t help but thinking how useful that practice might be to the art of process management and performance reporting. How well do we really implement the performance management mantra of Plan, Do, Check, Adjust? Are the adjustments we plan to make tested against our historical metrics? Sure we look at pre and post performance based on the process changes we make, but do we go back and see how the process would have performed during the times where our old processes failed? That’s the real test of whether or not the process change actually makes sense. It’s during the tough periods that our processes are really put to the test, right?

I suspect many of us make changes to our businesses processes based on a particular problem that appears to be hurting our business. Does that mean every process we change based on “gut feel” or political pressures is the wrong move? Of course not. But we’d all be a lot better served by our own little bit of “backtesting”…challenging our proposed process changes against history, and seeing whether the new change would have made a difference.

Certainly, this is not a black or white topic. There will be times wher we change things on “gut feel”, just like the trader that breaks out of his method for a trade or two. That will happen, and you need to be flexible. But most traders will tell you that they do their worst when they trade for any length of time on pure instinct. Following a well backtested methodology almost always produces superior results.

Let’s all take a lesson from all those traders who got burned over the past few weeks. The really bad traders have probably already implemented changes that will likely fail because they are reacting to what was just an anomoloy in the market. Good traders will realize that and most likely stick to their process for many profitable trades to come.

Next time you contemplate a change in your business process, try and do a little bit of your own backtesting in different environments and see how the changes would have held up. The answers may surprise you.

.