The NAS Trading approach to evaluating trading ideas – Plus a few notes on related issues

I wanted to take a few moments to outline the approach NAS Trading takes to evaluating trading ideas, particularly as it relates to creating studies and conducting historical backtests (simulating trade results using historical data).   This article is going to be a little different.  If you are primarily interested in reading about specific investment/trading ideas, you can  skip this one.

The simplest way to describe the NAS Trading approach to evaluating trading ideas is with a bullet-point style list.  At the end I will add some additional comments.  In no particular order:

• Start  the process with general concepts or a qualitative insight  - At NAS Trading, this has developed over long period of time based upon reading, observation, conceptual understanding, and the experience of running many thousands of studies on all types of markets and strategies over a 10 year time period
• Evaluate independent ideas independently before seeking to combining multiple factors into one test
• Simple is better – more parameters plus more optimization equals less reliable real-time results.  For example:  An immense number of patterns are possible using OHLC (Open, high, low, close) data.  I am highly skeptical of patterns that use a large number of, “if, then” type logical statements.  If you find yourself doing this, it might be better to further simplify and refine the idea.
• Check for parameter sensitivity – For ideas that contain variable parameters, the best ideas  show favorable results over a broad parameter set.
• Check for logically ordered results –  For example:  If a low value of a factor supposedly creates an above average forward return, does a high value create below average returns?
• Test for significance – Bootstrapping/resampling, Out of sample testing, and the T-Test are all useful tools.  I prefer tests for significance that allow for the most recent data to be used in idea formation.
• My preference is for the simple T-Test or resampling methods.
• Consider the number of hypotheses evaluated when considering significance
• NAS Trading never uses the, “Test a million ideas and see if anything sticks” approach – Though I do know others who use this method extremely effectively.
• Are “near neighbor” ideas effective?  I have found that the best trading ideas can be classified conceptually.   I have more confidence in a concept if I can evaluate it with multiple rule-sets, and all the results are similar.
• Test ideas across different markets – unless a good reason exists to think certain markets have different trading properties that will continue in the future.
• Testing generic strategies (trend following, mean reversion, etc)  plus simple statistical tests like auto-correlation can provide valuable feedback about market dynamics.
• Use trading ideas, don’t “believe in” them – Continue to learn and re-evaluate what you have found to be useful in the past

Backtest results and studies are simply information.  Studies can be interesting, informative, and useful without suggesting that the past will be like the future or that the “gold mine” trading rule has been discovered.

In fact, one of the greatest misconceptions is that strategy backtesting is about finding the ultimate trading system (You know, the one that shoots your equity to the moon with zero risk).  This is not the case.

It is better to view each trading idea or system as just one athlete on your team. You want a portfolio of ideas or systems that are not completely correlated (the less so the better) and have different sources of return.

This suggests that rather than looking for, “the ultimate holy grail system” it is better to be realistic about individual system/idea performance, and instead “round out your team” of valid trading approaches.  To complete our analogy:  If one of your athletes (trading approaches) begins to behave in a way that is negative and out of character, replace him with an up-and-comer from the bullpen.   Results and ideas must be continually evaluated in real time based upon new feedback.  This is simply part of the investing process and not a problem in and of itself.

Personally, I never seek to prove a trading idea to others partly because I have never fully proved one to myself.  I use a trading idea if I think it  has a very good chance of working in real time.  I continue to use it so long as this is true.  If it performs in a way that is out of character relative to my expectations, I move on.  End of story.

The trading world and the academic world are two different things in much the same way that the inventor is different from the academic scientist.   Practical inventors are typically experimental and judge ideas based upon their immediate effectiveness.  Successful inventors embrace both positive and negative feedback and use it as a guide.

On the other hand, an academic who puts forth a mistaken idea could suffer a serious loss of prestige and do permanent career damage.  Indeed, academics snickered at many of Edison’s findings (and blunders) yet ultimately Edison’s process proved to be immensely productive.

In trading and investing, the only judge is “what works” and this is dictated by the market.  Seeking approval or accolades is actively counter-productive.  This is because the best trading ideas are ones that are not overly obvious or satisfying to other market participants.   If you share a well thought out idea and have it shouted down, this feedback should be considered encouragement to further explore and evaluate that idea.

Traders who want a “sure thing” advantage need to get a position with an organization that has a structural advantage in the marketplace (For obvious reasons, these positions are very difficult to get).  For example, access to customer order flow is an advantage that allows large financial institutions to make a trading profit almost every single trading day.  Other structural advantages include a cost of funds advantage, a technological advantage, or a time and space advantage.

Yet even these massive advantages are not risk free:  Hubris or catastrophic error (such as what recently occurred at Knight Capital) can sink even the most advantaged competitors.

Moving forward, I have one article on deck that will discuss risk – both investment and trading risk, but also with a broader perspective.  However beyond this, NAS Trading will focus on creating value for subscribers with specific trading and investment ideas.

Good Trading.