The NAS Trading approach to evaluating trading ideas – Plus a few notes on related issues

I wanted to take a few moments to outline how I evaluating trading ideas, particularly as it relates to creating studies and conducting historical backtests (simulating trade results using historical data).   This article is going to be a little different.  If you are primarily interested in reading about specific investment/trading ideas, you can  skip this one.

The simplest way to describe my approach to evaluating trading ideas is with a bullet-point style list.  At the end I will add some additional comments.  In no particular order:

  • Start  the process with general concepts or a qualitative insight  - My process for doing this  is based upon observation, discretionary trading, reading, and conceptual understanding, which I have built up over a 12 year time period.
  • Evaluate independent ideas independently before seeking to combining multiple factors into one test
  • Simple is better – more parameters plus more optimization equals less reliable real-time results.  For example:  An immense number of patterns are possible using OHLC (Open, high, low, close) data.  I am highly skeptical of patterns that use a large number of, “if, then” type logical statements.  If you find yourself doing this, it might be better to further simplify and refine the idea.
  • Check for parameter sensitivity – For ideas that contain variable parameters, the best ideas  show favorable results over a broad parameter set.
  • Check for logically ordered results –  For example:  If a low value of a factor supposedly creates an above average forward return, does a high value create below average returns?
  • Test for significance – Bootstrapping/resampling, Out of sample testing, and the T-Test are all useful tools.  I prefer tests for significance that allow for the most recent data to be used in idea formation.  I almost never use the “out of sample” test that most people consider important.
  • My preference is for the simple T-Test or resampling methods.
  • Consider the number of hypotheses evaluated when considering significance
  • I never use the, “Test millions ideas and see if anything sticks” approach – Though I do know others who use this method extremely effectively.
  • Are “near neighbor” ideas effective?  I have found that the best trading ideas can be classified conceptually.   I have more confidence in a concept if I can evaluate it with multiple rule-sets, and all of the results are similar.
  • Test ideas across different markets – unless a good reason exists to think certain markets have different trading properties that will continue in the future.
  • Testing generic strategies (trend following, mean reversion, etc)  plus simple statistical tests like auto-correlation can provide valuable feedback about market dynamics.
  • Use trading ideas, don’t “believe in” them – Continue to learn and re-evaluate what you have found to be useful in the past

Backtest results and studies are simply information.  Studies can be interesting, informative, and useful without suggesting that the past will be like the future.

In fact, one of the greatest misconceptions is that strategy backtesting is about finding the ultimate trading system (You know, the one that shoots your equity to the moon with zero risk).  This is not the case.

It is better to view each trading idea or system as just one athlete on your team. You want a portfolio of ideas or systems that are not completely correlated (the less so the better) and have different sources of return.

Personally, I never seek to prove a trading idea to others partly because I have never fully proved one to myself.  I use a trading idea if I think it  has a very good chance of working.  I continue to use it so long as this is true.  If it performs in a way that is out of character relative to my expectations, I move on.  End of story.

The trading world and the academic world are two different things in much the same way that the inventor is different from the academic scientist.   Practical inventors are typically experimental and judge ideas based upon their immediate effectiveness.  Successful inventors embrace both positive and negative feedback and use it as a guide.  On the other hand, an academic who puts forth a mistaken idea could suffer a serious loss of prestige and do permanent career damage.  I told a friend with an academic background that his trading would  improve if he embraced  the freedom to look like a fool more often (not sure that went over well!).

In trading and investing, the only judge is “what works” and this is dictated by the market.  Seeking approval or accolades is counter-productive.  The best trading ideas are ones that are not overly obvious or satisfying to other market participants.   If you share a well thought out idea and have it shouted down, this feedback should encourage you to speed up your research and move to live trading – you might be on to something!

Traders who want a “sure thing” advantage need to get a position with an organization that has a structural advantage in the marketplace.  For example, access to customer order flow is an advantage that allows large financial institutions to make a trading profit almost every single trading day.  Other structural advantages include a cost of funds advantage, a technological advantage, or a time and space advantage.

Yet even these massive advantages are not risk free:  Hubris or catastrophic error (such as what recently occurred at Knight Capital) can sink even the most advantaged competitors.

Good Trading.

Leave A Reply (1 comment so far)

  • Atrad

    Hi Nat – Great post. I am just wondering if you can elaborate little more on what you mean by these three bullets. My background is not statistics.

    >> Consider the number of hypotheses evaluated when considering significance
    What do you mean by number of hypotheses evaluated? Is it same as the number of variables in the setup?

    >> Testing generic strategies (trend following, mean reversion, etc) plus simple statistical tests like auto-correlation can provide valuable feedback about market dynamics.

    Can you give an example of how these tests provide feedback to you about market dynamics.

    Why do you prefer resampling and tests for significance over bootstrapping? Does resampling here mean in each iteration the underlying price series data is resampled and then the setup/rules is tested on that resampled price data?

    I appreciate your sharing of knowledge. Your posts in general are high quality and provide actionable ideas and information.


Traders -