Backtesting Issues

I'm in the process of validating some investment ideas and thought I'd share some of the hidden biases that occur when comparing systems against each other and the market in general. I should know...these issues have bit me numerous times over my past 7 years of system testing. Maybe the testing methods below will help you as they have helped me.

Failure to Trade All Signals

• Every signal received from your idea should be tested. Which means taking trades for a stock you already own. If your idea is to buy 20-day highs and you receive your first signal for YHOO on 10/15/2007...you buy it. If you receive another signal on 10/16/2007 for YHOO...guess what? You buy it. Despite already owning the stock. Failure to do so means your filtering the idea. This is okay if you already know the idea is valid...but if so...then why are you testing in the first place?

• A big obstacle in taking all signals is money. First test the idea without money constraints. Money should not limit your trades. In fact, when testing...the only limit on your trades should be the idea.

Failure to Test the Other Side

• The Trading Digest covers idea reversals in their article on the 50/200 Moving Avererage Crossover system. In essence, any idea should be reversed, tested, and compared with the original idea.

• So, if the idea is selecting 20-day highs...the reverse of that idea is selecting stocks less than or equal to their 20-day highs. But, performing this test presents a problem. There are fewer stocks at their 20-day highs (idea #1) than stocks not at their 20-day highs (idea #2).

• It's unfair to compare 150 trades from idea #1 against 5,000 trades from idea #2. Whose to say randomly selecting 150 trades from the 5,000 trades in the idea #2 sample set wouldn't generate as good if not better results than idea #1?

• And that's just how to solve the problem...boot-strap it. Take those 5,000 trades and randomly select 150 trades. Compare those results against idea #1's sample set of 150. Of course, it wouldn't be wise to select one 150-set sample from 5,000. Take several 150-set samples...compile the results and use that for the comparison.

Remove the Best Trades

• With any system...the Pareto Principle is large and in charge. You'll find only 5% to 10% of the trades generate the majority of the idea's profits. What are the chances you would have missed a few of those trades? Throw money limits into the mix and real world events like vacations and that home remodeling project your wife has been bugging you about...and it's very easy to do. So, remove them from the test sample. Don't let a few trades in the sample play such a large role in the results.

Market Exposure

• When buying presents for your kids...the number one rule is every kid must receive the same number of presents. If you fail here...you will pay. Testing and comparing ideas are a lot like your kids. Give them equal time and exposure.

• Use the same signal starting point for the comparisons. If your testing date range is 01/01/1995 - 12/31/2004 and idea #1 kicks off it's first signal on 05/01/1995 and idea #2 kicks off it's first signal on 02/01/1996...no fair! Idea #1 received a 9 months head start on idea #2. Might not seem like much...but with markets and babies...9 months can make a difference.

• In fact, it usually pays to bucket the trades in similar timeframes. That way you can ensure not only the trades share similar signal start times but also similar bucket sizes. For example, to test 10 years of data...01/01/1995 - 12/31/2004:

Bucket 1 -> 01/01/1995 - 12/31/1996
Bucket 2 -> 01/01/1997 - 12/31/1998
Bucket 3 -> 01/01/1999 - 12/31/2000
Bucket 4 -> 01/01/2001 - 12/31/2002
Bucket 5 -> 01/01/2003 - 12/31/2004

• The buckets 1 - 5 above would be the testing date ranges. After the simulations are complete for each idea and corresponding bucket...verify idea #1's bucket 1 contains roughly the same number of trades as idea #2's bucket 1. If not, then boot-strap the larger bucket size down to the smaller bucket's size. Compare results. And remember, remove the top 5% or so trades from each idea's buckets.

[Clarification: The testing date ranges in the buckets listed above are not the start and end dates for your trades. The date ranges are the start and end dates for your idea. Meaning...all ideas triggered from 01/01/1999 to 12/31/2000 would be in Bucket 3. You would need to track those signals for as long as you stay in them...which may mean all the way to your end date of 12/31/2004. Make sense?

In essence, your overall begin and end dates are 01/01/1995 - 12/31/2004. Then you slice and dice the signals that occurred during that time frame based on entry date into buckets 1 - 5.]

Questions?

• What if the idea consists of several smaller ideas such as selecting stocks at their 20-day high and on that new high experience a 30-day volume surge? In this case, you would need to iterate through the above process twice.

• In the first pass, you'd compare the 20-day high stocks against stocks not at their 20-day high. In the second pass (if the first pass proves fruitful), compare the 20-day high stocks with 30-day volume surges against 20-day high stocks without 30-day volume surges.

As you can see, getting an idea through the validation process is quite an ordeal. Unfortunately for us system traders; very few ideas make it out alive. Thankfully, we don't have to cut those ideas from our sample set.

Later Trades,

MT

Quote of the Week - 10/10/2007

"...the most important quality for a trader to develop is discipline. As you've read, my stubborn ego and impatience prevented me from achieving lasting success and financial security. I hope my story has shown you that any fool can get lucky and quickly make a great deal of money. But, if playing the stock market was always that easy, there would be no need for research and hard work. Considering that all the information you need to be able to profit is available on the Internet, what sets successful traders apart is their ability to wade through all the muck. With regard to your sources, keep an open mind. As my losses demonstrate, if you allow your emotions and ego to control your trading, you are doomed to fail." -- Timothy Sykes from his book, An American Hedge Fund

What great words of advice! Reading Tim's book brought back the memories of what it was like trading in the greatest stock bubble of our time. In fact, I participated in the ISCO trade he mentions in the book. What a ride, indeed!

I look back and realize just how far I've come as a trader and how much further I still have to go. Tim is right...cutting through the muck is a tough chore for any trader/investor. The key to long-lasting success is keeping an open mind, learning from your failures, and working hard...really really hard. Of course, a sprinkle or two of luck never hurts.

Thanks for the book, Tim.

Later Trades,

MT

Recent Links for 09/21/2007

Newbie - converting csv files to arrays in NumPy Great message thread on how to convert csv files to numpy arrays.

Cookbook/InputOutput - Numpy and Scipy File processing examples using numpy, scipy, and matplotlib. How to read/write a numpy array from/to ascii/binary files.

Numpy Example List Examples of Numpy functions such as fromfile(), hsplit(), recarray(), shuffle(), sort(), split(), sqrt(), std(), tofile(), unique(), var(), vsplit(), where(), zeros(), empty(), and many more.

Introducing Plists: An Erlang module for doing list operations in parallel Could you spawn a trading system process for each stock of a given day's trading (a list)? What if you had 20,000 stocks for a given day? Can plists/erlang handle 20,000 processes without hitting memory constraints?

Recent Links for 09/19/2007

130/30 Strategy Backtested Disagree with comparison of 130/30 to non-leveraged benchmark/strategy. Also, the additional longs didn't improve the long portion of the 130/30 returns compared to the long-only strategy returns. Am I missing something?

Recent Links for 09/18/2007

Chapter 22. Struct and Array Modules Overview of the python struct and array modules

Building Skills in Programming Nice python tutorial.

Python Grimoire Nice python cookbook.

Recent Links for 09/17/2007

• One year ago I was a stock market analyst [Part 1]
  Very interesting how he was able to process 9,000 stocks in less than a minute.


• Goldblog - Python - Stock Quote Module
  Interesting python stock quote utility.


• Goldblog - Investment Advice
  Excellent investment advice. Keep it simple.


• NumPy for Numeric/numarray users
  Cheatsheet for NumPy Python users

Recent Links 09/17/2007

Goldblog - Investment Advice

• Excellent investment advice.  Keep it simple...
  
  • 
    There are 2 simple questions you must first answer:
    
    1. What is the time-frame in which you need access to your money? (next week? next year? 10 years? retirement?)
    
    2. How much risk and volatility are you comfortable with?
    
  • 
    The Answer:
    Asset Allocation... not fundamental analysis, not technical analysis, not market
    trending, not tips from brokers and analysts ... but straight up asset
    allocation.
    
    
  
  
  
  
  
  
  
  
   - post by taylortree

Recent Links for 09/15/2007

Links for 2007-09-15 [del.icio.us]

Posted: 16 Sep 2007 12:00 AM CDT

• Practical Common Lisp
  Excellent way to get started with Common Lisp.


• 9 Things You Simply Must Do
  Friend of mine sent me this great post on Dr. Cloud's 9 principles commonly practiced by successful people. My favorites?
  

Principle #2: Pull the Tooth - face your fears...don't put off today what you can do today.


Principle #4: Do Something

• ONLamp.com -- An Introduction to Erlang
  Great coverage of the Erlang language.


• Python Cheat Sheet
  Simple little python cheat sheet.

Recent Links 09/15/2007

Practical Common Lisp

• Great coverage on Common Lisp.
   - post by taylortree

Recent Links 09/06/2007

Quantmod - Quantitative Financial Modelling Framework for R

• Offers R language modules to...
  
  • calculate periodic returns
  • retrieve historic quotes from Yahoo, Google, FRED
  • there's even a tradeModel that looks interesting
  • and well documented.
    
  
  
  - post by taylortree

Recent Links 09/05/2007

Speed up R, Python, and MATLAB - Going Parallel

• Reduce runtime of Python, R, and MATLAB applications by 85%?  Process 10-100X  larger datasets?  With just a few code changes?  Not quite sure how...but something to explore in the future.  Their success story on speeding up MATLAB code for Monte Carlo Analysis looks pretty easy of a code change to me.  Read their blog for further insights into HPC...
  
  • Multicore:  Why all the Hubbub?
  • What is "Productivity" in High Performance Computing?
  
  
   - post by taylortree

Recent Links 09/04/2007

World Beta - Engineering Targeted Returns and Risk: More On The Endowment Style Of Investing  Annotated

• World Beta shares some links covering the endowment investing side of things...
  
  • A link to
    Frontier Capital Management
    - check out their knowledge section for more great papers similar to the ones Faber links to.
  • Faber mentions a great upcoming book covering the twelve top endowment CIO's .
  • from Alpha Magazine...Highbridge Capital Managment shares its office organization - putting traders and developers together.  I've always thought this would be a great idea in any shop.  By putting users and developers together - manual taks can be seen and automation can happen.
    
  
  
   - post by taylortree

Frontier Capital Management

SourceForge.net: tkdiff

• Great little file compare utility.  Graphic front end to the diff program.
  note:  tested this today against a large file/program (well, not that large in my line of work...but I guess to Google's)...couldn't handle it.  But, works great on small files.
   - post by taylortree

Google Mondrian: web-based code review and storage

• Online code review that works like a blog/wiki.  I wonder...is it possible to create a code review system similar to Mondrian within a source management toolset such as subversion?  Seems like most of the backend is there already...would only need to add some front end tools to display the changes being committed and allow comments on those changes.
   - post by taylortree

Recent Links 09/03/2007

ONLamp.com -- Numerical Python Basics

• Numpy basics.
   - post by taylortree

Programming in R

Finding Duplicate Elements in an Array :: Phil! Gregory  Annotated

• Interesting way to find duplicates in an array.  Enjoyed the links on the pigeonhold principle and Floyd's cycle-finding algorithm.
   - post by taylortree

Now, suppose that the array is of length n and only contains positive
integers less than n. We can be sure (by the pigeonhole principle)
that there is at least one duplicate.

So, how do we find the beginning of the cycle? The easiest approach is to
use Floyd's cycle-finding algorithm. It works roughly like this:
Start at the beginning of the sequence. Keep track of two values (call
them a_i and a_j). At
each step of the algorithm, move a_i one step
along the sequence, but move a_j two steps. Stop
when a_i = a_j.

Investor or Gambler?

Tom from InvestorGuide.com sent me an article of his to read regarding the differences between investing and gambling. Tom did a great job in discussing the two terms fairly. Very hard to write an article like that without exposing unknown biases.

I did find a couple of very minor areas where I disagreed with Tom's article. I shared those comments to Tom in an email. But, felt those comments would be helpful to readers of this site. First read Tom's article. Then my comments below...

Tom's article:
There's a big difference between buying a stock after thoroughly researching it and buying a stock by hitting it on a dartboard.

My comments:
Is there really a big difference...in outcome? Sure, the person may feel different about the investment...but based on outcome alone...historical evidence would suggest the odds of success are approximately the same.

Tom's Article:
Gambling - "Any activity in which money is put at risk for the purpose of making a profit, and which is characterized by some or most of the following...no net economic effect results."

My comments:
I would argue that each player in the stock market provides a positive economic effect. The investor provides long-term capital to companies in need of capital. The speculator and gambler provide liquidity. Sure there are negative effects from all players...investors prop up some companies that probably shouldn't receive further funding...and will eventually go bust. And speculators/gamblers can turn liquidity into a frothing market that can cause long-term problems after the swell has subsided.

Of course, your point is true that gamblers' short-term trades may be a net effect with each other...but that activity regardless of reason or length of hold...still provides liquidity for other players in the market.

Basically, remove any player from the game...and the market wouldn't be what it is.

That's it from here where I've got a softball game to prepare for this week. I haven't played softball since my college days. And haven't thrown many balls since my shoulder surgery. Should be an interesting show to say the least.

Later Trades,

MT

Weekend Readings - Search for Alpha

Sharing some great links found this morning while enjoying a Starbucks coffee and Krispy Kreme doughnut. :-)

A great Quantitative Primer from the Bionic Turtle.

Nice little profile on Parametric Portfolio Associates by Bloomberg Markets Magazine. I enjoyed reading their rebalancing alpha along with their contrarian weighting of individual stocks for each country. In fact, check out Parametric's Research & Whitepapers area...great information to be had. Their paper on Using Statistical Process Control to Monitor Active Managers is one to read a few times this weekend.

One view shared by Parametric coincides with my recent testing of alpha filtering in the U.S. Equities market. Parametric found that smaller countries exhibit less correlation to the broader markets. My guess is due to less investor attention, liquidity issues, and such. In essence the markets are less efficient which enable Parametric to extract alpha (non-commoditized beta) by overweighting these smaller countries in their portfolio compared to the index they track. I found that by increasing the frequency in my alpha calculations...the fewer stocks I found that could beat the market (high alpha). Increasing the frequency to daily in my alpha filter picked up only the most illiquid stocks in the market out of the 20,000 stocks in my database (including the delisted). What does this mean? These stocks are the less followed? Less efficient? Their returns are least effected by the broad market? Short-term...all other stocks are governed by the overall market returns? Only if you extend your time horizon do stocks capture more alpha from the market? My recent testing shows this to be a possibility.

So, what should an investor/trader do? Well, if you're a daytrader, you should pay much more attention to the market in general. The market acts as a powerful force in a stock's short-term returns. If you're an investor...extending your time horizon months if not years is a way to capture more (alpha) than the market gives (beta) to each individual stock. Think about it...anywhere the market is efficient...the market governs the behavior. Only when you drive outside of that efficient zone do you capture non-market behavior. Holding stocks for the very long-term is one road outside of the city.

World Beta finds an interview with Harvard's Endowment manager,El-Erian. El-Erian claims his biggest challenge is overcoming the popularity of the endowment model. Which I tend to agree.

Finally, for you R fans out there...check out the Econometric tools for performance and risk analysis created by Brian G. Peterson and Peter Carl. Need to calculate the Information Ratio, Sharpe Ratio, Sortino Ratio, max drawdowns, rolling returns, CAPM, Kelly Ratio, and Omega on your portfolio's returns? Then this PerformanceAnalytics package could be just the ticket. I have yet to work with this feature-rich package...but have it on my list of to-do's over the coming weeks.

That's it from TaylorTree...where the weather is beautiful, coffee is great, and I've got an afternoon of fishing in my future.

Later Trades,

MT

Manager Focus - Robert B. Gillam

I recently read an interview with McKinley Capital Management founder, Robert B. Gillam. Very interesting guy and investment style. Mr. Gillam setup shop in Alaska of all places. Despite the remote location...he runs a heavy "quant" shop. His work on ranking of stocks based on performance and risk (OR Index) bears further study from yours truly.

What drew me to Mr. Gillam's interview was his research on stocks making new highs and new lows. Mr. Gillam found stocks making new highs on any given day to have a greater than 70% chance of hitting another new high in the next 90 days. And stocks making new lows to have a greater than 65% chance of making another new low in 90 days. Then goes into the how and why of his research. Great stuff!

Check out these articles for further information on Mr. Gillam...

Background on Robert Gillam and McKinley Capital Management.

U.S. Blinders Hinder Stock Pickers. This is a great article covering Mr. Gillam's expansion into international investing.

Here's an article from Mr. Gillam's son, Robert A. Gillam...

STUDY CHALLENGES CONVENTION: Growth, value style distinctions also important in international investing from Pensions & Investments.
This breakup between styles reminds me of an interview with DFA's founder, Rex Sinquefield. Mr. Sinquefield discussed how value stocks were riskier than growth stocks despite their similar volatility. Thus the reason for value stock's higher returns over the years.

Have a great week!

MT

Weekend Linkfest!

The Beta in Alpha's clothing? (pdf) Bridgewater Associates' detail how various hedge fund's are charging Alpha prices for Beta returns.

How to differentiate Alpha from Beta in those hedge funds? Check out AllAboutAlpha's post titled, Mommy, Where do alphas come from? AllAboutAlpha does a great job of summarizing Andrew Lo's paper, Where Do Alphas Come from?: A New Measure of the Value of Active Investment Management. Lo explains how to differentiate active returns from passive and more importantly what value the active returns add to the total returns of the portfolio. Cool stuff!

Very interesting draft, When Do Stop-Loss Rules Stop Losses?, by Kathryn M. Kaminski and Andrew W. Lo. They find stop-losses improve returns and reduce volatility compared to buy & hold. And stop-out periods were distributed uniformly over time versus only during small market crashes.

MOSERS discusses how to handle residual cash in a portfolio in the following newsletters, Rebalancing and Cash Securitization and Rebalancing II. I was shocked to discover how big of an impact residual cash had on my returns during my market studies. A cash drag indeed! Very important to get that cash, no matter how small, back into the market in order to generate further market returns.

Enjoy your week!

MT

Quote of the Week - Busy Bee

"I really believe that success is just getting up one more time than you fall. It doesn't come from one brilliant idea, but from a bunch of small decisions that accumulate over the years. And you shouldn't underestimate the amount of work that's involved, the amount of fear that's involved." -- Roxanne Quimby

I love unconventional success stories. And Inc.com shares a great one with Burt's Bees founder, Roxanne Quimby.

Enjoy Quimby's story? Here are a few more interviews with the busy bee...

After Burt's Bees, What? from Business Week

Interview with Burt's Bees founder, Roxanne Quimby from Hilary Magazine.

MT

Sky is Falling...

According to this article, Jeremy Grantham believes the sky is falling.

"The bursting of this bubble will be across all countries and all assets, with the probable exception of high-grade bonds," Grantham warned. "Since no similar global event has occurred before, the stresses to the system are likely to be unexpected. All of this is likely to depress confidence and lower economic activity."

But, plenty of hedging is done as to the timing of the fall.

As for timing, he (Grantham) concedes that's impossible to predict. But here's the kicker: Even Grantham thinks you probably need to be bullish right now. The reason? Most bubbles, he notes, go through a short but dramatic "exponential phase" just before they burst. Like Japan in 1989 or the Internet in early 2000.

How does this type of prognostication help anyone...I asks ya's?

Grantham's bold statements and subsequent hedging of bets is something a lot of market pundits do. This allows them to zig and zag at the same time. If the market goes down big, Grantham claim's victory. If the market goes up big, Grantham claim's victory. A win-win scenario.

This type of hedge may prove useful in other pursuits. For example, wouldn't it be nice to lead a programming project and hedge the timeline of the project?

Yes, we will make the May 1st deadline unless we don't.

The above statement doesn't work because it's not wordy enough. Too simple...straight to the point.

We fully expect to meet the May 1st deadline. All components have been reviewed, tested, and verified to meet our stringent requirements. But, there is always the case that problems may arise due to circumstances outside our control. These problems may impact our schedule and possibly result in extending the deadline.

There, that's better. A great hedge!

Well, that's it for me this Saturday morning. I'm taking my daughter fishing today. Of course, the fishing trip may be delayed or cancelled if changes in the Earth's atmosphere produce drops of water from the sky. ;-)

Later Trades,

MT

Quote of the Week

"If you spend some time thinking about it ("ideal position sizing") you will realize that life just isn’t so simple. Every individual has different tolerances for different types of risks. A formula won’t capture all of them and a formula most certainly hides information that might be very valuable." -- Curtis Faith

Great article posted by Curtis today . Discusses some of what I found to be true in regards to the Sharpe Ratio and smoothness of returns. In fact, through much testing I have found using any type of volatility measure in my trading systems (entry, exit, position sizing, etc.) produces sub-par results when real-world results take effect. But, sometimes, those are the best measures we have. So, what do you do?

I think it's important to do what Curtis asks...

I’ll leave you with an exercise. Take your favorite position sizing methodology and then see what might happen if you happened to take the wrong side of a trade using that methodology on September 11th, 2001 or on Black Monday (October 19th, 1987).

In regard to formulas hiding valuable information. Pay attention to your stock data. There's a formula in there adjusting your time series for splits and possibly dividends. The longer term your system is...the more important this adjustment formula will play in your overall results. Common entry/exits formulas such as Average True Range (ATR) become rather out-of-date when processing IBM back in the 1960's. Something for all aspiring long-term trend-followers to consider when backtesting their systems.

Enjoy your week! I'm enjoying my first day of vacation since moving up to Missouri. And what a pretty day I picked out. What to do...what to do... ;-)

Later Trades,

MT