TaylorTree

Tuesday, December 11, 2007

To Design or Code?

"The one who does the work decides." -- KDE principle

Jeff Atwood over at the Coding Horror blog discusses a fascinating problem in software development. Doers and talkers. Designers and coders.

I believe all developers need to have a bit of both in their toolbox. Mainly, because the first design is most always changed due to scope creep (failure to see all the pieces to the puzzle). If you spend all your time talking about that first design...you never get to coding. And if you can't get to the coding...you'll fail to find those missing puzzle pieces. And fail to deliver a prototype for the customers to evaluate.

Designers, this means getting your hands dirty in order to create something to improve. Developing systems is an iterative process. Design, code, design, code. And yes, even code, design, code, design. The ultimate goal is to refine the process until you and your customers are satisfied. Whatever it takes. And yes, that means moving to the coding stage even when the optimal design has yet to reveal itself. It's really a Kaizen process. Small accomplished improvements to the initial design pays dividends to all.

Coders, this means before plunging forward hacking away at the problem...ask for feedback of your idea and possible alternatives to the problem. It's important to starting coding down the right path. One that encompasses as much information of the problem as possible. This means you'll need to bring some information to the design table yourself. Perhaps a bit of discovery coding must take place to figure out what elements are involved and possible problems or bottlenecks in your proposed solution. This also helps in keeping the design discussion focused.

So, how does this apply to investing? Well, how many investors/traders do you know that invest without a plan? Without a design? An overriding investment philosophy? Just plunge ahead into the market?

These type of investors would be well served by stepping back a bit and design their investment model. Then ask for feedback of their proposed design. It's okay to perform some discovery trading first. Determining how the market handles your ideas. But, gather what you need and then design. Then invest with the goal of continually refining your design.

What about investors/traders who are afraid of the market? Have not found an investment model that is perfect? And refuse to step a toe in the market waters until they feel 100% comfortable in their design? Problem with this thinking is knowledge requires experience. And nothing is ever 100%...especially in the market. So, create your investment manifesto and then try it out. You can't improve upon something that isn't there to improve. And you can't design a successful investment strategy if you don't have market experience.

Later trades,

MT

Saturday, December 08, 2007

Breakthrough Programming

“The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.” -- George Bernard Shaw

Scott over at the scottberkun.com blog links to an amazing paper on Managing for Breakthroughs in Productivity. The article discusses how breakthroughs occur...and don't occur. My favorite quote...

For breakthroughs to occur, people must be given a chance to do work than can not be proven: ambition and risk are necessary for breakthroughs. If individuals are not trusted to take risks, breakthroughs are unlikely.

Spot on! Programming and risk goes hand in hand. As do programming and ambition.

To be an effective programmer you must have the ambition to automate tasks...all tasks. That's your job. If you spend your time manually putting widgets together...then you're not really programming.

And in order to automate tasks...a programmer has to take risks. Cause you are programming something that has never been done by a computer before. At least that computer. And most likely, never been programmed by you before.

MT

Friday, November 30, 2007

Programming Culture

Hat tip to Howard Lindzon for sharing this post...Software Engineering Tips for Startups

Alex's tips for startups is one of the best posts I've read in a long time on building a programming culture. I realize the focus is on tech startups. But, I believe his points are applicable to all companies with programming departments.

Some of my favorite quotes...

So the first tip is to always have a strong technical co-founder. Someone who shares or invents the business along with others, but also has the technical feet on the ground. Someone who can make sure the business is mapped onto technology correctly.

Avoid hiring managers...What you need are experienced technical people who love coding. These are going to be natural mentors for your younger engineers. Mentors and not managers.

Coding becomes sculpting. Starting with a shapeless form you continuously refine the code to satisfy the business requirements and make sure that the system is designed and implemented correctly.

And probably my favorite in regard to hiring programmers...

...candidates need to demonstrate love for simple and elegant code.

Simple and elegant code = Simple and elegant company.

I would also like to add one more trait to look for in programmers. Willingness to share knowledge. Evidence of this sharing trait in the interview and in past performance.

For example, all programmers develop tools to make their jobs easier. But, do they develop those tools for themselves only? Or for others to use as well? We all know...developing anything for others to use is the harder path to follow. But, without knowledge transfer, the wheel will be re-invented...again, again, and again.

Have a great weekend,

MT

Wednesday, November 28, 2007

Quote of the Week - 11/28/2007

"The Things to do are: the things that need doing, that you see need to be done, and that no one else seems to see need to be done. Then you will conceive your own way of doing that which needs to be done — that no one else has told you to do or how to do it. This will bring out the real you that often gets buried inside a character that has acquired a superficial array of behaviors induced or imposed by others on the individual." -- Bucky Fuller

Great quote. What do you see that needs to be done? And more importantly...what are you doing about it? But, a precaution...doing what needs to be done that hasn't been done by others prompts others to stall/halt the doing. Do you have what it takes to overcome the others in order to get the doing done?

MT

Saturday, November 10, 2007

Quote of the Week - 11/10/2007

"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." -- Albert Einstein

Something to think about when developing systems.

MT

Sunday, October 21, 2007

Backtesting Issues

I'm in the process of validating some investment ideas and thought I'd share some of the hidden biases that occur when comparing systems against each other and the market in general. I should know...these issues have bit me numerous times over my past 7 years of system testing. Maybe the testing methods below will help you as they have helped me.

Failure to Trade All Signals

Every signal received from your idea should be tested. Which means taking trades for a stock you already own. If your idea is to buy 20-day highs and you receive your first signal for YHOO on 10/15/2007...you buy it. If you receive another signal on 10/16/2007 for YHOO...guess what? You buy it. Despite already owning the stock. Failure to do so means your filtering the idea. This is okay if you already know the idea is valid...but if so...then why are you testing in the first place?

A big obstacle in taking all signals is money. First test the idea without money constraints. Money should not limit your trades. In fact, when testing...the only limit on your trades should be the idea.

Failure to Test the Other Side

The Trading Digest covers idea reversals in their article on the 50/200 Moving Avererage Crossover system. In essence, any idea should be reversed, tested, and compared with the original idea.

So, if the idea is selecting 20-day highs...the reverse of that idea is selecting stocks less than or equal to their 20-day highs. But, performing this test presents a problem. There are fewer stocks at their 20-day highs (idea #1) than stocks not at their 20-day highs (idea #2).

It's unfair to compare 150 trades from idea #1 against 5,000 trades from idea #2. Whose to say randomly selecting 150 trades from the 5,000 trades in the idea #2 sample set wouldn't generate as good if not better results than idea #1?

And that's just how to solve the problem...boot-strap it. Take those 5,000 trades and randomly select 150 trades. Compare those results against idea #1's sample set of 150. Of course, it wouldn't be wise to select one 150-set sample from 5,000. Take several 150-set samples...compile the results and use that for the comparison.

Remove the Best Trades

With any system...the Pareto Principle is large and in charge. You'll find only 5% to 10% of the trades generate the majority of the idea's profits. What are the chances you would have missed a few of those trades? Throw money limits into the mix and real world events like vacations and that home remodeling project your wife has been bugging you about...and it's very easy to do. So, remove them from the test sample. Don't let a few trades in the sample play such a large role in the results.

Market Exposure

When buying presents for your kids...the number one rule is every kid must receive the same number of presents. If you fail here...you will pay. Testing and comparing ideas are a lot like your kids. Give them equal time and exposure.

Use the same signal starting point for the comparisons. If your testing date range is 01/01/1995 - 12/31/2004 and idea #1 kicks off it's first signal on 05/01/1995 and idea #2 kicks off it's first signal on 02/01/1996...no fair! Idea #1 received a 9 months head start on idea #2. Might not seem like much...but with markets and babies...9 months can make a difference.

In fact, it usually pays to bucket the trades in similar timeframes. That way you can ensure not only the trades share similar signal start times but also similar bucket sizes. For example, to test 10 years of data...01/01/1995 - 12/31/2004:

Bucket 1 -> 01/01/1995 - 12/31/1996
Bucket 2 -> 01/01/1997 - 12/31/1998
Bucket 3 -> 01/01/1999 - 12/31/2000
Bucket 4 -> 01/01/2001 - 12/31/2002
Bucket 5 -> 01/01/2003 - 12/31/2004

The buckets 1 - 5 above would be the testing date ranges. After the simulations are complete for each idea and corresponding bucket...verify idea #1's bucket 1 contains roughly the same number of trades as idea #2's bucket 1. If not, then boot-strap the larger bucket size down to the smaller bucket's size. Compare results. And remember, remove the top 5% or so trades from each idea's buckets.

[Clarification: The testing date ranges in the buckets listed above are not the start and end dates for your trades. The date ranges are the start and end dates for your idea. Meaning...all ideas triggered from 01/01/1999 to 12/31/2000 would be in Bucket 3. You would need to track those signals for as long as you stay in them...which may mean all the way to your end date of 12/31/2004. Make sense?

In essence, your overall begin and end dates are 01/01/1995 - 12/31/2004. Then you slice and dice the signals that occurred during that time frame based on entry date into buckets 1 - 5.]

Questions?

What if the idea consists of several smaller ideas such as selecting stocks at their 20-day high and on that new high experience a 30-day volume surge? In this case, you would need to iterate through the above process twice.

In the first pass, you'd compare the 20-day high stocks against stocks not at their 20-day high. In the second pass (if the first pass proves fruitful), compare the 20-day high stocks with 30-day volume surges against 20-day high stocks without 30-day volume surges.

As you can see, getting an idea through the validation process is quite an ordeal. Unfortunately for us system traders; very few ideas make it out alive. Thankfully, we don't have to cut those ideas from our sample set.

Later Trades,

MT

Wednesday, October 10, 2007

Quote of the Week - 10/10/2007

"...the most important quality for a trader to develop is discipline. As you've read, my stubborn ego and impatience prevented me from achieving lasting success and financial security. I hope my story has shown you that any fool can get lucky and quickly make a great deal of money. But, if playing the stock market was always that easy, there would be no need for research and hard work. Considering that all the information you need to be able to profit is available on the Internet, what sets successful traders apart is their ability to wade through all the muck. With regard to your sources, keep an open mind. As my losses demonstrate, if you allow your emotions and ego to control your trading, you are doomed to fail." -- Timothy Sykes from his book, An American Hedge Fund

What great words of advice! Reading Tim's book brought back the memories of what it was like trading in the greatest stock bubble of our time. In fact, I participated in the ISCO trade he mentions in the book. What a ride, indeed!

I look back and realize just how far I've come as a trader and how much further I still have to go. Tim is right...cutting through the muck is a tough chore for any trader/investor. The key to long-lasting success is keeping an open mind, learning from your failures, and working hard...really really hard. Of course, a sprinkle or two of luck never hurts.

Thanks for the book, Tim.

Later Trades,

MT

Friday, September 21, 2007

Recent Links for 09/21/2007

Newbie - converting csv files to arrays in NumPy

Great message thread on how to convert csv files to numpy arrays.

Cookbook/InputOutput - Numpy and Scipy

File processing examples using numpy, scipy, and matplotlib. How to read/write a numpy array from/to ascii/binary files.

Numpy Example List

Examples of Numpy functions such as fromfile(), hsplit(), recarray(), shuffle(), sort(), split(), sqrt(), std(), tofile(), unique(), var(), vsplit(), where(), zeros(), empty(), and many more.

Introducing Plists: An Erlang module for doing list operations in parallel

Could you spawn a trading system process for each stock of a given day's trading (a list)? What if you had 20,000 stocks for a given day? Can plists/erlang handle 20,000 processes without hitting memory constraints?

Wednesday, September 19, 2007

Recent Links for 09/19/2007

130/30 Strategy Backtested

Disagree with comparison of 130/30 to non-leveraged benchmark/strategy. Also, the additional longs didn't improve the long portion of the 130/30 returns compared to the long-only strategy returns. Am I missing something?

Tuesday, September 18, 2007

Recent Links for 09/18/2007

Chapter 22. Struct and Array Modules
Overview of the python struct and array modules

Building Skills in Programming
Nice python tutorial.

Python Grimoire
Nice python cookbook.

Monday, September 17, 2007

Sunday, September 16, 2007

Saturday, September 15, 2007

Thursday, September 06, 2007

Wednesday, September 05, 2007

Tuesday, September 04, 2007

Recent Links 09/04/2007

World Beta - Engineering Targeted Returns and Risk: More On The Endowment Style Of Investing Annotated

- post by taylortree

Frontier Capital Management

SourceForge.net: tkdiff

Great little file compare utility. Graphic front end to the diff program.
note: tested this today against a large file/program (well, not that large in my line of work...but I guess to Google's)...couldn't handle it. But, works great on small files.
- post by taylortree

Google Mondrian: web-based code review and storage

Online code review that works like a blog/wiki. I wonder...is it possible to create a code review system similar to Mondrian within a source management toolset such as subversion? Seems like most of the backend is there already...would only need to add some front end tools to display the changes being committed and allow comments on those changes.
- post by taylortree

Monday, September 03, 2007

Recent Links 09/03/2007

ONLamp.com -- Numerical Python Basics

Numpy basics.
- post by taylortree

Programming in R

Finding Duplicate Elements in an Array :: Phil! Gregory Annotated

Interesting way to find duplicates in an array. Enjoyed the links on the pigeonhold principle and Floyd's cycle-finding algorithm.
- post by taylortree

Now, suppose that the array is of length n and only contains positive
integers less than n. We can be sure (by the pigeonhole principle)
that there is at least one duplicate.

So, how do we find the beginning of the cycle? The easiest approach is to
use Floyd's cycle-finding algorithm. It works roughly like this:
Start at the beginning of the sequence. Keep track of two values (call
them a_i and a_j). At
each step of the algorithm, move a_i one step
along the sequence, but move a_j two steps. Stop
when a_i = a_j.

Wednesday, August 15, 2007

Investor or Gambler?

Tom from InvestorGuide.com sent me an article of his to read regarding the differences between investing and gambling. Tom did a great job in discussing the two terms fairly. Very hard to write an article like that without exposing unknown biases.

I did find a couple of very minor areas where I disagreed with Tom's article. I shared those comments to Tom in an email. But, felt those comments would be helpful to readers of this site. First read Tom's article. Then my comments below...

Tom's article:
There's a big difference between buying a stock after thoroughly researching it and buying a stock by hitting it on a dartboard.

My comments:
Is there really a big difference...in outcome? Sure, the person may feel different about the investment...but based on outcome alone...historical evidence would suggest the odds of success are approximately the same.

Tom's Article:
Gambling - "Any activity in which money is put at risk for the purpose of making a profit, and which is characterized by some or most of the following...no net economic effect results."

My comments:
I would argue that each player in the stock market provides a positive economic effect. The investor provides long-term capital to companies in need of capital. The speculator and gambler provide liquidity. Sure there are negative effects from all players...investors prop up some companies that probably shouldn't receive further funding...and will eventually go bust. And speculators/gamblers can turn liquidity into a frothing market that can cause long-term problems after the swell has subsided.

Of course, your point is true that gamblers' short-term trades may be a net effect with each other...but that activity regardless of reason or length of hold...still provides liquidity for other players in the market.

Basically, remove any player from the game...and the market wouldn't be what it is.

That's it from here where I've got a softball game to prepare for this week. I haven't played softball since my college days. And haven't thrown many balls since my shoulder surgery. Should be an interesting show to say the least.

Later Trades,

MT

Saturday, June 09, 2007

Weekend Readings - Search for Alpha

Sharing some great links found this morning while enjoying a Starbucks coffee and Krispy Kreme doughnut. :-)

A great Quantitative Primer from the Bionic Turtle.

Nice little profile on Parametric Portfolio Associates by Bloomberg Markets Magazine. I enjoyed reading their rebalancing alpha along with their contrarian weighting of individual stocks for each country. In fact, check out Parametric's Research & Whitepapers area...great information to be had. Their paper on Using Statistical Process Control to Monitor Active Managers is one to read a few times this weekend.

One view shared by Parametric coincides with my recent testing of alpha filtering in the U.S. Equities market. Parametric found that smaller countries exhibit less correlation to the broader markets. My guess is due to less investor attention, liquidity issues, and such. In essence the markets are less efficient which enable Parametric to extract alpha (non-commoditized beta) by overweighting these smaller countries in their portfolio compared to the index they track. I found that by increasing the frequency in my alpha calculations...the fewer stocks I found that could beat the market (high alpha). Increasing the frequency to daily in my alpha filter picked up only the most illiquid stocks in the market out of the 20,000 stocks in my database (including the delisted). What does this mean? These stocks are the less followed? Less efficient? Their returns are least effected by the broad market? Short-term...all other stocks are governed by the overall market returns? Only if you extend your time horizon do stocks capture more alpha from the market? My recent testing shows this to be a possibility.

So, what should an investor/trader do? Well, if you're a daytrader, you should pay much more attention to the market in general. The market acts as a powerful force in a stock's short-term returns. If you're an investor...extending your time horizon months if not years is a way to capture more (alpha) than the market gives (beta) to each individual stock. Think about it...anywhere the market is efficient...the market governs the behavior. Only when you drive outside of that efficient zone do you capture non-market behavior. Holding stocks for the very long-term is one road outside of the city.

World Beta finds an interview with Harvard's Endowment manager,El-Erian. El-Erian claims his biggest challenge is overcoming the popularity of the endowment model. Which I tend to agree.

Finally, for you R fans out there...check out the Econometric tools for performance and risk analysis created by Brian G. Peterson and Peter Carl. Need to calculate the Information Ratio, Sharpe Ratio, Sortino Ratio, max drawdowns, rolling returns, CAPM, Kelly Ratio, and Omega on your portfolio's returns? Then this PerformanceAnalytics package could be just the ticket. I have yet to work with this feature-rich package...but have it on my list of to-do's over the coming weeks.

That's it from TaylorTree...where the weather is beautiful, coffee is great, and I've got an afternoon of fishing in my future.

Later Trades,

MT