I’m happy to tell you that the trading tools upgrade announced in late-July is done — on my part. From conversations with Sean and Chris, we should be in position to release the upgrade in early-September.

I think it’s good to sit down and review one’s perspective on trading from time to time. Revisiting the code from 2007 gave me a chance to refine it with two more years of observation, insight and learning. Remarkably, the equations become more simplified each time as if I am converging on a set of fundamental truths of directional trading (which I think there is).
Even though I am not that old, I have traded for more than half my life professionally and this time, the process had a certain “arc of my career” feeling to it. But I’m not the only one. My favorite is William Eckhardt. If there is only one book you are ever allowed to read, I think it would be his interview in Jack Schwager’s The New Market Wizards: Conversations with America’s Top Traders.
Mr. Eckhardt is a standout in an industry filled with blowhards and snake oil salesmen. Not many traders (no, market makers and flash traders don’t count) extract so much from the market that they end up donating $20 million to their alma mater, so we had better listen up:

Anyone with average intelligence can learn to trade. This is not rocket science. However, it’s much easier to learn what you should do in trading than to do it. Good systems tend to violate normal human tendencies. Of the people who can learn the basics, only a small percentage will be successful traders.
If a betting game among a certain number of participants is played long enough, eventually one player will have all the money. If there is any skill involved, it will accelerate the process of concentrating all the stakes in a few hands. Something like this happens in the market. There is a persistent overall tendency for equity to flow from the many to the few. In the long run, the majority loses. The implication for the trader is that to win you have to act like the minority. If you bring normal human habits and tendencies to trading, you’ll gravitate toward the majority and inevitably lose.

In 1997, Eckhardt and his former trading partner, the legendary Richard Dennis, gave an interview to Barclay Hedge. The two provided insight into three very important topics.
On backtesting:

Q: Many CTAs are continually testing and searching for new ways to improve their trading or for new trading approaches. Much of this research is based on back testing. Is back testing overrated? How much data are necessary in order to have confidence in the results?
Dennis: Back testing is essential. The key question is what time periods should be tested. You can take the point of view that you should use all available data. How would you know that 19th century wheat prices are less relevant than today’s wheat prices?
I believe that was the right answer in 1983. Today, I find it hard to be agnostic on the questions of whether markets have changed and how they may have changed. The trends of the 1970s occurred in the absence of computer-generated, trend-following algorithms. The markets of the last ten years are distorted by the onslaught of the technical trader.
As a result, I back test only the last ten years. In the unlikely event markets are as good and undistorted as they were in the good old days, I’ll be happy to make less than I might if I had used that early data in an optimization. The trade-off is that if markets continue their perversity, I’m way more likely to have captured a sound way to handle these more difficult markets because I’m fitted only to them.
Eckhardt: I know of no way to validate conjectures concerning technical trading without back testing; however, this procedure is fraught with peril–we all know horror stories. Having adequate amounts of data for reliable inferences is only one of many problems facing the technical analyst, but it is as crucial as any. Statisticians tend to consider that more than about 30 instances constitutes a large sample statistic. For futures price research this is a recipe for disaster. The underlying probability distributions in this subject are so exotic and pathologic that those subtle techniques that statisticians use to squeeze significance out of sparse data are all decidedly out of place.
To make even moderately reliable judgments about a kind of trade, you need something like 300 instances. This is a minimum figure. I don’t feel comfortable acting on research results unless I have several thousand instances.

On strategy optimization:

Q: When you back test particular trading strategies, do you attempt to optimize? If so, how many parameters can you comfortably optimize before falling prey to curve-fitting?
Dennis: There is no escaping a priori decision making in research. There may be no absolutes, but some ideas come close. For example, it would be very hard to justify favoring long positions over short. And no amount of data will validate trading certain markets larger than others (liquidity considerations aside). Research that starts with concepts is much more likely to avoid curve-fitting than blind number-crunching.
Eckhardt: I prefer the term “over fitting”. This makes clear that you can under fit to data. Those CTAs who boast that they never optimize are doing precisely that–they are grossly under fitting. The topic of fitting raises profound theoretical and practical questions, but it boils down to this: you want to fit to reproducible features and not to accidental ones.
To derive an estimate of how much overall good versus bad fitting your optimization labors have produced, the technique statisticians call cross-validation is quite helpful. Of course, this will not tell you where good or bad features originate or how to alter the mixture favorably. For this, it is crucial to assess the quality of degrees of freedom, not only their sheer number. A degree of freedom that has uniformly graduated significance over a manifold of possibilities is better than one that is quirky or that vacillates in meaning for slightly dissimilar cases. It is also important how selective the influence of a degree of freedom is.
A preset profit objective, for instance, is a much more suspect degree of freedom than, say, a look back. The latter presumably impinges on every trade, whereas the influence of the former tends to be concentrated on a few highly profitable scenarios.
The philosophy of science teaches that all observation is theory-laden; there is simply no way to analyze data in a theoretically neutral manner. In fitting to historical data, theoretically unsound procedures can lead to radically invalid conclusions. This is probably why back testing has developed such a bad reputation.

On the impact of so many people using technical trading rules:

Q: How will the rapidly increasing rate of technological advancement and computerization impact a CTA’s ability to capture speculative profits in the futures market?
Dennis: I am less pessimistic than Bill on this question, because I am very pessimistic about the effect of politics on the economy and the implications for price volatility.
Consider one very recent example: After a meteoric rise in U.S. stock prices, it has been seriously suggested that the funding of Social Security be supplemented by permitting, for the first time, investment in the stock market. I promise you that this will be the “solution” to the problem because it is politically painless. It is also based on the ridiculous notion that stocks “always” outperform bonds if you wait long enough.
There will be millions of homeless and starving elderly if we commit to this madness. The pressure to inflate away debt and politicize monetary policy will be overwhelming. This regrettable chaos means volatility and trends.
Eckhardt: When I first began trading solely on the basis of price and was much more concerned than I should have been about the academic orthodoxy that futures market price change was pure white noise–a random walk–I made the following notebook entry: “How can the aggregate of traders and users arbitrage out a potentially unlimited number of nonlinear relationships?” The implication was that they could not. Twenty-five years later, I am less confident about the continuing correctness of this answer. What I failed to take into consideration was the staggering explosion in information processing. This will only continue. Eventually artificial intelligence devices, superior to any human researcher, will effectively uncover all exploitable nonlinear relationships of price to price. Such relationships will be mined until technical analysis is no longer profitable. There is an irony in that dogmatic “random walk” theorists, dead wrong for a century, will turn out to have been prescient–futures markets will have been driven to randomness. The process has already begun.
I feel these developments are nearly assured (assuming no disruption of civilization). What is less clear is whether this will happen as rapidly as I predict–in 10 to 20 years. In the meantime, profitable trading will only get harder as increasingly more astute traders pursue progressively weaker statistical regularities. This is why it is necessary for a CTA continually to improve just to hold his or her own. The only consolation I can offer is that there are profits to be made participating in this process of randomization.

His comment about random walk is so ironic. Burton Malkiel, the godfather of random walk, is now peddling alpha:

• Burt Malkiel Stir-Fries Stock, Takes Non-Random Walk in China
• Malkiel’s AlphaShares Bets on China’s Demand for KFC

I fully understand the desire for people to cash in on their fame. The market’s memory is very short, so presumably it’s OK to contradict yourself with a defense such as “the market has changed.” Still, it probably pays more to focus on the common denominators of success.
The opposite of random walks are trends. In reading a stack of books written by quants, the age old-argument for and against so-called trend trading with technical trading rules may simply depend on the data in question:

• The Profitability of Technical Trading Rules in US Futures Markets: A Data Snooping Free Test
  Numerous empirical studies have investigated the profitability of technical trading rules in a wide variety of markets, and many of them found positive profits. Despite positive evidence about profitability and improvements in testing procedures, skepticism about technical trading profits remains widespread among academics mainly due to data snooping problems. This study tries to mitigate the problems by confirming the results of a previous study and then replicating the original testing procedure on a new body of data. Results indicate that in 12 U.S. futures markets technical trading profits have gradually declined over time. Substantial technical trading profits during the 1978-1984 period are no longer available in the 1985-2003 period.
• A Reality Check on Technical Trading Rule Profits in US Futures Markets
  This paper investigates the profitability of technical trading rules in US futures markets over the 1985-2004 period. To account for data snooping biases, we evaluate statistical significance of performance across technical trading rules using White’s Bootstrap Reality Check test and Hansen’s Superior Predictive Ability test. These methods directly quantify the effect of data snooping by testing the performance of the best rule in the context of the full universe of technical trading rules. Results show that the best rules generate statistically significant economic profits only for two of 17 futures contracts traded in the US. This evidence indicates that technical trading rules generally have not been profitable in US futures markets after correcting for data snooping biases.
• Actual Farmer Market Timing
  One maxim that has been circulating among farmers is that most farmers sell in the lower third of the market. This maxim is soundly rejected using data from Oklahoma elevators. In fact, roughly half of producers sell in the upper third of the market. Thus, there does not seem to be a great need for producers to hire a market advisor to do their marketing for them. But, some farmers do store longer than is optimal and they could be encouraged to sell sooner after harvest. In the short run, farmers sold after price increases and held after price decreases. Price movements in the days after a large number of sales were no different than price movements after few sales. While farmers are noise traders in the short run, it does appear that they are responding to long-run market signals. Even though there may be room for improvement, it appears that farmers are doing a good job of deciding when to sell their wheat.

These papers sprung up from research performed by B. Wade Brorsen (Lukac, Louis P & Brorsen, B Wade & Irwin, Scott H, 1988. “A Test of Futures Market Disequilibrium Using Twelve Different Technical Trading Systems“).
While I fully expect the debate to continue, the one factor key to trader survival is certainly not in doubt:

• Optimal f and Portfolio Return Optimisation in US Futures Markets
  While considerable evidence has been produced concerning the efficacy of trading rules in futures markets, the results have generally not allowed for the reinvestment of profits as might be observed for real traders. Similarly, the determination of the appropriate capital allocation required per futures contract traded has been largely unstructured so making reported percentage returns questionable. This paper provides evidence of the profitability of a simple and publicly available trading rule in five futures markets but more importantly incorporates the ability to reinvest any profits via the ‘Optimal f’ technique described by Vince (1990). The results indicate that money management in speculative futures trading plays a more important role in trading rule profitability than previously considered by providing dramatic differences in profitability depending on how aggressively the trader capitalises each futures contract.