In
general terms, optimization is the process of finding the best system
parameters or markets in which a mechanical trading system produces the
highest historical return over a specified period of time. The theory is
that history repeats itself and what worked in the past in a specific
market will work in the future in that same market. While such an
assumption may be valid over very short time periods, history has
repeatedly shown that such trading methodologies are not sustainable and
invariably lose money over longer time periods.
Unfortunately, with the
raw computing power now available to the masses, the concept of
optimization has been taken to an illogical extreme by many system
developers. There are now a number of supposed mechanical trading
systems available in which each and every single parameter within the
system has been optimized for each and every commodity market on which
they were tested. Many of these systems, at first glance, report
seemingly incredible historical returns. Upon more diligent inspection,
however, it always turns out that such systems have been little more
than "curve-fitted" or modified to fit the specific data of
the markets on which they were tested.
We fervently believe that
such optimization in system development is fundamentally flawed,
misleading to the investing public, and constitutes little more than
self-delusion on the part of most traders. In additional to system
parameter modification, there are
many other non-obvious and subtle ways in which a system developer can
unwittingly or even knowingly optimize a system to the point that its future value is
suspect.
____________________________________________________
Commodity Portfolio Selection Optimization
Data Manipulation Optimization
False and Misleading Reporting of Data
How to Tell if a System has been
Optimized in TradeStation
____________________________________________________
The most common form of optimization is to select the best parameter input values for every different commodities. Historically, however, very few systems that have been optimized to any significant extent have stood the test of time and are still promoted today. Or if they still are being promoted, they undoubtedly have undergone substantial parameter input modifications to keep their reported track records profitable. To give you an idea of how widespread this package is, we performed an internet survey and found that over 95% of the publicly available trading methodologies involve some degree of optimization with parameter optimization accounting for 80% of all the optimized systems being promoted. For the many reasons discussed by others in writing far more eloquent than our own, we believe almost all of these reported track records are illusory and misleading.
As an example, several years ago we obtained a heavily-promoted system for evaluation that was originally offered to the unsuspecting public for many thousands of dollars. The system was an indicator crossover system that intuitively seemed like a good idea. Before evaluating the system, we talked with the developer and received unhedged assurances that the system only had a few minor optimized tweaks. When we received the system and looked at it carefully, however, every single one of the commodities used a radically different set of input parameters to obtain the positive results that they reported. One commodity used 5 and 55 days for the inputs while other commodities' inputs ranged from as little as 3 days to as much as 60 days. When all commodities were subsequently tested using the same inputs for all commodities, not one single set of parameters produced a positive return across the portfolio of 21 commodities. We subsequently tested the system again after a 1 year period and found that even using the original optimized parameters proposed by the developer the system failed to produce a positive portfolio return…oh well, one more system for the bookshelf!
It really goes to show that almost any mechanical trading system can be manipulated a show a spectacular track record by curve -fitting the system to the data. With today's computing power it is not a problem to run optimization schemes using literally thousands of different parameter combinations to get the best mix. Unfortunately, almost all such efforts are doomed to failure when the results are transferred to real time trading.
Our alternative approach is to demand that all system parameters be exactly the same for all commodities - no exceptions. We further constrain ourselves and minimize the number of different parameter inputs that are tested by demanding that every parameter be a Fibonacci number or a 10x multiple of a Fibonacci number. We are not Fibonacci adherents and none of our systems use Fibonacci price levels in any way in their analysis. It just seemed like a fair way to limit the number of different parameters that could be tested to ensure we were not inadvertently over-optimizing the systems even though we already required all parameters to be identical with different commodities. For instance, when looking at a parameter such as ADX, we only allowed ourselves the luxury of testing a 5 bar, 8 bar, 13 bar, 21 bar, 34, or 55 bar ADX (based on the Fibonacci numbers 5, 8, 13, 21, 34, and 55). Thus, we limited our parameter selection in this case to 6 possibilities out a possible 50 (55-5). We have to admit, though, we really didn't give up much by enforcing this limitation because it often turned out that a Fibonacci number would have been close to one of the best choices had we gone through a full parameter optimization procedure.
Our Guarantee – Every single system offered by longtermtrading.com comes with a published track record using the exact same parameters for every single commodity. Further, every parameter is a Fibonacci number (or a 10x multiple of a Fibonacci number). This includes all of the entry and exit parameters including the money management stops – no exceptions!!
No other vendor can honestly make this statement. Of course, you are free to optimize these systems to your heart’s content and this may improve system performance even more. For instance it appears that using money management stops of about $1,500 - $1,800 will improve the overall performance of most commodities in most systems compared to the nonoptimized $2,100 money management stop we use in all of our systems for every commodity. (Note: $2,100 is a 102 multiple of the Fibonacci number 21). Be forewarned, however. The use of such optimization techniques may improve the results in real time only if you are starting with a very robust, non-optimized system such as those offered by longtermtrading.com.
< back to top>________________________________________
Commodity Portfolio Selection Optimization
Another very common method of improving trading system results is to cherry pick the commodities on which the system is tested. For instance, in testing one system one might use corn and soybean oil as the commodities selected for the grain complex while soybeans and wheat are tested in another system. Of course, the reason for doing so is that the vendor can produce a track record with the highest reported net profit with the lowest drawdown - thus increasing the lure to potential purchasers. However, for the same reasons as parameter optimization, commodity portfolio selection optimization is highly misleading to investors.
Another popular method of improving published track records is to include results of commodities that have such low liquidity and daily trading volume that no reasonable trader would commit resources to that market. For instance, Rough Rice, T-Bills, and the Goldman Sachs or CRB Indexes are popular commodities to include in commodity trading track records. This is because they have had many broad and sustained price trends over the years that are easy for many trend-following trading systems to catch. It is not uncommon to show profits of $25,000- $75,000 over a 15 – 20 year time frame on a single contract basis with these commodities. The problem is that these commodities are often extremely illiquid markets with contract volume often down to a few hundred contracts per day. If every purchaser of every commodity trading system that included these commodities in its track record traded just one contract, I’m sure that the average volume would be closer to 20,000 contracts per day instead of only 100-200 per day that are actually traded. The real world reason almost all speculators shun these markets is because there is simply not enough commercial interest to provide any liquidity in the market. As a result, when any big speculative buying or selling comes into the pits, the locals and few commercials let the bid-ask spreads explode then eat you alive.
longtermtrading.com avoids this data optimization trap by testing all of its systems on the same rigidly selected, highly liquid 21-commodity portfolio. (See <How We Design Systems>). No other multi-system vendor can honestly make this statement.
Our Guarantee – Every single system offered by longtermtrading.com comes with a published track record generated by using the exact same 21-commodity portfolio for every system tested – no exceptions!!
Of course, you are free to use these systems on other less liquid commodities and the results are equally spectacular in many cases.
< back to top>________________________________________
Data
Manipulation Optimization
Less common than system parameter optimization or commodity selection optimization, data manipulation does occur in many different, often subtle, ways. For instance, a system promoter may find that its methodologies might only be profitable during periods of low inflation and high stock prices. Such a developer may choose to limit testing of such a system to only over the past several years or so. Surprisingly, using different rollover methods for different commodities or testing on backadjusted continuous contracts constructed in a different manner can often produce quite different results compared to testing across a portfolio of continuous contracts generated in an identical manner. longtermtrading.com does not make this mistake and tests every commodity back to 1970 or the original date trading began - whichever is later. Every continuous contract for each commodity is also generated in exactly the same manner.
Our Guarantee – Every single system offered by longtermtrading.com comes with a published track record generated by using back adjusted, continuous contracts that were created in an identical manner using CSI (Commodity Systems Inc.) raw data and software for back-adjustment. Contract months were rolled and back adjusted on the day that the new contract volume or open interest exceeded that of the old contract. The testing period includes the entire length of the commodity contract life (as contained in CSI's database) or going back to 1/1/1970 – whichever was later.
< back to top>________________________________________
False
and Misleading Reporting of Data
Although most vendors are fundamentally honest, it is not uncommon for some developers to intentionally mislead potential investors by reporting profitability and drawdown in erroneous ways. Some other developers are out-and-out liars. Their profitability and drawdown numbers are completely fictitious. Others are subtle masters of deceit and allude to riches by extrapolating profits from one very short test period with only a small group of commodities to an extended period of time in a large portfolio. Unfortunately, if a person sets out to intentionally deceive another, only the most rigorous due diligence on the part of the purchaser will detect this type of fraud.
More common is the occurrence where a developer will only report certain statistics that obscure a system's true performance. Misreporting the true expected drawdown of a system is most often seen. For instance, one software program that calculated portfolio profitability and drawdowns based on input from TradeStation individual commodity profitability reports is Portfolio Maximizer and related products. These programs are excellent in almost all respects and provide an enormous amount of trade statistical data for the advanced user. Its analysis engine now serves as the platform for the profitability reports within TradeStation 2000i and higher. However, Portfolio Maximizer and earlier related products reported both individual commodity and portfolio drawdowns in two very different ways - one of which I consider very misleading to novice traders.
Portfolio Maximizer reported drawdowns as Maximum Drawdown and Maximum Equity Drawdown. Maximum Drawdown is defined in the program as the largest intraday drawdown experienced by the system on a single closed out trade. This version of drawdown measures the highest open equity to the lowest unrealized equity low of the trade, based on a long position and reversed for a short position. In this context, the maximum drawdown refers to the largest unrealized loss experienced by any of the system’s trades. Although of potential usefulness in some analysis, this parameter is useless in determining the worst string of losses a trader may incur. After all, if the largest single trade loss is $1,500 but you had 10 of them in a series with only a few occasional profits interspersed, your drawdown would be closer to $15,000 instead of the $1,500 as reported by Portfolio Maximizer.
Although the developers of Portfolio Maximizer make it clear in their documentation that this is what they mean by Maximum Drawdown, a few vendors continue to report this lower number and implying that it represents a true individual commodity drawdown experienced by a trader. In fact, reporting portfolio drawdowns in this manner can underestimate true portfolio drawdowns by 75% or more! This is grossly misleading to the investing public.
A more true measure of drawdowns, Maximum Equity Drawdown, is defined in the Portfolio Maximizer program as the greatest equity drawdown experienced by the system from the highest high during a single trade to the lowest low during the same or consecutive trades. On an individual commodity basis, this drawdown measure closely mirrors TradeStation's drawdown calculation, but it additionally includes the initial run-up of the existing trade just prior to when the system’s drawdown begins. Therefore, Portfolio Maximizer’s calculation of Maximum Equity Drawdown on a single commodity is generally larger than those derived from TradeStation.
This information would tell you the maximum drawdown a trader would incur if, when starting to trade the system, the trader blindly went in and established the trade on the day that the trading started rather than waiting for a new entry signal. The problem is that nobody trades this way in the real world because of the huge risk to reward ratio inherent when entering positions already in place. The individual commodity drawdown calculation of greatest usefulness to a trader is that which would occur from the point of initiation of a new trade to the lowest intraday equity prior to making a new equity high. This is what a trader would experience in the real world, what TradeStation reports for individual commodities, and what longtermtrading.com reports for the individual commodities in its portfolio.
Most other portfolio analysis software reports drawdown as a simple bar-ending mark-to-market drawdown. Our portfolio analysis was performed using Portfolio MCS by Inside Edge Systems. It calculates drawdown in the bar-ending mark-to-market method described above and this is reported in our analysis summary for each portfolio. This assumes, however, that a trader arbitrarily starts trading a complete portfolio and enters positions in every commodity in the portfolio on the equity peak and then looks at the maximum drawdown until a new equity high is subsequently made.
This has the effect of overestimating the actual drawdown that would be seen by a more rational trader who, when starting trading, only establishes new positions as indicated by new entry orders. Maximum drawdown in that instance can then calculated from that first original entry point until new equity highs are made. This is how almost all traders would logically start trading a new system - that is, at the start of a new trade and not at some point in the middle of a trade. For our clients further understanding, we also report this figure for each of our systems weekly portfolios as "maximum closed trade-to-closed trade drawdown" to distinguish it from the conventional "maximum mark-to-market drawdown". Closed trade-to-closed trade drawdowns are calculated through the use of our money management spreadsheet templates included with each of our systems.
Our Guarantee– Every published track record disclosed by longtermtrading.com reports individual commodity drawdowns directly from TradeStation and true portfolio drawdowns using both bar-ending mark-to-market drawdowns and (for weekly data) as closed trade-to-closed trade drawdown.
< back to top>________________________________________
How
to Tell if a System has been Optimized in TradeStation
Fortunately, there are a number of simple methods and clues to see if a system is optimized in TradeStation and to what degree - even if you are dealing with black box software.
-
Look at the TradeStation P/L printouts for each commodity - In TradeStation P/L printouts for each commodity, the name of the system and the commodity data set used to generate the report is printed along the top of the printout of the printout. If a differently named system is used for each different commodity, the system is almost certainly optimized.
-
Look at the TradeStation input parameters required for each commodity. If you are asked to insert different parameters for different commodities or if you are asked to input a specific commodity symbol as a parameter, this is a very strong indicator that the system has been optimized. In the later case, the different parameters required for the different commodities are simply embedded within the TradeStation code and are activated by the trader's selection or input of the commodity symbol.
-
Ask very specific questions of the developer – Ask the system vendor point-blank, "Are any system parameters different in any way between any different commodities or are they all traded in exactly the same manner to generate the track record”. If the vendor unequivocally states there is no optimization and is willing to put it in writing, you may have a winner. If the vendor admits to even a small level of optimization, require the vendor to supply a track record in which all commodities were tested using the exact same parameters - no exceptions. Only by comparison of this non-optimized track record with an "optimized" version can a user determine to what extent optimization has improved the overall system performance. If you are not willing to purchase the system based on its non-optimized track record, do not be swayed by the optimized version' track record showing unbelievable returns…the results appear too good to be true because they are!
< back to top>_____________________________________________