How Much History Is Enough?

How much historical data is enough to decide whether a company's performance has actually been persistent? We originally tested this using three and five years of history and came away thinking five years was clearly better. Since then, while doing additional validation of our historical research, we found a problem with the benchmark data used in that experiment. We corrected it, strengthened the validation process and reran the research. The result changed enough that we withdrew the original article rather than simply changing a few numbers.

For the new test, we ran the same 10-company Persistfolio portfolio using one, three and five years of historical data. The portfolio construction and other research rules stayed the same. This gave us a reasonably clean way to see what happens when we change the amount of history available to the model.

The one-year result is probably the easiest to interpret. It was much more active than the longer-history versions, changing an average of 4.3 of its 10 holdings each quarter, yet all that activity didn't produce a better return. It gained 212.0% compared with 215.2% for the S&P 500. One year may simply be too short to tell the difference between performance that is persistent and performance that happens to have been strong recently.

Three years was much more interesting. The portfolio gained 355.1%, or 16.36% annualized, making it the strongest result in the test. This is also where correcting our earlier research mattered most. Our previous experiment had led us to conclude that five years was substantially better than three. With the corrected historical data, the result went the other way. Three years produced the higher return, although it also came with somewhat higher volatility, a larger maximum drawdown and more portfolio changes.

The five-year portfolio gained 291.4%, or 14.62% annualized. That was below the three-year result, but it was somewhat steadier and required fewer changes. So there is a real trade-off here. Three years responded more quickly and produced the strongest historical return, while five years gave up some of that return in exchange for a somewhat more stable portfolio. That is a much more useful result than simply saying that one period "won."

What about 10 years?

Persistfolio also uses a 10-year research horizon, but our historical database doesn't let us test it over the same full period. There isn't enough earlier data to give companies ten years of history at the beginning of the experiment. The usable 10-year test therefore doesn't begin until September 2019, so putting its cumulative return beside the figures above would be misleading.

We can, however, compare the five- and ten-year approaches over the period where both have enough data:

Over those five years, the return difference was very small: 122.1% for the five-year approach versus 119.8% for ten years. The ten-year version had somewhat lower volatility and a smaller maximum drawdown, but five years of comparable data isn't enough to make much more of it than that. In particular, it would be wrong to compare the 119.8% figure with the full-period 355.1% or 291.4% results above because they cover different periods.

What did we learn?

The main result isn't that three years is the new "best" setting. We tested several alternatives, and choosing three years simply because it produced the highest historical return would be a good way to start fitting the methodology to its backtest. What the experiment does show is that our original assumption—that more history would necessarily produce a better result—doesn't hold up very well.

One year looks too reactive. Three years produced the strongest historical result but also moved around more. Five years was somewhat steadier and still performed well. Ten years may provide even more stability, but we don't have enough comparable historical data to make a strong claim about it.

There is also a second lesson from this experiment. Our original three-versus-five-year study gave us a fairly convincing answer, and that answer turned out to be wrong after we found a problem in the historical data used for the test. We could have quietly replaced the old numbers, but that wouldn't accurately represent what happened. We removed the earlier research, corrected the problem, added additional validation and reran the experiment from the beginning.

That's probably more important than which row in the table has the largest number. Quantitative research isn't particularly useful if the objective is to defend the result you started with. Sometimes the useful result is finding out that you were wrong.

Persistfolio publishes quantitative investment research for informational purposes only. Results shown above are hypothetical historical results based on price returns and do not include dividends, fees, taxes or transaction costs. Historical and hypothetical performance does not guarantee future results and is not investment advice.

Learn more about Persistfolio: