I do a lot of backtesting in my articles. Digging into historical data can lead to interesting insights. But have you ever wondered how reliable historical data is? You might be surprised to learn that the answer is “not super great”.
Even in the present day, there are sometimes questions about data. In 2009 Jeremy Siegel (of Stocks For The Long Run Fame) said that the S&P was calculating its earnings incorrectly.
Many others disagreed with Siegel.
In this case, I think Siegel got it wrong. But the point is that reasonable people can disagree about modern data; there are almost always some judgment calls involved. And that’s when everything is easily available on the nearest computer.
The Cowles Commission (1871–1940)
Whenever you see someone (like me) say, “equity returns since 1871 have been X%”, did you ever wonder…why 1871? And where do the numbers come from?
In the 1938 the first edition of “Common-Stock Indexes” was published by the Cowles Commission for Research in Economics.
The purpose of the Cowles Commission common-stock indexes is to portray the average experience of those investing in this class of security in the United States from 1871 to 1937.
Basically, the Cowles Commission had a bunch of researchers going through 60–70 year old copies of newspapers looking for stock prices and dividend announcements.
Why 1871?
In the late 1800s there were really only three kinds of public companies: railroads, industrials, and utilities. It was also long before day traders and CNBC, so newspapers didn’t quote stock prices every single day for all the public companies.
In the 4 years prior to 1865 not more than 9 industrials and 1 utility are quoted in any given year.
Imagine, in the entire country only 10 companies (other than railroads) had their prices quoted at all. And even those didn’t have their prices quoted every month.
This led the Commission to declare that
information as to capital structure, earnings, and dividends of industrial and utility stocks is practically non-existent before 1871.
There was insufficient data on industrials and utility stocks…but plenty of data on railroad stocks. But the Commission decided that building an index that was really just a railroad stock index would be misleading. It didn’t help that in 1861 only 0.007% of the national wealth was in publicly traded companies. (The rest was in private companies.)
Prior to 1871 the All Stock index would have been almost completed dominated by one group, the railroads, and would have constituted such a small fraction of the national wealth that its economic significance would have not have been at all comparable with that of later years.
Which stocks?
With 1871 set as a cut off date, the next thing is to figure out what data you want to include. Cowles relies almost entirely on the New York Stock Exchange.
In the years 1929–33 the New York Stock Exchange accounted for about 67 per cent of the total volume of stock transactions on the 24 most important exchanges in this country.
(That’s right 24 exchanges! You didn’t know about the Hartford, Honolulu, Colorado Springs, Baltimore, New Orleans, Louisville, Pittsburgh, Milwaukee, and Cleveland stock exchanges?)
So they limited themselves to around 2/3rds of the available stocks in the country. About 50% of the ones they excluded came from the New York Curb Exchange. The reason they exclude it? Too much fraud makes the data useless.
Frauds practiced on the New York Curb were so severely criticized in the report of the Hughes Commission in 1909 that organization was begun for the purpose of reform. It was not, however, until 1921, when the activities of the Curb were moved indoors, that it was possible to exercise the necessary control.
Handling Edge Cases
Now you start to get into the details. What if Company A is just a shell holding company and owns 99% of Company B. By including both in the index, you are basically double counting Company B.
Cowles did a test, found that deduplicating only changed the index by 0.002%, and decided it wasn’t worth the effort.
What if Company A has price quotes available in January 1890 but then disappears from the historical record until December 1890. There were 518 cases where a stock temporarily disappeared like that.
The stock market was shut down for parts of World War 1. What exactly do you do about that?!?
Those issues, and others, had to be ironed out by the Cowles Commission.
Estimating Dividends & Earnings
Cowles also needed to estimate dividends and earnings for these companies. Like everything else, that wasn’t always a straightforward task.
It wasn’t always easy to tell when a company declared a dividend. Or how long after the declaration is actually paid it.
Analysis of 62 dividend payments from 12 companies in 1872–1873 showed the average lag to be 14.7 days, and a similar analysis of 53 dividend payments from 27 companies in 1895 showed the average lag to be 13.1 days.
What if the company didn’t pay the dividend in cash?
The question is relatively unimportant, arising in the case of less than 0.5 of 1 per cent of all dividends paid.
Calculating an index for January 1871
When we say the index has a value of X for January 1871, what does that mean? Is that for January 1st? January 31st? Or some other date?
Cowles actually calculates the average price per share for the month. Which is probably not what you expected. Cowles does this by taking the highest price seen during the month and averaging it with the lowest price seen during the month.
It may be objected than an average of high and low prices does not strictly represent the average price for all shares traded in the month, but it will be readily understood that so vast a task as a computation on the latter basis would be impracticable even if the data were available.
Why use the arithmetic mean and not the geometric or the harmonic mean?
The arithmetic average of the monthly high and low prices was employed, in preference to the geometric or harmonic average, in part, because of the greater ease of computation
Using averages like this isn’t great. We will come back to this in a later installment of “Making an Index”.
Calculating the index
Once you have all this data, how do you make an index out of it? That’s not as straightforward as you might think. Do you just add up the individual prices? Do you take the arithmetic average of them all? Do you weight them based on number of shares outstanding? Do you weight them based on number of shares traded? Do you take a geometric mean of something?
Cowles shows that, using the same raw data, different methods of index calculation can lead to very different results. Did the index go up by 183%? Or by 203%? And that’s just over a 19-month period.
Only a short time before Irving Fisher (often called the greatest economist America has ever produced) had published the seminal book The Making of Index Numbers. Based on Fisher’s arguments — and because calculating the geometric mean was harder — Cowles settled on the arithmetic mean weighted by the number of shares outstanding. This is the standard “capitalisation-weighting” that we’re used to today.
Future revisions by Cowles
The first edition came out in 1938. A revision with corrections quickly followed in 1939. Cowles published two further editions, which extended the data to 1939 and 1940. The full Cowles data set covers 1871–1940.
Along the way, you can see the large amounts of missing data & judgment calls that the Cowles Commission was forced to make. Some of them, especially averaging monthly data, can have relatively large impacts.
The Standard Statistics Bureau weekly price index (1918–1956)
The Standard Statistics Bureau created their first index in 1923. However, like most index providers, they also created a few years of historical data as well. So even though they started in 1923, their data goes back to 1918.
They started out by tracking 233 companies on a weekly basis.
For the period 1918–1940 the Cowles Commission relied on the data from the Standard Statistics Bureau. This posed a few problems:
- Cowles was creating a monthly average, whereas the Standard Statistics Bureau was reporting weekly point-in-time numbers.
- The Standard Statistics Bureau only reported on price not total return. That is, they didn’t track dividend payments.
- How do you splice the two different data sets? That is, how do you go from Cowles’ own price index in December 1917 to the Standard Statistics price index in January 1918?
Cowles had three solutions:
- Continue averaging, just using the weekly data from Standard Statistics.
- Continue researching dividend data and add that back in.
- Use a lot of math. There are 8 pages explaining all of it. Yikes!
So at this point we have a series of data from 1871–1940 using:
- Cowles’ dividend data
- Cowles’ monthly averaging
- Cowles’ price data from 1871–1917
- Standard’s price data from 1918–1940
- Calculated using Irving Fisher’s “ideal index formula”
Hopefully you were able to spot all of the ways that the final result is less than perfect. Missing data. Judgment calls. Averaging. Switching between data sets (from Cowles’ own to the Standard Statistics’ one). The underlying math. The timing of dividend payments. Whether to de-duplicate cross-holdings. Handling disappearing and reappearing companies. The small number of companies involved (only 45 in 1875). The dominance of a few industries (especially railroads). Concerns about fraud. And so on.
Does this mean we should throw it all away? No. But it is worth keeping in mind that our backtests always come with limits on what they can tell us. We’ll explore more of these limits in future instalments of the series…
The second instalment can now be found at…
