Index investing paper shows gap between theory and reality

In his famous roast of then-US president George W Bush at the White House Correspondents' Dinner in 2008, comedian Stephen Colbert mocked Bush's stubbornness. "The greatest thing about this man is he's steady," Colbert said. "You know where he stands. He believes the same thing Wednesday that he believed on Monday, no matter what happened Tuesday."

One could make a similar joke about Gary Gorton and K Geert Rouwenhorst, a pair of finance professors at the Yale School of Management in Connecticut. They are best known for a 2004 paper, Facts and fantasies about commodity futures, that helped unleash the commodity index investing boom of the mid-2000s. Reviewing 45 years of data, Gorton and Rouwenhorst found the performance of a basket of commodity futures had a low to negative correlation with that of stocks. That meant investors could achieve more stable returns by allocating part of their portfolio to commodities.

Many investors bought the argument, and hundreds of billions of dollars poured into commodity index products developed by firms such as New York-based AIG Financial Products – which, incidentally, supported Gorton and Rouwenhorst's research. All was well until 2008. That year, when stock markets crashed, commodity markets crashed too. For the next five years, commodities and equities were highly correlated. Investors who hoped commodities would be a safe haven from stock-market turmoil were disappointed.

So it raised a few eyebrows when Gorton and Rouwenhorst recently mounted a vigorous defence of their 2004 conclusions. In May, they released a follow-up paper that examined data from the past 10 years and found their original arguments "largely hold up". They conceded that correlations had spiked after the crisis, but they characterised that as a temporary blip, not a fundamental shift in market structure that might undermine their thesis.

Critics have savaged the new paper, saying it relies on a theoretical index that bears little resemblance to real-world index products. In an interview, Gorton and Rouwenhorst retorted, essentially, that their maths is correct and that their theories are meant to be considered over the long run. But that offers little consolation for investors of the past decade, whose "foray into commodity index investments has proved to be a huge and sometimes costly disappointment", as Barclays analysts wrote in a June 15 research note.

Gorton and Rouwenhorst's reputations may have dimmed in the past several years, as events have cast doubt on their theories. But at least there's no doubt what they believe.