As investors we seek capital gains that are maximized while achieving the lowest risk of losses, however markets are subject to forces we cannot predict. These dynamic forces can be selective, meaning they affect a small set of our holdings, or like 'mob rule' where all our holdings change value at once. Diversification is meant to mitigate the effects of such broad changes to the majority of our holdings at once. This is done by selecting investments that are known not to move in value together, or only to do so to a marginal degree. Market correlation is related to diversification because it is the measure of how much one type of investment rises when another type of investment also rises, called positive correlation. Or, where one investment may tend to fall while another rises, thus moving in the opposite direction, called negative correlation. These movements in value are always expressed as probabilities as they may be strongly or weakly correlated, but there is rarely a set relationship between them.
Typical Diversification Approaches with Stocks and Bonds
Most investors seek diversification and the most popular method is between stocks and bonds. While stocks are subject to market valuation, and the value of bonds may also vary with the market, bonds are more like loans where they will eventually pay their annual rate of return, called the 'coupon rate.' The stress on a portfolio due to valuation of bonds is therefore lower, because they eventually pay their interest and return the 'face value' of the bond so long as the company does not go bankrupt. But even in this case the bond holders are the first to be repaid compared to stock investors, so bonds remain the safer investment.
Problems with Diversification of Stocks
The biggest problem with diversifying stocks based on industries or market segments or even bonds is that the market correlation is still too high, and the capital gains achieved by bonds that are the most uncorrelated also tend to have the lowest yields. What good is diversification if it costs you so much of your gains? Most investors are torn by the high cost of 'insurance' that bonds provide, particularly in times of low inflation where bond returns may be much lower than capital gains of stocks.
There are always options with alternative investments like commodities such as gold, however here too most alternative investments have mediocre returns, or may be highly volatile. What then is a better alternative?
The Holy Grail of Diversification and Low Market Correlation
The ideal diversification is to achieve close to zero in the correlation, or depending on strategy to use negative correlation. This is hard to achieve, but with skill is available to those who are able to fully appreciate market forces. For example, there are industry segments where an ETF reflects a broad sample of an industry. At the same time particular stocks or preferred shares are highly correlated with such an ETF, but move apart under specific predictable conditions, and in a known direction. This type of investment can then be 'paired' with each other. An investor or fund may short the ETF while buying or going long the preferred shares of a stock in the same market segment. This produces what is called a 'differential.' Such a differential may be quite small, however in specific predictable conditions it may occur over a very short time-frame. The result is a correlation near zero, but with a high return given the short time-frame. If this is done repeatedly and regularly it is possible to nearly completely decouple returns from the broader market while achieving high capital gains.