There’s been drama recently in a normally quiet corner of the market.

This story got its start back in 2015, when Warren Buffett helped to merge food makers Kraft and Heinz. At first, it looked like a smart idea. Through cost-cutting, the combined company was expected to save more than $1 billion in annual operating expenses.

“This is my kind of transaction,” Buffett said at the time, “uniting two world-class organizations and delivering shareholder value. I’m excited by the opportunities for what this new combined organization will achieve.”

The excitement was short-lived, and many observers were skeptical from the start, mainly because Buffett had teamed up with a private equity firm called 3G to make the purchase. 3G had a reputation for being overly zealous when it came to cost-cutting.

Initially, Buffett defended 3G. They “could not be better partners,” he wrote in his 2015 annual letter. But within a few years, it became clear that the skeptics had been right. Sales at the combined company began falling, and in 2018, Buffett’s Berkshire Hathaway recorded a $15 billion write-down on the value of its Kraft Heinz holdings.

The following year, Buffett publicly acknowledged that the merger had been a mistake and that Berkshire had overpaid for its stake. “The business does not earn more because you pay more for it,” he said.

In the years since, Kraft Heinz has continued to struggle with declining sales. To address the problem, in January of this year, the company brought in a new CEO, Steve Cahillane, and tasked him with splitting the company back up again.

By that point, though, Buffett had changed his mind again. His view was that it was now better to leave the combined company intact rather than going through the costly exercise of trying to break it back up. A breakup, he said, wouldn’t create value. “It doesn’t do a thing, you know, for what the ketchup tastes like.” Despite his influence, though, the break-up plan appeared to be moving forward, and Cahillane took the helm on January 1 with that mandate.

Within weeks, Cahillane came around to Buffett’s point of view. The company’s woes were more fundamental, he told the board, and breaking it up wouldn’t address those core issues. Where things go next is an open question.

This story is notable because of Warren Buffett’s involvement, but it turns out not to be so unusual. Studies over the years have found that corporate mergers and acquisitions, on average, do not create value. According to a study by KPMG of more than 3,000 acquisitions, 57% of deals were found to detract from shareholder value rather than increase it. Other research puts the failure rate in the neighborhood of 70%. Aswath Damodaran, a finance professor at NYU, sums it up this way: “More value is destroyed by acquisitions than by any other action that companies take.”

Why do so many transactions detract from shareholder value? Economist Richard Thaler attributes it to what he calls the “winner’s curse.” This phenomenon was first identified in the petroleum industry, where competitive auctions are held for oil leases. Research found that the winners of these competitive auctions often ended up disappointed—not because they didn’t find any oil, but simply because they had overpaid.

Thaler explains that auctions—especially when there are large numbers of bidders—can cause some participants to become emotional, to the point that they become undisciplined and end up bidding too much. The winners in these situations are thus “cursed” because they’re the ones who were willing to overpay the most and thus tend to be most disappointed.

Thaler found that the winner’s curse dynamic appears across industries, and that is what explains the poor track record of corporate acquisitions. Competitive situations, whether it was in the Kraft-Heinz case, or in the one that recently played out in the competition for Paramount, can cause prices to go too high. That’s great for sellers but a key reason why acquirers often end up regretting their decisions and why a large number of corporate takeovers end up being reversed.

So why, despite all this data, do corporate managers—including even Warren Buffett—pursue these transactions? There are three key reasons.

The first is that they’re an easy way for companies to combat stagnant growth—much easier than the hard work of developing new products. This helps explain the Kraft-Heinz tie-up. According to a write-up in 2015, when the merger was first announced, many of Kraft’s businesses had been stalled out, delivering zero or even negative growth.

Another reason mergers and acquisitions are popular despite the odds: Corporate managers tend to overestimate the economic benefits—so-called synergies—that will result from a transaction. Consider companies like Kraft and Heinz. It was easy to make the argument that two companies in the same industry would be able to gain significant efficiencies by combining operations and realizing economies of scale. And since some number of transactions do succeed, even if it’s only a minority, it’s natural for corporate managers to believe that their transaction will be the one to beat the odds.

In a 1986 paper, economist Richard Roll identified a related phenomenon, which he dubbed “the hubris hypothesis.” The logic is as follows: Corporate managers who find themselves in a position to be making acquisitions are, by definition, probably doing well. Their stock prices are up, and they likely have cash in the bank. Because their businesses are strong, they’re more likely to feel self-confident in their ability to succeed with a merger or an acquisition even when the data suggests the odds are against them.

The lesson for individual investors? Companies will probably always pursue transactions like this that end up subtracting from shareholder value. But since there’s no way to predict when this will happen, I see this as yet another reason to choose broadly-diversified index funds, where any one company’s mistake generally won’t have too much of a negative impact.

Also, to the extent that the company being acquired is also in the index, passive fund investors can enjoy the benefits that accrue to that company’s shareholders.​