New York CNN Business  — 

US oil companies used to ramp up production at even the slightest hint of higher prices.

That drill-baby-drill strategy worked well for American drivers last decade, keeping prices at the pump relatively low. And it made the United States the king of the oil world, surpassing both Saudi Arabia and Russia in production.

But the strategy was terrible for the oil industry’s bottom line. Drillers repeatedly oversupplied the market, careening debt-riddled companies from one price crash to another. The oil-and-gas sector was easily the stock market’s biggest loser in the 2010s.

Today, the world is completely different. Gasoline prices have surged to seven-year highs and Wall Street banks are warning that $100 or even $120 oil is on its way. Yet US oil companies are in no rush to come to the rescue, leaving the White House facing pressure from its own party to intervene in energy markets.

What’s changed is that, under immense pressure from Wall Street shareholders, oil companies are finally trying to live within their means. Even though crude has surged above $85 a barrel amid roaring demand, drillers are only gradually adding supply.

“They have PTSD. They are scarred,” Robert McNally, president of consulting firm Rapidan Energy Group, said referring to US oil companies.

US oil output is lower than before Covid

Normally, the best cure for high prices is high prices, which incentivize more supply. And yet, even though US oil prices have surged by more than 65% this year, US oil production is about 14% below the levels of the end of 2019, before Covid erupted.

“Stop spending like drunk sailors. That’s the message from shareholders,” said Pavel Molchanov, an analyst at Raymond James.

That message was heard loud and clear. Despite higher prices, 50 of the largest oil companies have increased their annual budgets by just 1% relative to their initial plans, according to Raymond James. Instead of plowing money into expensive drilling projects, the oil-and-gas industry is focused on returning cash to shareholders.

“It’s not the government that is banning them from drilling more. It’s pressure from their shareholders,” said Molchanov.

And it’s not just shareholders that are demanding to get paid fatter dividends and rewarded with bigger share buybacks.

The rise of the ESG movement is forcing fossil fuels companies to rethink their futures.

Pressure from socially conscious investors hit a crescendo earlier this year when activist investors won board seats at ExxonMobil (XOM), America’s largest oil company. That vote sent shockwaves through the industry because it was the first proxy campaign at a major US company in which the case for change was built around a shift away from fossil fuels. And it was successful.

Case in point: On Tuesday, when much of the energy industry was focused on whether President Joe Biden would intervene in energy markets, Exxon CEO Darren Woods released a statement detailing how his company is investing $15 billion over the next six years to develop a “lower-carbon future.” That includes spending on reducing emissions in heavy industry and power generation, developing carbon capture and storage and researching hydrogen and biofuels.

Oil companies are ‘very confused’

Not only are oil companies plowing money into low-carbon businesses, the climate crisis has created vast uncertainty about the future demand for fossil fuels. Appetite for gasoline is surging right now as the economy reopens from Covid. But the rise of electric vehicles will change the outlook significantly in the years to come. Many expect oil demand to eventually peak, though there remains great debate over precisely when that will happen.

At the same time, governments around the world are setting ambitious targets for cutting emissions. Biden has set a goal of cutting carbon emissions by as much as 52% by 2030, rolled out new methane regulations and returned the United States to the Paris climate agreement.

The oil industry has pointed to how this regulatory uncertainty is depressing its ability to invest in future projects.

“Companies are very confused about what the right pace of investment should be. It’s difficult to plan when there’s such demand uncertainty,” said Francisco Blanch, head of global commodities at Bank of America.

Yet analysts said it’s not fair to pin the blame for sluggish US oil supply, and thus high prices, on the climate crackdown.

“Environmental regulation will play a huge role in the oil and gas industry in the next decade and beyond. But it does not by itself explain why commodity prices are going up,” said Molchanov, the Raymond James analyst.

‘They don’t want them to spoil the party’

It’s worth noting that US oil companies are cashing in on high prices.

Chevron (CVX) recently reported its fattest quarterly profit in eight years and record cash flow. Exxon’s revenue soared by 60% last quarter.

Numbers like those are viewed as validation of the oil industry’s newfound discipline.

“A lot of this has been driven by investor sentiment. They don’t want them to spoil the party,” said Helima Croft, head of global commodity strategy at RBC Capital Markets.

The slow return of US oil is a big problem for Biden, whose poll numbers have tumbled in part due to high prices on gasoline and other items.

Energy Secretary Jennifer Granholm hinted at frustration over US supply.

“But we have the same problem in fuels that the supply chains have, which is that the oil and gas companies are not flipping the switch as quickly as the demand requires,” Granholm told CNN’s Dana Bash on Sunday.

Unlike in OPEC nations, oil production is set by private enterprise and the free market, not government leaders.

Biden’s climate conundrum

Still, in theory, Biden could hold talks with oil CEOs and urge them to turn on the taps to ease the pain on American drivers.

And they could then ask him to slash red tape and ease off environmental regulations.

But that’s not really an option for Biden, who ran on the most aggressive climate agenda in US presidential history.

Climate scientists and even the International Energy Agency have said oil companies need to stop drilling now to reduce greenhouse gas emissions warming the planet.

Like all presidents, Biden wants to keep energy prices affordable and prevent prices at the pump from hitting levels that slow the economy.

And yet he can’t tell Big Oil to drill-baby-drill and still credibly argue his administration is doing its part to save the planet.

Therein lies the inherent tension facing Biden right now. His climate ambitions are colliding head-on with economic reality.