Editor's note: Julian Zelizer is a professor of history and public affairs at Princeton University. He is the author of "Jimmy Carter" and of the new book "Governing America."
Princeton, New Jersey (CNN) -- Back in 2010, the conservative columnist and CNN contributor David Frum was worried about what he saw in his own party. Frum, who had worked as a speechwriter for President George W. Bush, feared that Republicans would be tempted by tea party Republicans to shift far to the right to achieve short-term electoral gains that would cost the party in the long run.
"A party must champion the values of the voters it already has," Frum wrote, then warning, "But it must also speak to the votes it still needs to win."
For several years, the tea party helped energize a moribund Republican Party. After the 2008 election, with conservatives reeling from the dismal approval ratings of Bush and the defeat of John McCain, tea party activists injected some life into the Republican grass roots, bringing out voters in the primaries who were frustrated with the political status quo.
They pushed the GOP to focus more on the issue of deficit reduction and government spending, while creating immense pressure on the congressional leadership to avoid compromises. The midterm elections of 2010 put President Barack Obama on the defensive and constrained him for much of his first term.
But Frum was right about the long run. The tea party has extracted some very high costs from the party from which it will be difficult to recover. The most immediate cost was control of the Senate. In 2010 and 2012, tea party activists knocked out some powerful Republican candidates who probably would have been victorious and improved the chances for the party to win a majority in the Senate.
In 2010, the nation was riveted when Christine O'Donnell defeated U.S. Rep. Michael Castle in Delaware in the GOP primary. Most agree Castle would have won the Senate seat. O'Donnell could not. The seat went to a Democrat.
Nevada's Sharron Angle and Colorado's Ken Buck suffered similar fates. This year, powerhouse Sen. Richard Lugar of Indiana was defeated by Richard Mourdock, who uttered some explosive comments about abortion that fueled perceptions the GOP was far right. Democrats won the seat.
Besides the Senate, the electoral costs were also evident in the Republican presidential primaries, when Mitt Romney felt the need to make the kinds of "severely conservative" statements that cost him enormously in the election.
The tea party Republicans also made it difficult for Republicans to sign off on a $4 trillion "grand bargain" about taxes and spending when Obama's back was to the wall in 2011.
The most zealous Republicans were unwilling to capitulate to the president on the issue of tax increases in return for the spending cuts they sought. They gambled that Americans would elect a Republican in 2012, which would give them free rein on Capitol Hill.
But things turned out quite differently. Instead, Republicans don't have control of the White House and face the possibility of taxes going up for all taxpayers, as well as steep cuts in defense, if nothing is done.
They don't have the leverage that existed in the post-2010 election. By most indications, they will now have to settle for ending some of Bush's tax cuts and possibly domestic spending cuts that are not as deep as they could have gotten a few years ago.
If the automatic cuts go through, a product of the failure to reach a deal when the wind was at their backs, conservatives will see heavy reductions on defense, something not likely to please the Republican base. Even if an acceptable deal goes through, after the 2012 election Americans will likely see Obama as the driving force and Republicans taking a deal because they didn't have any other choice.
The tea party has also undermined efforts by the GOP to position itself as the party of governance rather than the party of extremism. The tea partiers are part of a dynamic in the party, that started with Barry Goldwater's run in 1964, which pushed out moderates and made it more difficult for the Republicans to build a long-term coalition, as Geoffrey Kabaservice documents in his book "Rule and Ruin."
The demand to remain ideologically pure has consistently pushed Republican leaders to move to avoid saying yes to almost anything, preventing House Speaker John Boehner and Senate Minority Leader Mitch McConnell from cutting deals.
The inaction of the GOP and the ideologically charged rhetoric from its members have played directly into arguments by scholars such as Thomas Mann and Norman Ornstein that the GOP is moving far to the right and is responsible for the dysfunctional state of politics in Washington.
Finally, the tea party is making it increasingly difficult for Republicans to win over hugely important voting blocs that can play a major role in 2016 and 2020.
In 2012, the Republicans paid dearly among Latino voters for the rising power of hard-line anti-immigration advocates. The tea party helped to drown out voices such as those of Bush who believed the party needed to broaden its appeal and reach out to new constituencies, not shrink its electoral map. They have also turned off many younger millennial voters, another key constituency finding less to like about the Republican Party.
All of these point to the ways in which the embrace of the tea party has taken its toll on the Republicans. Recently, observing the losses that resulted from tea party candidates, Frum asked: "Have we learned our lesson yet?"
The answer is uncertain. At a moment when the party needs to avoid an electoral free fall with more red states turning blue, or at least becoming swing states, party leaders need to do some hard thinking about whether one of their greatest challenges does not come from Democrats, but from within.
Follow us on Twitter @CNNOpinion.
Join us on Facebook/CNNOpinion.
The opinions expressed in this commentary are solely those of Julian Zelizer.