Conservative Republican Rep. Jim Jordan called off a second speaker’s vote Tuesday evening while he scrambled behind the scenes to try to win over 20 Republicans who voted against him earlier in the day. Jordan has scheduled another vote for speaker for 11 a.m. ET Wednesday, where the Ohio Republican and his allies hope to see his support increase even in the face of entrenched opposition. The level of GOP opposition to Jordan during Tuesday’s speaker’s vote – held exactly two weeks after the House ousted Kevin McCarthy – was a disappointment for Jordan’s allies who had expressed hopes that the number of holdouts would only be in the single digits. The 20 Republicans who voted against Jordan include House Appropriations Chairwoman Kay Granger of Texas, Rep. Mario Diaz-Balart of Florida, Rep. Ken Buck of Colorado and a quartet of New York Republicans in purple districts. The anti-Jordan contingent cast six votes for McCarthy, seven votes for Majority Leader Steve Scalise and three for former New York GOP Rep. Lee Zeldin, among other alternatives. Jordan’s opponents come from several factions. There are centrist Republicans concerned that the face of the House GOP would be a conservative hardliner, as well as lawmakers still furious at the small group of Republicans who forced out McCarthy and then opposed the speaker nomination of Scalise, who initially defeated Jordan inside the GOP conference, 113 to 99. Seven members of the House Appropriations Committee – which has fought Jordan’s opposition to spending bills for years – voted against Jordan. Jordan can only afford to lose four GOP votes. The House’s slim margin is what led to McCarthy’s removal at the hands of a band of eight GOP rebels – and now a similarly sized group of House Republicans could block Jordan’s ascension, too. After the first vote, the House recessed and Jordan shuffled between the speaker’s office and the majority whip’s office holding meetings, before Jordan said the next vote would be on Wednesday. “We’re making progress. I feel good about it. We’re going to keep going,” Jordan said Tuesday afternoon. “I had great conversations, great discussions with our colleagues.” One of Jordan’s meetings was with Scalise. Before the first vote, Jordan asked the majority leader to give him a nominating speech on the floor, and a source told CNN Scalise wouldn’t commit to doing so. A Scalise spokeswoman said he has been supportive of Jordan throughout the process. Several Jordan holdouts expressed openness to switching their votes, and one said he would support the Ohio Republican on the next ballot. But several Jordan opponents said they would not be swayed – and GOP sources say the opposition could grow because some members only committed to backing him on the first ballot. In January, 19 Republicans initially voted against McCarthy for speaker, before he eventually won a majority and the gavel after 15 votes. Jordan hasn’t signaled how long he plans to take his push to become speaker. Jordan and his allies felt they had made significant headway over the past several days, with the Ohio Republican pitching skeptical lawmakers one on one – and his allies outside Congress attacking the holdouts and threatening political consequences if they stand against a favorite of the Trump-aligned GOP base. Until the House selects a speaker, the chamber is in a paralysis, unable to consider legislation, such as passing additional military aid to Israel or government funding – with the threat of a shutdown now one month away thanks to McCarthy’s six-week stopgap spending deal that prompted the move against him. The extended impasse has intensified conversations among rank-and-file Republicans and Democrats about empowering a speaker pro tempore if Jordan fails again on the second ballot, though significant hurdles remain for such a move. Jordan tries to move opponents On the next vote, at least one of Jordan’s opponents said he would flip: GOP Rep. Doug LaMalfa of California, a longtime McCarthy ally who voted for McCarthy on the first ballot, told reporters he will vote for Jordan on the second ballot. “I’m not against him,” LaMalfa said of Jordan. “I was for Kevin McCarthy the whole time and I thought the process has been terrible, what has happened to him and the things that he’s been held up for. So I’m voting for Jim Jordan because he’s a good guy. He’s done good work on committee. And we need to move forward with this place today, and get our work done.” Rep. John James of Michigan, who voted for Oklahoma Rep. Tom Cole for speaker, said he would be open to supporting Jordan in future rounds of voting and that he planned to speak with him later. “I’m talking with Jim Jordan right now and we’re going to work it out,” James said. But multiple Republicans said they cannot be moved. In the aftermath of the initial vote, several Jordan opponents called for the House to immediately hold another floor vote for speaker. GOP Rep. Carlos Gimenez of Florida, who voted for McCarthy, said he will never vote for Jordan but would consider voting for other candidates. “I’m not going to be part of a coup,” Gimenez said. Rep. John Rutherford of Florida also indicated he would stick with Scalise going forward and believes the party should find a “consensus” candidate, such as interim Speaker Patrick McHenry. “I think now we’re gonna have to find a consensus candidate,” he said, adding: “I kind of like Patrick McHenry.” Nebraska Rep. Don Bacon said Monday that he would oppose Jordan because he “can’t get past the fact that a small group in our conference violated the rules to get rid of Kevin, and then blocked Steve.” A group of centrist New York Republicans – a key bloc of lawmakers in the House, who helped hand the GOP the majority – privately discussed their vote for speaker for weeks, conferring with one another and weighing the idea of voting as a bloc, sources told CNN. As they wrestled with whether to back Jordan, one thing that came up repeatedly and gave them pause: Jordan’s votes against key priorities in New York, including voting against aid for superstorm Sandy and against 9/11 health care funding for first responders. But they didn’t make a final decision on how they would vote until Tuesday morning, sources said. Four voted against Jordan. Democrats say they’re open to talks of a consensus candidate House Democrats, meanwhile, all voted for House Minority Leader Hakeem Jeffries, who got more votes than Jordan for speaker, 212 to 200. Jeffries told reporters Tuesday evening that his caucus would be open to reaching some form of agreement with “traditional Republicans” to reopen the House, saying that informal conversations have “accelerated” the last few days and that it’s his hope they will continue. Jeffries said he was open to further empowering interim speaker Patrick McHenry, a North Carolina Republican. “Our focus right now relates not just to any one individual, but to getting the institution reopened. I have respect for Patrick McHenry. I think he is respected on our side of the aisle. There are a whole host of other Republicans who are respected on our side of the aisle. Jim Jordan is not one of them,” he said. Rep. Steny Hoyer, the former House majority leader, also pointed to the interim speaker. “People are talking to one another. We’ve had a House that’s now been shut down by the extreme wing of their party,” Hoyer said. “One option is we vote for a speaker that we think would be responsible and effective. We have an acting speaker that I think wants to be responsible. So, there are a lot of options.” But the obstacles to forming some kind of coalition – even in the interim or for a short period of time – are immense. For one, Republicans would have to admit failure to find a unity candidate within their own rank. And Democrats would have to help Republicans at a moment when they could likely use this episode as a cudgel against the GOP in the next election. McHenry is also making clear to allies he isn’t interested in that route and wants the House focused on electing a permanent, Republican speaker, according to a source close to him. He was dismissive of the idea on Tuesday evening. “I voted for Speaker designate Jordan on the House floor, and it should be Jim Jordan,” he said. “I supported Steve Scalise before, I support Jim Jordan now. We need to get this done.” Still, two sources told CNN that momentum is building for the idea, and the circle of GOP members interested in it is growing. Jordan, however, dismissed the notion of a compromise over the speakership. “No one in our conference wants to see any type of coalition government with Democrats,” he said. “We’ve gotta have a speaker, and it can’t be some deal with the Democrats. The American people don’t want that. They elected Republicans in a majority.” Jordan’s supporters still believe he can convince enough opponents to come around, pointing to the several key votes Jordan won on Monday, including House Armed Services Chairman Mike Rogers of Alabama and Rep. Ann Wagner of Missouri, who had previously called Jordan a “nonstarter.” “If we can whittle it down to 10 in a hurry, and then get there – I think it’s more likely we do than we don’t,” Rep. Dan Meuser of Pennsylvania said of the Jordan opponents. “On the same note, we’ve got to be prepared on what’s next.” Jordan’s backers have urged the conference to unify around him. Rep. Scott Perry of Pennsylvania argued that those angry at what’s happened over the past two weeks shouldn’t take it out on Jordan, because he supported both McCarthy and Scalise. “Feelings are hurt,” Perry said. “But Jim didn’t have anything to do with that. So they need to assign their ire, if you will, to those who they think deserve it – but certainly not Jim Jordan.” This story and headline have been updated with additional developments.