Washington CNN  — 

Anthony Scaramucci, a former White House communications director who has recently withdrawn his support for President Donald Trump, said Monday that he is putting together a coalition of former Trump Cabinet officials to denounce the President ahead of the 2020 election.

“I’m in the process of putting together a team of people that feel the exact same way that I do. This is not a ‘Never Trump’ situation. This not just screeching rhetoric. This is – OK, the guy is unstable. Everyone inside knows it, everyone outside knows it. Let’s see if we can find a viable alternative,” Scaramucci told CNN’s Alisyn Camerota on “New Day.”

“Moreover, I have to get some former Cabinet officials in unity to speak up about it. They know it’s a crisis,” he added. Scaramucci did not provide any names of people associated with the coalition he said he is assembling, but predicted that by “middle to late fall” there will be a “trove” of people willing to speak out against Trump.

Last week, Scaramucci announced he no longer supports Trump’s reelection bid and that the top of the 2020 Republican presidential ticket may need to be replaced. During his “New Day” appearance on Monday, Scaramucci said that if more Republicans begin publicly opposing Trump, it would jumpstart a “process” that could eventually replace Trump on the GOP ticket.

Trump, in an apparent response to Scaramucci’s “New Day” interview, labeled his former aide a “nut job” on Twitter and said he “barely knew him,” even though he hired him as communications director and Scaramucci frequently defended him on television.

Trump is already facing a Republican primary challenge by former Massachusetts Gov. Bill Weld, who announced in April he was officially entering the race to take on Trump in 2020. Earlier this year, Weld, who was the vice presidential nominee on the Libertarian Party ticket in 2016, told CNN’s Jake Tapper on “The Lead” that it would be a “political tragedy” and he would “fear for the Republic” if the country had six more years of Trump as President.

Trump could also face a challenge from former South Carolina Rep. Mark Sanford, who is mulling a longshot bid against the President for the Republican primary. On Sunday, Sanford told NBC that Trump doesn’t deserve to be reelected because he is taking the country in the “wrong direction” and that there needs to be a “course correction.”

Scaramucci, who has not publicly endorsed a Republican challenger, told Camerota that the party has to “offer Republicans a viable candidate and you also have to offer them an off ramp,” adding that while it’s “tough” to support the President, “you want to be loyal” to the party.