JERUSALEM (CNN) -- Israel's defense minister said Wednesday that Prime Minister Ehud Olmert should step down while he defends himself against corruption allegations.
Defense Minister Ehud Barak says the corruption inquiry is too distracting for the Israeli prime minister.
Ehud Barak, who also heads Israel's Labor Party, said Olmert's leadership is compromised by the distraction of a corruption investigation. The prime minister should suspend himself, resign or take a vacation, he said.
Olmert's Kadima Party must then choose a new leader "and with the person that replaces him there, we will consider joining hands," Barak said.
Olmert responded Wednesday, according to Israeli daily newspaper Haaretz.
"Someone under investigation doesn't necessarily have to resign," Olmert said. "You can be sure that I have explanations for all the allegations against me, and every testimony will be refuted."
"I need to resign because someone said something against me? Every minute an investigation is launched and someone has to resign? If so, four prime ministers should have resigned in recent years," he said, according to the newspaper.
Olmert has said he would only resign if he is indicted on corruption charges.
But Barak said that if Olmert refuses to resign, the Labor Party will push for elections.
"In light of the situation created and the hefty challenges facing Israel," Barak said. "I don't think the prime minister can run both the country and take care of his personal situation."
Barak, a former prime minister, spoke a day after American businessman Morris Talansky testified that he gave Olmert $150,000 in loans and direct payments over 14 or 15 years.
"I only know that he loved expensive cigars. I know he loved pens, watches. I found it strange," Talansky testified Tuesday.
The probe is the fifth investigation into Olmert since he became prime minister two years ago. Olmert has not been convicted of wrongdoing.
He also is under investigation for a home in an upscale Jerusalem neighborhood that he bought while he was the city's mayor. He allegedly bought it below market value in return for favors.
Israeli officials also are investigating an alleged $10 million business deal involving his former business partner. Olmert is alleged to have taken part in the deal while he was labor minister.
He is also under investigation for allegedly appointing his political cronies to positions at Israel's Small Business Authority from 2003 to 2006 while he was labor minister.
Olmert was Jerusalem's mayor from 1993 to 2003 and served in several ministerial capacities -- including as minister of trade, labor and industry -- from 2003 to 2006 before taking over as prime minister after a stroke incapacitated Kadima founder Ariel Sharon.
A yearlong investigation into Olmert's role in the privatization of Israel's second-largest bank wrapped up in November with no charges filed.
CNN's Shira Medding contributed to this report.