(CNN) -- Dozens of international envoys walked out during a speech by Iran's president Monday as he accused Israel of having a "racist government" and committing genocide.
European Union delegates leave during Iranian President Mahmoud Ahmadinejad's speech Monday.
Mahmoud Ahmadinejad accused the West of making "an entire nation homeless under the pretext of Jewish suffering ... in order to establish a totally racist government in occupied Palestine."
Many delegates at the controversial U.N. anti-racism conference in Geneva, Switzerland, cheered his words as a minority of diplomats -- mostly from Europe -- collected their papers and briefcases and left the room. Cameras at the scene showed empty seats where delegates from France, Finland and Denmark had been sitting.
The British and Spanish delegations also walked out, both countries' foreign affairs divisions confirmed. Watch delegates make their exit »
U.N. Secretary-General Ban Ki-moon issued a statement criticizing Ahmadinejad for using the conference "to accuse, divide and even incite."
Ban said he had spoken to the Iranian president and asked him not to focus on "divisiveness" in his address.
"It is deeply regrettable that my plea to look to the future of unity was not heeded by the Iranian president," Ban said.
"This is the opposite of what this conference seeks to achieve. This makes it significantly more difficult to build constructive solutions to the very real problem of racism," Ban said in a statement following Ahmadinejad's speech.
During Monday's speech, Ahmadinejad paused a moment, then continued: "In fact, in compensation for the dire consequences of racism in Europe, they helped bring to power the most cruel and repressive racist regime in Palestine.
"It is all the more regrettable that a number of Western governments and the United States have committed themselves to defend those racist perpetrators of genocide," he said, echoing Tehran's official line on the Israeli-Palestinian conflict.
Israel was established in 1948 as a homeland for Jewish people after the Holocaust, on land also claimed by Palestinians.
Ahmadinejad said Zionism, the Jewish national movement, "personifies racism" and accused Zionists of wielding economic and political resources to silence opponents.
He also blasted the United States-led invasions of Iraq and Afghanistan. "Wasn't the military action against Iraq planned by the Zionists and their allies in the then-U.S. administration?" he demanded.
At least two protesters in brightly colored wigs interrupted Ahmadinejad as he began to speak, shouting: "You're a racist!" in accented English.
But some delegates cheered as he began his speech while security officers dragged the protesters from the chamber.
Later in the address, more protesters shouted at him from a balcony, leading him to pause and look down for a moment, a smile playing across his lips.
He also blamed the United States for the world economic crisis.
Israel withdrew its ambassador from Switzerland in protest before the conference, which the United States and a number of other countries are boycotting.
Alejandro Wolff, the U.S. deputy ambassador to the U.N., called Ahmadinejad's remarks "vile," "hateful" and "inciteful," and praised Ban's condemnation.
Speaking at a Holocaust remembrance service in Jerusalem, Israeli President Shimon Peres said Ahmadinejad's address "constitutes an acceptance of racism, rather than the fight against it."
"It is hard to fathom why despots such as Hitler the Nazi, Stalin the Bolshevik and Ahmadinejad the Persian chose the Jews as the main target for their hatred, their madness and their violence," Peres said. "Perhaps they targeted the Jewish people because of its spiritual power - a nation poor in material possessions, but rich in values -- for he who is infected with megalomania fears the power of the spirit."
New Israeli Prime Minister Binyamin Netanyahu declared during the Holocaust commemoration event that Israel will not allow "Holocaust deniers to carry out another Holocaust against Jews."
A number of European countries had vowed to walk out if Iran's president made offensive remarks.
"The UK unreservedly condemns Iranian President Ahmadinejad's offensive and inflammatory comments. Such outrageous, anti-Semitic remarks should have no place in a U.N. anti-racism forum," said Peter Gooderham, Britain's envoy to the U.N. in Geneva.
British Prime Minister Gordon Brown was quick to "unreservedly condemn" the Iranian president's words. Britain decided against boycotting the event being held in Geneva, although other Western countries including Australia, Canada, Germany, Italy, Israel, the Netherlands, New Zealand, Poland, Sweden, and the United States withdrew in advance over concerns that the conference would devolve into a platform for some nations to attack Israel.
French Foreign Minister Bernard Kouchner said he instructed the country's ambassador to the U.N. to leave the conference along with his European colleagues after Ahmadinejad spoke "in an unacceptable way."
Koucher explained that France "would not tolerate anyone holding the conference hostage and using it as a platform for making heinous remarks."
Canadian Prime Minister Stephen Harper told reporters that he believed the vocal opposition to Ahmadinejad's positions was a "very positive thing". The major problems with Iran, according to Harper, are continued threats against Israel and against the Israeli people, along with persistent nuclear ambitions.
Ahmadinejad was the first speaker at the conference because he was the only head of state to respond to the invitation, conference spokesman Ramu Damodaran told CNN.
"Invitations are sent to all member states. They decide at what level they wish to be represented," he explained. Ahmadinejad "was the only head of state who had confirmed as of today -- and when you arrange the list of speakers, heads of state get precedence over non-heads of states."
The United States, among others, is refusing to send any envoys at all to the Durban Review Conference.
While rejecting the boycott, Navi Pillay, the U.N. high commissioner for human rights, urged reporters "not allow this one intervention to mar the conference.
"I prefer to move on," she said at a news conference after Ahmadinejad's speech. She also criticized the delegates who walked out during his speech, saying his "unsavory remarks" did not "provide justification for anyone to walk out of the conference."
Pillay said Sunday that she regrets -- and is "shocked" by -- the United States' decision to boycott.
The U.S. State Department said Washington's decision was based in part on a conference document that "singles out" Israel in its criticism and conflicts with the United States' "commitment to unfettered free speech."
Australia, Canada, Germany, Poland and Italy are among those also boycotting the conference
CNN's Atika Shubert contributed to this report.