(CNN) -- The religious leader behind plans to build an Islamic center and mosque a few blocks from New York's ground zero said Wednesday night that America's national security depends on how it handles the controversy.
"If we move from that location, the story will be the radicals have taken over the discourse," Imam Feisal Abdul Rauf told CNN's Soledad O'Brien on "Larry King Live."
"The headlines in the Muslim world will be that Islam is under attack."
But some critics decried his assessment.
"The whole national security thing: that's a veiled threat," Andy Sullivan, a union construction worker who wants all New York construction workers to boycott the proposed Islamic center, said on CNN's "AC 360" Wednesday night. "He's saying 'you make me move' and, guess what, the whole radical Muslim world is coming after us."
"This is a turf war," Sullivan said.
The imam, who repeatedly said his mission was to promote peace and build a bridge among faiths, said he was also speaking about "radicals" on both sides of the debate on the Islamic center. "Our national security now hinges on how we negotiate this, how we speak about it."
"The battlefront is between moderates of all sides ... and the radicals on all sides," he said.
Moving the project to another location would strengthen Islamist radicals' ability to recruit followers and will likely increase violence against Americans, the imam said.
But Rosaleen Tallon, whose firefighter brother died on 9/11, finds that notion ironic.
"On 9/11, it didn't take a mosque for extremists to come and attack the World Trade Center and kill my brother," Tallon said on "AC 360." "What I'm finding here to be very disturbing is that now ... this mosque has to go up or there will be retribution."
Rauf said that "nothing is off the table" when asked whether he would consider moving the site.
"We are consulting, talking to various people about how to do this so that we negotiate the best and safest option."
The imam told O'Brien "had I known [the controversy] would happen we certainly would never have done this."
Asked if he meant he would not have picked the location, Rauf said, "we would not have done something that would create more divisiveness."
Worry over what some observers have termed "Islamophobia" has been heightened by a Gainesville, Florida, church's plan to burn copies of the Quran on Saturday, the ninth anniversary of the 2001 attacks on New York and Washington.
Rauf said he hopes the church reconsiders. "It is something that is not the right thing to do."
"With freedom comes responsibility," he said. "This is dangerous to our national security and is also the un-Christian thing to do. ... Jesus said to love your enemy. We are not your enemy."
Responding to a poll showing 71 percent of New Yorkers oppose the center's location, Rauf said he was going on television to explain his background and vision.
"I want to show them my face. Show them my track record."
The imam spoke of "a vision I've had for almost 15 years ... to establish a space that embodies the fundamental beliefs that we have as Jews, Christians and Muslims, which is to love our God and to love our neighbor -- to build a space where we have a culture of worship."
Rauf said he would continue speaking with families of those killed in the September 11, 2001, attacks. "You cannot heal a trauma by walking away from it," he said.
Opposition goes against "the fundamental American principle of separation of church and state," said Rauf, adding he has been surprised by the controversy.
The project, known as Park51, is slated to include a variety of facilities, including a prayer room, a performing arts center, gym, swimming pool and other public spaces. It is planned for a site two blocks from where the World Trade Center once stood.
A source familiar with Park51 told CNN's Allan Chernoff last week that the structure is being planned as an 11-story building. It will cover 120,000 square feet -- 10,000 feet of which would be designated for the Muslim prayer space. The developer is considering the possibility of an interfaith education/meditation/prayer space as well, the source said.
Opponents of the plan to build the center say it is too close to the site of the terror attacks and is an affront to the memory of those who died in the al Qaeda strike. Backers cite, among other things, First Amendment rights and the need to express religious tolerance.
Those who know Rauf describe him as a thoughtful man, a bridge builder who seeks to unite all faiths but who won't parse words when he sees religion used for nefarious ends.
But he has landed in controversy before.
He has chided the U.S. for killing civilians in Baghdad. He said in 2005 that the U.S. had more Muslim blood on its hands than "al Qaeda has on its hands of innocent non-Muslims."
He has also refused to accept Western governments' designation of Hamas as a terrorist group. Shortly after the 9/11 attacks, he told CBS' "60 Minutes" that "United States policies were an accessory to the crime."
Earlier this week, Rauf wrote a commentary published online by The New York Times.
"I have been struck by how the controversy has riveted the attention of Americans, as well as nearly everyone I met in my travels," Rauf wrote.
"We have all been awed by how inflamed and emotional the issue of the proposed community center has become," wrote Rauf, who had just returned from a State Department-sponsored Middle East trip to promote U.S.-Muslim relations.
"The level of attention reflects the degree to which people care about the very American values under debate: recognition of the rights of others, tolerance and freedom of worship."
Soledad O'Brien is the host of CNN's "In America" series that covers untold stories in underreported communities.