Washington (CNN) -- This month's deadly assault on the U.S. Consulate in eastern Libya was clearly a planned attack by terrorists, U.S. Defense Secretary Leon Panetta said Thursday.
"As we determined the details of what took place there and how that attack took place, it became clear that there were terrorists who planned that attack," Panetta said.
Speaking to reporters at the Pentagon, Panetta said investigators had not yet determined exactly what terrorists were involved in the attack on the U.S. Consulate in Benghazi.
"There's a lot of different kinds of terrorism in that part of the world," he said.
It's the administration's latest assessment of the September 11 consulate attack that killed the U.S. ambassador to Libya and three other Americans. The incident fueled increased global scrutiny of the North African nation and increased political sparring in the United States over the investigation into who was behind it.
The FBI is investigating the attack, although its agents are in Tripoli and have yet to access the consulate, which is still unsecured, according to senior law enforcement sources. Libyan officials have said they have brought in dozens of people for questioning in connection with the attack.
In the days after the assault on the Benghazi consulate, U.S. administration officials offered conflicting assessments of what led to the fatal security breach. Some top officials said the violence erupted spontaneously amid a large protest about a U.S.-made film that mocked the Muslim Prophet Mohammed.
Administration, intelligence officials and lawmakers have disagreed about whether the violence was the result of a mob gone awry, a planned terror attack or a combination of the two.
On September 14, White House spokesman Jay Carney said U.S. authorities had "no actionable intelligence" indicating the attack "was planned or imminent."
Two days later, Susan Rice, U.S. ambassador to the United Nations, said U.S. officials believed extremists carried out the attack after a spontaneous protest began outside the building.
"We do not have information at present that leads us to conclude that this was premeditated or preplanned," she said.
On September 19, Matthew Olsen, the nation's counterterrorism chief, told senators it had been a terrorist attack. The next day, Carney also said it was "self-evident that what happened in Benghazi was a terrorist attack."
The administration's response has drawn sharp criticism from Republican lawmakers, who have argued that it mischaracterized the attacks and gave unnecessary credence to the film.
"They were either incredibly naïve or willfully deceiving the American people. I don't know which," Republican Sen. John McCain told CNN's "Anderson Cooper 360" on Thursday.
Asked when he had concluded that terrorists were behind the consulate attack, Panetta said, "It took a while to really get some of the feedback from what exactly happened at that location."
An al Qaeda link?
A senior U.S. official told CNN that there was evidence within 24 hours of the attack that suggested it was the work of extremists either affiliated with al Qaeda groups or inspired by them.
"We started to get a strong sense of it," the official said. He declined to be identified due to the sensitivity of the information.
The efforts by al Qaeda, especially the Mali-based al Qaeda in the Islamic Maghreb, to extend its reach into Libya and elsewhere has been of concern to the United States, something that Secretary of State Hillary Clinton mentioned on Wednesday.
"For some time, al Qaeda in the Islamic Maghreb and other groups have launched attacks and kidnappings from northern Mali into neighboring countries," she said. "Now, with a larger safe haven and increased freedom to maneuver, terrorists are seeking to extend their reach and their networks in multiple directions. And they are working with other violent extremists to undermine the democratic transitions under way in North Africa, as we tragically saw in Benghazi."
That comment has been interpreted as Clinton tying those behind the Benghazi assault to al Qaeda. But a senior State Department official said Clinton was speaking more generally about how al Qaeda in the Islamic Maghreb is working to undermine democratic transitions such as the one in Libya.
"With regard to the specific issue of who was responsible for the Benghazi attack, as everybody in the administration has said, we can't go beyond our preliminary statements until we have the results of the FBI investigation," the State Department official said.
Key issue: Security concerns
Security concerns remain a top issue for the FBI and other U.S. officials in Libya.
A U.S. official told CNN on Thursday that security concerns were the primary reason that FBI investigators had not yet traveled to Benghazi.
Last week, Libya's prime minister elect said Libyan authorities had offered to provide security to FBI investigators in Benghazi. But in order to go to the eastern Libyan city, FBI and military officials have decided they would need more military protection in case there's another rocket attack on the consulate, Bob Baer, a CNN contributor and former CIA officer, told "AC 360."
"They would need heavy weapons to counter another one of those attacks," Baer said, citing Pentagon sources. "We don't really know who led the attack and whether they're still active or not."
A senior State Department official said Thursday that the United States was removing staff from its embassy in Tripoli for security reasons. The move is intended to be temporary, the official said, and the staff could return by next week.
A statement on the U.S. Embassy in Libya's website warned citizens to avoid areas of Tripoli and Benghazi where anti-extremist demonstrations were scheduled for Friday.
"Even demonstrations that are meant to be peaceful can become violent and unpredictable," the statement said.
U.S. intelligence analysts are still looking into whether local extremists quickly took advantage of an opportunity during the assault earlier this month to attack the consulate as they saw the initial demonstrations unfold outside the gate. It also may be the case, a senior U.S. official said, that there was no central "command and control" directing or ordering the attack.
Much of the initial intelligence about the incident came from intercepts of communications, according to the official. But even at that point, the information was "bits and pieces" and needed to be verified, the official said. Even now inside the national security community, there is no firm consensus on several key points, the official emphasized.
A key analytical question also now being posed is whether this ability to establish a stronghold and then rapidly attack targets of opportunity is potentially a new trend or capability by al Qaeda affiliates, especially in North Africa.
U.S. officials have said there was no "actionable intelligence" ahead of the September 11 attack, but there was a sense by intelligence agencies that groups in eastern Libya were seeking to "coalesce," Joint Chiefs of Staff Chairman Gen. Martin Dempsey said Thursday.
"There was a thread of intelligence reporting that groups in the environment in eastern Libya were seeking to coalesce but there wasn't anything specific and certainly not a specific threat to the consulate that I am aware of," Dempsey said.
When asked whether the State Department was aware of that information, Dempsey said it was shared among intelligence agencies and agency "partners."
That information first came to light in the month before the attack and was indicating extremist groups "may be collaborating and are more connected than they are disparate," a source familiar with Dempsey's thinking on that intelligence thread told CNN.
The United States had been conducting drone surveillance missions and monitoring cell phone traffic among extremist groups in eastern Libya for some time, U.S. officials told CNN.
CNN's Carol Cratty, Elise Labott and Jomana Karadsheh contributed to this report.