New Orleans, Louisiana (CNN) -- The Bush administration made a "fatal mistake" by talking up facts and figures without painting a broader picture of the obstacles in its widely criticized Hurricane Katrina response effort, ex-FEMA chief Michael Brown said Thursday.
Brown told CNN's Anderson Cooper that the talking points he and other federal officials used at the time didn't tell the whole story.
"They were factually correct, but weren't in context. We're moving all of this stuff in. We have teams here. Rescue teams are doing this," he said. "But we never explained to the people that it's not coming as fast as we want it to, and it's not enough, because of the number of people that were left behind in the aftermath of the storm."
Not making that clear was a "fatal mistake," Brown said.
"Had I said that at the time, I probably would have gotten the old hook and been pulled off the stage anyway, but the truth would have been out," he said.
Speaking five years after the storm hit, Brown sharply criticized former Secretary of Homeland Security Michael Chertoff's handling of the situation. Chertoff had attended an avian flu convention amid the disaster.
"Here is why that's so important. In the middle of any crisis," Brown said, "whether it's a natural disaster or man-made disaster, you need to have one person in charge. And that person needs to be on the ground with the team, understanding what's going on."
The former FEMA head told CNN that he winced when President George Bush said the now infamous line, "Brownie, you're doing a heck of a job."
"I knew the minute he said that, the media and everybody else would see a disconnect between what he was saying and what I was witnessing on the ground," Brown said. "That's the president's style. His attitude and demeanor is always one of being a cheerleader and trying to encourage people to keep moving. It was just the wrong time and the wrong place."
Brown headed FEMA under the Bush administration and resigned in September 2005, two weeks after Hurricane Katrina struck the Gulf Coast and 10 days after President Bush famously complimented him.
Chertoff was criticized by a House committee in 2006 for choosing Brown to lead the government response, despite inadequate training.
"If I knew then what I know now about Mr. Brown's agenda, I would have done something different," Chertoff said.
Hurricane Katrina's fifth anniversary is Sunday.
Brown, who is now on KOA-850 AM in Denver, Colorado, was in New Orleans this week broadcasting his radio show.