(CNN) -- Army Sgt. Bowe Bergdahl was physically abused during his five years in Taliban captivity and is suffering from psychological trauma, a senior U.S. official told CNN on Friday.
The information bolstered the White House argument that President Barack Obama needed to move quickly to secure Bergdahl's release in a May 31 exchange for five Taliban figures held at Guantanamo Bay.
According to the official, who spoke on condition of not being identified, Bergdahl tried to escape at one point but got caught, and his captors then confined him in small enclosure described as a cage or box.
At an American military hospital in Germany where he was taken after his handover near the Afghanistan-Pakistan border, doctors said Friday he remained in stable condition and continued to improve, but wasn't ready to travel back to the United States.
"There is no predetermined time line for Sgt. Bergdahl's recovery process," said a statement by the Landstuhl Regional Medical Center. "The duration will continue to be based on the pace of his healing and reintegration process."
Improving but still far to go
When he is ready, Bergdahl will be flown to the San Antonio Military Medical Complex in Texas, where he might be able to reunite with his parents for the first time. Bergdahl has yet to speak with them since his release.
According to military officials familiar with the process of reintegrating longtime captives, the first meeting with family or loved ones can often be the most overwhelming moment of the entire transition to freedom.
An initial family visit lasts only minutes due to the intensity and emotional overload for the former captive, the officials said.
Earlier, the Obama administration and Pentagon implored the American public and media not to prejudge Bergdahl as questions and criticism swirled around the secret swap.
National Security Adviser Susan Rice told CNN in an interview that Bergdahl was being prematurely condemned after members of his unit in Afghanistan said he deserted.
Both Rice and Pentagon spokesman Rear Adm. John Kirby said only Bergdahl knows the full truth of what happened and why.
"This is a young man whose circumstances we are still going to learn about," she told CNN in an interview. "He is, as all Americans, innocent until proven guilty. He's now being tried in the court of public opinion after having gone through enormously traumatic five years of captivity."
If the military finds Bergdahl deserted, he'll face consequences, she said.
"But in the meantime, let's remember this is a young man who volunteered to serve his country. He was taken as a prisoner of war. He suffered in captivity," Rice continued. "He's now trying to begin the process of recovery. Let's let that happen. And then let's know the facts including his side of the story, and then we can make a judgment."
An Army fact-finding investigation conducted after Bergdahl's 2009 disappearance concluded that he left his outpost deliberately and on his own free will, according to a U.S. military official briefed on the report.
But there has been no definitive finding that Bergdahl deserted, because that would require knowing his intent -- something Army officials couldn't do without talking to him.
Kirby told CNN's "New Day" that the Army is going to do a "comprehensive review" of what happened once Bergdahl is ready to participate.
"They're going to look at his situation again, the circumstances surrounding his disappearance," Kirby said. "And a key to that is going to be talking to Sgt. Bergdahl himself. Because there's really only one man who knows exactly what happened with respect to his disappearance and his captivity, and that's Sgt. Bergdahl himself."
However, that will take time, Kirby cautioned.
"The process will only go as fast as he and his doctors are ready to let it go. Nobody's going to be rushing him back into society and even back home," he said, later adding: "I don't think it does anybody any favors for folks to be out there speculating and criticizing him when he hasn't even had a chance to tell his own side of the story."
In Washington, Republican critics have complained that the exchange for Bergdahl gave up hardened terrorists capable of attacking U.S. troops and interests. In addition, legislators from both parties accuse the Obama administration of violating the National Defense Authorization Act by failing to provide 30 days' advance notice of a transfer of Guantanamo detainees.
Administration officials have given several reasons for the lack of notification -- a need to move quickly due to Bergdahl's poor health and the overall threat to his safety, and the likelihood that even a small leak of the plan would have gotten him killed.
At a classified briefing this week in defense of the decision to not notify Congress about the deal, officials made clear that Bergdahl's life was in danger if he remained in captivity and if word of the planned exchange got out.
"They had intelligence that, had even the fact of these discussions leaked out, there was a reasonable chance Bowe Bergdahl would have been killed," Sen. Angus King, an independent who caucuses with the Democratic majority, told CNN on Thursday.
No evidence for desertion
The question of what legislators should have known -- and when -- has helped propel a broader controversy over whether bringing home a soldier accused by comrades of deserting his unit was worth releasing the alleged terrorist leaders.
Under terms of the exchange, the five Taliban figures were handed over in Qatar, where the government says they will stay for at least a year and be monitored.
That fails to satisfy GOP Sen. Marco Rubio of Florida, who told Fox News on Thursday that the released Taliban figures will "rejoin the battlefield" to fight against the United States and its allies.
Kirby said Defense Secretary Chuck Hagel was satisfied with the conditions placed on the former Guantanamo detainees.
"He was comfortable with the assurances that we got from the Emir of Qatar, that this was in the national interest and the risk was substantially mitigated," Kirby said, adding: "Is it eliminated? No. It's of course not eliminated. ... But it is substantially mitigated. And while I can't go into the details of assurances that we got from the government of Qatar, again, the secretary believes that his concerns were satisfied."
Some soldiers involved in operations to find Bergdahl have said at least six soldiers were killed searching for him. But a U.S. official told CNN that Pentagon and Army officials have looked at the soldiers' claims, and "right now there is no evidence to back that up."
Wanderlust and escape
A U.S. official said Bergdahl may have tried to run away two times, but the only way to know for sure would come from talking to him.
According to a Taliban source, Bergdahl's captors stopped keeping a close eye on him in 2011, and Bergdahl used the opportunity to bolt, CNN's Jake Tapper reported.
Bergdahl survived on his own for three days, according to an account in the Daily Beast. Then the Taliban found him hiding in a trench he'd dug with his bare hands.
"He fought like a boxer," the Taliban source said of Bergdahl, who was overcome and dragged back in shackles.
According to the Army investigation, Bergdahl had previously wandered off or disappeared -- once in basic training in Irwin, California, when he slipped away to watch the sun set, and again after arriving in Afghanistan, when fellow soldiers said he took a stroll outside the wired perimeter of his outpost.
Under partisan fire
Republicans who once called for the administration to work for Bergdahl's return now say the swap released some of the worst Taliban leaders being held at Guantanamo Bay.
Despite their criticism and anger from lawmakers over being left out of the loop, the White House has argued Obama had the legal authority and moral high ground to do what he did.
Originally, the administration had given lawmakers a different line of reasoning, arguing that Bergdahl's deteriorating health dictated fast, uncomplicated action.
Officials reiterated that point in this week's classified briefing. Senators were shown the so-called proof of life video of Bergdahl, which the administration has said shows the soldier in deteriorating health.
"He could barely talk. He couldn't focus his eyes," King said. Other senators who saw the video at Wednesday's classified briefing, such as Republican Tom Coburn of Oklahoma, who is a doctor, said Bergdahl looked drugged, not sick.
Recovery and accusations
The accusations of desertion continue to hang over Bergdahl as he recovers. He is conversing in English and has become more engaged in treatment, a Pentagon spokesman said.
Former Army Staff Sgt. Justin Gerleve, Bergdahl's former squad leader, told CNN's "The Lead With Jake Tapper" that he believes Bergdahl is at least partly to blame for the deaths of soldiers who went to find him.
"I can't really say I blame Bergdahl to fullest extent, but if he wouldn't have deserted us, these soldiers very well could have been in a different place at a different time," he said.
Gerleve also said he heard radio intercepts that an American "was running around looking for people to speak English and wanted to seek out the Taliban."
Kirby said Friday that "we do not have any indication at this time that there were specific casualties caused by his disappearance or the search for him in the aftermath," but he noted the Army's review would look into that question.
History of a deal
Consideration of a deal with the Taliban dates back several years but failed to gain traction when the possibility first emerged in 2011, according to sources.
A Defense Department official has said that Robert Gates, Obama's first defense secretary, opposed the idea back then, as did his successor, Leon Panetta, and former Secretary of State Hillary Clinton.
King noted that the Taliban figures involved in the swap may have been freed soon in any event.
They were being held in Guantanamo as enemy combatants, he said. Legally, after hostilities in Afghanistan end, they would have to be set free.
Rice has been criticized for saying on Sunday that Bergdahl served with "honor and distinction," a statement called into question by his unit colleagues who said he deserted.
"I realize there has been lots of discussion and controversy around this," Rice said. "But what I was referring to was the fact that this was a young man who volunteered to serve his country in uniform at a time of war. That, in and of itself, is a very honorable thing."
Her comments last weekend weren't the first time she prompted criticism over an appearance on one of Washington's Sunday talk shows.
In September 2012, she appeared on several programs following the attack on the American diplomatic post in Benghazi, Libya -- and asserted the incident was prompted by a spontaneous protest against an anti-Muslim video made in the United States.
The administration later acknowledged the attack was a terrorist assault on the U.S. compound.
The "Susan Rice talking points" have become a political flashpoint, spurring congressional investigations, administration explanations, and -- potentially -- a major headache for former Secretary of State Hillary Clinton if she runs for president in 2016.
Rice has said she was simply providing the best information the administration had at the time -- and on Friday, she contended that just because the information she provided that day turned out to be inaccurate doesn't mean she's willfully misleading Americans.
"I'm upfront with the American people and I always do my best on behalf of my country and I do my best to tell the facts as I know them," she said.
The Bergdahl Files: A custom magazine
CNN's Jake Tapper, Dana Bash, Ben Brumfield, Matthew Chance and Martin Savidge contributed to this report.