(CNN) -- Defense Secretary Chuck Hagel returned to Congress on Wednesday to defend the deal that resulted in the release of Army Sgt. Bowe Bergdahl, saying the agreement to release five Afghan detainees was "fully consistent with U.S. law, our nation's interests and our military's core values."
"The President's decision to move forward with the transfer of these detainees was a tough call. I supported it, I stand by it," the former lawmaker told the House Armed Services Committee.
"We made the right decision, and we did it for the right reasons -- to bring home one of our own people," he said.
He said the highly criticized detainee release included "substantial mitigation of risk" to American interests, and said he did not take lightly the decision to authorize the release of the detainees.
"I take that responsibility damn seriously," he said.
"I would never sign off on any decision that I did not feel was in the best interests of this country, nor would the President of the United States, who made the final decision with the full support of his national security team," the defense chief told the committee.
Hagel, who signed off on the deal, said the effort to free the captive sergeant after five years of captivity were unaffected by questions about Bergdahl's disappearance, including statements from fellow soldiers that he walked off his post.
"Questions about Sgt. Bergdahl's capture are separate from our effort to recover him, because we do whatever it takes to recover any and every U.S. service member held in captivity," he told lawmakers.
He noted that Bergdahl has not been charged with any misconduct and that "his conduct will be judged on the facts, not political hearsay, posturing, charges or innuendo."
And he said that he was "offended and disappointed in how the Bergdahl family has been treated by some people" amid the controversy over the release.
Timeline of talks
The U.S. had talks with the Taliban involving the five Taliban detainees in 2011, Hagel told lawmakers. The Taliban later broke them off, and in September 2013, Qatar offered to mediate indirect talks, he said.
In November, the U.S. requested a proof-of-life video of Bergdahl. The video, received in January, was disturbing and "showed a deterioration in his physical appearance and mental state," Hagel said.
"Our entire intelligence community carefully analyzed every part of it and concluded that Sgt. Bergdahl's health was poor and possibly declining. This gave us growing urgency to act," he said.
The U.S. signed a memorandum of understanding with Qatar on May 12 regarding how Qatar would handle Taliban detainees transferred to its custody.
Soon after, "U.S. officials received a warning from the Qatari intermediaries that as we proceeded, time was not on our side. ... This indicated that the risks to Sgt. Bergdahl's safety were growing," he said.
'Trust has been broken'
Addressing concerns expressed by some lawmakers that the administration failed to notify Congress of the planned detainee release, Hagel said that Bergdahl's declining health and the sensitivity of the negotiations made it imperative to hold details close to the vest.
"We were told by the Qataris that a leak ... would end the negotiation for Bergdahl's release. We also knew that he would be extremely vulnerable during any movement, and our military personnel conducting the handoff would be exposed to a possible ambush or other deadly scenarios in very dangerous territory."
Hagel said he understood lawmakers' concerns about being left out of the loop.
"But under these exceptional circumstances -- a fleeting opportunity to protect the life of an American service member held captive and in danger for almost five years -- the national security team and the President of the United States agreed that we needed to act swiftly," he said.
He acknowledged the administration "could have done a better job of keeping you informed."
"I value the Defense Department's partnership with Congress and the trust we have developed over the years," Hagel told the committee. "I know that trust has been broken."
'A matter of courtesy'
Before Hagel's appearance, the Obama administration had already held two classified briefings this week for members of Congress -- one to the Senate Armed Services Committee on Tuesday and similar session on Monday with House members. The sessions have only intensified the rhetoric coming from Capitol Hill.
Members of both parties complain that the White House should have let Congress know ahead of time that the exchange was taking place. Some cite the 30 days' notice called for by the National Defense Authorization Act, while others say a simple heads-up would have been good enough.
"It's just a matter of courtesy, whether it was in law or not," said conservative Sen. Jeff Sessions of Alabama, a regular critic of the President. "A matter of this importance should have been discussed with at least key leaders in the Congress."
Rep. Adam Schiff, a California Democrat, told CNN's "Erin Burnett OutFront" that White House officials should have told congressional leaders, even if the law gave them wiggle room.
"I think the President had the constitutional authority ... to make this decision without consulting with Congress," Schiff said. "But I think it would have been wiser, far wiser, for the administration to have notified, certainly the leadership of Congress in the interest of having good relations."
Discretion was 'important'
But administration officials say the deal came together quickly, and the risks to Bergdahl and the military commandos involved in collecting him near the Afghanistan-Pakistan border required keeping the exchange top secret.
"This was a secret military mission in which disclosure of the mission could put into jeopardy not just the life of Sergeant Bergdahl but also the lives of the American servicemen who were involved in the mission," White House spokesman Josh Earnest said Tuesday. "So discretion on this matter was important."
However, House Armed Services Committee Chairman Buck McKeon of California called such reasoning for not notifying Congress "absurd."
"Nothing in the law requires the Secretary of Defense to disclose the physical details of the transfer," said McKeon, a Republican. "They are merely required to notify us in a classified manner that a transfer will take place, and of the conditions set to prevent a terrorist from re-entering the fight."
House Speaker John Boehner noted that he was briefed ahead of another significant and secret military mission -- the 2011 raid that killed al Qaeda leader Osama bin Laden.
"I was given a heads-up several days before" the bin Laden mission, Boehner said. "So this idea that they couldn't trust us to not leak things is just not true."
Democratic Sen. Joe Manchin, a moderate from West Virginia known for criticizing the administration at times, said his main concern was the release of five Taliban figures who could return to the battlefield against U.S forces and interests.
'There are going to be costs'
It's a concern shared by Boehner, who told reporters he had no doubt "that there are going to be costs, lost lives associated with what came out of this" and called the swap a de facto deal with Taliban terrorists.
On the Democratic side, Senate allies of President Barack Obama said the deal mediated by the government of Qatar came together too quickly to consult with Congress over an exchange that critics from both parties have labeled too costly.
"All I can tell you is this: (the government) knew a day ahead of time that the transfer was going to take place. They knew an hour ahead of time where it was going to take place," said Sen. Richard Durbin of Illinois, while his Democratic colleague from Michigan, Carl Levin, noted the military's top leadership fully backed the exchange.
"Whether you agree with them or not, its critically important that the American people know that the chairman of the Joint Chiefs and the vice chair of the Joint Chiefs strongly recommended this agreement knowing full well Bergdahl, one, had left his unit and knowing full well how bad these Taliban people were," Levin said.
'In a position to go after them'
Under the exchange, the released Taliban detainees were taken to Qatar, where the government says it will keep them under watch for at least a year.
But some lawmakers fear there's no guarantee.
"How we are sure that the Qataris are going to do what they are supposed to do and we are able to keep (the released Taliban figures) from engaging again?" Manchin told reporters.
A memorandum of understanding between the United States and Qatar on the deal that freed Bergdahl offered "no guarantee" that the five Taliban commanders exchanged for the captured Army sergeant would be tracked, a Democratic member of the House Intelligence Committee told CNN.
Schiff, who reviewed the agreement, said, based on his reading, that it is possible the former U.S. detainees "might disappear even during the first year in Qatar."
He told CNN's Jim Sciutto that the United States has "some capability to track them while in Qatar," but after the first year, he said, all bets are off.
"We have to expect them to return to the fight," Schiff said.
U.S. Secretary of State John Kerry defended the swap in an interview with CNN over the weekend, saying the former detainees would be monitored closely, and not just by officials in Qatar. He wouldn't say who else will be watching, but he said the United States is confident the conditions of their release will be honored.
National Security Council spokeswoman Caitlin Hayden said Tuesday that the Obama administration had received "assurances from Qatar" during the deal.
"We have acknowledged that some of these individuals could attempt to return to activities that are of concern. The President said so explicitly on his trip last week," she said. "But we have confidence that we will be in a position to go after them if, in fact, they are engaging in activities that threaten our defenses."
'Continuing to improve'
Bergdahl went missing on June 30, 2009, in Afghanistan's Paktika province, where he was deployed with the 1st Battalion, 501st Infantry Regiment, 4th Brigade Combat Team, 25th Infantry Division.
An Army fact-finding investigation conducted in the months after his disappearance concluded he left his outpost deliberately and of his own free will, according to an official who was briefed on the report.
The Army has no definitive finding that Bergdahl deserted because that would require knowing his intent -- something Army officials couldn't learn without talking to the soldier, a U.S. military official told CNN.
Bergdahl is "continuing to improve every day" as he recovers at a military hospital in Germany, Pentagon spokesman Col. Steve Warren said this week, but he hasn't spoken with his family yet.
The "decision to speak with the family is a decision the returnee has to make when he or she is emotionally in the right place to make the phone call," Warren said.
When Bergdahl is ready, he will be flown to the San Antonio Military Medical Center in Texas, where he may be able to reunite with his parents.
As Bergdahl recovers, details are slowly emerging about the conditions he endured during five years of captivity.
Citing an American official, The New York Times reported Sunday that Bergdahl told medical staff that the box he was kept in for weeks at a time was pitch black and like a shark cage.
CNN reported last week that Bergdahl has said he was kept in a small box after trying to escape, according to a senior U.S. official, who spoke on the condition of not being identified. The official also told CNN that Bergdahl suffers from psychological trauma caused by physical abuse.
CNN's Mariano Castillo, Paul Steinhauser, Catherine Shoichet, Virginia Nicolaidis, Kevin Liptak, Barbara Starr, Elise Labott, Qadir Sediqi, Dana Bash and Deirdre Walsh contributed to this report.
™ & © 2014 Cable News Network, Inc., a Time Warner Company. All rights reserved.