(CNN) -- "Mr. B" and his family dodged militias by moving from house to house in Baghdad -- but they couldn't escape being Sunni or the fact that Mr. B had served in Saddam Hussein's military.
A boy holds a toy gun Wednesday in front of a bombed building where his family has taken shelter in Baghdad.
Their home eventually was bombed, injuring Mr. B's second youngest son, who now bears a scar from belly to breastbone. Friends and neighbors were kidnapped, some killed. A friend's brother was tortured, his mutilated corpse dumped in the neighborhood, Mr. B told an aid group.
Mr. B, his wife and five children finally fled for Syria in 2006, according to the International Rescue Committee, which issued a report this week detailing the plight of Iraqi refugees on the five-year anniversary of the U.S.-led invasion of Iraq.
"How can we go back and live there after what we have seen?" Mr. B asked.
Like the other refugees interviewed for the report, Mr. B asked that his real name not be used, for fear of retribution. See photos of refugees young and old »
Mr. B and his family now live in a two-room apartment, bare but for two tattered divans, a small TV and thin mats on the floor, the IRC reported. Two of his children have heart problems. Another has diabetes. The son injured in the blast still suffers from trauma.
Mr. B fears letting his children, ages 4 to 12, play outdoors, he told the agency. He is supporting the family on a daily wage of about $4, which he musters by peddling gum and cigarettes on 11- to 12-hour shifts.
The story of Mr. B is hardly anomalous, according to the IRC. The group's report, "Five Years Later, a Hidden Crisis: Report of the IRC Commission on Iraqi Refugees," said more than 4 million Iraqis have been uprooted by the violence that has wracked their nation.
Many fled Iraq after their friends or family members were kidnapped, raped, tortured or murdered, the report said. There are also large numbers suffering from anxiety and depression; others are struggling to find ways to pay for food, basic services and health care.
The report accuses the international community, especially the United States, of ignoring "one of the largest humanitarian crises of our time."
"We believe the United States has a special responsibility to Iraqi refugees, if only to restore its credibility. The violence they flee is an unplanned-for byproduct of the American invasion of Iraq, and its chaotic aftermath," the report said.
The U.N. High Commissioner for Refugees has said the Iraqi refugee crisis is the most significant population displacement in the Middle East since Israel was established in 1948.
The Iraqi refugee population is the third largest in the world, the IRC says, topped only by the Afghan and Palestinian refugee populations. About 2 million Iraqis have fled, mostly to Syria and Jordan, and about 2.5 million more are displaced inside Iraq.
"Neither the U.S. nor the rest of the world is paying sufficient heed: External help provided by regional countries and major international donors has been half-hearted and woefully insufficient," according to an IRC statement.
The report calls for countries -- particularly European nations and the U.S. -- to grant asylum to thousands of refugees and asks that the United Nations hold a conference with government officials and international donors to assess the crisis.
It also suggests that the international community provide billions of dollars in aid, while working to improve conditions in Iraq so refugees can return.
"Contrary to media reports that indicate refugees are repatriating because of improved safety in Iraq, all Iraqis in Syria and Jordan queried by the commission found unimaginable the prospect of returning any time soon to ruined and occupied homes in still-volatile communities," the statement said.
In Amman, Jordan, the group spoke to "Mr. E," a logistician who fled Iraq after militants bombed the company he worked for, killing three staffers. He moved to the Jordanian capital to live in his uncle's house.
Asked by the IRC if he would return to Iraq, he replied, "Why? I'll just be kidnapped and killed. I get death threats on my phone."
The IRC compiled the report after meeting with refugees, aid workers and officials from the United Nations, U.S., Iraq, Jordan and Syria. Syrian President Bashar al-Assad was among the officials interviewed.
The report praises Syria and Jordan for their generosity, but the IRC asks that both nations do more to facilitate aid deliveries and to find employment for the refugees.
The report paints a picture of a refugee population living in fear and poverty.
"The ever-present fear of detention and deportation forces many of them to live hidden from society," the report said. "Worse, many feel they have no future, and that their lives and those of their children are hopeless."
In Amman, a widow who went by "Mrs. T" told the IRC that she left Baghdad after her daughter was sexually assaulted and her husband was tortured and murdered. She now lives in a cold, cramped apartment with her two daughters, their husbands and several grandchildren.
No one in the household is permitted to work legally, and they survive largely on help from aid agencies. But the handouts don't cover all their needs -- Mrs. T recently went blind because she was not able to afford treatment for her high blood pressure, she told the IRC.
The report said there generally are only three options for refugees: return home, remain abroad or resettle elsewhere.
"None of these is a good option. It is still too dangerous for many to go back, they cannot afford to remain as they become increasingly destitute, and yet only a very few will be resettled in other countries," it said.
The report was issued Tuesday, the same day that James Foley, the State Department's senior coordinator for Iraqi refugee issues, said he's confident that the U.S. can hit its goal of letting in 12,000 Iraqi refugees by October.
This fiscal year, the United States has had 1,876 arrivals. That number, however, is about 1,500 higher than reported in early February.
Foley -- who says the U.S. has a moral obligation to help Iraqis -- said the process is "getting faster and will get a lot faster as the year progresses."
Congress has criticized the slow pace of resettling Iraqi refugees. Only 1,608 Iraqi refugees came into the U.S. in the previous fiscal year, the State Department has said. The target was 7,000.
The IRC said the Bush administration's goal of resettling 12,000 Iraqis in the U.S. this year is "extremely meager" compared with those granted asylum from other war-torn regions -- such as Vietnam and the Balkans -- and from religious minorities, such as Russian Jews during the Cold War.
The commission, all of whose members hail from the United States, includes Morton Abramowitz, a former assistant secretary of state and ambassador to Turkey and Thailand; Jean Kennedy Smith, former ambassador to Ireland; and James Wolfensohn, former president of the World Bank. E-mail to a friend