Thirty-eight people have so far been confirmed dead from the disaster, in which the ship tipped over minutes after leaving port, according to the Philippine Coast Guard.
There were 173 passengers and 14 crew members on board the ship, the coast guard said. A total of 134 people have been rescued, it said Friday.
The coast guard is leading the search effort, with divers from the Red Cross and the military also involved.
Three Americans are among the survivors. They are Rhome and Chip Nuttall and Larry Drake, according to Philippine Red Cross Chairman Richard Gordon, who spoke to CNN from Manila.
The ferry, the MB Nirvana, had just departed Ormoc City in Leyte province bound for the town of Pilar on Camotes Island, about noon local time Thursday when the disaster occurred.
The boat, which is 27 meters (89 feet) long, was barely 200 meters (more than 650 feet) from the shore when it capsized, Philippine Coast Guard Lt. Christopher Ganet said.
High winds, seas hamper efforts
Video footage from the scene showed people of all ages, including a small child, being brought ashore in small inflatable craft. Some survivors were laid on the concrete waterfront, while others were wheeled away on stretchers.
Gordon said Red Cross staff members were on the pier in Ormoc with blankets and meals to assist those who made it back to shore.
Poor weather conditions hampered the initial rescue effort on Thursday.
"The winds are really bad, and the seas are really bad," Gordon said.
Investigators will examine what caused the boat to capsize.
A CNN Philippines reporter quoted the coast guard as saying the boat had left port too quickly and people stood up, throwing the boat off balance. Gordon said he understood the boat overturned because of high winds.
Gordon: Passengers were 'poor folks, simple folks'
Gwendolyn Pang, secretary-general of the Philippine Red Cross, described the vessel as a commercial "pump boat" or banka, which is essentially a canoe-style craft with outriggers powered by a small engine. They're widely used across the Philippines for transporting people and goods as well as for fishing.
Gordon said the boat would operate three times a day on the route from Ormoc to Camotes Island.
"The passengers would have been farmers or fishermen, or ordinary businessmen -- common folks," he said.
"They are not very well off, otherwise they would be on better vessels or take the plane. But obviously, these are poor folks, simple folks who are trying to eke out an existence."
This style of outrigger boat has no cabins, he said, which should make it easier for divers to find anyone trapped under the canopy or keel of the vessel.
Gordon said Red Cross staff members at the scene were checking who was on board and how many of them were children. Officials have also brought body bags in case they are needed, he said.
Many may not even be able to afford a proper burial, he added.
American survivor Rhome Nuttall told CNN that she was on the boat because she helps to run a medical mission in the area.
"There was a little wave, which is normal, but yesterday that caused the boat to flip to the right," she said. "I ran to the left hoping we could help to balance the boat. When we flipped, I said: 'OK, this is it. OK, this is it.'"
Water came in and washed her out of the boat. Nuttall said she couldn't find her husband for about five minutes, and that everyone panicked.
She remembered hearing a thud as the boat backed out. The vessel was loaded with people and cargo.
"I think it was that big thud. I think that was cargo that shifted and that caused the boat to flip," Nuttall said.
Hospitals take in survivors, bodies
Dr. Rolando Tomaro, medical director of Ormoc Doctors' Hospital, told CNN that four hospitals in the Ormoc area were receiving people caught up in the disaster.
The dead included an 8-month-old child and his 13-year-old brother, he said.
Ten survivors in the hospital were under observation, he said, but none of them were in a critical condition.
"We have to monitor them for signs of respiratory distress because of the near-drowning incident," he said. "They would have ingested seawater, so we are monitoring them. As of now, they are all stable."
Tomaro, who has been a surgeon at the hospital for 20 years, said it was the first time anything like this had occurred.