The seven-story building fell Friday in Nairobi as heavy rains inundated the capital.
So far, 135 people have been rescued, according to Nathan Kigotho of Kenya's National Disaster Operation Center.
The building in the residential area of Huruma in northeast Nairobi housed about 164 one-room apartments.
It's unclear how many were occupied at the time of the collapse, Red Cross spokesman Abbas Gullet told CNN affiliate KTN.
An additional 96 remain missing, said Anthony Mwangi of the Red Cross.
Among those pulled from the rubble were two children.
Hours after the collapse, a man emerged holding up a baby wrapped in a pink blanket
as the crowd erupted into cheers.
Another man followed him, carrying a toddler also pulled from the debris. Both children were rushed to waiting ambulances.
As the hours passed, rescuers heard sounds of hope as voices of trapped survivors called out, terrified but alive. Workers yanked chunks of concrete from where the building once stood, the cries spurring their efforts.
Shoes, pots, photo albums
Rescuers frantically scoured through the debris overnight to find survivors from the building.
"Don't cry, we're coming!" one man was heard yelling in Swahili, in video from the scene. "Let's move faster," one person implored others involved in the rescue effort.
Household items lay scattered as residents prayed and wept. Others carried debris using their bare hands.
"It's heartbreaking ... amongst the rubble you've got mattresses strewn around, children's shoes, pots and pans, photo albums," said Arnolda Shiundu, a spokeswoman for the Red Cross.
"Every time we move some rubble and we hear voices," she said, "it gives the multiagencies that are here the morale to keep on fighting so that we can get as many people -- as many survivors as possible."
Kenya's defense forces have taken control of the scene with help from the Kenya Police and various medical agencies.
It's unclear what caused the building to fail during the storm, but residents told the Red Cross the bottom floors caved in first.
Kenya Police spokesman Charles Owino said it was too early to give definitive answers, but he expected preliminary information soon.
"It's not a new building; it's an old building, so hopefully the planning department is able to tell us," he said.
"We don't how how much the heavy rains could have caused the building to fall. There could be other technical reasons."
Structure had been condemned
Kenyan President Uhuru Kenyatta visited the scene Saturday morning. He asked the country's disaster authorities to survey other houses in the area, determine if any were likely to collapse and move residents out, the National Police Service said.
He also ordered the owners of any unstable structures be arrested after it emerged the building that collapsed had previously been condemned by the National Construction Authority, the police statement said.