Of the fatalities, at least 24 were in the Weiguan Jinlong apartment building in the city of Tainan, according to presidential spokeswoman Ma Wei-kuo.
Dozens of people were still trapped in the building, one of 11 that collapsed after Saturday's magnitude-6.4 quake, the official Central News Agency reported.
President Ma Ying-jeou canceled his traditional Chinese New Year address to oversee rescue and recovery operations at the national disaster headquarters, his spokeswoman said.
Chien, her 3-year-old daughter and her husband were in their bedroom in Tainan --Taiwan's oldest city -- when the earthquake struck.
"I was trapped in a room in a building toppled by the quake," said the mother, who gave only her surname.
"The smell of gas was thick in the air, and I was worried that I would be killed by an explosion if not crushed to death in the collapsed building," she told CNA.
It was a frightening ordeal, one that she has dealt with before.
She lived in central Taiwan before moving to Tainan and survived the 1999 quake
that killed more than 2,000 people.
"I moved to Tainan after I got married and now I have encountered another major earthquake," she told CNA.
'Ring of fire'
Before rescuers freed them, Chien and her family were trapped for three hours in their sixth-floor apartment in the 16-story residential building.
In all, more than 500 people were injured, CNA reported. Ninety-two people remain hospitalized late Sunday, according to Tainan's disaster response office.
As of Sunday morning, at least 121 people were still unaccounted for in Taiwan, CNA reported. A cold wave moving into the area added to rescuers' sense of urgency.
Taiwan is in the so-called "Ring of Fire," an area in the Pacific Ocean where intense tectonic plate movement causes frequent earthquakes.
"Taiwan is very used to earthquakes and tremors, but this is far more significant than the island has seen in quite a while," Elise Hu, an NPR correspondent who was in Taipei when the quake hit, told CNN.
The Weiguan Jinlong building now looks like an accordion from above. Aerial footage from CNN affiliate SETTV
showed the collapsed building, with white smoke or dust still rising.
"The building essentially collapsed onto itself," Hu said. "When you see the aerial images around Tainan, the rest of the buildings are standing. But this particular apartment complex is as damaged as it is."
One woman told CNN affiliate EBC that rescuers had to cut a hole in order to help her family get out.
"Fortunately we were stuck under a space created by a baby crib and a closet door, so that things won't fall on us and air was able to get in," she said from the hospital, where she was being treated for a leg injury. "I was so afraid."
The country's interior minister and other officials said they would conduct an investigation into the building's collapse, according to CNA.
After residents raised concerns about the safety of the building, Tainan Mayor Lai Ching-te said he will order a probe as well.
The quake struck as many in Taiwan prepared to celebrate the Lunar New Year.
It's one of the country's biggest holidays, and some people have as many as nine days off, Hu said.
"If you can imagine something like this happening during Thanksgiving holiday weekend or Christmas travel, that's the equivalent of what's happening here in Taiwan right now," she said.