The fire started at 3:15 a.m. at the Puttingal temple in Kollam, in Kerala state .
Indian Prime Narendra Modi called the fire "heart-rending and shocking beyond words."
"My thoughts are with families of the deceased and prayers with the injured," he said in a statement.
Sparks from the display ignited a magazine of fireworks, causing a bigger blaze, said K. Laljy, a local police official.
The temple did not have permission for the fireworks celebration, according to M.S. Santosh, another police official.
Laljy said the fire was out and emergency personnel were at the scene.
The worshippers were celebrating a local temple festival, according to P. Prakash, the Kollam police commissioner. It's not uncommon for people to pray at temples early in the morning in southern India.
Rescue operations were underway, though it was unclear whether anyone was still inside the temple.
Modi said he plans to visit the site of the fire.
"I will be reaching Kerala soon to take stock of the situation arising due to the unfortunate fire tragedy," he tweeted.
His office said the next of kin of those killed will get about $3,000 in "ex-gratia relief." Those injured in the temple fire will receive 50,000 rupees ( $751) in compensation.
The temple is more than 100 years old. It's named after a goddess who people believed lived inside an ant hill.
India has seen a handful of lethal fire breakouts in recent years.
an explosion in central India caused by mining explosives killed nearly 90 people. And nine people were killed in a 2014 train fire.
Another blaze in February
shut down a performance in Mumbai. No casualties were reported.