The 35-year-old suspect, who was described as an illegal immigrant from Bangladesh, died from severe injuries before officers could rescue him, said L.L. Doungal, a police official in the remote Nagaland state, where the attack occurred Thursday evening.
"There were thousands of them. Many of them were students in uniform," Doungal said, referring to the mobs involved in the incident.
The suspect was accused of raping a local woman last month.
Since the attack, tensions have grown between the native Nagas and Bangladeshi migrants in the district, police said.
Thousands of protesters demanding cancellation of trade permits for Bengali-speaking settlers tore into the prison complex Thursday and pulled out the suspect, Doungal said.
Police initially used teargas and bamboo canes to rescue the man, but it didn't work, he said.
"There were students in uniform. So, we had to use minimum force," Doungal added.
Later, police opened fire, but it was too late to save the suspect.
Protesters had planned a public hanging, but the suspect died from injuries in the attack, according to authorities.
"We retrieved his body before it could be hung," Doungal said.
Dimapur town has been placed under a curfew. "The situation is still volatile," he said.
India made international headlines this month
when it banned a documentary showing an interview with a convicted rapist
, who blamed his victim and said she "should just be silent and allow the rape."
Outrage has grown following the incident, which gives a window into the violence against women in the nation.
The woman in the documentary incident was attacked by five men on a public bus in 2012. She later died from her injuries, sparking an outcry worldwide.
New Delhi police said the interview was banned because of its potential to breed disorder.