Police say they tried to stave off the mob with tear gas and opened fire, killing one person.
The crowd targeted a 35-year-old suspect, identified as an undocumented Bangladeshi settler, who died of his injuries before officers could rescue him, said L.L. Doungal, the director-general of police in the remote Nagaland state, where the attack took place.
The mob that broke into the main jail in the state's Dimapur district was comprised of "thousands" of people, Doungal told CNN. "Many of them were students in uniform," he said.
The suspect was accused of raping a local woman on February 23.
Tensions between the native Naga and undocumented Bangladeshi migrants in the district have been simmering for some time, police said.
On Thursday evening, thousands of people who were demanding the cancellation of trade permits for Bengali-speaking settlers, tore into the prison complex and pulled the suspect out.
Police initially used tear gas and bamboo canes to rescue the inmate, but were not successful, Doungal said.
"There were students in uniform. So, we had to use minimum force," he said.
The crowd planned to lynch the suspect, but he died from critical injuries suffered in the attack.
"We retrieved his body before it could be (hanged)," Doungal said.
Dimapur town has been placed under curfew, according to the officer, as the situation remains volatile.