The case was brought by 409 relatives of those killed or injured and survivors of the attack, in which more than 1,000 people were held hostage for more than two days in a school gymnasium rigged with explosives by heavily armed Chechen separatists.
Survivors and relatives of those caught up in the bloody attack have long called for answers about what happened and whether the security forces' armed response led to unnecessary deaths among the hostages.
Kremlin spokesman Dmitry Peskov said the court's judgment against a country which had suffered a terror attack was "unacceptable" and overly emotional.
In its ruling, the court said the Russian authorities had not done enough to keep the school safe.
"The authorities had been in possession of sufficiently specific information of a planned terrorist attack in the area, linked to an educational institution. Nevertheless, not enough had been done to disrupt the terrorists meeting and preparing; insufficient steps had been taken to prevent them travelling on the day of the attack; security at the school had not been increased; and neither the school nor the public had been warned of the threat," a press release on the judgment said.
The court also identified "serious shortcomings" in the investigation into the attack by the Russian authorities.
There had been no proper examination of how the victims had died, it said. Russian authorities did not carry out a full forensic examination of the majority of victims and failed to properly record the location of most of the hostages' bodies.
In addition, other evidence at the scene was not properly secured and recorded, the court said, causing "irreparable harm to subsequent analysis of the event."
The court also criticized "serious shortcomings in the planning and control of the security operation" by Russian authorities responding to the hostage situation.
"In the absence of proper legal rules, powerful weapons such as tank cannon, grenade launchers and flame-throwers had been used on the school. This had contributed to the casualties among the hostages," it said.
A number of domestic investigations have been held in the years since the attack on Beslan's Middle School Number One and, in 2006, the only surviving hostage-taker was sentenced to life in prison.
But the court ruled that the Russian authorities had breached the applicants' rights, because "the investigation had not been 'effective', as it had not been capable of leading to a determination of whether the force used by the State had been justified in the circumstances. It also noted that the public scrutiny aspect of the investigation had been breached by the victims' restricted access to it."
The court ordered Russia to pay the 409 applicants a total of 2.9 million euros (almost $3.1 million) in compensation.
The ECHR is not a European Union body. It was established in 1959 to rule on violations set out in the European Convention on Human Rights, of which Russia is a signatory. However in late 2015 the Russian parliament passed a law allowing it to overrule judgments for the court and allowing the Russian constitution to take precedence.
Asked if the Russian government would abide by the court's decision, Peskov said: "All the necessary legal actions about this decision will take place."