Myanmar, now ruled by a military junta that seized power in 2021, had argued that Gambia, which brought the suit, had no standing to do so at the top UN court, formally known as the International Court of Justice (ICJ).
But presiding Judge Joan Donoghue said all states that had signed the 1948 Genocide Convention could and must act to prevent genocide, and the court had jurisdiction in the case.
"Gambia, as a state party to the genocide convention, has standing," she said, reading a summary of the 13-judge panel's ruling.
The court will now proceed to hearing the merits of the case, a process that will take years.
Gambia took up the Rohingya's cause in 2019, backed by the 57-nation Organisation for Islamic Cooperation, in a suit aiming to hold Myanmar accountable and prevent further bloodshed.
Gambia Justice Minister Dawda Jallow said outside the courtroom he was "very happy" with the decision and was confident the suit would prevail.
Gambia became involved after his predecessor, Abubacarr Tambadou, a former prosecutor at the UN Rwanda tribunal, visited a refugee camp in Bangladesh and said that the stories he heard evoked memories of the genocide in Rwanda.
A representative for Myanmar said the state would do its "utmost" to protect the country's "national interest" in further proceedings.
Protesters outside the court's gates hoisted a red banner with the text "Free Burma" and yelled at cars carrying the junta's representatives leaving the building after the decision.
A UN fact-finding mission concluded that a 2017 military campaign by Myanmar that drove 730,000 Rohingya into neighboring Bangladesh had included "genocidal acts."
Myanmar has denied genocide, rejecting the UN findings as "biased and flawed." It says its crackdown was aimed at Rohingya rebels who had carried out attacks.
While the Hague court's decisions are binding and countries generally follow them, it has no way of enforcing them.
In a 2020 provisional decision it ordered Myanmar to protect the Rohingya from harm, a legal victory that established their right under international law as a protected minority.
However Rohingya groups and rights activists say there has been no meaningful attempt to end their systemic persecution.
Rohingya are still denied citizenship and freedom of movement in Myanmar. Tens of thousands have now been confined to squalid displacement camps for a decade.
Bangladesh's Foreign Ministry welcomed the judgment in a statement.
"For the victims living in the camps in Bangladesh as well as in Myanmar, they see the hope that justice will be delivered to them and that the perpetrators in the Myanmar military will be brought to accountability," said Ambia Parveen of the European Rohingya Council outside the court.
The junta has imprisoned democratic leader Aung San Suu Kyi, who defended Myanmar personally in 2019 hearings in The Hague.