Peter Greste, Mohamed Fadel Fahmy and Baher Mohamed were convicted in June of supporting the banned Muslim Brotherhood. Greste and Fahmy were sentenced to seven years in prison and Mohamed to 10 years.
TV cameras were not allowed in Thursday's hearing in the Court of Cassation, and the incarcerated journalists were also absent from the courtroom.
The men's families were disappointed at the outcome. They had hoped the court would throw out the case and release them or at least free them on bail pending retrial, but they remain behind bars. The retrial date has not yet been set.
"This was not good as we hoped for," Lois Greste, mother of Australian journalist Peter Greste, told reporters at the court.
'I believe in his innocence'
Lawyers for the three defendants asked the court for their release but told reporters that the chances were slim. Fahmy's lawyers asked for his deportation to Canada -- he is Egyptian-Canadian -- taking advantage of a new law allowing a president to deport defendants.
The judge overseeing the retrial will decide on their release, said Amr El-Deeb, a lawyer for Peter Greste and Mohamed.
Greste's brothers, Mike and Andrew Greste, expressed mixed feelings about the court's decision.
"It's a step in the right direction. Obviously, it's a little disappointing that we didn't get the most positive outcome, but on reflection, it's certainly the next best result for us," Mike Greste told CNN affiliate Seven Network Australia.
But Greste's family hopes Egypt's government will make use of other legal avenues to free his brother Peter sooner.
"Following instruction from Peter and the family, Peter's Egyptian lawyer lodged an application with the Egyptian Attorney General's office to have Peter deported, pursuant to a presidential decree that was issued in November," Andrew Greste said.
"We believe there is now a window of opportunity for the Egyptian president to exercise his presidential power under the decree," he Greste said.
Fahmy's fiancee, Marwa Omara, said they planned to marry despite his incarceration.
"We will hold our wedding vows inside the prison," she said. "I believe it's a message to the whole world that Mohamed is innocent, and I believe in his innocence. And it is a personal message to him that I love him and I know that he is innocent."
Fahmy's brother, Adel Fahmy, said the whole family had shared in the journalist's ordeal.
"It's the epitome of injustice what happened," he said. "He lost a year and more now of his life. We are suffering as a family severely for my brother, as all the families are for their loved ones. And I can't describe in words what we've been through this year."
'Unjustly in jail'
In a statement issued after the ruling, Al Jazeera urged Egyptian authorities to move swiftly to a retrial. Lawyers point out that the appeal process from this point could take another 12 to 18 months, the broadcaster said.
"Baher, Peter and Mohamed have been unjustly in jail for over a year now," an Al Jazeera spokesman said.
"The Egyptian authorities have a simple choice -- free these men quickly, or continue to string this out, all the while continuing this injustice and harming the image of their own country in the eyes of the world. They should choose the former."
There has been wide international condemnation of Egypt's prosecution of the three journalists, who deny wrongdoing.
The United States, Australia and European Union have all called for the men to be freed, as have rights groups including Amnesty International, the Committee to Protect Journalists and the International Press Institute.
The appeal was accepted because of procedural errors in the trial that led to the conviction. The men had appeared in a special court created to handle terrorism cases, which may, in retrospect, be considered illegal.
Prosecutors are now expected to present the same case in a new court.
The three journalists have been imprisoned in Cairo since December 2013 on charges that included conspiring with the Muslim Brotherhood, spreading false news and endangering national security.
They have steadfastly denied the charges, as has Al Jazeera, and the prosecution put forward no evidence to support the allegations.
Prosecutors were criticized for presenting evidence that was purportedly made up or irrelevant to the case, like videos of Greste's old news reports from other countries.
Amnesty International has said that Greste, Fahmy and Mohamed are pawns in a geopolitical dispute between Egypt
and Qatar, the tiny but oil-rich Middle Eastern country that finances Al Jazeera.
Qatar has long been perceived as a supporter of the Muslim Brotherhood, which was declared a terrorist organization just days before the Al Jazeera journalists were arrested at the Cairo hotel where they were working.
With Saudi mediation, however, there has been a recent improvement in Egyptian-Qatari relations.
President Abdel Fattah el-Sisi
met with a Qatari envoy in December, and the meeting was followed by Al Jazeera's decision to suspend its Egyptian Mubasher Masr channel.
This channel is already banned by an Egyptian court and was at the center of the journalists' trial, even though they work for the network's English-language channel.
Omara, Fahmy's fiancee, welcomed the apparent improvement in the two countries' relations.
"I hope the reconciliation between Egypt and Qatar can lead to a better outcome," she said. "It's very hard for us to feel that they will stay for a year in prison. It's very tough. They did nothing."