Iraqi-British architect Zaha Hadid's ultramodern design for the stadium -- likened by critics to a giant bike helmet -- has been the subject of fierce debate in Japan, particularly since the cost of the project rose to 250 billion yen ($2.02 billion).
Officials have hinted in recent weeks that the design could be scrapped.
Speaking to reporters outside his official residence Friday, Abe said the decision had been made to seek a new design as soon as possible, with the goal of minimizing costs and constructing the best stadium possible.
"I have made a decision to take the plan back to square one and reconsider," he said.
He cited the skyrocketing costs for the design, which drew criticism from citizens and athletes, as the reason for the rethink.
But a spokesman for Zaha Hadid Architects denied that the design was to blame for cost increases.
Abe said the scrapping of the design meant the stadium would not be ready in time for the 2019 Rugby World Cup, but said he was "certain" that the project would be ready in time for the 2020 Olympics and Paralympics.
A spokesman for World Rugby, organizers of the Rugby World Cup, said the group was "extremely disappointed" by the announcement that the stadium would not be able to host the tournament, which he said came "despite repeated assurances to the contrary."
"World Rugby is urgently seeking further detailed clarification from the Japan Rugby 2019 Organizing Committee and will need to consider the options relating to the impact of today's announcement."
The troubled stadium project has seen budget cuts, design revisions, demolition delays and serious cost blowouts since it was unveiled, as material and construction costs have soared.
Jim Heverin, project director at Zaha Hadid Architects, said the company's teams had worked to design a "new home for Japanese sport for the next 50 to 100 years," using experience gained working on the London 2012 Olympic Games and other big-name sports and cultural projects.
"It is not the case that the recently reported cost increases are due to the design, which uses standard materials and techniques well within the capability of Japanese contractors and meets the budget set by the Japan Sports Council," Heverin insisted.
"The real challenge for the stadium has been agreeing an acceptable construction cost against the backdrop of steep annual increases in construction costs in Tokyo and a fixed deadline," he said.
Japanese architects had been scathing of the design, with leading architect Arata Isozaki writing an open letter to the government body in charge of the games that the sight left him "in despair" and warning the stadium would be a "disgrace to future generations."
Amid concerns over the design's cost, sustainability and its suitability to its surrounds -- in the outer gardens of Tokyo's Meiji Shrine, on the site of the stadium used for the 1964 Olympics -- rival groups have proposed alternatives to the Hadid design.
One group, led by Pritzker Prize-winning architect Fumihiko Maki, who designed the Tokyo gymnasium for the 1964 Olympics, favors altering Hadid's design by removing two of its trademark arches running the length of the stadium. In this way, the group claimed it could reduce the cost of the stadium significantly.
"The biggest problem overall is the cost and the length of construction," the group said in a statement.
Another group has advocated retrofitting the existing stadium for the 1964 Olympics -- a solution advocated by Jeff Kingston, an Asian studies professor at Tokyo's Temple University.
Kingston described Hadid's design to CNN as a multi-billion dollar "white elephant waiting to happen," claiming the design "tramples on Japanese aesthetics."