Grindeanu said he will call a Sunday Cabinet meeting to scrap the measure, which he passed Tuesday without the vote of Parliament.
"I don't want to divide Romania," Grindeanu said in a televised address as crowds protested across the country, including in Bucharest's Victory Square just outside the government headquarters.
"Romania cannot be torn apart. Romania, in this moment, due to what's going on seems to be divided, but it's my last wish to get to anything like that," he said.
The decree would have decriminalized corruption that causes damage worth less than about $48,000 US (200,000 Romanian lei.)
It also would have halted all investigations for pending corruption offenses, prevented further cases related to these offenses to be brought into justice and freed some officials imprisoned for corruption.
Grindeanu said Saturday there were some problems in communicating the decree to the public. He said the justice minister, who championed the law, had taken full responsibility.
The news didn't calm the crowds. Massive rallies were expected again Sunday in Bucharest and other cities.
The sometimes-violent street protests stretched across the nation and were some of the largest in Romania since communism fell in 1989.
Before Grindeanu's announcement, the law was set to take effect in a little over a week.
The decree would have benefited some politicians, such as Liviu Dragnea -- president of the Social Democrat Party, which recently took power. Dragnea is under investigation over abuse of power allegations and had also previously received a two-year suspended sentence for an elections offense.
Concern over decree
European Commission President Jean-Claude Juncker expressed concern over the new law this week.
"The fight against corruption needs to be advanced, not undone," he said "We are following the latest developments in Romania with great concern."
The embassies of Belgium, Canada, France, Germany, the Netherlands and the United States issued a joint statement echoing Juncker's sentiments.