Karne Hesketh crossed out wide in the fourth minute of time added on to clinch Japan's second victory in tournament history, and first since beating fellow minnow Zimbabwe in 1991.
"The whole crowd was behind Japan. When South Africa gave a penalty and we decided to go for it, that's when we finally got their respect," Hesketh, who scored with his first touch of the game, said after the final whistle.
"You can't imagine something like that because it's so special. It was amazing."
The Springboks were expected to have few problems against a team ranked 13th in the world as they began their bid to become the first nation to win the title on three occasions.
But the 1995 and 2007 champions could not overcome the tenacious "Brave Blossoms," despite scoring four tries to three. It was the most points the Boks had ever conceded at the tournament.
Third-ranked South Africa led 12-10 at halftime, but each time Japan -- which had conceded an average of 48 points in its previous RWC matches -- kept fighting back.
Fullback Ayumu Goromaru kicked five penalties and two conversions, also scoring a superb try as he totaled 24 points.
Japan's brave attacking tactics paid off at the death, choosing to take a five-meter scrum instead of a simple penalty kick that would have ensured a draw.
"I could tell they were panicking," Japan skipper Michael Leitch said. "We wanted to scrum rather than take the kick because they were down a man and I personally wanted to go for a win rather than a draw. I didn't want to let the boys down."
That decision came after the video ref ruled that Japan's forwards had not managed to ground the ball over the try line, as the Boks held on desperately.
Pummeling the South African defense with the seconds ticking down, knowing that one mistake would bring the final whistle, the Japanese finally created space for New Zealand-born Hesketh to slide over the try line in the left corner.
It was a notable victory for Japan coach Eddie Jones, who was an assistant with the Springboks team that won the 2007 title, and who guided his native Australia to the 2003 final but lost to England.
"It was amazing at the end, I think even the Springboks supporters were backing us," he said.
"The crowd, the noise was unbelievable. That has to go down as one of the greatest games in World Cup history."
Japan, the 2019 World Cup host, will now have hopes of being one of the two teams from the group to qualify for the quarterfinals. It next faces Scotland on Wednesday, then Samoa on October 3 and the United States eight days later.
South Africa's only previous World Cup defeats had come against fellow two-time champions New Zealand and Australia, and England.
In Saturday's other results, Georgia also caused an upset by beating Tonga 17-10 for the 16th-ranked country's third victory in tournament history.
The Pacific Islanders had two tries disallowed, while their European opponents played the last eight minutes with only 14 men in Gloucester after fullback Merab Kvirikashvili was sin-binned.
Georgia's scrumhalf Vasil Lobzhanidze became the youngest player in World Cup history, aged 18 years and 340 days when he took the field at Kingsholm Stadium.
Six Nations champion Ireland defeated Canada 50-7 in the Pool D opener in Cardiff, Wales.
Three of Ireland's seven tries came while Canada captain Jamie Cudmore was in the sin-bin, with the score 29-0 at halftime.
France, beaten in the 2011 final by New Zealand, began its Pool D campaign with a 32-10 win against Italy at Twickenham.
Frederic Michalak converted both of Les Bleus' tries and kicked five penalties for 19-point haul, but winger Yoann Huget was carried off with an injury that is expected to end his tournament.