The Australian, 35, has long been lukewarm on the event and has now joined Fiji's Vijay Singh in opting out of this summer's showpiece.
"My decision has been taken as a result of an extremely busy playing schedule around the time of the Olympics and other commitments, both personal and professional," the world No. 7 announced in a statement from his company's base in the Bahamas.
"I have informed the Australian team captain and relevant authorities, who are understanding of my position, and I wish the Australian Olympic team the very best of luck in Rio."
Scott, who has won twice on the PGA Tour
this season, became the first Australian to win the Masters when he triumphed at Augusta in 2013 for his sole major title.
Last year he told Reuters the Olympic golf tournament was an "exhibition event" and said he preferred to concentrate on winning majors.
Golf Australia chief executive Stephen Pitt said: "We're obviously disappointed Adam will not play because he's one of the best players in the world but we understand his position."
In Scott's absence, world No. 1 Jason Day will lead Australia's two-man team, captained by Ian Baker-Finch, with Marc Leishman the next highest-ranked player at 34 in the world and Marcus Fraser at 66th.
The Olympic golf competition features men's and women's 72-hole strokeplay.
A total of 60 players will compete in each event, with qualification based on world ranking as of July 11, 2016.
The top 15 players of each gender will qualify up to a limit of four players per country. The remaining spots will go the highest-ranked players from countries that do not already have two golfers qualified.
The tournament takes place on a new Gil Hanse-designed course in Rio's western suburb of Barra da Tijuca. The venue has been hampered by controversy over land ownership and environmental issues but a test event held last month drew favorable reactions.
"I thought it was going to be way too hard and not fair but it's so interesting because you do get rewarded for good shots," Brazil's Ladies European Tour player Victoria Lovelady told CNN's Living Golf.
"The greens were extremely receptive and amazing to play on. It was really impressive how good everything was."
Singh, a three-time major winner, announced last week he would not compete in Rio because of fears over the Zika virus.
Northern Ireland's Rory McIlroy ended speculation in 2014 when he announced he would compete for Ireland instead of Great Britain.
The global golf calendar has been rejigged to accommodate the Games, with the year's final major, the U.S. PGA brought forward from its traditional August spot to take place two weeks after the British Open in July.