Seven-time grand slam tennis champion Maria Bueno believes there are other underlying reasons for their decision not to compete in Brazil.
"I think that the people who are pulling out because of the Zika virus must have other reasons," Bueno told CNN.
"Zika has become a very good excuse not to go to Rio but I don't think it's really that bad."
The world's top four golfers, Jason Day, Dustin Johnson, Jordan Spieth and Rory McIlroy
have all withdrawn -- as have a number of other high-profile stars.
Reigning Olympic long-jump champion Greg Rutherford has taken the precaution of freezing his sperm
as has U.S. volleyball coach John Speraw.
Brazilian officials have argued the risk of infection is seriously reduced during the Games, because they are being held in the South American winter month of August, when mosquito populations tend to die.
"It has gotten a lot of bad publicity but we are trying to do everything we can to prevent it, to keep the athletes safe, the tourists and everybody else who is going," Bueno said.
"We are pretty confident that it will go well, it will give the chance for the country to be united for something that means a lot to everybody."
It's not just golfers who have pulled out either.
Earlier this month, Canada's Milos Raonic
and Czech star Tomas Berdych both withdrew
from the Games after citing fears over Zika.
Several other big names such as French Open semifinalist Dominic Thiem, American world No. 16 John Isner and Spain's No. 20 Feliciano Lopez have all opted to to play in ATP Tour events instead as these tournaments offer ranking points -- while the Games do not.
Nick Kyrgios ruled himself out after a disagreement with the Australian Olympic committee while fellow Australian Bernard Tomic -- who was also criticized by the country's sporting authority -- has decided to play in a Mexican tournament instead.
Maria Bueno Stadium
That means plenty of players will miss out on playing at Rio's brand new state of the art Olympic Park which houses the venue's 10,000 capacity tennis court
-- named after Bueno.
Overall, there are 16 courts, 10 for competition, including two other show courts which have temporary seating of 5,000 and 3,000 respectively.
"I think it's fair to say that we've a lot of things to be concerned about today and Zika virus will certainly be one of them," 1996 gold medal winner Andre Agassi told CNN.
"The good news is it's winter down in Brazil so I don't think we have the same sort of potential devastation that would happen if this kind of tourism came down in summer time.
"Mosquitoes aren't quite as active by a long shot so that's good news. Everybody has a decision to make. You know everything comes with risk.
"Unfortunately the penalty of the risk can follow with a rough time in an athlete's life as they look to their future. I don't know how I would have handled it.
"You can't judge everybody else for how they do but I certainly understand the fear."
While most of the world's golfers are absent, many of tennis' big names will be going for gold in Brazil.
Serena Williams who won the ladies gold in 2012 and Andy Murray who won the men's event in London, are both set to compete in Rio.
'Something you can't pay for'
"The Olympics is something that everybody should play at least once, " former Wimbledon champion Goran Ivanisevic told CNN.
Ivanisevic, who won bronze in the singles and doubles at Barcelona in 1992, recalls his appearance at the Games fondly.
As flag bearer for Croatia in the country's first ever Games, he says it was something he will never forget.
"It's a great experience," he added. "I was in four Olympics. You represent your country and it's something that you're proud of.
"I know it's a busy schedule. I know it's not the most important thing in the sport but getting a medal, being there with other sportsmen, staying in the village, that's something that you can't pay for."
Seven grand slams
Tennis returned to the Seoul Games in 1988 after a 60 year absence.
Its return came too late for Bueno who was forced to retire through injury as the Open era started to take shape.
On top of her three singles titles at Wimbledon, Bueno won four U.S Opens and reached the finals of the French and Australian.
She also won 11 doubles titles and one mixed doubles title during an illustrious career.
For a girl from Sao Paulo who left home at 15 to chase a dream, her achievements remain astonishing.
"Things were very different to the way they are now," she said. "Just to give you an idea, the prize money was £15 ($20) for winning Wimbledon.
"Things didn't come easy and you had to face one difficulty after another every single day. That really prepared me for life."
Putting Brazil on the map
For thousands of youngsters, the name of Maria Bueno may not have been one they were familiar with before the construction of the new stadium.
Now 76, she is hoping to inspire another generation when the Games get underway in Rio.
"It will change a lot because everyone wants to know how the name got there," she said of the Maria Bueno stadium.
"It will make people think more about the history, about the fact that a few years ago someone put Brazil on the map.
"Being a woman that is an especially great thing. In those days, it meant something to achieve something like I did as a woman in Brazil.
"It will be good for children. I think TV has been helping because people will associate the name to the face, so there is a good chance that this will happen."