McIlroy added that he may not even tune in to watch golfers in action at the Games.
His comments came after Jordan Spieth announced his decision not to play at Rio
Monday -- a decision meaning none of the world's top four will be part of the sport's return to the Summer Games for the first time since 1904.
Speaking at a news conference before this week's Open Championship in Scotland, where the game's leading players have assembled for the third major tournament of the golf season, McIlroy said: "I don't feel like I've let the game down at all.
"I didn't get into golf to try and grow the game -- I got into golf to win championships and win major championships.
"I'll probably watch the Olympics, but I'm not sure golf will be one of the events I watch. Probably the events like track and field, swimming, diving -- the stuff that matters."
Told that Spieth had described his decision not to play in Rio as the most difficult he had ever made, McIlroy responded: "Honestly, I don't think it was as difficult a decision for me as it was for him.
"I get where different people come from, and different people have different opinions. But I'm very happy with the decision that I've made, and I have no regrets about it."
But International Olympic Committee (IOC) president Thomas Bach told German news agencies SID and DPA the withdrawals did not help "the attractiveness of the golf competition."
He said: "These are individual decisions taken by the players that stand in contrast to World Health Organisation recommendations.
"The IOC, of course, has to respect the decisions of the athletes. But it is obvious that this does not help the attractiveness of the golf competition.
"You can see that these are individual decisions just by looking at Martin Kaymer, who recently said that the Olympic golf tournament is a top priority for him this year."
In a statement sent to CNN, the IOC said: "The WHO -- which is the authority in this case -- has once again reaffirmed its advice that there should be no general restrictions on travel to Rio de Janeiro because of the Zika virus and has issued advice on a number of sensible measures that all visitors should take."
Two-time major winner Spieth, the world No. 3, had said last month he was "uncertain" about taking part amid Zika concerns.
While the symptoms of the virus -- which include a rash, headaches and joint pain -- are not severe, it has been linked to microcephaly in newborn babies and some cases of the muscle-weakening disease Guillain-Barré syndrome in adults.
Last month, Brazil's health minister told tourists and athletes that the risk of catching Zika in Rio was "almost zero."
However, that did not assuage the concerns of the world's leading golfers, many of whom have young families or are planning to start them.
The long line of withdrawals has raised questions about whether golf should be an Olympic sport.
Tiger Woods, arguably still golf's greatest draw despite the long-term back problems that have meant he cannot take part at Rio, said the Games "deserved" to have the best players involved.
But world No. 1 Jason Day said last month he would not compete in Rio.
In a statement, the Australian explained: "The reason for my decision is my concerns about the possible transmission of the Zika virus and the potential risks it might present to my wife's future pregnancies and to future members of our family.
"While it has always been a major goal to compete in the Olympics on behalf of my country, playing golf cannot take precedent over the safety of our family. I will not place them at risk."
Second-ranked Dustin Johnson, the U.S. Open champion, did likewise Friday, while other high-profile golfers to withdraw from the Rio Games include Adam Scott and Marc Leishman of Australia, South Africans Branden Grace, Louis Oosthuizen and Charl Schwartzel and Fiji's former world No. 1 Vijay Singh.