Just hours after Serena and Venus Williams had suffered a shock defeat in the first round of the women's doubles, world No. 1 Djokovic was beaten 7-6 7-6 by Argentina's Juan Martin del Potro.
Del Potro, who had spent 40 minutes stuck in an elevator in the athletes' village earlier Sunday, went up a level as he claimed a hugely impressive victory.
"Delpo was the better player and he deserved to win. That's sport," Djokovic told reporters.
"In the decisive moments, he just came up with some extraordinary tennis and I have to congratulate him.
"It's very sad and disappointing to go out of the tournament this early but I am glad that a good friend of mine, who has struggled with injuries, has won.
"It's one of the toughest losses in my life and career. The wounds are still fresh."
Djokovic's defeat came on a day where the Williams sisters, reigning champions, suffered their first ever defeat at the Games in 16 matches.
Seeded first at the tournament, the Americans were beaten 6-3 6-4 in 93 minutes.
The sisters, who won the women's doubles title in 2000, 2008 and 2012, were stunned in the first round by the Czech pairing of Lucie Safarova and Barbora Strycova.
"It was a really tough match for us and a really tough draw, but we're used to tough draws," Serena, who is still in the women's singles competition, told reporters.
"We honestly didn't play our best whatsoever tonight. We played terrible and that basically showed in the results.
"I don't know why. I wasn't playing the way I needed to play, I wasn't crossing the way I needed to cross, it was what it was.
"I wouldn't say it's devastating. We have a chance to compete for our country, we did the best that we can.
"It was a lot of fun and we will always remember these moments and these matches and at the end of the day I think that's what matters most."
This defeat capped a couple of miserable days for older sibling Venus, who was dumped out of the singles draw in the first round by Belgium's Kirsten Flipkens.
Earlier Sunday, Serena started the defense of her singles title with a 6-4 6-2 win over Australia's Daria Gavrilova.
She will play Alize Cornet in the second round after the French player overcame Sweden's Johanna Larsson.
No. 2 seed Angelique Kerber of Germany also progressed after defeating Colombia's Mariana Duque-Marino.
In the men's draw, defending champion Andy Murray set up a second-round clash against Argentina's Juan Monaco after cruising to a 6-3 6-2 win over Serbia's Viktor Troicki.
He is joined by 2008 champion Rafael Nadal of Spain, who overcame Argentina's Federico Delbonis in straight sets to book a date with Italy's Andreas Seppi.