McIlroy led Koepka by a stroke going into the final round but instead of an epic duel with the world No.1 he faltered with a 71 as the American surged past with a 65 to win the WGC-FedEx St Jude Invitational by three from countryman American Webb Simpson.
The 30-year-old McIlroy, who had gone clear after a blistering 62 Saturday, ended in a tie for fourth, five strokes adrift.
McIlroy missed the cut at the Open in his native Northern Ireland last week after a disastrous opening 79, followed by a scintillating second-round 65, willed on by the vocal home fans as he tried in vain to make the weekend.
The four-time major winner has climbed back up to world No.3 in a solid season in which he has won the Players Championship and the Canadian Open but he has also has failed to break 70 seven times in his 13 PGA Tour final rounds this season.
For Koepka, 29, it was payback for his tied fourth at the Open in a stunning season in which he was also second to Tiger Woods at the Masters, won the PGA Championship and was second at the US Open.
The powerful Floridian, who has won four of his last 10 majors, pulled alongside McIlroy with a birdie at the third and then took control with birdies at the fifth and sixth as his playing partner lost ground.
"It's incredible. To look at what I've done this year, just show consistency, try to take my game to a new level and I've done that," Koepka said, who leads the season-long FedEx Cup standings.
"To win this title against Rory was pretty special."