Hamilton took first place on the front row on the last lap, beating Rosberg by a slim four hundredths of a second margin.
The two former friends have enjoyed, or rather endured, a heated rivalry since falling out last season and Rosberg's annoyance at Hamilton's last ditch success was obvious.
The German appeared upset as he left his car and refused to shake Hamilton's hand.
He did, however, find time to congratulate fellow German driver Sebastian Vettel, who will start in third after the Ferrari driver surprisingly won the Malaysian GP two weeks ago.
"I don't actually mind him having more (poles)," a nonplussed Hamilton told AFP after the session.
"He can have the trophy for most poles. As long as I have the trophy for winning -- that's all that matters. Ultimately qualifying is not the end of the world."
"I am frustrated"
In the post qualifying press conference Rosberg appeared more magnanimous, but still shocked by Hamilton's last minute burst of speed.
"I was frustrated, I am frustrated, yeah," Rosberg admitted.
"It's so close and of course I would have wanted to have pole today... It's just four hundredths and that makes it even more annoying because it's just very, very close. A lap is never 100 per cent perfect and four hundredths is really the blink of an eye."
Kimi Raikkonen will join his teammate Vettel on the second row and will be looking for a repeat of Ferrari's performance in Malaysia, which shocked Mercedes and Hamilton in particular.
But Hamilton has been quickest all weekend and will enjoy a surprising amount of support in Shanghai, a track the British driver has always thrived on.
"Have you seen the banners that I have here?" he told the assembled press when asked why he liked racing at Shanghai so much.
"The support I have here's pretty unreal. I just like the track. I guess it just naturally suits my driving style."