Ivanovic ended the American's 25-match winning streak to reach her first grand slam quarter-final since the 2012 U.S. Open with an inspired display of clean hitting.
"It's amazing. I just tried to play the best I could," a thrilled Ivanovic said in a courtside interview.
"You guys have been amazing. I'm so thrilled," she told the crowd.
Ivanovic, a former world number one, had never taken a set off Williams in their four previous encounters.
A winner at Roland Garros at 2008, she was aggressive from the outset, breaking Williams at 2-2 in each of the first two sets and holding firm in the second to draw level.
A shell-shocked Williams was then broken at 1-0 in the deciding set when the Serbian laced a brilliant return down the line.
With the Rod Laver Arena crowd roaring their support for the underdog, Ivanovic coolly raised three match points and sealed the tournament's biggest upset when Williams missed a return.
Ivanovic will play the winner of Casey Dellacqua and Eugenie Bouchard for a place in the semi-finals.
"It was not easy playing such a champion and I had never beaten her before, but she is just a human and I know that she has pressure on her too," she added.
"So I just went out there and was swinging at the ball and I did that really well."