Widely billed as the group favourites and dark horse to lift the trophy, Belgium had to toil hard to erase the 0-1 deficit after Algeria had taken the lead in the 25th minute against the run of play through a penalty kick conversion by Sofiane Feghouli.
Belgium, whose most of players ply trade in top European clubs, like Chelsea star Eden Hazard and Manchester City defender Vincent Kompany, turned the match on its head with two goals in the space of 10 minutes.
Manchester United's Marouane Fellaini equalised in the 70th minute through a glancing header off Kevin De Bruyne's fine cross from the left and with momentum on Belgium's side, substitute Dries Mertens made a fierce finish with a low shot for the winning goal in the 80th minute.
Today's win was the first World Cup victory for Belgium after defeating Russia 3-2 in a group match in Shizuoka, Japan in 2002 and they richly deserved it as they had the overwhelming 65 per cent possession and dictated terms on their opponents except for conceding a surprise goal.
To the credit of Algeria, who were playing in their fourth World Cup, they frustrated a quality European side, full of players in top clubs, for 70 minutes though the resistance broke down in the last 20 minutes.
Algeria sat back in the first half with more men in the defence but were lucky to have taken the lead through a surprise goal in the 25th minute.
It looked like that Algeria will register their first World Cup win after their victory over Chile in 1982 but it was a matter of time a rampaging Belgium stamped their authority with incessant attack under which the Algerian defence crumbled finally.
Algeria dominated possession in the opening minutes, surging forward and moving the ball with flair. Belgium eventually settled down and took control of the game.
It was, however, Algeria who took a stunning lead from the only opening they got in the fist half. Left back Jan Vertonghen pulled down pacy Sofiane Feghouli on the edge of the box in the 25th minute and the Mexican referee Marco Rodriguez did not have any hesitation to point to the dreaded spot.
Velencia winger Feghouli coolly converted the penalty kick, placing the ball to the left of wrong-sided goalkeeper Thibaut Courtios.
Feghouli's goal is Algeria's first World Cup goal since Djamel Zidane scored against Northern Ireland in 1986. They have scoreless for 506 minutes, just 12 minutes shy of surpassing Bolivia's record for 517 scoreless World Cup minutes.
Belgium had a dominating 65 per cent possession in the first half but failed to break down the stubborn Algeria defence.
Otherwise, it was a half of few chances. Defensive midfielder Axel Witsel had Belgium's best two opportunities: powerful 25-yard shots parried by the Algeria goalkeeper in the 21st and 34th minutes.
Just before half time, Chadli Nacer had a fine chance off an assist from Eden Hazard but his shot was too feeble to create any problem to Algeria goalkeeper Rais M'bolhi.
In the second half, Belgium coach Marc Wilmots, who has never lost a competitive fixture as national team coach, made four substitutions.
Sluggish Romelu Lukaku was replaced by Divock Origi in the 58th and Fellaini came in for Mousa Dembele in the 65th minute and more muscular approach paid off.
Fellaini headed in a pinpoint cross from De Bruyne's and 10 minutes later it was Mertens who made use of the space left by the tiring Algerians to take the game away from the Africans.