A Reuters reporter saw a convoy of around 20 military transport vehicles and buses filled with armed rebels driving out of Kramatorsk where they had gone after apparently earlier fleeing the separatist stronghold of Slaviansk nearby.
Interior Minister Arsen Avakov said a large number of separatists had fled Slaviansk after sustained fire from Ukrainian forces.
"A significant number of militants have left Slaviansk... along the way, our battle groups are greeting them. They are suffering losses and surrendering," he said in a statement on his Facebook page.
A source close to the rebels told Reuters the rebels had been outnumbered by 50 to one. "(The Ukrainian forces) have greater numbers of troops and military hardware," the source said, speaking on condition of anonymity.
Aleksandr Borodai, a leader of the self-proclaimed Donetsk People's Republic, was quoted by the Interfax news agency as saying: "The punitive forces of Ukraine... moved into a large-scale offensive. Given the disproportionate numerical superiority of the enemy troops, units of the armed forces of the Donetsk People's Republic were forced to leave their previous positions on the northern sector of the front."
Ukrainian President Petro Poroshenko, after hearing a report from his newly-appointed chief of the armed forces general staff, ordered the Ukrainian flag to be run up on public buildings in Slaviansk.
The Ukrainian flag was later flying over Slaviansk, the website said, in place of the Russian one which had been there since rebels took over key buildings in April.
Slaviansk has been the strongest redoubt of militants fighting government forces in eastern Ukraine. Its re-capture would be Kiev's most notable military victory in three months of fighting in which more than 200 Ukrainian troops have been killed as well as hundreds of civilians and rebels.
Poroshenko's website said separatist fighters had come under mortar fire as they tried to break through government forces' lines. The separatists had lost one tank and other armoured vehicles, the statement said.
"The president gave the order... for the state flag to be raised on Slaviansk's town council offices," the website said, referring to one of the city's main rebel headquarters.
"Armed forces' intelligence and the National Guard are now working in Slaviansk," it said.
A Ukrainian paratrooper, Eduard, who was at an army checkpoint going into Slaviansk said: "It was a quiet night when suddenly a column of separatists appeared and began firing. They didn't have much luck. Artillery guns hit part of the town and part of those here. Some of them managed to get out."
Some of the rebels appeared to have moved to Kramatorsk but they were pulling out of there quickly on Saturday apparently fearing air strikes from Ukrainian planes overhead.
People in the town said they had begun to pull out around 4 a.m. and about 100 of them had left the town.
Uprisings in eastern Ukraine erupted in April as rebels took over state buildings, built a powerful arsenal of seized weapons and declared their independence from Kiev, calling the pro-European government illegitimate.
The crisis began when street protests ousted the Moscow-backed Viktor Yanukovich in late February for rejecting a landmark political and trade deal with the European Union in favour of closer ties with Russia.
Russia subsequently annexed Crimea and separatist revolts against the new Kiev authorities broke out with rebels declaring "people's republics" and saying they wanted to join Russia.
Slaviansk, a city of 130,000 people, became a hotbed of resistance under the military command of Igor Strelkov, a Muscovite appointed as defence minister of the self-declared Donetsk People's Republic.
It has increasingly appeared to be going its own way independently of the rebel groups controlling Donetsk, the main industrial hub, and Luhansk, and many of the rebel violations of a government ceasefire which expired last Monday appeared to come from Slaviansk.
But on Friday Strelkov made an impassioned appeal for help to Russia, whom Kiev blames for fanning the violence. He said without Moscow's aid the region the rebels lay claim to, known as Novorossiya (New Russia), would fall to Kiev's forces.
"Slaviansk will fall earlier than the rest," he wrote on a rebel website.
The government launched a new offensive against the separatists on Tuesday after Poroshenko allowed a 10-day unilateral ceasefire to expire without being extended.