"Jameen par aay gaye (now he has come on ground)," Yadav said while referring to Modi's allegation that the EC had failed to check "rigging and violence" in West Bengal, Bihar and western Uttar Pradesh.
"When people get the sense of defeat they start talking about irregularities in the game," he told reporters here.
Lauding EC for conducting free and fair polls for which it has drawn appreciation from across the world, he said those challenging EC were in fact "innocent" and "out to destroy its credibility".
On RJD President Lalu Prasad entering into verbal duel with officials over checking of vehicle of Rabri Devi, he said it was another example of showing disrespect to the EC.
"Even my vehicle were checked and two cases of poll code violation filed against me, but I never pointed fingers on EC," he said.
Yadav, who left Patna for Varanasi this morning to campaign against Modi, said he would campaign in the holy city from a non-political platform.
"I'll talk to voters from Nagrik Manch," he said.
Opinions are divided within JD(U) over extending support to the Aam Aadmi Party candidate Arvind Kejriwal in Varanasi.
Party General Secretary K C Tyagi's statement that JD(U) would support Kejriwal was not taken kindly by party leaders.
While Yadav himself had not endorsed Tyagi's statement, Bihar JD(U) President Basistha Narayan Singh had said it should have been made after wide discussions within the party.