Addressing a rally in Nizamabad, the Bharatiya Janata Party leader also accused the Congress of adopting a "divide and rule" policy.
"This family removed Narasimha Rao from public memory. They dislike Andhra Pradesh and believe in divide and rule," Modi said.
"Be it Andhra or Telangana, this one family has hatred towards the state. They do not even give respect to Narasimha Rao," he said.
Rao was prime minister during 1991-1996. He was the first Congress leader outside the Nehru-Gandhi family to serve as prime minister for five years.
Modi will be addressing a series of rallies in Andhra Pradesh and Maharashtra Tuesday.