The ferry was only licensed to carry 85 passengers, but had more than 200 on board when it went down in rough conditions in Munshiganj, around 30 kilometres (20 miles) south of Dhaka, on Monday.
Police said they had launched a manhunt for the owner of the vessel, which rescue workers have still not been able to locate more than 48 hours after the disaster.
"The owner A. B. Siddique Kalu and five others including the captain have been charged with causing death by overloading it beyond its capacity of only 85 passengers," local police chief Tofazzal Hossain told AFP.
If found guilty under the charge of culpable homicide not amounting to murder, they will face jail time. Ferry accidents are common in Bangladesh, with overcrowding and poor ship design and maintenance often to blame.
The charges came amid growing anger among the hundreds of distraught relatives who have gathered on the river bank for news of the missing. Survivor Shahidul Islam, 36, whose wife, son and niece are still missing, said the authorities were not doing enough to find the boat.
"It's been more than two days, but they are still not able to locate the boat," Islam told AFP. Fire service deputy director Amir Hossain Mojumder, however, defended the recovery work, blaming difficult weather conditions.
He said strong currents had stirred up silt in the river, reducing visibility, and might also have pushed the wreckage downstream. Around 100 survivors were pulled from the waters on Monday, and authorities on Wednesday revised up the number of missing to 133.
Bangladesh, one of Asia's poorest nations, is criss-crossed by more than 230 rivers. Boats are the main form of travel, especially in the southern and northeastern regions. Officials have said more than 95% of Bangladesh's hundreds of thousands of small and medium-sized boats do not meet minimum safety regulations.