A study involving more than 1,000 elderly people has found that those who began taking benzodiazepine had a 50% higher risk of developing dementia within 15 years.
In Britain the drugs are used for short-term insomnia and anxiety, but mostly for sedation and anxiety before surgery. Around 1.5 million people in Britain are believed to be taking the pills at any one time and around 9 million prescriptions were issued in England in 2011 for benzodiazepines which include diazepam and temazepam.
They should only be prescribed for short-term relief but patients often remain on them for years.
Experts said their "widespread use should be cautioned against".
Academics from Harvard University and the University of Bordeaux carried out a study on 1,063 men and women, with an average age 78, who were all free of dementia at the start of the trial. Over the following 20 years 253 developed dementia, 30 cases of which were in benzodiazepine users.
They accounted for other factors that affect dementia such as age, sex, educational level, marital status, wine consumption, diabetes, high blood pressure, cognitive decline, and depressive symptoms.
The results showed that 4.8 out of every 100 people who had taken the drugs developed dementia compared with 3.2 who had not.
Sophie Billioti de Gage, the lead author of the study, wrote in the British Medical Journal that the increased risk of dementia, if substantiated by other studies, "would constitute a substantial public health concern".
She said: "Benzodiazepines remain useful for the treatment of acute anxiety states and transient insomnia.
"However, increasing evidence shows that their use may induce adverse outcomes, mainly in elderly people, such as serious falls and fall-related fractures.
"Our data add to the accumulating evidence that use of benzodiazepines is associated with increased risk of dementia, which, given the high and often chronic consumption of these drugs in many countries would constitute a substantial public health concern."
Prof Tobias Kurth, who works jointly at Harvard and the University of Bordeaux said: "There is a potential that these drugs are really harmful. If it is really true that these drugs are causing dementia that will be huge. But one single study does not necessarily show everything that is going on, so there is no need to panic."
Prof Clive Ballard, director of research at the Alzheimer's Society, said: "This is the not the first time it has been suggested that these drugs could have a negative impact on cognition. With this long-term study adding to the evidence, it emphasises how important it is that we properly monitor how treatments for anxiety or sleep problems are used.
"While Alzheimer's Society is leading the way in calling for an end to the inappropriate use of anti-psychotic drugs for people with dementia, it is also vital that benzodiazepines are not automatically turned to as the alternative.
"Instead we need to empower care staff with the knowledge they need to understand dementia and the person behind the condition."