Researchers tested the memory and blood sugar levels of 141 apparently healthy people with an average age of 63.
None were suffering from diabetes, or experiencing pre-diabetic symptoms.
Participants with lower blood sugar levels were likely to have better scores in memory tests.
In one test, which involved recalling a list of 15 words 30 minutes after hearing them, higher blood sugar correlated with poorer memory.
Scans also showed that the hippocampus brain region, which is important to memory, was smaller in those with higher blood sugar.
"These results suggest that even for people within the normal range of blood sugar, lowering their blood sugar levels could be a promising strategy for preventing memory problems and cognitive decline as they age," lead researcher Dr Agnes Floel, from the Charite University Medicine in Berlin, Germany, said.
"Strategies such as lowering calorie intake and increasing physical activity should be tested."
The research appeared in the latest online edition of the journal Neurology.
Dr Clare Walton, from the Alzheimer's Society, said: "We already know that Type 2 diabetes is a risk factor for developing Alzheimer's disease but this new study suggests that higher blood sugar levels may also be linked to poor memory in people without diabetes.
"The research suggests that regulating blood sugar levels might be a way to improve people's memory, even if they don't have diabetes.
"However, before people without diabetes consider changing their diets or taking medication, more research is needed to test this theory.
"One in three people over 65 will develop dementia so investing in research like this is vital."
Dr Simon Ridley, from Alzheimer's Research UK, said: "Research is highlighting diabetes is a risk factor for cognitive decline but this study suggests that higher blood sugar could be linked to poorer memory performance, even when levels fall within the normal range.
"While we do not know whether the people in this study would have gone on to develop dementia, the findings serve as a warning that we should be conscious of the impact that subtle changes in our health could have on our brain.
"Understanding the risk factors for cognitive decline and dementia could help shape health campaigns to keep people healthier for longer.
"Current evidence suggests the best way to keep the brain healthy is to eat a balanced diet, take regular exercise, maintain a healthy weight, not smoke and keep blood pressure and cholesterol levels in check."