A fifth of clients who have not seen an IFA in five years blamed trust issues.
According to the Sun Life of Canada Sense Check at 60 report published this month, respondents also say their trust in pension companies had been eroded since the recession.
However, IFAs are seen as more trustworthy than pension providers with 43% of clients saying they trust advisers, while only 31% trust their pension provider.
In total, 65% of respondents say financial service providers must help remove the fear surrounding retirement planning.
Chris Cummings, who was director of AIFA at the time of the publication, writes in his introduction to the research: "As an industry, we suffer from the reputational damage wrought by pensions mis-selling and the various scandals and wrong-doings that have hit the headlines and undermined trust in the financial services
"If we are to help people meet their financial retirement goals, we can only do so from the position of being trusted.
"Trusted enough that people have the confidence to look to our sector for long term security, trusted enough to follow the advice they receive, and trusted enough to be seen by politicians and regulators as part of the solution, not part of the problem."
He adds: "The most important thing remains the same: people want to deal with
individuals they trust. Those who make a personal commitment to help and advise them today and over the longer term.
"So much about this market is based on the fact that it is long term by its nature and that applies to the advice too: people want an on-going relationship with their adviser: up to, at and after retirement. Good firms recognize this desire and offer a tailored service to match these client needs."
Joined as head of strategy, multi asset, in June
Group income protection
Nine in 10 do not have income protection
Set to become part of Single Financial Guidance Body