Some 80% of retirees who stick with their pension provider when they buy an annuity could get a better deal elsewhere, pointing to a huge failure in the open market option, the Financial Conduct Authority (FCA) has found.
The regulator's thematic review into the annuity market found eight out of ten people could get a more generous rate if they shopped around and bought an annuity from a different provider.
This equates to having an extra £1,500 saved into a pension just before retirement, it said.
The FCA found that for a pension pot of £17,700 (the average size in this review), buying an annuity from the existing pension provider would return an average annual income of £1,030.
However, by shopping around and switching provider, that annual income would increase by 6.8%, or £71, to £1,101.
It said one in six people could increase their retirement income by more than 10% if they changed provider. For people with severe health conditions the figure was "potentially much higher".
Currently, 60% of people buy an annuity from their current provider, with about 420,000 annuity sales every year, according to Association of British Insurers (ABI) data.
It described the market as "disorderly" and said further investigation was warranted.
A competition market study to "get to the heart of the issue" will now begin, the FCA added.
The review said providers expect to receive greater profit from the annuities they sell to their existing customers versus contracts conducted in the open market.
The FCA said: "Several insurers have retention of annuity business as part of their business strategy and one of the drivers of this may be the expected profitability of this business."
The regulator praised the efforts of the ABI to encourage shopping around but also said there are "significant barriers" in the market and that the "traditional method of disclosure may not be enough to change consumer behaviour".
These barriers included consumers lacking the confidence to switch provider, not fully understanding the decision they need to make, and behavioural biases such as inertia.
The review also found there was "virtually no market whatsoever for people with smaller pension pots".
People with pots of £5,000 or lower have less choice when shopping around as only a handful of providers offer them annuities, the FCA found.
Chief executive Martin Wheatley said: "The need to get an income in retirement unites us all. But once you've bought an annuity you can't change your mind. For most people getting the right annuity could mean the equivalent of an extra £1,500 in savings - so we need to understand why they aren't shopping around and switching.
"But this isn't true for everybody; our research showed that there is virtually no market whatsoever for people with smaller pension pots. This means that for those people who need to make every penny of their pension count, the market has closed the door on them."
He added: "There should be competition across the entire market, not just for those with the most money. That is why we will be using our new remit to conduct a competition market study and a review of sales practices in pension providers."
The thematic review gathered data from 25 firms - representing 98% of the annuities market.
Provider MGM Advantage said the FCA review did not go far enough or act quickly enough.
Pensions technical director Andrew Tully said: "This will potentially leave many thousands of retirees high and dry when navigating the annuity minefield.
"So often we see poor consumer outcomes through a lack of awareness or understanding of the options available. Although the review puts the spotlight firmly on the issues that need to be addressed, another year or two of customers sleepwalking into retirement is simply not good enough.
"There are some simple and practical steps we can take now to help those people looking to take benefits from their pensions. This could be done in tandem with the competition market study which will take some time."
Key Retirement Solutions group director Dean Mirfin said the findings would be of little surprise to industry players but would push the fact that advice is necessary at retirement.
"What we desperately hope is that consumers will now start to understand and believe the fact that shopping around is a vital part of retirement planning.
"The industry must work hard to deliver the best consumer outcomes. Some insurers have taken, and continue to take, further steps to achieve this but too many are still taking advantage of consumer apathy and their customers trust in them to do the right thing."
Hargreaves Landsown head of pensions research Tom McPhail said: "The FCA has confirmed what many independent observers have known for years: The retirement income process only works well for the minority of investors who fight their way out onto the open market and engage with a good quality broker; the problem is that the majority of investors aren't finding their way through to these good deals.
"Insurance companies have been the principal culprits in selling poor value products over the years and annuity brokers are working towards adopting minimum standards in how they deal with their customers.
"This is an opportunity for the FCA to initiate a fundamental rethink on how we help millions of investors to get the best value from their retirement savings."
Let's talk numbers...
The FCA found that for a pension pot of £17,700 (the average size in this review), buying an annuity from the current pension provider would return an average annual income of £1,030.
However, by shopping around for a better rate and switching provider, that annual income would increase by 6.8%, or £71 to £1,101.
One in six people could increase their retirement income by more than 10% if they changed provider. For people with severe health conditions the figure is potentially much higher.
For enhanced annuities, the average fund size was £26,800 with the average annual income achieved by sticking with existing provider at £1,630.
Some 91% could be better off by an average of £135 a year by switching - a 8.3% increase.
That's the equivalent of £2,428 extra in their pension pot just before they retire.
Let me break it down...
About 420,000 annuities are sold a year (ABI 2012), the FCA's review breaks this down as follows:
Out of scope of the review:
31 firms sell pension annuities (25 of which were included in the review)
Paul Bruns and Elaine Parkes
3,000 left to transfer
Record numbers of people aged 90 plus
From 3 to 10 October