ETFs are generally more expensive than many institutional index products and are an unattractive long-term investment option for most pension funds, according to Watson Wyatt.
The consultant says ETFs may also have tax implications that require specialist advice and often contain counterparty risks which investors may not be compensated for.
Increasing development within the indexation space would offer passive investors a broader range of options and better risk-adjusted returns than those currently available, according to the firm.
Watson Wyatt senior investment consultant Chris Sutton, who is former chief executive of iShares, says the case for including ETFs in institutional investment portfolios was "not yet obvious as we wait to see more competitive fees and transparent structures."
He adds: "In the meantime, they may be useful tools for transition and shorter-term exposure management. Where the ETF industry has engaged in product proliferation, we would rather press for genuine innovation in the investment content of index products.
"If investors are looking for more efficient market exposures their first step should be to review the indices underlying their existing investments, with a view to seeing if there are better alternatives."
The firm says in many markets passive funds have been structured with clearly defined tax positions for institutional investors, while the treatment of ETFs is much more variable, which typically necessitates tax advice.
Sutton adds: "ETF sponsors have also embedded significant amounts of counterparty risk inside ETFs, whether through stock lending or the way in which swaps are used. It is not clear that investors are being adequately compensated for having to take these risks when holding an ETF."
BlackRock global head of ETF research and implementation strategy Deborah Fuhr says: "I agree with the fact investors should look at a range of products within the passive space and therefore look at alternatives to ETFs. Nonetheless, I believe it is very hard to make blanket statements as the ones Watson Wyatt made."
She adds: "Investors should consider the size of the investment they are going to make, their time horizon and their benchmark. They might well conclude ETFs are a very attractive products for them and also that some ETFs suit their needs but others don't."
iShares head of Sales Strategy, Europe Nizam Hamid agrees. He says: "The growth of the ETF market in Europe has been largely driven by institutional investors, who represent around 70% of the total market.
"There are other products for institutions to use, but such growth of the ETF market suggests a lot of institutions can see the benefits of using ETFs compared to some other products, where you have implicit costs as opposed to easy to measure explicit costs."
'Right thing to do'
£69m spent on upgrades
European fintech market 'underserved'