As mortgage rates and home prices soar, investors are growing more weary and beginning to back away from the housing market.

This segment of the homebuying market is somewhat risk-adverse due to fears over home price appreciation, the potential for lower affordability levels and uncertainty as to whether there will be a sudden pullback in investor buying, Bank of America Merrill Lynch’s said in a report.

Investors have played an important role in stabilizing the market, particularly in hardest hit areas. As a result, they have been able to arbitrage between a market with excess supply of owner-occupied properties and a market with an undersupply of rentals.

However, investor demand is waning as the pipeline of distressed properties shrinks and affordability declines.

"This will be part of the transition back to a more normal housing market, but also another reason to expect slowing price appreciation in coming years," explained Chris Flanagan, Michelle Meyer and Justin Borst, mortgage-backed securities strategists for BofAML (BAC).

Institutional investor buying is concentrated in key markets today.

For example, of the 280 metropolitan statistical areas sampled, 10% of those metros contain 65% of total investor purchases.

As of the first quarter of 2013, the hottest market for institutional buying is Miami, as it makes up the highest share of total transactions, according to data compiled by RadarLogic.

When combining data from the top 20 MSAs, RadarLogic found that institutional investors made up 12% of sales during the first quarter of this year.

For the most part, home purchases by institutional investors have increased in value more rapidly than non-investor properties.

This is evident in Atlanta, Charlotte, Chicago and Cleveland – all markets that did not have the early frenzy of investor buying, the report noted.

In contrast, there is a smaller gap between home price appreciation for investors versus the overall aggregate in Las Vegas, Los Angeles and Miami, which is due to a greater share of investor buying in these markets and therefore these transactions "overwhelm the aggregate composite," BofAML analysts stated.

Overall, investor participation should start to decline given the drop in affordability and shrinking share of distressed homes.

This is evident in markets that are further along in the healing process of the housing recovery.

"In light of the changing prospects for US monetary policy, we have revised our 10-year Treasury yield forecasts higher," said analysts for Capital Economics.

They added, "Accordingly, recent gains in 30-year mortgage rates are unlikely to be unwound and could well be extended."

Nonetheless, there will be some winners in losers in this trade.

For instance, investors who are buying non-distressed properties are unlikely to realize the same return as those who were able to buy distressed properties in bulk because they acquired homes at the bottom of the market.

"The dynamics of investor buying and their subsequent sales will be important to monitor over the coming years," BofAML analysts concluded.