The residential mortgage-backed securities market is expected to continue its recovery well into next year as long as it’s not stunted by popping home price bubbles, regulatory uncertainty and volatile interest rates, Fitch Ratings RMBS analysts said Wednesday.

Fitch RMBS head analyst Rui Pereira contributed to a larger report on structured finance this week. In it, he notes RMBS transactions created before the financial crisis are likely to maintain their stable ratings in 2014 due to positive credit trends and an improving housing market.

The only outlier is the prime, pre-2005 vintage collateral, which remains a step apart due to "collateral adverse selection and growing tail risk," the report says.

But the RMBS market may become a minefield if outliers begin presenting themselves in 2014. These outliers include all of the regulatory uncertainty, the potential for volatile interest rates and a likely tapering of the Fed's agency mortgage-backed securities and Treasury purchases at some point.

Ashley Gam, an analyst with the Royal Bank of Scotland (RBS), said the big outlier is whether President Obama’s pick to lead the Federal Housing Finance Agency, Rep. Mel Watt, D-N.C., gets into place and offers principal write-downs or an expansion of the Home Affordable Refinance Program.

"The major risks will come in if he introduces borrower-friendly modifications," Gam told HousingWire. "One being expansion of the HARP program. That would mean expanding the HARP cut-off date, which would make more borrowers eligible," Gam said. If that were to take place, RMBS deals with 4 ½ to 5 coupons would feel it the most.

Gam doesn’t expect a potential Fed tapering of mortgage-backed securities purchases to hurt too much since it’s likely to come in small waves, possibly beginning in March, she said. But not knowing where the FHFA or other regulators could head in 2014 remains challenging.

"You have nowhere to hide. It’s sort of across the whole stack,” she advised. Previously, you could move to higher coupons if one part of the market is affected, but now there are other risks making that a challenge as well.

Fitch analysts highlight some of the same risks and are cautious about steep price declines if certain markets bubble over.

"While home price gains are well supported in most regions, the risk of overvaluation is increasing in some markets," the Fitch report says. "Home prices are expected to climb further, although other higher rates may temper growth."

In fact, if there are headwinds slowing the real estate and RMBS recoveries, Fitch sees them as steep declines in home prices, upticks in unemployment, rapidly rising mortgage rates and a significant contraction in mortgage credit availability.

Fitch believes home price appreciation will actually moderate next year as rates edge up, but for now, it’s calling RMBS stable thanks to solid macro conditions.

However, this doesn’t mean swift changes in any of the outliers couldn’t throw off investors. It’s just not happening yet. Right now, the total percentage of outstanding RMBS that's considered seriously delinquent is down to 24% from 32% three years earlier.

But Fitch is somewhat frightened by markets that are seeing dramatic double-digit home price growth year-over-year. One example is the the San Francisco-Bay Area, which saw prices rise more than 20% over the past year.

"Fitch does not believe recent home price growth in these markets is sustainable and currently view parts of coastal California as being more than 20% overvalued, according to our sustainable home price model," the ratings giant advised.

The result of shifting home prices could also impact where institutional investors put their money, Fitch said.

"Low rates and a steep drop in prices, coupled with the decline in homeownership, have attracted an estimated $15 billion of new capital to the single-family rental sector," the ratings agency concluded. "While these investors have been a significant source of demand, their decision to scale back growth or exit positions could drive greater volatility in affected markets."