Declining mortgage rates weakened most residential mortgage-backed securities in the fourth quarter by causing higher prepayment speeds and narrower spreads on new investments.

Analysts at investment bank KBW expect weaker results for most of the residential MBS real estate investment trusts in the fourth quarter versus the third quarter.

Residential MBS REITs underperformed the broader market in the fourth quarter with an average total return of 2.9% versus 11.7% for the S&P 500. Dividends, which are generally a good indicator of earnings, for most of the sector have been flat to down, analysts said.

On the commercial side, real estate fundamentals are outperforming residential real estate, analysts at the investment and advisory firm Forward Management said, adding that REITS could attract more investors in 2012.

"We are big believers in investing through publicly traded REITs and real estate development or operating companies," said Joel Beam with Forward Management. "The first reason is liquidity, which you don’t get with limited partnerships or private REITs."

KBW analysts remain positive on most of the residential MBS REITs based on dividend expectations, however, they do not expect meaningful improvements in book values.

Dividends continue to drive returns for the group as the average dividend yield came in at 3.8%. Redwood (RWT) and Chimera (CMI) were the worst performers in the sector with fourth quarter returns of 6.4% and 5.0%, respectively, but both have been the strongest performers in the space so far in the first quarter of 2012.

"We remain positive on most of the residential mortgage REITs," KBW analysts said. "While the sector has had a rally in 2012, most names trade at relatively modest premiums to book value."

They said with the Federal Reserve announcing its intention to keep short term rates on hold through 2014, investors will be more willing to take interest rate risk, causing interest in the RMBS REITs to continue to grow.

The declining mortgage rates also led to higher prepayments, with speeds appearing to have peaked in the fourth quarter following increases since the middle of 2011.

Fannie Mae MBS prepayment rate declined to 23.1% in December from 24.7% in November and 24.4% in October. The average prepayment rate for Freddie Mac MBS declined to 24.3% in December from 25.7% in November and 25.9% in October.

"While we currently expect prepayments to continue to slow in 2012, if the sharp drop in interest rates that has occurred over the last week persists, we could see another pickup in refinance activity," KBW analysts said.