According to the latest quarterly report from the Federal Housing Finance Agency, both Fannie Mae and Freddie Mac are reducing the money set aside to cushion any blow to their business.

The loss reserve fund once stood at $116 billion and it's now down to a four-year low of $72 billion (see chart at bottom).

It's fair to say this is the result of the government-sponsored enterprises betting on a continued housing recovery.

The report states that home prices are up, so they're making more selling REO. And as the fair value of housing keeps increasing, although that rate is expected to slow, the strength of the GSE portfolios will likewise appreciate.

Plus, the GSEs brought in more than $15 billion from the settlements of FHFA lawsuits regarding mortgage-backed securities with multiple mortgage lenders.

And post-housing bust originations are a greatly improved collateral pool. However, it should be noted that refiancings will continue to be a huge driver of business at Fannie and Freddie. The worst vintage, performance wise, is 2006 and 2007, so the indication is that the GSE will continue to look at these borrowers for HARP assistance.

In 2013, the reports state delinquent loans continued to decline. Serious delinquencies are down 27%, to around 256,000 loans.

"The combination of these factors resulted in a 23% or $21.5 billion decrease in the Enterprises' combined loss reserves," the report states.

The enterprises and Ginnie Mae still take the lion's share of mortgage-backed securities, but that footprint is reducing, as mandated by Congress.

However, at $1.2 trillion in issuance in 2013, these three will long be the primary mortgage finance players. That they believe so much in the housing recovery continuing should provide a boost of confidence to everyone.

Here's the graph showing this trend.

Fannie and Freddie loss reserves