The only profits Fannie Mae and Freddie Mac generated since entering conservatorship in 2008 came from financing multifamily mortgages, according to the inspector general of the Federal Housing Finance Agency.

The GSEs earned a $7 billion gain from their multifamily division between 2008 and the first quarter of 2012. It's more than overwhelmed by the $208 billion in total losses from the single-family mortgage business, $4 billion in investment losses and $16 billion in what the FHFA OIG called "other losses," according to a report released Thursday.

As of March 31, both GSEs drew more than $188.4 billion in bailouts from the Treasury Department and paid back $41 billion in dividend payments, according to their financial filings.

The losses before leading to conservatorship were staggering. In 2008 alone, the GSEs combined for more than $100 billion in losses ($58.7 billion for Fannie and $50.1 billion for Freddie).

"To put these losses into perspective, over the 37-year period from 1971 to mid 2008, Fannie Mae and Freddie Mac earned $95 billion, less than they lost in 2008 alone," the FHFA OIG said.

The FHFA estimates both Fannie and Freddie will end up drawing between $220 billion and $311 billion through 2014, depending on where house prices go.

But performance has turned around. Fannie actually drew an overall profit in the first quarter, the first time that happened since entering conservatorship in 2008.

Multifamily has been a major driver for the entire housing industry. According to the Mortgage Bankers Association, multifamily mortgage originations jumped 45% in the first quarter from one year prior.

"The low interest rates and stabilization in commercial real estate fundamentals that raised origination levels by 55% in 2011 are continuing to buoy activity in 2012," said Jamie Woodwell, MBA vice president of commercial real estate research earlier in the month.