Housing may actually add to U.S. GDP growth by the end of 2012, according to Fannie Mae Director of Mortgage Market Analysis Richard Koss.

"I think so, but it will be very small. In the past it was somewhere around 6% of GDP, but now it might bump up to something like 2%," Koss said at an American Action Forum conference in Washington, D.C. Thursday.

The Federal Reserve lowered its projection for GDP growth in 2012 to between 1.9% and 2.4%.

Taking a 5% baseline average, the U.S. economy holds a $350 billion hole or roughly 2.3% of GDP since the downturn in 2007, according to the Bipartisan Policy Center.

"This will be the first time in five years that we have a positive addition, but it will not be enough to change the far out decimal on your GDP forecast," Koss said.

Construction and home sales continue to lag. Since 2005, construction contributed less to GDP every year. In 2011, construction added just $440.5 billion to GDP, down from $612 billion in 2005, according to an American Action Forum paper released Thursday by analysts Andy Winkler and Douglas Holtz-Eakin.

The biggest drop came in 2009, when construction added just $456 billion to GDP, a nearly 13% drop from the previous year.

"The construction of owner-occupied, single family homes is today a mere fraction of its pre-bubble level. The resurgence of residential construction spending will not come until the excess of distressed properties clears the market and people have the disposable incomes to pursue homeownership," Winkler and Holtz-Eakin wrote.

New York Federal Reserve Executive Vice President Joseph Tracy sat on the same panel Thursday as Koss. Despite the very nominal addition to GDP growth housing is expected to have, Tracy said it's still a sign the housing market is beginning to pick back up.

"Even less of a drag is a positive contribution," Tracy said.