Housing will never return to pre-crisis levels with baby boomers aging and the next generation lacking the population numbers to support the credit and homeownership expansion that occurred in the decades leading up to the credit meltdown, Christopher Whalen of Tangent Capital Partners said Friday.

While speaking to an audience at HousingWire's REperform Summit, a mortgage servicing conference held in Dallas, Whalen said the U.S. economy since the 1980s  managed to substitute spending in other areas by leaning on housing.

"This is significant because if you look at the factors that drove housing, the biggest one was the baby boom," Whalen said. "The population growth after the boom basically drove consumption, and we had this remarkable increase in demand for housing."

But Whalen said the boomers heading into retirement are leaving smaller populations behind them, making it unlikely housing will ever get back to the peak levels experienced before the 2008 financial crisis.

"When you are looking at things like employment and job growth, you have to reduce your expectations of what you see in the future because the demographics are just not there," Whalen said.

Still, Whalen sees homeownership still having a place in the market, but said regulations are blocking certain aspects of the rebuilding.

"Credit availability is the second factor," Whalen said when discussing how the overall market will shift. "We have to tailor down our expectations. You are not going to have the degree of credit availability that you had back in 2005."

Despite housing still facing a great deal of uncertainty, Brian Montgomery, chairman of the Collingwood Group, told the same crowd that the nation can "expect mild to moderate improvement in the housing market."

Home prices are already ticking up, he noted. But Montgomery conceded that rules drafted to fix housing, including the CFPB's qualified mortgage rule, are still going to have a dramatic impact on the market.

While presidential candidate Mitt Romney wants to repeal Dodd-Frank, Montgomery is not counting on it.

"Unwinding legislation is extremely difficult," he said. "It's almost impossible to unwind the CFPB (Consumer Financial Protection Bureau), but it doesn't mean you can't try to soften what they are trying to do."