Nearly one quarter of working households in the United States spends more than half of their income on housing, says a report by the Center for Housing Policy.

The cost of owning a home (including mortgage and utilities) went down between 2008 to 2010, dropping 5%, but so did homeowners’ incomes, which decreased 2%—often as a result of layoffs or reduced hours. During this same time frame, the number of households nationwide spending more than half their income on housing rose from 21.8% (in 2008) to 23.6% (in 2010).

The report compared the housing costs for renters and homeowners who make within 120% of the median income for their area in all 50 states and the District of Columbia, and the 50 largest metropolitan areas in the U.S.

“The data show that homeowners have been hit hard by the housing crisis in more ways than just lost equity,” said Jeffrey Lubell, executive director of the Washington-based Center for Housing Policy, in a statement. “Many working homeowners have been laid off or had their hours cut.”

Housing costs may have dipped but unless homeowners were able to refinance their existing mortgages during that period they would not have been able to take advantage of lower prices, said Laura Williams, Research Associate at the Center for Housing Policy and lead researcher on the report.

“It’s a bit incorrect to say that housing costs decreased for many homeowners,” said Williams. “Many aren’t able to take advantage of them unless they were able to refinance over the same period. So they’re facing the same level of costs that they were before but with lower incomes.”

Nationwide, 24 markets noted a significant increase in the number of  households spending more than half their income on housing. California was the hardest hit, at 34%, followed by Florida at 33%, while Maine was the only state to note a significant decrease.

Three of the five largest metropolitan areas with the highest percentages of households with significant housing costs were located in California, joining Miami, Fla. and Pittsburgh, Pa.

Written by Lindsay Welbers