While consumer confidence struggled to rise 0.5% in May and there are signs of strength in other sectors, housing remains a drag on both consumer sentiment and the overall economy.

Or, it may be more accurate to say that a lack of consumer confidence is creating drag for housing.

The Conference Board says that it believes there are still solid positives for jobless claims, which point to improvement in the labor market, and the factory workweek, which point to gains for income and output. The report's leading credit index is also solid pointing to improved lending strength.

The big negative in the report, however, is weakness in building permits in what points to continuing struggles for the housing sector.

Analysts predicted 1.03 million annual housing starts for May, but with investors are pulling out of multifamily and homebuilder being wary of betting too heavy on single-family taking off, it was a miss. There were just 1.001 million starts in May.

The housing permits front looks even grimmer. Analysts expected 1.05 million, but what the industry delivered was 991,000, a miss of epic proportions.

For the industry, it should be a wakeup call.

Adding to the concern, mortgage rates are hovering at the lowest averages in almost a year, and buyers aren’t taking.

A week after unexpectedly jumping 10.3%, mortgage applications returned to their declining trend despite near year-low interest rates – tumbling a remarkable 9.2%, according to data from the Mortgage Bankers Association’s Weekly Mortgage Applications Survey for the week ending June 13, 2014.  

This is despite mortgage rates that should be tapping the so-called pent-up demand. Rates are hovering well below the 4.25% mark.

The 30-year, fixed rate mortgage averaged 4.17% for the week ended June 19, compared to 4.20% a week ago and 3.93% a year prior, Freddie Mac reported Thursday.

This is not a short-term anomaly, this is an ongoing pattern.

Even Federal Reserve Chair Janet Yellen noted this in her remarks after the Federal Open Markets Committee announcement Wednesday.

“The Committee currently judges that there is sufficient underlying strength in the broader economy to support ongoing improvement in labor market conditions,” she said. “Recovery in the housing sector remains slow.”