It’s difficult to gauge how the housing market viewed the Federal Reserve’s decision to keep acquiring mortgage-backed securities and Treasurys at its current pace on Wednesday.

After all, investors and the housing market do receive some fringe benefits from the monthly sugar injection of MBS purchases and from rates staying artificially low as the Fed buys up agency MBS, while simultaneously keeping the federal funds rate near zero.

But despite these lingering benefits, the HW 30 index – a compilation of housing and mortgage finance stocks – fell in the wake of the Fed’s announcement.

All the major HW homebuilders – including Toll Brothers (TOL), Lennar (LEN) and D.R. Horton (DHI) – tumbled, finishing down 1.64%, nearly 2% and 2.25%, respectively.

Perhaps, what spooked homebuilding investors is the Fed’s own reluctance to declare the housing market fully recovered. In its latest update, members of the Federal Open Market Committee showed some skepticism about the overall state of the economy.

For starters, the unemployment rate remained elevated.  

Furthermore, the Fed says current data suggests "household spending and business fixed investment advanced, while the recovery in the housing sector slowed somewhat in recent months."

The above statement may have sparked new fears, hitting stocks related to home buying and mortgages.

The list of falling stocks included, lender Impaq Mortgage Holdings (IMH), and MOVE Inc. (MOVE), the parent of realtor.com, which alone fell more than 4%.

Real estate website Zillow (Z) also finished down roughly 2%.

Overall, a strong majority of the stocks making up the HW 30 index fell on Wednesday.