Markets went haywire after the Federal Reserve gave investors a peek at the latest Federal Open Market Committee meeting minutes.

The minutes released Wednesday showed a solid majority of FOMC members on board with Chairman Ben Bernanke's proposal to ease back on mortgage-bond purchases in the fall if emerging data still supports the decision.

In response, the HW 30 stock index took a beating Wednesday afternoon.

The minutes showed all of the FOMC members 'broadly comfortable' with the timeline Federal Reserve chairman Ben Bernanke laid out for tapering asset purchases. The only dissenter was Esther George, who believes the markets should receive a timely forewarning before QE3 tapering begins.

When looking at just homebuilder stocks, investors in the space sold off right after the Fed minutes hit the wires. The SPDR S&P Homebuilders ETF (XHB) and the iShares Dow Jones US Home Construction (ITB) indices declined on fears that tapering would push rates higher, stifling home affordability.

"Less repurchasing on their part translates to higher mortgage rates and thus lower affordability across the housing spectrum, assuming buyers finance their homes," said John Benda, a research analyst with Susquehanna Financial Group.

Only three homebuilder stocks refused to let the FOMC minutes shake their market performance. Those solid performers included Beazer Homes USA (BZH), Hovnanian Enterprises (HOV) and Toll Brothers (TOL).

Homebuilders that deliver at the higher end of the average selling price spectrum — such as Toll Brothers — are ‘rate agnostic,’ meaning that their buyers typically purchase homes with cash and have stronger personal finance profiles than those at the lower end of the average selling price spectrum, Benda said.

This is why Toll Brothers is less disrupted by QE3 easing and rising rates.

On the other hand, builders like D.R. Horton, Inc. (DHI) often cater to buyers with floating debt and lower incomes, making these builders and their customers more sensitive to interest rate changes, Benda added.