All of the major builders faced stock declines, including D.R. Horton (DHI), Toll Brothers (TOL), Lennar (Len) , KB Home (KB) and Pulte Group (PHM).

Only 69,000 jobs were created in May, well below the 115,000 new positions reported in April. That April figure has since been revised to 77,000 new positions, well below the 200,000-plus job gains reported earlier in the year.

Even though builder stocks were rocked by declining jobs numbers, construction spending in April edged up 0.3% from March and 6.8% from last year, the government said.

Overall, the Commerce Department valued April construction spending at $820.7 billion, compared to $768.2 billion a year earlier. Residential construction also grew 2.8% from $249.1 billion last year to $256.1 billion in April.

The construction spending numbers paint a more optimistic picture for builders, but their stock prices still fell on signs of a slowing job market.

Pulte's stock fell almost 10% during morning trading, while Lennar, Toll Brothers and D.R. Horton's stock fell at least 6% after the weak jobs report. KB Home fell more than 5%.