While the majority of the companies on the HW 30 — an exclusive list of mortgage-related stocks — continued to rally on news that Congress is trying to work out a deal on the shutdown, one particular mortgage firm took a big hit.

The stock for Walter Investment Management (WAC) finished down nearly 4% Friday, after a Wells Fargo (WFC) analyst downgraded the mortgage servicer.

The potential impact of rising mortgage rates weighed heavily on the decision since rates could directly impact the firm's HARP refinancing business, explained Wells Fargo Securities analyst Joel Houck.

Additionally, the upcoming departure of Walter Investment chief financial officer and chief operating officer Charles Cauthen could become a distraction for the company.

"Based on these factors, we believe that the near-term upside in shares remains capped," Houck said.

Over the past month, Walter Investment has spiraled down 4.86%, while the company is down nearly 14% year-to-date.

Many lenders are feeling the pressure of the rapid tapering of refinance demand due to sharp increases in mortgages rates, forcing the size of the market to shrink. Consequently, the refi boom has faced its final curtain call.

Mega banks JPMorgan Chase (JPM) and Wells Fargo are also taking a beating as refi volumes fall.

As a result, these two top lenders are cutting mortgage staff.

Earlier this year, JPMorgan announced they would reduce head count by 13,000 to 15,000 employees within its mortgage unit through 2014.  

This number is on top of the 4,000-person reduction in the consumer banking unit.

Similarly, Wells Fargo announced plans to cut 20% of its mortgage production staff, with refinancing activity accounting for 70% of the bank’s mortgage applications during the first half the year, but that has dropped to roughly 50% with a decrease expected to happen in the future.

JPMorgan finished Friday down 0.02%, while Wells Fargo also underperformed a bit, falling 0.02%.