It’s looking like this is going to a rough year for mortgage servicers, thanks to 2016’s continually low interest rates and our friends over in Britain, Fitch Ratings said in a new report.

According to Fitch’s report, many of the largest bank and nonbank mortgage servicers saw their portfolios decline in the first quarter of 2016, and with mortgage interest rates remaining near historic lows ever since the Brexit, things aren’t likely to get much better for mortgage servicers this year.

As Fitch notes, refinance mortgage applications are on the rise in the first few weeks since Britain’s decision to leave the European Union, and Fitch suggests that trend will continue as long as rates stay low.

“Mortgage servicers are facing a difficult interest rate environment for the remainder of 2016,” Fitch Managing Director Roelof Slump said.  “Add in the far-reaching impact of the recent Brexit vote and challenges could be magnified, as sustained low rates can impact servicer portfolios.”

Slump said that mortgage servicers will have to reverse the trend of shrinking portfolios in order to remain financial successful.

“Maintaining portfolio size can be an important factor as mortgage servicing is a scale business,” Slump said. “Servicers that do not have associated origination platforms and service mostly performing loans are most exposed to portfolio runoff.” 

Slump noted that smaller servicers can quickly see growth and financial benefit by being involved in new areas while it takes “more to move the needle” for the largest servicers.

According to Fitch’s report, the first quarter wasn’t as rough on some servicers.

In particular, Roundpoint Mortgage Servicing showed more than 13% growth in loans serviced, according to Fitch’s data.

Additionally, Cenlar FSB, Specialized Loan Servicing and Shellpoint also exhibited solid growth in the first quarter, with those companies’ servicing portfolios seeing growth of between 5% and 8% during the first quarter.