After sitting through almost a dozen sessions in at an industry conference and talking one-on-one with at least twice that number in the MSR space,  one thought kept recurring to me:

There has to be an easier way to make a living.

No, not me. Covering this industry is engaging, rewarding and – frankly – kind of fun. The Information Management Network’s first-ever Residential Mortgage Servicing Rights conference was an intimate venue and the information was enlightening.

I mean there has to be an easier way to make a living for people working in the mortgage servicing space, be they servicers, securitizers or even investors. And, Odin forbid, you’re a nonbank servicer, why not run for the hills and maybe get into something less frustrating, like panning for gold in a bathtub or something?

And yet time and again, in panel after panel and discussion after discussion, and despite all the hurdles and uncertainty, it was amazing to see the determination of those in the space to make it work.

Oh sure, the reward can be there. As one panelist noted, MSRs can be an offset, a cheap hedge, and compared to IOs spreads there’s a higher return. When mortgage rates tick up there will be opportunities.

But until then – ugh. What a business.

The legislative, regulatory and cost-motive pressure on banks is to get out of the mortgage-servicing business. But those same legislative and regulatory pushers aren’t very happy that the result is that nonbank servicers and hedge funds are ready to jump in and take over mortgage servicing.

Federal and state regulators fret about oversight, capital requirements, and customer care in a manner that makes you wonder if they understand the business. One panelist said he didn’t think the Consumer Financial Protection Bureau even knows the difference between holding servicing rights and actually servicing.

Yes, there were something close to $1.03 trillion in mortgage servicing rights sold last year, and MSRs have grown into a $10 trillion market. Nonbank servicers own $1.4 trillion worth of MSRs, an increase of 69% over the past three months.

Of the 30 largest mortgage servicers, nonbank servicers represented 17% of all firms at the end of calendar 2013, up from 9% year-over-year. Mortgage servicing rights at U.S. commercial banks and thrifts have declined by more than half since their peak in mid-2008.

The Mortgage Bankers Association is keeping a close watch on ongoing regulatory activities aimed particularly at nonbank servicers, and they have their work cut out for them.

The CFPB was cited as a major concern at the conference, coming up in almost every session and dominating a few that weren’t really supposed to focus on regulations. The effect of a number of CFPB mortgage rules has made working the space look like a downhill slalom course.

Then there’s Benjamin Lawsky, superintendent of the New York Department of Financial Services, who is coming after nonbank mortgage servicers arguing that they are growing too fast and regulators need to protect homeowners by making sure these companies can handle the volume of business they are taking on.

Lawsky’s DFS put an indefinite freeze on the $2.7 billion MSR deal between Ocwen Financial Corp. (OCN) and Wells Fargo (WFC).

The Financial Stability Oversight Council, a collection of federal regulators from the Federal Reserve and the U.S. Department of Treasury, said in their 2013 annual report that banks are “reducing their mortgage servicing activity and removing potential risk from their balance sheets by selling servicing rights to nonbank servicers, which are not subject to the same capital requirements as banks."

And there are private sector concerns. Moody’s Investor Services is citing concerns that nonbank mortgage servicing companies, such as Ocwen FinancialNationstar (NSM) and Walter Investments (WAC), may begin originating nonprime mortgages, or grow existing originations platforms, even as they grow in the MSR space.

(Few in the public or private sector acknowledge that Ocwen has lead the way in loan modifcations under the Home Affordable Modification Program, and that Nationstar has helped 371,000 borrowers over the last four years.

All these new rules from Washington and the uncertainty out of the NY DFS on how mortgage servicers serve borrowers makes serving in the space all the more costly. (The DFS, though a state agency, has a huge reach given the number of banks chartered out of New York and the language of the DFS’s own charter.)

The regulatory environment is driving banks out of the space and (ironically or naturally depending on your temperament) making the space relatively hostile to those moving in to fill the gap. 

“If I had a choice, I would never be into bulk servicing again," Jamie Dimon, CEO of JPMorgan Chase & Co. (JPM) said in February.

Lawsky, while casting the biggest shadow in terms of uncertainty, seems to have some appreciation that banks are being driven out of servicing and that nonbank servicers are the ones meeting the demand.

“We're not saying that nonbank servicers can't do this. Our concern as a regulator is capacity," Lawsky told the Wall Street Journal a few days ago.

Someone has to do it, he seems to be saying, no matter how difficult state and federal regulators are making the terrain. 

Which brings my question full circle. Why, even though the regulators are moving in and there’s opportunity abounding, would anyone want to jump right into the very snake pit that banks are trying to climb out of?

“We are paid for that additional risk,” one of the attendees said.

Let's hope so. No one I talked to seemed daunted. (Granted, they probably wouldn't have attended the conference if they were on the fence.)

But a last question: What happens if the regulatory environment ends up driving the nonbanks out of the space? Who will step into the breach then?