To say a company survived the economic crisis is a giant understatement. But to think someone can up and start a new mortgage lender today, experts say, is a pipe dream.

Over the past seven years, mortgage lenders and servicers didn’t just survive, they battled every year to stay alive.

The financial crisis weeded out a lot of the lenders and servicers in the industry that brought on the destruction of housing and borrowers, leaving the loan survivors of those who strategized, executed and successfully implanted their battle plans.

Where does this leave the companies that decide to embark into the housing industry now?

During the Goldman Sachs (GS) Housing Finance Conference Thursday, David Spector, president and chief operating officer with PennyMac Mortgage Trust (PMT) said that the biggest barrier to entering the housing industry “is understanding the market that you are operating in today.”

“If you are going to start a business today and expect to hit it out of the park, you are going to be very disappointed,” said Spector.

A strong company begins with a foundation of a good team that has been in the industry a long time and understands it, he explained.

A significant key for the lenders that exist now is that they have been around for an average of seven years. They have attacked every new regulation and rule that has been put in front of them, and so far, they are winning.

To put it in perspective, Anthony Hsieh, chairman and CEO of loanDepot, said in the session that they have from six to 12 audits going on ever day, ranging from states, the Consumer Financial Protection Bureau and regulators.

Hsieh joked that he loves the CFPB and regulators (a comment to which the audience laughed).

But in reality, Hsieh wasn’t being sarcastic.

“I love them because it keeps my competition away. The barrier to entry is so difficult. If it wasn’t there would be a lot more companies like loanDepot and PennyMac,” said Hsieh. “Regulation and compliance is a serious thing, but it is just a reality of today’s world.”

But this does not mean the gateway into the housing industry is blocked. It is just difficult to enter a battle when everyone is has already set up their bases.

In a later panel discussion on mortgage innovation, Mike Cagney, CEO, SoFi, which was founded on student loan refinancing products and is moving into the mortgage market, said, “The barriers are out there.” However he added that he believes SoFi is one of those select lenders that can bring about change in housing.