On Monday, Raj Date, deputy director of the Consumer Financial Protection Bureau, speaking at the American Bankers Association conference in Orlando, Fla., about the bureau’s mission, resorted to a common argument when discussing the financial crisis:

“If you think back to the most problematic vintages of mortgages during the bubble, most of those problematic mortgages were originated not by supervised banks, but by mortgage brokers and finance companies who then sold those loans into capital market execution on Wall Street.”

And then he said this:

“Let me give you an example from the mortgage bubble: the yield-spread premium. Too often it was the case that mortgage brokers were paid more to give borrowers a worse deal. If a borrower could qualify for a loan at, say, 6%, a broker might juice that rate from 6% up to 8%. As a result, the most important, most visible person in the mortgage process for many borrowers — the mortgage broker — had a financial stake that was confusingly and perversely in direct opposition to the interest of the consumer himself.”

“If people are paid to treat customers poorly, it shouldn’t be surprising when they do," Date said.

Mortgage-broker bashing? No surprises there.

Date’s comments immediately brought to mind not just a story I wrote in April, but also HousingWire readers’ reaction to it. The story was about a letter JPMorgan Chief Executive Jamie Dimon wrote to shareholders in which he admits his bank contributed to the collapse of the American housing industry.

Dimon points out that early on in the crisis, JPMorgan stopped dealing with mortgage brokers, some of whom, he claims, “underwrote the worst of the mortgages and probably mis-sold mortgages more than most.”

Of course, many mortgage brokers responded sharply.

Marc Savitt, president of The Mortgage Center and the National Association of Independent Housing Professionals (and past president of the National Association of Mortgage Brokers) wrote me. “Perhaps Mr. Dimon is unaware, mortgage brokers don’t underwrite mortgage loans, but Chase underwriters do," Savitt said. "Brokers sold his products and under his guidelines. Jamie needs to stop pointing fingers and accept full responsibility for what Chase did during the housing crisis.”

John Hudson, from San Antonio, Texas-based Premier Nationwide Lending and member of NAMB, said, “This is amusing — perhaps Mr. Dimon needs to be reminded that mortgage brokers do not underwrite or fund mortgages…and that Chase was one of the largest pushers of subprime mortgages.”

However you feel about the blame placed on brokers, it’s clear that they — four years later — are still fighting an onslaught of accusations that seem to paint all brokers as one of the causes of the housing debacle as opposed to a minority.

The onslaught includes powerful players such as Dimon, who on Wednesday will attempt to explain to the Senate Banking Committee how and why, under his watch, the firm lost at least $3 billion on a bad trade. And it also has the CFPB, which is working to install regulations that prevent JPMorgan-like financial implosions, on its team, batting cleanup.

Brokers are slowly growing in numbers (NAMB eventually fell to 5,100 members from 25,000 in 2006). “We’re starting to see increased interest in being a mortgage broker and from wholesale lenders who are coming back that were gone,” NAMB President Donald Frommeyer says. “Some of the goals of these wholesale lenders is volume and the only way they’ll get the volume numbers is to use brokers.”

Ultimately, the quality of that volume will determine the strength and length of their comeback.