The mortgage industry has severed thousands of jobs due to the significant drop in refinance applications. USA Today posted an article overlooking just how significant the wound in job layoffs is, with the sector announcing 49,000 layoffs in the first nine months of 2013.

The refinancing downturn is not the only reason banks are retrenching. With businesses uncertain about the economy, interest rates and policies in Washington, trading and mergers have slowed, says Jeff Harte, an analyst at Sandler O'Neill.

Meanwhile, Harte says, new rules have cut the debit card fees banks charge retailers and limited overdraft charges. And the Dodd-Frank reform is adding costs as banks meet stricter standards.