More than two million homeowners remain eligible for the government's Home Affordable Refinance Program, making them a key part of the market as refi volumes drop overall.

As a whole, refinance applications have slipped in recent weeks, according to data from a mortgage industry trade group. But, the percentage of HARP loans compared to non-HARP applications escalated at the same time, explained Mike Fratantoni, vice president of single-family research with the Mortgage Bankers Association.

A survey by loanDepot found that 51% of HARP borrowers were turned down at least once before successfully refinancing their mortgages.

"Many underwater HARP-eligible homeowners have been turned down before, and they now assume they can't get help," said Jim Svinth, chief economist for loanDepot.

"Requirements have changed, making it possible for loanDepot to give financial relief to more homeowners. If a homeowner is current on their mortgage payments, and the loan is owned by Fannie Mae or Freddie Mac, help is available," Svinth added.

Since the program was announced in 2011, 87% of borrowers have managed to save $1,200 or more a year after refinancing, loanDepot explained.

But as time continues, the pool of borrowers is expected to slip as rates rise again.

"The program was extended to the end of 2015, but the numbers will begin to dwindle by the end of next year as rates go up. And at some point, it is not attractive to even someone that is underwater," Fratantoni said.

WIth mortgage rafts edging closer to 5% in May, HARP demand fell. "But now that rates are back closer to 4.5%, I would not be surprised if lenders turn back to the list of who could be the next potential HARP borrower," Fratantoni added.