A new mortgage refinance plan the White House will send to Congress would cost the largest banks between $5 billion and $10 billion.

The plan, first announced in President Obama's State of the Union address last week, would allow private-label mortgages, those not guaranteed by Fannie Mae and Freddie Mac, to refinance through the Federal Housing Administration.

"I want to be clear: this plan, like the other actions we’ve taken, will not help the neighbors down the street who bought a house they couldn’t afford, then walked away and left a foreclosed home behind.  It will not help those who bought multiple homes just to speculate and make a quick buck," Obama said in a speech at Falls Church, Va. Wednesday. "What this plan will do is help millions of responsible homeowners who make their payments on time but find themselves trapped under falling home values or wrapped in red tape."

Only borrowers who have been current on their mortgage for the past six months will qualify and cannot have missed more than one payment over the past year. Eligible borrowers must have a credit score of at least 580. According to the White House, roughly nine in 10 borrowers have a credit score to meet the criteria.

The current mortgage cannot exceed the FHA conforming loan limits which were restored to the previously elevated level in November. In the lowest cost areas, the mortgage cannot exceed $271,050 and cannot be more than $729,750 in the most expensive neighborhoods.

The White House also said Tuesday it would work with Congress to establish clearer guidelines for banks wanting to refinance borrowers deep underwater, those with loan-to-value ratios above 140%.

A separate fund will be created for the FHA, so that the refinance program will not effect its Mutual Mortgage Insurance fund, which is already in danger of reaching negative territory.

The fee charged to banks will be based on "their size and riskiness of their activities," according to the White House.

Appraisal costs were eliminated. And the plan directs Fannie and Freddie to require the same process of new servicers as larger ones to provide more competition for borrowers searching for the lowest rate.

Last year, the Federal Housing Finance Agency eliminated restrictions on the Home Affordable Refinance Program to help more underwater borrowers refinance. To date, the program has allowed more than 900,000 borrowers refinance into lower-rate loans.

Isaac Boltansky, an analyst at Compass Point, a research firm in Washington, said estimates show another 3.5 million borrowers would be eligible for the new program, in addition to a new HARP.

"Opponents will coalesce around their opposition to expanding the FHA's balance sheet and, thereby, the expansion of the government's risk exposure to the mortgage market," Boltansky said.