As the Obama administration works to construct a plan to refinance millions of underwater borrowers into lower-rate mortgages, a Senate subcommittee heard the hidden risks and difficulties of building such a program Wednesday.

Nearly 11 million borrowers currently owe more on their mortgage than the home is worth, according to CoreLogic (CLGX: 14.56 +0.62%). Nearly every panelist testifying Wednesday said these borrowers along with the overhang of more than 1.25 million vacant homes and 3.5 million loans in the foreclosure process to be the obstacle holding back the housing recovery and the overall economy.

Senators Barbara Boxer (D-Calif.) and Johnny Isakson (R-Ga.) reiterated before the subcommittee the need to adopt their plan, which would eliminate the loan-to-value restrictions and fees for refinancing Fannie Mae and Freddie Mac loans into lower rates.

"Our bill is based on a very simple premise: if you have paid your mortgage through this difficult time and it has a high interest rate, but you've never missed a payment and you're underwater, you should be rewarded with a program like this," Boxer said Wednesday. "You should have a chance to refinance at the current levels."

The Congressional Budget Office studied a possible refinance plan and found that such a program would actually save the government-sponsored enterprises about $3.9 billion, though Anthony Sanders, an economist at George Mason University said could be a slight overestimate.

But the program would also cost the Federal Reserve $4.5 billion in the reduction of interest payments on mortgage-backed securities it bought during the credit crisis of 2008, leaving a $600 million net loss to taxpayers.

Getting Congress, which is looking for at the very least $1.5 trillion in deficit reductions before the end of the year, to eat those losses would be the first obstacle.

Rep and warranty risk

David Stevens, the CEO of the Mortgage Bankers Association, told the subcommittee that no proposal addresses the representation and warranty risk of refinancing a mortgage. If the Obama administration's program does not force Fannie and Freddie to waive its right to force lenders to buy back the refinanced mortgage should it slip into default, lenders may not participate.

"All lenders are necessarily cautious with respect to protecting their capital base given the widespread uncertainties in this environment," Stevens said. "MBA believes policy makers should consider setting a clear limit on the duration of an originator’s repurchase obligation following the origination date."

Moody's Analytics Chief Economist Mark Zandi, who supports the Boxer-Isakson bill, agreed.

"I think it would worth thinking about urging Fannie and Freddie to waive rep and warranty claims," he said.

Not enough

Christopher Mayer, a professor of finance and real estate at Columbia Business School said a refinance program developed by other professors at the university would cut about $70 billion in mortgage payments for roughly 25 million borrowers, an average of $2,800 in savings.

"This plan would function like a long-lasting tax cut for these 25 or 30 million American families," Mayer said. "Empirical evidence suggests that consumers spend a larger portion of permanent increases in income than temporary increases."

Mayer and others in support of such a refinancing plan point to the boost to the overall economy getting borrowers into lower-rate mortgages would have.

"The Boxer bill is not about the housing market. It's about trying to create consumption by lowering monthly payment to go out and buy other things," said Mark Calabria, director of financial regulation studies at the Cato Institute. "But a mortgage is one person's liability and another person's asset. So it's not clear to me as an economist if the effect on consumption will be zero."

Meaning if a borrower receives more cash in their pocket from paying a lower rate, the investor in that very mortgage would have less because the basis points on his or her portfolio would be cut.

Sanders agreed: "I'm not sure it's going to have the stimulative effect that it would need."

Let's just revamp HARP

Many panelists suggested the Obama administration should simply revamp and expand the Home Affordable Refinance Program.

HARP launched in March 2009 to allow current borrowers who are up to 125% loan-to-value ratio. So far, roughly 838,400 Fannie and Freddie loans received a refinance, according to data released by the Federal Housing Finance Agency.

But the program was expected to reach between 4 million and 5 million borrowers.

Zandi said the average coupon rate on a Fannie and Freddie security is 4.5%. If that is refinanced down to 4.25%, borrowers would be saving up to $10 billion in interest payments.

"That's not going to solve our problems but it's not insignificant. Roll back the loan level price adjustments, waive rep and warranties, and be proactive. Let borrowers know about this option. Right now they don't do that," Zandi said. "These are reasonable, not difficult to do."

Stevens said the private sector has refinanced roughly 4 million mortgages, nearly four times the amount of public programs. While he warned that using the GSEs to support the housing recovery could only further jeopardize their viability, he did say HARP should be expanded.

"GSEs could still expand lending guidelines, and the origination deadline for HARP qualification could be extended," Stevens said.

Richard Smith, the CEO of Realogy Corporation, which holds a 25% share of the real estate market, said, he too would like to see HARP revised to include more borrowers even before reforming the GSEs themselves.

"The expansion of HARP we're very much in favor of," Smith said. "GSE reform can be handled at a later date."

Action likely to come?

A slew of other obstacles remain for refinancing borrowers so deep underwater. Stevens said existing requirements under the "To-Be-Announced" market and current tax law make pricing securities with loans in excess of 125% LTV nearly "insurmountable."

Calabria said because the Boxer-Isakson bill has no cap on LTV, someone with one as high as 300% could theoretically get a refinance.

"The Federal Housing Finance Agency is having a hard enough time raising the limit on HARP," he said. "I have a hard time seeing it removed all together."

Despite the wave of panelists appearing Wednesday to speak before the subcommittee, there were at most two Senators posing questions against of backdrop of empty seats around them. The scene alluded to the near impossibility of moving yet another major housing program through Congress.

Still, Boxer herself begged lawmakers to do something.

"Let's get in front of this crisis for once," she said.

Write to Jon Prior.

Follow him on Twitter @JonAPrior.