If JPMorgan Chase (JPM) and Bank of America (BAC) fail to improve performance in the Home Affordable Modification Program, the amount of payments they would lose totals more than $131 million, according to the Special Inspector General for the Troubled Asset Relief Program. In the middle of last year, the Treasury withheld funds from Chase, BofA and Wells Fargo (WFC) for their poor performance in the program during the first quarter. The Treasury compliance team looked at how each participating servicer performed when contacting homeowners, evaluating them for the program and how they assisted with any questions or document submissions. Wells made immediate improvements the next quarter and the withheld funds were returned. BofA made some headway in the third quarter, enough for the Treasury to say the servicer needed only "moderate improvement" to regain its $63.7 million. Chase however has not made enough progress, and the Treasury warned the bank in December it could permanently withhold $67.3 million owed in HAMP payments. Since its inception in March 2009, just over 900,000 permanent HAMP modifications were granted and will likely fall well short of the 3 million to 4 million originally estimated. Roughly $29.9 billion in TARP funds have been allocated for HAMP and other foreclosure prevention programs, but only $2.3 billion has been spent as of Dec. 31. "JPMorgan Chase's continuing refusal to comply with program requirements is deeply troubling and there must be serious repercussions," SIGTARP said in a report Thursday. "Permanently withholding incentives from a servicer, as recommended by SIGTARP, sends a strong signal to JPMorgan Chase and all other servicers that failure to perform at acceptable levels will result in serious consequences." The watchdog reiterated, as it had since the program first started showing underwhelming signs, that more can be done in the last year of the program to modify as many loans as possible, such as permanently reducing payments in the future and even clawing back funds paid in the past. "SIGTARP continues to urge Treasury to take more aggressive steps to ensure that all servicers act in accordance with the program rules and the contracts they signed, for which they are being paid by the taxpayers," SIGTARP said. Write to Jon Prior. Follow him on Twitter @JonAPrior.