The five biggest mortgage servicers have cancelled a planned negotiating session with representatives of the 50 State Attorneys General in apparent protest over a federal regulator filing suit against them, a source familiar with the matter tells TIME. The banks canceled the meeting on Tuesday afternoon in protest over the announcement last Friday that the Federal Housing Finance Agency would bring a broad case against 17 firms, including those in talks with the State AGs. The FHFA, which oversees mortgage giants Fannie Mae and Freddie Mac, alleges the firms violated securities law by misrepresenting the value of bundles of high-risk mortgages they sold. FHFA did not say how much the case might be worth, but outside analysts have said it could potentially produce billions of dollars in compensatory damages from the firms.