Freddie Mac and the Federal Housing Finance Agency did not implement stronger oversight of mortgage servicers when they had the chance, according to an FHFA Office of Inspector General report.

Freddie contracts with more than 1,400 servicers, but the largest banks handle the majority of its $1.8 trillion portfolio. In 2010, evidence surfaced of mishandled foreclosures, lost paperwork and improper signatures at the big servicing firms. The top five servicers agreed to a $25 billion settlement with the state attorneys general in February.

Freddie did not install its servicing scorecard program until July 2011 and rated the overall performance as below standard or unacceptable, according to its most recent report. Documentation the government-sponsored enterprise provided to the Inspector General shows weak servicer oversight and risk management played a significant role in the subpar performance.

The FHFA agreed with this conclusion, but not enough was done, the report concludes.

"FHFA noted that numerous GSE executives have indicated that servicer performance would be much improved if the enterprises were to take a more aggressive approach with respect to their servicers," according to the report released Tuesday.

Instead, Freddie put "account plans" in place, establishing actions servicers would need to take to improve their scores.

In January, Freddie developed a servicing business plan and projected it would save the GSE an undisclosed amount of losses by urging more servicers to modify mortgages or approve more short sales.

"However, Freddie Mac has not implemented its business plan in its entirety," according to the report. "The enterprise can enhance its results by implementing its loss mitigation goals among all — or at least a greater cross-section — of its servicers."

In a letter to the inspector general, FHFA Deputy Director of Enterprise Regulation Jon Greenlee wrote the FHFA has been proactive in its supervision of Freddie. However, he did say "a dedicated examination staff to monitor and coordinate oversight activities would enhance the agency's overall efficiency and effectiveness."

"FHFA's ability to implement this authority may be impaired without direct access to evidence of unsound practices or non-compliance with applicable standards," according to the inspector general.

Greenlee said the FHFA would consider asking Congress for legislation to strengthen its ability to examine GSE servicers. He said such a law change could provide the FHFA the same authority and rights federal regulators have.

He also said the agency is conducting a review of a key, third-party service provider for one of the GSEs and will give a final update in January 2013.

"Through its ongoing supervisory process, FHFA will continue to make Freddie Mac's oversight of servicers a key priority to determine if risks are being properly managed and that credit loss savings are reasonably achieved," Greenlee said.