Originations for borrowers with credit scores below 620 mostly disappeared in recent years, eliminating low credit scores from the housing market, Elizabeth Duke, member of the Board of Governors for the Federal Reserve System, said while speaking at the Housing Policy Executive Council.

Previously, Duke said, "Borrowers with lower credit scores have typically represented a significant segment of first-time homebuyers."

Between 2007 and 2012, originations for borrowers with a credit score between 620 and 680 tumbled nearly 90%, compared to a 30% drop for borrowers with a score greater than 780.

Meanwhile, the median credit score skyrocketed from 730 in 2007 to a whopping 770 in 2013.

With few lending channels, borrowers have turned to mortgages insured or guaranteed by the Federal Housing Administration, the Department of Veteran Affairs and the Rural Housing Service, Duke said.

As a result, the share of purchase mortgages guaranteed or insured by the FHA, the VA, or the RHS escalated from 5% in 2006 to more than 40% in 2011.

According to Duke, much of the dip in the average credit score can be attributed to tighter credit conditions and a rise in unemployment rates.

In addition, according to the Federal Reserve's April survey of loan officers and bank lenders, banks are less likely to originate mortgages with FICO scores in the 620 range, even with a 10% or 20% down payment.

"The path to easier credit conditions is somewhat murky," Duke explained. "Some of the forces damping mortgage credit availability, such as capacity constraints and concerns about economic conditions or house prices, are likely to unwind through normal cyclical forces."