The suit involves a $369 million pot of cash the state of California received as part of the settlement in 2012 between 49 states and the country's five largest mortgage servicers.
At the time, California was facing a state budget shortfall. Brown diverted most of the $369 million away from a housing-related fund in order to help plug the budget gap, according to the lawsuit.
The lawsuit's plaintiffs are the National Asian American Coalition, the National Hispanic Christian Leadership Conference, and Cor Community Development Corp., a group that's affiliated with thousands of black churches.
They allege that Brown's decision violated state law. Their suit also charges that the governor's budget maneuver impinged on the state attorney general's authority to settle lawsuits on behalf of the state.
The plaintiffs are asking that the money be returned to its original fund, in order to be used for mortgage counseling and other housing-related purposes such as Short Sale Assistance, Loan Modification Assistance, Principal Reduction and Deficiency Waivers.
Since 2012, California's fiscal outlook has improved considerably, and the state currently has a budget surplus.
"The governor had no legal right to divert these funds in the first place and, even if he did, he certainly has the statutory duty to replenish them in this year of surplus," the lawsuit states.
The governor's office referred comment on the lawsuit to the California Department of Finance. A spokesman for that agency said in an email: "While we haven't yet seen the complaint, we're confident that our budget actions are legally sound."
A spokesman for Attorney General Kamala Harris declined to comment on the suit.
Initially, Harris, a Democrat who negotiated a hefty share of the national mortgage settlement for California, spoke out against the governor's decision to divert the funds.
"While the state is undeniably facing a difficult budget gap," the attorney general said in a May 2012 Facebook post, "these funds should be used to help Californians stay in their homes."
But Harris has not argued publicly that the governor's decision was unlawful.
The lawsuit was spearheaded by longtime public interest lawyer Robert Gnaidza and Neil Barofsky, the former special inspector general for the Troubled Asset Relief Program who has loudly criticized the Obama administration for failing to provide more assistance to victims of the foreclosure crisis.
During a conference call with reporters, Gnaidza acknowledged that Harris finds herself in a difficult position.
"She did a better job than any other AG in the country in getting a good settlement," he said. "But I think she also feels a loyalty as attorney general to the governor."
Faith Bautista, president of the National Asian American Coalition, said she's focused on getting assistance for the two million households that still face the prospect of losing their homes. "Homeowners need help," she said.
Other states besides California have also used money they received from the mortgage settlement for purposes unrelated to housing.
When asked if he's considering suing other states, Gnaidza said: "There are at least 15 states, such as Georgia and Nevada, where this could be done. I suspect there will be other suits within a few months."
I can help you short sale your property and get back on your feet.