California has passed the first major state law specifically regulating how lawyers use artificial intelligence, requiring attorneys to verify AI-generated content for accuracy, protect client confidentiality, and prevent discriminatory outcomes before submitting work to courts. Senate Bill 574 (SB 574), which takes effect soon, codifies ethical obligations that already exist in the legal profession but applies them directly to AI-related conduct, addressing growing concerns about AI hallucinations, data privacy breaches, and algorithmic bias in legal practice.

What Went Wrong That Prompted This Law?

The legal profession has experienced several high-profile failures with generative AI over the past two years. Attorneys have filed court briefs containing completely fabricated court cases and fake legal citations generated by AI systems like ChatGPT, Gemini, and Claude. In some instances, judges imposed monetary sanctions on the lawyers involved and questioned their professional competence.

The problem stems from how generative AI actually works. Unlike traditional legal research databases, these systems do not truly "understand" the law. Instead, they predict and generate language based on patterns in their training data. This can produce what experts call an AI "hallucination," where the system generates plausible-sounding but entirely false information with complete confidence.

Beyond accuracy concerns, lawyers have also faced criticism for uploading confidential client information into public AI platforms without understanding how that data might be stored, retained, or used to train future models. Additionally, concerns have emerged about bias and discrimination in AI-generated content, particularly when AI tools are used to assist with decisions that affect people's rights and freedoms.

What Does SB 574 Actually Require Attorneys to Do?

• Verify AI Accuracy: Attorneys must take reasonable steps to verify the accuracy of AI-generated material and correct any erroneous or hallucinated content before submitting it to courts or clients.
• Protect Client Confidentiality: The law prohibits attorneys from entering confidential, personally identifying, or other nonpublic information into public generative AI systems, directly addressing concerns about lawyers uploading client documents, medical records, financial information, discovery materials, or litigation strategies into publicly accessible AI platforms.
• Prevent Algorithmic Bias: Attorneys must ensure that their use of AI does not unlawfully discriminate against protected individuals or groups, reflecting broader concerns regarding algorithmic bias and fairness in AI systems.
• Consider Disclosure: The bill requires attorneys to consider whether disclosure is appropriate when AI is used to create content intended for the public, signaling growing concern regarding transparency when AI-generated content is used in professional settings.

One lesser-known aspect of SB 574 is that it also regulates arbitrators. The bill prohibits arbitrators from delegating any part of their decision-making responsibilities to generative AI systems and prevents them from relying on AI-generated information outside the evidentiary record without appropriate disclosure to the parties.

How Should Lawyers Implement These New Requirements?

• Establish Verification Protocols: Create internal procedures for independently checking all AI-generated research, summaries, and citations before using them in legal work, treating AI outputs with the same scrutiny as any other source material.
• Restrict Data Input: Develop clear policies about what types of information can and cannot be entered into AI systems, with particular attention to confidential client data, privileged communications, and personally identifying information.
• Audit AI Tools for Bias: Regularly test and monitor AI systems used in legal practice to identify potential discriminatory outcomes, particularly when AI is used to assist with hiring, client profiling, or other decision-making processes that could affect protected groups.
• Document AI Use: Maintain records of when and how AI was used in legal work, making it easier to demonstrate compliance with ethical obligations and to identify problems if they arise later.

Why Does This Matter Beyond California?

Although SB 574 is a California law, its impact will likely extend far beyond the state. Many law firms operate nationally, and legal ethics rules in one large state often influence developments elsewhere. The legislation essentially sends a message to the entire legal profession that AI is a tool and not a substitute for professional judgment.

For clients, SB 574 provides several important protections. First, it reinforces that lawyers remain fully accountable for the quality and accuracy of their work, meaning clients should not bear the consequences of an attorney blindly relying on flawed AI-generated information. Second, the bill strengthens protections for confidential client information at a time when data privacy concerns continue to grow. Third, the legislation recognizes that AI can create risks of bias, misinformation, and unfair treatment, and it imposes duties upon attorneys to monitor and verify AI-generated material to reduce those risks.

What About the Broader AI Accountability Conversation?

The legal profession's struggle with AI accountability reflects a broader challenge across industries. In Malaysia, for example, the government launched national guidelines on AI governance and ethics in September 2024, outlining seven core principles intended to guide responsible AI development and deployment: fairness, reliability and safety, privacy and security, inclusiveness, transparency, accountability, and pursuit of human benefit.

Malaysia is also developing its first dedicated AI Governance Bill, which will introduce a risk-based model covering areas such as AI-related harm, incident reporting, and ethical principles. The bill will establish responsibilities for entities that develop or deploy AI systems and create a mechanism for reporting AI-related incidents.

Research into AI in judicial systems reveals a fundamental tension between algorithmic consistency and human judgment. A recent study examining AI's role in courtrooms found that individuals from science and technology backgrounds often prioritize the data-driven objectivity that AI offers, while those with humanities backgrounds emphasize due process and moral responsibility. This "perception gap" highlights how different professional communities evaluate the fairness and legitimacy of AI-assisted decision-making in high-stakes settings.

The study noted that while some argue AI can address system overloads such as court backlogs and administrative delays, legal scholars warn that AI use in sentencing and risk assessment raises serious concerns about fairness and transparency. A prominent example is the COMPAS risk assessment algorithm, a proprietary "black box" system whose internal logic remains undisclosed. Investigations into COMPAS have produced contradictory results regarding bias, with some studies finding a 45% false positive rate for Black defendants compared to 23% for White defendants, often stemming from "proxy" data such as housing stability, employment, and education that are deeply correlated with historical discrimination.

Despite these disparities, a mutual consensus exists regarding ethical implications. AI should not replace human judgment in sentencing but serve to enhance rulings. Transparency is an essential condition for ethical use, particularly regarding life and liberty. And an algorithm may be statistically accurate yet still erode public confidence.

As AI technology continues to evolve rapidly, its use within the legal profession is almost certain to expand. SB 574 represents an important step toward ensuring that expansion happens responsibly, with clear rules protecting clients and preserving human oversight over decisions that affect people's rights and freedoms.