AI-Generated Legal Advice: Liability, Accountability and Professional Ethics

1. Introduction: The Rise of AI in the Legal Profession

Artificial Intelligence (AI) is rapidly transforming the manner in which legal professionals conduct research, analyse information and deliver legal services. Tasks that traditionally required hours of manual research—such as identifying statutory provisions, locating judicial precedents, reviewing contracts, analysing compliance requirements and preparing preliminary legal drafts—can now be performed within seconds with the assistance of generative AI. The growing availability of AI tools has also made legal information more accessible to the general public, allowing individuals to obtain quick and relatively inexpensive responses to legal questions.

However, the increasing use of AI in the legal sector presents a fundamental dilemma: when an AI-generated legal answer is wrong and causes harm, who should be held responsible? Unlike a lawyer, an AI system has no professional licence, no ethical obligation, no independent legal judgment and no conventional legal personality capable of bearing professional responsibility. Consequently, the use of AI cannot become a mechanism for transferring human responsibility to a machine. The central challenge for the modern legal profession is therefore not simply whether AI should be used, but how it can be used while preserving accuracy, confidentiality, professional accountability and the integrity of the justice system.

2. Understanding AI-Generated Legal Advice

AI-generated legal advice refers broadly to legal information, analysis, interpretation or recommendations produced wholly or partly through an artificial intelligence system. Modern large language models are capable of processing enormous quantities of information and generating responses that appear highly authoritative and professionally written. They can assist with legal research, summarisation of judgments, preparation of contractual clauses, compliance checklists, due diligence and preliminary legal drafting.

The difficulty is that AI systems are fundamentally designed to generate plausible responses rather than to function as courts, lawyers or authoritative legal databases. As a result, an AI system may confidently produce an incorrect statutory provision, misunderstand a legal principle, rely upon outdated law or generate a case citation that does not exist. These errors are commonly described as AI hallucinations. In ordinary circumstances, an incorrect answer may simply be inconvenient. In law, however, the consequences can be far more serious. An incorrect limitation period, an outdated regulatory requirement or a fabricated precedent may cause a party to lose a legal remedy, file an incorrect pleading or make a commercially damaging decision.

3. The Problem of AI Hallucinations: When Plausible Information Is Completely False

One of the greatest risks associated with generative AI is that incorrect information is often presented in a confident and convincing manner. Unlike an obvious factual error, an AI hallucination may contain a realistic case name, a plausible citation, a judicial name and even apparently accurate quotations. This makes the error particularly difficult to detect for a person who does not independently verify the information.

The legal profession is especially vulnerable to this problem because legal research depends heavily upon authoritative sources. A lawyer cannot simply rely upon whether a judgment “sounds correct.” The existence, citation, ratio and relevant paragraph of the judgment must be independently verified from an authentic source.

This danger is no longer theoretical. Courts across jurisdictions have already encountered cases where lawyers or litigants relied upon AI-generated fictitious authorities.

4. Real-Life Example: Mata v. Avianca and the Cost of Trusting AI

One of the most famous examples is the 2023 decision in Mata v. Avianca, Inc. before the United States District Court for the Southern District of New York. Lawyers representing the plaintiff submitted a court filing containing several apparently genuine judicial authorities. It was subsequently discovered that the cited cases did not exist and had been generated by ChatGPT. The lawyers had relied upon the AI-generated material without adequately verifying the authorities.

The Court made it clear that there is nothing inherently improper about using technology or even reliable AI tools as an aid to legal research. The problem was the lawyers' failure to perform their professional gatekeeping function. The Court ultimately imposed a $5,000 monetary penalty on the respondents and criticised the submission of fabricated authorities.

The significance of Mata v. Avianca goes far beyond the monetary penalty. It demonstrates a fundamental principle: AI may assist a lawyer, but it cannot replace the lawyer's duty to verify the accuracy of the work product. If a lawyer signs a pleading, files a written submission or makes an argument before a court, professional responsibility remains with the lawyer.

5. A Landmark Indian Development: The Supreme Court's 2026 Decision

The issue has now acquired direct and significant relevance in India. In July 2026, the Supreme Court of India dealt with a case involving reliance by the NCLT and NCLAT upon six citations that were found to be either non-existent or associated with non-existent paragraphs. The Supreme Court examined the use of AI-generated or hallucinated legal material and set aside the orders that had been affected by such material.

The Supreme Court adopted an exceptionally strong position on the issue. It emphasised that courts must maintain zero tolerance towards AI-generated fake or hallucinated precedents being cited or relied upon without verification. It also treated the use of such material by advocates without verification as professional misconduct and stated that reliance upon such material by a court or tribunal seriously compromises the sanctity of adjudication.

This judgment is particularly important because it establishes that the issue is not merely about technological accuracy. It concerns the integrity of judicial decision-making itself. If a court relies upon a judgment that never existed, the problem extends beyond the individual litigant: it affects the credibility of the judicial process and the administration of justice.

6. Liability When AI-Generated Legal Advice Is Incorrect

The question of liability becomes particularly complex when an individual suffers financial or legal loss after relying upon AI-generated advice. The answer will depend upon the circumstances, including who generated the advice, who relied upon it, whether a lawyer was involved, the contractual terms governing the AI service and the applicable law.

Where a lawyer uses AI as a professional tool and subsequently provides the AI-generated conclusion to a client, the lawyer generally cannot escape responsibility simply by stating that the answer came from an AI system. The lawyer remains responsible for exercising professional judgment and verifying the material before communicating it as legal advice.

For example, suppose a lawyer asks an AI system to determine the limitation period applicable to a client's claim. The AI incorrectly provides a limitation period that has been amended by a recent statutory change. If the lawyer accepts the answer without verification and the client subsequently loses the opportunity to file the claim within the correct period, the fact that the error originated with AI would not automatically eliminate the lawyer's professional responsibility.

The precise legal consequences would depend upon the applicable professional, contractual and negligence principles. However, the underlying principle is clear: delegating research to AI does not mean delegating professional responsibility to AI.

7. AI as an Assistant, Not an Autonomous Lawyer

The most appropriate approach is to treat AI as a highly sophisticated legal assistant rather than an autonomous lawyer. AI can assist in identifying issues, summarising large volumes of documents, generating preliminary drafts and suggesting potential authorities. However, the final determination of what the law actually is must remain with a qualified professional.

Consider a simple example. A lawyer may ask AI: “Identify the Supreme Court judgments dealing with a particular provision of the Companies Act.” AI may generate ten possible cases. The lawyer can then use those suggestions as a starting point for research. However, every case must subsequently be checked against an authoritative legal database or the actual judgment before being cited.

The distinction is therefore between AI-assisted research and AI-dependent legal practice. The former can improve efficiency; the latter can create significant professional and legal risks.

8. Human Oversight and Professional Accountability

Human oversight is the most important safeguard against AI-related legal errors. Every material AI-generated legal conclusion should pass through a process of human verification before it is relied upon in professional work.

This should include checking the relevant statutory provision, confirming whether the legislation is currently in force, verifying case citations, reading the actual judgment rather than relying upon an AI summary and ensuring that the legal proposition is applicable to the specific facts of the matter.

This becomes particularly important in litigation. Courts rely upon advocates to present accurate law and credible authorities. If fabricated judgments are introduced into the judicial process, the court may waste valuable time investigating authorities that do not exist, while the opposing party may incur unnecessary costs in responding to false propositions.

Therefore, human verification is not merely a best practice; it is an essential component of responsible AI-assisted legal practice.

9. Professional Ethics and the Duties of Lawyers

The use of AI must also be considered from the perspective of professional ethics. In India, advocates operate within the broader framework of the Advocates Act, 1961 and professional standards governing their conduct. Their duties extend beyond simply achieving a favourable outcome for a client. They include duties towards the client, the court, the opposing party and the administration of justice.

AI does not alter these fundamental obligations.

A lawyer who uses AI remains responsible for ensuring that submissions made to the court are accurate and that advice provided to a client is based upon proper professional judgment. If an advocate knowingly or negligently presents fabricated authorities, misleading information or unverified legal propositions, the fact that AI generated the content cannot automatically excuse the conduct.

The emergence of AI therefore makes professional competence increasingly dependent not only upon knowledge of law but also upon technological literacy. A modern lawyer must understand both the capabilities and limitations of the tools being used.

10. Confidentiality: The Hidden Risk of Uploading Legal Documents to AI

Confidentiality is another major concern. Lawyers routinely handle sensitive information such as contracts, corporate strategies, personal information, financial records, intellectual property, litigation documents and privileged communications. Uploading such information into an AI system without understanding its data-handling practices may create serious risks.

For instance, imagine a lawyer uploads an unpublished merger agreement containing confidential commercial terms into a public AI platform and asks the system to identify potential regulatory concerns. Before doing so, the lawyer should understand how the platform processes the information, whether inputs are retained, whether the information may be used for model improvement, where the data is stored and what security protections apply.

The convenience of obtaining an instant AI-generated analysis cannot justify compromising a client's confidentiality. Legal professionals must therefore assess AI systems from both a legal and cybersecurity perspective before incorporating them into professional workflows.

11. AI and Client Confidentiality: A Practical Scenario

Consider a corporate lawyer advising a company on a proposed acquisition. The transaction involves confidential financial statements, employee information and commercially sensitive negotiations. The lawyer uploads the entire due diligence folder into an AI system without reviewing the platform's data-retention policy.

Even if the AI produces an excellent legal analysis, the lawyer may have created a separate confidentiality and data-protection risk. The question is no longer whether the AI gave the correct legal answer; it is whether the lawyer was entitled to disclose that information to the AI system in the first place.

This illustrates an important principle: an AI system should be evaluated not only for the quality of its output but also for the risks associated with the information provided to it.

12. AI and Access to Justice

Despite these concerns, AI has enormous potential to improve access to justice. Legal services can be expensive and inaccessible to many individuals. AI tools can help people understand basic legal concepts, identify potentially relevant legislation, organise documents and prepare preliminary questions before consulting a lawyer.

For example, a person who receives a legal notice may use AI to understand the basic terminology used in the notice and identify the general area of law involved. This may help the individual have a more informed discussion with a lawyer.

However, legal information is not the same as legal advice. A chatbot may explain what a particular provision generally means, but it may not understand the complete factual circumstances, strategic considerations, evidentiary issues or procedural risks associated with an individual case.

AI can therefore democratise access to legal information without necessarily replacing professional legal representation.

13. The Risk of Over-Reliance and Automation Bias

Another emerging concern is automation bias—the tendency of individuals to trust information simply because it has been generated by a sophisticated technological system.

A lawyer may subconsciously assume that an AI system trained on millions of documents must be more accurate than human research. This assumption can be dangerous. AI may process information at extraordinary speed, but speed does not equal accuracy.

The problem becomes particularly serious when the output is presented in professional language. A fabricated judgment written in convincing legal terminology may appear more credible than an obviously incorrect answer. Consequently, professionals must develop the habit of questioning AI output rather than merely accepting it.

The appropriate mindset should be: “AI gives me a starting point; authoritative sources give me the law.”

14. Regulatory Challenges in India

India does not currently have a single comprehensive legislative framework specifically governing every aspect of AI-generated legal advice. Instead, different legal and regulatory principles may become relevant depending upon the circumstances, including professional conduct rules, contractual principles, consumer protection provisions, intellectual property law and data-protection requirements.

The Digital Personal Data Protection Act, 2023 is particularly relevant where personal data is processed through AI systems. However, questions surrounding professional responsibility, AI-generated misinformation, liability for harmful outputs and the use of AI in judicial and legal processes may require further regulatory development.

A future framework could potentially establish standards relating to transparency, human oversight, auditability, data protection, disclosure of AI use, verification of legal authorities and accountability for AI-assisted professional work.

15. What Should Responsible AI Use in Legal Practice Look Like?

A responsible legal AI framework should follow a simple principle: the more consequential the legal decision, the greater the degree of human oversight required.

For routine tasks such as summarising a publicly available judgment or organising research notes, AI may be used extensively. For higher-risk tasks—such as preparing court pleadings, providing final legal opinions, calculating limitation periods, interpreting complex regulatory provisions or handling confidential client information—human verification should be mandatory.

Law firms and legal departments should ideally establish internal AI policies covering approved AI tools, confidential information, verification requirements, record-keeping, client disclosure and responsibility for final outputs.

16. A Possible Model for AI-Assisted Legal Work

A practical AI-assisted legal workflow can be understood through five stages: Prompt, Review, Verify, Apply and Approve.

First, the professional uses AI to identify potential issues or generate preliminary research. Second, the output is critically reviewed for obvious inconsistencies. Third, every important legal authority is independently verified from an authoritative source. Fourth, the verified law is applied to the actual facts of the case. Finally, the responsible lawyer approves the final advice or document.

This approach preserves the efficiency of AI while ensuring that professional judgment remains at the centre of the process.

17. The Future of AI and the Legal Profession

The future is unlikely to be a simple choice between lawyers and AI. Instead, the legal profession is likely to evolve towards a model in which lawyers who effectively use technology work alongside AI-powered systems.

AI may increasingly perform repetitive research, document review, summarisation and analytical tasks, allowing lawyers to devote more time to strategy, negotiation, advocacy, client counselling and complex legal judgment. However, the value of a lawyer will increasingly depend upon the ability to question, verify and intelligently use AI-generated information.

The lawyer of the future may therefore need to possess three complementary forms of competence: legal knowledge, professional judgment and technological literacy.

18. Conclusion: Accountability Must Remain Human

AI-generated legal advice represents one of the most significant technological developments in modern legal practice. Its ability to reduce costs, accelerate research and expand access to legal information presents enormous opportunities. At the same time, fabricated authorities, outdated law, confidentiality risks and excessive reliance upon automated outputs create serious professional and ethical challenges.

The recent Supreme Court of India's approach to AI-hallucinated precedents demonstrates that these concerns are no longer theoretical. The judiciary has made it clear that the integrity of the adjudicatory process cannot be compromised by unverified AI-generated material.

The lesson is therefore straightforward: AI can assist with legal work, but it cannot assume legal responsibility. A lawyer may delegate a task to technology, but cannot delegate the professional duty to verify, evaluate and take responsibility for the final result.

The future of AI in law should consequently not be measured by how effectively machines can replace lawyers, but by how effectively technology can augment human legal expertise without undermining professional ethics, confidentiality, accountability and the administration of justice. The ultimate objective should be a legal system in which technology makes lawyers more efficient, legal information more accessible and justice more effective—while ensuring that, whenever something goes wrong, responsibility remains clearly attributable to the human decision-makers who chose to rely upon the technology.

Disclaimer: The information published on this website is provided for general informational and educational purposes only and does not constitute legal advice, a legal opinion, or a substitute for professional consultation. While reasonable efforts are made to ensure accuracy, the content may contain errors, omissions, outdated information or incorrect legal references. Readers are advised to independently verify the information and consult a qualified legal professional before relying upon it. No advocate-client or professional relationship is created merely by accessing or using this website.

Prerna Yadav

Legalmantra.net

Print