Introduction
Artificial Intelligence (AI) is rapidly changing the way legal professionals conduct research, analyse documents and manage large amounts of information. Courts are also becoming part of this technological transformation. From digital case management to AI-assisted legal research, technology is increasingly being used to make the justice system faster and more accessible. Artificial Intelligence is rapidly transforming the legal system, assisting courts and lawyers with legal research, document analysis, translation, and case management.
However, the use of AI in courts raises a much deeper question: Can artificial intelligence ever replace human legal judgment?
The question is no longer purely theoretical. In Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. (2026), the Supreme Court dealt with a case in which the National Company Law Tribunal (NCLT) had relied upon several authorities that were later found to be non-existent or incorrectly reproduced. The Supreme Court found that some of the material appeared to have been AI-generated and emphasised the serious danger of relying upon such material without verification.
The judgment provides an important starting point for understanding the proper role of AI in the Indian judicial system.
How Is AI Being Used in Indian Courts?
Technology is not new to the Indian justice system. Courts have already adopted electronic filing, video conferencing, digital case records and online access to judgments.
AI represents the next stage of technological development. The Supreme Court’s AI initiatives include tools designed to assist judges with legal research, document analysis, translation, transcription and case management. The Department of Justice has stated that the Supreme Court’s AI Committee has been working on such tools, including Legal Research Analysis Assistant (LegRAA) and Digital Courts 2.1. These systems are being used in controlled settings with attention to privacy and security.
In April 2026, the Supreme Court and the Department of Justice also organised a National Conference on Judicial Process Re-engineering and Digital Transformation. During the conference, Justice Rajesh Bindal emphasised that technology should function as an aid rather than a substitute and should not override judicial reasoning.
Therefore, the present approach is not to reject AI completely but to determine where AI can assist courts and where human judgment must remain central.
The Pooja Ramesh Singh Case
The most important recent development came in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd.
The dispute originated from insolvency proceedings. The NCLT had relied upon six judgments while deciding the matter. During the proceedings before the Supreme Court, it was discovered that some of the cited judgments did not actually exist, while other citations were genuine but had been attributed with paragraphs that could not be found in the original judgments.
The Supreme Court independently examined the authorities and confirmed the problem. The Court found that some of the material was apparently AI-generated or hallucinated.
This was particularly serious because the material had entered the judicial decision-making process itself.
The Supreme Court therefore set aside the orders of the NCLT and NCLAT and directed that the matter be considered afresh.
What Is an AI Hallucination?
An AI hallucination occurs when an AI system generates information that appears convincing but is actually false, inaccurate or unsupported.
In legal research, this can be particularly dangerous.
For example, an AI system may produce:
- a case that does not exist;
- an incorrect citation of a genuine case;
- a paragraph that was never written by the court;
- an incorrect interpretation of a statutory provision; or
- a mixture of information from different judgments.
The problem is that such information may appear professionally written and legally convincing.
A lawyer or judge who fails to verify the information may therefore unknowingly rely upon fiction as if it were legal authority.
Why Is This More Serious in a Court?
A mistake in ordinary research can be corrected relatively easily. A mistake in judicial decision-making has much greater consequences.
Courts determine legal rights and obligations. Their decisions can affect a person’s liberty, property, employment, reputation and other important interests.
For this reason, a judgment cannot be based upon authorities that do not exist.
The Supreme Court in Pooja Ramesh Singh adopted a particularly strict approach. It stated that there should be zero tolerance for AI-generated or hallucinated precedent being cited or relied upon without verification. The Court treated the use of such material as a serious problem affecting the integrity of adjudication.
The Court also made an important distinction: legitimate use of AI is not prohibited. The concern is the presentation of false or hallucinated material as genuine legal authority.
Can AI Understand Law Like a Judge?
AI can process enormous amounts of information within seconds. It can identify patterns, summarise documents and locate potentially relevant authorities.
However, legal judgment involves more than finding information.
A judge must examine the facts of a particular case, understand the arguments of both sides, assess the credibility and relevance of evidence, interpret legislation and apply legal principles to circumstances that may not fit neatly into existing categories.
There may also be constitutional values involved, such as fairness, equality, liberty and human dignity.
These decisions require the exercise of judgment rather than simply the processing of information.
The Supreme Court in Pooja Ramesh Singh recognised this distinction while discussing the growing ability of AI to assist, and potentially substitute for, aspects of human intellectual work. The Court emphasised that judicial reasoning requires careful human thought and discrimination between what is true and what is false.
The Human Element in Judicial Reasoning
Judicial decision-making is not merely a technical exercise.
Two cases may involve the same statutory provision but have very different factual circumstances. A judge must understand the context before deciding how the law should apply.
For example, a court deciding a constitutional case may have to balance competing rights. The answer may not be found in a single precedent or generated through a simple algorithm.
AI can provide information; the judge must decide what that information means in the particular case.
This distinction is important because an AI system may identify a large number of authorities but cannot be allowed to determine the outcome merely on the basis of patterns or probability.
What About Bias in AI?
Another major concern is algorithmic bias.
AI systems are trained using large datasets. If the underlying data contains biases or gaps, the system may reproduce or even amplify those patterns.
This becomes especially sensitive in the justice system.
A technological system used for case classification, risk assessment or other judicial functions could potentially affect how cases are prioritised or understood. The concern is not limited to whether the technology is accurate; courts must also consider fairness, transparency and accountability.
The Supreme Court has therefore been considering safeguards surrounding the use of AI in courts. Its 2026 discussions on judicial digitalisation have emphasised controlled and responsible technological integration.
Privacy and Confidentiality
AI in courts also creates an important data privacy question.
Court records may contain sensitive personal information, medical details, financial information and confidential legal communications. Uploading such material to an external AI system could create risks relating to data storage, access and confidentiality.
The Supreme Court’s AI-related initiatives have therefore focused on controlled systems and the protection of judicial data. The Department of Justice has stated that certain AI tools developed for the judiciary use court-generated data and incorporate privacy and ethical safeguards.
This demonstrates that technological efficiency cannot be considered separately from data protection and confidentiality.
AI as an Assistant, Not a Judge
The emerging Indian approach appears to draw a distinction between AI-assisted justice and AI-driven justice.
AI can potentially assist judges by:
- searching large databases;
- identifying relevant judgments;
- summarising lengthy documents;
- translating legal material;
- converting speech into text;
- identifying procedural information; and
- organising case-related data.
These functions can reduce repetitive work and allow judges and lawyers to spend more time on legal reasoning.
However, the final decision must remain with the human judge.
The Supreme Court’s 2026 conference expressly emphasised technology as an aid rather than a replacement for judicial reasoning.
The Need for Regulation
The growing use of AI makes regulation increasingly important.
The Supreme Court has already invited comments on Draft Regulations for Use of Artificial Intelligence (AI) in Courts, 2026. This indicates that the judiciary is moving towards a more structured framework governing how AI should be used.
Such regulations can potentially address questions such as:
- Who is responsible when AI produces incorrect information?
- Should lawyers be required to disclose the use of AI?
- How should confidential judicial data be protected?
- What level of verification should be required before an AI-generated authority is cited?
- Which judicial functions can safely use AI and which should remain exclusively human?
These questions will become increasingly important as AI systems become more sophisticated.
The Future of AI and the Judiciary
The future is unlikely to be a choice between completely accepting AI and completely rejecting it.
Instead, the legal system will have to determine how AI can be used without compromising the independence, fairness and integrity of judicial decision-making.
The experience of Pooja Ramesh Singh demonstrates why human verification remains essential. Even highly advanced AI systems can produce convincing but incorrect legal material.
At the same time, completely ignoring AI may prevent courts from benefiting from technology that can improve research, accessibility and administrative efficiency.
The challenge, therefore, is to create a system in which technology supports judicial reasoning without replacing it.
Conclusion
Artificial Intelligence has the potential to transform the Indian justice system. It can assist with legal research, document analysis, translation, transcription and case management. The Indian judiciary is already exploring these possibilities through controlled technological initiatives.
However, judicial judgment cannot be reduced to information processing. The decision of a court involves facts, law, constitutional values, human circumstances and reasoned interpretation.
The Supreme Court’s decision in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. provides a significant warning about the risks of blindly relying on AI-generated legal material. A fabricated judgment may look genuine, but once it enters the judicial process, it can undermine the very foundation of adjudication.
The future of AI in courts, therefore, should not necessarily be about replacing judges. It should be about assisting judges while preserving human responsibility for the final decision.
The most important principle may be simple: AI can assist the courtroom, but the judgment must remain human.
