AI Hallucinations and the Misinformation Dilemma

Sharisha Sahay

Research Analyst - Policy & Advocacy, CyberPeace

PUBLISHED ON

May 14, 2025

As AI language models become more powerful, they are also becoming more prone to errors. One increasingly prominent issue is AI hallucinations, instances where models generate outputs that are factually incorrect, nonsensical, or entirely fabricated, yet present them with complete confidence. Recently, ChatGPT released two new models—o3 and o4-mini, which differ from earlier versions as they focus more on step-by-step reasoning rather than simple text prediction. With the growing reliance on chatbots and generative models for everything from news summaries to legal advice, this phenomenon poses a serious threat to public trust, information accuracy, and decision-making.

What Are AI Hallucinations?

AI hallucinations occur when a model invents facts, misattributes quotes, or cites nonexistent sources. This is not a bug but a side effect of how Large Language Models (LLMs) work, and it is only the probability that can be reduced, not their occurrence altogether. Trained on vast internet data, these models predict what word is likely to come next in a sequence. They have no true understanding of the world or facts, they simulate reasoning based on statistical patterns in text. What is alarming is that the newer and more advanced models are producing more hallucinations, not fewer. seemingly counterintuitive. This has been prevalent reasoning-based models, which generate answers step-by-step in a chain-of-thought style. While this can improve performance on complex tasks, it also opens more room for errors at each step, especially when no factual retrieval or grounding is involved.

As per reports shared on TechCrunch, it mentioned that when users asked AI models for short answers, hallucinations increased by up to 30%. And a study published in eWeek found that ChatGPT hallucinated in 40% of tests involving domain-specific queries, such as medical and legal questions. This was not, however, limited to this particular Large Language Model, but also similar ones like DeepSeek. Even more concerning are hallucinations in multimodal models like those used for deepfakes. Forbes reports that some of these models produce synthetic media that not only look real but are also capable of contributing to fabricated narratives, raising the stakes for the spread of misinformation during elections, crises, and other instances.

It is also notable that AI models are continually improving with each version, focusing on reducing hallucinations and enhancing accuracy. New features, such as providing source links and citations, are being implemented to increase transparency and reliability in responses.

The Misinformation Dilemma

The rise of AI-generated hallucinations exacerbates the already severe problem of online misinformation. Hallucinated content can quickly spread across social platforms, get scraped into training datasets, and re-emerge in new generations of models, creating a dangerous feedback loop. However, it helps that the developers are already aware of such instances and are actively charting out ways in which we can reduce the probability of this error. Some of them are:

Retrieval-Augmented Generation (RAG): Instead of relying purely on a model’s internal knowledge, RAG allows the model to “look up” information from external databases or trusted sources during the generation process. This can significantly reduce hallucination rates by anchoring responses in verifiable data.
Use of smaller, more specialised language models: Lightweight models fine-tuned on specific domains, such as medical records or legal texts. They tend to hallucinate less because their scope is limited and better curated.

Furthermore, transparency mechanisms such as source citation, model disclaimers, and user feedback loops can help mitigate the impact of hallucinations. For instance, when a model generates a response, linking back to its source allows users to verify the claims made.

Conclusion

AI hallucinations are an intrinsic part of how generative models function today, and such a side-effect would continue to occur until foundational changes are made in how models are trained and deployed. For the time being, developers, companies, and users must approach AI-generated content with caution. LLMs are, fundamentally, word predictors, brilliant but fallible. Recognising their limitations is the first step in navigating the misinformation dilemma they pose.

References

‍

PUBLISHED ON

May 14, 2025

Related Blogs

Navigating the Treacherous Terrain of Online Romance Scams in India

February 23, 2024

Introduction

Romance scams have been rised in India. A staggering 66 percent of individuals in India have been ensnared by the siren songs of deceitful online dating schemes. These are not the attempts of yesteryears but rather a new breed of scams, seamlessly weaving the threads of traditional deceit with the sinew of cutting-edge technologies such as generative AI and deep fakes. A report by Tenable highlights the rise of romance scams in India, which now combine traditional tactics with advanced technologies like generative AI and deepfakes. Over 69% of Indians struggle to distinguish between artificial and authentic human voices. Scammers are using celebrity impersonations and platforms like Facebook to lure victims into a false sense of security.

The Romance Scam

A report by Tenable, the exposure management company, illuminates the disturbing evolution of these romance scams. It reveals a reality: AI-generated deep lakes have attained a level of sophistication where an astonishing 69 percent of Indians confess to struggling to discern between artificial and authentic human voices. This technological prowess has armed scammers with the tools to craft increasingly convincing personas, enabling them to perpetrate their nefarious acts with alarming success.

In 2023 alone, 43 percent of Indians reported falling victim to AI voice scams, with a staggering 83 percent of those targeted suffering financial loss. The scammers, like puppeteers, manipulate their digital marionettes with a deftness that is both awe-inspiring and horrifying. They have mastered the art of impersonating celebrities and fabricating personas that resonate with their targets, particularly preying on older demographics who may be more susceptible to their charms.

Social media platforms, which were once heralded as the town squares of the 21st century, have unwittingly become fertile grounds for these fraudulent activities. They lure victims into a false sense of security before the scammers orchestrate their deceitful symphonies. Chris Boyd, a staff research engineer at Tenable, issues a stern warning against the lure of private conversations, where the protective layers of security are peeled away, leaving individuals exposed to the machinations of these digital charlatans.

The Vulnerability of Individuals

The report highlights the vulnerability of certain individuals, especially those who are older, widowed, or experiencing memory loss. These individuals are systematically targeted by heartless criminals who exploit their longing for connection and companionship. The importance of scrutinising requests for money from newfound connections is underscored, as is the need for meticulous examination of photographs and videos for any signs of manipulation or deceit.

'Increasing awareness and maintaining vigilance are our strongest weapons against these heartless manipulations, 'safeguarding love seekers from the treacherous web of AI-enhanced deception.'

The landscape of love has been irrevocably altered by the prevalence of smartphones and the deep proliferation of mobile internet. Finding love has morphed into a digital odyssey, with more and more Indians turning to dating apps like Tinder, Bumble, and Hinge. Yet, as with all technological advancements, there lurks a shadowy underbelly. The rapid adoption of dating sites has provided potential scammers with a veritable goldmine of opportunity.

It is not uncommon these days to hear tales of individuals who have lost their life savings to a person they met on a dating site or who have been honey-trapped and extorted by scammers on such platforms. A new study, titled 'Modern Love' and published by McAfee ahead of Valentine's Day 2024, reveals that such scams are rampant in India, with 39 percent of users reporting that their conversations with a potential love interest online turned out to be with a scammer.

The study also found that 77 percent of Indians have encountered fake profiles and photos that appear AI-generated on dating websites or apps or on social media, while 26 percent later discovered that they were engaging with AI-generated bots rather than real people. 'The possibilities of AI are endless, and unfortunately, so are the perils,' says Steve Grobman, McAfee’s Chief Technology Officer.

Steps to Safeguard

Scammers have not limited their hunting grounds to dating sites alone. A staggering 91 percent of Indians surveyed for the study reported that they, or someone they know, have been contacted by a stranger through social media or text message and began to 'chat' with them regularly. Cybercriminals exploit the vulnerability of those seeking love, engaging in long and sophisticated attempts to defraud their victims.

McAfee offers some steps to protect oneself from online romance and AI scams:

Scrutinise any direct messages you receive from a love interest via a dating app or social media.
Be on the lookout for consistent, AI-generated messages which often lack substance or feel generic.
Avoid clicking on any links in messages from someone you have not met in person.
Perform a reverse image search of any profile pictures used by the person.
Refrain from sending money or gifts to someone you haven’t met in person, even if they send you money first.
Discuss your new love interest with your trusted friend. It can be easy to overlook red flags when you are hopeful and excited.

Conclusion

The path is fraught with illusions, and only by arming oneself with knowledge and scepticism can one hope to find true connection without falling prey to the mirage of deceit. As we navigate this treacherous terrain, let us remember that the most profound connections are often those that withstand the test of time and the scrutiny of truth.

References

‍

Big Tech vs Government: The Rising Tensions Over Digital Freedom in India

April 2, 2025

Introduction

Big Tech has been pushing back against regulatory measures, particularly regarding data handling practices. X Corp (formerly Twitter) has taken a prominent stance in India. The platform has filed a petition against the Central and State governments, challenging content-blocking orders and opposing the Center’s newly launched Sahyog portal. The X Corp has furthermore labelled the Sahyog Portal as a 'censorship portal' that enables government agencies to issue blocking orders using a standardized template.

The key regulations governing the tech space in India include the IT Act of 2000, IT Rules 2021 and 2023 (which stress platform accountability and content moderation), and the DPDP Act 2023, which intersects with personal data governance. This petition by the X Corp raises concerns for digital freedom, platform accountability, and the evolving regulatory frameworks in India.

Elon Musk vs Indian Government: Key Issues at Stake

The 2021 IT Rules, particularly Rule 3(1)(d) of Part II, outline intermediaries' obligations regarding ‘Content Takedowns’. Intermediaries must remove or disable access to unlawful content within 36 hours of receiving a court order or government notification. Notably, the rules do not require government takedown requests to be explicitly in writing, raising concerns about potential misuse.

X’s petition also focuses on the Sahyog Portal, a government-run platform that allows various agencies and state police to request content removal directly. They contend that the failure to comply with such orders can expose intermediaries' officers to prosecution. This has sparked controversy, with platforms like Elon Musk’s X arguing that such provisions grant the government excessive control, potentially undermining free speech and fostering undue censorship.

The broader implications include geopolitical tensions, potential business risks for big tech companies, and significant effects on India's digital economy, user engagement, and platform governance. Balancing regulatory compliance with digital rights remains a crucial challenge in this evolving landscape.

The Global Context: Lessons from Other Jurisdictions

The ‘EU's Digital Services Act’ establishes a baseline 'notice and takedown' system. According to the Act, hosting providers, including online platforms, must enable third parties to notify them of illegal content, which they must promptly remove to retain their hosting defence. The DSA also mandates expedited removal processes for notifications from trusted flaggers, user suspension for those with frequent violations, and enhanced protections for minors. Additionally, hosting providers have to adhere to specific content removal obligations, including the elimination of terrorist content within one hour and deploying technology to detect known or new CSAM material and remove it.

In contrast to the EU, the US First Amendment protects speech from state interference but does not extend to private entities. Dominant digital platforms, however, significantly influence discourse by moderating content, shaping narratives, and controlling advertising markets. This dual role creates tension as these platforms balance free speech, platform safety, and profitability.

India has adopted a model closer to the EU's approach, emphasizing content moderation to curb misinformation, false narratives, and harmful content. Drawing from the EU's framework, India could establish third-party notification mechanisms, enforce clear content takedown guidelines, and implement detection measures for harmful content like terrorist material and CSAM within defined timelines. This would balance content regulation with platform accountability while aligning with global best practices.

Key Concerns and Policy Debates

As the issue stands, the main concerns that arise are:

The need for transparency in government orders for takedowns, the reasons and a clear framework for why they are needed and the guidelines for doing so.
The need for balancing digital freedom with national security and the concerns that arise out of it for tech companies. Essentially, the role platforms play in safeguarding the democratic values enshrined in the Constitution of India.
This court ruling by the Karnataka HC will have the potential to redefine the principles upon which the intermediary guidelines function under the Indian laws.

Potential Outcomes and the Way Forward

While we wait for the Hon’ble Court’s directives and orders in response to the filed suit, while the court's decision could favour either side or lead to a negotiated resolution, the broader takeaway is the necessity of collaborative policymaking that balances governmental oversight with platform accountability. This debate underscores the pressing need for a structured and transparent regulatory framework for content moderation. Additionally, this case also highlights the importance of due process in content regulation and the need for legal clarity for tech companies operating in India. Ultimately, a consultative and principles-based approach will be key to ensuring a fair and open digital ecosystem.

References

Digital Arrest Fraud

December 7, 2023

Introduction

In the rapidly evolving landscape of cyber threats, a novel menace has surfaced the concept of Digital Arrest. The impostors impersonating law enforcement officers deceive the victims into believing that their bank account, SIM card, Aadhaar card, or bank card has been used unlawfully. They coerce victims into paying them money. Digital Arrest involves the virtual restraint of individuals. These suspensions can vary from restricted access to the account(s), and digital platforms, to implementing measures to prevent further digital activities or being restrained on video calling or being monitored through video calling. In the era of digitisation where the technology is growing on an exponential phase, various existing loopholes are being utilised by the wrongdoers which has given rise to this sinister trend known as “digital arrest fraud”. In this scam, the defrauder manipulates the victims, who impersonate law enforcement officials and further traps the victims into a web of deception involving threats of imminent digital restraint and coerced financial transactions.

Recognizing the Danger of Digital Arrest

A recent case involving an interactive voice response (IVR) call that targeted a victim sheds light on the complexities of the "digital arrest" cybercrime. The victim was notified by the scammers—who were pretending to be law enforcement officers—that a SIM card in her name had apparently been utilised in a criminal incident in Mumbai. The call proceeded to a video conversation with an FBI agent who falsely accused her of being involved in money laundering. The victim was forced into a web of dishonesty because she now believed she was involved in a criminal case, underscoring the psychological manipulation these hackers were using.

Recent incidents of digital arrest fraud

Recently, a complaint was registered at the Noida Cyber Crime Police Station made by a 50-year-old victim, who was deceived of over Rs 11 lakh and exposed to "digital arrest". By using the identities of an IPS officer in the CBI and the founder of an airline that was grounded, the attackers, masquerading as law enforcement officers, falsely accused the victim of being involved in a fake money-laundering case. She was told that she had another SIM card in her name that was used for fraudulent activities in Mumbai. The complaint made by the victim asserted “Victim’s call was transferred to a person (who identified himself as a Mumbai Police officer) who conducted the initial interrogation over the call and then on Skype VC, where she stayed from 9:30 AM to around 7 in the evening. The woman ended up transferring around ₹11.11 lakh. The scammers then ended contact with her, after which she realised she had been scammed.
Another recent case of digital arrest fraud came from Faridabad. Where a 23-year-old girl got a call from a fraudster posing as a Lucknow customs officer. The caller said that a package was being shipped to Cambodia that included cards and passports associated with the victim's Aadhaar number. The victim was forced to believe that she was a part of illegal activity, which included trafficking in humans. Under the guise of police officials, the hackers made up allegations before extorting money from the victim. After that, she was told by a man acting as a CBI official that she needed to pay five per cent of the total which was Rs 15 lakh. She said the cybercriminals instructed her not to log off Skype. In the meantime, she ended up transferring Rs 2.5 lakh to a bank account shared by cybercriminals.

Measures to protect oneself from digital arrest

Sustaining a practical and observant approach towards cybersecurity is the key to lowering the peril of being targeted and experiencing digital arrest. Following are certain best practices for ensuring the same:

Cyber Hygiene: This includes maintaining cyber hygiene by regularly updating passwords, and software and also enabling two-factor authentications to reduce the chances of unauthorized access.

Phishing Attempts: These can be evaded by refraining from clicking on dubious links or downloading attachments from unknown sources and also authenticating the legitimacy of emails and messages before sharing any personal information.

Secured devices: By installing reputable antivirus and anti-malware solutions and keeping operating systems and applications up to date with the latest security protocols.

Virtual Private Networks (VPNs): VPNs can be employed to encrypt internet connections thus enhancing privacy and security. However one must be cautious of free VPN services and OTP only for trustworthy providers.

Monitor online services: A regular review of online accounts for any unauthorized or unlawful activities and setting up alerts for any changes to account settings or login attempts may help in the early detection of cybercrime and coping with it.

Secure communication channels: Using secure communication techniques such as encryption can be done for the protection of sensitive information. Sharing of passwords and other information must be cautiously done especially in public forums.

Awareness: The increasing prevalence of cybercrime known as "digital arrest" underscores the need for preventive measures and increased public awareness. Educational initiatives that draw attention to prevalent cyber threats—especially those that include law enforcement impersonation—can enable people to identify and fend off scams of this kind. The collaboration of law enforcement agencies and telecommunication companies can effectively limit the access points used by fraudsters by identifying and blocking susceptible calls.

Conclusion

The rise of Digital Arrest presents a noteworthy and innovative threat to cybersecurity by taking advantage of people's weaknesses through deceitful impersonation and coercive measures. The case in Noida is a prime example of the boldness and skill of cybercriminals who use fear and false information to trick victims into thinking they are in danger of suffering harsh legal repercussions and taking large amounts of money. In order to combat this increasing cybercrime, people need to take a proactive and watchful stance when it comes to cybersecurity. Cyber hygiene techniques, such as two-factor authentication and frequent password changes, are essential for lowering the possibility of unwanted access. Important precautions include being aware of phishing efforts, protecting devices with reliable antivirus software, and using Virtual Private Networks (VPNs) to increase privacy. Cybercriminals and fraudsters often use fear as a powerful tool to manipulate people and exploit their vulnerabilities for illicit gains in the realms of cybercrime and financial fraud. To protect themselves against the sneaky threat of Digital Arrest, netizens must traverse the constantly changing cyber threat landscape with collective knowledge, educated practices, and strong cybersecurity measures.

AI Hallucinations and the Misinformation Dilemma

What Are AI Hallucinations?

The Misinformation Dilemma

Conclusion

References

Related Blogs

Introduction

The Romance Scam

The Vulnerability of Individuals

Steps to Safeguard

McAfee offers some steps to protect oneself from online romance and AI scams:

Conclusion

References

Introduction

Elon Musk vs Indian Government: Key Issues at Stake

The Global Context: Lessons from Other Jurisdictions

Key Concerns and Policy Debates

Potential Outcomes and the Way Forward

References

Introduction

Recognizing the Danger of Digital Arrest

Recent incidents of digital arrest fraud

Measures to protect oneself from digital arrest

Conclusion

References:

Become a part of our vision to make the digital world safe for all!

Awareness

Engagement

Play your part for CyberPeace