#FactCheck-AI-Generated Video Falsely Shared as Delivery Boy Saving Child From Speeding Truck in Mumbai

As AI language models become more powerful, they are also becoming more prone to errors. One increasingly prominent issue is AI hallucinations, instances where models generate outputs that are factually incorrect, nonsensical, or entirely fabricated, yet present them with complete confidence. Recently, ChatGPT released two new models—o3 and o4-mini, which differ from earlier versions as they focus more on step-by-step reasoning rather than simple text prediction. With the growing reliance on chatbots and generative models for everything from news summaries to legal advice, this phenomenon poses a serious threat to public trust, information accuracy, and decision-making.

What Are AI Hallucinations?

AI hallucinations occur when a model invents facts, misattributes quotes, or cites nonexistent sources. This is not a bug but a side effect of how Large Language Models (LLMs) work, and it is only the probability that can be reduced, not their occurrence altogether. Trained on vast internet data, these models predict what word is likely to come next in a sequence. They have no true understanding of the world or facts, they simulate reasoning based on statistical patterns in text. What is alarming is that the newer and more advanced models are producing more hallucinations, not fewer. seemingly counterintuitive. This has been prevalent reasoning-based models, which generate answers step-by-step in a chain-of-thought style. While this can improve performance on complex tasks, it also opens more room for errors at each step, especially when no factual retrieval or grounding is involved.

As per reports shared on TechCrunch, it mentioned that when users asked AI models for short answers, hallucinations increased by up to 30%. And a study published in eWeek found that ChatGPT hallucinated in 40% of tests involving domain-specific queries, such as medical and legal questions. This was not, however, limited to this particular Large Language Model, but also similar ones like DeepSeek. Even more concerning are hallucinations in multimodal models like those used for deepfakes. Forbes reports that some of these models produce synthetic media that not only look real but are also capable of contributing to fabricated narratives, raising the stakes for the spread of misinformation during elections, crises, and other instances.

It is also notable that AI models are continually improving with each version, focusing on reducing hallucinations and enhancing accuracy. New features, such as providing source links and citations, are being implemented to increase transparency and reliability in responses.

The Misinformation Dilemma

The rise of AI-generated hallucinations exacerbates the already severe problem of online misinformation. Hallucinated content can quickly spread across social platforms, get scraped into training datasets, and re-emerge in new generations of models, creating a dangerous feedback loop. However, it helps that the developers are already aware of such instances and are actively charting out ways in which we can reduce the probability of this error. Some of them are:

1. Retrieval-Augmented Generation (RAG): Instead of relying purely on a model’s internal knowledge, RAG allows the model to “look up” information from external databases or trusted sources during the generation process. This can significantly reduce hallucination rates by anchoring responses in verifiable data.
2. Use of smaller, more specialised language models: Lightweight models fine-tuned on specific domains, such as medical records or legal texts. They tend to hallucinate less because their scope is limited and better curated.

Furthermore, transparency mechanisms such as source citation, model disclaimers, and user feedback loops can help mitigate the impact of hallucinations. For instance, when a model generates a response, linking back to its source allows users to verify the claims made.

Conclusion

AI hallucinations are an intrinsic part of how generative models function today, and such a side-effect would continue to occur until foundational changes are made in how models are trained and deployed. For the time being, developers, companies, and users must approach AI-generated content with caution. LLMs are, fundamentally, word predictors, brilliant but fallible. Recognising their limitations is the first step in navigating the misinformation dilemma they pose.

References

• https://www.eweek.com/news/ai-hallucinations-increase/
• https://www.resilience.org/stories/2025-05-11/better-ai-has-more-hallucinations/ 
• https://www.ekathimerini.com/nytimes/1269076/ai-is-getting-more-powerful-but-its-hallucinations-are-getting-worse/ 
• https://techcrunch.com/2025/05/08/asking-chatbots-for-short-answers-can-increase-hallucinations-study-finds/ 
• https://en.as.com/latest_news/is-chatgpt-having-robot-dreams-ai-is-hallucinating-and-producing-incorrect-information-and-experts-dont-know-why-n/
• https://www.newscientist.com/article/2479545-ai-hallucinations-are-getting-worse-and-theyre-here-to-stay/
• https://www.forbes.com/sites/conormurray/2025/05/06/why-ai-hallucinations-are-worse-than-ever/ 
• https://towardsdatascience.com/how-i-deal-with-hallucinations-at-an-ai-startup-9fc4121295cc/ 
• https://www.informationweek.com/machine-learning-ai/getting-a-handle-on-ai-hallucinations 

‍

Introduction
‍

The recent investigation of Patan Cyber Crime Police as part of Operation Mule Hunt 2.0 reveals the sheer scale and intricacy of India's burgeoning cyber fraud economy. Police found that a total of 13 current accounts were being operated at a cooperative bank in the Patan district of Gujarat and used for siphoning 398.43 crore of cyber fraud transaction data on 228 cybercrime cases across states. Further investigations against 14 current account holders and intermediaries show the indispensability of mule accounts in laundering criminal money. The recent incident cannot be taken as isolated; the story points at a formalised and industrialised fraud economy with a robust banking infrastructure, a growing payment gateway, and complex networks.
‍

What Is a Mule Account and Why Should You Care?
‍

The term "mule account" is benign but plays a critical role in modern cybercrime networks. The Reserve Bank of India defines a mule account as a bank account that serves as a vehicle to transfer money proceeds from unlawful transactions and can be operated by people coerced by the prospect of high earnings or by way of inducement.

This mechanism can be witnessed through the investigation of the Patan cybercrime incident, where an investor can be defrauded by a fake investment website, employment fraud, or a digital arrest scheme. After transactions from the victim account, funds would quickly flow into the mule account, which would be held by a legitimate KYC customer. These transactions would then be passed on, between 1 lakh and 5 lakh transactions within hours, to multiple accounts as alleged by the Indian Cyber Crime Coordination Centre (I4C) before they get difficult to trace by being passed through informal channels or converted to cryptocurrency.

In the Patan case, it is alleged that the middlemen enticed locals and offered commissions to open firms and current accounts at Harij Nagrik Sahakari Bank and subsequently gave up their ATM cards, checkbooks, SIMs, and net banking facilities to the operators of the account. It is estimated that such accounts channeled an amount of 398.43 crore to 228 Indian cybercrime cases.

‍

The Scale of India's Mule Account Crisis

‍

The scale of the mule account ecosystem is reflected in India's rapidly worsening cybercrime statistics. As of data from the National Cyber Crime Reporting Portal (NCCRP), a total of 22.68 lakh complaints were registered in 2024, a jump by 42% from 2023. This was not even half the rate of financial loss, which jumped by 206% in 2023 (22,845 crore) and stood at 22,495 crore in 2025 (complaints jumped to 28.15 lakh). The increase in fraudulent transactions therefore outweighs the stability in financial losses significantly.

Mule accounts are the backbone of this crime network. To curb this phenomenon, the Indian Cyber Crime Coordination Centre (I4C) launched a Suspect Registry along with Indian banks and financial institutions in September 2024. 24.67 lakh accounts of suspected mules were identified in this, preventing over 8,031 crore in fraudulent transactions. Despite these efforts, a recent statement from the ED found over 12,000 crore being routed via mule accounts, shell firms, and cryptocurrency.

This isn't isolated to certain banks. 2024 alone saw over 65,000 mule accounts detected in Karnataka. By analyzing the Citizen Financial Cyber Frauds Reporting and Management System, about 40,000 such accounts were detected in SBI branches, and thousands more were detected across the PNB, Canara Bank, Kotak Mahindra Bank, and Airtel Payments Bank. The Patan case also clearly highlights that cooperative banks' lack of compliance and lower levels of transaction-monitoring systems contribute to easily creating and using mule accounts.

‍

Operation Mule Hunt: Gujarat's Coordinated Offensive

‍

This bust in Patan is just one manifestation of a much wider coordinated effort by the state government. Operation Mule Hunt 1.0, which ran from November to December 2025 across the state of Gujarat, was a month-long campaign by Gujarat Police's Cyber Centre of Excellence (CCOE) that unearthed 2,289 crore of fraudulent transactions, led to the registration of 565 FIRs, arrest of 638 accused, and impounding of 913 mule accounts with connections to over 4,000 cases of cybercrime nationwide.

This was followed up with the second installment of the operation, which was kicked off in all districts of Gujarat in 2026. The two-week campaign, which began across the state on January 8 this year, resulted in the Surat City Police alone arresting 77 people and uncovering close to 23.85 crore in fraudulent transactions. In what looks like one of the single largest single-district bust-ups in the operation, the Patan incident itself, with a staggering 398.43 crore routed through only 13 accounts, is remarkable.

The extraordinary nature of the operation is seen in the intelligence capabilities that drove it. It wasn't that police accidentally stumbled upon the Patan network; they worked back on it. After using data from the union government’s inter-agency platform, SAMANVAYA, a coordination platform for data on cybercrimes and the NCCRP, they traced suspicious clusters of transactions in the Harij Nagrik Sahakari Bank accounts to build a chain of evidence connecting the accountholders to the middlemen and, from the middlemen, to the whole ring of fraud. Twenty accused have been chargesheeted under the Bharatiya Nyaya Sanhita (BNS), and fourteen have been arrested, while six are still absconding.

‍

The Human Cost Nobody Talks About

‍

Behind every crore of scam money lies a real person who actually lost the real money. Of the 75%+ fraud losses incurred in 2025, 75% are from investment scams alone. Victims of stock trading scams lost ₹4,636 crore, spread across 2.28 lakh complaints filed in 2024. "Digital arrest" scams, in which fraudsters posing as law enforcement officials psychologically blackmail the victims to transfer money,  claimed ₹2,576 crore between 2022 and the first quarter of 2025.

For the victims it's never about the money: it's the retired teacher's lifetime savings from Chhattisgarh, the small trader's capital from Rajkot, the emergency money of the Bhopal family, or just savings from an ordinary person. And the mule accounts' networks are why most of it is never retrieved. Once the money is thrown into the layering chain, it's exponentially more difficult to trace it after every jump.

Then there's another category of victims that often gets overlooked, and they are the mule account holders themselves, many being semi-literate people from semi-urban or rural backgrounds approached with ₹10,000 in commission and with no awareness about the legalities of lending their bank details. With the BNS now they stand to get convicted for grave crimes, but the awareness of this trap is very low.

‍

Recommendations and Suggestions

‍

This isn't something India is facing passively. I4C, along with RBI, has developed Mule Account Hunter software. This software can be used by banks for the detection of suspect accounts through the use of behavioral analysis, device intel, and transaction pattern recognition. The Union Home Minister has directly asked all cooperative banks across the country to adopt this software at the earliest. Failure to do so, he warned, would make consumer safety from cyber fraud incomplete.

Apart from technology, three other areas need to go hand in hand: stringent KYC enforcement for cooperative and small finance banks; the prime locations of the mule recruitment network; greater awareness for the masses regarding the criminal liability one takes up when lending their accounts; and efficient inter-agency coordination so that the intelligence gathered on platforms like SAMANVAYA is converted into arrests before the accounts are dumped and the network reforms in another location.

Operation Mule Hunt 2.0 proves that this is feasible. 13 accounts in a small district of Gujarat. 398 crore. 228 victims. 14 arrested. The pipeline did exist, and it has been broken.

Yet, even as one network is broken, another is forming, somewhere right now. The accounts will appear legitimate. The holders of these accounts may not even realize what they have got into. That is the true danger of the mule accounts and work that cannot stop.

‍

Conclusion

‍

The Patan investigation has clearly shown that mule accounts have now moved from being a subsidiary tool of financial crime to becoming the infrastructure that underpins the economy of cyber-fraud in India. Every financial fraud, including investment fraud, digital arrest fraud, and phishing scams, is backed by a string of real bank accounts where the proceeds of crime are transferred and the trail is obscured. Though attempts such as the I4C Suspect Registry have made attempts to break down this network, it remains an overwhelming task. Robust KYC norms, real-time monitoring of transactions, and coordination between banks, police, and regulators are the key in preventing further industrialisation of cyber financial fraud in India.

‍

References

1. https://timesofindia.indiatimes.com/city/ahmedabad/operation-mule-hunt-2-0-gujarat-
2. police-bust-rs-398-43-crore-cyber-fraud-14-held/articleshow/131594240.cms?utm_source=contentofinterest&utm_medium=text&utm_campaign=cppst
3. https://the420.in/india-cybercrime-2024-42-percent-spike-sims-imei-mule-accounts/
4. https://www.thehansindia.com/news/national/ed-explains-how-mule-accounts-and-crypto-networks-enabled-12000-crore-cyber-fraud-1047606
5. https://www.zigram.tech/article/mule-accounts-tier-1-tier-2-cities-india/
6. https://risk.lexisnexis.com/global/en/insights-resources/article/stopping-money-mules-in-india
7. https://timesofindia.indiatimes.com/city/ahmedabad/operation-mule-hunt-2-0-gujarat-police-bust-rs-398-43-crore-cyber-fraud-14-held/articleshow/131594240.cms

‍

‍Introduction 

‍

In September 2025, social media feeds were flooded with strikingly vintage saree-type portraits. These images were not taken by professional photographers, but AI-generated images. More than a million people turned to the "Nano Banana" AI tool of Google Gemini, uploading their ordinary selfies and watching them transform into Bollywood-style, cinematic, 1990s posters. The popularity of this trend is evident, as are the concerns of law enforcement agencies and cybersecurity experts regarding risks of infringement of privacy, unauthorised data sharing, and threats related to deepfake misuse.

‍

What is the Trend?

‍

This trend in AI sarees is created using Google Geminis' Nano Banana image-editing tool, editing and morphing uploaded selfies into glitzy vintage portraits in traditional Indian attire. A user would upload a clear photograph of a solo subject and enter prompts to generate images of cinematic backgrounds, flowing chiffon sarees, golden-hour ambience, and grainy film texture, reminiscent of classic Bollywood imagery. Since its launch, the tool has processed over 500 million images, with the saree trend marking one of its most popular uses. Photographs are uploaded to an AI system, which uses machine learning to alter the pictures according to the description specified. The transformed AI portraits are then shared by users on their Instagram, WhatsApp, and other social media platforms, thereby contributing to the viral nature of the trend.

‍

Law Enforcement Agency Warnings

‍

1. A few Indian police agencies have issued strong advisories against participation in such trends. IPS Officer VC Sajjanar warned the public: "The uploading of just one personal photograph can make greedy operators go from clicking their fingers to joining hands with criminals and emptying one's bank account." His advisory had further warned that sharing personal information through trending apps can lead to many scams and fraud.
2. Jalandhar Rural Police issued a comprehensive warning stating that such applications put the user at risk of identity theft and online fraud when personal pictures are uploaded. A senior police officer stated: "Once sensitive facial data is uploaded, it can be stored, analysed, and even potentially misused to open the way for cyber fraud, impersonation, and digital identity crimes.

‍

The Cyber Crime Police also put out warnings on social media platforms regarding how photo applications appear entertaining but can pose serious risks to user privacy. They specifically warned that selfies uploaded can lead to data misuse, deepfake creation, and the generation of fake profiles, which are punishable under Sections 66C and 66D of the IT Act 2000.

‍

Consequences of Such Trends

‍

The massification of AI photo trends has several severe effects on private users and society as a whole. Identity fraud and theft are the main issues, as uploaded biometric information can be used by hackers to generate imitated identities, evading security measures or committing financial fraud. The facial recognition information shared by means of these trends remains a digital asset that could be abused years after the trend has passed. ‘Deepfake’ production is another tremendous threat because personal images shared on AI platforms can be utilised to create non-consensual artificial media. Studies have found that more than 95,000 deepfake videos circulated online in 2023 alone, a 550% increase from 2019. The images uploaded can be leveraged to produce embarrassing or harmful content that can cause damage to personal reputation, relationships, and career prospects.

‍

Financial exploitation is also when fake applications in the guise of genuine AI tools strip users of their personal data and financial details. Such malicious platforms tend to look like well-known services so as to trick users into divulging sensitive information. Long-term privacy infringement also comes about due to the permanent retention and possible commercial exploitation of personal biometric information by AI firms, even when users close down their accounts.

‍

Privacy Risks

‍

A few months ago, the Ghibli trend went viral, and now this new trend has taken over. Such trends may subject users to several layers of privacy threats that go far beyond the instant gratification of taking pleasing images. Harvesting of biometric data is the most critical issue since facial recognition information posted on these sites becomes inextricably linked with user identities. Under Google's privacy policy for Gemini tools, uploaded images might be stored temporarily for processing and may be kept for longer periods if used for feedback purposes or feature development. 

Illegal data sharing happens when AI platforms provide user-uploaded content to third parties without user consent. A Mozilla Foundation study in 2023 discovered that 80% of popular AI apps had either non-transparent data policies or obscured the ability of users to opt out of data gathering. This opens up opportunities for personal photographs to be shared with anonymous entities for commercial use. Exploitation of training data includes the use of personal photos uploaded to enhance AI models without notifying or compensating users. Although Google provides users with options to turn off data sharing within privacy settings, most users are ignorant of these capabilities. Integration of cross-platform data increases privacy threats when AI applications use data from interlinked social media profiles, providing detailed user profiles that can be taken advantage of for purposeful manipulation or fraud. Inadequacy of informed consent continues to be a major problem, with users engaging in trends unaware of the entire context of sharing information. Studies show that 68% of individuals show concern regarding the misuse of AI app data, but 42% use these apps without going through the terms and conditions.

‍

CyberPeace Expert Recommendations

‍

While the Google Gemini image trend feature operates under its own terms and conditions, it is important to remember that many other tools and applications allow users to generate similar content. Not every platform can be trusted without scrutiny, so users who engage in such trends should do so only on trustworthy platforms and make reliable, informed choices. Above all, following cybersecurity best practices and digital security principles remains essential.

‍

Here are some best practices:-

‍

1.Immediate Protection Measures for User

In a nutshell, protection of personal information may begin by not uploading high-resolution personal photos into AI-based applications, especially those trained for facial recognition. Instead, a person can play with stock images or non-identifiable pictures to the degree that it satisfies the program's creative features without compromising biometric security. Strong privacy settings should exist on every social media platform and AI app by which a person can either limit access to their data, content, or anything else.

‍

2.Organisational Safeguards

AI governance frameworks within organisations should enumerate policies regarding the usage of AI tools by employees, particularly those concerning the upload of personal data. Companies should appropriately carry out due diligence before the adoption of an AI product made commercially available for their own use in order to ensure that such a product has its privacy and security levels as suitable as intended by the company. Training should instruct employees regarding deepfake technology.

‍

3.Technical Protection Strategies

Deepfake detection software should be used. These tools, which include Microsoft Video Authenticator, Intel FakeCatcher, and Sensity AI, allow real-time detection with an accuracy higher than 95%. Use blockchain-based concepts to verify content to create tamper-proof records of original digital assets so that the method of proposing deepfake content as original remains very difficult.

‍

4.Policy and Awareness Initiatives

For high-risk transactions, especially in banks and identity verification systems, authentication should include voice and face liveness checks to ensure the person is real and not using fake or manipulated media. Implement digital literacy programs to empower users with knowledge about AI threats, deepfake detection techniques, and safe digital practices. Companies should also liaise with law enforcement, reporting purported AI crimes, thus offering assistance in combating malicious applications of synthetic media technology.

‍

5.Addressing Data Transparency and Cross-Border AI Security

Regulatory systems need to be called for requiring the transparency of data policies in AI applications, along with providing the rights and choices to users regarding either Biometric data or any other data. Promotion must be given to the indigenous development of AI pertaining to India-centric privacy concerns, assuring the creation of AI models in a secure, transparent, and accountable manner. In respect of cross-border AI security concerns, there must be international cooperation for setting common standards of ethical design, production, and use of AI. With the virus-like contagiousness of AI phenomena such as saree editing trends, they portray the potential and hazards of the present-day generation of artificial intelligence. While such tools offer newer opportunities, they also pose grave privacy and security concerns, which should have been considered quite some time ago by users, organisations, and policy-makers. Through the setting up of all-around protection mechanisms and keeping an active eye on digital privacy, both individuals and institutions will reap the benefits of this AI innovation, and they shall not fall on the darker side of malicious exploitation. 

‍

References

‍

1. https://www.hindustantimes.com/trending/amid-google-gemini-nano-banana-ai-trend-ips-officer-warns-people-about-online-scams-101757980904282.html%202
2. https://www.moneycontrol.com/news/india/viral-banana-ai-saree-selfies-may-risk-fraud-warn-jalandhar-rural-police-13549443.html
3. https://www.parliament.nsw.gov.au/researchpapers/Documents/Sexually%20explicit%20deepfakes.pdf
4. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year
5. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year
6. https://socradar.io/top-10-ai-deepfake-detection-tools-2025/

‍