AI in the Hands of Bad Actors: ChatGPT’s Misuse by State-Linked Cyber Operatives
Introduction
Artificial Intelligence (AI) is fast transforming our future in the digital world, transforming healthcare, finance, education, and cybersecurity. But alongside this technology, bad actors are also weaponising it. More and more, state-sponsored cyber actors are misusing AI tools such as ChatGPT and other generative models to automate disinformation, enable cyberattacks, and speed up social engineering operations. This write-up explores why and how AI, in the form of large language models (LLMs), is being exploited in cyber operations associated with adversarial states, and the necessity for international vigilance, regulation, and AI safety guidelines.
The Shift: AI as a Cyber Weapon
State-sponsored threat actors are misusing tools such as ChatGPT to turbocharge their cyber arsenal.
- Phishing Campaigns using AI- Generative AI allows for highly convincing and grammatically correct phishing emails. Unlike the shoddily written scams of yesteryears, these AI-based messages are tailored according to the victim's location, language, and professional background, increasing the attack success rate considerably. Example: It has recently been reported by OpenAI and Microsoft that Russian and North Korean APTs have employed LLMs to create customised phishing baits and malware obfuscation notes.
- Malware Obfuscation and Script Generation- Big Language Models (LLMs) such as ChatGPT may be used by cyber attackers to help write, debug, and camouflage malicious scripts. While the majority of AI instruments contain safety mechanisms to guard against abuse, threat actors often exploit "jailbreaking" to evade these protections. Once such constraints are lifted, the model can be utilised to develop polymorphic malware that alters its code composition to avoid detection. It can also be used to obfuscate PowerShell or Python scripts to render them difficult for conventional antivirus software to identify. Also, LLMs have been employed to propose techniques for backdoor installation, additional facilitating stealthy access to hijacked systems.
- Disinformation and Narrative Manipulation
State-sponsored cyber actors are increasingly employing AI to scale up and automate disinformation operations, especially on election, protest, and geopolitical dispute days. With LLMs' assistance, these actors can create massive amounts of ersatz news stories, deepfake interview transcripts, imitation social media posts, and bogus public remarks on online forums and petitions. The localisation of content makes this strategy especially perilous, as messages are written with cultural and linguistic specificity, making them credible and more difficult to detect. The ultimate aim is to seed societal unrest, manipulate public sentiments, and erode faith in democratic institutions.
Disrupting Malicious Uses of AI – OpenAI Report (June 2025)
OpenAI released a comprehensive threat intelligence report called "Disrupting Malicious Uses of AI" and the “Staying ahead of threat actors in the age of AI”, which outlined how state-affiliated actors had been testing and misusing its language models for malicious intent. The report named few advanced persistent threat (APT) groups, each attributed to particular nation-states. OpenAI highlighted that the threat actors used the models mostly for enhancing linguistic quality, generating social engineering content, and expanding operations. Significantly, the report mentioned that the tools were not utilized to produce malware, but rather to support preparatory and communicative phases of larger cyber operations.
AI Jailbreaking: Dodging Safety Measures
One of the largest worries is how malicious users can "jailbreak" AI models, misleading them into generating banned content using adversarial input. Some methods employed are:
- Roleplay: Simulating the AI being a professional criminal advisor
- Obfuscation: Concealing requests with code or jargon
- Language Switching: Proposing sensitive inquiries in less frequently moderated languages
- Prompt Injection: Lacing dangerous requests within innocent-appearing questions
These methods have enabled attackers to bypass moderation tools, transforming otherwise moral tools into cybercrime instruments.
Conclusion
As AI generations evolve and become more accessible, its application by state-sponsored cyber actors is unprecedentedly threatening global cybersecurity. The distinction between nation-state intelligence collection and cybercrime is eroding, with AI serving as a multiplier of adversarial campaigns. AI tools such as ChatGPT, which were created for benevolent purposes, can be targeted to multiply phishing, propaganda, and social engineering attacks. The cross-border governance, ethical development practices, and cyber hygiene practices need to be encouraged. AI needs to be shaped not only by innovation but by responsibility.
References
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
- https://www.bankinfosecurity.com/openais-chatgpt-hit-nation-state-hackers-a-28640
- https://oecd.ai/en/incidents/2025-06-13-b5e9
- https://www.microsoft.com/en-us/security/security-insider/meet-the-experts/emerging-AI-tactics-in-use-by-threat-actors
- https://www.wired.com/story/youre-not-ready-for-ai-hacker-agents/
- https://www.cert-in.org.in/PDF/Digital_Threat_Report_2024.pdf
- https://cdn.openai.com/threat-intelligence-reports/5f73af09-a3a3-4a55-992e-069237681620/disrupting-malicious-uses-of-ai-june-2025.pdf
Related Blogs
Introduction
A 33-year-old MBA graduate and 36-year-old software engineer set up the cybercrime hub in one bedroom. They formed the nameless private enterprise two years ago and hired the two youngsters as employees. The police revealed that the fraudsters moved Rs 854 crore rapidly through 84 bank accounts in the last two years. They were using eight mobile phones active during the day and night for their malicious operations. This bad actors group came in the eyes of the police when a 26-year-old woman filed a complaint, she was lured and cheated for Rs 8.5 lakh on the pretext of making small investments for high returns. It led to cyber crime police on their doorstep. The police discovered that they were operating a massive cyber fraud network from that single room, targeting a large number of people for committing cyber fraud through offering investment schemes and luring innocent people.
How cybercrime fraudsters lured the victims?
The Bangalore police have busted a cyber fraud scam worth 854 Crore rupees. And police have arrested 6 accused. These bad actors illegally deceived numerous victims on the pretext of investment schemes. The gang used to lure them through WhatsApp and Telegram. Initially, the people were asked to invest small amounts, promising daily profits ranging from 1 thousand to 5 thousand rupees. As the trust grew, thousands of victims indulged in investments ranging from 1 lack to 10 lack rupees. This Money luring modus operandi was used by the fraudsters to attract them and get the victims to invest more and more. The amount invested by the victims was deposited into various bank accounts by the fraudsters. When the victims tried to withdraw their amount after depositing they were unable to do so. Soon after the amount was received, the accused gang would launder the money and divert it to other accounts.
Be cautious of online investment fraud
It concerns all of us who used to invest online. The Bangalore police have busted cyber crime or cyber investment fraud of 854 crore rupees. The 6 members of the gang that the police have arrested used to approach victims through WhatsApp and telegram to convince them to invest small amounts, from 1 thousand to 10 thousand at the bare minimum and promising them returns or profit amount per day and later lock this amount and diverting it into different bank accounts, ensuring that those get invested never get access to it again. Now, this went on in the country receiving a large number of cases that have been registered from various states in the country.
Advisory and best practices
- It is important to mention that there could be several other cybercrime investment frauds like this that you may not even be aware of. Hence, this incident of massive online investment fraud operated from the IT capital of the country definitely acts as an eye-opener for all of us. We urge people to be cautious and raise the alarm about any such cyber crime or investment fraud that they see in the cyber world today.
- In the age of the internet, where there is a large number of mobile users in the country, and users look for a source of income on the internet and use it to invest their money, it is important to be aware of such fraud and be cautious and take proper precautions before investing in any such online scheme. It is always advisable to invest only in legitimate sources and after conducting due diligence.
- Be cautious and do your research: Whenever you are investing in any scheme or in digital currency, make sure to verify the authenticity or legitimacy of the person or company who is offering such service. Check the reviews, official website, and feedback from authentic sources. Find out whether the agents or brokers who contact you are licensed to operate in your state and are compliant with regulators or other investors.
- Verify the credentials: Check the genuineness by checking the licenses, registration and certification of the person or company offering such services, whether he is authorised or not.
- Be Skeptical of offers which seem to be too good: If it sounds too good, be cautious and inquire about its authenticity, such as unsolicited offers. Be especially careful if you receive an unsolicited pitch to invest in a particular company or see it praised online but if you could not find current financial information about it from independent sources. It could be a fraudulent scheme. It is advisable to compare promised yields with current returns on well-known stock indexes.
- Seek Expert Advice: If you are a beginner in online investment, you may seek advice from reliable resources such as financial advisors who can provide more clarity on aspects of investment and guidance to help you make informed decisions.
- Avoid Unreliable Platforms: Be cautious and stick to authorised established agencies. Be cautious when dealing with a person or company lacking sufficient user reviews and credible security measures.
- Protect yourself online: Protect yourself online. Fraudsters target users on online and social marketing sites and commit various online frauds; hence, it's important to be cautious and protect yourself online. So be cautious and make your own sound decision after all analysis while investing in any such services.
- Report Suspicious Accounts: If you encounter any social media accounts, social media groups or profiles which seem suspicious and engaged in fraudulent services, you must report such profiles to the respective platform immediately.
- Report cyber crimes to law enforcement agencies: A powerful resource available to victims of cybercrime is the National Cyber Crime Reporting Portal, equipped with a 24x7 helpline number, 1930. This portal serves as a centralised platform for reporting cybercrimes, including financial fraud.
Conclusion:
This recent cyber investment fraud worth Rs 854 Crore, orchestrated by a group of fraudsters operating from a single room, serves as a stark reminder of the risks posed by bad actors. This incident underscores the importance of being vigilant when it comes to online investments and financial transactions. As we navigate the vast and interconnected landscape of the internet, it is imperative that we exercise due diligence and employ best practices to protect ourselves. We need to be cautious and protected from falling victim to these fraudulent schemes, actively reporting suspicious accounts and cybercrimes to relevant authorities through resources like the National Cyber Crime Reporting Portal will contribute to helping stop these types of cyber crimes. Knowledge and awareness are some of the biggest factors we have in fighting back against such cyber frauds in this digital age and making a safer digital environment for everyone.
References
- https://www.news18.com/india/bengaluru-cyber-crime-rs-854-crore-84-banks-accounts-fraud-network-one-bedroom-house-yelahanka-karnataka-8618426.html
- https://indianexpress.com/article/cities/bangalore/cyber-crime-bengaluru-links-over-5000-cases-india-8982753/lite/
Introduction
In a setback to the Centre, the Bombay High Court on Friday 20th September 2024, struck down the provisions under IT Amendment Rules 2023, which empowered the Central Government to establish Fact Check Units (FCUs) to identify ‘fake and misleading’ information about its business on social media platforms.
Chronological Overview
- On 6th April 2023, the Ministry of Electronics and Information Technology (MeitY) notified the Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Amendment Rules, 2023 (IT Amendment Rules, 2023). These rules introduced new provisions to establish a fact-checking unit with respect to “any business of the central government”. This amendment was done In exercise of the powers conferred by section 87 of the Information Technology Act, 2000. (IT Act).
- On 20 March 2024, the Central Government notified the Press Information Bureau (PIB) as FCU under rule 3(1)(b)(v) of the Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Amendment Rules 2023 (IT Amendment Rules 2023).
- The next day on 21st March 2024, the Supreme Court stayed the Centre's decision on notifying PIB -FCU, considering the pendency of the proceedings before the High Court of Judicature at Bombay. A detailed analysis covered by CyberPeace on the Supreme Court Stay decision can be accessed here.
- In the latest development, the Bombay High Court on 20th September 2024, struck down the provisions under IT Amendment Rules 2023, which empowered the Central Government to establish Fact Check Units (FCUs) to identify ‘fake and misleading’ information about its business on social media platforms.
Brief Overview of Bombay High Court decision dated 20th September 2024
Justice AS Chandurkar was appointed as the third judge after a split verdict in January 2023 by a division bench consisting of Justices Gautam Patel and Neela Gokhal. As a Tie-breaker judge' Justice AS Chandurkar delivered the decision striking down provisions for setting up a Fact Check Unit under IT amendment 2023 rules. Striking down the Centre's proposed fact check unit provision, Justice A S Chandurkar of Bombay High Court also opined that there was no rationale to undertake an exercise in determining whether information related to the business of the Central govt was fake or false or misleading when in digital form but not doing the same when such information was in print. It was also contended that there is no justification to introduce an FCU only in relation to the business of the Central Government. Rule 3(1)(b)(v) has a serious chilling effect on the exercise of the freedom of speech and expression under Article 19(1)(a) of the Constitution since the communication of the view of the FCU will result in the intermediary simply pulling down the content for fear of consequences or losing the safe harbour provision given under IT Act.
Justice Chandurkar held that the expressions ‘fake, false or misleading’ are ‘vague and overbroad’, and that the ‘test of proportionality’ is not satisfied. Rule 3(1)(b)(v), was violative of Articles 14 and 19 (1) (a) and 19 (1) (g) of the Constitution and it is “ultra vires”, or beyond the powers, of the IT Act.
Role of Expert Organisations in Curbing Mis/Disinformation and Fake News
In light of the recent developments, and the rising incidents of Mis/Disinformation and Fake News it becomes significantly important that we all stand together in the fight against these challenges. The actions against Mis/Disinformation and fake news should be strengthened by collective efforts, the expert organisations like CyberPeace Foundation plays an key role in enabling and encouraging netizens to exercise caution and rely on authenticated sources, rather than solely rely on govt FCU to block the content.
Mis/Disinformation and Fake News should be stopped, identified and countered by netizens at the very first stage of its spread. In light of the Bombay High Court's decision to stuck down the provision related to setting up the FCU by the Central Government, it entails that the government's intention to address misinformation related solely to its business/operations may not have been effectively communicated in the eyes of the judiciary.
It is high time to exercise collective efforts against Mis/Disinformation and Fake News and support expert organizations who are actively engaged in conducting proactive measures, and campaigns to target these challenges, specifically in the online information landscape. CyberPeace actively publishes fact-checking reports and insights on Prebunking and Debunking, conducts expert sessions and takes various key steps aimed at empowering netizens to build cognitive defences to recognise the susceptible information, disregard misleading claims and prevent further spreads to ensure the true online information landscape.
References:
- https://www.scconline.com/blog/post/2024/09/20/bombay-high-court-it-rules-amendment-2023-fact-check-units-article14-article19-legal-news/#:~:text=Bombay%20High%20Court%3A%20A%20case,grounds%20that%20it%20violated%20constitutional
- https://indianexpress.com/article/cities/mumbai/bombay-hc-strikes-down-it-act-amendment-fact-check-unit-9579044/
- https://www.cyberpeace.org/resources/blogs/supreme-court-stay-on-centres-notification-of-pibs-fact-check-unit-under-it-amendment-rules-2023
Introduction
Twitter is a popular social media plate form with millions of users all around the world. Twitter’s blue tick system, which verifies the identity of high-profile accounts, has been under intense scrutiny in recent years. The platform must face backlash from its users and brands who have accused it of basis, inaccuracy, and inconsistency in its verification process. This blog post will explore the questions raised on the verification process and its impact on users and big brands.
What is Twitter’s blue trick System?
The blue tick system was introduced in 2009 to help users identify the authenticity of well-known public figures, Politicians, celebrities, sportspeople, and big brands. The Twitter blue Tick system verifies the identity of high-profile accounts to display a blue badge next to your username.
According to a survey, roughly there are 294,000 verified Twitter Accounts which means they have a blue tick badge with them and have also paid the subscription for the service, which is nearly $7.99 monthly, so think about those subscribers who have paid the amount and have also lost their blue badge won’t they feel cheated?
The Controversy
Despite its initial aim, the blue tick system has received much criticism from consumers and brands. Twitter’s irregular and non-transparent verification procedure has sparked accusations of prejudice and inaccuracy. Many Twitter users have complained that the network’s verification process is random and favours account with huge followings or celebrity status. In contrast, others have criticised the platform for certifying accounts that promote harmful or controversial content.
Furthermore, the verification mechanism has generated user confusion, as many need to understand the significance of the blue tick badge. Some users have concluded that the blue tick symbol represents a Twitter endorsement or that the account is trustworthy. This confusion has resulted in users following and engaging with verified accounts that promote misleading or inaccurate data, undermining the platform’s credibility.
How did the Blue Tick Row start in India?
On 21 May 2021, when the government asked Twitter to remove the blue badge from several profiles of high-profile Indian politicians, including the Indian National Congress Party Vice-President Mr Rahul Ghandhi.
The blue badge gives the users an authenticated identity. Many celebrities, including Amitabh Bachchan, popularly known as Big B, Vir Das, Prakash Raj, Virat Kohli, and Rohit Sharma, have lost their blue tick despite being verified handles.
What is the Twitter policy on blue tick?
To Twitter’s policy, blue verification badges may be removed from accounts if the account holder violates the company’s verification policy or terms of service. In such circumstances, Twitter typically notifies the account holder of the removal of the verification badge and the reason for the removal. In the instance of the “Twitter blue badge row” in India, however, it appears that Twitter did not notify the impacted politicians or their representatives before revoking their verification badges. Twitter’s lack of communication has exacerbated the controversy around the episode, with some critics accusing the company of acting arbitrarily and not following due process.
Is there a solution?
The “Twitter blue badge row” has no simple answer since it involves a complex convergence of concerns about free expression, social media policies, and government laws. However, here are some alternatives:
- Establish clear guidelines: Twitter should develop and constantly implement clear guidelines and policies for the verification process. All users, including politicians and government officials, would benefit from greater transparency and clarity.
- Increase transparency: Twitter’s decision-making process for deleting or restoring verification badges should be more open. This could include providing explicit reasons for badge removal, notifying impacted users promptly, and offering an appeals mechanism for those who believe their credentials were removed unfairly.
- Engage in constructive dialogue: Twitter should engage in constructive dialogue with government authorities and other stakeholders to address concerns about the platform’s content moderation procedures. This could contribute to a more collaborative approach to managing online content, leading to more effective and accepted policies.
- Follow local rules and regulations: Twitter should collaborate with the Indian government to ensure it conforms to local laws and regulations while maintaining freedom of expression. This could involve adopting more precise standards for handling requests for material removal or other actions from governments and other organisations.
Conclusion
To sum up, the “Twitter blue tick row” in India has highlighted the complex challenges that Social media faces daily in handling the conflicting interests of free expression, government rules, and their own content moderation procedures. While Twitter’s decision to withdraw the blue verification badges of several prominent Indian politicians garnered anger from the government and some public members, it also raised questions about the transparency and uniformity of Twitter’s verification procedure. In order to deal with this issue, Twitter must establish clear verification procedures and norms, promote transparency in its decision-making process, participate in constructive communication with stakeholders, and adhere to local laws and regulations. Furthermore, the Indian government should collaborate with social media platforms to create more effective and acceptable laws that balance the necessity for free expression and the protection of citizens’ rights. The “Twitter blue tick row” is just one example of the complex challenges that social media platforms face in managing online content, and it emphasises the need for greater collaboration among platforms, governments, and civil society organisations to develop effective solutions that protect both free expression and citizens’ rights.
