Photo: VCG

Online rumormongers are now advancing with the assistance of artificial intelligence (AI) models. The emerging AI-powered paid posting industry can influence the views and decisions of ordinary internet users with bunks of fake comments in a short period through making up more than 4,000 harmful messages within a day.

AI has infiltrated into every phase of the paid posting industry by enabling those “internet water army” to register social media accounts in bulk bypassing the real-name system, providing maintenance services of keeping the accounts and eventually monetizing the posts, the Shanghai-based Jiefang Daily reported.

With the rapid development of AI large models in recent years, online rumormongers have strengthened their capabilities by integrating with AI technology. They can easily disguise themselves as ordinary internet users, and posts and comments generated by AI technology can be even more provocative than those written by real people, Jiefang Daily reported.

Yang Bing (pseudonym), an employee who used to work at a studio, said paid posters who could be hired by either movie investors or actor’s fans. She told Jiefang Daily that the content posted by these “water army” is merely aimed at achieving their employers’ goals, either to create hype or attack opponents. With the development of AI large models, Yang has felt that real-people “water army” are being gradually replaced by AI-empowered posters.

Ye Xing, a marketing director from Geetest, an internet security service provider said they have been fighting against the underground industry for 12 years and intercepts a total of 2.17 billion attacks launched by AI-empowered “water army”, Jiefang Daily reported.

Online rumormongers can generate automated scripts to generate online content in batches, deliberately manipulating the public opinion through the use of AI models – they are low-cost yet highly efficient, capable of operating 24/7 with multitasking skills, according to Ye.

With the assistance of AI large models, an individual or a small group of people can now organize an AI-driven “online water army.”

A task that once required several part-time paid posters’ hours of effort in generating 1,000 comments or posts can now be accomplished by AI in just a few minutes, Yang noted.

In June 2024, police in Nanchang, East China’s Jiangxi Province investigated a case involving the spread of rumors and fabricated information disrupting public order. The suspects produced a massive number of articles in a short period of time, peaking at 4,000 to 7,000 articles per day with AI softwares. Preliminary estimates indicated that their revenue exceeded 10,000 yuan ($1,382) per day, according to a China Central Television report.

China’s cyberspace administration departments have stepped up efforts in combating paid posting since 2024, investigating issues such as recruitment, promotion, traffic manipulation, and controlled reviews. They have shut down or removed over 400 websites and platforms, deleted 4.82 million pieces of illegal content, and handled 2.39 million accounts and online stores, along with 52,000 online groups.

Xu Xiaoke, a professor at the Center for Computation Communication Research (Zhuhai), School of Journalism and Communication of Beijing Normal University, said it was easy to distinguish real-person posters and AI-driven paid posters based on the mobile phone IP and geographic location information. However, such methods can no longer precisely crack down on today AI-powered paid posters who can generate content in bulk with algorithms and even simulate real users’ behavior.

In response, China has released a set of guidelines on labeling internet content that is generated or composed by AI technology, which are set to take effect on September 1. The guidelines, issued by authorities including the Cyberspace Administration of China, regulate the labeling of AI-generated online content throughout its production and dissemination processes, requiring providers to add visible marks to their content in appropriate locations.

Besides, social media accounts registered by AI algorithms should be clearly marked by social media platforms and operators of the platforms should supervise the content posted by these accounts and promptly identify and mark fake or illegal content, Xu told the Global Times on Monday.