Make AI Great Again: How the Overuse of AI in Content Moderation on Digital Platforms is Harming User Experience

By Mehar Khursheed

Concern for man himself and his fate must always form the chief interest of all technical endeavors, concern for the great unsolved problems of the organization of labor and the distribution of goods – in order that the creations of our mind shall be a blessing and not a curse to mankind. Never forget this in the midst of your diagrams and equations.

– Albert Einstein

Imagination is the beginning of creation. You imagine what you desire, you will what you imagine and at last you create what you will.

– George Bernard Shaw

The GSMA’s Mobile Gender Gap Report 2022 highlights that women in Pakistan are 33% less likely than men to own a mobile phone and 38% less likely to use mobile internet. According to UNICEF, ‘Evidence shows that 9 out of 10 adolescent girls and young women are offline in low-income countries.’ This underrepresentation and lack of access for some demographics already ensures that the majority of the content and information being created and promoted on digital platforms caters to the dominant demographic i.e. straight adult men.

Yet when the internet became a (somewhat) household convenience many believed it would be the great equalizer. It was thought that people of all identities and backgrounds would now have an identical weight assigned to their opinions and needs. Of course, it didn’t quite work out that way. Creating a platform free from biases, discrimination, bigotry and threats would require that those who created i tand those who used it be just as free too. As we know, women, children, the disabled and elderly, queer folk, people from marginalized religions and ethnicities, class and caste backgrounds are all disproportionately vulnerable to being abused on, and excluded from digital spaces and platforms, reflecting, if not amplifying, real-life social hierarchy and policies.

For instance, Facebook, the most populous digital platform (3 billion active users monthly), that was created in 2004 for the purpose of objectifying and rating women, has only further widened the gender gap on and within digital platforms. In January 2025 its owner and co-creator Mark Zuckerburg blatantly declared “I think a lot of the corporate world is pretty culturally neutered,” and “I think having a culture that celebrates the (masculine) aggression a bit more has its own merits.” Forbes has expanded on this dog whistle; ‘the question for Zuckerberg should be: is the emperor wearing new clothes, or being explicit about what he was wearing all along? This question comes at a particularly significant juncture in the ongoing discussion about the intersection of politics and technology. In an industry racing to shape humanity’s future through Artificial Intelligence while facing renewed political pressure, Zuckerberg’s words present a crucial contradiction- that behind the veneer of tech’s innovative exterior remain some deeply conventional beliefs about power and leadership.’

Artificial Intelligence is thought to be able to address every problem, from world hunger to deep emotional dilemmas about the human condition. More than that, despite every movie on the subject warning us not to, AI is being given the reins to much of our offline and online life. It is being authorised to moderate social media platforms, flag inappropriate content, or take down harmful posts. Meta further announced that it will be firing its fact-checkers and ‘instead rely on its users and AI’ to moderate community standards. Community standards are what (allegedly) ensure that vulnerable and marginalised demographics are not further attacked on the platform in the form of bullying, harassment, doxxing, racist and sexist content and comments. Even before AI was given so much of the content moderation responsibility, Meta was not known to be the most compassionate when it came to taking down content that would constitute as digital hate crimes, but now under the banner of free speech and user control, all manner of extremely right-wing vitriol has been made permissible. It would be laughable if it wasn’t all so grotesquely tragic.

Generative AI like Chatgpt and Gemini have replaced conventional search engines and now every digital platform is vying to have its own version one of the most powerful ones in this category is Grok, available on X. However just like the internet, creation will do what its creators and consumers value most. Since AI learns on training data, who trains it to do that, what determines how well or how ‘reasonably’ it functions? Additionally, studies are showing that increased and frequent use of generative AI (like ChatGPT and Gemini) can be seen to be reducing critical thinking among users.

AI will base its results and responses on what it can glean from the information available to it online and the internet is fast becoming an arena where even credible, public health and scientific knowledge is being expunged at an official and global level. This limits and skews what AI can gather, since there is no end to misinformation online, and as a consequence, much of AI results are biased, limited and even simply incorrect.

More than this, when Generative AI is used for the purposes of trolling or harassment it is capable of automated manufacturing of harassment on a wide variety of online social platforms and interfaces, in a sense ‘stalking’ the victim’s entire digital existence rapidly and being able to disseminate threatening comments, extremely personal and specific posts, deep-fakes, sexually explicit messages and emails. While historically the impact of harassment may have been limited when done manually, as it would take an individual several days if not weeks to sift through the posts and comments and online footprint of the person being targeted, but generative AI can comb through all this in a matter of minutes. Saadia, an active user on Twitter, now X, was a recent victim of this type of online harassment. ‘Even though I did not use my real face as my profile picture, the harasser found my Linkedin profile from my Twitter/X username.’ She said that from there the harassment became ‘extremely personal and sexually explicit with deep-fakes created using my Linkedin profile picture, with threats of these being sent to my places of work and my university.’ Depending on the malintent of the attacker, these automated messages can then lead to significant harm not just to the targeted but also to other users from similar vulnerable demographics who will further self-censor and avoid these platforms if threats such as this and worse, persist against members of their community, who are left with little to no recourse to stem the voluminous tide of abuse, nor identify the person(s) behind it. Reports indicate that harassment using automated troll bots is an ever-growing problem, and it seems imminent that real-time, autonomous conversational agents take over as a primary vehicle for, or driver of, harassment.

What does this mean for other women plus other marginalized and vulnerable demographics in the Global South and more specifically, Pakistan? One of the major reasons why the internet is presently and always has been so unsafe for women, children, the disabled and elderly, queer folk, people from marginalized religions and ethnicities, class and caste backgrounds, is because not just the users but the people behind the screens, the decision makers and developers are majorly not from these demographics. Worldwide, women in the tech industry only make up about 35% of the workforce and in Pakistan, that number is below 20%. Add racial, religious and gender minorities to this mix and the result is almost non-existent. This means that the content being generated and the content being moderated is majority of the time not by people from these marginalized demographics. The AI that is trained to flag harmful content or targeted harassment is not trained by people from these groups either and therefore the nuance needed to understand human emotion, colloquial context, racial, religious, ethnic and gender frameworks are not fed into these models.

In fact, what is happening is the complete opposite of that. As we watch the termination of  Diversity, Equity and Inclusion programs across the world, slashing funding for institutions that protect civil liberties and eliminating social welfare organisations, the same disdain for human rights, diversity and dignity is now reflected with even more gusto on online platforms. AI content creation, moderation, generation and promotion have been majorly very white. This means that the content AI makes, moderates, and promotes is largely, and by design, in favour of a certain demographic. Despite there already being a racial, gendered and class disbalance in the online world (similar to the real world), using a lack of human understanding and overly relying on AI to do the work of content moderation creates an environment that is not just unsafe but overtly hostile for certain groups of people.

I interviewed several Pakistani individuals from diverse backgrounds who are all active social media users to investigate what their online experience was when they first began using these platforms, especially in regards to reporting the abuse they were targeted with and the response to that reporting, and what they have to deal with in the present day.

Zainab, a religious minority from Karachi who mostly uses Instagram and Facebook has explained that she experienced regular harassment and targeted attacks based on her faith. When she reported these attacks and profiles as an individual these posts and accounts were not contained or removed. She had to enlist her community of over 20+ people to report the attacks which then resulted in those attacks being reviewed and consequently removed. Now, however, the response time to these attacks is ineffectual and she chooses to simply block or ignore these accounts.

Bubbles, a trans woman, artist, educator and activist also from Karachi says that ‘content moderation in the past seemed more hands-on. Content was effectively moderated, filtering out disturbing or malicious content. Now everything seems to be left to bots who don’t do the same job.’ When asked whether her experience in reporting seems to have changed over the years she felt that ‘Yes I have reported (these attacks) several times. Before Elon Musk took over, reports on X used to be effective but they are useless now. Facebook reports also used to be swift but they rarely help now. Instagram may be slightly better at this point.’ Concluding that ‘I don’t even report it anymore because nothing happens. And also at some point, you get desensitized even to your own harassment especially when you know speaking up won’t help.’

Anwar, a man from a faith-based minority also responded that despite him reporting the attacks he faced online ‘nothing happened, so I deleted my comments. Initially, I would report the people who hate commented but now I just block them because I know Meta ne kuch nahi karna.’

Rimsha, an active online user of digital platforms for the past 15 years, explained that ‘when I used to regularly comment on feminist posts, sometimes there were threatening messages and racist replies. Nothing ever happened whenever I reported them on either Instagram or Facebook.’ She also said that ‘reporting as spam works better than reporting as harassment. But blocking works the best. I still choose to report any comment I see that’s harmful but now I know it will do nothing.’

There has been a marked trend in the decreased lack of confidence that users from marginalised communities have in reporting abuse and threats for content moderation on digital platforms as the years have progressed and the bots have taken charge. Digital platforms are meant to inspire a sense of freedom and creative expression in those who use them. People from marginalised and vulnerable communities should be able to use these spaces without fear for their safety and sanity. However, the over-reliance on AI and removing human individuals from teams of content moderation has been at the expense of those who are already the subject of targeted attacks and violence in the offline world. AI itself is not the root of any evil or inefficacy, it is the lack of diversity in the teams who are training these programs to be responsible for our safety that are lacking. Women and other marginalised groups’ underrepresentation in generative AI usage is what leads to systems trained on data that inadequately sample women’s preferences and needs, ultimately widening existing gender disparities in technology adoption, economic opportunity and user experience on digital platforms. Even when women do get to use these tools, their efficacy could be hindered by the fact that these generative AI systems are often trained on biased data (Guilbeault et al., 2024), including data from their predominantly male users. This can result in self-reinforcing gender gaps as these systems learn from, adapt to, and so sample from more men (Cao, Koning, and Nanda, 2023).

AI has been an exciting new tool for many of us but the implications it has for those who are already vulnerable have been difficult to ignore despite there being conditions laid out for AI ethics, those are not the easiest to implement especially if the humans behind the technology keep being removed. The people to pay the price for this excess reliance on AI are those who have already been sidelined and subjected to abuse in online spaces but now the threats are much worse and much more dangerous. Human understanding and involvement cannot be underestimated in technology and AI cannot be said to be aiding humanity if only a certain group of humans are allowed to design, develop and benefit from that technology.