More than 60 Journalists and Media Practitioners Across Pakistan Condemn the Sexist and Derogatory Statements Made by Dr. Omer Adil Against Women in Media
July 25, 2024July 2024 Newsletter: DRF at International Conference on Addressing Technology Facilitated Violence Against Women (TFVAW)
August 15, 2024Submission: Research Department – Digital Rights Foundation
Aleena Afzaal – Sr. Research Associate
Abdullah B. Tariq – Research Associate
Submission Date: April 30, 2024
Legal Context:
Given the borderless nature of digital content, Meta should consider international legal developments as a framework for its policies. The European Union’s Digital Services Act and specific statutes from the U.S. state of California, such as AB 602, provide precedents for regulating digital content and protecting individuals against non-consensual use of their images.
Irregular responses in two different cases (How such cases affect people in different regions):
It is important to note that the two cases relating to deepfake videos of women public figures were approached and dealt with differently potentially due to difference in ethnicity and identity: one being from the Global North and the other belonging to the global majority identity. The American public figure case received a relatively immediate response whereas the case of resemblance to a public figure in India was not highlighted or amplified as quickly. Despite the technical discrepancies, it cannot be ignored that in the latter case, an Instagram account with several similar images remained unflagged for a long time. Additionally, one question that arises continuously from a string of these cases is why have tech platforms not adopted technological mechanisms that can flag sensitive content, particularly deepfakes circulating on different platforms. The harms prevailing due to the emerging technologies particularly generative AI content need to be viewed through a more intersectional lens. Women and marginalized groups in the global majority particularly from South Asia are more vulnerable to attacks online with a significant impact on their online and offline safety rather than individuals from the global North. While female security and inclusion is crucial, the potential otherization of the community is concerning and needs to be revisited.
Moreover, taking cultural context into account, the level of scrutiny and criticism a South Asian female is subjected to in such events is higher as compared to a woman of American descent. In India, a woman is viewed as good only if she is able to maintain the respect and honor of her family. Female bodies are sexualized and any attack on them is considered to be an attack on men and the community’s honor. Several cases have come forward in the past where women and young girls in India have taken their own lives as a result of leaked photos. In the wider Indian subcontinent region, cases have arisen where women have been subjected to honor killing as a consequence of being romantically involved with a man, their explicit photos being leaked and more. Such cases in the region showcase an underlying problem where women and honor are used as interchangeable terms and need to be taken into consideration when handling issues of similar nature. Public figures or not, women are more prone to being targeted by AI-generated content and deepfakes. Recently, incidents have come forward where deepfakes of two female public figures in Pakistan have been made widely available across different social media platforms. As far as Meta’s platforms are concerned, these deepfakes have been uploaded with nudity being covered with the use of stickers and emojis however in the comments section, users have offered and/or asked to share the link to view the originally created content. It is crucial that platforms like Meta have mechanisms in place where content and comments amplifying technology-facilitated gender-based violence are also flagged. Considering the higher probability combined with the societal consequences, it is essential for Meta to give greater consideration to cases involving deepfakes and AI-generated content showcasing characteristics of technology-facilitated gender-based violence more importance on the platform, particularly with countries from the global majority where the risk of potential harm is higher than others. Human reviewers should also be made aware of the language and cultural context of the cases under consideration. Trusted partners of Meta should be entrusted with the task of escalating the cases, where the response time of prioritized cases is expedited and addressed at the earliest.
Clarification and Expansion of Community Guidelines:
Meta’s current community standards need to be more explicit in defining violations involving AI-generated content. There is an urgent need to develop a specific section for public-facing community guidelines on the platform to address deepfakes. Detailing examples and outlining repercussions would clarify the company’s stance for users and content moderators alike. Public figures are at a higher risk of being victims of deep fake content due to their vast exposure (reference imagery) in online spaces. Thus, the policy rationale and the consequent actions need to be similar in the case of public figures and private individuals considering the sensitivity of such content regardless of an individual’s public exposure. It is equally important that Meta revises its policy regarding sensitive content where the person being imitated is not tagged. The policy needs to be inclusive of such content as the potential harms remain. Regular updates to these guidelines are crucial as AI technology evolves.
Technical Mechanisms for Enhanced Detection and Response:
- Implementing cutting-edge machine learning techniques to detect deepfake content (image, video and audio) can significantly reduce the spread of harmful content. These algorithms should focus on detecting common deepfake anomalies and be regularly updated to keep pace with technological advancements. A two pronged approach can be utilized for detecting and flagging harmful content on their platforms. Larger investments should be placed in automated detection systems to efficiently categorize and identify generative AI content and be adaptable to future advancements.
- Detected Gen AI content should be marked on Meta platforms to avoid confusion or the spread of misinformation. Meta needs to reassess its appeals pipeline and allow for extended review times, especially for content that contains any human likeness. Moreover, Meta needs to reassess its appeals pipeline and allow for extended review times, especially for content that contains any human likeness.
- Collaborating with AI developers to embed watermarks in AI-generated content can help automatically identify and segregate unauthorized content. This would bolster Meta’s ability to preemptively block the dissemination of harmful material.
- Expanding this database to include international cases and allowing for real-time updates can enhance its effectiveness in identifying and removing known violating content swiftly.
- Meta should build on and enhance the capacity of its trusted partners particularly in terms of escalating content to the platform and having a robust and quick escalation channel in case of emergencies or content that is life-threatening. Meta needs to have emergency response mechanisms in place and have policy teams who are sensitized to deal with matters of utmost urgency particularly when it relates to marginalized groups and vulnerable communities.
The current challenges faced by Meta in managing AI-generated content are largely due to its lack of specificity in its policies to encapsulate generative AI content. The community standards in their current state fail to address the complexities of AI-generated content and the adverse impacts it can have on people and communities. Meta’s clear differentiation in its policy application rationale for two different cases raises concerns over irregular and inefficient content moderation policies. While we acknowledge that content in both these cases is no longer on the platform, the urgency displayed in taking down content from the second case compared to the delay in the removal from the first case highlights the dire need for stringent and equitable response of social media platforms on gen-AI content. Moreover, in the second case the deepfake video of an American woman public figure was removed under the policy “Bullying and Harassment, specifically for “derogatory sexualised photoshop or drawings”” – Greater discourse is required over what classifies as “derogatory” in this context. In the absence of a derogatory element, will an AI-generated image that involves sexualisation and nudity be available to view on the platform? If so, then how is Meta perceiving the consensual privacy and dignity of public figures on its platforms? These are the questions that need to be addressed and outlined in Meta’s content moderation policies, especially in terms of tech-facilitated gender-based violence.
Meta’s Media Matching Service Banks are restricted by the database of known images, which renders them highly ineffective against newly generated deepfake content. With tools to create generative AI content becoming increasingly accessible, the technology to flag and address such content needs to catch up as soon as possible. It is essential for Meta to expand its database to encompass a wider array of AI-generated content types and implement real-time updates.
In conclusion, Meta’s automated detection systems struggle to keep pace with rapidly advancing sophisticated technologies used in deepfake content. For Meta to ensure safety on its platforms for marginalized groups and communities, it is essential for them to revisit their content moderation policies pertaining to generative AI content while enhancing and investing in its trusted civil society partners to escalate content towards the platform.
