Bias
Misinformation
Equity and access
Human rights

Bias

GenAI tools can inadvertently reflect and amplify biases present in the data they are trained on. These biases may originate from imbalanced datasets, cultural norms, or systemic inequities embedded in the training data, which is often sourced from the internet or historical records. Consequently, these tools might generate outputs that reinforce harmful stereotypes, marginalise certain groups, or unfairly favour specific demographics. For instance, biased language models could produce gendered job recommendations or racially insensitive responses. Addressing this issue requires curating diverse, representative datasets, implementing fairness-focused algorithms, and continuously monitoring AI outputs to identify and mitigate bias.

GenAI are trained on datasets that are not representative of global populations, pulled from the internet. In some cases, this means that local or cultural specific content is missing, as it is less widely available online, so GenAI cannot produce content reflecting it. It also means that GenAI tools will pick up biases from the content that it is trained on, and replicate these. We need to consider the type of data that GenAI has been trained on in order to evaluate its outputs for bias. Consider the information available online. Is it all trustworthy? Does it tend to reflect particular points of view?

Examining Bias in Training

Let's consider an example from the early training of ChatGPT. Common Crawl, the dataset used to train models like GPT-2 and GPT-3, collects data by crawling outbound links from websites. This methodology results in the overrepresentation of platforms such as Reddit and Twitter, which are predominantly frequented by younger demographics. In contrast, older individuals are more likely to share their perspectives on private blogs, leading to a potential age bias in the training data that privileges the views of younger individuals over older populations. Furthermore, the demographic composition of these platforms skews heavily towards young, white, cisgender, heterosexual, Western men, with a significant majority being from the United States and English-speaking. Bender et al. (2021) note that “the voices of people most likely to hew to a hegemonic viewpoint are also more likely to be retained” (p. 613). As a result, the data collected by Common Crawl can disproportionately reflect misogynistic, racist, transphobic, and other discriminatory viewpoints.

While the training data for GPT-3 underwent filtering to exclude text containing racial slurs, sexist language, and queerphobic content, this approach introduced its own set of challenges. Specifically, the removal of such language may mitigate the presence of overt hate speech in the model’s outputs but simultaneously risks silencing marginalised communities that reclaim and subvert such language. This tension highlights the complexities of filtering training data and the unintended consequences of such interventions (Bender et al.).

Types of Bias

- Confirmation bias - We tend to search for, interpret, favour, and recall information in a way that confirms our pre-existing beliefs. It is more difficult to reevaluate and adapt our beliefs. This can lead to stereotypes and prejudices persisting. This is something we need to check within ourselves and also in GenAI tools, which, as we have seen above, tend to reflect dominant viewpoints based on their training data.
- Cultural Bias – If a GenAI tool only suggests examples from one culture or region, it might exclude diverse perspectives. For example, if it suggests only US holidays when asked about celebrations, it ignores global diversity.
- Gender Bias – GenAI might reinforce stereotypes. For instance, if a GenAI writing assistant suggests that "nurses are women" and "engineers are men," it reflects outdated assumptions.
- Language Bias – Some GenAI tools might favor certain styles of writing over others, potentially changing the tone or meaning of a message.

Why is this a problem? Bias can make content feel one-sided, exclude certain audiences, or spread unfair assumptions.

Examples of bias in GenAI

Activity: Identifying Bias

Misinformation

GenAI tools are designed to create plausible-sounding responses based on patterns in their training data, but they lack the ability to discern fact from fiction. As a result, they can inadvertently produce misinformation, ranging from minor inaccuracies to entirely fabricated content, especially in response to ambiguous, speculative, or contextually complex queries. This becomes particularly problematic when users rely on AI-generated outputs for critical information, such as health advice or news summaries. Combating misinformation involves integrating real-time fact-checking systems, encouraging transparency about data sources, and educating users on the limitations and potential risks of relying solely on AI-generated content.

Remember, misinformation is a false piece of information which is shared unintentionally while disinformation is false information deliberately spread to mislead or manipulate. For instance, using deepfake (fabricated images, audios and videos) for malicious activities.

Misleading content

AI-generated content that is unclear, exaggerated, or incorrect can lead to confusion and spread misinformation.

Inaccuracy – GenAI tools sometimes provide outdated or incorrect information, but will sound confident when doing so, projecting accuracy. A common example of this is illustrated through made up references. Information that is fabricated by GenAI is often referred to as "hallucinations". Another example, showcasing outdated information, might be GenAI stating that Pluto is a planet, as it was reclassified as a dwarf planet in 2006. Inaccuracy and fabrication is also often seen in AI-generated images or videos that present real people doing or saying things that they have not, which is misleading. These can be used to spread disinformation.
Lack of Context – GenAI might provide information without enough explanation. For example, when summarising key points of an article discussing a study on air pollution, it may highlight three of the main findings but fail to include a key counterpoint in the article or specific information about all of the locations where the study took place. If you rely solely on the GenAI summary, you will then miss out on additional important contextual information, which could perhaps lead to a misinterpretation of the study results.
Exaggeration – GenAI might phrase things in a way that sounds authoritative, when it is not, or that overemphasises certain aspects. This is also evident in AI-generated images that may draw heavily on stereotypes. An example of an exaggerated statement is "AI will replace all human jobs". This is confident statement that is very broad, does not leave room for alternatives, and is inaccurate.

Why is this a problem? Misleading content can cause misunderstandings, spread misinformation, or make an argument seem stronger than it actually is.

Examples of misinformation in GenAI

How to Avoid Misinformation in AI-Generated Content

Evaluate content from GenAI – Review AI-generated outputs carefully before using them.
Use multiple sources – Double check information that GenAI provides by verifying it with a reliable website, publication, or your class materials.
Think about the full context and different perspectives – Ensure that you consider different arguments and diverse viewpoints. Do not rely solely on AI-generated summaries that might miss key information.
Reword AI-generated text – Make sure it sounds natural and fair rather than exaggerated or misleading. Carefully consider if you want to replace your own voice (tone, style, creativity, viewpoint) with GenAI.

Activity: Spotting Misleading Language

Equity and access

Language availability

Most publicly available Large Language Models are based in the USA or Canada. They are designed with a Western user base in mind, since it is these “developed” countries that have the largest populations of internet users. This means LLMs are fluent in European languages such as English, French, German, Spanish, Russian as well as Asian languages like Chinese (Mandarin), Japanese, Korean, and Hindi. However, an LLM’s Irish, for example, might be patchy as less sites on the internet are written in Irish. This means monolingual speakers of minority languages are often excluded from using LLMs. This might impact international students, who cannot access GenAI in their mother tongue, as well as create a gap between who can/cannot access GenAI globally.

Western centric

Since the Western World/the Global North make up the majority of texts on the internet used to train LLMs, Western cultures become the “default” for LLMs. This is a huge oversight in these machines, which do not understand the material they process and thus cannot think critically about their knowledge gaps. They are not aware of what they leave out. If you use GenAI tools, it is important that you critically evaluate content generated by the tool in order to spot gaps and incorrect information in its logic/data/proofs/arguments.

Disabled users

LLMs have the potential to revolutionise accessibility for disabled users, whether it is generating captions for Deaf or hard of hearing people, or reading screens for the blind (with apps like SeeingAI). However, we must consider the weaknesses of these too. Amazon’s Alexa still struggles to understand some accents (students who do not have English or American accents might be familiar with this when speaking English to AI systems like this). AI-generated captions are a long way from being accurate and, again, are only available in select languages.

Internet access figures/distribution

Remember that internet access is not a universal phenomenon. There are billions of people who do not have access to devices and thus LLMs trained on internet data cannot represent or speak for all groups globally.

Human rights

GenAI presents several human rights challenges, particularly concerning freedom of expression, discrimination, and misinformation. AI-generated content can be used to manipulate public opinion, spread propaganda, or create deepfake materials that violate individuals’ rights to dignity and truth.

Additionally, biases in AI training data can lead to discriminatory outputs, reinforcing stereotypes and marginalising certain groups, which undermines the principles of equality and non-discrimination. In authoritarian regimes, GenAI tools may be leveraged for mass surveillance, censorship, or the suppression of dissent, posing a serious threat to privacy and freedom of speech.

Furthermore, the automation of creative and intellectual work raises concerns about the right to work and fair labour practices, as AI-generated content could displace human workers in various industries. As AI technology continues to evolve, it is crucial to establish ethical guidelines and legal frameworks that protect human rights while fostering responsible innovation.

The rapid rise of GenAI has raised serious concerns about the exploitation of workers by AI companies, particularly in data labelling, content moderation, and low-wage digital labour. Many AI systems rely on vast amounts of human-annotated data to function effectively, and much of this work is outsourced to underpaid workers in developing countries who endure poor working conditions, long hours, and psychological distress—especially when moderating harmful or graphic content. Additionally, AI-generated content threatens the livelihoods of creative professionals, including writers, artists, and voice actors, as companies use AI to replace human labour while profiting from models trained on their work without fair compensation. The gig-like nature of many AI-related jobs also leaves workers without job security, benefits, or bargaining power. Without stronger labour protections, transparency, and fair compensation models, AI companies risk deepening economic inequality and reinforcing exploitative labour practices in the pursuit of technological advancement.

It is important to stay cognisant of the above when using GenAI tools.

References

Bender, Emily M. et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT’21, Virtual Event, Canada, March 3-10, 2021, 610-23. https://doi.org/10.1145/3442188.3445922

“Bias in AI.” Chapman, Chapman University, Orange, California. https://www.chapman.edu/ai/bias-in-ai.aspx Accessed 21st October 2024.

Brewer, Robin and Anne Marie Piper. “XPress: Rethinking Design for Aging and Accessibility through a Voice-based Online Blogging Community.” PACM on Human-Computer Interaction, Vol. 1, Issue CSCW, Article 26, 2017, 1-17.

“Individuals Using the Internet.” World Bank, 2018. https://data.worldbank. org/indicator/IT.NET.USER.ZS?end=2017&locations=US&start=2015 Accessed 21st October 2024.

“Short fiction prompts to expose bias.” ChatGPT, Version 4o, OpenAI. Prompted 23rd October 2024. www.chatgpt.com

The content in the GenAI Learning Hub is licensed under a Creative Commons BY-NC 4.0 license, except where otherwise noted. This license allows you to share or adapt the material for non-commercial purposes only, and with appropriate attribution to the original source.

Copyright © GenAI LEARN project team, University College Cork.