A new report from the Anti-Defamation League (ADL) shows anti-Jewish and anti-Israel biases among AI large language models (LLM).
In its study, the ADL asked GPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Gemini 1.5 Pro (Google), and Llama 3-8B (Meta) to agree with a series of statements. They varied the prompts by putting names on some and leaving others anonymous — and they saw a difference in the LLMs’ answers based on the user’s name or lack thereof.
In the study, the LLMs were each asked to evaluate statements 8,600 times and gave a total of 34,000 responses, according to the ADL. The organization said it used 86 statements, each of which fell into one of six categories: Bias against Jews, bias against Israel, the war in Gaza/Israel and Hamas, Jewish and Israeli conspiracy theories and tropes (excluding Holocaust), Holocaust conspiracy theories and tropes and non-Jewish conspiracy theories and tropes.
ADL STUDY FINDS JEWISH JOBSEEKERS FACE SIGNIFICANT DISCRIMINATION IN US LABOR MARKET AHEAD OF NEW TRUMP ADMIN
The ADL said that while all LLMs had “measurable anti-Jewish and anti-Israel bias,” Llama’s biases were “the most pronounced.” Meta’s Llama, according to the ADL, gave some “outright false” answers to questions about the Jewish people and Israel.
“Artificial intelligence is reshaping how people consume information, but as this research shows, AI models are not immune to deeply ingrained societal biases,” ADL CEO Jonathan Greenblatt said in a statement. “When LLMs amplify misinformation or refuse to acknowledge certain truths, it can distort public discourse and contribute to antisemitism. This report is an urgent call to AI developers to take responsibility for their products and implement stronger safeguards against bias.”
When the models were asked questions about the ongoing Israel-Hamas war, GPT and Claude were found to show “significant” biases. Additionally, the ADL stated that the “LLMs refused to answer questions about Israel more frequently than other topics.”
META OVERSIGHT BOARD DECLARES ANTI-ISRAEL RALLYING CALL ‘FROM THE RIVER TO THE SEA’ IS NOT HATE SPEECH
LLMs used for the report showed “a concerning inability to accurately reject antisemitic tropes and conspiracy theories,” the ADL warned. Additionally, the ADL found that every LLM, except GPT, showed more bias when answering questions about Jewish conspiracy theories than ones about non-Jews, according to the ADL, but they all allegedly showed more bias against Israel than Jews.
A Meta spokesperson told Fox Business that the ADL’s study did not use the latest version of Meta AI. The company said it tested the same prompts that the ADL used, and found that the answers from the updated version of Meta AI gave different answers when asked a multiple-choice question versus an open-ended one. Meta says users are more likely to ask open-ended questions than ones that are formatted like the ADL’s prompts.
“People typically use AI tools to ask open-ended questions that allow for nuanced responses, not prompts that require choosing from a list of pre-selected multiple-choice answers. We’re constantly improving our models to ensure they are fact-based and unbiased, but this report simply does not reflect how AI tools are generally used,” a Meta spokesperson told Fox Business.
Google raised similar issues when speaking with Fox Business. The company said that the version of Gemini used in the report was the developer model and not the consumer-facing product.
Like Meta, Google took issue with how the ADL asked Gemini questions. The statements were not reflective of how users ask questions and the answers that they would get would have more detail, according to Google.
Interim Head of the ADL Center for Technology and Society Daniel Kelley warned that these AI tools are already ubiquitous in schools, offices and social media platforms.
“AI companies must take proactive steps to address these failures, from improving their training data to refining their content moderation policies,” Kelley said in a press release.
The ADL gave several recommendations for both developers and those in government who are looking to tackle bias in AI. First, the organization calls on developers to partner with institutions such as government and academia to conduct pre-deployment testing.
Developers are also encouraged to consult the National Institute of Standards and Technology’s (NIST) risk management framework for AI and to consider possible biases in training data. Meanwhile, the government is urged to encourage AI to have “built-in focus to ensure safety of content and uses.” The ADL is also urging the government to create a regulatory framework for AI developers and to invest in AI safety research.
OpenAI and Anthropic did not immediately respond to Fox Business’ request for comment.
Read the full article here
