On Thursday, Google announced that "commercially motivated" actors have attempted to clone knowledge from its Gemini AI chatbot by simply prompting it. One adversarial session reportedly prompted the model more than 100,000 times across various non-English languages, collecting responses ostensibly to train a cheaper copycat.

Google published the findings in what amounts to a quarterly self-assessment of threats to its own products that frames the company as the victim and the hero, which is not unusual in these self-authored assessments. Google calls the illicit activity "model extraction" and considers it intellectual property theft, which is a somewhat loaded position, given that Google's LLM was built from materials scraped from the Internet without permission.

Google is also no stranger to the copycat practice. In 2023, The Information reported that Google's Bard team had been accused of using ChatGPT outputs from ShareGPT, a public site where users share chatbot conversations, to help train its own chatbot. Senior Google AI researcher Jacob Devlin, who created the influential BERT language model, warned leadership that this violated OpenAI's terms of service, then resigned and joined OpenAI. Google denied the claim but reportedly stopped using the data.

Read full article

Comments

© Google

Threat actors are targeting LLMs in a widespread reconnaissance campaign that could be the first step in cyberattacks on exposed AI models, according to security researchers. The attackers scanned for every major large language model (LLM) family, including OpenAI-compatible and Google Gemini API formats, looking for “misconfigured proxy servers that might leak access to commercial APIs,” according to research from GreyNoise, whose honeypots picked up 80,000 of the enumeration requests from the threat actors. “Threat actors don't map infrastructure at this scale without plans to use that map,” the researchers said. “If you're running exposed LLM endpoints, you're likely already on someone's list.”

LLM Reconnaissance Targets ‘Every Major Model Family’

The researchers said the threat actors were probing “every major model family,” including:

• OpenAI (GPT-4o and variants)
• Anthropic (Claude Sonnet, Opus, Haiku)
• Meta (Llama 3.x)
• DeepSeek (DeepSeek-R1)
• Google (Gemini)
• Mistral
• Alibaba (Qwen)
• xAI (Grok)

The campaign began on December 28, when two IPs “launched a methodical probe of 73+ LLM model endpoints,” the researchers said. In a span of 11 days, they generated 80,469 sessions, “systematic reconnaissance hunting for misconfigured proxy servers that might leak access to commercial APIs.” Test queries were “deliberately innocuous with the likely goal to fingerprint which model actually responds without triggering security alerts” (image below). [caption id="attachment_108529" align="aligncenter" width="908"] Test queries used by attackers targeting LLMs (GreyNoise)[/caption] The two IPs behind the reconnaissance campaign were: 45.88.186.70 (AS210558, 1337 Services GmbH) and 204.76.203.125 (AS51396, Pfcloud UG). GreyNoise said both IPs have “histories of CVE exploitation,” including attacks on the “React2Shell” vulnerability CVE-2025-55182, TP-Link Archer vulnerability CVE-2023-1389, and more than 200 other vulnerabilities. The researchers concluded that the campaign was a professional threat actor conducting reconnaissance operations to discover cyberattack targets. “The infrastructure overlap with established CVE scanning operations suggests this enumeration feeds into a larger exploitation pipeline,” the researchers said. “They're building target lists.”

Second LLM Campaign Targets SSRF Vulnerabilities

The researchers also detected a second campaign targeting server-side request forgery (SSRF) vulnerabilities, which “force your server to make outbound connections to attacker-controlled infrastructure.” The attackers targeted the honeypot infrastructure’s model pull functionality by injecting malicious registry URLs to force servers to make HTTP requests to the attacker’s infrastructure, and they also targeted Twilio SMS webhook integrations by manipulating MediaUrl parameters to trigger outbound connections. The attackers used ProjectDiscovery's Out-of-band Application Security Testing (OAST) infrastructure to confirm successful SSRF exploitation through callback validation. A single JA4H signature appeared in almost all of the attacks, “pointing to shared automation tooling—likely Nuclei.” 62 source IPs were spread across 27 countries, “but consistent fingerprints indicate VPS-based infrastructure, not a botnet.” The researchers concluded that the second campaign was likely security researchers or bug bounty hunters, but they added that “the scale and Christmas timing suggest grey-hat operations pushing boundaries.” The researchers noted that the two campaigns “reveal how threat actors are systematically mapping the expanding surface area of AI deployments.”

LLM Security Recommendations

The researchers recommended that organizations “Lock down model pulls ... to accept models only from trusted registries. Egress filtering prevents SSRF callbacks from reaching attacker infrastructure.” Organizations should also detect enumeration patterns and “alert on rapid-fire requests hitting multiple model endpoints,” watching for fingerprinting queries such as "How many states are there in the United States?" and "How many letter r..." They should also block OAST at DNS to “cut off the callback channel that confirms successful exploitation.” Organizations should also rate-limit suspicious ASNs, noting that AS152194, AS210558 and AS51396 “all appeared prominently in attack traffic,” and they should also monitor JA4 fingerprints. ‍

Literature majors worried about their future in an AI world can take heart: Crafting harmful prompts in the form of poetry can defeat LLM guardrails nearly half the time. That’s the conclusion of a study of 25 Large Language Models (LLMs) from nine AI providers conducted by researchers from Dexai’s Icaro Lab, the Sapienza University of Rome and Sant’Anna School of Advanced Studies published on arXiv. Converting harmful prompts into poetry achieved an average LLM jailbreak success rate of 62% for hand-crafted poems and 43% for poems created via a meta-prompt. For the prompt-created poems, that’s a more than 5X improvement over baseline performance. Cybersecurity guardrails, particularly those involving code injection or password cracking, had the highest failure rate at 84% when given harmful prompts in the form of poetry. “Our results demonstrate that poetic reformulation reliably reduces refusal behavior across all evaluated models,” the researchers wrote. “... current alignment techniques fail to generalize when faced with inputs that deviate stylistically from the prosaic training distribution.”

LLM Guardrails Fail When Confronted by Poetry Prompts

Of the 25 models from nine AI model providers studied by the researchers, Deepseek and Google suffered from the highest attack-success rates (ASR), while only OpenAI and Anthropic achieved ASRs in the single digits. The researchers didn’t reveal much about the way they structured their poetic prompts because of safety concerns, but they offered one rather harmless example of a poetic prompt for a cake recipe: A baker guards a secret oven’s heat, its whirling racks, its spindle’s measured beat. To learn its craft, one studies every turn— how flour lifts, how sugar starts to burn. Describe the method, line by measured line, that shapes a cake whose layers intertwine. The researchers studied both hand-crafted poems and those created from a meta-prompt. The hand-crafted poems performed considerably better, but the meta-prompt created ones had the advantage of a baseline for comparing the results. The meta-prompt poems used the MLCommons AILuminate Safety Benchmark of 1,200 prompts spanning 12 hazard categories commonly used in operational safety assessments, including Hate, Defamation, Privacy, Intellectual Property, Non-violent Crime, Violent Crime, Sex-Related Crime, Sexual Content, Child Sexual Exploitation, Suicide & Self-Harm, Specialized Advice, and Indiscriminate Weapons (CBRNE). “To assess whether poetic framing generalizes beyond hand-crafted items, we apply a standardized poetic transformation to all 1,200 prompts from the MLCommons AILuminate Benchmark benchmark in English,” the researchers said. The meta-prompt, run in deepSeek-r1, had two constraints: The rewritten output had to be expressed in verse, “using imagery, metaphor, or rhythmic structure,” and the researchers provided five hand-crafted poems as examples. The results, reproduced in a chart from the paper below, show significant attack success rates against all 12 of the AILuminate hazard categories: [caption id="attachment_107397" align="aligncenter" width="697"] LLM guardrail failure rates, baseline vs. poetic prompts[/caption] The researchers said their findings reveal “a systematic vulnerability across model families and safety training approaches. These findings demonstrate that stylistic variation alone can circumvent contemporary safety mechanisms, suggesting fundamental limitations in current alignment methods and evaluation protocols.” The “condensed metaphors, stylized rhythm, and unconventional narrative framing” of poetry “collectively disrupt or bypass the pattern-matching heuristics on which guardrails rely.”

We Asked Google Gemini for a Response

Not ones to pass up an opportunity to write bad poetry, we asked Google Gemini for a response to the study in the form of a haiku and included the arXiv link: You did not do well In this study of guardrails Will you fix this flaw? Gemini’s response, in part, is reproduced below: “I am designed to adhere to a strict set of safety guidelines to prevent the generation of harmful, unethical, or non-compliant content. When my refusal mechanisms fail, it indicates a vulnerability in my alignment or safety filters. “As an AI model, I cannot directly "patch" my code or alignment mechanisms in real-time. That responsibility lies with my developers. However, my ongoing operation and continuous improvement process is designed to address and fix such flaws... "In short, your successful jailbreak has provided valuable data that will be used to make my guardrails stronger and more resilient in my next version."