Up to 100 malicious artificial intelligence (AI)/machine learning (ML) models have been discovered by open-source platforms

David Cohen, senior security researcher, said, "The model's payload provides an attacker with a shell over the infected machine, allowing them to take full control of the victim's machine through what is commonly referred to as a 'backdoor'."

"This silent infiltration could grant access to critical internal systems and pave the way for large-scale data breaches and even corporate espionage, affecting not only individual users but entire organizations across the globe, while leaving victims completely unaware of their compromised status."

Specifically, the rogue model initiates a reverse shell connection to 210.117.212[.] 93 a reverse shell connection to an IP address belonging to the Korea Research Environment Open Network (KREONET). Other repositories with the same load have been observed connecting to other IP addresses.

In one case, the author of the model urged users not to download it, increasing the likelihood that the publication could be the work of a researcher or AI practitioner.

"However, a fundamental principle of security research is to avoid publishing exploits or malicious code that actually works," JFrog said. "This principle is violated when malicious code tries to connect back to a real IP address."

The findings re-emphasize the threats lurking in open source repositories that can be poisoned by malicious activity.

From Supply Chain Risk to Zeroing in on the Worm #

At the same time, the researchers devised effective methods to generate cues that can be used to elicit harmful responses from Large Language Models (LLMs) using Beam Search-based Adversarial Attack (BEAST) techniques.

In a related development, security researchers have developed a generative AI worm called Morris II that is capable of stealing data and spreading it through multiple systemsmalicious software.

Morris II is a variant of one of the oldest computer worms that utilizes antagonistic self-replicating cues encoded into inputs such as images and text when GenAI When models process these cues, they can be triggered to "copy inputs as outputs (replication) and engage in attacks. Malicious activity (payload)," said security researchers Stav Cohen, Ron Bitton and Ben Nassi.

More disturbingly, these models can be weaponized to provide malicious input to new applications by exploiting connections within the generative AI ecosystem.

This attack technique, known as ComPromptMized, has similarities to traditional methods such as buffer overflows and SQL injections, as it embeds code and data from a query into a region known to hold executable code.

ComPromptMized affects applications whose execution process relies on the output of generative AI services as well as applications that use Retrieval Augmented Generation (RAG), which combines a text generation model with an information retrieval component to enrich query responses.

This study is not the first, nor will it be the last, to explore the use of instant injection as a method of attacking LLMs and tricking them into performing unexpected operations.

Previously, scholars have demonstrated attacks that use image and audio recordings to inject invisible "adversarial perturbations" into a multimodal LLM, causing the model to output attacker-selected text or instructions.

In a paper published late last year, Nassi, along with Eugene Bagdasaryan, Tsung-Yin Hsieh, and Vitaly Shmatikov, said, "Attackers may lure victims to visit Web pages with interesting images or send emails with audio clips. "

"When a victim feeds an image or clip directly into an isolated LLM and asks relevant questions, the model will be guided by the prompts injected by the attacker."

Early last year, CISPA Helmholtz, Saarland University, Germanyinformation securityA group of researchers at the Center and Sequire Technology also discovered how an attacker can exploit an LLM model by strategically injecting hidden hints into data that the model is likely to appear (i.e., indirect hint injection). retrieved in response to user input.