Google open-sources artificial intelligence (AI)-assisted fuzzy testing framework

Preface

In recent years.cyber securityVulnerabilities are frequent, causing huge losses to individuals and businesses. To help developers and researchers find vulnerabilities faster, Google has open-sourced itsAI-assistedfuzzy testThe Framework. The framework utilizes the Large Language Model (LLM) to generate fuzzy test cases for real-world C and C++ projects and benchmarks them using Google's OSS-Fuzz service, which has long been a leading resource for automated discovery of vulnerabilities in open source software.

AI-enabled fuzz testing significantly improves vulnerability discovery efficiency

To automate some aspects of manual fuzz testing, Google began using LLM in August 2023 to "write project-specific code to improve fuzz test coverage and find more vulnerabilities," which resulted in an increase of 301 TP3T in code coverage for over 300 OSS-Fuzz C/C++ projects.

Google said, "So far, the extended fuzz test coverage provided by the improvements generated by LLM has enabled OSS-Fuzz to discover two new vulnerabilities in cJSON and libplist, two widely used projects that have been fuzz tested for years."

Open source frameworks offer a wide range of applications

The open source tool supports Vertex AI code-bison, Vertex AI code-bison-32k, Gemini Pro, OpenAI GPT-3.5-turbo, and OpenAI GPT-4. Additionally, Google said the tool uses the latest data from production environments to evaluate generated fuzzy tests based on four metrics targets, which include compilability, runtime crashes, runtime coverage, and runtime line coverage differences from existing manually-written fuzzy test targets in OSS-Fuzz.

Google notes, "All in all, the framework successfully utilized LLM to generate effective fuzzy test targets (producing non-zero coverage increases) for 160 C/C++ projects. The maximum line coverage increase was 291 TP3T higher than the existing manually written targets."

The open source framework allows researchers and developers to test the validity of generated fuzzy test objectives using their own prompts and measure the results against the OSS-Fuzz C/C++ project.

More than vulnerability discovery, AI powers vulnerability remediation

In addition to fuzzing tests for vulnerability discovery, Google is also looking at how LLM can be used for vulnerability patching and has proposed a project to build an automated pipeline of LLM-generated and tested fixes.

Google said, "This AI-driven patching approach solves the 15% targeted vulnerability and saves engineers a significant amount of time. The potential of this technique should apply to most or all categories of the software development process."

OSS-Fuzz, a traditional fuzzy testing tool.

Introduces OSS-Fuzz, another well-known fuzzy testing tool open-sourced by Google.OSS-Fuzz is a Continuous Integration/Continuous Delivery (CI/CD) pipeline that automates the integration of fuzzy testing into the development process of open source software projects. It utilizes a variety of fuzz testing engines and memory error detection tools to find security vulnerabilities and stability issues.

As of August 2023, OSS-Fuzz has helped identify and fix over 10,000 vulnerabilities and 36,000 bugs in 1,000 projects.

Summary and reference

Google's open-source AI-assisted Fuzzing framework is a major innovation in the field of software development security, OSS-Fuzz represents the latest advances in fuzz testing and provides a powerful tool to improve the security of open-source software This not only improves the efficiency of the existing Fuzzing methodology, but also accelerates vulnerability discovery and remediation process by automatically generating Fuzz targets. This not only improves the efficiency of existing Fuzzing methods, but also accelerates the vulnerability discovery and remediation process through automatically generated Fuzz targets. As the application of AI technology in the software development process continues to deepen, we have reason to believe that with the continuous development of AI technology, fuzzy testing will become smarter and more efficient, helping to build a more secure cyberspace.

Project: https://github.com/google/oss-fuzz