debian-mirror-gitlab/doc/user/application_security/api_fuzzing/index.md

stage	group	info	type
Secure	Fuzz Testing	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers	reference, howto

Web API Fuzz Testing (ULTIMATE)

You can add web API fuzzing to your GitLab CI/CD pipelines. This helps you discover bugs and potential security issues that other QA processes may miss. API fuzzing performs fuzz testing of API operation parameters. Fuzz testing sets operation parameters to unexpected values in an effort to cause unexpected behavior and errors in the API backend.

We recommend that you use fuzz testing in addition to GitLab Secure's other security scanners and your own test processes. If you're using GitLab CI/CD, you can run fuzz tests as part your CI/CD workflow.

Requirements

• One of the following web API types:
  • REST API
  • SOAP
  • GraphQL
  • Form bodies, JSON, or XML
• An OpenAPI definition, or HTTP Archive (HAR) of requests to test

When fuzzing scans run

When using the API-Fuzzing.gitlab-ci.yml template, the fuzz job runs last, as shown here. To ensure API fuzzing scans the latest code, your CI pipeline should deploy changes to a test environment in one of the jobs preceding the fuzz job:

stages:
  - build
  - test
  - deploy
  - fuzz

Note that if your pipeline is configured to deploy to the same web server on each run, running a pipeline while another is still running could cause a race condition in which one pipeline overwrites the code from another. The API to scan should be excluded from changes for the duration of a fuzzing scan. The only changes to the API should be from the fuzzing scanner. Be aware that any changes made to the API (for example, by users, scheduled tasks, database changes, code changes, other pipelines, or other scanners) during a scan could cause inaccurate results.

Configuration

There are two ways to perform scans. See the configuration section for the one you wish to use:

• OpenAPI v2 specification
• HTTP Archive (HAR)

Examples of both configurations can be found here:

• Example OpenAPI v2 specification project
• Example HTTP Archive (HAR) project

OpenAPI Specification

The OpenAPI Specification (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing by using an OpenAPI specification to provide information about the target API to test. OpenAPI specifications are provided as a filesystem resource or URL.

Follow these steps to configure API fuzzing in GitLab with an OpenAPI specification:

1. To use API fuzzing, you must include the API-Fuzzing.gitlab-ci.yml template that's provided as part of your GitLab installation. To do so, add the following to your .gitlab-ci.yml file:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   

2. Add the configuration file gitlab-api-fuzzing-config.yml to your repository's root as .gitlab-api-fuzzing.yml.

3. The configuration file has several testing profiles defined with varying amounts of fuzzing. We recommend that you start with the Quick-10 profile. Testing with this profile completes quickly, allowing for easier configuration validation.

   Provide the profile by adding the FUZZAPI_PROFILE variable to your .gitlab-ci.yml file, substituting Quick-10 for the profile you choose:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
   

4. Provide the location of the OpenAPI v2 specification. You can provide the specification as a file or URL. Specify the location by adding the FUZZAPI_OPENAPI variable:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
     FUZZAPI_OPENAPI: test-api-specification.json
   

5. The target API instance's base URL is also required. Provide it by using the FUZZAPI_TARGET_URL variable or an environment_url.txt file.

   Adding the URL in an environment_url.txt file at your project's root is great for testing in dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its domain in an environment_url.txt file. API fuzzing automatically parses that file to find its scan target. You can see an example of this in our Auto DevOps CI YAML.

   Here's an example of using FUZZAPI_TARGET_URL:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
     FUZZAPI_OPENAPI: test-api-specification.json
     FUZZAPI_TARGET_URL: http://test-deployment/
   

This is a minimal configuration for API Fuzzing. From here you can:

• Run your first scan.
• Add authentication.
• Learn how to handle false positives.

DANGER: Danger: NEVER run fuzz testing against a production server. Not only can it perform any function that the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting data. Only run fuzzing against a test server.

HTTP Archive (HAR)

The HTTP Archive format (HAR) is an archive file format for logging HTTP transactions. When used with GitLab's API fuzzer, HAR must contain records of calling the web API to test. The API fuzzer extracts all the requests and uses them to perform testing.

You can use various tools to generate HAR files:

• Fiddler: Web debugging proxy
• Insomnia Core: API client
• Chrome: Browser
• Firefox: Browser

DANGER: Warning: HAR files may contain sensitive information such as authentication tokens, API keys, and session cookies. We recommend that you review the HAR file contents before adding them to a repository.

Follow these steps to configure API fuzzing to use a HAR file that provides information about the target API to test:

1. To use API fuzzing, you must include the API-Fuzzing.gitlab-ci.yml template that's provided as part of your GitLab installation. To do so, add the following to your .gitlab-ci.yml file:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   

2. Add the configuration file gitlab-api-fuzzing-config.yml to your repository's root as .gitlab-api-fuzzing.yml.

3. The configuration file has several testing profiles defined with varying amounts of fuzzing. We recommend that you start with the Quick-10 profile. Testing with this profile completes quickly, allowing for easier configuration validation.

   Provide the profile by adding the FUZZAPI_PROFILE variable to your .gitlab-ci.yml file, substituting Quick-10 for the profile you choose:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
   

4. Add the FUZZAPI_HAR variable and set it to the HAR file's location:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
     FUZZAPI_HAR: test-api-recording.har
   

5. The target API instance's base URL is also required. Provide it by using the FUZZAPI_TARGET_URL variable or an environment_url.txt file.

   Adding the URL in an environment_url.txt file at your project's root is great for testing in dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its domain in an environment_url.txt file. API fuzzing automatically parses that file to find its scan target. You can see an example of this in our Auto DevOps CI YAML.

   Here's an example of using FUZZAPI_TARGET_URL:

   include:
     - template: API-Fuzzing.gitlab-ci.yml
   
   variables:
     FUZZAPI_PROFILE: Quick-10
     FUZZAPI_HAR: test-api-recording.har
     FUZZAPI_TARGET_URL: http://test-deployment/
   

This is a minimal configuration for API Fuzzing. From here you can:

• Run your first scan.
• Add authentication.
• Learn how to handle false positives.

DANGER: Danger: NEVER run fuzz testing against a production server. Not only can it perform any function that the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting data. Only run fuzzing against a test server.

Authentication

Authentication is handled by providing the authentication token as a header or cookie. You can provide a script that performs an authentication flow or calculates the token.

HTTP Basic Authentication

HTTP basic authentication is an authentication method built into the HTTP protocol and used in-conjunction with transport layer security (TLS). To use HTTP basic authentication, two variables are added to your .gitlab-ci.yml file:

• FUZZAPI_HTTP_USERNAME: The username for authentication.
• FUZZAPI_HTTP_PASSWORD: The password for authentication.

For the password, we recommended that you create a CI/CD variable (for example, TEST_API_PASSWORD) set to the password. You can create CI/CD variables from the GitLab projects page at Settings > CI/CD, in the Variables section.

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_HAR: test-api-recording.har
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_HTTP_USERNAME: testuser
  FUZZAPI_HTTP_PASSWORD: $TEST_API_PASSWORD

Bearer Tokens

Bearer tokens are used by several different authentication mechanisms, including OAuth2 and JSON Web Tokens (JWT). Bearer tokens are transmitted using the Authorization HTTP header. To use bearer tokens with API fuzzing, you need one of the following:

• A token that doesn't expire
• A way to generate a token that lasts the length of testing
• A Python script that API fuzzing can call to generate the token

Token doesn't expire

If the bearer token doesn't expire, you can provide it using the FUZZAPI_OVERRIDES_ENV variable. The FUZZAPI_OVERRIDES_ENV content is a JSON snippet that provides headers and cookies that should be added to outgoing HTTP requests made by API fuzzing.

Create a CI/CD variable, for example TEST_API_BEARERAUTH, with the value {"headers":{"Authorization":"Bearer dXNlcm5hbWU6cGFzc3dvcmQ="}} (substitute your token). You can create CI/CD variables from the GitLab projects page at Settings > CI/CD in the Variables section.

Set FUZZAPI_OVERRIDES_ENV in your .gitlab-ci.yml file:

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: $TEST_API_BEARERAUTH

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API's application logs.

Token generated at test-runtime

If the bearer token must be generated, and the resulting token doesn't expire during testing, you can provide to API fuzzing a file containing the token. This file can be generated by a prior stage and job, or as part of the API fuzzing job.

API fuzzing expects to receive a JSON file with the following structure:

{
  "headers" : {
    "Authorization" : "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

This file can be generated by a prior stage and provided to API fuzzing through the FUZZAPI_OVERRIDES_FILE variable.

Set FUZZAPI_OVERRIDES_FILE in your .gitlab-ci.yml file:

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API's application logs.

Token has short expiration

If the bearer token must be generated and expires prior to the scan's completion, you can provide a program or script for the API fuzzer to execute on a provided interval. The provided script runs in an Alpine Linux container that has Python 3 and Bash installed. If the Python script requires additional packages, it must detect this and install the packages at runtime.

The script must create a JSON file containing the bearer token in a specific format:

{
  "headers" : {
    "Authorization" : "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

You must provide three variables, each set for correct operation:

• FUZZAPI_OVERRIDES_FILE: File generated by the provided command.
• FUZZAPI_OVERRIDES_CMD: Command to generate JSON file.
• FUZZAPI_OVERRIDES_INTERVAL: Interval in seconds to run command.

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick-10
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
  FUZZAPI_OVERRIDES_CMD: renew_token.py
  FUZZAPI_OVERRIDES_INTERVAL: 300

To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and the test API's application logs.

Configuration files

To get started quickly, GitLab provides you with the configuration file gitlab-api-fuzzing-config.yml. This file has several testing profiles that perform various amounts of testing. The run time of each increases as the numbers go up. To use a configuration file, add it to your repository's root as .gitlab-api-fuzzing.yml.

Profile	Scan Type
Quick-10	Fuzzing 10 times per parameter
Medium-20	Fuzzing 20 times per parameter
Medium-50	Fuzzing 50 times per parameter
Long-100	Fuzzing 100 times per parameter

Available variables

Environment variable	Description
FUZZAPI_VERSION	Specify API Fuzzing container version. Defaults to latest.
FUZZAPI_TARGET_URL	Base URL of API testing target.
FUZZAPI_CONFIG	API Fuzzing configuration file. Defaults to .gitlab-apifuzzer.yml.
FUZZAPI_PROFILE	Configuration profile to use during testing. Defaults to Quick.
FUZZAPI_REPORT	Scan report filename. Defaults to gl-api_fuzzing-report.xml.
FUZZAPI_OPENAPI	OpenAPI specification file or URL.
FUZZAPI_HAR	HTTP Archive (HAR) file.
FUZZAPI_OVERRIDES_FILE	Path to a JSON file containing overrides.
FUZZAPI_OVERRIDES_ENV	JSON string containing headers to override.
FUZZAPI_OVERRIDES_CMD	Overrides command.
FUZZAPI_OVERRIDES_INTERVAL	How often to run overrides command in seconds. Defaults to 0 (once).
FUZZAPI_HTTP_USERNAME	Username for HTTP authentication.
FUZZAPI_HTTP_PASSWORD	Password for HTTP authentication.

Overrides

API Fuzzing provides a method to add or override headers and cookies for all outbound HTTP requests made. You can use this to inject semver headers, authentication, and so on. The authentication section includes examples of using overrides for that purpose.

Overrides uses a JSON document to define the headers and cookies:

{
  "headers": {
    "header1": "value",
    "header2": "value"
  },
  "cookies": {
    "cookie1": "value",
    "cookie2": "value"
  }
}

Example usage for setting a single header:

{
  "headers": {
    "Authorization": "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  }
}

Example usage for setting both a header and cookie:

{
  "headers": {
    "Authorization": "Bearer dXNlcm5hbWU6cGFzc3dvcmQ="
  },
  "cookies": {
    "flags": "677"
  }
}

You can provide this JSON document as a file or environment variable. You may also provide a command to generate the JSON document. The command can run at intervals to support values that expire.

Using a file

To provide the overrides JSON as a file, the FUZZAPI_OVERRIDES_FILE environment variable is set. The path is relative to the job current working directory.

Example .gitlab-ci.yml:

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json

Using an environment variable

To provide the overrides JSON as an environment variable, use the FUZZAPI_OVERRIDES_ENV variable. This allows you to place the JSON as CI/CD variables that can be masked and protected.

In this example .gitlab-ci.yml, the JSON is provided directly:

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: '{"headers":{"X-API-Version":"2"}}'

In this example .gitlab-ci.yml, the CI/CD variable SECRET_OVERRIDES provides the JSON. This is a group or instance level environment variable defined in the UI:

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_ENV: $SECRET_OVERRIDES

Using a command

If the value must be generated or regenerated on expiration, you can provide a program or script for the API fuzzer to execute on a specified interval. The provided script runs in an Alpine Linux container that has Python 3 and Bash installed. If the Python script requires additional packages, it must detect this and install the packages at runtime. The script creates the overrides JSON file as defined above.

You must provide three variables, each set for correct operation:

• FUZZAPI_OVERRIDES_FILE: File generated by the provided command.
• FUZZAPI_OVERRIDES_CMD: Command to generate JSON file.
• FUZZAPI_OVERRIDES_INTERVAL: Interval in seconds to run command.

include:
  - template: API-Fuzzing.gitlab-ci.yml

variables:
  FUZZAPI_PROFILE: Quick
  FUZZAPI_OPENAPI: test-api-specification.json
  FUZZAPI_TARGET_URL: http://test-deployment/
  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
  FUZZAPI_OVERRIDES_CMD: renew_token.py
  FUZZAPI_OVERRIDES_INTERVAL: 300

Running your first scan

When configured correctly, a CI/CD pipeline contains a Fuzz stage and a apifuzzer_fuzz job. The job only fails when an invalid configuration is provided. During normal operation, the job always succeeds even if faults are identified during fuzz testing.

Faults are displayed on the Tests pipeline tab with the suite name API-Fuzzing. The Name field on the Tests page includes the fuzz-tested operation and parameter. The Trace field contains a writeup of the identified fault. This writeup contains information on what the fuzzer tested and how it detected something wrong.

To prevent an excessive number of reported faults, the API fuzzing scanner limits the number of faults it reports to one per parameter.

Fault Writeup

The faults that API fuzzing finds aren't associated with a specific vulnerability type. They require investigation to determine what type of issue they are and if they should be fixed. See handling false positives for information about configuration changes you can make to limit the number of false positives reported.

This table contains a description of fields in an API fuzzing fault writeup.

Writeup Item	Description
Operation	The operation tested.
Parameter	The field modified. This can be a path segment, header, query string, or body element.
Endpoint	The endpoint being tested.
Check	Check module producing the test. Checks can be turned on and off.
Assert	Assert module that detected a failure. Assertions can be configured and turned on and off.
CWE	Fuzzing faults always have the same CWE.
OWASP	Fuzzing faults always have the same OWASP ID.
Exploitability	Fuzzing faults always have an unknown exploitability.
Impact	Fuzzing faults always have an unknown risk impact.
Description	Verbose description of what the check did. Includes the original parameter value and the modified (mutated) value.
Detection	Why a failure was detected and reported. This is related to the Assert that was used.
Original Request	The original, unmodified HTTP request. Useful when reviewing the actual request to see what changes were made.
Actual Request	The request that produced the failure. This request has been modified in some way by the Check logic.
Actual Response	The response to the actual request.
Recorded Request	An unmodified request.
Recorded Response	The response to the unmodified request. You can compare this with the actual request when triaging this fault.

Handling False Positives

False positives can be handled in two ways:

• Turn off the Check producing the false positive. This prevents the check from generating any faults. Example checks are the JSON Fuzzing Check, and Form Body Fuzzing Check.
• Fuzzing checks have several methods of detecting when a fault is identified, called Asserts. Asserts can also be turned off and configured. For example, the API fuzzer by default uses HTTP status codes to help identify when something is a real issue. If an API returns a 500 error during testing, this creates a fault. This isn't always desired, as some frameworks return 500 errors often.

Turn off a Check

Checks perform testing of a specific type and can be turned on and off for specific configuration profiles. The provided configuration files define several profiles that you can use. The profile definition in the configuration file lists all the checks that are active during a scan. To turn off a specific check, simply remove it from the profile definition in the configuration file. The profiles are defined in the Profiles section of the configuration file.

Example profile definition:

Profiles:
  - Name: Quick-10
    DefaultProfile: Quick
    Routes:
      - Route: *Route0
        Checks:
          - Name: FormBodyFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
          - Name: GeneralFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
          - Name: JsonFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
          - Name: XmlFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true

To turn off the General Fuzzing Check you can remove these lines:

- Name: GeneralFuzzingCheck
  Configuration:
    FuzzingCount: 10
    UnicodeFuzzing: true

This results in the following YAML:

- Name: Quick-10
  DefaultProfile: Quick
  Routes:
    - Route: *Route0
      Checks:
        - Name: FormBodyFuzzingCheck
          Configuration:
            FuzzingCount: 10
            UnicodeFuzzing: true
        - Name: JsonFuzzingCheck
          Configuration:
            FuzzingCount: 10
            UnicodeFuzzing: true
        - Name: XmlFuzzingCheck
          Configuration:
            FuzzingCount: 10
            UnicodeFuzzing: true

Turn off an Assertion for a Check

Assertions detect faults in tests produced by checks. Many checks support multiple Assertions such as Log Analysis, Response Analysis, and Status Code. When a fault is found, the Assertion used is provided. To identify which Assertions are on by default, see the Checks default configuration in the configuration file. The section is called Checks.

This example shows the FormBody Fuzzing Check:

Checks:
  - Name: FormBodyFuzzingCheck
    Configuration:
      FuzzingCount: 30
      UnicodeFuzzing: true
    Assertions:
      - Name: LogAnalysisAssertion
      - Name: ResponseAnalysisAssertion
      - Name: StatusCodeAssertion

Here you can see three Assertions are on by default. A common source of false positives is StatusCodeAssertion. To turn it off, modify its configuration in the Profiles section. This example provides only the other two Assertions (LogAnalysisAssertion, ResponseAnalysisAssertion). This prevents FormBodyFuzzingCheck from using StatusCodeAssertion:

Profiles:
  - Name: Quick-10
    DefaultProfile: Quick
    Routes:
      - Route: *Route0
        Checks:
          - Name: FormBodyFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
            Assertions:
              - Name: LogAnalysisAssertion
              - Name: ResponseAnalysisAssertion
          - Name: GeneralFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
          - Name: JsonFuzzingCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true
          - Name: XmlInjectionCheck
            Configuration:
              FuzzingCount: 10
              UnicodeFuzzing: true

Glossary

• Assert: Assertions are detection modules used by checks to trigger a fault. Many assertions have configurations. A check can use multiple Assertions. For example, Log Analysis, Response Analysis, and Status Code are common Assertions used together by checks. Checks with multiple Assertions allow them to be turned on and off.
• Check: Performs a specific type of test, or performed a check for a type of vulnerability. For example, the JSON Fuzzing Check performs fuzz testing of JSON payloads. The API fuzzer is comprised of several checks. Checks can be turned on and off in a profile.
• Fault: During fuzzing, a failure identified by an Assert is called a fault. Faults are investigated to determine if they are a security vulnerability, a non-security issue, or a false positive. Faults don't have a known vulnerability type until they are investigated. Example vulnerability types are SQL Injection and Denial of Service.
• Profile: A configuration file has one or more testing profiles, or sub-configurations. You may have a profile for feature branches and another with extra testing for a main branch.