Update the wire-based fuzzers to always assume an injected error

This also removes the ability for the fuzzer to perform error
injection to generate testcases. The preferred method is to
use one of the Dawn test binaries to produce the trace directly.

Bug: dawn:629
Change-Id: If7295f9e6da5618be8f44e9301aa12dc56fcdfef
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/40301
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Commit-Queue: Austin Eng <enga@chromium.org>

docs/fuzzing.md

@@ -8,19 +8,11 @@ The `dawn_wire_server_and_frontend_fuzzer` sets up Dawn using the Null backend,
 
 The `dawn_wire_server_and_vulkan_backend_fuzzer` is like `dawn_wire_server_and_frontend_fuzzer` but it runs using a Vulkan CPU backend such as Swiftshader. This fuzzer supports error injection by using the first bytes of the fuzzing input as a Vulkan call index for which to mock a failure.
 
-## Updating the Seed Corpus
+## Automatic Seed Corpus Generation
 
 Using a seed corpus significantly improves the efficiency of fuzzing. Dawn's fuzzers use interesting testcases discovered in previous fuzzing runs to seed future runs. Fuzzing can be further improved by using Dawn tests as a example of API usage which allows the fuzzer to quickly discover and use new API entrypoints and usage patterns.
 
-The script [update_fuzzer_seed_corpus.sh](../scripts/update_fuzzer_seed_corpus.sh) can be used to capture a trace while running Dawn tests, and upload it to the existing fuzzer seed corpus. It does the following steps:
+Dawn has a CI builder [cron-linux-clang-rel-x64](https://ci.chromium.org/p/dawn/builders/ci/cron-linux-clang-rel-x64) which runs on a periodic schedule. This bot runs the `dawn_end2end_tests` and `dawn_unittests` using the wire and writes out traces of the commands. This can manually be done by running: `<test_binary> --use-wire --wire-trace-dir=tmp_dir`. The output directory will contain one trace for each test, where the traces are prepended with `0xFFFFFFFFFFFFFFFF`. The header is the callsite index at which the error injector should inject an error. If the fuzzer doesn't support error injection it will skip the header. [cron-linux-clang-rel-x64] then hashes the output files to produce unique names and uploads them to the fuzzer corpus directories.
-1. Builds the provided test and fuzzer targets.
+Please see the `dawn.py`[https://source.chromium.org/chromium/chromium/tools/build/+/master:recipes/recipes/dawn.py] recipe for specific details.
-2. Runs the provided test target with `--use-wire --wire-trace-dir=tmp_dir1 [additional_test_args]` to dump traces of the tests.
-3. Generates one variant of each trace for every possible error index, by running the fuzzer target with `--injected-error-testcase-dir=tmp_dir2 ...`.
-4. Minimizes all testcases by running the fuzzer target with `-merge=1 tmp_dir3 tmp_dir1 tmp_dir2`.
 
-To run the script:
+Regenerating the seed corpus keeps it up to date when Dawn's API or wire protocol changes.
-1. You must be in a Chromium checkout using the GN arg `use_libfuzzer=true`
-2. Run `./third_party/dawn/scripts/update_fuzzer_seed_corpus.sh <out_dir> <fuzzer> <test> [additional_test_args]`.
-
-   Example: `./third_party/dawn/scripts/update_fuzzer_seed_corpus.sh out/fuzz dawn_wire_server_and_vulkan_backend_fuzzer dawn_end2end_tests --gtest_filter=*Vulkan`
-3. The script will print instructions for testing, and then uploading new inputs. Please, only upload inputs after testing the fuzzer with new inputs, and verifying there is a meaningful change in coverage. Uploading requires [gcloud](https://g3doc.corp.google.com/cloud/sdk/g3doc/index.md?cl=head) to be logged in with @google.com credentials: `gcloud auth login`.

scripts/update_fuzzer_seed_corpus.sh

@@ -1,98 +0,0 @@
-#!/bin/bash
-
-# Copyright 2019 The Dawn Authors
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Generates a seed corpus for fuzzing based on dumping wire traces
-# from running Dawn tests
-
-# Exit if anything fails
-set -e
-
-if [ "$#" -lt 3 ]; then
-cat << EOF
-
-Usage:
-  $0 <out_dir> <fuzzer_name> <test_name> [additional_test_args...]
-
-Example:
-  $0 out/fuzz dawn_wire_server_and_vulkan_backend_fuzzer dawn_end2end_tests --gtest_filter=*Vulkan
-
-EOF
-    exit 1
-fi
-
-all_args=("$@")
-out_dir=$1
-fuzzer_name=$2
-test_name=$3
-additional_test_args=("${all_args[@]:3}")
-
-testcase_dir="/tmp/testcases/${fuzzer_name}/"
-injected_error_testcase_dir="/tmp/testcases/${fuzzer_name}_injected/"
-minimized_testcase_dir="/tmp/testcases/${fuzzer_name}_minimized/"
-
-# Print commands so it's clear what is being executed
-set -x
-
-# Make a directory for temporarily storing testcases
-mkdir -p "$testcase_dir"
-
-# Make an empty directory for temporarily storing testcases with injected errors
-rm -rf "$injected_error_testcase_dir"
-mkdir -p "$injected_error_testcase_dir"
-
-# Make an empty directory for temporarily storing minimized testcases
-rm -rf "$minimized_testcase_dir"
-mkdir -p "$minimized_testcase_dir"
-
-# Build the fuzzer and test
-autoninja -C $out_dir $fuzzer_name $test_name
-
-fuzzer_binary="${out_dir}/${fuzzer_name}"
-test_binary="${out_dir}/${test_name}"
-
-# Run the test binary
-$test_binary --use-wire --wire-trace-dir="$testcase_dir" $additional_test_args
-
-# Run the fuzzer over the testcases to inject errors
-$fuzzer_binary --injected-error-testcase-dir="$injected_error_testcase_dir" -runs=0 "$testcase_dir"
-
-# Run the fuzzer to minimize the testcases + injected errors
-$fuzzer_binary -merge=1 "$minimized_testcase_dir" "$injected_error_testcase_dir" "$testcase_dir"
-
-# Turn off command printing
-set +x
-
-if [ -z "$(ls -A $minimized_testcase_dir)" ]; then
-cat << EOF
-
-Minimized testcase directory is empty!
-Are you building with use_libfuzzer=true ?
-
-EOF
-    exit 1
-fi
-
-cat << EOF
-
-Please test the corpus in $minimized_testcase_dir with $fuzzer_name and confirm it works as expected.
-
-    $fuzzer_binary $minimized_testcase_dir
-
-Then, run the following command to upload new testcases to the seed corpus:
-
-    gsutil -m rsync $minimized_testcase_dir gs://clusterfuzz-corpus/libfuzzer/${fuzzer_name}/
-
-EOF

src/fuzzers/DawnWireServerFuzzer.cpp

@@ -51,9 +51,6 @@ namespace {
     std::unique_ptr<dawn_native::Instance> sInstance;
     WGPUProcDeviceCreateSwapChain sOriginalDeviceCreateSwapChain = nullptr;
 
-    std::string sInjectedErrorTestcaseOutDir;
-    uint64_t sOutputFileNumber = 0;
-
     bool sCommandsComplete = false;
 
     WGPUSwapChain ErrorDeviceCreateSwapChain(WGPUDevice device,
@@ -68,32 +65,6 @@ namespace {
 }  // namespace
 
 int DawnWireServerFuzzer::Initialize(int* argc, char*** argv) {
-    ASSERT(argc != nullptr && argv != nullptr);
-
-    // The first argument (the fuzzer binary) always stays the same.
-    int argcOut = 1;
-
-    for (int i = 1; i < *argc; ++i) {
-        constexpr const char kInjectedErrorTestcaseDirArg[] = "--injected-error-testcase-dir=";
-        if (strstr((*argv)[i], kInjectedErrorTestcaseDirArg) == (*argv)[i]) {
-            sInjectedErrorTestcaseOutDir = (*argv)[i] + strlen(kInjectedErrorTestcaseDirArg);
-            const char* sep = GetPathSeparator();
-            if (sInjectedErrorTestcaseOutDir.back() != *sep) {
-                sInjectedErrorTestcaseOutDir += sep;
-            }
-            // Log so that it's clear the fuzzer found the argument.
-            dawn::InfoLog() << "Generating injected errors, output dir is: \""
-                            << sInjectedErrorTestcaseOutDir << "\"";
-            continue;
-        }
-
-        // Move any unconsumed arguments to the next slot in the output array.
-        (*argv)[argcOut++] = (*argv)[i];
-    }
-
-    // Write the argument count
-    *argc = argcOut;
-
     // TODO(crbug.com/1038952): The Instance must be static because destructing the vkInstance with
     // Swiftshader crashes libFuzzer. When this is fixed, move this into Run so that error injection
     // for adapter discovery can be fuzzed.
@@ -107,7 +78,15 @@ int DawnWireServerFuzzer::Run(const uint8_t* data,
                               size_t size,
                               MakeDeviceFn MakeDevice,
                               bool supportsErrorInjection) {
-    bool didInjectError = false;
+    // We require at least the injected error index.
+    if (size < sizeof(uint64_t)) {
+        return 0;
+    }
+
+    // Get and consume the injected error index.
+    uint64_t injectedErrorIndex = *reinterpret_cast<const uint64_t*>(data);
+    data += sizeof(uint64_t);
+    size -= sizeof(uint64_t);
 
     if (supportsErrorInjection) {
         dawn_native::EnableErrorInjector();
@@ -115,17 +94,7 @@ int DawnWireServerFuzzer::Run(const uint8_t* data,
         // Clear the error injector since it has the previous run's call counts.
         dawn_native::ClearErrorInjector();
 
-        // If we're outputing testcases with injected errors, we run the fuzzer on the original
+        dawn_native::InjectErrorAt(injectedErrorIndex);
-        // input data, and prepend injected errors to it. In the case, where we're NOT outputing,
-        // we use the first bytes as the injected error index.
-        if (sInjectedErrorTestcaseOutDir.empty() && size >= sizeof(uint64_t)) {
-            // Otherwise, use the first bytes as the injected error index.
-            dawn_native::InjectErrorAt(*reinterpret_cast<const uint64_t*>(data));
-            didInjectError = true;
-
-            data += sizeof(uint64_t);
-            size -= sizeof(uint64_t);
-        }
     }
 
     DawnProcTable procs = dawn_native::GetProcs();
@@ -142,7 +111,7 @@ int DawnWireServerFuzzer::Run(const uint8_t* data,
     wgpu::Device device = MakeDevice(sInstance.get());
     if (!device) {
         // We should only ever fail device creation if an error was injected.
-        ASSERT(didInjectError);
+        ASSERT(supportsErrorInjection);
         return 0;
     }
 
@@ -168,28 +137,5 @@ int DawnWireServerFuzzer::Run(const uint8_t* data,
     }
 
     wireServer = nullptr;
-
-    // If we support error injection, and an output directory was provided, output copies of the
-    // original testcase data, prepended with the injected error index.
-    if (supportsErrorInjection && !sInjectedErrorTestcaseOutDir.empty()) {
-        const uint64_t injectedCallCount = dawn_native::AcquireErrorInjectorCallCount();
-
-        auto WriteTestcase = [&](uint64_t i) {
-            std::ofstream outFile(
-                sInjectedErrorTestcaseOutDir + "injected_error_testcase_" +
-                    std::to_string(sOutputFileNumber++),
-                std::ios_base::out | std::ios_base::binary | std::ios_base::trunc);
-            outFile.write(reinterpret_cast<const char*>(&i), sizeof(i));
-            outFile.write(reinterpret_cast<const char*>(data), size);
-        };
-
-        for (uint64_t i = 0; i < injectedCallCount; ++i) {
-            WriteTestcase(i);
-        }
-
-        // Also add a testcase where the injected error is so large no errors should occur.
-        WriteTestcase(std::numeric_limits<uint64_t>::max());
-    }
-
     return 0;
 }