`-o` will emit the files to the given root output directory
`--verbose` will print what's going on, to help with debugging
Omitting these flags will behave as before.
Also consolidate the utils package into fileutils. These were two packages with near identical functionality.
Change-Id: I855dd4b57807fb9239a52e7f357842d4ba2517ee
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/107687
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
This CL updates the intrinsics lexer to allow negative values for int
and float numerics. This allows doing `@test_value(-2)` in the def file.
Change-Id: I2cad9b25a2932057ce9bc51dec6c32231e06f0a0
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/107440
Commit-Queue: Dan Sinclair <dsinclair@chromium.org>
Reviewed-by: Ben Clayton <bclayton@google.com>
Auto-Submit: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
• Add `cts validate` command used to check for expectation collisions.
Can be used as a presubmit check.
This is more tightly checked than the previous logic, as this works on just
the expectations, instead of results.
• Fix an issue where the test result reduction could introduce collisions with
'Skip' expectations.
To fix this, the update process first adds 'consumed' results for the skipped
tests, preventing test tree reduction for that part of the tree.
• Fix a bug in the generation of 'New failures' and 'New flakes' which produced
more expectations than was necessary.
The issue here was that the tree roots could contain overlaps, and roots could
be processed before sub-trees, resulting in inefficient expectations.
• Fix collisions in the expectations file, and update with results from
the most recent roll.
Change-Id: I7b64553408998fb4416458ce564fc49c8f6d4d07
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101860
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Austin Eng <enga@chromium.org>
may_exonerate indicates that a test failed for a known issue that
we could exonerate. Merging of test results now removes results
with may_exonerate unless all of them were tagged as such. So, if
for example, a test fails for a known timeout issue, but has a
subsequent pass, the timeout will be ignored.
This serves to reduce the impact of known, hard-to-fix issues and
allow the CTS roller to make progress with less noise.
Change-Id: I5103a666496398a17b3aa6ccf3f267421e40ba97
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101804
Reviewed-by: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Austin Eng <enga@chromium.org>
Also parallelize some of the more lengthy file generation steps
Bug: dawn:1479
Change-Id: I7674fca4958e4d9948e287008916c4b0d33e1ca1
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/97022
Reviewed-by: Austin Eng <enga@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Always report the Javascript duration from the page instead of
the total duration from ResultDB. Reporting of the total duration
could be added back in the future, but usages of the tooling are
currently interested in the Javascript duration.
Bug: chromium:1336333
Change-Id: I74e91e7a9c29f3a4c45ed67898cc6f2de62e19a4
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/94762
Reviewed-by: Ben Clayton <bclayton@google.com>
Commit-Queue: Austin Eng <enga@chromium.org>
These sources need to be generated in Dawn's source tree
so that Chromium's //third_party/webgpu-cts DEP can be
removed.
Bug: chromium:1333969
Change-Id: I03c8cba691bcbfac00839f0cdd40fab6198ec83f
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/91060
Commit-Queue: Austin Eng <enga@chromium.org>
Reviewed-by: Ben Clayton <bclayton@google.com>
Adds an "-aggregate" flags to ./tools/run/cts time to report results
aggregated by test
Change-Id: Id8d15188846b3bff79ac4c65a09e5d6740acde73
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/93300
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Ben Clayton <bclayton@google.com>
Commit-Queue: Austin Eng <enga@chromium.org>
For commands like `cts update`, `cts export`:
If a build completed with `StatusInfraFailure` or `StatusCanceled` and we're trying to obtain the results, re-kick the build.
Bug: dawn:1342
Change-Id: I84565b06eb70e92f3b75a0ab2befd7ed83315140
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/92940
Auto-Submit: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: Austin Eng <enga@chromium.org>
The CTS runner can randomly pick between these two GPUs, leading to unstable expectations.
Just use 'intel' for now. We might get a more targetted tag in the future, but this will do for today.
Bug: dawn:1387
Bug: dawn:1444
Change-Id: I2c77ab0060976c40e34b1597b5594894867b7edd
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/92242
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Auto-Submit: Ben Clayton <bclayton@google.com>
Instead of Failure.
If we have duplicate results (same query, same tags), but with different results, then we have to merge them. Previously we'd merge Pass and Slow as Failure, which was just odd. Fix this so the combined result is Slow.
Bug: dawn:1342
Change-Id: If9b035bb3c1e59b7a93a5c8b194f854405e656d6
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/89960
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Auto-Submit: Ben Clayton <bclayton@google.com>
• Add the included trybots in the CL description. All of these trybots are tested by the roll, but the final CQ-submit wouldn't necessarily test all of the variants before landing. This would mean that the 'cts export' could miss some results, as it takes the last PS with any results.
• Add --force flag to cts roll to force a roll. Useful for testing.
• Emit timing diagnostics for tests labelled 'Slow' instead of unhelpfully stating they pass.
• Enable the --cl and --ps flags for cts export
• Export with the most recent data to the top of the spreadsheet
Bug: dawn:1401
Change-Id: Id926367ab805bfb9f3032fce9cce7f00daf7a5d4
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88661
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
• Strip 'release-x64' tags.
These are always paired with 'release', and all
other tests just have the 'release' tag. This
means that results with the 'release' +
'release-x64' tags are a subset of just the
'release' tags, leading to many broken
assumptions in the expectation update.
• Don't consider tests with the error reason
'asyncio.exceptions.TimeoutError' as slow, but
as a failure. The CTS expectation updater will
already mark tests exceeding the config
threshold as slow. This reduces the amount of
data requested from resultdb, and prevents
failed rolls when results are actually
deadlocking.
• Fix inferring of patchset in GetResults().
LatestPatchest() was called but was assigned to
a nested scope.
Bug: dawn:1401
Change-Id: I57b2c9a029b31430d63a056f7b13c4bf1bc5b437
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88450
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Updates a Google spreadsheet with latest CTS statistics.
Bug: dawn:1342
Change-Id: I01a9c99032c755d6a78af674675d8c9cbfc79019
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88443
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Add the common utilities for starting and fetching build results.
'cts time' provides views on the longest running tests, along with
a histogram view of all test times.
Bug: dawn:1342
Change-Id: Ia3707f7f062ea26a2406e3163a26e1cc7e30e3b2
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88319
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
This CL introduces the Config structure, and config file.
Used by all the sub-commands.
Bug: dawn:1342
Change-Id: Iaad5a02bc2ceba2c63265cf0d1086dc7901e7ed7
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88317
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Antonio Maiorano <amaiorano@google.com>
