3 Commits

Author SHA1 Message Date
Ben Clayton
aad2e9c0b5 tools/cts: Add cts validate, improvements & fixes
• Add `cts validate` command used to check for expectation collisions.
  Can be used as a presubmit check.
  This is more tightly checked than the previous logic, as this works on just
  the expectations, instead of results.

• Fix an issue where the test result reduction could introduce collisions with
  'Skip' expectations.
  To fix this, the update process first adds 'consumed' results for the skipped
  tests, preventing test tree reduction for that part of the tree.

• Fix a bug in the generation of 'New failures' and 'New flakes' which produced
  more expectations than was necessary.
  The issue here was that the tree roots could contain overlaps, and roots could
  be processed before sub-trees, resulting in inefficient expectations.

• Fix collisions in the expectations file, and update with results from
  the most recent roll.

Change-Id: I7b64553408998fb4416458ce564fc49c8f6d4d07
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101860
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Austin Eng <enga@chromium.org>
2022-09-17 19:30:29 +00:00
Austin Eng
a48e46f3a4 CTS: Handle may_exonerate tag
may_exonerate indicates that a test failed for a known issue that
we could exonerate. Merging of test results now removes results
with may_exonerate unless all of them were tagged as such. So, if
for example, a test fails for a known timeout issue, but has a
subsequent pass, the timeout will be ignored.

This serves to reduce the impact of known, hard-to-fix issues and
allow the CTS roller to make progress with less noise.

Change-Id: I5103a666496398a17b3aa6ccf3f267421e40ba97
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101804
Reviewed-by: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Austin Eng <enga@chromium.org>
2022-09-14 14:29:07 +00:00
Ben Clayton
665c07ac6a tools: Add the 'cts update' sub-command
Updates the expectations file with new results.

Bug: dawn:1342
Change-Id: I05cab48eb1cf0f9ab7e652a1b7c3da361c45b0b1
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88442
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
2022-04-29 19:09:17 +00:00