appendConsumedResultsForSkippedTests() adds fake results for expectations that are 'Skip'ed, so that these sub-trees aren't collapsed as all passing.
However, this code was not handling the fact that there might actually be results for the cases.
This happened because there was an expectation collision in the expectations.txt file, and tests were not being skipped even though there was an expectation with a Skip (a collision expectation was used instead).
Change-Id: I2a1543d231db44dc8aa6683d051f884f4cb96853
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/113520
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Ben Clayton <bclayton@google.com>
Reviewed-by: Austin Eng <enga@chromium.org>
`-o` will emit the files to the given root output directory
`--verbose` will print what's going on, to help with debugging
Omitting these flags will behave as before.
Also consolidate the utils package into fileutils. These were two packages with near identical functionality.
Change-Id: I855dd4b57807fb9239a52e7f357842d4ba2517ee
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/107687
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
This CL updates the intrinsics lexer to allow negative values for int
and float numerics. This allows doing `@test_value(-2)` in the def file.
Change-Id: I2cad9b25a2932057ce9bc51dec6c32231e06f0a0
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/107440
Commit-Queue: Dan Sinclair <dsinclair@chromium.org>
Reviewed-by: Ben Clayton <bclayton@google.com>
Auto-Submit: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
• Add `cts validate` command used to check for expectation collisions.
Can be used as a presubmit check.
This is more tightly checked than the previous logic, as this works on just
the expectations, instead of results.
• Fix an issue where the test result reduction could introduce collisions with
'Skip' expectations.
To fix this, the update process first adds 'consumed' results for the skipped
tests, preventing test tree reduction for that part of the tree.
• Fix a bug in the generation of 'New failures' and 'New flakes' which produced
more expectations than was necessary.
The issue here was that the tree roots could contain overlaps, and roots could
be processed before sub-trees, resulting in inefficient expectations.
• Fix collisions in the expectations file, and update with results from
the most recent roll.
Change-Id: I7b64553408998fb4416458ce564fc49c8f6d4d07
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101860
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Austin Eng <enga@chromium.org>
may_exonerate indicates that a test failed for a known issue that
we could exonerate. Merging of test results now removes results
with may_exonerate unless all of them were tagged as such. So, if
for example, a test fails for a known timeout issue, but has a
subsequent pass, the timeout will be ignored.
This serves to reduce the impact of known, hard-to-fix issues and
allow the CTS roller to make progress with less noise.
Change-Id: I5103a666496398a17b3aa6ccf3f267421e40ba97
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/101804
Reviewed-by: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Austin Eng <enga@chromium.org>
- Run format on the current expectations file to make it up to date.
- It's a lot easier to read the failures when they are grouped by the bug.
Change-Id: I333a0b237ccd405d204e47809eb7ab723824fddd
No-Try: True
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/98241
Auto-Submit: Loko Kung <lokokung@google.com>
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Commit-Queue: Loko Kung <lokokung@google.com>
• Add the included trybots in the CL description. All of these trybots are tested by the roll, but the final CQ-submit wouldn't necessarily test all of the variants before landing. This would mean that the 'cts export' could miss some results, as it takes the last PS with any results.
• Add --force flag to cts roll to force a roll. Useful for testing.
• Emit timing diagnostics for tests labelled 'Slow' instead of unhelpfully stating they pass.
• Enable the --cl and --ps flags for cts export
• Export with the most recent data to the top of the spreadsheet
Bug: dawn:1401
Change-Id: Id926367ab805bfb9f3032fce9cce7f00daf7a5d4
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/88661
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Implement the logic to update an existing expectations file with new
results.
Bug: dawn:1342
Change-Id: Idcbad57946712539cc5f0d238f89a2adf42f5aa0
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/87702
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
MinimalVariantTags accepts a list of tag-sets (e.g GPU tags, OS tags, etc),
and returns an optimized list of variants, folding together variants that
have identical result query-to-status mappings, and removing redundant tags.
Bug: dawn:1342
Change-Id: I759c82e9a0631a9d321d376656e5a2dbbf5f5507
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/87643
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Implement an expectation parser, data structures and writer.
Bug: dawn:1342
Change-Id: I53587a9b55346ccf1543e15c9cec5ff68c6849ad
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/87641
Reviewed-by: Dan Sinclair <dsinclair@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Add result.List.StatusTree() for building a query.Tree[Status].
Add helpers for serializing results.
Add helpers for merging and de-duplicating results.
Change the interface of result.List.ReplaceDuplicates() so that the
merging function takes a status set instead of a list of results.
Bug: dawn:1342
Change-Id: I77580ec5fd4c8f12109fb6e9e83afea8b740260c
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/87240
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
A tree of query to data. Has utilities for reducing the tree based on a
custom merger function.
Bug: dawn:1342
Change-Id: If1c0503be05ee04bcf55dd5bdc9aa3caf6fb56ee
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/87222
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Provides a type to represent the CTS query strings.
100% test coverage.
Bug: dawn:1342
Change-Id: I3769b094ba64221a7b79dd38f76daf0125ee9e28
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/85221
Reviewed-by: Austin Eng <enga@chromium.org>
Commit-Queue: Ben Clayton <bclayton@google.com>
Ben Clayton
dan sinclair
Austin Eng
Loko Kung