15 Commits

Author SHA1 Message Date
James Price
facb5ced00 tools: Add --filename-column-width to test runner
If provided, this option truncates long filenames to avoid unpleasant
line wrapping behavior for those of us that use narrow terminals.

Change-Id: I684b91b7aa4b0ae86a4cf9ed9f047b685ab97550
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/57100
Auto-Submit: James Price <jrprice@google.com>
Reviewed-by: Ben Clayton <bclayton@google.com>
Commit-Queue: James Price <jrprice@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
2021-07-06 18:31:15 +00:00
Ben Clayton
c2da12bb20 tools: Add --fxc option to test-runner
And add a reasonable timeout to tests.

Change-Id: I362d44f2a799562d236a80f5c35d50d93e8d094a
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/56769
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: David Neto <dneto@google.com>
2021-07-02 21:13:24 +00:00
Ben Clayton
288c8ad32e test-runner: Fix printing on windows
Colors were not being escaped properly for windows (which has special non-ANSI color code handling)

Change-Id: I03c34b1205b26b1192af9c05162bbe03035172bd
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/56500
Auto-Submit: Ben Clayton <bclayton@google.com>
Reviewed-by: Antonio Maiorano <amaiorano@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
2021-06-29 20:55:16 +00:00
Ben Clayton
230295dea4 test-runner: Don't print PASS rows by default
If a full test row PASSes, then omit this unless the --verbose flag is used.
Add a summary below the table for PASS, SKIP and FAIL for each column.

Change-Id: Ie88003696c59614cf8e5efa3c80dca37a8e4d3f0
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/56380
Auto-Submit: Ben Clayton <bclayton@google.com>
Commit-Queue: Antonio Maiorano <amaiorano@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Antonio Maiorano <amaiorano@google.com>
2021-06-29 13:10:25 +00:00
Ben Clayton
c2c46a0533 test-runner: Consider wgsl output as validated
The resolver is the closest thing to a validator in this case.
Enables `--generate-expected` for WGSL files that were previously marked as SKIP.

Change-Id: Id7979ec39571ff52a2beb4255d88e4eb0962b82a
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/54650
Auto-Submit: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: James Price <jrprice@google.com>
Reviewed-by: James Price <jrprice@google.com>
2021-06-17 12:53:54 +00:00
Ben Clayton
1987fd80f4 test-runner: Print results as soon as they're available
Gives feedback much eariler.

Also move the error messages after the table

Change-Id: If462fd9bbc72ff9428d6ee77c8ec6c9338f7f27e
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/54000
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Auto-Submit: Ben Clayton <bclayton@google.com>
Reviewed-by: James Price <jrprice@google.com>
2021-06-09 19:14:09 +00:00
Antonio Maiorano
089c9e34a5 test-runner: on Windows, use forward slash separators in paths passed to tint
This ensures that we don't fail when comparing against expected output
that contains paths in them.

Bug: tint:872
Change-Id: I93d4b940eae2c22175ef3add94bb28f907156ca0
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/53600
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Antonio Maiorano <amaiorano@google.com>
Reviewed-by: Ben Clayton <bclayton@google.com>
2021-06-07 19:43:04 +00:00
Antonio Maiorano
6a665d1b85 test-runner: allow using '/' in filter on Windows, and fix using '\'
Change-Id: I635952a6c04bc305a896143333b2ae141bb6a2b0
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/53400
Kokoro: Kokoro <noreply+kokoro@google.com>
Reviewed-by: Ben Clayton <bclayton@google.com>
Commit-Queue: Antonio Maiorano <amaiorano@google.com>
2021-06-07 14:51:22 +00:00
Ben Clayton
3db1820f0b test-runner: Don't overwrite SKIPs with --generate-expected
Unless the test also passed validation.
Prevents clobbering expected validation failures when you've only checked tint didn't error.

Change-Id: I44be0c8b1f9ef1a0df51bba81f5ad694715204b8
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/53383
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: James Price <jrprice@google.com>
Reviewed-by: Antonio Maiorano <amaiorano@google.com>
2021-06-04 21:58:07 +00:00
Ben Clayton
b175d91c7e test-runner: Pass a relative path to tint
Avoids absolute paths being emitted in diagnostics

Change-Id: Ic62af27e079ceac8d89735486edcaca75014b8c9
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/53382
Auto-Submit: Ben Clayton <bclayton@google.com>
Reviewed-by: James Price <jrprice@google.com>
Commit-Queue: Ben Clayton <bclayton@chromium.org>
Kokoro: Kokoro <noreply+kokoro@google.com>
2021-06-04 20:02:45 +00:00
Ben Clayton
2270b434bc [tools] test-runner: Add --generate-skip flag
Generates expected cases that fail, using a SKIP header

Change-Id: I2738fa2cf47c715fe6e978edf51aa928c5e3b216
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/53049
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Ben Clayton <bclayton@chromium.org>
Reviewed-by: James Price <jrprice@google.com>
2021-06-03 10:47:24 +00:00
Ben Clayton
a70c14dbce main: Replace --dawn-validation with --validate
Performs output validation with spirv-val for SPIR-V (as before), HLSL
validation with DXC, and MSL validation with the Metal Shader Compiler.

Disable HLSL tests that fail to validate

Bug: tint:812
Change-Id: If78c351b4e23c7fb50d333eacf9ee7cc81d18564
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/51280
Reviewed-by: Antonio Maiorano <amaiorano@google.com>
Reviewed-by: Ben Clayton <bclayton@google.com>
Kokoro: Kokoro <noreply+kokoro@google.com>
Commit-Queue: Antonio Maiorano <amaiorano@google.com>
2021-06-02 21:02:34 +00:00
Ben Clayton
a53a3b8c19 test-runner: Add option to limit number of threads
Fixed: tint:830
Change-Id: I46696d0f72760549743f95a35ea21fef96269146
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/51924
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: David Neto <dneto@google.com>
2021-05-21 18:02:33 +00:00
Ben Clayton
c0587b6f2d test-runner: Return non-zero code on failure
Kokoro will use this to fail presubmits on new failures.

Change-Id: I32cf49d0380c7b751ae4410ccd91b20407262e9f
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/51482
Commit-Queue: Ben Clayton <bclayton@chromium.org>
Reviewed-by: James Price <jrprice@google.com>
2021-05-19 17:06:54 +00:00
Ben Clayton
57b2a06ba7 tools/test-all.sh: Reimplement in golang
Makes future development easier.

New features:
* A more compact and cleaner results view
* Concurrent testing, much quicker across multiple cores
* Supports comparing output against an expected file, including a text diff of differences. Also has a flag for updating the expected outputs
* Advanced file-globbing support, including scanning for files in subdirectories
* Skip lists are now no longer hidden away in the tool, but defined as a SKIP header in the *.expected.* file

Change-Id: I4fac80bb084a720ec9a307b4acf9f73792973a1d
Reviewed-on: https://dawn-review.googlesource.com/c/tint/+/50903
Commit-Queue: Ben Clayton <bclayton@google.com>
Reviewed-by: David Neto <dneto@google.com>
2021-05-14 19:48:43 +00:00