Updates for decomp.dev (replaces frogress)

Resolves #52

.github.example/workflows/build.yml

@@ -37,23 +37,7 @@ jobs:
       run: |
         python configure.py --map --version ${{ matrix.version }} \
             --binutils /binutils --compilers /compilers
-        ninja all_source build/${{ matrix.version }}/progress.json \
-            build/${{ matrix.version }}/report.json
-
-    # Upload progress if we're on the main branch
-    - name: Upload progress
-      # If you're using a different branch, change this to match
-      if: github.ref == 'refs/heads/main'
-      continue-on-error: true
-      env:
-        # Replace with your project slug
-        PROGRESS_SLUG: prime
-        # Set the API key in your repository secrets
-        PROGRESS_API_KEY: ${{ secrets.PROGRESS_API_KEY }}
-      run: |
-        python tools/upload_progress.py -b https://progress.decomp.club/ \
-          -p $PROGRESS_SLUG -v ${{ matrix.version }} \
-          build/${{ matrix.version }}/progress.json
+        ninja all_source progress build/${{ matrix.version }}/report.json
 
     # Upload map files
     - name: Upload map

README.example.md

@@ -1,5 +1,5 @@
 Some Game  
-[![Build Status]][actions] ![Progress] ![DOL Progress] ![RELs Progress] [![Discord Badge]][discord]
+[![Build Status]][actions] [![Code Progress]][progress] [![Data Progress]][progress] [![Discord Badge]][discord]
 =============
 
 <!--
@@ -7,24 +7,13 @@ Replace with your repository's URL.
 -->
 [Build Status]: https://github.com/zeldaret/tww/actions/workflows/build.yml/badge.svg
 [actions]: https://github.com/zeldaret/tww/actions/workflows/build.yml
-<!---
-Code progress URL:
-https://progress.decomp.club/data/[project]/[version]/all/?mode=shield&measure=code
-URL encoded then appended to: https://img.shields.io/endpoint?label=Code&url=
--->
-[Progress]: https://img.shields.io/endpoint?label=Code&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fall%2F%3Fmode%3Dshield%26measure%3Dcode
-<!---
-DOL progress URL:
-https://progress.decomp.club/data/[project]/[version]/dol/?mode=shield&measure=code
-URL encoded then appended to: https://img.shields.io/endpoint?label=DOL&url=
--->
-[DOL Progress]: https://img.shields.io/endpoint?label=DOL&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fdol%2F%3Fmode%3Dshield%26measure%3Dcode
 <!--
-REL progress URL:
-https://progress.decomp.club/data/[project]/[version]/modules/?mode=shield&measure=code
-URL encoded then appended to: https://img.shields.io/endpoint?label=RELs&url=
+decomp.dev progress badges
+See https://decomp.dev/api for an API overview.
 -->
-[RELs Progress]: https://img.shields.io/endpoint?label=RELs&url=https%3A%2F%2Fprogress.decomp.club%2Fdata%2Ftww%2FGZLE01%2Fmodules%2F%3Fmode%3Dshield%26measure%3Dcode
+[Code Progress]: https://decomp.dev/zeldaret/tww.svg?mode=shield&measure=code&label=Code
+[Data Progress]: https://decomp.dev/zeldaret/tww.svg?mode=shield&measure=data&label=Data
+[progress]: https://decomp.dev/zeldaret/tww
 <!--
 Replace with your Discord server's ID and invite URL.
 -->

README.md

@@ -25,27 +25,12 @@ References
 - [Discord: GC/Wii Decompilation](https://discord.gg/hKx3FJJgrV) (Come to `#dtk` for help!)
 - [objdiff](https://github.com/encounter/objdiff) (Local diffing tool)
 - [decomp.me](https://decomp.me) (Collaborate on matches)
-- [frogress](https://github.com/decompals/frogress) (Decompilation progress API)
+- [decomp.dev](https://decomp.dev) (Decompilation progress hub and API)
 - [wibo](https://github.com/decompals/wibo) (Minimal Win32 wrapper for Linux)
 - [sjiswrap](https://github.com/encounter/sjiswrap) (UTF-8 to Shift JIS wrapper)
 
-Projects using this structure:
-
-- [zeldaret/tww](https://github.com/zeldaret/tww)
-- [zeldaret/oot-gc](https://github.com/zeldaret/oot-gc)
-- [zeldaret/ss](https://github.com/zeldaret/ss)
-- [PrimeDecomp/prime](https://github.com/PrimeDecomp/prime)
-- [PrimeDecomp/echoes](https://github.com/PrimeDecomp/echoes)
-- [DarkRTA/rb3](https://github.com/DarkRTA/rb3)
-- [doldecomp/melee](https://github.com/doldecomp/melee)
-- [doldecomp/sadx](https://github.com/doldecomp/sadx)
-- [InputEvelution/wp](https://github.com/InputEvelution/wp)
-- [NWPlayer123/AnimalCrossing-dtk](https://github.com/NWPlayer123/AnimalCrossing-dtk)
-- [Rainchus/marioparty4](https://github.com/Rainchus/marioparty4)
-- [Rainchus/ttyd_dtk](https://github.com/Rainchus/ttyd_dtk)
-- [Sage-of-Mirrors/zmansion](https://github.com/Sage-of-Mirrors/zmansion)
-- [bfbbdecomp/bfbb](https://github.com/bfbbdecomp/bfbb)
-- [EstexNT/rhf-dtk](https://github.com/EstexNT/rhf-dtk)
+Nearly all active GC/Wii decompilation projects use this structure, and will be useful
+for reference. A list of active GC/Wii projects can be found on [decomp.dev](https://decomp.dev).
 
 Features
 --------
@@ -56,7 +41,7 @@ Features
 - Feature-rich analyzer: Many time-consuming tasks are automated, allowing you to focus on the decompilation itself. See [Analyzer features](https://github.com/encounter/decomp-toolkit#analyzer-features).
 - REL support: RELs each have their own `symbols.txt` and `splits.txt`, and will automatically be built and linked against the main binary.
 - No manual assembly: decomp-toolkit handles splitting the DOL into relocatable objects based on the configuration. No game assets are committed to the repository.
-- Progress calculation and upload script for [frogress](https://github.com/decompals/frogress).
+- Progress calculation and integration with [decomp.dev](https://decomp.dev).
 - Integration with [objdiff](https://github.com/encounter/objdiff) for a diffing workflow.
 - CI workflow template for GitHub Actions.
 

configure.py

@@ -158,7 +158,6 @@ config.asflags = [
     "-I include",
     f"-I build/{config.version}/include",
     f"--defsym BUILD_VERSION={version_num}",
-    f"--defsym VERSION_{config.version}",
 ]
 config.ldflags = [
     "-fp hardware",
@@ -307,7 +306,7 @@ if args.mode == "configure":
     # Write build.ninja and objdiff.json
     generate_build(config)
 elif args.mode == "progress":
-    # Print progress and write progress.json
+    # Print progress information
     calculate_progress(config)
 else:
     sys.exit("Unknown mode: " + args.mode)

docs/github_actions.md

@@ -3,8 +3,8 @@
 This repository includes [.github.example/workflows/build.yml](/.github.example/workflows/build.yml) as an example CI workflow. To use it for your project, follow the setup instructions below.
 
 - [Build Repository](#build-repository)
-- [Progress](#progress)
 - [Workflow](#workflow)
+- [decomp.dev](#decompdev)
 
 ## Build Repository
 
@@ -25,18 +25,6 @@ This repository will be used to build and store the CI build container.
 4. Under "Manage Actions access", add your project's main repository with the "Read" role:  
     ![GitHub package Actions access](images/github_package_settings_access.png)
 
-## Progress
-
-1. In the [GC/Wii Decompilation Discord](https://discord.gg/hKx3FJJgrV), visit `#frogress` and request an API key for your project.  
-    Please provide the following:
-    - Project name
-    - Repository URL
-    - Game ID(s)
-    - Whether the game has RELs
-
-2. On GitHub, visit your repo's `/settings/secrets/actions/new` and add a new secret with the name `PROGRESS_API_KEY`:  
-    ![GitHub Actions secrets](images/github_actions_secrets.png)
-
 ## Workflow
 
 1. Rename `.github.example` to `.github`.
@@ -45,8 +33,14 @@ This repository will be used to build and store the CI build container.
 
 3. In `build.yml`, replace `GAMEID` with your game's ID. (Or list of IDs, for multi-version support.)
 
-4. In `build.yml`, update `PROGRESS_SLUG` to match the project name on [frogress](#progress).
+4. Commit and push the changes to your repository.
 
-5. Commit and push the changes to your repository.
+If everything is set up correctly, the workflow will build all versions on every push or pull request.
 
-If everything is set up correctly, the workflow will build all versions on every push or pull request, and upload progress on pushes the `main` branch.
+## decomp.dev
+
+Once the build workflow is running on the main branch, you can add your game to <https://decomp.dev>.
+
+Visit <https://decomp.dev/manage/new>, select your GitHub repository and fill out the required fields.
+
+If you have questions or issues, try asking in the [GC/Wii Decompilation Discord](https://discord.gg/hKx3FJJgrV) #decomp.dev channel.

tools/project.py

@@ -196,9 +196,8 @@ class ProjectConfig:
             None  # Callback to add/remove/reorder units within a module
         )
 
-        # Progress output, progress.json and report.json config
+        # Progress output and report.json config
         self.progress = True  # Enable report.json generation and CLI progress output
-        self.progress_all: bool = True  # Include combined "all" category
         self.progress_modules: bool = True  # Include combined "modules" category
         self.progress_each_module: bool = (
             False  # Include individual modules, disable for large numbers of modules
@@ -431,7 +430,6 @@ def generate_build_ninja(
     n.comment("Tooling")
 
     build_path = config.out_path()
-    progress_path = build_path / "progress.json"
     report_path = build_path / "report.json"
     build_tools_path = config.build_dir / "tools"
     download_tool = config.tools_dir / "download_tool.py"
@@ -1188,7 +1186,7 @@ def generate_build_ninja(
             description="PROGRESS",
         )
         n.build(
-            outputs=progress_path,
+            outputs="progress",
             rule="progress",
             implicit=[
                 ok_path,
@@ -1385,7 +1383,7 @@ def generate_build_ninja(
         if config.non_matching:
             n.default(link_outputs)
         elif config.progress:
-            n.default(progress_path)
+            n.default("progress")
         else:
             n.default(ok_path)
     else:
@@ -1825,7 +1823,7 @@ def generate_compile_commands(
         json.dump(clangd_config, w, indent=2, default=default_format)
 
 
-# Calculate, print and write progress to progress.json
+# Print progress information from objdiff report
 def calculate_progress(config: ProjectConfig) -> None:
     config.validate()
     out_path = config.out_path()
@@ -1917,35 +1915,3 @@ def calculate_progress(config: ProjectConfig) -> None:
     if summary_file:
         summary_file.write("```\n")
         summary_file.close()
-
-    # Generate and write progress.json
-    progress_json: Dict[str, Any] = {}
-
-    def add_category(id: str, measures: Dict[str, Any]) -> None:
-        progress_json[id] = {
-            "code": measures.get("complete_code", 0),
-            "code/total": measures.get("total_code", 0),
-            "data": measures.get("complete_data", 0),
-            "data/total": measures.get("total_data", 0),
-            "matched_code": measures.get("matched_code", 0),
-            "matched_code/total": measures.get("total_code", 0),
-            "matched_data": measures.get("matched_data", 0),
-            "matched_data/total": measures.get("total_data", 0),
-            "matched_functions": measures.get("matched_functions", 0),
-            "matched_functions/total": measures.get("total_functions", 0),
-            "fuzzy_match": int(measures.get("fuzzy_match_percent", 0) * 100),
-            "fuzzy_match/total": 10000,
-            "units": measures.get("complete_units", 0),
-            "units/total": measures.get("total_units", 0),
-        }
-
-    if config.progress_all:
-        add_category("all", report_data["measures"])
-    else:
-        # Support for old behavior where "dol" was the main category
-        add_category("dol", report_data["measures"])
-    for category in report_data.get("categories", []):
-        add_category(category["id"], category["measures"])
-
-    with open(out_path / "progress.json", "w", encoding="utf-8") as w:
-        json.dump(progress_json, w, indent=2)

tools/upload_progress.py

@@ -1,79 +0,0 @@
-#!/usr/bin/env python3
-
-###
-# Uploads progress information to https://github.com/decompals/frogress.
-#
-# Usage:
-#   python3 tools/upload_progress.py -b https://progress.decomp.club/ -p [project] -v [version] build/[version]/progress.json
-#
-# If changes are made, please submit a PR to
-# https://github.com/encounter/dtk-template
-###
-
-import argparse
-import json
-import os
-import requests
-import subprocess
-import sys
-
-
-def get_git_commit_timestamp() -> int:
-    return int(
-        subprocess.check_output(["git", "show", "-s", "--format=%ct"])
-        .decode("ascii")
-        .rstrip()
-    )
-
-
-def get_git_commit_sha() -> str:
-    return subprocess.check_output(["git", "rev-parse", "HEAD"]).decode("ascii").strip()
-
-
-def generate_url(args: argparse.Namespace) -> str:
-    url_components = [args.base_url.rstrip("/"), "data"]
-
-    for arg in [args.project, args.version]:
-        if arg != "":
-            url_components.append(arg)
-
-    return str.join("/", url_components) + "/"
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Upload progress information.")
-    parser.add_argument("-b", "--base_url", help="API base URL", required=True)
-    parser.add_argument("-a", "--api_key", help="API key (env var PROGRESS_API_KEY)")
-    parser.add_argument("-p", "--project", help="Project slug", required=True)
-    parser.add_argument("-v", "--version", help="Version slug", required=True)
-    parser.add_argument("input", help="Progress JSON input")
-
-    args = parser.parse_args()
-    api_key = args.api_key or os.environ.get("PROGRESS_API_KEY")
-    if not api_key:
-        raise KeyError("API key required")
-    url = generate_url(args)
-
-    entries = []
-    with open(args.input, "r") as f:
-        data = json.load(f)
-        entries.append(
-            {
-                "timestamp": get_git_commit_timestamp(),
-                "git_hash": get_git_commit_sha(),
-                "categories": data,
-            }
-        )
-
-    print("Publishing entry to", url)
-    json.dump(entries[0], sys.stdout, indent=4)
-    print()
-    r = requests.post(
-        url,
-        json={
-            "api_key": api_key,
-            "entries": entries,
-        },
-    )
-    r.raise_for_status()
-    print("Done!")