decompals
/
frogress
mirror of https://github.com/decompals/frogress.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

0.2.0: Fixes and improvements (#33) 

Fixes #27, #29, #13
This commit is contained in:
Ethan Roseman 2023-05-25 18:50:09 +09:00 committed by GitHub
parent b45cb756c9
commit 19e625ad27
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
12 changed files with 771 additions and 701 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
.github/workflows/ci.yml vendored
View File

@ -5,14 +5,14 @@ on:
- main
pull_request:
jobs:
full_test_and_build:
name: unit tests
tests:
name: tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.9
python-version: '3.10'
- uses: snok/install-poetry@v1
- run: poetry install
- name: Run backend tests
@ -22,10 +22,10 @@ jobs:
name: mypy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.9
python-version: '3.10'
- uses: snok/install-poetry@v1
- run: |-
poetry install
@ -34,7 +34,11 @@ jobs:
name: black
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: psf/black@stable
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
src: "."
python-version: '3.10'
- uses: snok/install-poetry@v1
- run: |-
poetry install
poetry run black . --check

2
CONTRIBUTING.md
View File

@ -1,7 +1,7 @@
# Contributing
## Dependencies
- Python >=3.9
- Python >=3.10
- [Poetry](https://python-poetry.org/docs/master/#installing-with-the-official-installer)
## Setup

68
GUIDE.md
View File

@ -1,8 +1,8 @@
# Guide on how to monitor decomp progress using Frogress
# Guide on how to monitor decomp progress using frogress
## Overview
It will guide you to onboard your decomp project to Frogress.
This guide will provide a flow for how to use frogress for your project.
```mermaid
sequenceDiagram
@ -12,11 +12,11 @@ sequenceDiagram
Contributor->>+CI: Trigger
CI->>CI: Build
CI->>CI: Calculate progress
CI->>-Frogress: Upload progress
Note over CI,Frogress: POST /data/project:/version:/
Monitor->>Frogress: Fetch progress
Note over Monitor,Frogress: GET /data/project:/version:/?mode=all
Frogress->>Monitor: Return progress
CI->>-frogress: Upload progress
Note over CI,frogress: POST /data/project:/version:/
Monitor->>frogress: Fetch progress
Note over Monitor,frogress: GET /data/project:/version:/?mode=all
frogress->>Monitor: Return progress
Monitor->>Monitor: Render progress
```
@ -24,7 +24,7 @@ sequenceDiagram
## Steps
1. Contact Frogress admin to create project and assign api_key
1. Contact frogress admin (Ethan) to add your project to the database and obtain an api_key
2. Create schema with `cli.py`
@ -45,9 +45,7 @@ sequenceDiagram
./cli.py create version fireemblem8 us
```
2.3 Create category (optional)
Default category: `default`
2.3 Create category
```bash
# Usage
@ -56,14 +54,14 @@ sequenceDiagram
./cli.py create category fireemblem8 us default
```
3. Upload progress in CI
3. Configure CI to upload data on build
3.1 API
```
POST https://progress.deco.mp/data/project:/version:/
```
```python
{
"api_key": "",
@ -80,39 +78,47 @@ sequenceDiagram
```
3.2 Example
https://github.com/FireEmblemUniverse/fireemblem8u/pull/307
4. Supplement historical data (optional)
4. Supplement historical data (optional, one-time)
Calculate progress for historical commits and upload it to Frogress if you would like to draw historical curve.
Calculate progress for historical commits and upload it to frogress for the purpose of visualizing and tracking historical data
[Example](https://github.com/laqieer/fireemblem8u/blob/master/.github/workflows/supplement-progress.yml)
5. Prune duplicated data (optional)
5. Fetch project data
Background: https://github.com/decompals/frogress/issues/27
```
# Usage
./cli.py prune -h
# Example
./cli.py prune fireemblem8 us
```
5.1 API
6. Fetch project data
6.1 API
There are 3 "modes" for returning results: `all`, `latest`, and `shield`. The first two are pretty self explanatory, and `shield` can be used to generate badges for your repo's README.md (shields.io).
```
GET https://progress.deco.mp/data/project:/version:/?mode=all
```
```
GET https://progress.deco.mp/data/project:/version:/?mode=latest
```
Note that the category and measure are required for mode `shield`.
```
GET https://progress.deco.mp/data/project:/version:/category:/?mode=shield&measure=MEASURE
```
https://progress.deco.mp/data/fireemblem8/us/?mode=all
**Note:** Specifying a category in the request URL is optional for `all` and `latest` modes. If the category is not specified, results from all categories will be returned. However, for `shield`, it is necessary to specify the category and measure in the request url, as shown above.
6.2 Build a website
`all` example:
Build a website to render progress graph using a library such as [uPlot](https://github.com/leeoniya/uPlot) and [Chart.js](https://www.chartjs.org).
`https://progress.deco.mp/data/fireemblem8/us/?mode=all`
`shield` example:
`https://progress.deco.mp/data/dukezh/us/default/?mode=shield&measure=bytes`
5.2 Build a website
Build a website to display your progress!
You can use the "latest" mode to retrieve just the latest datapoint or render full graphs using a library such as [uPlot](https://github.com/leeoniya/uPlot) or [Chart.js](https://www.chartjs.org)
- https://pikmin.dev
- https://axiodl.com

98
cli.py
View File

@ -83,93 +83,6 @@ def delete_category(args: argparse.Namespace) -> None:
print(response.text)
def prune_entries(args: argparse.Namespace) -> None:
# Get entries
url = f"{domain}/data/{args.project}/{args.version}?mode=all"
debug("GET " + url)
response = requests.get(url)
print(response.text)
categories = response.json().get(args.project, {}).get(args.version, {})
if len(categories) == 0:
return
# Filter entries
filtered = {}
for category, entries in categories.items():
if len(entries) == 0:
continue
for entry in entries:
if entry["git_hash"] not in filtered:
filtered[entry["git_hash"]] = {
"git_hash": entry["git_hash"],
"timestamp": entry["timestamp"],
"categories": {},
}
if category not in filtered[entry["git_hash"]]["categories"]:
filtered[entry["git_hash"]]["categories"][category] = entry["measures"]
else:
for measure, value in entry["measures"].items():
if (
measure
not in filtered[entry["git_hash"]]["categories"][category]
):
filtered[entry["git_hash"]]["categories"][category][
measure
] = value
entries = list(filtered.values())
# Clear entries
for category in categories.keys():
# Delete categories
url = f"{domain}/projects/{args.project}/{args.version}/{category}/"
data = {"api_key": api_key}
debug("DELETE " + url)
response = requests.delete(url, json=data)
print(response.text)
# Recreate categories
data["name"] = category
debug("POST " + url)
response = requests.post(url, json=data)
print(response.text)
# Upload entries
url = f"{domain}/data/{args.project}/{args.version}/"
data = {"api_key": api_key, "entries": entries}
debug("POST " + url)
response = requests.post(url, json=data)
print(response.status_code, response.text)
# Check entries
url = f"{domain}/data/{args.project}/{args.version}?mode=all"
debug("GET " + url)
response = requests.get(url)
print(response.text)
def main() -> None:
parser = argparse.ArgumentParser()
@ -235,17 +148,6 @@ def main() -> None:
delete_category_parser.add_argument("slug", help="the slug for the category")
delete_category_parser.set_defaults(func=delete_category)
# Prune entries
prune_parser = subparsers.add_parser(
"prune",
help="prune entries to remove duplicates",
)
prune_parser.add_argument(
"project", help="the project for which to prune duplicated entries"
)
prune_parser.add_argument("version", help="the slug for the version")
prune_parser.set_defaults(func=prune_entries)
args = parser.parse_args()
args.func(args)

45
frog_api/migrations/0010_alter_entry_category_alter_project_discord_and_more.py Normal file
View File

@ -0,0 +1,45 @@
# Generated by Django 4.2.1 on 2023-05-25 08:56
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
(
"frog_api",
"0001_squashed_0009_alter_project_discord_alter_project_repository_and_more",
),
]
operations = [
migrations.AlterField(
model_name="entry",
name="category",
field=models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE, to="frog_api.category"
),
),
migrations.AlterField(
model_name="project",
name="discord",
field=models.URLField(blank=True),
),
migrations.AlterField(
model_name="project",
name="repository",
field=models.URLField(blank=True),
),
migrations.AlterField(
model_name="project",
name="website",
field=models.URLField(blank=True),
),
migrations.AddConstraint(
model_name="entry",
constraint=models.UniqueConstraint(
fields=("timestamp", "git_hash", "category"), name="unique entry"
),
),
]

5
frog_api/models.py
View File

@ -77,6 +77,11 @@ class Entry(models.Model):
class Meta:
verbose_name_plural = "Entries"
ordering = ["-timestamp"]
constraints = [
models.UniqueConstraint(
fields=["timestamp", "git_hash", "category"], name="unique entry"
)
]
def __str__(self) -> str:
time_string = datetime.utcfromtimestamp(self.timestamp).strftime(

150
frog_api/tests.py
View File

@ -41,42 +41,44 @@ class CreateCategoryTests(APITestCase):
self.assertEqual(Category.objects.count(), 2)
class CreateEntriesTests(APITestCase):
def test_create_entries(self) -> None:
"""
Ensure that the entry creation endpoint works
"""
create_json = {
"api_key": "test_key_123",
"entries": [
{
"categories": {
"default": {
"code_matching": 103860,
"code_total": 4747584,
"asm": 4597948,
"nonmatching_functions_count": 49,
"assets_identified": 0,
"assets_total": 40816656,
"code_decompiled": 120152,
"assets_debinarised": 0,
},
"actors": {
"code_matching": 103860,
"code_total": 4747584,
},
},
"timestamp": 1615435438,
"git_hash": "e788bfecbfb10afd4182332db99bb562ea75b1de",
}
],
SAMPLE_PROJECT_SLUG = "oot"
SAMPLE_VERSION_SLUG = "us"
SAMPLE_DATA = {
"api_key": "test_key_123",
"entries": [
{
"categories": {
"default": {
"code_matching": 103860,
"code_total": 4747584,
"asm": 4597948,
"nonmatching_functions_count": 49,
"assets_identified": 0,
"assets_total": 40816656,
"code_decompiled": 120152,
"assets_debinarised": 0,
},
"actors": {
"code_matching": 103860,
"code_total": 4747584,
},
},
"timestamp": 1615435438,
"git_hash": "e788bfecbfb10afd4182332db99bb562ea75b1de",
}
],
}
class CreateEntriesTests(APITestCase):
def create_project_metadata(self, project_slug: str, version_slug: str) -> None:
# Create a test Project, Version, and Categories
project = Project(slug="oot", name="Ocarina of Time", auth_key="test_key_123")
project = Project(
slug=project_slug, name="Ocarina of Time", auth_key="test_key_123"
)
project.save()
version = Version(slug="us", name="US", project=project)
version = Version(slug=version_slug, name="US", project=project)
version.save()
category1 = Category(slug="default", name="Default", version=version)
@ -85,14 +87,94 @@ class CreateEntriesTests(APITestCase):
category2 = Category(slug="actors", name="Actors", version=version)
category2.save()
def test_create_entries(self) -> None:
"""
Ensure that the entry creation endpoint works
"""
self.create_project_metadata(SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG)
response = self.client.post(
reverse("version-data", args=[project.slug, version.slug]),
create_json,
reverse("version-data", args=[SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG]),
SAMPLE_DATA,
format="json",
)
# Confirm we created the entries and that they are in the DB
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
self.assertEqual(Entry.objects.count(), 2)
self.assertEqual(Measure.objects.count(), 10)
def test_create_duplicate_entries(self) -> None:
"""
Ensure that it's impossible to make duplicate entries
"""
self.create_project_metadata(SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG)
response = self.client.post(
reverse("version-data", args=[SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG]),
SAMPLE_DATA,
format="json",
)
# Confirm we created the entries and that they are in the DB
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
self.assertEqual(Entry.objects.count(), 2)
self.assertEqual(Measure.objects.count(), 10)
response = self.client.post(
reverse("version-data", args=[SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG]),
SAMPLE_DATA,
format="json",
)
# Ensure we got a bad request code since we shouldn't be able to create duplicate entries
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Entry.objects.count(), 2)
self.assertEqual(Measure.objects.count(), 10)
def test_atomicity(self) -> None:
"""
Ensure that if some entries fail to be created, none are created
"""
self.create_project_metadata(SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG)
yucky_data = {
"api_key": "test_key_123",
"entries": [
{
"categories": {
"actors": {
"code_matching": 103860,
"code_total": 4747584,
},
},
"timestamp": 1615435438,
"git_hash": "e788bfecbfb10afd4182332db99bb562ea75b1de",
},
{
"categories": {
"actors": {
"code_matching": 103860,
"code_total": 4747584,
},
},
"timestamp": 1615435438,
"git_hash": "e788bfecbfb10afd4182332db99bb562ea75b1de",
},
],
}
response = self.client.post(
reverse("version-data", args=[SAMPLE_PROJECT_SLUG, SAMPLE_VERSION_SLUG]),
yucky_data,
format="json",
)
# Confirm no entries or measures are created
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(Entry.objects.count(), 0)
self.assertEqual(Measure.objects.count(), 0)

162
frog_api/views/data.py
View File

@ -1,18 +1,20 @@
from typing import Any
from enum import Enum
from typing import Any, Optional
from django.db import models
from django.db import IntegrityError, models, transaction
from django.template.defaultfilters import title
from rest_framework import status
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
from frog_api.cache import (
get_entries_cache,
invalidate_entries_cache,
set_entries_cache,
get_entries_cache,
)
from frog_api.exceptions import (
InvalidDataException,
NoEntriesException,
)
from frog_api.models import Entry, Measure, Project, Version
from frog_api.exceptions import InvalidDataException, NoEntriesException
from frog_api.models import Category, Entry, Measure, Project, Version
from frog_api.serializers.model_serializers import EntrySerializer
from frog_api.serializers.request_serializers import CreateEntriesSerializer
from frog_api.views.common import (
@ -21,18 +23,13 @@ from frog_api.views.common import (
get_version,
validate_api_key,
)
from rest_framework import status
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
DEFAULT_CATEGORY_SLUG = "default"
DEFAULT_CATEGORY_NAME = "Default"
EntryT = dict[str, Any]
def get_latest_entry(
project_slug: str, version_slug: str, category_slug: str
) -> list[dict[str, Any]]:
) -> Optional[EntryT]:
project = get_project(project_slug)
version = get_version(version_slug, project)
category = get_category(category_slug, version)
@ -40,14 +37,23 @@ def get_latest_entry(
entry = Entry.objects.filter(category=category).first()
if entry is None:
raise NoEntriesException(project_slug, version_slug, category_slug)
return None
return [EntrySerializer(entry).data]
return EntrySerializer(entry).data
def get_latest_entry_throw(
project_slug: str, version_slug: str, category_slug: str
) -> EntryT:
entry = get_latest_entry(project_slug, version_slug, category_slug)
if entry is None:
raise NoEntriesException(project_slug, version_slug, category_slug)
return entry
def get_all_entries(
project_slug: str, version_slug: str, category_slug: str
) -> list[dict[str, Any]]:
) -> list[EntryT]:
data = get_entries_cache(project_slug, version_slug, category_slug)
if data:
return data # type: ignore
@ -66,10 +72,16 @@ def get_all_entries(
def get_versions_digest_for_project(project: Project) -> dict[Any, Any]:
versions = {}
for version in Version.objects.filter(project=project):
category_slug = DEFAULT_CATEGORY_SLUG
entry = get_latest_entry(project.slug, version.slug, category_slug)
if entry is not None:
versions[version.slug] = {"default": entry}
category_entries: dict[str, list[EntryT]] = {}
for category in Category.objects.filter(version=version):
entry = get_latest_entry(project.slug, version.slug, category.slug)
if entry is not None:
category_entries[category.slug] = [entry]
if len(category_entries) > 0:
versions[version.slug] = category_entries
return versions
@ -109,8 +121,11 @@ class ProjectDataView(APIView):
def get_progress_shield(
request: Request, project_slug: str, version_slug: str, category_slug: str
) -> dict[str, Any]:
latest = get_latest_entry(project_slug, version_slug, category_slug)
latest_measures = latest[0]["measures"]
latest = get_latest_entry_throw(project_slug, version_slug, category_slug)
assert latest is not None
latest_measures = latest["measures"]
project = get_project(project_slug)
version = get_version(version_slug, project)
@ -157,7 +172,10 @@ def get_progress_shield(
return {"schemaVersion": 1, "label": label, "message": message, "color": color}
VALID_MODES: set[str] = {"latest", "all", "shield"}
class Mode(Enum):
LATEST = "latest"
ALL = "all"
SHIELD = "shield"
class VersionDataView(APIView):
@ -195,12 +213,16 @@ class VersionDataView(APIView):
value = categories[cat][measure_type]
if type(value) != int:
raise InvalidDataException(
f"{cat}:{measure_type} must be an integer"
f"{cat}:{measure_type} must be an integer, not {type(value): {value}}"
)
to_save.append(Measure(entry=entry, type=measure_type, value=value))
for s in to_save:
s.save()
try:
with transaction.atomic():
for s in to_save:
s.save()
except IntegrityError as e:
raise InvalidDataException(f"Integrity error: {e}")
invalidate_entries_cache(project_slug, version_slug, data)
@ -211,26 +233,38 @@ class VersionDataView(APIView):
Return the most recent entry for overall progress for a version of a project.
"""
category_slug = DEFAULT_CATEGORY_SLUG
mode_str = self.request.query_params.get("mode", Mode.LATEST.value)
mode = self.request.query_params.get("mode", "latest")
if mode not in VALID_MODES:
raise InvalidDataException(f"Invalid mode specified: {mode}")
try:
mode: Mode = Mode(mode_str)
except ValueError:
raise InvalidDataException(f"Invalid mode specified: {mode_str}")
if mode == "latest":
entries = get_latest_entry(project_slug, version_slug, category_slug)
elif mode == "all":
entries = get_all_entries(project_slug, version_slug, category_slug)
elif mode == "shield":
return Response(
get_progress_shield(
self.request, project_slug, version_slug, category_slug
project = get_project(project_slug)
version = get_version(version_slug, project)
categories_data: dict[str, list[EntryT]]
match mode:
case Mode.LATEST:
categories_data = {}
for category in Category.objects.filter(version=version):
entry = get_latest_entry(project_slug, version_slug, category.slug)
if entry is not None:
categories_data[category.slug] = [entry]
response_json = {project_slug: {version_slug: categories_data}}
return Response(response_json)
case Mode.ALL:
categories_data = {}
for category in Category.objects.filter(version=version):
entries = get_all_entries(project_slug, version_slug, category.slug)
categories_data[category.slug] = entries
response_json = {project_slug: {version_slug: categories_data}}
return Response(response_json)
case Mode.SHIELD:
raise InvalidDataException(
"Category must be specified for shield output"
)
)
response_json = {project_slug: {version_slug: {category_slug: entries}}}
return Response(response_json)
def post(self, request: Request, project_slug: str, version_slug: str) -> Response:
result = VersionDataView.create_entries(
@ -257,21 +291,27 @@ class CategoryDataView(APIView):
Return data for a specific category and a version of a project.
"""
mode = self.request.query_params.get("mode", "latest")
if mode not in VALID_MODES:
raise InvalidDataException(f"Invalid mode specified: {mode}")
mode_str = self.request.query_params.get("mode", Mode.LATEST.value)
if mode == "latest":
entries = get_latest_entry(project_slug, version_slug, category_slug)
elif mode == "all":
entries = get_all_entries(project_slug, version_slug, category_slug)
elif mode == "shield":
return Response(
get_progress_shield(
self.request, project_slug, version_slug, category_slug
try:
mode: Mode = Mode(mode_str)
except ValueError:
raise InvalidDataException(f"Invalid mode specified: {mode_str}")
match mode:
case Mode.LATEST:
entries = [
get_latest_entry_throw(project_slug, version_slug, category_slug)
]
response_json = {project_slug: {version_slug: {category_slug: entries}}}
return Response(response_json)
case Mode.ALL:
entries = get_all_entries(project_slug, version_slug, category_slug)
response_json = {project_slug: {version_slug: {category_slug: entries}}}
return Response(response_json)
case Mode.SHIELD:
return Response(
get_progress_shield(
self.request, project_slug, version_slug, category_slug
)
)
)
response_json = {project_slug: {version_slug: {category_slug: entries}}}
return Response(response_json)

10
frog_api/views/structure.py
View File

@ -19,8 +19,6 @@ from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
from frog_api.views.data import DEFAULT_CATEGORY_NAME, DEFAULT_CATEGORY_SLUG
class RootStructureView(APIView):
def get(self, request: Request, format: Any = None) -> Response:
@ -76,14 +74,6 @@ class VersionStructureView(APIView):
name=request_ser.data["name"],
)
version.save()
# Create the default category
default_cat = Category(
version=version,
slug=DEFAULT_CATEGORY_SLUG,
name=DEFAULT_CATEGORY_NAME,
)
default_cat.save()
return Response(status=status.HTTP_201_CREATED)
def delete(

2
mypy.ini
View File

@ -1,6 +1,6 @@
[mypy]
# The mypy configurations: https://mypy.readthedocs.io/en/latest/config_file.html
python_version = 3.9
python_version = 3.10
check_untyped_defs = True
disallow_any_generics = True

902
poetry.lock generated
View File

File diff suppressed because it is too large Load Diff

2
pyproject.toml
View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "frogress"
version = "0.1.0"
version = "0.2.0"
description = "Progress API for decompilation projects"
authors = ["Ethan Roseman <ethteck@gmail.com>"]
license = "MIT"