Commit Graph

97 Commits

Author SHA1 Message Date
4611a4b501 Vendor nintendo-lz crate to fix issues & avoid old deps 2024-10-04 21:02:04 -06:00
b184fee73f Migrate SectionIndex/SymbolIndex to u32
This halves the size of structs like SectionAddress.
2024-10-04 20:40:50 -06:00
ef7e0db095 VFS fixes and improvements, update README.md
`vfs ls`: Now displays size, detected file format, and decompressed
size (if applicable). `-r`/`--recursive` lists files recursively.
`-s`/`--short` prints only filenames.

`vfs cp`: Copies files recursively when the source is a directory.
`--no-decompress` disables automatic decompression for Yay0/Yaz0.
`-q` disables printing copied files.

`rarc` and `u8` commands are now thin wrappers over `vfs ls` and
`vfs cp`. For example, `rarc list file.arc` is now equivalent to
`vfs ls file.arc:`. `rarc extract file.arc -o out` is equivalent
to `vfs cp file.arc: out`.
2024-10-04 18:15:24 -06:00
f91c2a1474 Load objects from disc image & vfs module
Revamps support for container paths and centralizes logic into a VFS (virtual file system) module.
The new VFS architecture supports disc images and any layer of nesting.

For example, the following command works:
`dtk dol info 'Interactive Multi-Game Demo Disc - July 2002 (USA).rvz:files/zz_StarFox051702_e3.tgc:files/default.dol'`
This opens a TGC file inside an RVZ disc image, then reads `default.dol` in the FST.

Another example:
`dtk rel info 'Legend of Zelda, The - The Wind Waker (USA).rvz:files/RELS.arc:mmem/f_pc_profile_lst.rel'`
This opens a RARC archive inside an RVZ disc image, loads the Yaz0-compressed REL and
decompresses it on the fly.

This all operates in memory with minimal overhead, with no need to extract temporary files.

Supported container formats:
- Disc images (ISO/GCM, WIA/RVZ, WBFS, CISO, NFS, GCZ, TGC)
- RARC/SZS and U8 (.arc)

Supported compression formats:
- Yaz0 (SZS)
- Yay0 (SZP)
- NLZSS (.lz)

Additionally, projects can utilize a new configuration key `object_base`:
```
object: orig/GZLE01/sys/main.dol
modules:
- object: orig/GZLE01/files/RELS.arc:rels/mmem/f_pc_profile_lst.rel
```
becomes
```
object_base: orig/GZLE01
object: sys/main.dol
modules:
- object: files/RELS.arc:mmem/f_pc_profile_lst.rel
```
When loading the objects, decomp-toolkit will automatically check the `object_base`
directory for any disc images. (They can be named anything, but must be in the folder
root) If one is found, all objects will be fetched from the disc image itself, rather
than having to extract the files manually.

While still a work in progress, two new `vfs` commands were added: `vfs ls` and `vfs cp`.
These commands are very barebones currently, but allow listing directory contents and
extracting files from decomp-toolkit's vfs representation:
```
❯ dtk vfs ls disc.rvz:
files
sys

❯ dtk vfs ls disc.rvz:sys
boot.bin
bi2.bin
apploader.bin
fst.bin
main.dol

❯ dtk vfs cp disc.rvz:sys/main.dol .
```
2024-10-03 21:50:35 -06:00
ac45676770 Fixes for updated object crate
object stopped including the ELF null
symbol and null section in the respective
iterators. We relied on this behavior for
building certain vectors in order of
symbol index. Adjust this logic to
restore the correct behavior.
2024-09-29 12:00:44 -06:00
cfcd146dfa Add map config for generating symbols/splits
Useful for extracting information from
map files that aren't fully well-formed,
such as ones from existing decompilation
projects.
2024-09-09 20:38:25 -06:00
d4f695ffc7 dol diff: Loosen @ symbol match criteria 2024-09-09 20:36:46 -06:00
8b793b5616 Update CI workflow & all dependencies 2024-09-09 20:36:18 -06:00
Wesley Moret
da6a514fac RSO: make command (#67)
Allow to create rso file from relocatable elf

No sel file support yet
2024-08-06 21:15:03 -06:00
riidefi
cfeacd2c3a elf2dol: Support section name denylist (#64) 2024-07-17 20:02:34 -06:00
c484952912 Support generating RELs with non-sequential module IDs
Uses the module ID from the configuration, if present,
falling back to the standard behavior of sequential
module IDs.
2024-06-24 17:43:33 -06:00
761a940f9e Match original "exec" for REL sections
mwld writes empty code sections as
NULL type in the PLF, but sometimes
the original REL has the exec flag
set for these sections. Match the
original value.
2024-06-10 17:39:40 -06:00
e359ea1e9e Improve REL relocation error handling 2024-06-10 17:37:06 -06:00
eb7c3e9d9f Detect _savevr/_restvr + check in RELs
Some games include a copy of MW
runtime.c in RELs. Easy to check.

Some games also include _savevr/
_restvr for AltiVec. This is weird
but we can handle it as well.
2024-06-10 00:43:00 -06:00
4ea4ec86d0 Partially revert "Rework section alignment handling"
It turns out we can't actually
trust mwld to put our alignment
values in the PLF. Sad.
2024-06-10 00:37:17 -06:00
46cf0be183 Add U8 (newer .arc) support
Supports the U8 .arc format, just
like the older RARC format.

`u8 list`, `u8 extract` and support
for U8 archive paths in config.yml
2024-06-03 20:34:35 -06:00
255123796e Instruction disassembly in dol diff
When a function diff is detected in
`dol diff`, objdiff-core is used to
print a detailed view highlighting
any differences.

Resolves #28
2024-06-03 20:31:06 -06:00
61cd72cd11 Support address in config symbol references
Example: `symbol_name!.data:0x1234`
Allows disambiguating local symbols
with the same name.

Supported in `extract` and
`add_relocations` in config.yml

Resolves #58
2024-06-03 18:08:06 -06:00
c3f3ea58e8 Support block_relocations and add_relocations in config.yml
This allows more granular control over generated relocations.

Also optimizes relocation address validity checks,
leading to ~20% faster relocation analysis.

Config example:
```
block_relocations:
# Block any relocation pointing to this address.
- target: .data:0x80130140
# Block any relocation originating from this address.
- source: .text:0x80047160
  # (optional) End address to make it a range.
  end: .text:0x800471A8

add_relocations:
# Inserts or overwrites a relocation.
# From: `subi r3, r3, 0x7657`
# To: `li r3, mesWInsert-0x1@sda21`
- source: .text:0x800473F4
  type: sda21
  target: mesWInsert
  addend: -1
```

Resolves #33
Resolves #52
2024-05-19 22:49:40 -06:00
cadmic
876b78bfa6 Reconstruct "erased" tags from DWARF debugging info (#51)
* Read erased tags

* cargo +nightly fmt

* .filter(...).next() -> find(...)

* Plumb both data and address endians through

* Rename has_erased_parent -> is_erased_root and comment
2024-05-15 23:06:30 -06:00
4dacf2f39a Add disc commands: info, extract, convert, verify
Utilizing https://github.com/encounter/nod-rs
2024-05-01 00:12:20 -06:00
dc7e307c44 Rework section alignment handling
- Honor splits.txt alignment values when writing ldscript.lcf
- Add alignment values to ldscript_partial.lcf, remove hacky code from rel make
- Guess alignment values in DOL loader

Fixes #27
2024-04-30 18:04:17 -06:00
989293a477 Add Yay0/Yaz0 compression & decompression
Uses orthrus-ncompress

Fixes #6
2024-04-30 18:03:00 -06:00
d0f39f1d82 Support global labels in REL creation (_savegpr, etc) 2024-04-03 00:13:15 -06:00
7fd039722d Format & use syntect regex-fancy
The onig crate no longer builds
in CI with cargo-zigbuild.
2024-03-13 18:30:15 -06:00
d727110262 Update all dependencies & support new demangle flag
- Support `--mw-extensions` flag for demangle
- Fix relocation handling for object crate upgrade
2024-03-13 18:23:09 -06:00
b829e15438 Rework .splitmeta, now .note.split
Uses actual ELF .note format, which is
more standard and handled better by mwld.
2024-03-04 18:12:20 -07:00
4f8a9e6fab Write .splitmeta section in split objects
This enables showing the original address of
symbols in objdiff, as well as `elf disasm`
on split objects retaining the original
addresses.
2024-02-28 22:27:38 -07:00
Robin Avery
a94edb573f Emit an error code when dol diff does not match (#43)
* Emit an error code when `dol diff` does not match

* `exit(1)` instead of `bail`
2024-02-28 19:18:58 -07:00
Chippy
807dc6a440 Rename and output GCC extensions (#40)
Outputs GCC Source Info extensions
Add asm language to support DWARF emitted by MWCC MIPS assembler
Update MwOverlayBranch handling
2024-02-20 20:56:49 -07:00
1superchip
2784859c4f Handle more CompileUnit attributes in DWARF (#38)
* Handle more DWARF CompileUnit attributes

Handle comp_dir and add two unknown attributes

* Remove comment about unknown attributes
2024-02-19 16:13:35 -07:00
9a6cb70ff8 Add ar extract command 2024-02-09 15:00:09 -07:00
4a84975648 dol apply: Don't apply gap symbols
Also don't overwrite "unknown" scope
with global.

Fixes #30
2024-01-24 23:23:05 -07:00
26cc6a13b4 shasum: Support -o in hash mode
Fixes #32
2024-01-24 23:21:55 -07:00
7b7c8ce251 Add noexport attr to symbols.txt
Rename `auto_force_active` to `export_all`.
This is a better solution to code_merging, as
individual functions can be marked `noexport`.
2024-01-14 13:58:18 -07:00
968f50ebed Add auto_force_active option (disable to support -code_merging)
Fixes #13
2024-01-13 21:54:10 -07:00
Benjamin Moir
6b59c677d4 Add support for MwOverlayBranch (#24)
* Add support for MwOverlayBranch

* Make clippy happy

* cargo fmt
2024-01-07 08:02:01 -07:00
458d0599f5 dwarf dump: Emit address range for compile units and functions 2024-01-06 17:51:09 -07:00
Sewer
d43b95a0e4 Added: Support for Little Endian DWARF, Inline Arrays & Additional MetroWerks Extensions (#14)
* Added: Minimal Required Changes to Dump Sonic Heroes (PS2)

* Added: Switch for Little Endian DWARF Dump

* Added: Support for Array Ordering

* Added: Big/Little Endian Variants of u32_from_bytes

* Changed: Detect bitness from .elf header.

* Changed: Use Proper MetroWerks Names

* Changed: Use Endianness from File API

* Refactor DWARF endian handling & some fixes

* Undo accidental formatting changes

---------

Co-authored-by: Luke Street <luke@street.dev>
2023-12-11 13:07:29 -05:00
0cfc5df20b Overhauled common BSS support & more
- With a map, attempts to detect and handle common BSS automatically
- With a map, attempts to detect and correct inflated common BSS bug (< GC 2.7 linker)
- Support for "stripped" symbols, sometimes required to match inflated common BSS sizes
- Warns on duplicated TUs in a map (other than common BSS)
- Automatically adds `comment:0` to `.s` TUs from a map (avoids linker crash)
2023-11-29 18:31:35 -05:00
5c22c8850e Support .BINARY section
Resolves #12
2023-11-29 18:14:48 -05:00
4cb6f4f85d dwarf dump: Better support for GCC/ProDG & various fixes
- Displays subroutine "static" and "inline" if present
- Displays subroutine labels, blocks and inlines if present
- Displays struct member visibility if present
- Skips tags that can't be processed rather than bailing
2023-11-27 16:35:12 -05:00
dd60128ba0 Extract embedded assets to binary and C header
Adds an "extract" list to project configuration:
```
extract:
- symbol: SomeData
  binary: Lib/SomeData.bin
  header: Lib/SomeData.inc
```
This example extracts the data of symbol
`SomeData` to `out_dir/bin/Lib/SomeData.bin`,
and a C array representation to
`out_dir/include/Lib/SomeData.inc`.

Resolves #11
2023-11-26 00:35:05 -05:00
ebff47924f Add links field to config for overriding REL linkage
By default, every REL is linked with every other REL. Some games, like Mario Party, link RELs individually, so the module IDs are not unique. To support this, the `links` field can be used to override which _other_ modules are included in a module's analysis.
2023-11-18 23:37:50 -05:00
28af4872ab rel make: Add --quiet/-q, --names/-n args
`-n` is a somewhat ugly hack to only select certain modules from `config.yml`, for games with overlapping module IDs
2023-11-18 23:34:44 -05:00
8659b56da4 Skip writing objects and ldscripts if unchanged
Avoids unnecessary rebuilds with build systems like ninja & make that track mtime
2023-11-18 17:21:08 -05:00
363fa2ac72 Use custom syntect packs to reduce binary size
- Custom syntax set containing only C/C++
- Custom themeset containing only Solarized (Dark)
- Disable default syntect features
2023-11-18 16:12:03 -05:00
8f559c8921 Update dependencies & CI changes
- Use separate release-lto profile for CI
- A couple of clippy fixes
2023-11-18 15:03:24 -05:00
5e13998e93 Major dwarf dump rework
- Supports games with C++ DWARF info
- Syntax highlighting when printing to console (disable with `--no-color`)
- Overall improvements to parsing and output
2023-11-18 13:56:58 -05:00
4935708b61 Fix REL v2 creation (and v1, hopefully)
- Adjusts `write_rel` to use the proper ordering for relocations and imports based on the REL version.
- Adds `-r`/`--relocations` switch to `rel info` that prints (very) verbose relocation information.
2023-11-09 01:16:37 -05:00