Commit Graph

217 Commits

Author SHA1 Message Date
Luke Street 4611a4b501 Vendor nintendo-lz crate to fix issues & avoid old deps 2024-10-04 21:02:04 -06:00
Luke Street b184fee73f Migrate SectionIndex/SymbolIndex to u32
This halves the size of structs like SectionAddress.
2024-10-04 20:40:50 -06:00
Luke Street 1f4b452bd5 Fix disc VFS layout & update nod 2024-10-04 20:33:46 -06:00
Luke Street f346239b81 Update README.md 2024-10-04 19:17:46 -06:00
Luke Street ef7e0db095 VFS fixes and improvements, update README.md
`vfs ls`: Now displays size, detected file format, and decompressed
size (if applicable). `-r`/`--recursive` lists files recursively.
`-s`/`--short` prints only filenames.

`vfs cp`: Copies files recursively when the source is a directory.
`--no-decompress` disables automatic decompression for Yay0/Yaz0.
`-q` disables printing copied files.

`rarc` and `u8` commands are now thin wrappers over `vfs ls` and
`vfs cp`. For example, `rarc list file.arc` is now equivalent to
`vfs ls file.arc:`. `rarc extract file.arc -o out` is equivalent
to `vfs cp file.arc: out`.
2024-10-04 18:15:24 -06:00
Luke Street 281b0f7104 Decode extab entries as comment in assembly output 2024-10-03 22:33:21 -06:00
Luke Street 71701b5667 Update all dependencies 2024-10-03 22:24:54 -06:00
Luke Street f91c2a1474 Load objects from disc image & `vfs` module
Revamps support for container paths and centralizes logic into a VFS (virtual file system) module.
The new VFS architecture supports disc images and any layer of nesting.

For example, the following command works:
`dtk dol info 'Interactive Multi-Game Demo Disc - July 2002 (USA).rvz:files/zz_StarFox051702_e3.tgc:files/default.dol'`
This opens a TGC file inside an RVZ disc image, then reads `default.dol` in the FST.

Another example:
`dtk rel info 'Legend of Zelda, The - The Wind Waker (USA).rvz:files/RELS.arc:mmem/f_pc_profile_lst.rel'`
This opens a RARC archive inside an RVZ disc image, loads the Yaz0-compressed REL and
decompresses it on the fly.

This all operates in memory with minimal overhead, with no need to extract temporary files.

Supported container formats:
- Disc images (ISO/GCM, WIA/RVZ, WBFS, CISO, NFS, GCZ, TGC)
- RARC/SZS and U8 (.arc)

Supported compression formats:
- Yaz0 (SZS)
- Yay0 (SZP)
- NLZSS (.lz)

Additionally, projects can utilize a new configuration key `object_base`:
```
object: orig/GZLE01/sys/main.dol
modules:
- object: orig/GZLE01/files/RELS.arc:rels/mmem/f_pc_profile_lst.rel
```
becomes
```
object_base: orig/GZLE01
object: sys/main.dol
modules:
- object: files/RELS.arc:mmem/f_pc_profile_lst.rel
```
When loading the objects, decomp-toolkit will automatically check the `object_base`
directory for any disc images. (They can be named anything, but must be in the folder
root) If one is found, all objects will be fetched from the disc image itself, rather
than having to extract the files manually.

While still a work in progress, two new `vfs` commands were added: `vfs ls` and `vfs cp`.
These commands are very barebones currently, but allow listing directory contents and
extracting files from decomp-toolkit's vfs representation:
```
❯ dtk vfs ls disc.rvz:
files
sys

❯ dtk vfs ls disc.rvz:sys
boot.bin
bi2.bin
apploader.bin
fst.bin
main.dol

❯ dtk vfs cp disc.rvz:sys/main.dol .
```
2024-10-03 21:50:35 -06:00
Amber Brault 26f52f65b7
Automatically check for invalid extab relocations (#75)
* Begin work on extab reloc analysis code

* Refactoring

* Make format happy

* Address feedback + improvements
2024-10-03 01:13:23 -06:00
Luke Street c106123877 Use mimalloc when targeting musl
Also removes the armv7 linux build.
If you were using this, let me know!

Updates all dependencies
2024-09-29 12:20:28 -06:00
Luke Street dfda3d5ea3 Fix todo! in FileIterator::next_path 2024-09-29 12:02:55 -06:00
Luke Street 68f4552e44 Better support for SMG/TP maps
These maps are post-processed and have
various issues. This includes some hacks
to work around those issues as much as
possible.
2024-09-29 12:02:26 -06:00
Luke Street ac45676770 Fixes for updated object crate
object stopped including the ELF null
symbol and null section in the respective
iterators. We relied on this behavior for
building certain vectors in order of
symbol index. Adjust this logic to
restore the correct behavior.
2024-09-29 12:00:44 -06:00
Luke Street e430cb56f5 Remove features from CI 2024-09-09 22:05:33 -06:00
Luke Street 0719c73ef8 Resolve clippy issue 2024-09-09 20:39:54 -06:00
Luke Street cfcd146dfa Add map config for generating symbols/splits
Useful for extracting information from
map files that aren't fully well-formed,
such as ones from existing decompilation
projects.
2024-09-09 20:38:25 -06:00
Luke Street d4f695ffc7 dol diff: Loosen @ symbol match criteria 2024-09-09 20:36:46 -06:00
Luke Street 8b793b5616 Update CI workflow & all dependencies 2024-09-09 20:36:18 -06:00
Luke Street 9dfdbb9301 Fix v1-2 REL alignment regression
Alignment after section data and
before relocations / import table
is exclusive to REL v3.
2024-09-05 00:26:14 -06:00
Luke Street c403931f0f Update nod for TGC disc support 2024-09-04 20:09:56 -06:00
Luke Street d9817f63d5 Fix .note.split warnings for older mwld versions
Prior to mwld GC 3.0a3, the linker doesn't support ELF .note sections
properly. With GC 2.7, it crashes if the section type is SHT_NOTE.
Use the same section type as .mwcats.* so the linker ignores it.
2024-09-04 19:56:22 -06:00
Chippy a112eb1829
DWARF: Add PureVirtual/Virtual Block2 attributes (#70) 2024-08-26 17:41:25 -07:00
Luke Street b6a29fa910 Add split `order` attribute for manual reordering
Example in splits.txt:
```
file1.cpp: order:0
  ...

file2.cpp: order:1
  ...

file3.cpp: order:2
  ...
```

This ensures that file2.cpp is always
anchored in between 1 and 3 when resolving
the final link order.
2024-08-11 20:38:11 -06:00
Wesley Moret da6a514fac
RSO: `make` command (#67)
Allow to create rso file from relocatable elf

No sel file support yet
2024-08-06 21:15:03 -06:00
riidefi cfeacd2c3a
elf2dol: Support section name denylist (#64) 2024-07-17 20:02:34 -06:00
First Last c3c7c2b062
Properly locate ProDG .bss sections (partial addressing of #62) (#63)
* Locate ProDG .bss sections (partial addressing of #62)

* Support both correct and incorrect memset calls
2024-07-16 23:14:46 -06:00
Luke Street c484952912 Support generating RELs with non-sequential module IDs
Uses the module ID from the configuration, if present,
falling back to the standard behavior of sequential
module IDs.
2024-06-24 17:43:33 -06:00
Luke Street 4d039140f2 Version 0.9.2 2024-06-10 17:41:05 -06:00
Luke Street 9c12efa64e Update orthrus-ncompress 2024-06-10 17:40:46 -06:00
Luke Street af3bcf5f9e Create gap symbols at the end of sections
Before, the end of the section
was not considered, so missing
symbols at the end of a section
would omit that data entirely.
2024-06-10 17:40:39 -06:00
Luke Street 761a940f9e Match original "exec" for REL sections
mwld writes empty code sections as
NULL type in the PLF, but sometimes
the original REL has the exec flag
set for these sections. Match the
original value.
2024-06-10 17:39:40 -06:00
Luke Street 3841004947 Fix writing empty v3 RELs
Resolves #59
2024-06-10 17:38:11 -06:00
Luke Street e359ea1e9e Improve REL relocation error handling 2024-06-10 17:37:06 -06:00
Luke Street 8813a2c9e4 Version 0.9.1 2024-06-10 00:43:44 -06:00
Luke Street eb7c3e9d9f Detect _savevr/_restvr + check in RELs
Some games include a copy of MW
runtime.c in RELs. Easy to check.

Some games also include _savevr/
_restvr for AltiVec. This is weird
but we can handle it as well.
2024-06-10 00:43:00 -06:00
Luke Street 3289b2a3aa Ignore invalid instructions
This could be a bad idea, but
it's unclear how to properly
handle these cases.

Resolves #55
2024-06-10 00:40:37 -06:00
Luke Street b44aa78c49 Fix REL alignment after section data 2024-06-10 00:38:06 -06:00
Luke Street 4ea4ec86d0 Partially revert "Rework section alignment handling"
It turns out we can't actually
trust mwld to put our alignment
values in the PLF. Sad.
2024-06-10 00:37:17 -06:00
Luke Street 4dd2ebf85a Rename ObjSections::count -> len 2024-06-09 16:10:34 -06:00
Luke Street 46cf0be183 Add U8 (newer .arc) support
Supports the U8 .arc format, just
like the older RARC format.

`u8 list`, `u8 extract` and support
for U8 archive paths in config.yml
2024-06-03 20:34:35 -06:00
Luke Street 255123796e Instruction disassembly in `dol diff`
When a function diff is detected in
`dol diff`, objdiff-core is used to
print a detailed view highlighting
any differences.

Resolves #28
2024-06-03 20:31:06 -06:00
Luke Street d63111466b SN GCC DWARF improvements
Now emits all DWARF tags for
Spongebob Squarepants: Creature
from the Krusty Krab properly.

Resolves #46
2024-06-03 18:48:28 -06:00
Luke Street 61cd72cd11 Support address in config symbol references
Example: `symbol_name!.data:0x1234`
Allows disambiguating local symbols
with the same name.

Supported in `extract` and
`add_relocations` in config.yml

Resolves #58
2024-06-03 18:08:06 -06:00
Luke Street 4701de32f6 Check for existing function with `bl`
Ensures that the analyzer won't
create a function when the target
is already contained within a
function. Useful with manual asm
that would otherwise trip up the
analyzer.

Partial work for #56
2024-06-03 18:06:10 -06:00
Luke Street 6b60c13834 Change REL "invalid relocation" to warning
Also ensures that `block_relocations`
works to remove the warning

Resolves #53
2024-06-03 18:03:39 -06:00
Luke Street 53c6d74a6b Fix addic/addic. handling in relocation tracker
Resolves #57
2024-06-03 18:01:55 -06:00
Luke Street d9770c48b2 Version 0.8.3 2024-05-19 23:09:16 -06:00
Luke Street c3f3ea58e8 Support `block_relocations` and `add_relocations` in `config.yml`
This allows more granular control over generated relocations.

Also optimizes relocation address validity checks,
leading to ~20% faster relocation analysis.

Config example:
```
block_relocations:
# Block any relocation pointing to this address.
- target: .data:0x80130140
# Block any relocation originating from this address.
- source: .text:0x80047160
  # (optional) End address to make it a range.
  end: .text:0x800471A8

add_relocations:
# Inserts or overwrites a relocation.
# From: `subi r3, r3, 0x7657`
# To: `li r3, mesWInsert-0x1@sda21`
- source: .text:0x800473F4
  type: sda21
  target: mesWInsert
  addend: -1
```

Resolves #33
Resolves #52
2024-05-19 22:49:40 -06:00
Luke Street e1c80655b7 Fix lcf generation for empty RELs
Also, handle alignment of 0 in rel make
2024-05-19 22:28:14 -06:00
Luke Street bf0d1a823f Version 0.8.2: Fix asm output 2024-05-17 07:12:05 -06:00