Commit Graph

234 Commits

Author SHA1 Message Date
Luke Street 91aa36c120 Clean up VFS error handling 2024-11-07 09:00:52 -07:00
Luke Street 9fc56d847f Add `rename` field to extract configuration
Allows renaming, for example, local statics from `test$1234`
to `test` for inclusion in the source function.
2024-11-07 08:44:24 -07:00
Luke Street 1cc38ad621 Add WAD support to VFS & `wad` commands 2024-11-07 08:43:20 -07:00
Luke Street 146c4d2f8c Add `header_type` and `custom_type` to extract config
Extract configuration is now emitted in the output config, so
tooling can load and perform their own tasks on extracted assets
without having to parse YAML.

`header_type`:
- `symbol` (default): Emit a full symbol declaration.
- `raw`: Emit raw array data (for wrapping in your own declaration)
- `none`: Don't emit a header at all. (For custom processing)

`custom_type`/`custom_data`: Passed through to the output config
as-is for custom tasks.
2024-10-28 17:44:07 -06:00
Luke Street f984bc3fb2 Cleanup find_object_base & better error handling 2024-10-18 00:41:17 -06:00
cadmic 8823c2225e
Follow symlinks when looking for disc images (#78) 2024-10-18 00:37:54 -06:00
Luke Street fa5068fd6d Fix ELF relocation processing
Another bug caused by object removing ELF null section / symbol
from their iterators.
2024-10-18 00:24:34 -06:00
Luke Street 3ada073da1 Update dependencies 2024-10-18 00:19:06 -06:00
Luke Street 5c7560bcea Fix address analysis with negative `add` operands
Resolves #76
2024-10-14 09:40:46 -06:00
Luke Street 8d8d801b2f A couple process_elf fixes 2024-10-13 14:15:13 -06:00
Luke Street bee4570a4c Always check for extracted files in object resolution
Fixes an issue where extracted files would not be found after
removing the disc image from the orig dir.
2024-10-13 13:36:01 -06:00
Luke Street 18bd608fe8 Extract files from disc image to `object_base` by default
When `extract_objects` is enabled, objects will be extracted once
from a disc image into `object_base`, and then used directly from
`object_base` going forward.

This allows users to delete the disc image from their `orig` dir
once the initial build completes.
2024-10-13 00:53:45 -06:00
Luke Street 1a9736f8d9 ci: Add Rust workspace cache 2024-10-12 23:32:51 -06:00
Luke Street 4fe2608e07 Make `selfile` relative to `object_base` 2024-10-12 23:31:41 -06:00
Luke Street 601c8e1a5e ci: Setup python venv for cargo-zigbuild 2024-10-10 22:37:55 -06:00
Luke Street 2e524e6806 Use typed-path in place of std Path/PathBuf
This allows handling path conversions in a more structured way,
as well as avoiding needless UTF-8 checks. All argument inputs
use `Utf8NativePathBuf`, while all config entries use
`Utf8UnixPathBuf`, ensuring that we deserialize/serialize using
forward slashes. We can omit `.display()` and lossy UTF-8
conversions since all paths are known valid UTF-8.
2024-10-04 23:38:15 -06:00
Luke Street 64d0491256 Remove unused metroidbuildinfo command 2024-10-04 21:08:34 -06:00
Luke Street 4611a4b501 Vendor nintendo-lz crate to fix issues & avoid old deps 2024-10-04 21:02:04 -06:00
Luke Street b184fee73f Migrate SectionIndex/SymbolIndex to u32
This halves the size of structs like SectionAddress.
2024-10-04 20:40:50 -06:00
Luke Street 1f4b452bd5 Fix disc VFS layout & update nod 2024-10-04 20:33:46 -06:00
Luke Street f346239b81 Update README.md 2024-10-04 19:17:46 -06:00
Luke Street ef7e0db095 VFS fixes and improvements, update README.md
`vfs ls`: Now displays size, detected file format, and decompressed
size (if applicable). `-r`/`--recursive` lists files recursively.
`-s`/`--short` prints only filenames.

`vfs cp`: Copies files recursively when the source is a directory.
`--no-decompress` disables automatic decompression for Yay0/Yaz0.
`-q` disables printing copied files.

`rarc` and `u8` commands are now thin wrappers over `vfs ls` and
`vfs cp`. For example, `rarc list file.arc` is now equivalent to
`vfs ls file.arc:`. `rarc extract file.arc -o out` is equivalent
to `vfs cp file.arc: out`.
2024-10-04 18:15:24 -06:00
Luke Street 281b0f7104 Decode extab entries as comment in assembly output 2024-10-03 22:33:21 -06:00
Luke Street 71701b5667 Update all dependencies 2024-10-03 22:24:54 -06:00
Luke Street f91c2a1474 Load objects from disc image & `vfs` module
Revamps support for container paths and centralizes logic into a VFS (virtual file system) module.
The new VFS architecture supports disc images and any layer of nesting.

For example, the following command works:
`dtk dol info 'Interactive Multi-Game Demo Disc - July 2002 (USA).rvz:files/zz_StarFox051702_e3.tgc:files/default.dol'`
This opens a TGC file inside an RVZ disc image, then reads `default.dol` in the FST.

Another example:
`dtk rel info 'Legend of Zelda, The - The Wind Waker (USA).rvz:files/RELS.arc:mmem/f_pc_profile_lst.rel'`
This opens a RARC archive inside an RVZ disc image, loads the Yaz0-compressed REL and
decompresses it on the fly.

This all operates in memory with minimal overhead, with no need to extract temporary files.

Supported container formats:
- Disc images (ISO/GCM, WIA/RVZ, WBFS, CISO, NFS, GCZ, TGC)
- RARC/SZS and U8 (.arc)

Supported compression formats:
- Yaz0 (SZS)
- Yay0 (SZP)
- NLZSS (.lz)

Additionally, projects can utilize a new configuration key `object_base`:
```
object: orig/GZLE01/sys/main.dol
modules:
- object: orig/GZLE01/files/RELS.arc:rels/mmem/f_pc_profile_lst.rel
```
becomes
```
object_base: orig/GZLE01
object: sys/main.dol
modules:
- object: files/RELS.arc:mmem/f_pc_profile_lst.rel
```
When loading the objects, decomp-toolkit will automatically check the `object_base`
directory for any disc images. (They can be named anything, but must be in the folder
root) If one is found, all objects will be fetched from the disc image itself, rather
than having to extract the files manually.

While still a work in progress, two new `vfs` commands were added: `vfs ls` and `vfs cp`.
These commands are very barebones currently, but allow listing directory contents and
extracting files from decomp-toolkit's vfs representation:
```
❯ dtk vfs ls disc.rvz:
files
sys

❯ dtk vfs ls disc.rvz:sys
boot.bin
bi2.bin
apploader.bin
fst.bin
main.dol

❯ dtk vfs cp disc.rvz:sys/main.dol .
```
2024-10-03 21:50:35 -06:00
Amber Brault 26f52f65b7
Automatically check for invalid extab relocations (#75)
* Begin work on extab reloc analysis code

* Refactoring

* Make format happy

* Address feedback + improvements
2024-10-03 01:13:23 -06:00
Luke Street c106123877 Use mimalloc when targeting musl
Also removes the armv7 linux build.
If you were using this, let me know!

Updates all dependencies
2024-09-29 12:20:28 -06:00
Luke Street dfda3d5ea3 Fix todo! in FileIterator::next_path 2024-09-29 12:02:55 -06:00
Luke Street 68f4552e44 Better support for SMG/TP maps
These maps are post-processed and have
various issues. This includes some hacks
to work around those issues as much as
possible.
2024-09-29 12:02:26 -06:00
Luke Street ac45676770 Fixes for updated object crate
object stopped including the ELF null
symbol and null section in the respective
iterators. We relied on this behavior for
building certain vectors in order of
symbol index. Adjust this logic to
restore the correct behavior.
2024-09-29 12:00:44 -06:00
Luke Street e430cb56f5 Remove features from CI 2024-09-09 22:05:33 -06:00
Luke Street 0719c73ef8 Resolve clippy issue 2024-09-09 20:39:54 -06:00
Luke Street cfcd146dfa Add map config for generating symbols/splits
Useful for extracting information from
map files that aren't fully well-formed,
such as ones from existing decompilation
projects.
2024-09-09 20:38:25 -06:00
Luke Street d4f695ffc7 dol diff: Loosen @ symbol match criteria 2024-09-09 20:36:46 -06:00
Luke Street 8b793b5616 Update CI workflow & all dependencies 2024-09-09 20:36:18 -06:00
Luke Street 9dfdbb9301 Fix v1-2 REL alignment regression
Alignment after section data and
before relocations / import table
is exclusive to REL v3.
2024-09-05 00:26:14 -06:00
Luke Street c403931f0f Update nod for TGC disc support 2024-09-04 20:09:56 -06:00
Luke Street d9817f63d5 Fix .note.split warnings for older mwld versions
Prior to mwld GC 3.0a3, the linker doesn't support ELF .note sections
properly. With GC 2.7, it crashes if the section type is SHT_NOTE.
Use the same section type as .mwcats.* so the linker ignores it.
2024-09-04 19:56:22 -06:00
Chippy a112eb1829
DWARF: Add PureVirtual/Virtual Block2 attributes (#70) 2024-08-26 17:41:25 -07:00
Luke Street b6a29fa910 Add split `order` attribute for manual reordering
Example in splits.txt:
```
file1.cpp: order:0
  ...

file2.cpp: order:1
  ...

file3.cpp: order:2
  ...
```

This ensures that file2.cpp is always
anchored in between 1 and 3 when resolving
the final link order.
2024-08-11 20:38:11 -06:00
Wesley Moret da6a514fac
RSO: `make` command (#67)
Allow to create rso file from relocatable elf

No sel file support yet
2024-08-06 21:15:03 -06:00
riidefi cfeacd2c3a
elf2dol: Support section name denylist (#64) 2024-07-17 20:02:34 -06:00
First Last c3c7c2b062
Properly locate ProDG .bss sections (partial addressing of #62) (#63)
* Locate ProDG .bss sections (partial addressing of #62)

* Support both correct and incorrect memset calls
2024-07-16 23:14:46 -06:00
Luke Street c484952912 Support generating RELs with non-sequential module IDs
Uses the module ID from the configuration, if present,
falling back to the standard behavior of sequential
module IDs.
2024-06-24 17:43:33 -06:00
Luke Street 4d039140f2 Version 0.9.2 2024-06-10 17:41:05 -06:00
Luke Street 9c12efa64e Update orthrus-ncompress 2024-06-10 17:40:46 -06:00
Luke Street af3bcf5f9e Create gap symbols at the end of sections
Before, the end of the section
was not considered, so missing
symbols at the end of a section
would omit that data entirely.
2024-06-10 17:40:39 -06:00
Luke Street 761a940f9e Match original "exec" for REL sections
mwld writes empty code sections as
NULL type in the PLF, but sometimes
the original REL has the exec flag
set for these sections. Match the
original value.
2024-06-10 17:39:40 -06:00
Luke Street 3841004947 Fix writing empty v3 RELs
Resolves #59
2024-06-10 17:38:11 -06:00
Luke Street e359ea1e9e Improve REL relocation error handling 2024-06-10 17:37:06 -06:00