It was discovered that certain extab actions contain
uninitalized data from the compiler. This provides
a way to zero out uninitialized data in DOL or object
files. Usage: `dtk extab clean input.dol output.dol`
A `clean_extab` setting was added to config.yml, so
projects can link the cleaned objects and target the
cleaned DOL hash.
* add shiftjis as possible data type for symbols
* usage of anyhow:bail! -> bail!
* revise output of sjis strings
* rename shiftjis internally, symbols now uses sjis instead of shiftjis
* remove sjis decoding error check as the output is a comment
* run cargo fmt
A few issues were causing linker errors:
- Auto-splits could contain symbols that have a
higher alignment than the split itself. Detect
this and create a new auto-split at these symbols.
- The analyzer can miss objects in between other
objects if there are no direct relocations to them.
In these cases, non-zero data could just get totally
lost. Detect and create symbols for these.
When a link order is unresolvable, decomp-toolkit
now prints out all of the TUs in a cycle, easing
debugging.
Example:
```
Cyclic dependency encountered while resolving link order: Dolphin/os/__start.c -> Dolphin/os/__ppc_eabi_init.cpp -> Dolphin/pad/PadClamp.c -> Dolphin/pad/pad.c
```
Thanks @simonlindholm for the toposort impl.
Extract configuration is now emitted in the output config, so
tooling can load and perform their own tasks on extracted assets
without having to parse YAML.
`header_type`:
- `symbol` (default): Emit a full symbol declaration.
- `raw`: Emit raw array data (for wrapping in your own declaration)
- `none`: Don't emit a header at all. (For custom processing)
`custom_type`/`custom_data`: Passed through to the output config
as-is for custom tasks.
When `extract_objects` is enabled, objects will be extracted once
from a disc image into `object_base`, and then used directly from
`object_base` going forward.
This allows users to delete the disc image from their `orig` dir
once the initial build completes.
This allows handling path conversions in a more structured way,
as well as avoiding needless UTF-8 checks. All argument inputs
use `Utf8NativePathBuf`, while all config entries use
`Utf8UnixPathBuf`, ensuring that we deserialize/serialize using
forward slashes. We can omit `.display()` and lossy UTF-8
conversions since all paths are known valid UTF-8.
`vfs ls`: Now displays size, detected file format, and decompressed
size (if applicable). `-r`/`--recursive` lists files recursively.
`-s`/`--short` prints only filenames.
`vfs cp`: Copies files recursively when the source is a directory.
`--no-decompress` disables automatic decompression for Yay0/Yaz0.
`-q` disables printing copied files.
`rarc` and `u8` commands are now thin wrappers over `vfs ls` and
`vfs cp`. For example, `rarc list file.arc` is now equivalent to
`vfs ls file.arc:`. `rarc extract file.arc -o out` is equivalent
to `vfs cp file.arc: out`.
Revamps support for container paths and centralizes logic into a VFS (virtual file system) module.
The new VFS architecture supports disc images and any layer of nesting.
For example, the following command works:
`dtk dol info 'Interactive Multi-Game Demo Disc - July 2002 (USA).rvz:files/zz_StarFox051702_e3.tgc:files/default.dol'`
This opens a TGC file inside an RVZ disc image, then reads `default.dol` in the FST.
Another example:
`dtk rel info 'Legend of Zelda, The - The Wind Waker (USA).rvz:files/RELS.arc:mmem/f_pc_profile_lst.rel'`
This opens a RARC archive inside an RVZ disc image, loads the Yaz0-compressed REL and
decompresses it on the fly.
This all operates in memory with minimal overhead, with no need to extract temporary files.
Supported container formats:
- Disc images (ISO/GCM, WIA/RVZ, WBFS, CISO, NFS, GCZ, TGC)
- RARC/SZS and U8 (.arc)
Supported compression formats:
- Yaz0 (SZS)
- Yay0 (SZP)
- NLZSS (.lz)
Additionally, projects can utilize a new configuration key `object_base`:
```
object: orig/GZLE01/sys/main.dol
modules:
- object: orig/GZLE01/files/RELS.arc:rels/mmem/f_pc_profile_lst.rel
```
becomes
```
object_base: orig/GZLE01
object: sys/main.dol
modules:
- object: files/RELS.arc:mmem/f_pc_profile_lst.rel
```
When loading the objects, decomp-toolkit will automatically check the `object_base`
directory for any disc images. (They can be named anything, but must be in the folder
root) If one is found, all objects will be fetched from the disc image itself, rather
than having to extract the files manually.
While still a work in progress, two new `vfs` commands were added: `vfs ls` and `vfs cp`.
These commands are very barebones currently, but allow listing directory contents and
extracting files from decomp-toolkit's vfs representation:
```
❯ dtk vfs ls disc.rvz:
files
sys
❯ dtk vfs ls disc.rvz:sys
boot.bin
bi2.bin
apploader.bin
fst.bin
main.dol
❯ dtk vfs cp disc.rvz:sys/main.dol .
```
object stopped including the ELF null
symbol and null section in the respective
iterators. We relied on this behavior for
building certain vectors in order of
symbol index. Adjust this logic to
restore the correct behavior.
Prior to mwld GC 3.0a3, the linker doesn't support ELF .note sections
properly. With GC 2.7, it crashes if the section type is SHT_NOTE.
Use the same section type as .mwcats.* so the linker ignores it.
Example in splits.txt:
```
file1.cpp: order:0
...
file2.cpp: order:1
...
file3.cpp: order:2
...
```
This ensures that file2.cpp is always
anchored in between 1 and 3 when resolving
the final link order.
mwld writes empty code sections as
NULL type in the PLF, but sometimes
the original REL has the exec flag
set for these sections. Match the
original value.
Some games include a copy of MW
runtime.c in RELs. Easy to check.
Some games also include _savevr/
_restvr for AltiVec. This is weird
but we can handle it as well.
This allows more granular control over generated relocations.
Also optimizes relocation address validity checks,
leading to ~20% faster relocation analysis.
Config example:
```
block_relocations:
# Block any relocation pointing to this address.
- target: .data:0x80130140
# Block any relocation originating from this address.
- source: .text:0x80047160
# (optional) End address to make it a range.
end: .text:0x800471A8
add_relocations:
# Inserts or overwrites a relocation.
# From: `subi r3, r3, 0x7657`
# To: `li r3, mesWInsert-0x1@sda21`
- source: .text:0x800473F4
type: sda21
target: mesWInsert
addend: -1
```
Resolves#33Resolves#52
