Compare commits

..

41 Commits
v1.2.0 ... main

Author SHA1 Message Date
b56b399201 Support loading Wii Menu (BootStage) DOL files
Also adds support to elf2dol to extract the inner DOL
from a BootStage DOL.
2025-06-23 18:21:42 -06:00
8620099731 Version v1.6.2 2025-06-09 22:45:48 -06:00
ae00c35ec3 Better jump table error context 2025-06-09 22:45:48 -06:00
ba2589646e Relax prolog/epilog sequence checks
Some games (e.g. Excite Truck) have very aggressive float
scheduling that can create large gaps between prolog
instructions. This allows arbitrary instructions in the
sequence checks, provided they're not a branch and don't
touch r0/r1.

Resolves #105
2025-06-09 22:45:48 -06:00
cadmic
7bc0bc474d
Continue analyzing functions after unknown jumps (#106)
* Continue analyzing functions after unknown jumps
2025-06-09 22:45:21 -06:00
cadmic
d969819b78
Guess endianness of "erased" DWARF info (#104) 2025-06-09 22:44:39 -06:00
f4a67ee619 Version v1.6.1 2025-06-04 22:04:30 -06:00
Robin Avery
d92a892c2b
Relax string size requirement for auto symbols (#102) 2025-06-04 20:01:39 -07:00
cadmic
5e33fea49f
Allow specifying replacement bytes in dtk extab clean (#103)
* Allow specifying replacement bytes in dtk extab clean

* Simplify extab padding replacement

* Reword log message

* clippy has bad taste

* Don't specify revision number for cwextab

---------

Co-authored-by: Amber Brault <celestialamber1@gmail.com>
2025-06-04 20:01:05 -07:00
9cafb77d3f Add dtk extab clean & config.yml clean_extab
It was discovered that certain extab actions contain
uninitalized data from the compiler. This provides
a way to zero out uninitialized data in DOL or object
files. Usage: `dtk extab clean input.dol output.dol`

A `clean_extab` setting was added to config.yml, so
projects can link the cleaned objects and target the
cleaned DOL hash.
2025-06-01 20:23:07 -06:00
Dávid Balatoni
20e877c9ec
Some ProDG improvements (#101) 2025-06-01 16:43:13 -06:00
88d0e6b789 cargo clippy --fix 2025-06-01 16:42:00 -06:00
f212b35d28 Fix BSS symbol data check in add_padding_symbols 2025-06-01 16:40:40 -06:00
9c681557f5 Write ldscript_template path to ouput depfile 2025-05-30 19:18:46 -06:00
5505120148 Fix padding symbols on REL section boundaries
Regression introduced in 6819a8b
2025-05-11 22:15:02 -06:00
c1cbdd56d1 Version v1.5.0 2025-05-07 23:01:09 -06:00
robojumper
97302e5127
dtk dol apply: skip updating anonymous symbols by default (#97)
* Add --relaxed flag to dtk dol apply to skip updating anonymous symbols

* Invert --relaxed switch -> --full
2025-05-07 21:02:15 -06:00
Rainchus
18987ed330
add shiftjis as possible data type for symbols (#95)
* add shiftjis as possible data type for symbols

* usage of anyhow:bail! -> bail!

* revise output of sjis strings

* rename shiftjis internally, symbols now uses sjis instead of shiftjis

* remove sjis decoding error check as the output is a comment

* run cargo fmt
2025-04-16 23:59:21 -06:00
dbalatoni13
614d4f2efc
Add new options to rel make (#94) 2025-03-31 17:15:16 -06:00
ddd9dbb0ba clippy fixes 2025-03-21 16:14:44 -06:00
a064ddfd68 Quick & dirty ALF support in elf2dol 2025-03-21 16:13:04 -06:00
bb18a4b253 Auto-split and padding symbol fixes
A few issues were causing linker errors:
- Auto-splits could contain symbols that have a
higher alignment than the split itself. Detect
this and create a new auto-split at these symbols.
- The analyzer can miss objects in between other
objects if there are no direct relocations to them.
In these cases, non-zero data could just get totally
lost. Detect and create symbols for these.
2025-02-04 23:45:07 -07:00
7e15810af1 Version 1.4.0 2025-01-27 19:37:48 -07:00
51a7fbd85b Add WAD support to object_base
This allows WAD projects to use the auto-extraction
feature: decomp-toolkit will extract all `object`s
from a disc file or WAD file that exists in the
configured `object_base`.
2025-01-27 19:33:44 -07:00
a6c7001a52 Remove elf split command
This command was an early experiment and only worked
correctly on one game. The project system has long
superseded it. ELFs are better supported by
generating a project config via `elf config` and
using the DOL as the object target.

Closes #1
Closes #4
Closes #21
2025-01-27 19:11:40 -07:00
f270e9c53d clippy fix 2025-01-27 19:09:19 -07:00
b9642321a1 dol diff: Conditionally warn on symbol size mismatch
When a symbol's data matches but the size differs
due to padding bytes, print a warning and continue.

Resolves #35
2025-01-27 18:56:24 -07:00
e55ade10ec Ignore globalized symbols in dol diff
Resolves #61
2025-01-27 18:36:19 -07:00
6c3887c7e6 Improve split cyclic dependency errors
When a link order is unresolvable, decomp-toolkit
now prints out all of the TUs in a cycle, easing
debugging.

Example:
```
Cyclic dependency encountered while resolving link order: Dolphin/os/__start.c -> Dolphin/os/__ppc_eabi_init.cpp -> Dolphin/pad/PadClamp.c -> Dolphin/pad/pad.c
```

Thanks @simonlindholm for the toposort impl.
2025-01-27 18:06:25 -07:00
9a6348ec49 Set alignment for floats/doubles in analyzer
Resolves #60
2025-01-27 18:06:25 -07:00
589e59a620 Warn rather than fail on .note.split/.comment reading 2025-01-27 18:06:25 -07:00
6819a8bd5f Improve logic for generating padding symbols 2025-01-27 17:54:06 -07:00
a22e878258 Treat a branch to the start of a func as a tail call
Resolves #74
2025-01-27 17:54:06 -07:00
93000b28bd Check for conflicting splits with different file extensions
Resolves #81
2025-01-27 17:54:06 -07:00
0c6eff42cf Don't emit split alignment warnings for auto-splits
Resolves #84
2025-01-27 17:54:06 -07:00
04b60d319c vfs ls: Fix column sizing with Unicode chars 2025-01-27 17:54:06 -07:00
d1b35c4d18 clippy & cargo-deny fixes 2025-01-27 17:03:12 -07:00
NWPlayer123
8fb56c2fa4
Update Errors (#87)
Co-authored-by: NWPlayer123 <NWPlayer123@users.noreply.github.com>
2025-01-24 23:07:20 -07:00
91aa36c120 Clean up VFS error handling 2024-11-07 09:00:52 -07:00
9fc56d847f Add rename field to extract configuration
Allows renaming, for example, local statics from `test$1234`
to `test` for inclusion in the source function.
2024-11-07 08:44:24 -07:00
1cc38ad621 Add WAD support to VFS & wad commands 2024-11-07 08:43:20 -07:00
58 changed files with 2533 additions and 900 deletions

View File

@ -61,7 +61,7 @@ jobs:
continue-on-error: ${{ matrix.checks == 'advisories' }}
steps:
- uses: actions/checkout@v4
- uses: EmbarkStudios/cargo-deny-action@v1
- uses: EmbarkStudios/cargo-deny-action@v2
with:
command: check ${{ matrix.checks }}

93
Cargo.lock generated
View File

@ -249,7 +249,7 @@ dependencies = [
"encode_unicode",
"lazy_static",
"libc",
"unicode-width",
"unicode-width 0.1.14",
"windows-sys 0.52.0",
]
@ -339,28 +339,31 @@ checksum = "c2e06f9bce634a3c898eb1e5cb949ff63133cbb218af93cc9b38b31d6f3ea285"
[[package]]
name = "cwextab"
version = "1.0.2"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5aa7f13cc2fcb2bcfd3abc51bdbbf8f1fb729a69ed8c05ecbaa1a42197d1842"
checksum = "9dd95393b8cc20937e4757d9c22b89d016613e934c60dcb073bd8a5aade79fcf"
dependencies = [
"thiserror",
"thiserror 2.0.12",
]
[[package]]
name = "decomp-toolkit"
version = "1.2.0"
version = "1.7.0"
dependencies = [
"aes",
"anyhow",
"ar",
"argp",
"base16ct",
"base64",
"byteorder",
"cbc",
"crossterm",
"cwdemangle",
"cwextab",
"dyn-clone",
"enable-ansi-support",
"encoding_rs",
"filetime",
"fixedbitset 0.5.7",
"flagset",
@ -381,7 +384,6 @@ dependencies = [
"once_cell",
"orthrus-ncompress",
"owo-colors",
"petgraph",
"ppc750cl",
"rayon",
"regex",
@ -399,6 +401,7 @@ dependencies = [
"tracing-attributes",
"tracing-subscriber",
"typed-path",
"unicode-width 0.2.0",
"xxhash-rust",
"zerocopy",
]
@ -552,7 +555,7 @@ version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "14dbbfd5c71d70241ecf9e6f13737f7b5ce823821063188d7e46c41d371eebd5"
dependencies = [
"unicode-width",
"unicode-width 0.1.14",
]
[[package]]
@ -569,9 +572,9 @@ checksum = "d2fabcfbdc87f4758337ca535fb41a6d701b65693ce38287d856d1674551ec9b"
[[package]]
name = "hashbrown"
version = "0.15.0"
version = "0.15.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e087f84d4f86bf4b218b927129862374b72199ae7d8657835f1e89000eea4fb"
checksum = "bf151400ff0baff5465007dd2f3e717f3fe502074ca563069ce3a6629d07b289"
dependencies = [
"foldhash",
]
@ -629,7 +632,7 @@ dependencies = [
"instant",
"number_prefix",
"portable-atomic",
"unicode-width",
"unicode-width 0.1.14",
]
[[package]]
@ -870,7 +873,7 @@ dependencies = [
"miniz_oxide",
"rayon",
"sha1",
"thiserror",
"thiserror 1.0.64",
"zerocopy",
"zstd",
]
@ -943,7 +946,7 @@ dependencies = [
"proc-macro-crate",
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1107,7 +1110,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "479cf940fbbb3426c32c5d5176f62ad57549a0bb84773423ba8be9d089f5faba"
dependencies = [
"proc-macro2",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1121,9 +1124,9 @@ dependencies = [
[[package]]
name = "proc-macro2"
version = "1.0.88"
version = "1.0.95"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c3a7fc5db1e57d5a779a352c8cdb57b29aa4c40cc69c3a68a7fedc815fbf2f9"
checksum = "02b3e5e68a3a1a02aad3ec490a98007cbc13c37cbe84a3cd7b8e406d76e7f778"
dependencies = [
"unicode-ident",
]
@ -1155,7 +1158,7 @@ dependencies = [
"prost",
"prost-types",
"regex",
"syn 2.0.79",
"syn 2.0.101",
"tempfile",
]
@ -1169,7 +1172,7 @@ dependencies = [
"itertools",
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1371,7 +1374,7 @@ checksum = "243902eda00fad750862fc144cea25caca5e20d615af0a81bee94ca738f1df1f"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1382,7 +1385,7 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1405,7 +1408,7 @@ checksum = "6c64451ba24fc7a6a2d60fc75dd9c83c90903b19028d4eff35e88fc1e86564e9"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1524,7 +1527,7 @@ dependencies = [
"heck",
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1546,7 +1549,7 @@ dependencies = [
"proc-macro2",
"quote",
"rustversion",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1581,9 +1584,9 @@ dependencies = [
[[package]]
name = "syn"
version = "2.0.79"
version = "2.0.101"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89132cd0bf050864e1d38dc3bbc07a0eb8e7530af26344d3d2bbbef83499f590"
checksum = "8ce2b7fc941b3a24138a0a7cf8e858bfc6a992e7978a068a5c760deb0ed43caf"
dependencies = [
"proc-macro2",
"quote",
@ -1606,7 +1609,7 @@ dependencies = [
"serde",
"serde_derive",
"serde_json",
"thiserror",
"thiserror 1.0.64",
"walkdir",
]
@ -1629,7 +1632,16 @@ version = "1.0.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d50af8abc119fb8bb6dbabcfa89656f46f84aa0ac7688088608076ad2b459a84"
dependencies = [
"thiserror-impl",
"thiserror-impl 1.0.64",
]
[[package]]
name = "thiserror"
version = "2.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "567b8a2dae586314f7be2a752ec7474332959c6460e02bde30d702a66d488708"
dependencies = [
"thiserror-impl 2.0.12",
]
[[package]]
@ -1640,7 +1652,18 @@ checksum = "08904e7672f5eb876eaaf87e0ce17857500934f4981c4a0ab2b4aa98baac7fc3"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
name = "thiserror-impl"
version = "2.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f7cf42b4507d8ea322120659672cf1b9dbb93f8f2d4ecfd6e51350ff5b17a1d"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.101",
]
[[package]]
@ -1689,7 +1712,7 @@ checksum = "34704c8d6ebcbc939824180af020566b01a7c01f80641264eba0999f6c2b6be7"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1752,7 +1775,7 @@ dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]
@ -1788,6 +1811,12 @@ version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"
[[package]]
name = "unicode-width"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fc81956842c57dac11422a97c3b8195a1ff727f06e85c84ed2e8aa277c9a0fd"
[[package]]
name = "unsafe-libyaml"
version = "0.2.11"
@ -1844,7 +1873,7 @@ dependencies = [
"once_cell",
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
"wasm-bindgen-shared",
]
@ -1866,7 +1895,7 @@ checksum = "26c6ab57572f7a24a4985830b120de1594465e5d500f24afe89e16b4e833ef68"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
"wasm-bindgen-backend",
"wasm-bindgen-shared",
]
@ -2079,7 +2108,7 @@ checksum = "3ca22c4ad176b37bd81a565f66635bde3d654fe6832730c3e52e1018ae1655ee"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.79",
"syn 2.0.101",
]
[[package]]

View File

@ -3,7 +3,7 @@ name = "decomp-toolkit"
description = "Yet another GameCube/Wii decompilation toolkit."
authors = ["Luke Street <luke@street.dev>"]
license = "MIT OR Apache-2.0"
version = "1.2.0"
version = "1.7.0"
edition = "2021"
publish = false
repository = "https://github.com/encounter/decomp-toolkit"
@ -25,6 +25,8 @@ strip = "debuginfo"
codegen-units = 1
[dependencies]
encoding_rs = "0.8"
aes = "0.8"
anyhow = { version = "1.0", features = ["backtrace"] }
ar = { git = "https://github.com/bjorn3/rust-ar.git", branch = "write_symbol_table" }
argp = "0.3"
@ -32,9 +34,10 @@ base16ct = "0.2"
base64 = "0.22"
byteorder = "1.5"
typed-path = "0.9"
cbc = "0.1"
crossterm = "0.28"
cwdemangle = "1.0"
cwextab = "1.0"
cwextab = "1.1"
dyn-clone = "1.0"
enable-ansi-support = "0.2"
filetime = "0.2"
@ -58,7 +61,6 @@ object = { version = "0.36", features = ["read_core", "std", "elf", "write_std"]
once_cell = "1.20"
orthrus-ncompress = "0.2"
owo-colors = { version = "4.1", features = ["supports-colors"] }
petgraph = { version = "0.6", default-features = false }
ppc750cl = "0.3"
rayon = "1.10"
regex = "1.11"
@ -75,6 +77,7 @@ syntect = { version = "5.2", features = ["parsing", "regex-fancy", "dump-load"],
tracing = "0.1"
tracing-attributes = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
unicode-width = "0.2"
xxhash-rust = { version = "0.8", features = ["xxh3"] }
zerocopy = { version = "0.8", features = ["derive"] }

View File

@ -52,6 +52,9 @@ project structure and build system that uses decomp-toolkit under the hood.
- [yay0 compress](#yay0-compress)
- [yaz0 decompress](#yaz0-decompress)
- [yaz0 compress](#yaz0-compress)
- [wad info](#wad-info)
- [wad extract](#wad-extract)
- [wad verify](#wad-verify)
## Goals
@ -294,6 +297,8 @@ Dumps DWARF 1.1 information from an ELF file. (Does **not** support DWARF 2+)
```shell
$ dtk dwarf dump input.elf
# or, to include data that was stripped by MWLD
$ dtk dwarf dump input.elf --include-erased
```
### elf disasm
@ -474,6 +479,7 @@ Supported containers:
- Disc images (see [disc info](#disc-info) for supported formats)
- RARC archives (older .arc)
- U8 archives (newer .arc)
- WAD files (Wii VC)
Supported compression formats are handled transparently:
- Yay0 (SZP) / Yaz0 (SZS)
@ -562,3 +568,31 @@ $ dtk yaz0 compress input.bin -o output.bin.yaz0
# or, for batch processing
$ dtk yaz0 compress rels/* -o rels
```
### wad info
Prints information about a WAD file.
```shell
$ dtk wad info input.wad
```
### wad extract
> [!NOTE]
> [vfs cp](#vfs-cp) is more flexible and supports WAD files.
> This command is now equivalent to `dtk vfs cp input.wad: output_dir`
Extracts the contents of a WAD file.
```shell
$ dtk wad extract input.wad -o output_dir
```
### wad verify
Verifies the contents of a WAD file.
```shell
$ dtk wad verify input.wad
```

View File

@ -74,6 +74,7 @@ ignore = [
#{ id = "RUSTSEC-0000-0000", reason = "you can specify a reason the advisory is ignored" },
#"a-crate-that-is-yanked@0.1.1", # you can also ignore yanked crate versions if you wish
#{ crate = "a-crate-that-is-yanked@0.1.1", reason = "you can specify why you are ignoring the yanked crate" },
{ id = "RUSTSEC-2024-0384", reason = "unmaintained transient dependency, will be updated in next nod version" },
]
# If this is true, then cargo deny will use the git executable to fetch advisory database.
# If this is false, then it uses a built-in git library.

View File

@ -1,6 +1,6 @@
use std::{
cmp::min,
collections::BTreeMap,
collections::{BTreeMap, BTreeSet},
fmt::{Debug, Display, Formatter, UpperHex},
ops::{Add, AddAssign, BitAnd, Sub},
};
@ -191,7 +191,7 @@ impl AnalyzerState {
};
obj.add_symbol(
ObjSymbol {
name: format!("jumptable_{}", address_str),
name: format!("jumptable_{address_str}"),
address: addr.address as u64,
section: Some(addr.section),
size: size as u64,
@ -275,7 +275,7 @@ impl AnalyzerState {
let (section_index, _) = obj
.sections
.at_address(entry)
.context(format!("Entry point {:#010X} outside of any section", entry))?;
.context(format!("Entry point {entry:#010X} outside of any section"))?;
self.process_function_at(obj, SectionAddress::new(section_index, entry))?;
}
// Locate bounds for referenced functions until none are left
@ -530,7 +530,7 @@ pub fn locate_sda_bases(obj: &mut ObjInfo) -> Result<bool> {
let (section_index, _) = obj
.sections
.at_address(entry as u32)
.context(format!("Entry point {:#010X} outside of any section", entry))?;
.context(format!("Entry point {entry:#010X} outside of any section"))?;
let entry_addr = SectionAddress::new(section_index, entry as u32);
let mut executor = Executor::new(obj);
@ -572,6 +572,26 @@ pub fn locate_sda_bases(obj: &mut ObjInfo) -> Result<bool> {
Some((sda2_base, sda_base)) => {
obj.sda2_base = Some(sda2_base);
obj.sda_base = Some(sda_base);
obj.add_symbol(
ObjSymbol {
name: "_SDA2_BASE_".to_string(),
address: sda2_base as u64,
size_known: true,
flags: ObjSymbolFlagSet(ObjSymbolFlags::Global.into()),
..Default::default()
},
true,
)?;
obj.add_symbol(
ObjSymbol {
name: "_SDA_BASE_".to_string(),
address: sda_base as u64,
size_known: true,
flags: ObjSymbolFlagSet(ObjSymbolFlags::Global.into()),
..Default::default()
},
true,
)?;
Ok(true)
}
None => Ok(false),
@ -581,7 +601,7 @@ pub fn locate_sda_bases(obj: &mut ObjInfo) -> Result<bool> {
/// ProDG hardcodes .bss and .sbss section initialization in `entry`
/// This function locates the memset calls and returns a list of
/// (address, size) pairs for the .bss sections.
pub fn locate_bss_memsets(obj: &mut ObjInfo) -> Result<Vec<(u32, u32)>> {
pub fn locate_bss_memsets(obj: &ObjInfo) -> Result<Vec<(u32, u32)>> {
let mut bss_sections: Vec<(u32, u32)> = Vec::new();
let Some(entry) = obj.entry else {
return Ok(bss_sections);
@ -589,7 +609,7 @@ pub fn locate_bss_memsets(obj: &mut ObjInfo) -> Result<Vec<(u32, u32)>> {
let (section_index, _) = obj
.sections
.at_address(entry as u32)
.context(format!("Entry point {:#010X} outside of any section", entry))?;
.context(format!("Entry point {entry:#010X} outside of any section"))?;
let entry_addr = SectionAddress::new(section_index, entry as u32);
let mut executor = Executor::new(obj);
@ -632,3 +652,50 @@ pub fn locate_bss_memsets(obj: &mut ObjInfo) -> Result<Vec<(u32, u32)>> {
)?;
Ok(bss_sections)
}
/// Execute VM from specified entry point following inner-section branches and function calls,
/// noting all branch targets outside the current section.
pub fn locate_cross_section_branch_targets(
obj: &ObjInfo,
entry: SectionAddress,
) -> Result<BTreeSet<SectionAddress>> {
let mut branch_targets = BTreeSet::<SectionAddress>::new();
let mut executor = Executor::new(obj);
executor.push(entry, VM::new(), false);
executor.run(
obj,
|ExecCbData { executor, vm, result, ins_addr, section: _, ins: _, block_start: _ }| {
match result {
StepResult::Continue | StepResult::LoadStore { .. } => {
Ok(ExecCbResult::<()>::Continue)
}
StepResult::Illegal => bail!("Illegal instruction @ {}", ins_addr),
StepResult::Jump(target) => {
if let BranchTarget::Address(RelocationTarget::Address(addr)) = target {
if addr.section == entry.section {
executor.push(addr, vm.clone_all(), true);
} else {
branch_targets.insert(addr);
}
}
Ok(ExecCbResult::EndBlock)
}
StepResult::Branch(branches) => {
for branch in branches {
if let BranchTarget::Address(RelocationTarget::Address(addr)) =
branch.target
{
if addr.section == entry.section {
executor.push(addr, branch.vm, true);
} else {
branch_targets.insert(addr);
}
}
}
Ok(ExecCbResult::Continue)
}
}
},
)?;
Ok(branch_targets)
}

View File

@ -183,8 +183,7 @@ fn get_jump_table_entries(
let (section_index, _) =
obj.sections.at_address(entry_addr).with_context(|| {
format!(
"Invalid jump table entry {:#010X} at {:#010X}",
entry_addr, cur_addr
"Invalid jump table entry {entry_addr:#010X} at {cur_addr:#010X}"
)
})?;
entries.push(SectionAddress::new(section_index, entry_addr));
@ -245,7 +244,9 @@ pub fn uniq_jump_table_entries(
return Ok((BTreeSet::new(), 0));
}
let (entries, size) =
get_jump_table_entries(obj, addr, size, from, function_start, function_end)?;
get_jump_table_entries(obj, addr, size, from, function_start, function_end).with_context(
|| format!("While fetching jump table entries starting at {addr:#010X}"),
)?;
Ok((BTreeSet::from_iter(entries.iter().cloned()), size))
}

View File

@ -2,7 +2,7 @@ use anyhow::Result;
use crate::{
obj::{ObjDataKind, ObjInfo, ObjSectionKind, ObjSymbolKind, SymbolIndex},
util::split::is_linker_generated_label,
util::{config::is_auto_symbol, split::is_linker_generated_label},
};
pub fn detect_objects(obj: &mut ObjInfo) -> Result<()> {
@ -134,7 +134,9 @@ pub fn detect_strings(obj: &mut ObjInfo) -> Result<()> {
StringResult::None => {}
StringResult::String { length, terminated } => {
let size = if terminated { length + 1 } else { length };
if !symbol.size_known || symbol.size == size as u64 {
if symbol.size == size as u64
|| (is_auto_symbol(symbol) && symbol.size > size as u64)
{
let str = String::from_utf8_lossy(&data[..length]);
log::debug!("Found string '{}' @ {}", str, symbol.name);
symbols_set.push((symbol_idx, ObjDataKind::String, size));
@ -142,7 +144,9 @@ pub fn detect_strings(obj: &mut ObjInfo) -> Result<()> {
}
StringResult::WString { length, str } => {
let size = length + 2;
if !symbol.size_known || symbol.size == size as u64 {
if symbol.size == size as u64
|| (is_auto_symbol(symbol) && symbol.size > size as u64)
{
log::debug!("Found wide string '{}' @ {}", str, symbol.name);
symbols_set.push((symbol_idx, ObjDataKind::String16, size));
}

View File

@ -31,9 +31,8 @@ impl AnalysisPass for FindTRKInterruptVectorTable {
Ok(ret) => ret,
Err(_) => continue,
};
if data.starts_with(TRK_TABLE_HEADER.as_bytes())
&& data[TRK_TABLE_HEADER.as_bytes().len()] == 0
{
let trk_table_bytes = TRK_TABLE_HEADER.as_bytes();
if data.starts_with(trk_table_bytes) && data[trk_table_bytes.len()] == 0 {
log::debug!("Found gTRKInterruptVectorTable @ {:#010X}", start);
state.known_symbols.entry(start).or_default().push(ObjSymbol {
name: "gTRKInterruptVectorTable".to_string(),
@ -102,7 +101,7 @@ impl AnalysisPass for FindSaveRestSleds {
for i in reg_start..reg_end {
let addr = start + (i - reg_start) * step_size;
state.known_symbols.entry(addr).or_default().push(ObjSymbol {
name: format!("{}{}", label, i),
name: format!("{label}{i}"),
address: addr.address as u64,
section: Some(start.section),
size_known: true,

View File

@ -45,6 +45,17 @@ type BlockRange = Range<SectionAddress>;
type InsCheck = dyn Fn(Ins) -> bool;
/// Stop searching for prologue/epilogue sequences if the next instruction
/// is a branch or uses r0 or r1.
fn is_end_of_seq(next: &Ins) -> bool {
next.is_branch()
|| next
.defs()
.iter()
.chain(next.uses().iter())
.any(|a| matches!(a, ppc750cl::Argument::GPR(ppc750cl::GPR(0 | 1))))
}
#[inline(always)]
fn check_sequence(
section: &ObjSection,
@ -52,29 +63,26 @@ fn check_sequence(
ins: Option<Ins>,
sequence: &[(&InsCheck, &InsCheck)],
) -> Result<bool> {
let mut found = false;
let ins = ins
.or_else(|| disassemble(section, addr.address))
.with_context(|| format!("Failed to disassemble instruction at {addr:#010X}"))?;
for &(first, second) in sequence {
let Some(ins) = ins.or_else(|| disassemble(section, addr.address)) else {
continue;
};
if !first(ins) {
continue;
}
let Some(next) = disassemble(section, addr.address + 4) else {
continue;
};
if second(next)
// Also check the following instruction, in case the scheduler
// put something in between.
|| (!next.is_branch()
&& matches!(disassemble(section, addr.address + 8), Some(ins) if second(ins)))
{
found = true;
break;
let mut current_addr = addr.address + 4;
while let Some(next) = disassemble(section, current_addr) {
if second(next) {
return Ok(true);
}
if is_end_of_seq(&next) {
// If we hit a branch or an instruction that uses r0 or r1, stop searching.
break;
}
current_addr += 4;
}
}
Ok(found)
Ok(false)
}
fn check_prologue_sequence(
@ -89,15 +97,19 @@ fn check_prologue_sequence(
}
#[inline(always)]
fn is_stwu(ins: Ins) -> bool {
// stwu r1, d(r1)
ins.op == Opcode::Stwu && ins.field_rs() == 1 && ins.field_ra() == 1
// stwu[x] r1, d(r1)
matches!(ins.op, Opcode::Stwu | Opcode::Stwux) && ins.field_rs() == 1 && ins.field_ra() == 1
}
#[inline(always)]
fn is_stw(ins: Ins) -> bool {
// stw r0, d(r1)
ins.op == Opcode::Stw && ins.field_rs() == 0 && ins.field_ra() == 1
}
check_sequence(section, addr, ins, &[(&is_stwu, &is_mflr), (&is_mflr, &is_stw)])
check_sequence(section, addr, ins, &[
(&is_stwu, &is_mflr),
(&is_mflr, &is_stw),
(&is_mflr, &is_stwu),
])
}
impl FunctionSlices {
@ -148,8 +160,29 @@ impl FunctionSlices {
}
if check_prologue_sequence(section, addr, Some(ins))? {
if let Some(prologue) = self.prologue {
if prologue != addr && prologue != addr - 4 {
bail!("Found duplicate prologue: {:#010X} and {:#010X}", prologue, addr)
let invalid_seq = if prologue == addr {
false
} else if prologue > addr {
true
} else {
// Check if any instructions between the prologue and this address
// are branches or use r0 or r1.
let mut current_addr = prologue.address + 4;
loop {
if current_addr == addr.address {
break false;
}
let next = disassemble(section, current_addr).with_context(|| {
format!("Failed to disassemble {current_addr:#010X}")
})?;
if is_end_of_seq(&next) {
break true;
}
current_addr += 4;
}
};
if invalid_seq {
bail!("Found multiple functions inside a symbol: {:#010X} and {:#010X}. Check symbols.txt?", prologue, addr)
}
} else {
self.prologue = Some(addr);
@ -180,7 +213,11 @@ impl FunctionSlices {
ins.op == Opcode::Or && ins.field_rd() == 1
}
if check_sequence(section, addr, Some(ins), &[(&is_mtlr, &is_addi), (&is_or, &is_mtlr)])? {
if check_sequence(section, addr, Some(ins), &[
(&is_mtlr, &is_addi),
(&is_mtlr, &is_or),
(&is_or, &is_mtlr),
])? {
if let Some(epilogue) = self.epilogue {
if epilogue != addr {
bail!("Found duplicate epilogue: {:#010X} and {:#010X}", epilogue, addr)
@ -227,7 +264,7 @@ impl FunctionSlices {
})?;
}
self.check_epilogue(section, ins_addr, ins)
.with_context(|| format!("While processing {:#010X}: {:#?}", function_start, self))?;
.with_context(|| format!("While processing {function_start:#010X}: {self:#?}"))?;
if !self.has_conditional_blr && is_conditional_blr(ins) {
self.has_conditional_blr = true;
}
@ -340,7 +377,14 @@ impl FunctionSlices {
function_end.or_else(|| self.end()),
)?;
log::debug!("-> size {}: {:?}", size, entries);
if (entries.contains(&next_address) || self.blocks.contains_key(&next_address))
let max_block = self
.blocks
.keys()
.next_back()
.copied()
.unwrap_or(next_address)
.max(next_address);
if entries.iter().any(|&addr| addr > function_start && addr <= max_block)
&& !entries.iter().any(|&addr| {
self.is_known_function(known_functions, addr)
.is_some_and(|fn_addr| fn_addr != function_start)
@ -703,7 +747,7 @@ impl FunctionSlices {
}
}
// If we discovered a function prologue, known tail call.
if slices.prologue.is_some() {
if slices.prologue.is_some() || slices.has_r1_load {
log::trace!("Prologue discovered; known tail call: {:#010X}", addr);
return TailCallResult::Is;
}

View File

@ -269,7 +269,8 @@ impl Tracker {
possible_missed_branches: &mut BTreeMap<SectionAddress, Box<VM>>,
) -> Result<ExecCbResult<()>> {
let ExecCbData { executor, vm, result, ins_addr, section: _, ins, block_start: _ } = data;
let is_function_addr = |addr: SectionAddress| addr >= function_start && addr < function_end;
// Using > instead of >= to treat a branch to the beginning of the function as a tail call
let is_function_addr = |addr: SectionAddress| addr > function_start && addr < function_end;
let _span = debug_span!("ins", addr = %ins_addr, op = ?ins.op).entered();
match result {
@ -416,9 +417,13 @@ impl Tracker {
Ok(ExecCbResult::Continue)
}
StepResult::Jump(target) => match target {
BranchTarget::Return => Ok(ExecCbResult::EndBlock),
BranchTarget::Unknown
| BranchTarget::Return
| BranchTarget::JumpTable { address: RelocationTarget::External, .. } => {
let next_addr = ins_addr + 4;
if next_addr < function_end {
possible_missed_branches.insert(ins_addr + 4, vm.clone_all());
}
Ok(ExecCbResult::EndBlock)
}
BranchTarget::Address(addr) => {
@ -575,7 +580,7 @@ impl Tracker {
let relocation_target = relocation_target_for(obj, from, None).ok().flatten();
if !matches!(relocation_target, None | Some(RelocationTarget::External)) {
// VM should have already handled this
panic!("Relocation already exists for {:#010X} (from {:#010X})", addr, from);
panic!("Relocation already exists for {addr:#010X} (from {from:#010X})");
}
}
// Remainder of this function is for executable objects only
@ -667,7 +672,7 @@ impl Tracker {
0
};
let new_name =
if module_id == 0 { name.to_string() } else { format!("{}:{}", name, module_id) };
if module_id == 0 { name.to_string() } else { format!("{name}:{module_id}") };
log::debug!("Renaming {} to {}", section.name, new_name);
section.name = new_name;
}
@ -733,55 +738,65 @@ impl Tracker {
);
}
}
let data_kind = self
let (data_kind, inferred_alignment) = self
.data_types
.get(&target)
.map(|dt| match dt {
DataKind::Unknown => ObjDataKind::Unknown,
DataKind::Word => ObjDataKind::Byte4,
DataKind::Half => ObjDataKind::Byte2,
DataKind::Byte => ObjDataKind::Byte,
DataKind::Float => ObjDataKind::Float,
DataKind::Double => ObjDataKind::Double,
DataKind::Unknown => (ObjDataKind::Unknown, None),
DataKind::Word => (ObjDataKind::Byte4, None),
DataKind::Half => (ObjDataKind::Byte2, None),
DataKind::Byte => (ObjDataKind::Byte, None),
DataKind::Float => (ObjDataKind::Float, Some(4)),
DataKind::Double => (ObjDataKind::Double, Some(8)),
})
.unwrap_or_default();
let (target_symbol, addend) = if let Some(symbol) =
self.special_symbol(obj, target.address, reloc_kind)
{
(symbol, 0)
} else if let Some((symbol_idx, symbol)) =
obj.symbols.for_relocation(target, reloc_kind)?
{
let symbol_address = symbol.address;
// TODO meh
if data_kind != ObjDataKind::Unknown
&& symbol.data_kind == ObjDataKind::Unknown
&& symbol_address as u32 == target.address
let (target_symbol, addend) =
if let Some(symbol) = self.special_symbol(obj, target.address, reloc_kind) {
(symbol, 0)
} else if let Some((symbol_idx, symbol)) =
obj.symbols.for_relocation(target, reloc_kind)?
{
obj.symbols.replace(symbol_idx, ObjSymbol { data_kind, ..symbol.clone() })?;
}
(symbol_idx, target.address as i64 - symbol_address as i64)
} else {
// Create a new label
let name = if obj.module_id == 0 {
format!("lbl_{:08X}", target.address)
let symbol_address = symbol.address;
if symbol_address as u32 == target.address
&& ((data_kind != ObjDataKind::Unknown
&& symbol.data_kind == ObjDataKind::Unknown)
|| (symbol.align.is_none() && inferred_alignment.is_some()))
{
let mut new_symbol = symbol.clone();
if symbol.data_kind == ObjDataKind::Unknown {
new_symbol.data_kind = data_kind;
}
if symbol.align.is_none() {
if let Some(inferred_alignment) = inferred_alignment {
if symbol_address as u32 % inferred_alignment == 0 {
new_symbol.align = Some(inferred_alignment);
}
}
}
obj.symbols.replace(symbol_idx, new_symbol)?;
}
(symbol_idx, target.address as i64 - symbol_address as i64)
} else {
format!(
"lbl_{}_{}_{:X}",
obj.module_id,
obj.sections[target.section].name.trim_start_matches('.'),
target.address
)
// Create a new label
let name = if obj.module_id == 0 {
format!("lbl_{:08X}", target.address)
} else {
format!(
"lbl_{}_{}_{:X}",
obj.module_id,
obj.sections[target.section].name.trim_start_matches('.'),
target.address
)
};
let symbol_idx = obj.symbols.add_direct(ObjSymbol {
name,
address: target.address as u64,
section: Some(target.section),
data_kind,
..Default::default()
})?;
(symbol_idx, 0)
};
let symbol_idx = obj.symbols.add_direct(ObjSymbol {
name,
address: target.address as u64,
section: Some(target.section),
data_kind,
..Default::default()
})?;
(symbol_idx, 0)
};
let reloc = ObjReloc { kind: reloc_kind, target_symbol, addend, module: None };
let section = &mut obj.sections[addr.section];
if replace {

View File

@ -127,16 +127,16 @@ fn extract(args: ExtractArgs) -> Result<()> {
}
std::fs::create_dir_all(&out_dir)?;
if !args.quiet {
println!("Extracting {} to {}", path, out_dir);
println!("Extracting {path} to {out_dir}");
}
let mut file = open_file(path, false)?;
let mut archive = ar::Archive::new(file.map()?);
while let Some(entry) = archive.next_entry() {
let mut entry = entry.with_context(|| format!("Processing entry in {}", path))?;
let mut entry = entry.with_context(|| format!("Processing entry in {path}"))?;
let file_name = std::str::from_utf8(entry.header().identifier())?;
if !args.quiet && args.verbose {
println!("\t{}", file_name);
println!("\t{file_name}");
}
let mut file_path = out_dir.clone();
for segment in file_name.split(&['/', '\\']) {
@ -146,7 +146,7 @@ fn extract(args: ExtractArgs) -> Result<()> {
std::fs::create_dir_all(parent)?;
}
let mut file = File::create(&file_path)
.with_context(|| format!("Failed to create file {}", file_path))?;
.with_context(|| format!("Failed to create file {file_path}"))?;
std::io::copy(&mut entry, &mut file)?;
file.flush()?;
@ -154,7 +154,7 @@ fn extract(args: ExtractArgs) -> Result<()> {
}
}
if !args.quiet {
println!("Extracted {} files", num_files);
println!("Extracted {num_files} files");
}
Ok(())
}

View File

@ -47,6 +47,7 @@ use crate::{
diff::{calc_diff_ranges, print_diff, process_code},
dol::process_dol,
elf::{process_elf, write_elf},
extab::clean_extab,
file::{
buf_copy_with_hash, buf_writer, check_hash_str, touch, verify_hash, FileIterator,
FileReadInfo,
@ -59,7 +60,7 @@ use crate::{
split::{is_linker_generated_object, split_obj, update_splits},
IntoCow, ToCow,
},
vfs::{open_file, open_file_with_fs, open_fs, ArchiveKind, Vfs, VfsFile},
vfs::{detect, open_file, open_file_with_fs, open_fs, ArchiveKind, FileFormat, Vfs, VfsFile},
};
#[derive(FromArgs, PartialEq, Debug)]
@ -132,6 +133,9 @@ pub struct ApplyArgs {
#[argp(positional, from_str_fn(native_path))]
/// linked ELF
elf_file: Utf8NativePathBuf,
#[argp(switch)]
/// always update anonymous local symbol names, even if they are similar
full: bool,
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
@ -290,12 +294,17 @@ pub struct ModuleConfig {
pub block_relocations: Vec<BlockRelocationConfig>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub add_relocations: Vec<AddRelocationConfig>,
/// Process exception tables and zero out uninitialized data.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub clean_extab: Option<bool>,
}
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
pub struct ExtractConfig {
/// The name of the symbol to extract.
pub symbol: String,
/// Optionally rename the output symbol. (e.g. symbol$1234 -> symbol)
pub rename: Option<String>,
/// If specified, the symbol's data will be extracted to the given file.
/// Path is relative to `out_dir/bin`.
#[serde(with = "unix_path_serde_option", default, skip_serializing_if = "Option::is_none")]
@ -385,6 +394,7 @@ pub struct OutputModule {
#[derive(Serialize, Deserialize, Debug, Clone, Default)]
pub struct OutputExtract {
pub symbol: String,
pub rename: Option<String>,
#[serde(with = "unix_path_serde_option")]
pub binary: Option<Utf8UnixPathBuf>,
#[serde(with = "unix_path_serde_option")]
@ -528,9 +538,11 @@ pub fn info(args: InfoArgs) -> Result<()> {
apply_selfile(&mut obj, file.map()?)?;
}
println!("{}:", obj.name);
if !obj.name.is_empty() {
println!("{}:", obj.name);
}
if let Some(entry) = obj.entry {
println!("Entry point: {:#010X}", entry);
println!("Entry point: {entry:#010X}");
}
println!("\nSections:");
println!("\t{: >10} | {: <10} | {: <10} | {: <10}", "Name", "Address", "Size", "File Off");
@ -572,6 +584,7 @@ struct ModuleInfo<'a> {
config: &'a ModuleConfig,
symbols_cache: Option<FileReadInfo>,
splits_cache: Option<FileReadInfo>,
dep: Vec<Utf8NativePathBuf>,
}
type ModuleMapByName<'a> = BTreeMap<String, ModuleInfo<'a>>;
@ -811,17 +824,29 @@ struct AnalyzeResult {
splits_cache: Option<FileReadInfo>,
}
fn load_analyze_dol(config: &ProjectConfig, object_base: &ObjectBase) -> Result<AnalyzeResult> {
let object_path = object_base.join(&config.base.object);
fn load_dol_module(
config: &ModuleConfig,
object_base: &ObjectBase,
) -> Result<(ObjInfo, Utf8NativePathBuf)> {
let object_path = object_base.join(&config.object);
log::debug!("Loading {}", object_path);
let mut obj = {
let mut file = object_base.open(&config.base.object)?;
let mut file = object_base.open(&config.object)?;
let data = file.map()?;
if let Some(hash_str) = &config.base.hash {
if let Some(hash_str) = &config.hash {
verify_hash(data, hash_str)?;
}
process_dol(data, config.base.name())?
process_dol(data, config.name())?
};
if config.clean_extab.unwrap_or(false) {
log::debug!("Cleaning extab for {}", config.name());
clean_extab(&mut obj, std::iter::empty())?;
}
Ok((obj, object_path))
}
fn load_analyze_dol(config: &ProjectConfig, object_base: &ObjectBase) -> Result<AnalyzeResult> {
let (mut obj, object_path) = load_dol_module(&config.base, object_base)?;
let mut dep = vec![object_path];
if let Some(comment_version) = config.mw_comment_version {
@ -948,7 +973,7 @@ fn split_write_obj(
DirBuilder::new()
.recursive(true)
.create(out_dir)
.with_context(|| format!("Failed to create out dir '{}'", out_dir))?;
.with_context(|| format!("Failed to create out dir '{out_dir}'"))?;
let obj_dir = out_dir.join("obj");
let entry = if module.obj.kind == ObjKind::Executable {
module.obj.entry.and_then(|e| {
@ -968,9 +993,19 @@ fn split_write_obj(
entry,
extract: Vec::with_capacity(module.config.extract.len()),
};
let mut object_paths = BTreeMap::new();
for (unit, split_obj) in module.obj.link_order.iter().zip(&split_objs) {
let out_obj = write_elf(split_obj, config.export_all)?;
let out_path = obj_dir.join(obj_path_for_unit(&unit.name));
let obj_path = obj_path_for_unit(&unit.name);
let out_path = obj_dir.join(&obj_path);
if let Some(existing) = object_paths.insert(obj_path, unit) {
bail!(
"Duplicate object path: {} and {} both resolve to {}",
existing.name,
unit.name,
out_path,
);
}
out_config.units.push(OutputUnit {
object: out_path.with_unix_encoding(),
name: unit.name.clone(),
@ -1014,7 +1049,8 @@ fn split_write_obj(
if header_kind != HeaderKind::None {
if let Some(header) = &extract.header {
let header_string = bin2c(symbol, section, data, header_kind);
let header_string =
bin2c(symbol, section, data, header_kind, extract.rename.as_deref());
let out_path = base_dir.join("include").join(header.with_encoding());
if let Some(parent) = out_path.parent() {
DirBuilder::new().recursive(true).create(parent)?;
@ -1026,6 +1062,7 @@ fn split_write_obj(
// Copy to output config
out_config.extract.push(OutputExtract {
symbol: symbol.name.clone(),
rename: extract.rename.clone(),
binary: extract.binary.clone(),
header: extract.header.clone(),
header_type: header_kind.to_string(),
@ -1037,9 +1074,10 @@ fn split_write_obj(
// Generate ldscript.lcf
let ldscript_template = if let Some(template_path) = &module.config.ldscript_template {
let template_path = template_path.with_encoding();
Some(fs::read_to_string(&template_path).with_context(|| {
format!("Failed to read linker script template '{}'", template_path)
})?)
let template = fs::read_to_string(&template_path)
.with_context(|| format!("Failed to read linker script template '{template_path}'"))?;
module.dep.push(template_path);
Some(template)
} else {
None
};
@ -1055,8 +1093,7 @@ fn split_write_obj(
let out_path = asm_dir.join(asm_path_for_unit(&unit.name));
let mut w = buf_writer(&out_path)?;
write_asm(&mut w, split_obj)
.with_context(|| format!("Failed to write {}", out_path))?;
write_asm(&mut w, split_obj).with_context(|| format!("Failed to write {out_path}"))?;
w.flush()?;
}
}
@ -1073,7 +1110,7 @@ fn write_if_changed(path: &Utf8NativePath, contents: &[u8]) -> Result<()> {
return Ok(());
}
}
fs::write(path, contents).with_context(|| format!("Failed to write file '{}'", path))?;
fs::write(path, contents).with_context(|| format!("Failed to write file '{path}'"))?;
Ok(())
}
@ -1227,6 +1264,7 @@ fn split(args: SplitArgs) -> Result<()> {
config: &config.base,
symbols_cache: result.symbols_cache,
splits_cache: result.splits_cache,
dep: Default::default(),
}
};
let mut function_count = dol.obj.symbols.by_kind(ObjSymbolKind::Function).count();
@ -1241,6 +1279,7 @@ fn split(args: SplitArgs) -> Result<()> {
config: &config.modules[idx],
symbols_cache: result.symbols_cache,
splits_cache: result.splits_cache,
dep: Default::default(),
}),
Entry::Occupied(_) => bail!("Duplicate module name {}", result.obj.name),
};
@ -1422,6 +1461,10 @@ fn split(args: SplitArgs) -> Result<()> {
}
// Write dep file
dep.extend(dol.dep);
for module in modules.into_values() {
dep.extend(module.dep);
}
{
let dep_path = args.out_dir.join("dep");
let mut dep_file = buf_writer(&dep_path)?;
@ -1591,42 +1634,49 @@ fn validate(obj: &ObjInfo, elf_file: &Utf8NativePath, state: &AnalyzerState) ->
/// Check if two symbols' names match, allowing for differences in compiler-generated names,
/// like @1234 and @5678, or init$1234 and init$5678.
fn symbol_name_fuzzy_eq(a: &ObjSymbol, b: &ObjSymbol) -> bool {
if a.name == b.name {
fn symbol_name_fuzzy_eq(a: &str, b: &str) -> bool {
if a == b {
return true;
}
// Match e.g. @1234 and @5678
if a.name.starts_with('@') && b.name.starts_with('@') {
if a.starts_with('@') && b.starts_with('@') {
return true;
}
// Match e.g. init$1234 and init$5678
if let (Some(a_dollar), Some(b_dollar)) = (a.name.rfind('$'), b.name.rfind('$')) {
if a.name[..a_dollar] == b.name[..b_dollar] {
if let (Some(a_dollar), Some(b_dollar)) = (a.rfind('$'), b.rfind('$')) {
if a[..a_dollar] == b[..b_dollar] {
if let (Ok(_), Ok(_)) =
(a.name[a_dollar + 1..].parse::<u32>(), b.name[b_dollar + 1..].parse::<u32>())
(a[a_dollar + 1..].parse::<u32>(), b[b_dollar + 1..].parse::<u32>())
{
return true;
}
}
}
// Match e.g. symbol and symbol_80123456 (globalized symbol)
if let Some(a_under) = a.rfind('_') {
if &a[..a_under] == b && is_hex(&a[a_under + 1..]) {
return true;
}
}
if let Some(b_under) = b.rfind('_') {
if a == &b[..b_under] && is_hex(&b[b_under + 1..]) {
return true;
}
}
false
}
fn is_hex(s: &str) -> bool {
s.chars().all(|c| c.is_ascii_digit() || matches!(c, 'a'..='f' | 'A'..='F'))
}
fn diff(args: DiffArgs) -> Result<()> {
log::info!("Loading {}", args.config);
let mut config_file = open_file(&args.config, true)?;
let config: ProjectConfig = serde_yaml::from_reader(config_file.as_mut())?;
let object_base = find_object_base(&config)?;
log::info!("Loading {}", object_base.join(&config.base.object));
let mut obj = {
let mut file = object_base.open(&config.base.object)?;
let data = file.map()?;
if let Some(hash_str) = &config.base.hash {
verify_hash(data, hash_str)?;
}
process_dol(data, config.base.name())?
};
let (mut obj, _object_path) = load_dol_module(&config.base, &object_base)?;
if let Some(symbols_path) = &config.base.symbols {
apply_symbols_file(&symbols_path.with_encoding(), &mut obj)?;
@ -1656,7 +1706,7 @@ fn diff(args: DiffArgs) -> Result<()> {
});
let mut found = false;
if let Some((_, linked_sym)) = linked_sym {
if symbol_name_fuzzy_eq(linked_sym, orig_sym) {
if symbol_name_fuzzy_eq(&linked_sym.name, &orig_sym.name) {
if linked_sym.size != orig_sym.size &&
// TODO validate common symbol sizes
// (need to account for inflation bug)
@ -1672,7 +1722,7 @@ fn diff(args: DiffArgs) -> Result<()> {
);
}
found = true;
} else if linked_sym.kind == orig_sym.kind && linked_sym.size == orig_sym.size {
} else if linked_sym.kind == orig_sym.kind {
// Fuzzy match
let orig_data = orig_section.data_range(
orig_sym.address as u32,
@ -1682,7 +1732,12 @@ fn diff(args: DiffArgs) -> Result<()> {
linked_sym.address as u32,
linked_sym.address as u32 + linked_sym.size as u32,
)?;
if orig_data == linked_data {
let len = orig_data.len().min(linked_data.len());
if orig_data[..len] == linked_data[..len]
// Ignore padding differences
&& orig_data[len..].iter().all(|&b| b == 0)
&& linked_data[len..].iter().all(|&b| b == 0)
{
found = true;
}
}
@ -1746,7 +1801,11 @@ fn diff(args: DiffArgs) -> Result<()> {
linked_sym.address as u32,
linked_sym.address as u32 + linked_sym.size as u32,
)?;
if orig_data != linked_data {
let len = orig_data.len().min(linked_data.len());
if orig_data[..len] != linked_data[..len]
|| orig_data[len..].iter().any(|&b| b != 0)
|| linked_data[len..].iter().any(|&b| b != 0)
{
log::error!(
"Data mismatch for {} (type {:?}, size {:#X}) at {:#010X}",
orig_sym.name,
@ -1786,6 +1845,15 @@ fn diff(args: DiffArgs) -> Result<()> {
}
std::process::exit(1);
} else if orig_data.len() != linked_data.len() {
log::error!(
"Size mismatch for {} (type {:?}) at {:#010X}: Expected {:#X}, found {:#X}",
orig_sym.name,
orig_sym.kind,
orig_sym.address,
orig_data.len(),
linked_data.len()
);
}
}
@ -1793,21 +1861,38 @@ fn diff(args: DiffArgs) -> Result<()> {
Ok(())
}
fn are_local_anonymous_names_similar<'a>(left: &'a ObjSymbol, right: &'a ObjSymbol) -> bool {
if left.flags.scope() != ObjSymbolScope::Local || right.flags.scope() != ObjSymbolScope::Local {
return false;
}
let is_at_symbol =
|name: &str| name.starts_with('@') && name[1..].chars().all(|c| c.is_numeric());
if is_at_symbol(&left.name) && is_at_symbol(&right.name) {
// consider e.g. @8280 -> @8536 equal
return true;
}
let split_dollar_symbol = |name: &'a str| -> Option<&'a str> {
name.rsplit_once('$')
.and_then(|(prefix, suffix)| suffix.chars().all(|c| c.is_numeric()).then_some(prefix))
};
// consider e.g. __arraydtor$3926 -> __arraydtor$7669 equal
match (split_dollar_symbol(&left.name), split_dollar_symbol(&right.name)) {
(Some(left_prefix), Some(right_prefix)) => left_prefix == right_prefix,
_ => false,
}
}
fn apply(args: ApplyArgs) -> Result<()> {
log::info!("Loading {}", args.config);
let mut config_file = open_file(&args.config, true)?;
let config: ProjectConfig = serde_yaml::from_reader(config_file.as_mut())?;
let object_base = find_object_base(&config)?;
log::info!("Loading {}", object_base.join(&config.base.object));
let mut obj = {
let mut file = object_base.open(&config.base.object)?;
let data = file.map()?;
if let Some(hash_str) = &config.base.hash {
verify_hash(data, hash_str)?;
}
process_dol(data, config.base.name())?
};
let (mut obj, _object_path) = load_dol_module(&config.base, &object_base)?;
let Some(symbols_path) = &config.base.symbols else {
bail!("No symbols file specified in config");
@ -1843,7 +1928,9 @@ fn apply(args: ApplyArgs) -> Result<()> {
let mut updated_sym = orig_sym.clone();
let is_globalized = linked_sym.name.ends_with(&format!("_{:08X}", linked_sym.address));
if (is_globalized && !linked_sym.name.starts_with(&orig_sym.name))
|| (!is_globalized && linked_sym.name != orig_sym.name)
|| (!is_globalized
&& (linked_sym.name != orig_sym.name
&& (args.full || !are_local_anonymous_names_similar(linked_sym, orig_sym))))
{
log::info!(
"Changing name of {} (type {:?}) to {}",
@ -2079,7 +2166,7 @@ impl ObjectBase {
}
base.join(path.with_encoding())
}
ObjectBase::Vfs(base, _) => Utf8NativePathBuf::from(format!("{}:{}", base, path)),
ObjectBase::Vfs(base, _) => Utf8NativePathBuf::from(format!("{base}:{path}")),
}
}
@ -2096,7 +2183,7 @@ impl ObjectBase {
}
ObjectBase::Vfs(vfs_path, vfs) => {
open_file_with_fs(vfs.clone(), &path.with_encoding(), true)
.with_context(|| format!("Using disc image {}", vfs_path))
.with_context(|| format!("Using disc image {vfs_path}"))
}
}
}
@ -2114,30 +2201,36 @@ pub fn find_object_base(config: &ProjectConfig) -> Result<ObjectBase> {
if let Some(base) = &config.object_base {
let base = base.with_encoding();
// Search for disc images in the object base directory
for result in fs::read_dir(&base).with_context(|| format!("Reading directory {}", base))? {
let entry = result.with_context(|| format!("Reading entry in directory {}", base))?;
for result in fs::read_dir(&base).with_context(|| format!("Reading directory {base}"))? {
let entry = result.with_context(|| format!("Reading entry in directory {base}"))?;
let Ok(path) = check_path_buf(entry.path()) else {
log::warn!("Path is not valid UTF-8: {:?}", entry.path());
continue;
};
let file_type =
entry.file_type().with_context(|| format!("Getting file type for {}", path))?;
entry.file_type().with_context(|| format!("Getting file type for {path}"))?;
let is_file = if file_type.is_symlink() {
// Also traverse symlinks to files
fs::metadata(&path)
.with_context(|| format!("Getting metadata for {}", path))?
.with_context(|| format!("Getting metadata for {path}"))?
.is_file()
} else {
file_type.is_file()
};
if is_file {
let mut file = open_file(&path, false)?;
let format = nodtool::nod::Disc::detect(file.as_mut())
.with_context(|| format!("Detecting file type for {}", path))?;
if let Some(format) = format {
file.rewind()?;
let fs = open_fs(file, ArchiveKind::Disc(format))?;
return Ok(ObjectBase::Vfs(path, fs));
let format = detect(file.as_mut())
.with_context(|| format!("Detecting file type for {path}"))?;
match format {
FileFormat::Archive(ArchiveKind::Disc(format)) => {
let fs = open_fs(file, ArchiveKind::Disc(format))?;
return Ok(ObjectBase::Vfs(path, fs));
}
FileFormat::Archive(ArchiveKind::Wad) => {
let fs = open_fs(file, ArchiveKind::Wad)?;
return Ok(ObjectBase::Vfs(path, fs));
}
_ => {}
}
}
}
@ -2156,7 +2249,7 @@ fn extract_objects(config: &ProjectConfig, object_base: &ObjectBase) -> Result<U
{
let target_path = extracted_path(&target_dir, &config.base.object);
if !fs::exists(&target_path)
.with_context(|| format!("Failed to check path '{}'", target_path))?
.with_context(|| format!("Failed to check path '{target_path}'"))?
{
object_paths.push((&config.base.object, config.base.hash.as_deref(), target_path));
}
@ -2164,7 +2257,7 @@ fn extract_objects(config: &ProjectConfig, object_base: &ObjectBase) -> Result<U
if let Some(selfile) = &config.selfile {
let target_path = extracted_path(&target_dir, selfile);
if !fs::exists(&target_path)
.with_context(|| format!("Failed to check path '{}'", target_path))?
.with_context(|| format!("Failed to check path '{target_path}'"))?
{
object_paths.push((selfile, config.selfile_hash.as_deref(), target_path));
}
@ -2172,7 +2265,7 @@ fn extract_objects(config: &ProjectConfig, object_base: &ObjectBase) -> Result<U
for module_config in &config.modules {
let target_path = extracted_path(&target_dir, &module_config.object);
if !fs::exists(&target_path)
.with_context(|| format!("Failed to check path '{}'", target_path))?
.with_context(|| format!("Failed to check path '{target_path}'"))?
{
object_paths.push((&module_config.object, module_config.hash.as_deref(), target_path));
}
@ -2191,12 +2284,12 @@ fn extract_objects(config: &ProjectConfig, object_base: &ObjectBase) -> Result<U
let mut file = object_base.open(source_path)?;
if let Some(parent) = target_path.parent() {
fs::create_dir_all(parent)
.with_context(|| format!("Failed to create directory '{}'", parent))?;
.with_context(|| format!("Failed to create directory '{parent}'"))?;
}
let mut out = fs::File::create(&target_path)
.with_context(|| format!("Failed to create file '{}'", target_path))?;
.with_context(|| format!("Failed to create file '{target_path}'"))?;
let hash_bytes = buf_copy_with_hash(&mut file, &mut out)
.with_context(|| format!("Failed to extract file '{}'", target_path))?;
.with_context(|| format!("Failed to extract file '{target_path}'"))?;
if let Some(hash) = hash {
check_hash_str(hash_bytes, hash).with_context(|| {
format!("Source file failed verification: '{}'", object_base.join(source_path))
@ -2225,3 +2318,21 @@ fn extracted_path(target_dir: &Utf8NativePath, path: &Utf8UnixPath) -> Utf8Nativ
}
target_path
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_symbol_name_fuzzy_eq() {
assert!(symbol_name_fuzzy_eq("symbol", "symbol"));
assert!(symbol_name_fuzzy_eq("@1234", "@5678"));
assert!(symbol_name_fuzzy_eq("symbol$1234", "symbol$5678"));
assert!(symbol_name_fuzzy_eq("symbol", "symbol_80123456"));
assert!(symbol_name_fuzzy_eq("symbol_80123456", "symbol"));
assert!(!symbol_name_fuzzy_eq("symbol", "symbol2"));
assert!(!symbol_name_fuzzy_eq("symbol@1234", "symbol@5678"));
assert!(!symbol_name_fuzzy_eq("symbol", "symbol_80123456_"));
assert!(!symbol_name_fuzzy_eq("symbol_80123456_", "symbol"));
}
}

View File

@ -104,16 +104,16 @@ fn dump(args: DumpArgs) -> Result<()> {
// TODO make a basename method
let name = name.trim_start_matches("D:").replace('\\', "/");
let name = name.rsplit_once('/').map(|(_, b)| b).unwrap_or(&name);
let file_path = out_path.join(format!("{}.txt", name));
let file_path = out_path.join(format!("{name}.txt"));
let mut file = buf_writer(&file_path)?;
dump_debug_section(&args, &mut file, &obj_file, debug_section)?;
file.flush()?;
} else if args.no_color {
println!("\n// File {}:", name);
println!("\n// File {name}:");
dump_debug_section(&args, &mut stdout(), &obj_file, debug_section)?;
} else {
let mut writer = HighlightWriter::new(syntax_set.clone(), syntax.clone(), theme);
writeln!(writer, "\n// File {}:", name)?;
writeln!(writer, "\n// File {name}:")?;
dump_debug_section(&args, &mut writer, &obj_file, debug_section)?;
}
}
@ -209,26 +209,25 @@ where
}
writeln!(w, "\n/*\n Compile unit: {}", unit.name)?;
if let Some(producer) = unit.producer {
writeln!(w, " Producer: {}", producer)?;
writeln!(w, " Producer: {producer}")?;
}
if let Some(comp_dir) = unit.comp_dir {
writeln!(w, " Compile directory: {}", comp_dir)?;
writeln!(w, " Compile directory: {comp_dir}")?;
}
if let Some(language) = unit.language {
writeln!(w, " Language: {}", language)?;
writeln!(w, " Language: {language}")?;
}
if let (Some(start), Some(end)) = (unit.start_address, unit.end_address) {
writeln!(w, " Code range: {:#010X} -> {:#010X}", start, end)?;
writeln!(w, " Code range: {start:#010X} -> {end:#010X}")?;
}
if let Some(gcc_srcfile_name_offset) = unit.gcc_srcfile_name_offset {
writeln!(
w,
" GCC Source File Name Offset: {:#010X}",
gcc_srcfile_name_offset
" GCC Source File Name Offset: {gcc_srcfile_name_offset:#010X}"
)?;
}
if let Some(gcc_srcinfo_offset) = unit.gcc_srcinfo_offset {
writeln!(w, " GCC Source Info Offset: {:#010X}", gcc_srcinfo_offset)?;
writeln!(w, " GCC Source Info Offset: {gcc_srcinfo_offset:#010X}")?;
}
writeln!(w, "*/")?;
@ -269,7 +268,7 @@ where
continue;
}
match tag_type_string(&info, &typedefs, &tag_type, child.is_erased) {
Ok(s) => writeln!(w, "{}", s)?,
Ok(s) => writeln!(w, "{s}")?,
Err(e) => {
log::error!(
"Failed to emit tag {:X} (unit {}): {}",
@ -360,7 +359,7 @@ fn blend_fg_color(fg: Color, bg: Color) -> Color {
impl Write for HighlightWriter<'_> {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
let str = from_utf8(buf).map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
let str = from_utf8(buf).map_err(std::io::Error::other)?;
for s in str.split_inclusive('\n') {
self.line.push_str(s);
if self.line.ends_with('\n') {
@ -377,7 +376,7 @@ impl Write for HighlightWriter<'_> {
let ops = self
.parse_state
.parse_line(&self.line, &self.syntax_set)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
.map_err(std::io::Error::other)?;
let iter = HighlightIterator::new(
&mut self.highlight_state,
&ops[..],

View File

@ -1,11 +1,11 @@
use std::{
collections::{btree_map, hash_map, BTreeMap, HashMap},
collections::{btree_map, BTreeMap, HashMap},
fs,
fs::DirBuilder,
io::{Cursor, Write},
};
use anyhow::{anyhow, bail, ensure, Context, Result};
use anyhow::{anyhow, ensure, Context, Result};
use argp::FromArgs;
use objdiff_core::obj::split_meta::{SplitMeta, SPLITMETA_SECTION};
use object::{
@ -14,7 +14,7 @@ use object::{
FileFlags, Object, ObjectSection, ObjectSymbol, RelocationTarget, SectionFlags, SectionIndex,
SectionKind, SymbolFlags, SymbolIndex, SymbolKind, SymbolScope, SymbolSection,
};
use typed_path::{Utf8NativePath, Utf8NativePathBuf};
use typed_path::Utf8NativePathBuf;
use crate::{
obj::ObjKind,
@ -22,7 +22,7 @@ use crate::{
asm::write_asm,
comment::{CommentSym, MWComment},
config::{write_splits_file, write_symbols_file},
elf::{process_elf, write_elf},
elf::process_elf,
file::{buf_writer, process_rsp},
path::native_path,
reader::{Endian, FromReader},
@ -47,7 +47,6 @@ enum SubCommand {
Disasm(DisasmArgs),
Fixup(FixupArgs),
Signatures(SignaturesArgs),
Split(SplitArgs),
Info(InfoArgs),
}
@ -75,18 +74,6 @@ pub struct FixupArgs {
out_file: Utf8NativePathBuf,
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Splits an executable ELF into relocatable objects.
#[argp(subcommand, name = "split")]
pub struct SplitArgs {
#[argp(positional, from_str_fn(native_path))]
/// input file
in_file: Utf8NativePathBuf,
#[argp(positional, from_str_fn(native_path))]
/// output directory
out_dir: Utf8NativePathBuf,
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Generates configuration files from an executable ELF.
#[argp(subcommand, name = "config")]
@ -128,7 +115,6 @@ pub fn run(args: Args) -> Result<()> {
SubCommand::Config(c_args) => config(c_args),
SubCommand::Disasm(c_args) => disasm(c_args),
SubCommand::Fixup(c_args) => fixup(c_args),
SubCommand::Split(c_args) => split(c_args),
SubCommand::Signatures(c_args) => signatures(c_args),
SubCommand::Info(c_args) => info(c_args),
}
@ -159,14 +145,15 @@ fn disasm(args: DisasmArgs) -> Result<()> {
let mut files_out = buf_writer(&args.out.join("link_order.txt"))?;
for (unit, split_obj) in obj.link_order.iter().zip(&split_objs) {
let out_path = asm_dir.join(file_name_from_unit(&unit.name, ".s"));
let out_name = file_stem_from_unit(&unit.name);
let out_path = asm_dir.join(format!("{out_name}.s"));
log::info!("Writing {}", out_path);
let mut w = buf_writer(&out_path)?;
write_asm(&mut w, split_obj)?;
w.flush()?;
writeln!(files_out, "{}", file_name_from_unit(&unit.name, ".o"))?;
writeln!(files_out, "{out_name}.o")?;
}
files_out.flush()?;
}
@ -179,38 +166,7 @@ fn disasm(args: DisasmArgs) -> Result<()> {
Ok(())
}
fn split(args: SplitArgs) -> Result<()> {
let obj = process_elf(&args.in_file)?;
ensure!(obj.kind == ObjKind::Executable, "Can only split executable objects");
let mut file_map = HashMap::<String, Vec<u8>>::new();
let split_objs = split_obj(&obj, None)?;
for (unit, split_obj) in obj.link_order.iter().zip(&split_objs) {
let out_obj = write_elf(split_obj, false)?;
match file_map.entry(unit.name.clone()) {
hash_map::Entry::Vacant(e) => e.insert(out_obj),
hash_map::Entry::Occupied(_) => bail!("Duplicate file {}", unit.name),
};
}
let mut rsp_file = buf_writer(Utf8NativePath::new("rsp"))?;
for unit in &obj.link_order {
let object = file_map
.get(&unit.name)
.ok_or_else(|| anyhow!("Failed to find object file for unit '{}'", unit.name))?;
let out_path = args.out_dir.join(file_name_from_unit(&unit.name, ".o"));
writeln!(rsp_file, "{}", out_path)?;
if let Some(parent) = out_path.parent() {
DirBuilder::new().recursive(true).create(parent)?;
}
fs::write(&out_path, object).with_context(|| format!("Failed to write '{}'", out_path))?;
}
rsp_file.flush()?;
Ok(())
}
fn file_name_from_unit(str: &str, suffix: &str) -> String {
fn file_stem_from_unit(str: &str) -> String {
let str = str.strip_suffix(ASM_SUFFIX).unwrap_or(str);
let str = str.strip_prefix("C:").unwrap_or(str);
let str = str.strip_prefix("D:").unwrap_or(str);
@ -222,8 +178,7 @@ fn file_name_from_unit(str: &str, suffix: &str) -> String {
.or_else(|| str.strip_suffix(".o"))
.unwrap_or(str);
let str = str.replace('\\', "/");
let str = str.strip_prefix('/').unwrap_or(&str);
format!("{str}{suffix}")
str.strip_prefix('/').unwrap_or(&str).to_string()
}
const ASM_SUFFIX: &str = " (asm)";
@ -447,7 +402,7 @@ fn signatures(args: SignaturesArgs) -> Result<()> {
Ok(Some(signature)) => signature,
Ok(None) => continue,
Err(e) => {
eprintln!("Failed: {:?}", e);
eprintln!("Failed: {e:?}");
continue;
}
};
@ -590,13 +545,13 @@ fn info(args: InfoArgs) -> Result<()> {
.context("While reading .note.split section")?;
println!("\nSplit metadata (.note.split):");
if let Some(generator) = &meta.generator {
println!("\tGenerator: {}", generator);
println!("\tGenerator: {generator}");
}
if let Some(module_name) = &meta.module_name {
println!("\tModule name: {}", module_name);
println!("\tModule name: {module_name}");
}
if let Some(module_id) = meta.module_id {
println!("\tModule ID: {}", module_id);
println!("\tModule ID: {module_id}");
}
if let Some(virtual_addresses) = &meta.virtual_addresses {
println!("\tVirtual addresses:");

View File

@ -6,16 +6,20 @@ use object::{Architecture, Endianness, Object, ObjectKind, ObjectSection, Sectio
use typed_path::Utf8NativePathBuf;
use crate::{
util::{file::buf_writer, path::native_path},
util::{
dol::{process_dol, write_dol},
file::buf_writer,
path::native_path,
},
vfs::open_file,
};
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Converts an ELF file to a DOL file.
/// Converts an ELF, ALF, or BootStage file to a DOL file.
#[argp(subcommand, name = "elf2dol")]
pub struct Args {
#[argp(positional, from_str_fn(native_path))]
/// path to input ELF
/// path to input ELF, ALF or BootStage file
elf_file: Utf8NativePathBuf,
#[argp(positional, from_str_fn(native_path))]
/// path to output DOL
@ -48,7 +52,12 @@ const MAX_DATA_SECTIONS: usize = 11;
pub fn run(args: Args) -> Result<()> {
let mut file = open_file(&args.elf_file, true)?;
let obj_file = object::read::File::parse(file.map()?)?;
let data = file.map()?;
if data.len() >= 4 && data[0..4] != object::elf::ELFMAG {
return convert_dol_like(args, data);
}
let obj_file = object::read::File::parse(data)?;
match obj_file.architecture() {
Architecture::PowerPc => {}
arch => bail!("Unexpected architecture: {arch:?}"),
@ -153,6 +162,14 @@ pub fn run(args: Args) -> Result<()> {
Ok(())
}
/// Converts a DOL-like format (ALF or BootStage) to a DOL file.
fn convert_dol_like(args: Args, data: &[u8]) -> Result<()> {
let obj = process_dol(data, "")?;
let mut out = buf_writer(&args.dol_file)?;
write_dol(&obj, &mut out)?;
Ok(())
}
#[inline]
const fn align32(x: u32) -> u32 { (x + 31) & !31 }

78
src/cmd/extab.rs Normal file
View File

@ -0,0 +1,78 @@
use std::io::Write;
use anyhow::{Context, Result};
use argp::FromArgs;
use typed_path::Utf8NativePathBuf;
use crate::{
util,
util::{
dol::{process_dol, write_dol},
elf::{is_elf_file, process_elf, write_elf},
file::buf_writer,
path::native_path,
},
vfs::open_file,
};
#[derive(FromArgs, PartialEq, Debug)]
/// Commands for processing extab (exception table) data.
#[argp(subcommand, name = "extab")]
pub struct Args {
#[argp(subcommand)]
command: SubCommand,
}
#[derive(FromArgs, PartialEq, Debug)]
#[argp(subcommand)]
enum SubCommand {
Clean(CleanArgs),
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Rewrites extab data in a DOL or ELF file, replacing any uninitialized padding bytes.
#[argp(subcommand, name = "clean")]
pub struct CleanArgs {
#[argp(positional, from_str_fn(native_path))]
/// Path to input file
input: Utf8NativePathBuf,
#[argp(positional, from_str_fn(native_path))]
/// Path to output file
output: Utf8NativePathBuf,
#[argp(option, short = 'p')]
/// Data to replace padding bytes with, encoded as a hexadecimal string. If not specified, padding bytes will be zeroed instead.
padding: Option<String>,
}
pub fn run(args: Args) -> Result<()> {
match args.command {
SubCommand::Clean(clean_args) => clean_extab(clean_args),
}
}
fn clean_extab(args: CleanArgs) -> Result<()> {
let is_elf = is_elf_file(&args.input)?;
let mut obj = if is_elf {
process_elf(&args.input)?
} else {
let mut file = open_file(&args.input, true)?;
let name = args.input.file_stem().unwrap_or_default();
process_dol(file.map()?, name)?
};
let padding: Vec<u8> = match args.padding {
None => Vec::new(),
Some(padding_str) => {
hex::decode(padding_str).context("Failed to decode padding bytes from hex")?
}
};
let num_cleaned = util::extab::clean_extab(&mut obj, padding.iter().copied())?;
tracing::debug!("Cleaned {num_cleaned} extab symbols");
let mut out = buf_writer(&args.output)?;
if is_elf {
let data = write_elf(&obj, false)?;
out.write_all(&data).context("Failed to write ELF")?;
} else {
write_dol(&obj, &mut out).context("Failed to write DOL")?;
}
Ok(())
}

View File

@ -175,7 +175,7 @@ fn symbol(args: SymbolArgs) -> Result<()> {
if let Some(vec) = entries.unit_references.get_vec(&symbol_ref) {
println!("\nGenerated in TUs:");
for x in vec {
println!(">>> {}", x);
println!(">>> {x}");
}
}
println!("\n");

View File

@ -6,6 +6,7 @@ pub mod dol;
pub mod dwarf;
pub mod elf;
pub mod elf2dol;
pub mod extab;
pub mod map;
pub mod nlzss;
pub mod rarc;
@ -14,5 +15,6 @@ pub mod rso;
pub mod shasum;
pub mod u8_arc;
pub mod vfs;
pub mod wad;
pub mod yay0;
pub mod yaz0;

View File

@ -59,7 +59,7 @@ fn decompress(args: DecompressArgs) -> Result<()> {
path.as_path().to_cow()
};
fs::write(out_path.as_ref(), data)
.with_context(|| format!("Failed to write '{}'", out_path))?;
.with_context(|| format!("Failed to write '{out_path}'"))?;
}
Ok(())
}

View File

@ -104,6 +104,9 @@ pub struct MakeArgs {
#[argp(option, short = 'n')]
/// (optional) module names
names: Vec<String>,
#[argp(option, short = 'v')]
/// (optional) REL version (default is 3)
version: Option<u32>,
#[argp(switch, short = 'w')]
/// disable warnings
no_warn: bool,
@ -313,7 +316,7 @@ fn make(args: MakeArgs) -> Result<()> {
.unwrap_or(idx as u32);
load_obj(file.map()?)
.map(|o| LoadedModule { module_id, file: o, path: path.clone() })
.with_context(|| format!("Failed to load '{}'", path))
.with_context(|| format!("Failed to load '{path}'"))
})
.collect::<Result<Vec<_>>>()?;
@ -364,7 +367,7 @@ fn make(args: MakeArgs) -> Result<()> {
let _span = info_span!("file", path = %module_info.path).entered();
let mut info = RelWriteInfo {
module_id: module_info.module_id,
version: 3,
version: args.version.unwrap_or(3),
name_offset: None,
name_size: None,
align: None,
@ -392,7 +395,7 @@ fn make(args: MakeArgs) -> Result<()> {
let rel_path = module_info.path.with_extension("rel");
let mut w = buf_writer(&rel_path)?;
write_rel(&mut w, &info, &module_info.file, relocations)
.with_context(|| format!("Failed to write '{}'", rel_path))?;
.with_context(|| format!("Failed to write '{rel_path}'"))?;
w.flush()?;
}

View File

@ -143,7 +143,7 @@ fn make_rso(
let si = sym
.section_index()
.with_context(|| format!("Failed to find symbol `{}` section index", name))?;
.with_context(|| format!("Failed to find symbol `{name}` section index"))?;
let addr = sym.address();
*index = si.0 as u8;
@ -204,8 +204,7 @@ fn make_rso(
let mut rso_sections: Vec<RsoSectionHeader> =
vec![RsoSectionHeader::default() /* ELF null section */];
for section in file.sections() {
let is_valid_section =
section.name().is_ok_and(|n| RSO_SECTION_NAMES.iter().any(|&s| s == n));
let is_valid_section = section.name().is_ok_and(|n| RSO_SECTION_NAMES.contains(&n));
let section_size = section.size();
if !is_valid_section || section_size == 0 {
@ -321,8 +320,7 @@ fn make_rso(
let mut exported_relocations: Vec<RsoRelocation> = vec![];
for section in file.sections() {
let is_valid_section =
section.name().is_ok_and(|n| RSO_SECTION_NAMES.iter().any(|&s| s == n));
let is_valid_section = section.name().is_ok_and(|n| RSO_SECTION_NAMES.contains(&n));
if !is_valid_section {
continue;
}

View File

@ -45,14 +45,13 @@ pub fn run(args: Args) -> Result<()> {
check(&args, file.as_mut())?;
}
if let Some(out_path) = &args.output {
touch(out_path)
.with_context(|| format!("Failed to touch output file '{}'", out_path))?;
touch(out_path).with_context(|| format!("Failed to touch output file '{out_path}'"))?;
}
} else {
let mut w: Box<dyn Write> = if let Some(out_path) = &args.output {
Box::new(
buf_writer(out_path)
.with_context(|| format!("Failed to open output file '{}'", out_path))?,
.with_context(|| format!("Failed to open output file '{out_path}'"))?,
)
} else {
Box::new(stdout())

View File

@ -4,6 +4,7 @@ use anyhow::{anyhow, bail, Context};
use argp::FromArgs;
use size::Size;
use typed_path::{Utf8NativePath, Utf8NativePathBuf, Utf8UnixPath};
use unicode_width::UnicodeWidthStr;
use crate::{
util::{file::buf_copy, path::native_path},
@ -72,7 +73,7 @@ fn column_widths<const N: usize>(entries: &[Columns<N>]) -> [usize; N] {
let mut widths = [0usize; N];
for text in entries {
for (i, column) in text.iter().enumerate() {
widths[i] = widths[i].max(column.len());
widths[i] = widths[i].max(column.width_cjk());
}
}
widths
@ -84,7 +85,7 @@ fn file_info(
metadata: &VfsMetadata,
) -> anyhow::Result<Columns<5>> {
let format =
detect(file).with_context(|| format!("Failed to detect file format for {}", filename))?;
detect(file).with_context(|| format!("Failed to detect file format for {filename}"))?;
let mut info: Columns<5> = [
Size::from_bytes(metadata.len).to_string(),
filename.to_string(),
@ -96,9 +97,9 @@ fn file_info(
let mut decompressed = decompress_file(file, kind)?;
let metadata = decompressed
.metadata()
.with_context(|| format!("Failed to fetch metadata for {}", filename))?;
.with_context(|| format!("Failed to fetch metadata for {filename}"))?;
let format = detect(decompressed.as_mut())
.with_context(|| format!("Failed to detect file format for {}", filename))?;
.with_context(|| format!("Failed to detect file format for {filename}"))?;
info[3] = format!("Decompressed: {}", Size::from_bytes(metadata.len));
info[4] = format.to_string();
}
@ -111,11 +112,11 @@ pub fn ls(args: LsArgs) -> anyhow::Result<()> {
OpenResult::File(mut file, path) => {
let filename = path.file_name().ok_or_else(|| anyhow!("Path has no filename"))?;
if args.short {
println!("{}", filename);
println!("{filename}");
} else {
let metadata = file
.metadata()
.with_context(|| format!("Failed to fetch metadata for {}", path))?;
.with_context(|| format!("Failed to fetch metadata for {path}"))?;
files.push(file_info(filename, file.as_mut(), &metadata)?);
}
}
@ -130,10 +131,14 @@ pub fn ls(args: LsArgs) -> anyhow::Result<()> {
for (i, column) in entry.iter().enumerate() {
if widths[i] > 0 {
if written > 0 {
print!("{}", SEPARATOR);
print!("{SEPARATOR}");
}
written += 1;
print!("{:width$}", column, width = widths[i]);
print!("{column}");
let remain = widths[i].saturating_sub(column.width_cjk());
if remain > 0 {
print!("{:width$}", "", width = remain);
}
}
}
println!();
@ -156,25 +161,25 @@ fn ls_directory(
let display_path = base_filename.join(&filename);
let metadata = fs
.metadata(&entry_path)
.with_context(|| format!("Failed to fetch metadata for {}", entry_path))?;
.with_context(|| format!("Failed to fetch metadata for {entry_path}"))?;
match metadata.file_type {
VfsFileType::File => {
let mut file = fs
.open(&entry_path)
.with_context(|| format!("Failed to open file {}", entry_path))?;
.with_context(|| format!("Failed to open file {entry_path}"))?;
if args.short {
println!("{}", display_path);
println!("{display_path}");
} else {
files.push(file_info(display_path.as_str(), file.as_mut(), &metadata)?);
}
}
VfsFileType::Directory => {
if args.short {
println!("{}/", display_path);
println!("{display_path}/");
} else {
files.push([
" ".to_string(),
format!("{}/", display_path),
format!("{display_path}/"),
"Directory".to_string(),
String::new(),
String::new(),
@ -201,7 +206,7 @@ pub fn cp(mut args: CpArgs) -> anyhow::Result<()> {
OpenResult::File(file, path) => {
let dest = if dest_is_dir {
fs::create_dir_all(&dest)
.with_context(|| format!("Failed to create directory {}", dest))?;
.with_context(|| format!("Failed to create directory {dest}"))?;
let filename =
path.file_name().ok_or_else(|| anyhow!("Path has no filename"))?;
dest.join(filename)
@ -229,12 +234,12 @@ fn cp_file(
if let FileFormat::Compressed(kind) = detect(file.as_mut())? {
if auto_decompress {
file = decompress_file(file.as_mut(), kind)
.with_context(|| format!("Failed to decompress file {}", dest))?;
.with_context(|| format!("Failed to decompress file {dest}"))?;
compression = Some(kind);
}
}
let metadata =
file.metadata().with_context(|| format!("Failed to fetch metadata for {}", dest))?;
file.metadata().with_context(|| format!("Failed to fetch metadata for {dest}"))?;
if !quiet {
if let Some(kind) = compression {
println!(
@ -249,10 +254,10 @@ fn cp_file(
}
}
let mut dest_file =
File::create(dest).with_context(|| format!("Failed to create file {}", dest))?;
File::create(dest).with_context(|| format!("Failed to create file {dest}"))?;
buf_copy(file.as_mut(), &mut dest_file)
.with_context(|| format!("Failed to copy file {}", dest))?;
dest_file.flush().with_context(|| format!("Failed to flush file {}", dest))?;
.with_context(|| format!("Failed to copy file {dest}"))?;
dest_file.flush().with_context(|| format!("Failed to flush file {dest}"))?;
Ok(())
}
@ -263,18 +268,18 @@ fn cp_recursive(
auto_decompress: bool,
quiet: bool,
) -> anyhow::Result<()> {
fs::create_dir_all(dest).with_context(|| format!("Failed to create directory {}", dest))?;
fs::create_dir_all(dest).with_context(|| format!("Failed to create directory {dest}"))?;
let entries = fs.read_dir(path)?;
for filename in entries {
let entry_path = path.join(&filename);
let metadata = fs
.metadata(&entry_path)
.with_context(|| format!("Failed to fetch metadata for {}", entry_path))?;
.with_context(|| format!("Failed to fetch metadata for {entry_path}"))?;
match metadata.file_type {
VfsFileType::File => {
let file = fs
.open(&entry_path)
.with_context(|| format!("Failed to open file {}", entry_path))?;
.with_context(|| format!("Failed to open file {entry_path}"))?;
cp_file(file, &entry_path, &dest.join(filename), auto_decompress, quiet)?;
}
VfsFileType::Directory => {

108
src/cmd/wad.rs Normal file
View File

@ -0,0 +1,108 @@
use anyhow::Result;
use argp::FromArgs;
use size::Size;
use typed_path::Utf8NativePathBuf;
use crate::{
cmd::vfs,
util::{
path::native_path,
wad::{process_wad, verify_wad},
},
vfs::open_file,
};
#[derive(FromArgs, PartialEq, Debug)]
/// Commands for processing Wii WAD files.
#[argp(subcommand, name = "wad")]
pub struct Args {
#[argp(subcommand)]
command: SubCommand,
}
#[derive(FromArgs, PartialEq, Debug)]
#[argp(subcommand)]
enum SubCommand {
Extract(ExtractArgs),
Info(InfoArgs),
Verify(VerifyArgs),
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Extracts WAD file contents.
#[argp(subcommand, name = "extract")]
pub struct ExtractArgs {
#[argp(positional, from_str_fn(native_path))]
/// WAD file
file: Utf8NativePathBuf,
#[argp(option, short = 'o', from_str_fn(native_path))]
/// output directory
output: Option<Utf8NativePathBuf>,
#[argp(switch)]
/// Do not decompress files when copying.
no_decompress: bool,
#[argp(switch, short = 'q')]
/// Quiet output. Don't print anything except errors.
quiet: bool,
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Views WAD file information.
#[argp(subcommand, name = "info")]
pub struct InfoArgs {
#[argp(positional, from_str_fn(native_path))]
/// WAD file
file: Utf8NativePathBuf,
}
#[derive(FromArgs, PartialEq, Eq, Debug)]
/// Verifies WAD file integrity.
#[argp(subcommand, name = "verify")]
pub struct VerifyArgs {
#[argp(positional, from_str_fn(native_path))]
/// WAD file
file: Utf8NativePathBuf,
}
pub fn run(args: Args) -> Result<()> {
match args.command {
SubCommand::Info(c_args) => info(c_args),
SubCommand::Verify(c_args) => verify(c_args),
SubCommand::Extract(c_args) => extract(c_args),
}
}
fn info(args: InfoArgs) -> Result<()> {
let mut file = open_file(&args.file, true)?;
let wad = process_wad(file.as_mut())?;
println!("Title ID: {}", hex::encode(wad.ticket().title_id));
println!("Title key: {}", hex::encode(wad.title_key));
println!("Fake signed: {}", wad.fake_signed);
for content in wad.contents() {
println!(
"Content {:08x}: Offset {:#X}, size {}",
content.content_index.get(),
wad.content_offset(content.content_index.get()),
Size::from_bytes(content.size.get())
);
}
Ok(())
}
fn verify(args: VerifyArgs) -> Result<()> {
let mut file = open_file(&args.file, true)?;
let wad = process_wad(file.as_mut())?;
verify_wad(&wad, file.as_mut())?;
println!("Verification successful");
Ok(())
}
fn extract(args: ExtractArgs) -> Result<()> {
let path = Utf8NativePathBuf::from(format!("{}:", args.file));
let output = args.output.unwrap_or_else(|| Utf8NativePathBuf::from("."));
vfs::cp(vfs::CpArgs {
paths: vec![path, output],
no_decompress: args.no_decompress,
quiet: args.quiet,
})
}

View File

@ -80,7 +80,7 @@ fn compress(args: CompressArgs) -> Result<()> {
path.as_path().to_cow()
};
fs::write(out_path.as_ref(), data)
.with_context(|| format!("Failed to write '{}'", out_path))?;
.with_context(|| format!("Failed to write '{out_path}'"))?;
}
Ok(())
}
@ -92,7 +92,7 @@ fn decompress(args: DecompressArgs) -> Result<()> {
let data = {
let mut file = open_file(&path, true)?;
decompress_yay0(file.map()?)
.with_context(|| format!("Failed to decompress '{}' using Yay0", path))?
.with_context(|| format!("Failed to decompress '{path}' using Yay0"))?
};
let out_path = if let Some(output) = &args.output {
if single_file {
@ -104,7 +104,7 @@ fn decompress(args: DecompressArgs) -> Result<()> {
path.as_path().to_cow()
};
fs::write(out_path.as_ref(), data)
.with_context(|| format!("Failed to write '{}'", out_path))?;
.with_context(|| format!("Failed to write '{out_path}'"))?;
}
Ok(())
}

View File

@ -80,7 +80,7 @@ fn compress(args: CompressArgs) -> Result<()> {
path.as_path().to_cow()
};
fs::write(out_path.as_ref(), data)
.with_context(|| format!("Failed to write '{}'", out_path))?;
.with_context(|| format!("Failed to write '{out_path}'"))?;
}
Ok(())
}
@ -92,7 +92,7 @@ fn decompress(args: DecompressArgs) -> Result<()> {
let data = {
let mut file = open_file(&path, false)?;
decompress_yaz0(file.map()?)
.with_context(|| format!("Failed to decompress '{}' using Yaz0", path))?
.with_context(|| format!("Failed to decompress '{path}' using Yaz0"))?
};
let out_path = if let Some(output) = &args.output {
if single_file {
@ -104,7 +104,7 @@ fn decompress(args: DecompressArgs) -> Result<()> {
path.as_path().to_cow()
};
fs::write(out_path.as_ref(), data)
.with_context(|| format!("Failed to write '{}'", out_path))?;
.with_context(|| format!("Failed to write '{out_path}'"))?;
}
Ok(())
}

View File

@ -96,6 +96,7 @@ enum SubCommand {
Dwarf(cmd::dwarf::Args),
Elf(cmd::elf::Args),
Elf2Dol(cmd::elf2dol::Args),
Extab(cmd::extab::Args),
Map(cmd::map::Args),
Nlzss(cmd::nlzss::Args),
Rarc(cmd::rarc::Args),
@ -106,6 +107,7 @@ enum SubCommand {
Vfs(cmd::vfs::Args),
Yay0(cmd::yay0::Args),
Yaz0(cmd::yaz0::Args),
Wad(cmd::wad::Args),
}
// Duplicated from supports-color so we can check early.
@ -171,6 +173,7 @@ fn main() {
SubCommand::Dwarf(c_args) => cmd::dwarf::run(c_args),
SubCommand::Elf(c_args) => cmd::elf::run(c_args),
SubCommand::Elf2Dol(c_args) => cmd::elf2dol::run(c_args),
SubCommand::Extab(c_args) => cmd::extab::run(c_args),
SubCommand::Map(c_args) => cmd::map::run(c_args),
SubCommand::Nlzss(c_args) => cmd::nlzss::run(c_args),
SubCommand::Rarc(c_args) => cmd::rarc::run(c_args),
@ -181,6 +184,7 @@ fn main() {
SubCommand::Vfs(c_args) => cmd::vfs::run(c_args),
SubCommand::Yay0(c_args) => cmd::yay0::run(c_args),
SubCommand::Yaz0(c_args) => cmd::yaz0::run(c_args),
SubCommand::Wad(c_args) => cmd::wad::run(c_args),
});
if let Err(e) = result {
eprintln!("Failed: {e:?}");

View File

@ -5,7 +5,7 @@ use itertools::Itertools;
use crate::{
obj::{ObjInfo, ObjSection, SectionIndex},
util::{nested::NestedVec, split::default_section_align},
util::split::default_section_align,
};
/// Marks a split point within a section.
@ -107,7 +107,9 @@ impl ObjSplits {
}
pub fn push(&mut self, address: u32, split: ObjSplit) {
self.splits.nested_push(address, split);
let out = self.splits.entry(address).or_default();
out.push(split);
out.sort_by_key(|s| s.end);
}
pub fn remove(&mut self, address: u32) -> Option<Vec<ObjSplit>> { self.splits.remove(&address) }

View File

@ -175,8 +175,10 @@ pub enum ObjDataKind {
Float,
Double,
String,
ShiftJIS,
String16,
StringTable,
ShiftJISTable,
String16Table,
Int,
Short,
@ -401,7 +403,7 @@ impl ObjSymbols {
pub fn iter_ordered(&self) -> impl DoubleEndedIterator<Item = (SymbolIndex, &ObjSymbol)> {
self.symbols_by_section
.iter()
.flat_map(|v| v.iter().map(|(_, v)| v))
.flat_map(|v| v.values())
.flat_map(move |v| v.iter().map(move |u| (*u, &self.symbols[*u as usize])))
}
@ -448,7 +450,7 @@ impl ObjSymbols {
self.symbols_by_section
.get(section_idx as usize)
.into_iter()
.flat_map(|v| v.iter().map(|(_, v)| v))
.flat_map(|v| v.values())
.flat_map(move |v| v.iter().map(move |u| (*u, &self.symbols[*u as usize])))
}

View File

@ -161,7 +161,7 @@ impl FromReader for AlfSymbolKind {
match u32::from_reader(reader, e)? {
0 => Ok(Self::Function),
1 => Ok(Self::Object),
v => Err(Error::new(ErrorKind::InvalidData, format!("invalid ALF symbol kind: {}", v))),
v => Err(Error::new(ErrorKind::InvalidData, format!("invalid ALF symbol kind: {v}"))),
}
}
}

View File

@ -442,12 +442,12 @@ where
match parse_extab(symbols, entry, section) {
Ok(s) => {
for line in s.trim_end().lines() {
writeln!(w, " * {}", line)?;
writeln!(w, " * {line}")?;
}
}
Err(e) => {
log::warn!("Failed to decode extab entry {}: {}", symbol.name, e);
writeln!(w, " * Failed to decode extab entry: {}", e)?;
writeln!(w, " * Failed to decode extab entry: {e}")?;
}
}
writeln!(w, " */")?;
@ -505,7 +505,7 @@ where
}
current_symbol_kind = find_symbol_kind(current_symbol_kind, symbols, vec)?;
current_data_kind = find_data_kind(current_data_kind, symbols, vec)
.with_context(|| format!("At address {:#010X}", sym_addr))?;
.with_context(|| format!("At address {sym_addr:#010X}"))?;
entry = entry_iter.next();
} else if current_address > sym_addr {
let dbg_symbols = vec.iter().map(|e| &symbols[e.index as usize]).collect_vec();
@ -660,14 +660,46 @@ where W: Write + ?Sized {
'\x0D' => write!(w, "\\r")?,
'\\' => write!(w, "\\\\")?,
'"' => write!(w, "\\\"")?,
c if c.is_ascii_graphic() || c.is_ascii_whitespace() => write!(w, "{}", c)?,
_ => write!(w, "\\{:03o}", b)?,
c if c.is_ascii_graphic() || c.is_ascii_whitespace() => write!(w, "{c}")?,
_ => write!(w, "\\{b:03o}")?,
}
}
writeln!(w, "\"")?;
Ok(())
}
use encoding_rs::SHIFT_JIS;
fn write_string_shiftjis<W>(w: &mut W, data: &[u8]) -> Result<()>
where W: Write + ?Sized {
if data.last() != Some(&0x00) {
bail!("Non-terminated Shift-JIS string");
}
let raw_data = &data[..data.len() - 1];
// Decode then write SJIS as comment above byte array
let (cow, _, _) = SHIFT_JIS.decode(raw_data);
write!(w, "\t# ")?;
for c in cow.chars() {
match c {
'#' => write!(w, "\\#")?,
_ => write!(w, "{c}")?,
}
}
write!(w, "\n\t.byte ")?;
for (i, &b) in data.iter().enumerate() {
write!(w, "0x{b:02X}")?;
if i + 1 != data.len() {
write!(w, ", ")?;
}
}
writeln!(w)?;
Ok(())
}
fn write_string16<W>(w: &mut W, data: &[u16]) -> Result<()>
where W: Write + ?Sized {
if matches!(data.last(), Some(&b) if b == 0) {
@ -689,7 +721,7 @@ where W: Write + ?Sized {
'\x0D' => write!(w, "\\r")?,
'\\' => write!(w, "\\\\")?,
'"' => write!(w, "\\\"")?,
c if c.is_ascii_graphic() || c.is_ascii_whitespace() => write!(w, "{}", c)?,
c if c.is_ascii_graphic() || c.is_ascii_whitespace() => write!(w, "{c}")?,
_ => write!(w, "\\{:#X}", c as u32)?,
}
}
@ -705,6 +737,12 @@ where W: Write + ?Sized {
ObjDataKind::String => {
return write_string(w, data);
}
ObjDataKind::ShiftJIS => {
if data.is_empty() || data.last() != Some(&0x00) {
anyhow::bail!("Non-terminated Shift-JIS string");
}
return write_string_shiftjis(w, data);
}
ObjDataKind::String16 => {
if data.len() % 2 != 0 {
bail!("Attempted to write wstring with length {:#X}", data.len());
@ -734,6 +772,12 @@ where W: Write + ?Sized {
}
return Ok(());
}
ObjDataKind::ShiftJISTable => {
for slice in data.split_inclusive(|&b| b == 0) {
write_string_shiftjis(w, slice)?;
}
return Ok(());
}
_ => {}
}
let chunk_size = match data_kind {
@ -742,12 +786,14 @@ where W: Write + ?Sized {
ObjDataKind::Byte | ObjDataKind::Byte8 | ObjDataKind::Double => 8,
ObjDataKind::String
| ObjDataKind::String16
| ObjDataKind::ShiftJIS
| ObjDataKind::StringTable
| ObjDataKind::ShiftJISTable
| ObjDataKind::String16Table => unreachable!(),
};
for chunk in remain.chunks(chunk_size) {
if data_kind == ObjDataKind::Byte || matches!(chunk.len(), 1 | 3 | 5..=7) {
let bytes = chunk.iter().map(|c| format!("{:#04X}", c)).collect::<Vec<String>>();
let bytes = chunk.iter().map(|c| format!("{c:#04X}")).collect::<Vec<String>>();
writeln!(w, "\t.byte {}", bytes.join(", "))?;
} else {
match chunk.len() {

View File

@ -50,15 +50,26 @@ impl fmt::Display for HeaderKind {
}
/// Converts a binary blob into a C array.
pub fn bin2c(symbol: &ObjSymbol, section: &ObjSection, data: &[u8], kind: HeaderKind) -> String {
pub fn bin2c(
symbol: &ObjSymbol,
section: &ObjSection,
data: &[u8],
kind: HeaderKind,
rename: Option<&str>,
) -> String {
match kind {
HeaderKind::None => String::new(),
HeaderKind::Symbol => bin2c_symbol(symbol, section, data),
HeaderKind::Symbol => bin2c_symbol(symbol, section, data, rename),
HeaderKind::Raw => bin2c_raw(data),
}
}
fn bin2c_symbol(symbol: &ObjSymbol, section: &ObjSection, data: &[u8]) -> String {
fn bin2c_symbol(
symbol: &ObjSymbol,
section: &ObjSection,
data: &[u8],
rename: Option<&str>,
) -> String {
let mut output = String::new();
output.push_str(PROLOGUE);
output.push_str(&format!(
@ -72,7 +83,11 @@ fn bin2c_symbol(symbol: &ObjSymbol, section: &ObjSection, data: &[u8]) -> String
output.push_str("const ");
}
output.push_str("unsigned char ");
output.push_str(symbol.demangled_name.as_deref().unwrap_or(symbol.name.as_str()));
if let Some(rename) = rename {
output.push_str(rename);
} else {
output.push_str(symbol.demangled_name.as_deref().unwrap_or(symbol.name.as_str()));
}
output.push_str(&format!("[] ATTRIBUTE_ALIGN({}) = {{", symbol.align.unwrap_or(4)));
for (i, byte) in data.iter().enumerate() {
if i % 16 == 0 {
@ -80,7 +95,7 @@ fn bin2c_symbol(symbol: &ObjSymbol, section: &ObjSection, data: &[u8]) -> String
} else {
output.push(' ');
}
output.push_str(&format!("0x{:02X},", byte));
output.push_str(&format!("0x{byte:02X},"));
}
output.push_str("\n};\n");
output
@ -96,7 +111,7 @@ fn bin2c_raw(data: &[u8]) -> String {
output.push(' ');
}
}
output.push_str(&format!("0x{:02X},", byte));
output.push_str(&format!("0x{byte:02X},"));
}
output.push('\n');
output

View File

@ -58,7 +58,7 @@ impl FromReader for MWComment {
if magic != MAGIC {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Invalid .comment section magic: {:?}", magic),
format!("Invalid .comment section magic: {magic:?}"),
));
}
// 0xB
@ -78,7 +78,7 @@ impl FromReader for MWComment {
value => {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Invalid value for pool_data: {}", value),
format!("Invalid value for pool_data: {value}"),
))
}
};
@ -93,7 +93,7 @@ impl FromReader for MWComment {
v => {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Expected header size {:#X}, got {:#X}", HEADER_SIZE, v),
format!("Expected header size {HEADER_SIZE:#X}, got {v:#X}"),
))
}
}
@ -102,7 +102,7 @@ impl FromReader for MWComment {
if flags & !7 != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Unexpected flag value {:#X}", flags),
format!("Unexpected flag value {flags:#X}"),
));
}
if flags & 1 == 1 {
@ -221,14 +221,14 @@ impl FromReader for CommentSym {
if value != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Unexpected value after active_flags (1): {:#X}", value),
format!("Unexpected value after active_flags (1): {value:#X}"),
));
}
let value = u8::from_reader(reader, e)?;
if value != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Unexpected value after active_flags (2): {:#X}", value),
format!("Unexpected value after active_flags (2): {value:#X}"),
));
}
Ok(out)

View File

@ -282,11 +282,11 @@ where W: Write + ?Sized {
write!(w, " data:{kind}")?;
}
if let Some(hash) = symbol.name_hash {
write!(w, " hash:{:#010X}", hash)?;
write!(w, " hash:{hash:#010X}")?;
}
if let Some(hash) = symbol.demangled_name_hash {
if symbol.name_hash != symbol.demangled_name_hash {
write!(w, " dhash:{:#010X}", hash)?;
write!(w, " dhash:{hash:#010X}")?;
}
}
if symbol.flags.is_hidden() {
@ -329,8 +329,10 @@ fn symbol_data_kind_to_str(kind: ObjDataKind) -> Option<&'static str> {
ObjDataKind::Float => Some("float"),
ObjDataKind::Double => Some("double"),
ObjDataKind::String => Some("string"),
ObjDataKind::ShiftJIS => Some("sjis"),
ObjDataKind::String16 => Some("wstring"),
ObjDataKind::StringTable => Some("string_table"),
ObjDataKind::ShiftJISTable => Some("sjis_table"),
ObjDataKind::String16Table => Some("wstring_table"),
ObjDataKind::Int => Some("int"),
ObjDataKind::Short => Some("short"),
@ -382,8 +384,10 @@ fn symbol_data_kind_from_str(s: &str) -> Option<ObjDataKind> {
"float" => Some(ObjDataKind::Float),
"double" => Some(ObjDataKind::Double),
"string" => Some(ObjDataKind::String),
"sjis" => Some(ObjDataKind::ShiftJIS),
"wstring" => Some(ObjDataKind::String16),
"string_table" => Some(ObjDataKind::StringTable),
"sjis_table" => Some(ObjDataKind::ShiftJISTable),
"wstring_table" => Some(ObjDataKind::String16Table),
"int" => Some(ObjDataKind::Int),
"short" => Some(ObjDataKind::Short),
@ -435,10 +439,10 @@ where W: Write + ?Sized {
for unit in obj.link_order.iter().filter(|unit| all || !unit.autogenerated) {
write!(w, "\n{}:", unit.name)?;
if let Some(comment_version) = unit.comment_version {
write!(w, " comment:{}", comment_version)?;
write!(w, " comment:{comment_version}")?;
}
if let Some(order) = unit.order {
write!(w, " order:{}", order)?;
write!(w, " order:{order}")?;
}
writeln!(w)?;
let mut split_iter = obj.sections.all_splits().peekable();
@ -454,14 +458,14 @@ where W: Write + ?Sized {
write!(w, "\t{:<11} start:{:#010X} end:{:#010X}", section.name, addr, end)?;
if let Some(align) = split.align {
if align != default_section_align(section) as u32 {
write!(w, " align:{}", align)?;
write!(w, " align:{align}")?;
}
}
if split.common {
write!(w, " common")?;
}
if let Some(name) = &split.rename {
write!(w, " rename:{}", name)?;
write!(w, " rename:{name}")?;
}
if split.skip {
write!(w, " skip")?;
@ -710,7 +714,7 @@ where R: BufRead + ?Sized {
ensure!(
section.contains_range(start..end)
|| (start == section_end && end == section_end),
"Section {} ({:#010X}..{:#010X}) does not contain range {:#010X}..{:#010X}",
"Section {} ({:#010X}..{:#010X}) does not contain range {:#010X}..{:#010X}. Check splits.txt?",
name,
section.address,
section.address + section.size,
@ -779,7 +783,7 @@ pub mod signed_hex_serde {
if *value < 0 {
serializer.serialize_str(&format!("-{:#X}", -value))
} else {
serializer.serialize_str(&format!("{:#X}", value))
serializer.serialize_str(&format!("{value:#X}"))
}
}
@ -787,7 +791,7 @@ pub mod signed_hex_serde {
where D: Deserializer<'de> {
struct SignedHexVisitor;
impl<'de> serde::de::Visitor<'de> for SignedHexVisitor {
impl serde::de::Visitor<'_> for SignedHexVisitor {
type Value = i64;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
@ -860,7 +864,7 @@ impl<'de> serde::Deserialize<'de> for SectionAddressRef {
where D: serde::Deserializer<'de> {
struct SectionAddressRefVisitor;
impl<'de> serde::de::Visitor<'de> for SectionAddressRefVisitor {
impl serde::de::Visitor<'_> for SectionAddressRefVisitor {
type Value = SectionAddressRef;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {

View File

@ -209,7 +209,7 @@ fn print_line(ins_diff: &ObjInsDiff, base_addr: u64) -> Vec<Span> {
pad_to = 5;
}
DiffText::Address(addr) => {
label_text = format!("{:x}:", addr);
label_text = format!("{addr:x}:");
pad_to = 5;
}
DiffText::Opcode(mnemonic, _op) => {

File diff suppressed because it is too large Load Diff

View File

@ -358,6 +358,7 @@ pub struct Tag {
pub kind: TagKind,
pub is_erased: bool, // Tag was deleted but has been reconstructed
pub is_erased_root: bool, // Tag is erased and is the root of a tree of erased tags
pub data_endian: Endian, // Endianness of the tag data (could be different from the address endianness for erased tags)
pub attributes: Vec<Attribute>,
}
@ -554,6 +555,7 @@ where
kind: TagKind::Padding,
is_erased,
is_erased_root: false,
data_endian,
attributes: Vec::new(),
});
return Ok(tags);
@ -563,26 +565,42 @@ where
let tag = TagKind::try_from(tag_num).context("Unknown DWARF tag type")?;
if tag == TagKind::Padding {
if include_erased {
// Erased entries that have become padding are little-endian, and we
// have to guess the length and tag of the first entry. We assume
// the entry is either a variable or a function, and read until we
// find the high_pc attribute. Only MwGlobalRef will follow, and
// these are unlikely to be confused with the length of the next
// entry.
// Erased entries that have become padding could be either
// little-endian or big-endian, and we have to guess the length and
// tag of the first entry. We assume the entry is either a variable
// or a function, and read until we find the high_pc attribute. Only
// MwGlobalRef will follow, and these are unlikely to be confused
// with the length of the next entry.
let mut attributes = Vec::new();
let mut is_function = false;
// Guess endianness based on first attribute
let data_endian = if is_erased {
data_endian
} else {
// Peek next two bytes
let mut buf = [0u8; 2];
reader.read_exact(&mut buf)?;
let attr_tag = u16::from_reader(&mut Cursor::new(&buf), data_endian)?;
reader.seek(SeekFrom::Current(-2))?;
match AttributeKind::try_from(attr_tag) {
Ok(_) => data_endian,
Err(_) => data_endian.flip(),
}
};
while reader.stream_position()? < position + size as u64 {
// Peek next two bytes
let mut buf = [0u8; 2];
reader.read_exact(&mut buf)?;
let attr_tag = u16::from_reader(&mut Cursor::new(&buf), Endian::Little)?;
let attr_tag = u16::from_reader(&mut Cursor::new(&buf), data_endian)?;
reader.seek(SeekFrom::Current(-2))?;
if is_function && attr_tag != AttributeKind::MwGlobalRef as u16 {
break;
}
let attr = read_attribute(reader, Endian::Little, addr_endian)?;
let attr = read_attribute(reader, data_endian, addr_endian)?;
if attr.kind == AttributeKind::HighPc {
is_function = true;
}
@ -594,12 +612,13 @@ where
kind,
is_erased: true,
is_erased_root: true,
data_endian,
attributes,
});
// Read the rest of the tags
while reader.stream_position()? < position + size as u64 {
for tag in read_tags(reader, Endian::Little, addr_endian, include_erased, true)? {
for tag in read_tags(reader, data_endian, addr_endian, include_erased, true)? {
tags.push(tag);
}
}
@ -616,6 +635,7 @@ where
kind: tag,
is_erased,
is_erased_root: false,
data_endian,
attributes,
});
}
@ -1145,8 +1165,8 @@ fn structure_type_string(
struct_def_string(info, typedefs, t)?
} else if include_keyword {
match t.kind {
StructureKind::Struct => format!("struct {}", name),
StructureKind::Class => format!("class {}", name),
StructureKind::Struct => format!("struct {name}"),
StructureKind::Class => format!("class {name}"),
}
} else {
name.clone()
@ -1178,7 +1198,7 @@ fn enumeration_type_string(
if name.starts_with('@') {
enum_def_string(t)?
} else if include_keyword {
format!("enum {}", name)
format!("enum {name}")
} else {
name.clone()
}
@ -1203,7 +1223,7 @@ fn union_type_string(
if name.starts_with('@') {
union_def_string(info, typedefs, t)?
} else if include_keyword {
format!("union {}", name)
format!("union {name}")
} else {
name.clone()
}
@ -1306,7 +1326,7 @@ pub fn subroutine_type_string(
write!(parameters, "{}{}", ts.prefix, ts.suffix)?;
}
if let Some(location) = &parameter.location {
write!(parameters, " /* {} */", location)?;
write!(parameters, " /* {location} */")?;
}
}
if t.var_args {
@ -1322,7 +1342,7 @@ pub fn subroutine_type_string(
let base_name = tag
.string_attribute(AttributeKind::Name)
.ok_or_else(|| anyhow!("member_of tag {} has no name attribute", member_of))?;
out.member = format!("{}::", base_name);
out.member = format!("{base_name}::");
}
Ok(out)
}
@ -1337,7 +1357,7 @@ pub fn subroutine_def_string(
if is_erased {
out.push_str("// Erased\n");
} else if let (Some(start), Some(end)) = (t.start_address, t.end_address) {
writeln!(out, "// Range: {:#X} -> {:#X}", start, end)?;
writeln!(out, "// Range: {start:#X} -> {end:#X}")?;
}
let rt = type_string(info, typedefs, &t.return_type, true)?;
if t.local {
@ -1361,15 +1381,15 @@ pub fn subroutine_def_string(
let base_name = tag
.string_attribute(AttributeKind::Name)
.ok_or_else(|| anyhow!("member_of tag {} has no name attribute", member_of))?;
write!(out, "{}::", base_name)?;
write!(out, "{base_name}::")?;
// Handle constructors and destructors
if let Some(name) = t.name.as_ref() {
if name == "__dt" {
write!(out, "~{}", base_name)?;
write!(out, "~{base_name}")?;
name_written = true;
} else if name == "__ct" {
write!(out, "{}", base_name)?;
write!(out, "{base_name}")?;
name_written = true;
}
}
@ -1398,7 +1418,7 @@ pub fn subroutine_def_string(
write!(parameters, "{}{}", ts.prefix, ts.suffix)?;
}
if let Some(location) = &parameter.location {
write!(parameters, " /* {} */", location)?;
write!(parameters, " /* {location} */")?;
}
}
if t.var_args {
@ -1420,7 +1440,7 @@ pub fn subroutine_def_string(
ts.suffix
)?;
if let Some(location) = &variable.location {
write!(var_out, " // {}", location)?;
write!(var_out, " // {location}")?;
}
writeln!(var_out)?;
}
@ -1435,7 +1455,7 @@ pub fn subroutine_def_string(
.get(&reference)
.ok_or_else(|| anyhow!("Failed to locate reference tag {}", reference))?;
if tag.kind == TagKind::Padding {
writeln!(out, " // -> ??? ({})", reference)?;
writeln!(out, " // -> ??? ({reference})")?;
continue;
}
let variable = process_variable_tag(info, tag)?;
@ -1477,13 +1497,13 @@ fn subroutine_block_string(
) -> Result<String> {
let mut out = String::new();
if let Some(name) = &block.name {
write!(out, "{}: ", name)?;
write!(out, "{name}: ")?;
} else {
out.push_str("/* anonymous block */ ");
}
out.push_str("{\n");
if let (Some(start), Some(end)) = (block.start_address, block.end_address) {
writeln!(out, " // Range: {:#X} -> {:#X}", start, end)?;
writeln!(out, " // Range: {start:#X} -> {end:#X}")?;
}
let mut var_out = String::new();
for variable in &block.variables {
@ -1496,7 +1516,7 @@ fn subroutine_block_string(
ts.suffix
)?;
if let Some(location) = &variable.location {
write!(var_out, " // {}", location)?;
write!(var_out, " // {location}")?;
}
writeln!(var_out)?;
}
@ -1635,9 +1655,9 @@ pub fn struct_def_string(
};
if let Some(name) = t.name.as_ref() {
if name.starts_with('@') {
write!(out, " /* {} */", name)?;
write!(out, " /* {name} */")?;
} else {
write!(out, " {}", name)?;
write!(out, " {name}")?;
}
}
let mut wrote_base = false;
@ -1665,7 +1685,7 @@ pub fn struct_def_string(
}
out.push_str(" {\n");
if let Some(byte_size) = t.byte_size {
writeln!(out, " // total size: {:#X}", byte_size)?;
writeln!(out, " // total size: {byte_size:#X}")?;
}
let mut vis = match t.kind {
StructureKind::Struct => Visibility::Public,
@ -1751,9 +1771,9 @@ pub fn enum_def_string(t: &EnumerationType) -> Result<String> {
let mut out = match t.name.as_ref() {
Some(name) => {
if name.starts_with('@') {
format!("enum /* {} */ {{\n", name)
format!("enum /* {name} */ {{\n")
} else {
format!("enum {} {{\n", name)
format!("enum {name} {{\n")
}
}
None => "enum {\n".to_string(),
@ -1769,9 +1789,9 @@ pub fn union_def_string(info: &DwarfInfo, typedefs: &TypedefMap, t: &UnionType)
let mut out = match t.name.as_ref() {
Some(name) => {
if name.starts_with('@') {
format!("union /* {} */ {{\n", name)
format!("union /* {name} */ {{\n")
} else {
format!("union {} {{\n", name)
format!("union {name} {{\n")
}
}
None => "union {\n".to_string(),
@ -2028,9 +2048,9 @@ fn process_array_tag(info: &DwarfInfo, tag: &Tag) -> Result<ArrayType> {
(AttributeKind::Sibling, _) => {}
(AttributeKind::SubscrData, AttributeValue::Block(data)) => {
subscr_data =
Some(process_array_subscript_data(data, info.e, tag.is_erased).with_context(
|| format!("Failed to process SubscrData for tag: {:?}", tag),
)?)
Some(process_array_subscript_data(data, info.e).with_context(|| {
format!("Failed to process SubscrData for tag: {tag:?}")
})?)
}
(AttributeKind::Ordering, val) => match val {
AttributeValue::Data2(d2) => {
@ -2056,11 +2076,7 @@ fn process_array_tag(info: &DwarfInfo, tag: &Tag) -> Result<ArrayType> {
Ok(ArrayType { element_type: Box::from(element_type), dimensions })
}
fn process_array_subscript_data(
data: &[u8],
e: Endian,
is_erased: bool,
) -> Result<(Type, Vec<ArrayDimension>)> {
fn process_array_subscript_data(data: &[u8], e: Endian) -> Result<(Type, Vec<ArrayDimension>)> {
let mut element_type = None;
let mut dimensions = Vec::new();
let mut data = data;
@ -2101,8 +2117,7 @@ fn process_array_subscript_data(
SubscriptFormat::ElementType => {
let mut cursor = Cursor::new(data);
// TODO: is this the right endianness to use for erased tags?
let type_attr =
read_attribute(&mut cursor, if is_erased { Endian::Little } else { e }, e)?;
let type_attr = read_attribute(&mut cursor, e, e)?;
element_type = Some(process_type(&type_attr, e)?);
data = &data[cursor.position() as usize..];
}
@ -2456,10 +2471,7 @@ fn process_subroutine_parameter_tag(info: &DwarfInfo, tag: &Tag) -> Result<Subro
) => kind = Some(process_type(attr, info.e)?),
(AttributeKind::Location, AttributeValue::Block(block)) => {
if !block.is_empty() {
location = Some(process_variable_location(
block,
if tag.is_erased { Endian::Little } else { info.e },
)?);
location = Some(process_variable_location(block, tag.data_endian)?);
}
}
(AttributeKind::MwDwarf2Location, AttributeValue::Block(_block)) => {
@ -2514,10 +2526,7 @@ fn process_local_variable_tag(info: &DwarfInfo, tag: &Tag) -> Result<SubroutineV
) => kind = Some(process_type(attr, info.e)?),
(AttributeKind::Location, AttributeValue::Block(block)) => {
if !block.is_empty() {
location = Some(process_variable_location(
block,
if tag.is_erased { Endian::Little } else { info.e },
)?);
location = Some(process_variable_location(block, tag.data_endian)?);
}
}
(AttributeKind::MwDwarf2Location, AttributeValue::Block(_block)) => {
@ -2615,13 +2624,13 @@ pub fn process_type(attr: &Attribute, e: Endian) -> Result<Type> {
match (attr.kind, &attr.value) {
(AttributeKind::FundType, &AttributeValue::Data2(type_id)) => {
let fund_type = FundType::parse_int(type_id)
.with_context(|| format!("Invalid fundamental type ID '{:04X}'", type_id))?;
.with_context(|| format!("Invalid fundamental type ID '{type_id:04X}'"))?;
Ok(Type { kind: TypeKind::Fundamental(fund_type), modifiers: vec![] })
}
(AttributeKind::ModFundType, AttributeValue::Block(ops)) => {
let type_id = u16::from_bytes(ops[ops.len() - 2..].try_into()?, e);
let fund_type = FundType::parse_int(type_id)
.with_context(|| format!("Invalid fundamental type ID '{:04X}'", type_id))?;
.with_context(|| format!("Invalid fundamental type ID '{type_id:04X}'"))?;
let modifiers = process_modifiers(&ops[..ops.len() - 2])?;
Ok(Type { kind: TypeKind::Fundamental(fund_type), modifiers })
}
@ -2762,7 +2771,7 @@ pub fn tag_type_string(
match ud {
UserDefinedType::Structure(_)
| UserDefinedType::Enumeration(_)
| UserDefinedType::Union(_) => Ok(format!("{};", ud_str)),
| UserDefinedType::Union(_) => Ok(format!("{ud_str};")),
_ => Ok(ud_str),
}
}
@ -2789,9 +2798,9 @@ fn variable_string(
out.push(';');
if include_extra {
let size = variable.kind.size(info)?;
out.push_str(&format!(" // size: {:#X}", size));
out.push_str(&format!(" // size: {size:#X}"));
if let Some(addr) = variable.address {
out.push_str(&format!(", address: {:#X}", addr));
out.push_str(&format!(", address: {addr:#X}"));
}
}
Ok(out)

View File

@ -17,10 +17,10 @@ use object::{
elf::{ProgramHeader, Rel, SectionHeader, SectionIndex, SymbolIndex, Writer},
StringId,
},
Architecture, Endianness, Object, ObjectKind, ObjectSection, ObjectSymbol, Relocation,
Architecture, Endianness, File, Object, ObjectKind, ObjectSection, ObjectSymbol, Relocation,
RelocationFlags, RelocationTarget, SectionKind, Symbol, SymbolKind, SymbolScope, SymbolSection,
};
use typed_path::Utf8NativePath;
use typed_path::{Utf8NativePath, Utf8NativePathBuf};
use crate::{
array_ref,
@ -49,7 +49,7 @@ enum BoundaryState {
pub fn process_elf(path: &Utf8NativePath) -> Result<ObjInfo> {
let mut file = open_file(path, true)?;
let obj_file = object::read::File::parse(file.map()?)?;
let obj_file = File::parse(file.map()?)?;
let architecture = match obj_file.architecture() {
Architecture::PowerPc => ObjArchitecture::PowerPc,
arch => bail!("Unexpected architecture: {arch:?}"),
@ -106,49 +106,14 @@ pub fn process_elf(path: &Utf8NativePath) -> Result<ObjInfo> {
});
}
let mw_comment = if let Some(comment_section) = obj_file.section_by_name(".comment") {
let data = comment_section.uncompressed_data()?;
if data.is_empty() {
None
} else {
let mut reader = Cursor::new(&*data);
let header = MWComment::from_reader(&mut reader, Endian::Big)
.context("While reading .comment section")?;
log::debug!("Loaded .comment section header {:?}", header);
let mut comment_syms = Vec::with_capacity(obj_file.symbols().count());
comment_syms.push(CommentSym::from_reader(&mut reader, Endian::Big)?); // ELF null symbol
for symbol in obj_file.symbols() {
let comment_sym = CommentSym::from_reader(&mut reader, Endian::Big)?;
log::debug!("Symbol {:?} -> Comment {:?}", symbol, comment_sym);
comment_syms.push(comment_sym);
}
ensure!(
data.len() - reader.position() as usize == 0,
".comment section data not fully read"
);
Some((header, comment_syms))
}
} else {
let mw_comment = load_comment(&obj_file).unwrap_or_else(|e| {
log::warn!("Failed to read .comment section: {e:#}");
None
};
let split_meta = if let Some(split_meta_section) = obj_file.section_by_name(SPLITMETA_SECTION) {
let data = split_meta_section.uncompressed_data()?;
if data.is_empty() {
None
} else {
let metadata = SplitMeta::from_section(
split_meta_section,
obj_file.endianness(),
obj_file.is_64(),
)
.context("While reading .note.split section")?;
log::debug!("Loaded .note.split section");
Some(metadata)
}
} else {
});
let split_meta = load_split_meta(&obj_file).unwrap_or_else(|e| {
log::warn!("Failed to read .note.split section: {e:#}");
None
};
});
let mut symbols: Vec<ObjSymbol> = vec![];
let mut symbol_indexes: Vec<Option<ObjSymbolIndex>> = vec![None /* ELF null symbol */];
@ -199,7 +164,7 @@ pub fn process_elf(path: &Utf8NativePath) -> Result<ObjInfo> {
hash_map::Entry::Vacant(e) => e.insert(0),
};
*index += 1;
let new_name = format!("{}_{}", file_name, index);
let new_name = format!("{file_name}_{index}");
// log::info!("Renaming {} to {}", file_name, new_name);
file_name.clone_from(&new_name);
match section_starts.entry(new_name.clone()) {
@ -310,8 +275,8 @@ pub fn process_elf(path: &Utf8NativePath) -> Result<ObjInfo> {
continue;
}
symbol_indexes.push(Some(symbols.len() as ObjSymbolIndex));
let align = mw_comment.as_ref().map(|(_, vec)| vec[symbol.index().0].align);
symbols.push(to_obj_symbol(&obj_file, &symbol, &section_indexes, align)?);
let comment_sym = mw_comment.as_ref().map(|(_, vec)| &vec[symbol.index().0 - 1]);
symbols.push(to_obj_symbol(&obj_file, &symbol, &section_indexes, comment_sym)?);
}
let mut link_order = Vec::<ObjUnit>::new();
@ -384,6 +349,42 @@ pub fn process_elf(path: &Utf8NativePath) -> Result<ObjInfo> {
Ok(obj)
}
fn load_split_meta(obj_file: &File) -> Result<Option<SplitMeta>> {
let Some(split_meta_section) = obj_file.section_by_name(SPLITMETA_SECTION) else {
return Ok(None);
};
let data = split_meta_section.uncompressed_data()?;
if data.is_empty() {
return Ok(None);
}
let metadata =
SplitMeta::from_section(split_meta_section, obj_file.endianness(), obj_file.is_64())?;
log::debug!("Loaded .note.split section");
Ok(Some(metadata))
}
fn load_comment(obj_file: &File) -> Result<Option<(MWComment, Vec<CommentSym>)>> {
let Some(comment_section) = obj_file.section_by_name(".comment") else {
return Ok(None);
};
let data = comment_section.uncompressed_data()?;
if data.is_empty() {
return Ok(None);
}
let mut reader = Cursor::new(&*data);
let header = MWComment::from_reader(&mut reader, Endian::Big)?;
log::debug!("Loaded .comment section header {:?}", header);
CommentSym::from_reader(&mut reader, Endian::Big)?; // Null symbol
let mut comment_syms = Vec::with_capacity(obj_file.symbols().count());
for symbol in obj_file.symbols() {
let comment_sym = CommentSym::from_reader(&mut reader, Endian::Big)?;
log::debug!("Symbol {:?} -> Comment {:?}", symbol, comment_sym);
comment_syms.push(comment_sym);
}
ensure!(data.len() == reader.position() as usize, "Section data not fully read");
Ok(Some((header, comment_syms)))
}
pub fn write_elf(obj: &ObjInfo, export_all: bool) -> Result<Vec<u8>> {
let mut out_data = Vec::new();
let mut writer = Writer::new(Endianness::Big, false, &mut out_data);
@ -861,7 +862,7 @@ fn to_obj_symbol(
obj_file: &object::File<'_>,
symbol: &Symbol<'_, '_>,
section_indexes: &[Option<usize>],
align: Option<u32>,
comment_sym: Option<&CommentSym>,
) -> Result<ObjSymbol> {
let section = match symbol.section_index() {
Some(idx) => Some(obj_file.section_by_index(idx)?),
@ -891,6 +892,9 @@ fn to_obj_symbol(
if symbol.scope() == SymbolScope::Linkage {
flags = ObjSymbolFlagSet(flags.0 | ObjSymbolFlags::Hidden);
}
if comment_sym.is_some_and(|c| c.active_flags & 0x8 != 0) {
flags = ObjSymbolFlagSet(flags.0 | ObjSymbolFlags::Exported);
}
let section_idx = section.as_ref().and_then(|section| section_indexes[section.index().0]);
Ok(ObjSymbol {
name: name.to_string(),
@ -907,7 +911,7 @@ fn to_obj_symbol(
SymbolKind::Section => ObjSymbolKind::Section,
_ => bail!("Unsupported symbol kind: {:?}", symbol),
},
align,
align: comment_sym.map(|c| c.align),
..Default::default()
})
}
@ -1005,3 +1009,10 @@ fn write_relocatable_section_data(w: &mut Writer, section: &ObjSection) -> Resul
w.write(&section.data[current_address..]);
Ok(())
}
pub fn is_elf_file(path: &Utf8NativePathBuf) -> Result<bool> {
let mut file = open_file(path, true)?;
let mut magic = [0; 4];
file.read_exact(&mut magic)?;
Ok(magic == elf::ELFMAG)
}

57
src/util/extab.rs Normal file
View File

@ -0,0 +1,57 @@
use anyhow::{Context, Result};
use itertools::Itertools;
use crate::obj::ObjInfo;
pub fn clean_extab(obj: &mut ObjInfo, mut padding: impl Iterator<Item = u8>) -> Result<usize> {
let (extab_section_index, extab_section) = obj
.sections
.iter_mut()
.find(|(_, s)| s.name == "extab")
.ok_or_else(|| anyhow::anyhow!("No extab section found"))?;
let mut num_cleaned = 0;
for (_symbol_index, symbol) in obj
.symbols
.for_section(extab_section_index)
.filter(|(_, s)| s.size > 0)
.sorted_by_key(|(_, s)| s.address)
{
let data = extab_section.symbol_data(symbol)?;
let decoded = cwextab::decode_extab(data).with_context(|| {
format!(
"Failed to decode {} (extab {:#010X}..{:#010X})",
symbol.name,
symbol.address,
symbol.address + symbol.size
)
})?;
let mut updated = false;
for action in &decoded.exception_actions {
// Check if the current action has padding
if let Some(padding_offset) = action.get_struct_padding_offset() {
let index = padding_offset as usize;
let section_offset = (symbol.address - extab_section.address) as usize
+ action.action_offset as usize;
let mut clean_data: Vec<u8> = action.get_exaction_bytes(false);
// Write the two padding bytes
clean_data[index] = padding.next().unwrap_or(0);
clean_data[index + 1] = padding.next().unwrap_or(0);
let orig_data =
&mut extab_section.data[section_offset..section_offset + clean_data.len()];
orig_data.copy_from_slice(&clean_data);
updated = true;
}
}
if updated {
tracing::debug!(
"Replaced uninitialized bytes in {} (extab {:#010X}..{:#010X})",
symbol.name,
symbol.address,
symbol.address + symbol.size
);
num_cleaned += 1;
}
}
Ok(num_cleaned)
}

View File

@ -26,7 +26,7 @@ pub fn buf_writer(path: &Utf8NativePath) -> Result<BufWriter<File>> {
if let Some(parent) = path.parent() {
DirBuilder::new().recursive(true).create(parent)?;
}
let file = File::create(path).with_context(|| format!("Failed to create file '{}'", path))?;
let file = File::create(path).with_context(|| format!("Failed to create file '{path}'"))?;
Ok(BufWriter::new(file))
}

View File

@ -47,11 +47,11 @@ pub fn generate_ldscript(
let out = template
.unwrap_or(LCF_TEMPLATE)
.replace("$ORIGIN", &format!("{:#X}", origin))
.replace("$ORIGIN", &format!("{origin:#X}"))
.replace("$SECTIONS", &section_defs)
.replace("$LAST_SECTION_SYMBOL", &last_section_symbol)
.replace("$LAST_SECTION_NAME", &last_section_name)
.replace("$STACKSIZE", &format!("{:#X}", stack_size))
.replace("$STACKSIZE", &format!("{stack_size:#X}"))
.replace("$FORCEACTIVE", &force_active.join("\n "))
.replace("$ARENAHI", &format!("{:#X}", obj.arena_hi.unwrap_or(0x81700000)));
Ok(out)
@ -74,7 +74,7 @@ pub fn generate_ldscript_partial(
// Some RELs have no entry point (`.text` was stripped) so mwld requires at least an empty
// `.init` section to be present in the linker script, for some reason.
if obj.entry.is_none() {
section_defs = format!(".init :{{}}\n {}", section_defs);
section_defs = format!(".init :{{}}\n {section_defs}");
}
let mut force_files = Vec::with_capacity(obj.link_order.len());

View File

@ -802,10 +802,7 @@ pub fn apply_map(mut result: MapInfo, obj: &mut ObjInfo) -> Result<()> {
}
ObjSectionKind::Data | ObjSectionKind::ReadOnlyData => {
if section.elf_index == 4 {
if result
.section_symbols
.get(".rodata")
.map_or(false, |m| !m.is_empty())
if result.section_symbols.get(".rodata").is_some_and(|m| !m.is_empty())
{
".rodata"
} else {

View File

@ -10,6 +10,7 @@ pub mod diff;
pub mod dol;
pub mod dwarf;
pub mod elf;
pub mod extab;
pub mod file;
pub mod lcf;
pub mod map;
@ -18,13 +19,16 @@ pub mod nested;
pub mod nlzss;
pub mod path;
pub mod rarc;
pub mod read;
pub mod reader;
pub mod rel;
pub mod rso;
pub mod signatures;
pub mod split;
pub mod take_seek;
pub mod toposort;
pub mod u8_arc;
pub mod wad;
#[inline]
pub const fn align_up(value: u32, align: u32) -> u32 { (value + (align - 1)) & !(align - 1) }
@ -92,7 +96,7 @@ pub enum Bytes<'a> {
Owned(Box<[u8]>),
}
impl<'a> Bytes<'a> {
impl Bytes<'_> {
pub fn into_owned(self) -> Box<[u8]> {
match self {
Bytes::Borrowed(s) => Box::from(s),
@ -101,7 +105,7 @@ impl<'a> Bytes<'a> {
}
}
impl<'a> AsRef<[u8]> for Bytes<'a> {
impl AsRef<[u8]> for Bytes<'_> {
fn as_ref(&self) -> &[u8] {
match self {
Bytes::Borrowed(s) => s,

View File

@ -209,7 +209,7 @@ impl<'a> RarcView<'a> {
)
})?;
let c_string = CStr::from_bytes_until_nul(name_buf)
.map_err(|_| format!("RARC: name at offset {} not null-terminated", offset))?;
.map_err(|_| format!("RARC: name at offset {offset} not null-terminated"))?;
Ok(c_string.to_string_lossy())
}

26
src/util/read.rs Normal file
View File

@ -0,0 +1,26 @@
use std::{io, io::Read};
use zerocopy::{FromBytes, FromZeros, IntoBytes};
#[inline(always)]
pub fn read_from<T, R>(reader: &mut R) -> io::Result<T>
where
T: FromBytes + IntoBytes,
R: Read + ?Sized,
{
let mut ret = <T>::new_zeroed();
reader.read_exact(ret.as_mut_bytes())?;
Ok(ret)
}
#[inline(always)]
pub fn read_box_slice<T, R>(reader: &mut R, count: usize) -> io::Result<Box<[T]>>
where
T: FromBytes + IntoBytes,
R: Read + ?Sized,
{
let mut ret = <[T]>::new_box_zeroed_with_elems(count)
.map_err(|_| io::Error::from(io::ErrorKind::OutOfMemory))?;
reader.read_exact(ret.as_mut().as_mut_bytes())?;
Ok(ret)
}

View File

@ -20,6 +20,15 @@ impl From<object::Endianness> for Endian {
}
}
impl Endian {
pub fn flip(self) -> Self {
match self {
Endian::Big => Endian::Little,
Endian::Little => Endian::Big,
}
}
}
pub const DYNAMIC_SIZE: usize = 0;
pub const fn struct_size<const N: usize>(fields: [usize; N]) -> usize {
@ -263,6 +272,18 @@ impl ToWriter for Vec<u8> {
fn write_size(&self) -> usize { self.len() }
}
impl<const N: usize> ToWriter for [u32; N] {
fn to_writer<W>(&self, writer: &mut W, e: Endian) -> io::Result<()>
where W: Write + ?Sized {
for &value in self {
value.to_writer(writer, e)?;
}
Ok(())
}
fn write_size(&self) -> usize { N * u32::STATIC_SIZE }
}
pub fn write_vec<T, W>(writer: &mut W, vec: &[T], e: Endian) -> io::Result<()>
where
T: ToWriter,

View File

@ -364,7 +364,7 @@ where
reader.seek(SeekFrom::Start(header.section_info_offset as u64))?;
for idx in 0..header.num_sections {
let section = RelSectionHeader::from_reader(reader, Endian::Big)
.with_context(|| format!("Failed to read REL section header {}", idx))?;
.with_context(|| format!("Failed to read REL section header {idx}"))?;
sections.push(section);
}
Ok(sections)
@ -390,7 +390,7 @@ where R: Read + Seek + ?Sized {
reader.seek(SeekFrom::Start(offset as u64))?;
let mut data = vec![0u8; size as usize];
reader.read_exact(&mut data).with_context(|| {
format!("Failed to read REL section {} data with size {:#X}", idx, size)
format!("Failed to read REL section {idx} data with size {size:#X}")
})?;
reader.seek(SeekFrom::Start(position))?;
data
@ -405,7 +405,7 @@ where R: Read + Seek + ?Sized {
text_section = Some(idx as u8);
(".text".to_string(), ObjSectionKind::Code, true)
} else {
(format!(".section{}", idx), ObjSectionKind::Data, false)
(format!(".section{idx}"), ObjSectionKind::Data, false)
};
sections.push(ObjSection {
name,

View File

@ -147,14 +147,14 @@ impl FromReader for RsoHeader {
if next != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Expected 'next' to be 0, got {:#X}", next),
format!("Expected 'next' to be 0, got {next:#X}"),
));
}
let prev = u32::from_reader(reader, e)?;
if prev != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Expected 'prev' to be 0, got {:#X}", prev),
format!("Expected 'prev' to be 0, got {prev:#X}"),
));
}
let num_sections = u32::from_reader(reader, e)?;
@ -170,7 +170,7 @@ impl FromReader for RsoHeader {
if bss_section != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Expected 'bssSection' to be 0, got {:#X}", bss_section),
format!("Expected 'bssSection' to be 0, got {bss_section:#X}"),
));
}
let prolog_offset = u32::from_reader(reader, e)?;
@ -440,7 +440,7 @@ where R: Read + Seek + ?Sized {
// println!("Section {} offset {:#X} size {:#X}", idx, offset, size);
sections.push(ObjSection {
name: format!(".section{}", idx),
name: format!(".section{idx}"),
kind: if offset == 0 {
ObjSectionKind::Bss
} else if section.exec() {

View File

@ -6,7 +6,6 @@ use std::{
use anyhow::{anyhow, bail, ensure, Context, Result};
use itertools::Itertools;
use objdiff_core::obj::split_meta::SplitMeta;
use petgraph::{graph::NodeIndex, Graph};
use sanitise_file_name::sanitize_with_options;
use tracing_attributes::instrument;
@ -17,7 +16,7 @@ use crate::{
ObjSplit, ObjSymbol, ObjSymbolFlagSet, ObjSymbolFlags, ObjSymbolKind, ObjSymbolScope,
ObjUnit, SectionIndex, SymbolIndex,
},
util::{align_up, comment::MWComment},
util::{align_up, comment::MWComment, toposort::toposort},
};
/// Create splits for function pointers in the given section.
@ -27,7 +26,25 @@ fn split_ctors_dtors(obj: &mut ObjInfo, start: SectionAddress, end: SectionAddre
let mut current_address = start;
let mut referenced_symbols = vec![];
// ProDG ctor list can start with -1
if matches!(read_u32(ctors_section, current_address.address), Some(0xFFFFFFFF)) {
current_address += 4;
}
while current_address < end {
// ProDG hack when the end address is not known
if matches!(read_u32(ctors_section, current_address.address), Some(0)) {
while current_address < end {
ensure!(
matches!(read_u32(ctors_section, current_address.address), Some(0)),
"{} data detected at {:#010X} after null pointer",
ctors_section.name,
current_address,
);
current_address += 4;
}
break;
}
let function_addr = read_address(obj, ctors_section, current_address.address)?;
log::debug!("Found {} entry: {:#010X}", ctors_section.name, function_addr);
@ -386,6 +403,20 @@ fn create_gap_splits(obj: &mut ObjInfo) -> Result<()> {
new_split_end.address = symbol.address as u32;
break;
}
// Create a new split if we need to adjust the alignment
if let Some(align) = symbol.align {
if current_address & (align - 1) != 0 {
log::debug!(
"Auto split {:#010X}..{:#010X} for symbol {} with alignment {}",
current_address,
symbol.address,
symbol.name,
align
);
new_split_end.address = symbol.address as u32;
break;
}
}
}
ensure!(
@ -530,8 +561,9 @@ fn validate_splits(obj: &ObjInfo) -> Result<()> {
/// Add padding symbols to fill in gaps between splits and symbols.
fn add_padding_symbols(obj: &mut ObjInfo) -> Result<()> {
for (section_index, section, addr, _split) in obj.sections.all_splits() {
if section.name == ".ctors" || section.name == ".dtors" {
let mut splits = obj.sections.all_splits().peekable();
while let Some((section_index, section, addr, split)) = splits.next() {
if section.name == ".ctors" || section.name == ".dtors" || addr == split.end {
continue;
}
@ -545,19 +577,32 @@ fn add_padding_symbols(obj: &mut ObjInfo) -> Result<()> {
})?
.is_none()
{
let next_split_address = splits
.peek()
.filter(|(i, _, _, _)| *i == section_index)
.map(|(_, _, addr, _)| *addr as u64)
.unwrap_or(section.address + section.size);
let next_symbol_address = obj
.symbols
.for_section_range(section_index, addr + 1..)
.for_section_range(section_index, addr + 1..next_split_address as u32)
.find(|&(_, s)| s.size_known && s.size > 0)
.map(|(_, s)| s.address)
.unwrap_or(section.address + section.size);
.unwrap_or(next_split_address);
if next_symbol_address <= addr as u64 {
continue;
}
let symbol_name = format!(
"pad_{:02}_{:08X}_{}",
section_index,
addr,
section.name.trim_start_matches('.')
);
log::debug!("Adding padding symbol {} at {:#010X}", symbol_name, addr);
log::debug!(
"Adding padding symbol {} at {:#010X}-{:#010X}",
symbol_name,
addr,
next_symbol_address
);
obj.symbols.add_direct(ObjSymbol {
name: symbol_name,
address: addr as u64,
@ -602,20 +647,56 @@ fn add_padding_symbols(obj: &mut ObjInfo) -> Result<()> {
if let Some(&(_, next_symbol)) = iter.peek() {
(
next_symbol.name.as_str(),
next_symbol.address,
next_symbol.address + next_symbol.size,
next_symbol.address as u32,
(next_symbol.address + next_symbol.size) as u32,
next_symbol.align.unwrap_or(1),
)
} else {
(
section.name.as_str(),
section.address + section.size,
section.address + section.size,
(section.address + section.size) as u32,
(section.address + section.size) as u32,
1,
)
};
let aligned_end = align_up((symbol.address + symbol.size) as u32, next_align);
match aligned_end.cmp(&(next_address as u32)) {
// Check if symbol is missing data between the end of the symbol and the next symbol
let symbol_end = (symbol.address + symbol.size) as u32;
if !matches!(section.kind, ObjSectionKind::Code | ObjSectionKind::Bss)
&& next_address > symbol_end
{
let data = section.data_range(symbol_end, next_address)?;
if data.iter().any(|&x| x != 0) {
log::debug!(
"Non-zero data between {:#010X}..{:#010X}, creating new symbol",
symbol_end,
next_address
);
let name = if obj.module_id == 0 {
format!("lbl_{symbol_end:08X}")
} else {
format!(
"lbl_{}_{}_{:X}",
obj.module_id,
section.name.trim_start_matches('.'),
symbol_end
)
};
to_add.push(ObjSymbol {
name,
address: symbol_end as u64,
section: Some(section_index),
size: (next_address - symbol_end) as u64,
size_known: true,
kind: ObjSymbolKind::Object,
..Default::default()
});
continue;
}
}
let aligned_end = align_up(symbol_end, next_align);
match aligned_end.cmp(&next_address) {
Ordering::Less => {
let symbol_name = format!(
"gap_{:02}_{:08X}_{}",
@ -628,7 +709,7 @@ fn add_padding_symbols(obj: &mut ObjInfo) -> Result<()> {
name: symbol_name,
address: aligned_end as u64,
section: Some(section_index),
size: next_address - aligned_end as u64,
size: (next_address - aligned_end) as u64,
size_known: true,
flags: ObjSymbolFlagSet(
ObjSymbolFlags::Global
@ -826,14 +907,16 @@ fn resolve_link_order(obj: &ObjInfo) -> Result<Vec<ObjUnit>> {
to: i64,
}
let mut graph = Graph::<String, SplitEdge>::new();
let mut unit_to_index_map = BTreeMap::<String, NodeIndex>::new();
let mut unit_to_index_map = BTreeMap::<&str, usize>::new();
let mut index_to_unit = vec![];
for (_, _, _, split) in obj.sections.all_splits() {
unit_to_index_map.insert(split.unit.clone(), NodeIndex::new(0));
}
for (unit, index) in unit_to_index_map.iter_mut() {
*index = graph.add_node(unit.clone());
unit_to_index_map.entry(split.unit.as_str()).or_insert_with(|| {
let idx = index_to_unit.len();
index_to_unit.push(split.unit.as_str());
idx
});
}
let mut graph = vec![vec![]; index_to_unit.len()];
for (_section_index, section) in obj.sections.iter() {
let mut iter = section.splits.iter().peekable();
@ -856,12 +939,9 @@ fn resolve_link_order(obj: &ObjInfo) -> Result<Vec<ObjUnit>> {
b.unit,
b_addr
);
let a_index = *unit_to_index_map.get(&a.unit).unwrap();
let b_index = *unit_to_index_map.get(&b.unit).unwrap();
graph.add_edge(a_index, b_index, SplitEdge {
from: a_addr as i64,
to: b_addr as i64,
});
let a_index = *unit_to_index_map.get(a.unit.as_str()).unwrap();
let b_index = *unit_to_index_map.get(b.unit.as_str()).unwrap();
graph[a_index].push(b_index);
}
}
}
@ -880,41 +960,22 @@ fn resolve_link_order(obj: &ObjInfo) -> Result<Vec<ObjUnit>> {
}
}
}
let mut iter = ordered_units
.into_iter()
.filter_map(|(order, unit)| unit_to_index_map.get(&unit).map(|&index| (order, index)))
.peekable();
while let (Some((a_order, a_index)), Some((b_order, b_index))) = (iter.next(), iter.peek()) {
graph.add_edge(a_index, *b_index, SplitEdge { from: a_order as i64, to: *b_order as i64 });
let mut iter =
ordered_units.values().filter_map(|unit| unit_to_index_map.get(unit.as_str())).peekable();
while let (Some(&a_index), Some(&&b_index)) = (iter.next(), iter.peek()) {
graph[a_index].push(b_index);
}
// use petgraph::{
// dot::{Config, Dot},
// graph::EdgeReference,
// };
// let get_edge_attributes = |_, e: EdgeReference<SplitEdge>| {
// let &SplitEdge { from, to } = e.weight();
// let section_name = &obj.section_at(from).unwrap().name;
// format!("label=\"{} {:#010X} -> {:#010X}\"", section_name, from, to)
// };
// let dot = Dot::with_attr_getters(
// &graph,
// &[Config::EdgeNoLabel, Config::NodeNoLabel],
// &get_edge_attributes,
// &|_, (_, s)| format!("label=\"{}\"", s),
// );
// println!("{:?}", dot);
match petgraph::algo::toposort(&graph, None) {
match toposort(&graph) {
Ok(vec) => Ok(vec
.iter()
.map(|&idx| {
let name = &graph[idx];
if let Some(existing) = obj.link_order.iter().find(|u| &u.name == name) {
let name = index_to_unit[idx];
if let Some(existing) = obj.link_order.iter().find(|u| u.name == name) {
existing.clone()
} else {
ObjUnit {
name: name.clone(),
name: name.to_string(),
autogenerated: obj.is_unit_autogenerated(name),
comment_version: None,
order: None,
@ -923,8 +984,8 @@ fn resolve_link_order(obj: &ObjInfo) -> Result<Vec<ObjUnit>> {
})
.collect_vec()),
Err(e) => Err(anyhow!(
"Cyclic dependency (involving {}) encountered while resolving link order",
graph[e.node_id()]
"Cyclic dependency encountered while resolving link order: {}",
e.iter().map(|&idx| index_to_unit[idx]).join(" -> ")
)),
}
}
@ -1034,13 +1095,15 @@ pub fn split_obj(obj: &ObjInfo, module_name: Option<&str>) -> Result<Vec<ObjInfo
}) as u64;
if current_address & (align as u32 - 1) != 0 {
log::warn!(
"Alignment for {} {} expected {}, but starts at {:#010X}",
split.unit,
section.name,
align,
current_address
);
if !split.autogenerated {
log::warn!(
"Alignment for {} {} expected {}, but starts at {:#010X}",
split.unit,
section.name,
align,
current_address
);
}
while align > 4 {
align /= 2;
if current_address & (align as u32 - 1) == 0 {
@ -1420,7 +1483,7 @@ fn auto_unit_name(
if unit_exists(&unit_name, obj, new_splits) {
let mut i = 1;
loop {
let new_unit_name = format!("{}_{}", unit_name, i);
let new_unit_name = format!("{unit_name}_{i}");
if !unit_exists(&new_unit_name, obj, new_splits) {
unit_name = new_unit_name;
break;

117
src/util/toposort.rs Normal file
View File

@ -0,0 +1,117 @@
/// Topological sort algorithm based on DFS:
/// https://en.wikipedia.org/wiki/Topological_sorting#Depth-first_search
/// Finds either an ordering of the vertices such that all edges go from
/// lower to higher indexed nodes, or a cycle.
///
/// Implementation by Simon Lindholm
/// https://gist.github.com/simonlindholm/08664dd783ad4b9f23532fdd5e352b42
pub fn toposort(graph: &[Vec<usize>]) -> Result<Vec<usize>, Vec<usize>> {
let n = graph.len();
#[derive(Copy, Clone)]
enum State {
Unvisited,
Active(usize, usize),
Finished,
}
let mut state = vec![State::Unvisited; n + 1];
state[n] = State::Active(0, usize::MAX);
let mut ret = Vec::new();
let mut cur = n;
loop {
let State::Active(eind, par) = state[cur] else { panic!("unexpected state 1") };
let adj;
if cur == n {
if eind == n {
break;
}
adj = eind;
} else {
if eind == graph[cur].len() {
state[cur] = State::Finished;
ret.push(cur);
cur = par;
continue;
}
adj = graph[cur][eind];
};
state[cur] = State::Active(eind + 1, par);
match state[adj] {
State::Unvisited => {
state[adj] = State::Active(0, cur);
cur = adj;
}
State::Active(..) => {
let mut cycle = Vec::new();
while cur != adj {
cycle.push(cur);
let State::Active(_, par) = state[cur] else { panic!("unexpected state 2") };
cur = par;
}
cycle.push(cur);
cycle.reverse();
return Err(cycle);
}
State::Finished => {}
};
}
ret.reverse();
Ok(ret)
}
#[cfg(test)]
mod tests {
use std::collections::HashSet;
use super::*;
#[test]
fn test_correct() {
let mut rng_state: usize = 0;
let mut rand = || {
rng_state = rng_state.wrapping_mul(123156351724123181_usize);
rng_state = rng_state.wrapping_add(670143798154186239_usize);
rng_state >> 32
};
for _ in 0..10000 {
let n = rand() % 20;
let mut g = vec![vec![]; n];
let mut g_set = HashSet::new();
if n != 0 {
let m = rand() % 50;
for _ in 0..m {
let a = rand() % n;
let b = rand() % n;
g[a].push(b);
g_set.insert((a, b));
}
}
match toposort(&g) {
Ok(order) => {
assert_eq!(order.len(), n);
let mut node_to_order = vec![usize::MAX; n];
// Every node should occur exactly once...
for (i, &node) in order.iter().enumerate() {
assert!(node < n);
assert_eq!(node_to_order[node], usize::MAX);
node_to_order[node] = i;
}
// and the edges should go in forward order in the list
for i in 0..n {
for &j in &g[i] {
assert!(node_to_order[i] < node_to_order[j]);
}
}
}
Err(cycle) => {
// The found cycle should exist in the graph
assert!(!cycle.is_empty());
for i in 0..cycle.len() {
let a = cycle[i];
let b = cycle[(i + 1) % cycle.len()];
assert!(g_set.contains(&(a, b)));
}
}
}
}
}
}

View File

@ -149,7 +149,7 @@ impl<'a> U8View<'a> {
let mut idx = 1;
let mut stop_at = None;
while let Some(node) = self.nodes.get(idx).copied() {
if self.get_name(node).map_or(false, |name| name.eq_ignore_ascii_case(current)) {
if self.get_name(node).is_ok_and(|name| name.eq_ignore_ascii_case(current)) {
current = next_non_empty(&mut split);
if current.is_empty() {
return Some((idx, node));

220
src/util/wad.rs Normal file
View File

@ -0,0 +1,220 @@
use std::{
io,
io::{BufRead, Read, Seek},
};
use aes::cipher::{BlockDecryptMut, KeyIvInit};
use anyhow::{bail, Result};
use nodtool::nod::{Ticket, TmdHeader};
use sha1::{Digest, Sha1};
use size::Size;
use zerocopy::{big_endian::*, FromBytes, FromZeros, Immutable, IntoBytes, KnownLayout};
use crate::{
array_ref_mut, static_assert,
util::read::{read_box_slice, read_from},
};
// TODO: other WAD types?
pub const WAD_MAGIC: [u8; 8] = [0x00, 0x00, 0x00, 0x20, 0x49, 0x73, 0x00, 0x00];
#[derive(Clone, Debug, PartialEq, FromBytes, IntoBytes, Immutable, KnownLayout)]
#[repr(C, align(4))]
pub struct WadHeader {
pub header_size: U32,
pub wad_type: [u8; 0x2],
pub wad_version: U16,
pub cert_chain_size: U32,
pub _reserved1: [u8; 0x4],
pub ticket_size: U32,
pub tmd_size: U32,
pub data_size: U32,
pub footer_size: U32,
}
static_assert!(size_of::<WadHeader>() == 0x20);
#[derive(Clone, Debug, PartialEq, FromBytes, IntoBytes, Immutable, KnownLayout)]
#[repr(C, align(4))]
pub struct ContentMetadata {
pub content_id: U32,
pub content_index: U16,
pub content_type: U16,
pub size: U64,
pub hash: HashBytes,
}
static_assert!(size_of::<ContentMetadata>() == 0x24);
impl ContentMetadata {
#[inline]
pub fn iv(&self) -> [u8; 0x10] {
let mut iv = [0u8; 0x10];
*array_ref_mut!(iv, 0, 2) = self.content_index.get().to_be_bytes();
iv
}
}
const ALIGNMENT: usize = 0x40;
#[inline(always)]
pub fn align_up(value: u64, alignment: u64) -> u64 { (value + alignment - 1) & !(alignment - 1) }
pub type HashBytes = [u8; 20];
pub type KeyBytes = [u8; 16];
type Aes128Cbc = cbc::Decryptor<aes::Aes128>;
#[derive(Debug, Clone)]
pub struct WadFile {
pub header: WadHeader,
pub title_key: KeyBytes,
pub fake_signed: bool,
pub raw_cert_chain: Box<[u8]>,
pub raw_ticket: Box<[u8]>,
pub raw_tmd: Box<[u8]>,
pub content_offset: u64,
}
impl WadFile {
pub fn ticket(&self) -> &Ticket {
Ticket::ref_from_bytes(&self.raw_ticket).expect("Invalid ticket alignment")
}
pub fn tmd(&self) -> &TmdHeader {
TmdHeader::ref_from_prefix(&self.raw_tmd).expect("Invalid TMD alignment").0
}
pub fn contents(&self) -> &[ContentMetadata] {
let (_, cmd_data) =
TmdHeader::ref_from_prefix(&self.raw_tmd).expect("Invalid TMD alignment");
<[ContentMetadata]>::ref_from_bytes(cmd_data).expect("Invalid CMD alignment")
}
pub fn content_offset(&self, content_index: u16) -> u64 {
let contents = self.contents();
let mut offset = self.content_offset;
for content in contents.iter().take(content_index as usize) {
offset = align_up(offset + content.size.get(), ALIGNMENT as u64);
}
offset
}
pub fn trailer_offset(&self) -> u64 {
let contents = self.contents();
let mut offset = self.content_offset;
for content in contents.iter() {
offset = align_up(offset + content.size.get(), ALIGNMENT as u64);
}
offset
}
}
pub fn process_wad<R>(reader: &mut R) -> Result<WadFile>
where R: BufRead + Seek + ?Sized {
let header: WadHeader = read_from(reader)?;
let mut offset = align_up(header.header_size.get() as u64, ALIGNMENT as u64);
reader.seek(io::SeekFrom::Start(offset))?;
let raw_cert_chain: Box<[u8]> = read_box_slice(reader, header.cert_chain_size.get() as usize)?;
offset = align_up(offset + header.cert_chain_size.get() as u64, ALIGNMENT as u64);
reader.seek(io::SeekFrom::Start(offset))?;
let raw_ticket: Box<[u8]> = read_box_slice(reader, header.ticket_size.get() as usize)?;
offset = align_up(offset + header.ticket_size.get() as u64, ALIGNMENT as u64);
reader.seek(io::SeekFrom::Start(offset))?;
let raw_tmd: Box<[u8]> = read_box_slice(reader, header.tmd_size.get() as usize)?;
offset = align_up(offset + header.tmd_size.get() as u64, ALIGNMENT as u64);
let content_offset = offset;
let mut file = WadFile {
header,
title_key: [0; 16],
fake_signed: false,
raw_cert_chain,
raw_ticket,
raw_tmd,
content_offset,
};
let mut title_key_found = false;
if file.ticket().header.sig.iter().all(|&x| x == 0) {
// Fake signed, try to determine common key index
file.fake_signed = true;
let contents = file.contents();
if let Some(smallest_content) = contents.iter().min_by_key(|x| x.size.get()) {
let mut ticket = file.ticket().clone();
for i in 0..2 {
ticket.common_key_idx = i;
let title_key = ticket.decrypt_title_key()?;
let offset = file.content_offset(smallest_content.content_index.get());
reader.seek(io::SeekFrom::Start(offset))?;
if verify_content(reader, smallest_content, &title_key)? {
file.title_key = title_key;
title_key_found = true;
break;
}
}
}
if !title_key_found {
bail!("Failed to determine title key for fake signed WAD");
}
}
if !title_key_found {
let title_key = file.ticket().decrypt_title_key()?;
file.title_key = title_key;
}
Ok(file)
}
pub fn verify_wad<R>(file: &WadFile, reader: &mut R) -> Result<()>
where R: Read + Seek + ?Sized {
for content in file.contents() {
let content_index = content.content_index.get();
println!(
"Verifying content {:08x} (size {})",
content_index,
Size::from_bytes(content.size.get())
);
let offset = file.content_offset(content_index);
reader.seek(io::SeekFrom::Start(offset))?;
if !verify_content(reader, content, &file.title_key)? {
bail!("Content {:08x} hash mismatch", content_index);
}
}
Ok(())
}
fn verify_content<R>(
reader: &mut R,
content: &ContentMetadata,
title_key: &KeyBytes,
) -> Result<bool>
where
R: Read + ?Sized,
{
let mut buf = <[[u8; 0x10]]>::new_box_zeroed_with_elems(0x200)
.map_err(|_| io::Error::from(io::ErrorKind::OutOfMemory))?;
// Read full padded size for decryption
let read_size = align_up(content.size.get(), 0x40);
let mut decryptor = Aes128Cbc::new(title_key.into(), (&content.iv()).into());
let mut digest = Sha1::default();
let mut read = 0;
while read < read_size {
let len = buf.len().min(usize::try_from(read_size - read).unwrap_or(usize::MAX));
debug_assert_eq!(len % 0x10, 0);
reader.read_exact(&mut buf.as_mut_bytes()[..len])?;
for block in buf.iter_mut().take(len / 0x10) {
decryptor.decrypt_block_mut(block.into());
}
// Only hash up to content size
let hash_len = (read + len as u64).min(content.size.get()).saturating_sub(read) as usize;
if hash_len > 0 {
digest.update(&buf.as_bytes()[..hash_len]);
}
read += len as u64;
}
Ok(HashBytes::from(digest.finalize()) == content.hash)
}

View File

@ -112,7 +112,7 @@ impl Seek for WindowedFile {
}
#[inline]
fn stream_position(&mut self) -> io::Result<u64> { Ok(self.pos) }
fn stream_position(&mut self) -> io::Result<u64> { Ok(self.pos - self.begin) }
}
impl VfsFile for WindowedFile {

View File

@ -10,7 +10,7 @@ use nodtool::{
nod::{Disc, DiscStream, Fst, NodeKind, OwnedFileStream, PartitionBase, PartitionMeta},
};
use typed_path::Utf8UnixPath;
use zerocopy::IntoBytes;
use zerocopy::{FromZeros, IntoBytes};
use super::{
next_non_empty, StaticFile, Vfs, VfsError, VfsFile, VfsFileType, VfsMetadata, VfsResult,
@ -252,19 +252,21 @@ impl DiscFile {
Self { inner: DiscFileInner::Stream(file), mtime }
}
fn convert_to_mapped(&mut self) {
fn convert_to_mapped(&mut self) -> io::Result<()> {
match &mut self.inner {
DiscFileInner::Stream(stream) => {
let pos = stream.stream_position().unwrap();
stream.seek(SeekFrom::Start(0)).unwrap();
let mut data = vec![0u8; stream.len() as usize];
stream.read_exact(&mut data).unwrap();
let mut cursor = Cursor::new(Arc::from(data.as_slice()));
let pos = stream.stream_position()?;
stream.seek(SeekFrom::Start(0))?;
let mut data = <[u8]>::new_box_zeroed_with_elems(stream.len() as usize)
.map_err(|_| io::Error::from(io::ErrorKind::OutOfMemory))?;
stream.read_exact(&mut data)?;
let mut cursor = Cursor::new(Arc::from(data));
cursor.set_position(pos);
self.inner = DiscFileInner::Mapped(cursor);
}
DiscFileInner::Mapped(_) => {}
};
Ok(())
}
}
@ -304,7 +306,7 @@ impl Seek for DiscFile {
impl VfsFile for DiscFile {
fn map(&mut self) -> io::Result<&[u8]> {
self.convert_to_mapped();
self.convert_to_mapped()?;
match &mut self.inner {
DiscFileInner::Stream(_) => unreachable!(),
DiscFileInner::Mapped(data) => Ok(data.get_ref()),
@ -331,7 +333,7 @@ impl VfsFile for DiscFile {
pub fn nod_to_io_error(e: nod::Error) -> io::Error {
match e {
nod::Error::Io(msg, e) => io::Error::new(e.kind(), format!("{}: {}", msg, e)),
nod::Error::Io(msg, e) => io::Error::new(e.kind(), format!("{msg}: {e}")),
e => io::Error::new(io::ErrorKind::InvalidData, e),
}
}

View File

@ -3,6 +3,7 @@ mod disc;
mod rarc;
mod std_fs;
mod u8_arc;
mod wad;
use std::{
error::Error,
@ -22,12 +23,14 @@ use rarc::RarcFs;
pub use std_fs::StdFs;
use typed_path::{Utf8NativePath, Utf8UnixPath, Utf8UnixPathBuf};
use u8_arc::U8Fs;
use wad::WadFs;
use crate::util::{
ncompress::{YAY0_MAGIC, YAZ0_MAGIC},
nlzss,
rarc::RARC_MAGIC,
u8_arc::U8_MAGIC,
wad::WAD_MAGIC,
};
pub trait Vfs: DynClone + Send + Sync {
@ -105,8 +108,8 @@ impl Display for VfsError {
fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
match self {
VfsError::NotFound => write!(f, "File or directory not found"),
VfsError::IoError(e) => write!(f, "{}", e),
VfsError::Other(e) => write!(f, "{}", e),
VfsError::IoError(e) => write!(f, "{e}"),
VfsError::Other(e) => write!(f, "{e}"),
VfsError::NotADirectory => write!(f, "Path is a file, not a directory"),
VfsError::IsADirectory => write!(f, "Path is a directory, not a file"),
}
@ -126,8 +129,8 @@ impl Display for FileFormat {
fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
match self {
FileFormat::Regular => write!(f, "File"),
FileFormat::Compressed(kind) => write!(f, "Compressed: {}", kind),
FileFormat::Archive(kind) => write!(f, "Archive: {}", kind),
FileFormat::Compressed(kind) => write!(f, "Compressed: {kind}"),
FileFormat::Archive(kind) => write!(f, "Archive: {kind}"),
}
}
}
@ -154,6 +157,7 @@ pub enum ArchiveKind {
Rarc,
U8,
Disc(nod::Format),
Wad,
}
impl Display for ArchiveKind {
@ -161,7 +165,8 @@ impl Display for ArchiveKind {
match self {
ArchiveKind::Rarc => write!(f, "RARC"),
ArchiveKind::U8 => write!(f, "U8"),
ArchiveKind::Disc(format) => write!(f, "Disc ({})", format),
ArchiveKind::Disc(format) => write!(f, "Disc ({format})"),
ArchiveKind::Wad => write!(f, "WAD"),
}
}
}
@ -169,18 +174,19 @@ impl Display for ArchiveKind {
pub fn detect<R>(file: &mut R) -> io::Result<FileFormat>
where R: Read + Seek + ?Sized {
file.seek(SeekFrom::Start(0))?;
let mut magic = [0u8; 4];
let mut magic = [0u8; 8];
match file.read_exact(&mut magic) {
Ok(_) => {}
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => return Ok(FileFormat::Regular),
Err(e) => return Err(e),
}
file.seek_relative(-4)?;
file.seek_relative(-8)?;
match magic {
YAY0_MAGIC => Ok(FileFormat::Compressed(CompressionKind::Yay0)),
YAZ0_MAGIC => Ok(FileFormat::Compressed(CompressionKind::Yaz0)),
RARC_MAGIC => Ok(FileFormat::Archive(ArchiveKind::Rarc)),
U8_MAGIC => Ok(FileFormat::Archive(ArchiveKind::U8)),
_ if magic.starts_with(&YAY0_MAGIC) => Ok(FileFormat::Compressed(CompressionKind::Yay0)),
_ if magic.starts_with(&YAZ0_MAGIC) => Ok(FileFormat::Compressed(CompressionKind::Yaz0)),
_ if magic.starts_with(&RARC_MAGIC) => Ok(FileFormat::Archive(ArchiveKind::Rarc)),
_ if magic.starts_with(&U8_MAGIC) => Ok(FileFormat::Archive(ArchiveKind::U8)),
WAD_MAGIC => Ok(FileFormat::Archive(ArchiveKind::Wad)),
_ => {
let format = nod::Disc::detect(file)?;
file.seek(SeekFrom::Start(0))?;
@ -222,13 +228,13 @@ pub fn open_path_with_fs(
let file_type = match fs.metadata(segment) {
Ok(metadata) => metadata.file_type,
Err(VfsError::NotFound) => return Err(anyhow!("{} not found", current_path)),
Err(e) => return Err(e).context(format!("Failed to open {}", current_path)),
Err(e) => return Err(e).context(format!("Failed to open {current_path}")),
};
match file_type {
VfsFileType::File => {
file = Some(
fs.open(segment)
.with_context(|| format!("Failed to open {}", current_path))?,
.with_context(|| format!("Failed to open {current_path}"))?,
);
}
VfsFileType::Directory => {
@ -242,7 +248,7 @@ pub fn open_path_with_fs(
}
let mut current_file = file.take().unwrap();
let format = detect(current_file.as_mut())
.with_context(|| format!("Failed to detect file type for {}", current_path))?;
.with_context(|| format!("Failed to detect file type for {current_path}"))?;
if let Some(&next) = split.peek() {
match next {
"nlzss" => {
@ -250,7 +256,7 @@ pub fn open_path_with_fs(
file = Some(
decompress_file(current_file.as_mut(), CompressionKind::Nlzss)
.with_context(|| {
format!("Failed to decompress {} with NLZSS", current_path)
format!("Failed to decompress {current_path} with NLZSS")
})?,
);
}
@ -259,7 +265,7 @@ pub fn open_path_with_fs(
file = Some(
decompress_file(current_file.as_mut(), CompressionKind::Yay0)
.with_context(|| {
format!("Failed to decompress {} with Yay0", current_path)
format!("Failed to decompress {current_path} with Yay0")
})?,
);
}
@ -268,7 +274,7 @@ pub fn open_path_with_fs(
file = Some(
decompress_file(current_file.as_mut(), CompressionKind::Yaz0)
.with_context(|| {
format!("Failed to decompress {} with Yaz0", current_path)
format!("Failed to decompress {current_path} with Yaz0")
})?,
);
}
@ -277,16 +283,15 @@ pub fn open_path_with_fs(
return Err(anyhow!("{} is not an archive", current_path))
}
FileFormat::Compressed(kind) => {
file =
Some(decompress_file(current_file.as_mut(), kind).with_context(
|| format!("Failed to decompress {}", current_path),
)?);
file = Some(
decompress_file(current_file.as_mut(), kind)
.with_context(|| format!("Failed to decompress {current_path}"))?,
);
// Continue the loop to detect the new format
}
FileFormat::Archive(kind) => {
fs = open_fs(current_file, kind).with_context(|| {
format!("Failed to open container {}", current_path)
})?;
fs = open_fs(current_file, kind)
.with_context(|| format!("Failed to open container {current_path}"))?;
// Continue the loop to open the next segment
}
},
@ -296,7 +301,7 @@ pub fn open_path_with_fs(
return match format {
FileFormat::Compressed(kind) if auto_decompress => Ok(OpenResult::File(
decompress_file(current_file.as_mut(), kind)
.with_context(|| format!("Failed to decompress {}", current_path))?,
.with_context(|| format!("Failed to decompress {current_path}"))?,
segment.to_path_buf(),
)),
_ => Ok(OpenResult::File(current_file, segment.to_path_buf())),
@ -332,6 +337,7 @@ pub fn open_fs(mut file: Box<dyn VfsFile>, kind: ArchiveKind) -> io::Result<Box<
disc.open_partition_kind(nod::PartitionKind::Data).map_err(nod_to_io_error)?;
Ok(Box::new(DiscFs::new(disc, partition, metadata.mtime)?))
}
ArchiveKind::Wad => Ok(Box::new(WadFs::new(file)?)),
}
}

View File

@ -55,10 +55,10 @@ impl StdFile {
pub fn new(path: Utf8NativePathBuf) -> Self { StdFile { path, file: None, mmap: None } }
pub fn file(&mut self) -> io::Result<&mut BufReader<fs::File>> {
if self.file.is_none() {
self.file = Some(BufReader::new(fs::File::open(&self.path)?));
}
Ok(self.file.as_mut().unwrap())
Ok(match self.file {
Some(ref mut file) => file,
None => self.file.insert(BufReader::new(fs::File::open(&self.path)?)),
})
}
}
@ -86,13 +86,15 @@ impl Seek for StdFile {
impl VfsFile for StdFile {
fn map(&mut self) -> io::Result<&[u8]> {
if self.file.is_none() {
self.file = Some(BufReader::new(fs::File::open(&self.path)?));
}
if self.mmap.is_none() {
self.mmap = Some(unsafe { memmap2::Mmap::map(self.file.as_ref().unwrap().get_ref())? });
}
Ok(self.mmap.as_ref().unwrap())
let file = match self.file {
Some(ref mut file) => file,
None => self.file.insert(BufReader::new(fs::File::open(&self.path)?)),
};
let mmap = match self.mmap {
Some(ref mmap) => mmap,
None => self.mmap.insert(unsafe { memmap2::Mmap::map(file.get_ref())? }),
};
Ok(mmap)
}
fn metadata(&mut self) -> io::Result<VfsMetadata> {

366
src/vfs/wad.rs Normal file
View File

@ -0,0 +1,366 @@
use std::{
io,
io::{BufRead, Cursor, Read, Seek, SeekFrom},
sync::Arc,
};
use aes::cipher::{BlockDecryptMut, KeyIvInit};
use filetime::FileTime;
use nodtool::nod::DiscStream;
use typed_path::Utf8UnixPath;
use zerocopy::FromZeros;
use crate::{
array_ref,
util::wad::{align_up, process_wad, ContentMetadata, WadFile},
vfs::{
common::{StaticFile, WindowedFile},
Vfs, VfsError, VfsFile, VfsFileType, VfsMetadata, VfsResult,
},
};
#[derive(Clone)]
pub struct WadFs {
file: Box<dyn VfsFile>,
wad: WadFile,
mtime: Option<FileTime>,
}
enum WadFindResult<'a> {
Root,
Static(&'a [u8]),
Content(u16, &'a ContentMetadata),
Window(u64, u64),
}
impl WadFs {
pub fn new(mut file: Box<dyn VfsFile>) -> io::Result<Self> {
let mtime = file.metadata()?.mtime;
let wad = process_wad(file.as_mut())
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
Ok(Self { file, wad, mtime })
}
fn find(&self, path: &str) -> Option<WadFindResult> {
let filename = path.trim_start_matches('/');
if filename.contains('/') {
return None;
}
if filename.is_empty() {
return Some(WadFindResult::Root);
}
let filename = filename.to_ascii_lowercase();
if let Some(id) = filename.strip_suffix(".app") {
if let Ok(content_index) = u16::from_str_radix(id, 16) {
if let Some(content) = self.wad.contents().get(content_index as usize) {
return Some(WadFindResult::Content(content_index, content));
}
}
return None;
}
let title_id = hex::encode(self.wad.ticket().title_id);
match filename.strip_prefix(&title_id) {
Some(".tik") => Some(WadFindResult::Static(&self.wad.raw_ticket)),
Some(".tmd") => Some(WadFindResult::Static(&self.wad.raw_tmd)),
Some(".cert") => Some(WadFindResult::Static(&self.wad.raw_cert_chain)),
Some(".trailer") => {
if self.wad.header.footer_size.get() == 0 {
return None;
}
Some(WadFindResult::Window(
self.wad.trailer_offset(),
self.wad.header.footer_size.get() as u64,
))
}
_ => None,
}
}
}
impl Vfs for WadFs {
fn open(&mut self, path: &Utf8UnixPath) -> VfsResult<Box<dyn VfsFile>> {
if let Some(result) = self.find(path.as_str()) {
match result {
WadFindResult::Root => Err(VfsError::IsADirectory),
WadFindResult::Static(data) => {
Ok(Box::new(StaticFile::new(Arc::from(data), self.mtime)))
}
WadFindResult::Content(content_index, content) => {
let offset = self.wad.content_offset(content_index);
Ok(Box::new(WadContent::new(
AesCbcStream::new(
self.file.clone(),
offset,
content.size.get(),
&self.wad.title_key,
&content.iv(),
),
self.mtime,
)))
}
WadFindResult::Window(offset, len) => {
Ok(Box::new(WindowedFile::new(self.file.clone(), offset, len)?))
}
}
} else {
Err(VfsError::NotFound)
}
}
fn exists(&mut self, path: &Utf8UnixPath) -> VfsResult<bool> {
Ok(self.find(path.as_str()).is_some())
}
fn read_dir(&mut self, path: &Utf8UnixPath) -> VfsResult<Vec<String>> {
let path = path.as_str().trim_start_matches('/');
if !path.is_empty() {
return Err(VfsError::NotFound);
}
let title_id = hex::encode(self.wad.ticket().title_id);
let mut entries = Vec::new();
entries.push(format!("{title_id}.tik"));
entries.push(format!("{title_id}.tmd"));
entries.push(format!("{title_id}.cert"));
if self.wad.header.footer_size.get() > 0 {
entries.push(format!("{title_id}.trailer"));
}
for content in self.wad.contents() {
entries.push(format!("{:08x}.app", content.content_index.get()));
}
Ok(entries)
}
fn metadata(&mut self, path: &Utf8UnixPath) -> VfsResult<VfsMetadata> {
if let Some(result) = self.find(path.as_str()) {
match result {
WadFindResult::Root => {
Ok(VfsMetadata { file_type: VfsFileType::Directory, len: 0, mtime: self.mtime })
}
WadFindResult::Static(data) => Ok(VfsMetadata {
file_type: VfsFileType::File,
len: data.len() as u64,
mtime: self.mtime,
}),
WadFindResult::Content(_, content) => Ok(VfsMetadata {
file_type: VfsFileType::File,
len: content.size.get(),
mtime: self.mtime,
}),
WadFindResult::Window(_, len) => {
Ok(VfsMetadata { file_type: VfsFileType::File, len, mtime: self.mtime })
}
}
} else {
Err(VfsError::NotFound)
}
}
}
#[derive(Clone)]
enum WadContentInner {
Stream(AesCbcStream),
Mapped(Cursor<Arc<[u8]>>),
}
#[derive(Clone)]
struct WadContent {
inner: WadContentInner,
mtime: Option<FileTime>,
}
impl WadContent {
fn new(inner: AesCbcStream, mtime: Option<FileTime>) -> Self {
Self { inner: WadContentInner::Stream(inner), mtime }
}
fn convert_to_mapped(&mut self) -> io::Result<()> {
match &mut self.inner {
WadContentInner::Stream(stream) => {
let pos = stream.stream_position()?;
stream.seek(SeekFrom::Start(0))?;
let mut data = <[u8]>::new_box_zeroed_with_elems(stream.len() as usize)
.map_err(|_| io::Error::from(io::ErrorKind::OutOfMemory))?;
stream.read_exact(&mut data)?;
let mut cursor = Cursor::new(Arc::from(data));
cursor.set_position(pos);
self.inner = WadContentInner::Mapped(cursor);
}
WadContentInner::Mapped(_) => {}
};
Ok(())
}
}
impl BufRead for WadContent {
fn fill_buf(&mut self) -> io::Result<&[u8]> {
match &mut self.inner {
WadContentInner::Stream(stream) => stream.fill_buf(),
WadContentInner::Mapped(data) => data.fill_buf(),
}
}
fn consume(&mut self, amt: usize) {
match &mut self.inner {
WadContentInner::Stream(stream) => stream.consume(amt),
WadContentInner::Mapped(data) => data.consume(amt),
}
}
}
impl Read for WadContent {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
match &mut self.inner {
WadContentInner::Stream(stream) => stream.read(buf),
WadContentInner::Mapped(data) => data.read(buf),
}
}
}
impl Seek for WadContent {
fn seek(&mut self, pos: SeekFrom) -> io::Result<u64> {
match &mut self.inner {
WadContentInner::Stream(stream) => stream.seek(pos),
WadContentInner::Mapped(data) => data.seek(pos),
}
}
}
impl VfsFile for WadContent {
fn map(&mut self) -> io::Result<&[u8]> {
self.convert_to_mapped()?;
match &mut self.inner {
WadContentInner::Stream(_) => unreachable!(),
WadContentInner::Mapped(data) => Ok(data.get_ref()),
}
}
fn metadata(&mut self) -> io::Result<VfsMetadata> {
match &mut self.inner {
WadContentInner::Stream(stream) => Ok(VfsMetadata {
file_type: VfsFileType::File,
len: stream.len(),
mtime: self.mtime,
}),
WadContentInner::Mapped(data) => Ok(VfsMetadata {
file_type: VfsFileType::File,
len: data.get_ref().len() as u64,
mtime: self.mtime,
}),
}
}
fn into_disc_stream(self: Box<Self>) -> Box<dyn DiscStream> { self }
}
#[derive(Clone)]
struct AesCbcStream {
inner: Box<dyn VfsFile>,
position: u64,
content_offset: u64,
content_size: u64,
key: [u8; 0x10],
init_iv: [u8; 0x10],
last_iv: [u8; 0x10],
block_idx: u64,
block: Box<[u8; 0x200]>,
}
impl AesCbcStream {
fn new(
inner: Box<dyn VfsFile>,
content_offset: u64,
content_size: u64,
key: &[u8; 0x10],
iv: &[u8; 0x10],
) -> Self {
let block = <[u8; 0x200]>::new_box_zeroed().unwrap();
Self {
inner,
position: 0,
content_offset,
content_size,
key: *key,
init_iv: *iv,
last_iv: [0u8; 0x10],
block_idx: u64::MAX,
block,
}
}
#[inline]
fn len(&self) -> u64 { self.content_size }
#[inline]
fn remaining(&self) -> u64 { self.content_size.saturating_sub(self.position) }
}
impl Read for AesCbcStream {
fn read(&mut self, mut buf: &mut [u8]) -> io::Result<usize> {
let mut total = 0;
while !buf.is_empty() {
let block = self.fill_buf()?;
if block.is_empty() {
break;
}
let len = buf.len().min(block.len());
buf[..len].copy_from_slice(&block[..len]);
buf = &mut buf[len..];
self.consume(len);
total += len;
}
Ok(total)
}
}
impl BufRead for AesCbcStream {
fn fill_buf(&mut self) -> io::Result<&[u8]> {
if self.position >= self.content_size {
return Ok(&[]);
}
let block_size = self.block.len();
let current_block = self.position / block_size as u64;
if current_block != self.block_idx {
let block_offset = current_block * block_size as u64;
let mut iv = [0u8; 0x10];
if current_block == 0 {
// Use the initial IV for the first block
self.inner.seek(SeekFrom::Start(self.content_offset))?;
iv = self.init_iv;
} else if self.block_idx.checked_add(1) == Some(current_block) {
// Shortcut to avoid seeking when reading sequentially
iv = self.last_iv;
} else {
// Read the IV from the previous block
self.inner.seek(SeekFrom::Start(self.content_offset + block_offset - 0x10))?;
self.inner.read_exact(&mut iv)?;
}
let aligned_size = align_up(self.content_size, 0x10);
let remaining = aligned_size.saturating_sub(block_offset);
let read = remaining.min(block_size as u64) as usize;
self.inner.read_exact(&mut self.block[..read])?;
self.last_iv = *array_ref!(self.block, read - 0x10, 0x10);
let mut decryptor =
cbc::Decryptor::<aes::Aes128>::new((&self.key).into(), (&iv).into());
for aes_block in self.block[..read].chunks_exact_mut(0x10) {
decryptor.decrypt_block_mut(aes_block.into());
}
self.block_idx = current_block;
}
let offset = (self.position % block_size as u64) as usize;
let len = self.remaining().min((block_size - offset) as u64) as usize;
Ok(&self.block[offset..offset + len])
}
fn consume(&mut self, amt: usize) { self.position = self.position.saturating_add(amt as u64); }
}
impl Seek for AesCbcStream {
fn seek(&mut self, pos: SeekFrom) -> io::Result<u64> {
self.position = match pos {
SeekFrom::Start(p) => p,
SeekFrom::End(p) => self.content_size.saturating_add_signed(p),
SeekFrom::Current(p) => self.position.saturating_add_signed(p),
};
Ok(self.position)
}
}