XTool (2020) 0.9.6

Changes
- added tracking for sizes of stream patches
- added stream coverage tracking
- added option to specify multiple oodle libraries to use (-oodl#)
- fixed issues with reassign stream feature
- fixed minor issues with DirectStorage codecs

Notes
Patreon now seems to have virus scanner in attachments that I upload and has false detected xtool, this has delayed the 0.9.6 update because I was stuck in a situation where I wondered, should I add a password to the archives to prevent scanners from blocking the post? But then that would raise suspicions to some people using this tool especially with some programs like cpu-z and such getting malware recently (it's good to have trust issues) and so, I want to make it clear. The software is provided as is and I shall not be held liable for any damages.

With that out of the way,

xtool now tells you how many streams were patched and how big the patches are that way you can play a bit more with -df parameter to maximise ratio for imperfect streams.

Streams: 5359 / 5547 (95,43%)

Patched: 5359 (0,15 MB)

Нажмите для раскрытия...

It also tells you the stream coverage in percentages, 95,43% as an example for that input means 95% of the file had streams and 5% was something else, this is good feature that tells you if said file contained something else other than streams, maybe audio, video or encrypted data.

Streams: 17850 / 17850 (7,72%)

Нажмите для раскрытия...

ForzaHorizon6\media\Tracks\Brio\GeoChunk0.minizip

This file is 37GB, only 7% contains zlib streams, what is the remaining 93%? That's for you to investigate...

I've also added the feature that allows you to specify multiple oodle libraries for precompression, if one fails the next one is attempt because there's some games that seem to use more than one dll which makes complete precompression in one go not possible, requiring multiple passes of xtool which can be slower and use more memory.

Keep in mind, that the same parameters for oodle dlls is required when decoding and the primary oodle dll -oodle### should be a newer version meaning, do not use oo2core_4_win64.dll for the first one and oo2core_6_win64.dll for -oodl2###

gameresources_20_patch2.streamdb

Нажмите для раскрытия...

xtool.exe precomp -mkraken:l6 -c32mb -oodleoo2core_9_win64.dll -oodl2oo2core_9_win64_2.dll - -

with one library

Streams: 1045 / 1154 (96,90%)

Size: 356 MB >> 534 MB

Нажмите для раскрытия...

with both libraries

Streams: 1150 / 1154 (96,90%)

Size: 356 MB >> 605 MB

Нажмите для раскрытия...

Changes
- updated DirectStorage gdeflate and nvCOMP libgdeflate codecs

Notes
This is a complete implementation of both libraries meaning compression level can be specified and as such, the game Ninja Gaiden 4 now requires you to specify the correct level for precompression which is level 6 because stream patching takes place. Don't mind the streams that get left behind for that game because they are just too small to be patched without incurring negative ratio.

- Added internal version check when dragging liblz4, libzstd and oodle dlls onto xtool
- Added cls config
- Fixed an issue with plugins not being loaded correctly
- Updated library checker when using UI
- Updated leviathan codec

Changes
- updated io functions (find, erase and replace)
- UI has returned and should now work through wine

Notes
I have compiled 64-bit cls version of xtool and included it in the package... in case there's a need for it when inno setup 7 rolls out.

- memory optimisations when encoding
- low memory now has 3 settings to indicate how much less memory to use as a tradeoff for less speed

Notes
The ideal use case of the addition of this option is to allow users to set high chunk sizes to capture the most streams but if you really do not have enough ram, you can still make use of it however I do not recommend using anything above lm2, and lm2 itself should only be used for extreme cases like stray gods game or any other game that has above normal precompression ratios or if you're still using HDD then you can use -lm in conjuction with prefetch cache option.

A common scenario, Forza Horizon 5 Hot Wheels expansion

-mzlib -c256mb -t4 -lm0 took 3 minutes and 58 seconds and used 4,43 GB ram

-mzlib -c256mb -t4 -lm1 took 4 minutes and 29 seconds and used 2,29 GB ram

-mzlib -c256mb -t4 -lm2 took 5 minutes and 12 seconds and used 1,51 GB ram

-mzlib -c256mb -t4 -lm3 took 5 minutes and 27 seconds and used 704 MB ram

Нажмите для раскрытия...

A more extreme scenario, Stray Gods

-mzlib -c256mb -t4 -lm0 took 4 minutes and 57 seconds and used 50,98 GB ram

-mzlib -c256mb -t4 -lm1 took 5 minutes and 35 seconds and used 24,93 GB ram

-mzlib -c256mb -t4 -lm2 took 6 minutes and 39 seconds and used 1,43 GB ram

-mzlib -c256mb -t4 -lm3 took 8 minutes and 7 seconds and used 622,39 MB ram

Нажмите для раскрытия...

Changes
- minor code optimisation and fixes to prevent errors in certain scenarios
- fixed issues with reassign stream feature
- fixed issues library config based plugins
- removed recompress stream feature

Notes
Reassign stream was added because of inconsistent game engines like Unreal Engine because you could write a program/library to work on a list of games then a few months later, they change something in how the files are stored which renders your tool outdated because it no longer works on the newer games. unreal.dll is a plugin to support ue3-5 games and for as long as I remember, ue3 games were compressed with lzo then came mass effect legendary edition and it used leviathan, so I updated the plugin and then Gears of War Reloaded came out and it used lz4hc so rather than updating the plugin (I've lost source code), you can reassign all those ue3 streams to lz4hc, this feature was bugged but now it is fixed and should work like so:

Usually, you'd use -mue3 to precompress the game but it doesn't work because now the game uses lz4hc so, you need to notify xtool that streams need to be reassigned to a different codec and the new method becomes -mue3+ue3=lz4hc:l9.

Library config based plugins were added in 0.8.8 because I'm a person who seems to always lose their source codes due to hard drives dying, so what I've done is allowed dlls to be imported through ini which xtool can make use of, so rather than write code internally in xtool or through a plugin which is bound to have its code lost, I can just share a ini just like other config based plugins and if there's something that needs changing or updating, either the user can do it themselves or I can just use notepad to provide a quick solution. I've provided a sample.


Recompress stream was removed because it was an overly ambitious feature I thought of added, the idea behind it was that since some compression algorithms, zlib, zstd, lz4, and oodle to a certain extent do not error out if the incorrect compressed size was provided. So what does this mean? It means if a game could not be precompressed at all, like some special edition of zstd or oodle was used and let's assume it was compressed using level 6, recompress feature was meant to allow you to recompress the streams with a stronger level, let's assume level 7 and by doing this, you ensure that your new stream will be smaller and could replace the level 6 stream that could not be precompressed, a new game file is produced and that will be your new game files and then you can use xtool normally to precompress the game and repack it.

So, what's the problem? Well by doing this, you're opening a can of worms because you now have to test if the game works without issues (maybe from start to finish) because the game files were "modified", it is lossless but it is no longer bit-perfect and you might as well use other tools to rebuild the game files after unpacking them because that's even safer due to headers being updated too and this is why it was removed. 0.9.0 is the last version to have this so if you want to use it, then look for that version here.

- added support for lz4 multicore edition
- added libgdeflate support
- improved processing speed when depth is used with deduplication/dictionary
- fixed issues with lzo codec
- fixed issues with stream patching
- fast-lzma2 now works internally without needing dll
- updated internal zstd version to 1.5.7 to improve stream patching speed and sizes
- temporarily removed UI mode until it gets updated to run under wine

- fixed issues with base directory and file/directory pathing in general
- updated verbose mode log

- added zipcrypto encryption support
- added library config based plugin support

Changes
- updated media codecs (flac, brunsli, packjpg...)