XTool (2020) 0.9.3

Update available

Changes
- memory usage optimizations

Notes
Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1

Код:

XTool is created by Razor12911

Streams: 1415315 / 1415335
Time: 00:07:20 (CPU 00:44:37)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

0.7.2

Код:

XTool is created by Razor12911

NVIDIA GeForce GTX 1060 6GB (4.50 GB loaded)
Streams: 1415315 / 1415335
Time: 00:05:31 (CPU 00:46:09)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

Update available

Changes
- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner

Update available

Changes
- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.

Update available

Changes
- added library checker (trial and error)
- improved user interface
- fixed bugs related to oodle scanner
- skip verification no longer applies to encryption codecs

Notes
Library checker allows you to find out what library was used through trial and error, all you do is pick a directory with a list of either lz4, zstd or oodle libraries and set output to none then every single library found in that directory is loaded by xtool one by one while showing you how many streams were processed by each and their respective precompressed outputs. This should allow you to maximize compression in one click rather than doing it manually.

More syntax changes as per request from Cesar82:
--zlib= can also be -zb
--lz4=#, -l4#
--lzo=#, -lo#
--zstd=#, -zs#
--oodle=#, -od#
--srepmem=75p (when decoding) as requested by Gehrman

Announcement
This is the last update for xtool, I am at a point where I think I have done enough for this project. I have dragged development of the project longer than I should have but I guess it's a habit of mine of not leaving something unfinished and this is the creative vision I had for this tool from the start, it took longer than I expected but I'm glad it's done.

So what does that mean I am leaving the forum? No, I'll stick around for the time being, it's just that the main project is no longer getting updates... and I had to stop at the magic number 69

. So if there are bugs and issues, you'd have to refer to the older releases which are made available on the main post.

Update available

Changes
- added advanced configuration based plugin support
- added UI mode when xtool.exe is launched with xtoolui.dll present
- added skip verification mode
- xtool now enforces w15 deflate stream detection by default
- fixed oodle scanner exceptions when incomplete stream is detected
- fixed issue with zlib codec not accepting streams from database plugins
- updated command line syntax

Notes
advanced configuration plugin is a way of writing more complex ini files for stream detection without the need of coding skills or the need of creating a library based plugin.

spidey plugin is an example, regular configuration files could not add support for this game however, advanced configuration allows this and here's an example of how that looks like

Код:

[StreamList1]
Name=lz4
Codec=lz4
BigEndian=0
Signature=0x1000352415344
Structure1=Signature(8),Count(4),Unk1(4),Unk2(16)
StructureN=DPos(8),CPos(8),DSize(4),CSize(4),Unk3(8)
StructureS=Stream
CounterStart1=1
CounterEnd1=Count
CounterStep1=1
StreamPosition=CPos
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=

The user interface mode has been added and it's not meant to compete with tools like DiskSpanGUI or other similar programs but it's meant to help newbies operate the program as everything is made as simple as possible. The program can still be useful for regular users too, you can use the user interface mode to get theoretical outputs without doing actual (pre)compression, if you set output to none (leave blank in cli mode) and enable skip verification then essentially, you are performing a scan on the input (similar to what Drop and Scan for Zlib does) but for all supported codecs.

An example is Cyberpunk 2077 as shown below.

From here, you can find out the output size if you were to do precompression (in about 5 mins or less, all depends on drive speed).

More uses? Well people have a problem deciding how much chunk size to use, you can change the chunk size and see how the stream found or how the size differs to decide what chunk size to use.

The UI is a bit borked at the moment but the next update will improve upon it.

xtool enforces w15 streams because I figured out that this is what causes reflate to produce crc errors because there are more false positives this way. So how does this affect results, it doesn't because 99% of games use w15 anyways except Dishonored 2 and Dishonored Death of the Outsiders (set w10 for them)

Syntax has been updated as per request from Cesar82:
--dedup can also be -dd
--dedup=#, -dd#
--dbase, -db
--diff=#,-df#
--verbose, -v
--skip, -s

Update available

Changes
- fixed oodle scanner exceptions when incorrect library is used
- fixed issues with deduplication feature

Update available

Changes
- fixed issues with exporting precompression database
- fixed issues with deduplication feature consuming a lot of system memory
- fixed oodle codec from auto enabling selkie method
- fixed reflate related checksum issues due to false positives

Update available

Changes
- updated oodle scanner
- remove xdelta support from oodle and lzo codecs (crc mismatch often generates large diff files)

Notes
I've been getting reports of xtool taking a long time to process oodle streams (or getting stuck) so I reworked the scanner. I reverse engineered the code of oo2rec (I lost the source code) and rewrote parts of the code so xtool should produce better results while being faster.

There is a parameter rework as well, some people may know of the "n#" parameter which when set increases the chances of xtool capturing a stream. The default value now is 32, can be increased to 64, 128... Up to you, there is no limit just keep in mind that increasing this value also means longer precompression times.

Results on DOOM Eternal's "gameresources_4_1.streamdb"

oo2reck

Код:

Compressed 1 file, 1,353,405,353 => 2,584,084,262 bytes. Ratio 190.93%
Compression time: cpu 1.86 sec/real 590.20 sec = 0%. Speed 2.29 mB/s

xtool 0.4.7-0.6.4
Rumour has it that it's still processing as we speak (stuck on 5.7%)

xtool 0.6.5 (-mkraken:l6:n128)

Код:

Compressed 1 file, 1,353,405,353 => 2,589,860,614 bytes. Ratio 191.36%
Compression time: cpu 1.80 sec/real 370.32 sec = 0%. Speed 3.65 mB/s

Update available

Changes

- added universal lz4f scanner
- fixed issues with database feature
- fixed issues with executable plugin support
- updated lzo codecs

Update available

Changes
- added feature to inject libraries to main executable

Notes
You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage
xtool.exe inject dll_file

More notes
Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.