XTool (2020)

XTool (2020) 0.9.2

Нет прав для скачивания
Update available

Changes

- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner
  • Like
Реакции: Yaroslav950
Update available

Changes

- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.
Update available

Changes
- added library checker (trial and error)
- improved user interface
- fixed bugs related to oodle scanner
- skip verification no longer applies to encryption codecs

Notes
Library checker allows you to find out what library was used through trial and error, all you do is pick a directory with a list of either lz4, zstd or oodle libraries and set output to none then every single library found in that directory is loaded by xtool one by one while showing you how many streams were processed by each and their respective precompressed outputs. This should allow you to maximize compression in one click rather than doing it manually.

xt3.PNG


xt4.PNG


More syntax changes as per request from Cesar82:
--zlib= can also be -zb
--lz4=#, -l4#
--lzo=#, -lo#
--zstd=#, -zs#
--oodle=#, -od#
--srepmem=75p (when decoding) as requested by Gehrman

Announcement
This is the last update for xtool, I am at a point where I think I have done enough for this project. I have dragged development of the project longer than I should have but I guess it's a habit of mine of not leaving something unfinished and this is the creative vision I had for this tool from the start, it took longer than I expected but I'm glad it's done.

So what does that mean I am leaving the forum? No, I'll stick around for the time being, it's just that the main project is no longer getting updates... and I had to stop at the magic number 69
biggrin.gif
. So if there are bugs and issues, you'd have to refer to the older releases which are made available on the main post.
Update available

Changes

- added advanced configuration based plugin support
- added UI mode when xtool.exe is launched with xtoolui.dll present
- added skip verification mode
- xtool now enforces w15 deflate stream detection by default
- fixed oodle scanner exceptions when incomplete stream is detected
- fixed issue with zlib codec not accepting streams from database plugins
- updated command line syntax

Notes
advanced configuration plugin is a way of writing more complex ini files for stream detection without the need of coding skills or the need of creating a library based plugin.

spidey plugin is an example, regular configuration files could not add support for this game however, advanced configuration allows this and here's an example of how that looks like

Код:
[StreamList1]
Name=lz4
Codec=lz4
BigEndian=0
Signature=0x1000352415344
Structure1=Signature(8),Count(4),Unk1(4),Unk2(16)
StructureN=DPos(8),CPos(8),DSize(4),CSize(4),Unk3(8)
StructureS=Stream
CounterStart1=1
CounterEnd1=Count
CounterStep1=1
StreamPosition=CPos
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=

The user interface mode has been added and it's not meant to compete with tools like DiskSpanGUI or other similar programs but it's meant to help newbies operate the program as everything is made as simple as possible. The program can still be useful for regular users too, you can use the user interface mode to get theoretical outputs without doing actual (pre)compression, if you set output to none (leave blank in cli mode) and enable skip verification then essentially, you are performing a scan on the input (similar to what Drop and Scan for Zlib does) but for all supported codecs.

An example is Cyberpunk 2077 as shown below.

xt1.PNG


xt2.PNG



From here, you can find out the output size if you were to do precompression (in about 5 mins or less, all depends on drive speed).

More uses? Well people have a problem deciding how much chunk size to use, you can change the chunk size and see how the stream found or how the size differs to decide what chunk size to use.

The UI is a bit borked at the moment but the next update will improve upon it.

xtool enforces w15 streams because I figured out that this is what causes reflate to produce crc errors because there are more false positives this way. So how does this affect results, it doesn't because 99% of games use w15 anyways except Dishonored 2 and Dishonored Death of the Outsiders (set w10 for them)

Syntax has been updated as per request from Cesar82:
--dedup can also be -dd
--dedup=#, -dd#
--dbase, -db
--diff=#,-df#
--verbose, -v
--skip, -s
Update available

Changes

- fixed oodle scanner exceptions when incorrect library is used
- fixed issues with deduplication feature
Update available

Changes

- fixed issues with exporting precompression database
- fixed issues with deduplication feature consuming a lot of system memory
- fixed oodle codec from auto enabling selkie method
- fixed reflate related checksum issues due to false positives
Update available

Changes

- updated oodle scanner
- remove xdelta support from oodle and lzo codecs (crc mismatch often generates large diff files)

Notes
I've been getting reports of xtool taking a long time to process oodle streams (or getting stuck) so I reworked the scanner. I reverse engineered the code of oo2rec (I lost the source code) and rewrote parts of the code so xtool should produce better results while being faster.

There is a parameter rework as well, some people may know of the "n#" parameter which when set increases the chances of xtool capturing a stream. The default value now is 32, can be increased to 64, 128... Up to you, there is no limit just keep in mind that increasing this value also means longer precompression times.

Results on DOOM Eternal's "gameresources_4_1.streamdb"

oo2reck
Код:
Compressed 1 file, 1,353,405,353 => 2,584,084,262 bytes. Ratio 190.93%
Compression time: cpu 1.86 sec/real 590.20 sec = 0%. Speed 2.29 mB/s

xtool 0.4.7-0.6.4
Rumour has it that it's still processing as we speak (stuck on 5.7%)
rolleyes.gif


xtool 0.6.5 (-mkraken:l6:n128)
Код:
Compressed 1 file, 1,353,405,353 => 2,589,860,614 bytes. Ratio 191.36%
Compression time: cpu 1.80 sec/real 370.32 sec = 0%. Speed 3.65 mB/s
Update available

Changes


- added universal lz4f scanner
- fixed issues with database feature
- fixed issues with executable plugin support
- updated lzo codecs
Update available

Changes

- added feature to inject libraries to main executable

Notes
You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage
xtool.exe inject dll_file

More notes
Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.
Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example
xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -
Назад
Сверху