XTool (2020)

XTool (2020) 0.8.7

Нет прав для скачивания
Update available

Changes

- fixed issues with exporting precompression database
- fixed issues with deduplication feature consuming a lot of system memory
- fixed oodle codec from auto enabling selkie method
- fixed reflate related checksum issues due to false positives
Update available

Changes

- updated oodle scanner
- remove xdelta support from oodle and lzo codecs (crc mismatch often generates large diff files)

Notes
I've been getting reports of xtool taking a long time to process oodle streams (or getting stuck) so I reworked the scanner. I reverse engineered the code of oo2rec (I lost the source code) and rewrote parts of the code so xtool should produce better results while being faster.

There is a parameter rework as well, some people may know of the "n#" parameter which when set increases the chances of xtool capturing a stream. The default value now is 32, can be increased to 64, 128... Up to you, there is no limit just keep in mind that increasing this value also means longer precompression times.

Results on DOOM Eternal's "gameresources_4_1.streamdb"

oo2reck
Код:
Compressed 1 file, 1,353,405,353 => 2,584,084,262 bytes. Ratio 190.93%
Compression time: cpu 1.86 sec/real 590.20 sec = 0%. Speed 2.29 mB/s
xtool 0.4.7-0.6.4
Rumour has it that it's still processing as we speak (stuck on 5.7%)


xtool 0.6.5 (-mkraken:l6:n128)
Код:
Compressed 1 file, 1,353,405,353 => 2,589,860,614 bytes. Ratio 191.36%
Compression time: cpu 1.80 sec/real 370.32 sec = 0%. Speed 3.65 mB/s
Update available

Changes


- added universal lz4f scanner
- fixed issues with database feature
- fixed issues with executable plugin support
- updated lzo codecs
Update available

Changes

- added feature to inject libraries to main executable

Notes
You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage
xtool.exe inject dll_file

More notes
Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.
Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example
xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -
Changes
- added wav stream detector
- added flac codec
- added jpg stream detector
- added packjpg, brunsli, jojpeg codec
- added feature that allows input to be a directory
- added feature to extract detected streams
- updated database feature
- updated deduplication feature
- IO function decode updated

Notes
I have added wav audio compression as an alternative of msc for people who has issues with it or don't want to use Freearc's tta codec as it processes wav files individually. One thing to note, it's not as good as tta, tak or even frog as it is based on flac codec (It was the only open source codec I could work with...).

Also added jpg image compression codecs, you can pick between packjpg, brunsli or jojpeg. packjpg seems to have a memory leak, brunsli is fast but cannot deal with large jpg images and jojpeg can be used if you're after ratio as it is paq based (seems to be buggy so stick to packjpg or brunsli for now)

A new parameter is added --extract=[path], this will extract all detected streams to a directory... if you're interested in the streams themselves.

database and deduplication features have been updated and can now be used at all times without worrying about crc collisions.

Special thanks to KaktoR for running several tests and Shelwien for providing compiled libraries for brunsli and jojpeg.
Update available

Changes

- added png stream preprocessor
- removed grittibanzli codec (since nobody uses it)

Notes
Xtool is able to process deflate/zlib streams and png images do contain these streams however, they are split up into several blocks which at times does prevent the program from being able to process them. The png stream preprocessor's job is to concatenate (rejoin all these blocks into a single stream) which can then be processed by xtool, so if you know your input contains these images then it's best to include -mpng into the method chain and use -d1 to first preprocess the streams then process them using zlib, reflate or preflate (preflate is the preferred method to use).

Results

without png preprocessor:

Код:
Compressed 1 file, 7,232,549 => 8,291,632 bytes. Ratio 114.64% -mreflate
Compressed 1 file, 7,232,549 => 7,232,655 bytes. Ratio 100.00% -mpreflate
with png preprocessor:
Код:
Compressed 1 file, 7,232,549 => 26,075,119 bytes. Ratio 360.52% -mpng+reflate -d1
Compressed 1 file, 7,232,549 => 25,289,529 bytes. Ratio 349.66% -mpng+preflate -d1
Update available

Changes
- added IO functions (archive, execute)
- fixed issue in patch io function
- removed compression on patch diff files

Notes
More IO functions added. Archive behaves like -m0 but allows you to the archive to stdout and read from stdin when decoding (if you ever need that). Then there is execute which allows you execute several instances of another executable in parallel mode while all their inputs are fed from one source and all their output is fed to one destination.

Archive
Код:
xtool archive files1 files2... archive
There is nothing to add here as I have personal uses for this but the example would be

Код:
xtool archive game\* mygame.xtl
xtool decode mygame.xtl extracted\*
Execute
Код:
xtool execute [parameters] input output [exec_syntax]
Too lazy to write description but, here's an example

Код:
xtool.exe execute -c64mb -t8 UI.sb UI.bin bcm.exe -b64 [filein] [fileout]
xtool.exe decode -t100p UI.bin UI_dec.sb bcm.exe -d [filein] [fileout]
The left side of the syntax is to command xtool and after specifying input and output files, the right side begins and here you can write the command line that should be used to perform execution.

[filein], [fileout], [stdin], [stdout] can be used and denote what IO the program being executed uses.
[] is used to avoid conflicting with Freearc's <>

Freearc example would look like this

Код:
[External compressor:xbcm]
header = 0
packcmd = xtool.exe execute { -option} - - <stdin> <stdout> bcm.exe -b64 [filein] [fileout]
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout> bcm.exe -d [filein] [fileout]
and the method would be -mxbcm:c64mb:t75p
  • Like
Реакции: jonnyyankee
Update available

Changes

- added IO functions (find, extract, patch)
- generate database feature and IO functions now can search for streams larger than chunk size

Notes
More IO functions introduced. Find simply helps you track positions of extracted files from a given input, while extract uses a generated decode file data input to extract streams in case if you wanted another person to extract their very own streams using your own findings and patch, well patch compares two inputs and generates a diff file which can allow you to patch an input to make it similar to the new data.

Find
Код:
xtool find [parameters] extracted_streams original_data [decode_data]
Find can be used to search for streams to then produce decode data which you can upload here on the forum in cases where you wanted to inform people what files should be extracted if they wanted for example to separate audio files per language from a game.

An example where I wanted to make a guide of how to separate english, german, italian, russian and spanish languages from the game would be something like:

Код:
xtool find "extracted\en\*" "game\*" en.xtl
xtool find "extracted\de\*" "game\*" de.xtl
xtool find "extracted\it\*" "game\*" it.xtl
xtool find "extracted\es\*" "game\*" es.xtl
xtool find "extracted\ru\*" "game\*" ru.xtl
These files are then uploaded for people if they wanted to extract the same data by themselves by using xtool via the "extract" function or for later use by yourself.

Extract
Код:
xtool extract decode_data original_data extracted_streams
Extract works mostly hand in hand with with the find function as its job is to use the decode_data to extract the data that was used for its generation.

A use case for it would be similar to the find example but from another user's perspective, so if someone uploaded decode data to use to extract my very own streams from their investigations, rather than them giving me position ranges for languages or the tools to use, we can just use their uploaded decode data to do it ourselves

Код:
xtool extract en.xtl "game\*" "extracted\en\*"
xtool extract de.xtl "game\*" "extracted\de\*"
xtool extract it.xtl "game\*" "extracted\it\*"
xtool extract es.xtl "game\*" "extracted\es\*"
xtool extract ru.xtl "game\*" "extracted\ru\*"
Then from here, we use these extracted files, to even use the erase function introduced in the version prior to blank out the sectors to prepare our own repack

Код:
xtool erase "extracted\*" "game\*" languages.xtl
which we can then restore after installation using
Код:
xtool decode languages.xtl "extracted\*" "game\*"
Patch
Код:
xtool patch [parameters] old_data new_data patch_data
xtool decode patch_data old_data
Patch's use case can be the generation of update patches in cases where you already made a repack but for whatever reason refuse to re-repack the game so instead choose to generate a patch file which can be used to update the old repack with newer files

Код:
xtool patch "Sims4\*" "Sims4_updated\*" sims4_wedding_stories.patch
Then after the installation of the original repack, you can add an additional archive that has sims4_wedding_stories.patch which will be used to update the game

Код:
xtool decode sims4_wedding_stories.patch "Sims4\*"
I noticed after compiling xtool.exe that xdelta is set to compress these patch files which means that they cannot be precompressed nor compressed with something better, however this compression in the next version will be removed.
  • Like
Реакции: jonnyyankee
Update available

Changes


- added IO functions (erase, replace)
- fixed external executable support bugs

Notes
Xtool has introduced IO functions, which should help you perform file and folder operations such as erase and replace (for now, more will be added). I actually wanted to add these functions a long time ago as they could be useful for repacking however I delayed them time after time because precompression needed more attention but as there have been no bug reports for the past 3 weeks I decided to start working on them.

To summarise, Erase is a feature that fills a given input with zeroes in case you wanted to repack certain data separately and Replace is a feature that replaces existing data within certain files with another.
Xtool will search for locations of the extracted streams and store these positions for when you use decode function to revert the changes (yes, the process is reversible)

Erase
Код:
xtool erase extracted_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data
Use cases for erase is the removal of language/videos files from archives in games after using another extraction tool, so rather than manually searching for positions and storing them, erase will make this process automated.

An example is after extracting video files from an archive and then wanting to remove credits video, the syntax would be

Код:
xtool erase credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak credits.xtl
xtool decode credits.xtl credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak
decode will fill the zeroes with the original data
Replace
Код:
xtool replace old_streams new_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data
The use cases for replace is possibly game file modification.
An example is having several modified files and the original files and you wanted to bulk replace these files, you'd have to prepare two folders with the same file structure and the syntax would be

Код:
xtool replace "textures\original\*" "textures\modified\*" "game\*" mod_tex.xtl
xtool decode mod_tex.xtl "textures\original\*" "game\*"
decode will fill the modified data sectors with the original to restore the file its original form.
PS

If you are using these features for a repack in which people would select languages or video credits for example, if the users did not select any of these to be installed then there would be missing files. Xtool's decode command would still work and will try to restore the original data using whatever data that was selected and available without problems.
  • Like
Реакции: jonnyyankee
Сверху