Update available
Changes
- added IO functions (find, extract, patch)
- generate database feature and IO functions now can search for streams larger than chunk size
Notes
More IO functions introduced. Find simply helps you track positions of extracted files from a given input, while extract uses a generated decode file data input to extract streams in case if you wanted another person to extract their very own streams using your own findings and patch, well patch compares two inputs and generates a diff file which can allow you to patch an input to make it similar to the new data.
Find
Find can be used to search for streams to then produce decode data which you can upload here on the forum in cases where you wanted to inform people what files should be extracted if they wanted for example to separate audio files per language from a game.Код:xtool find [parameters] extracted_streams original_data [decode_data]
An example where I wanted to make a guide of how to separate english, german, italian, russian and spanish languages from the game would be something like:
These files are then uploaded for people if they wanted to extract the same data by themselves by using xtool via the "extract" function or for later use by yourself.Код:xtool find "extracted\en\*" "game\*" en.xtl xtool find "extracted\de\*" "game\*" de.xtl xtool find "extracted\it\*" "game\*" it.xtl xtool find "extracted\es\*" "game\*" es.xtl xtool find "extracted\ru\*" "game\*" ru.xtl
Extract
Extract works mostly hand in hand with with the find function as its job is to use the decode_data to extract the data that was used for its generation.Код:xtool extract decode_data original_data extracted_streams
A use case for it would be similar to the find example but from another user's perspective, so if someone uploaded decode data to use to extract my very own streams from their investigations, rather than them giving me position ranges for languages or the tools to use, we can just use their uploaded decode data to do it ourselves
Then from here, we use these extracted files, to even use the erase function introduced in the version prior to blank out the sectors to prepare our own repackКод:xtool extract en.xtl "game\*" "extracted\en\*" xtool extract de.xtl "game\*" "extracted\de\*" xtool extract it.xtl "game\*" "extracted\it\*" xtool extract es.xtl "game\*" "extracted\es\*" xtool extract ru.xtl "game\*" "extracted\ru\*"
Код:xtool erase "extracted\*" "game\*" languages.xtl which we can then restore after installation using
PatchКод:xtool decode languages.xtl "extracted\*" "game\*"
Patch's use case can be the generation of update patches in cases where you already made a repack but for whatever reason refuse to re-repack the game so instead choose to generate a patch file which can be used to update the old repack with newer filesКод:xtool patch [parameters] old_data new_data patch_data xtool decode patch_data old_data
Then after the installation of the original repack, you can add an additional archive that has sims4_wedding_stories.patch which will be used to update the gameКод:xtool patch "Sims4\*" "Sims4_updated\*" sims4_wedding_stories.patch
I noticed after compiling xtool.exe that xdelta is set to compress these patch files which means that they cannot be precompressed nor compressed with something better, however this compression in the next version will be removed.Код:xtool decode sims4_wedding_stories.patch "Sims4\*"
Update available
Changes
- added IO functions (erase, replace)
- fixed external executable support bugs
Notes
Xtool has introduced IO functions, which should help you perform file and folder operations such as erase and replace (for now, more will be added). I actually wanted to add these functions a long time ago as they could be useful for repacking however I delayed them time after time because precompression needed more attention but as there have been no bug reports for the past 3 weeks I decided to start working on them.
To summarise, Erase is a feature that fills a given input with zeroes in case you wanted to repack certain data separately and Replace is a feature that replaces existing data within certain files with another.
Xtool will search for locations of the extracted streams and store these positions for when you use decode function to revert the changes (yes, the process is reversible)
Erase
Use cases for erase is the removal of language/videos files from archives in games after using another extraction tool, so rather than manually searching for positions and storing them, erase will make this process automated.Код:xtool erase extracted_streams original_data [decode_data] xtool decode decode_data extracted_streams original_data
An example is after extracting video files from an archive and then wanting to remove credits video, the syntax would be
ReplaceКод:xtool erase credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak credits.xtl xtool decode credits.xtl credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak decode will fill the zeroes with the original data
An example is having several modified files and the original files and you wanted to bulk replace these files, you'd have to prepare two folders with the same file structure and the syntax would beКод:xtool replace old_streams new_streams original_data [decode_data] xtool decode decode_data extracted_streams original_data The use cases for replace is possibly game file modification.
PSКод:xtool replace "textures\original\*" "textures\modified\*" "game\*" mod_tex.xtl xtool decode mod_tex.xtl "textures\original\*" "game\*" decode will fill the modified data sectors with the original to restore the file its original form.
If you are using these features for a repack in which people would select languages or video credits for example, if the users did not select any of these to be installed then there would be missing files. Xtool's decode command would still work and will try to restore the original data using whatever data that was selected and available without problems.
Update available
Changes
- fixed issue with storing correct recompression information when stream patching is performed
Notes
This issue was brought up by Masquerade and L0v3craft when they were trying to precompress WRC 10 and upon decoding xtool would produce an error so thanks to them, it has been resolved however I'd like to add more useful information regarding this game and why even with xtool helping you with deltas will not allow you to process all streams based on how xtool was designed.
We all know that xtool patches streams if they cannot be restored correctly however it sets its own boundaries, by default the patch file cannot be 5% larger than the original size else the patch fails as highlighted here with some streams being missed. https://fileforums.com/showpost.php?...&postcount=248
This 5% can be changed by user via the command --diff=5p, with 5p being 5%, so to capture all these other streams, you'll have to increase this percentage.
CHUNK_57.PKG
using -mwrc10
Код:Compressed 1 file, 99,435,664 => 189,185,421 bytes. Ratio 190.26% Compression time: cpu 0.14 sec/real 3.71 sec = 4%. Speed 26.82 mB/s using -mwrc10 --diff=20p
Код:Compressed 1 file, 99,435,664 => 263,029,277 bytes. Ratio 264.52% Compression time: cpu 0.09 sec/real 3.61 sec = 3%. Speed 27.53 mB/s Verbose information displays
--diff=20pКод:[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed [0] - Patching stream at 0000000000005476 (514 >> 514) [26] has failed Patching this 514 byte stream produced a 26 byte patch file, which is 5.1% and that is larger than the 5% limit so increasing this means you are allowing xtool to accept this stream.
Код:[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed [0] - Patched stream at 0000000000005476 (514 >> 514) [26] successfully
Update available
Changes
- updated oodle scanner
- updated external executable support
- updated configuration based plugin support to add depth information
- updated verbose mode
Notes
Given that very small number of people are able to make their own plugins and codecs to use in xtool, I've added an example that imports codecs from QuickBMS after you have generated your database and have no codec to use with it.
For compression:
For encryption:Код:[cpk,snappy,rfpk] Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein> Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>
You'll need to check http://aluigi.altervista.org/papers/quickbms.txt for the list of supported compressors built in so that you don't pick something like lz2k and expect it to work (quickbms itself has to be able to both compress and decompress). You can add more codecs like I have done here "cpk,snappy,rfpk", if you wanted to add lzss for example, it then becomes "cpk,snappy,rfpk,lzss" then from there you can use Bms2Xtl for scripts that have this comtype and possibly expect xtool to use quickbms with no real issues (hopefully).Код:[blowfish,xxtea] Encode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 0 <ressize>; log <fileout> 0 <insize>" "" <filein> Decode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 1 <ressize>; log <fileout> 0 <insize>" "" <filein>
Also note that if for whatever reason there is crc mismatch, xtool will still help you by applying xdelta automatically.
Also also note that this generate temps files meaning the process is slower because it has to both read and write to disk therefore, only use this approach if there is nothing else that you can to do to improve the situation.
Update available
Changes
- generate database feature fixed
- fixed external executable support issues
- fixed lz4f level setting bug
Update available
Changes
- removed debugging code from encryption and executable codec
- fixed issue with depth when using search codec
- fixed external executable support issues
Update available
Changes
- fixed issue of status not reporting when encoding
- added depth method support for search support
- fixed zlib encoding issues for different window bits (thanks KaktoR)
- fixed zlib memory leak issue (thanks KaktoR)
- updated all internal codecs to support information relayed by external codecs
- updated lz4f codec and removed temporarily removed support for universal scanning
- added option to change recompression level to be used by reflate
- updated external executable support
- generate database feature currently bugged, wait for next update
- search database structure changed, older database files will no longer work with newer releases
Notes
Older database files will not work with future updates, use the older version 0.3.21 for old database files as you wait for them to be updated
A temp file may be created when using database files which can be the size of the largest file in the data to be processed.
Update available
Changes
- project made open source
- added external executable support
- added generate database feature
- fixed search support bug
Notes
The source code will be made available on GitHub
https://github.com/Razor12911/xtool
Update available
Changes
- updated search support (speed improvements)
- updated command line parser
- added partial universal scanner for lzo1x streams
- added universal scanner for lz4f streams
- fixed issue with configuration files failing to execute without conditions
Notes
The previous update is still up, in case if there are issues with this one as I'm rusty af at the moment. (I haven't coded in months)
The lzo1x codec uses LZO1X-999 (level 1-9) and should only be used if you have no alternatives for precompression such as no plugins as this thing can be slow at times.
Update available
Changes
- fixed library support bug
- x86 build discontinued (has bugs from nowhere)