XTool 2019 (Plugins) 2019-08-13

Mickey1s · 13 Авг 2019

Пользователь Mickey1s разместил новый ресурс:

XTool 2019 (Plugins) - List of plugins for xtool

Game Series/Game Engine
Unity Engine

Game specific

IO

Upcoming plugins:

Anvil/Scimitar Engine
Dunia Engine
Call of Duty series
Frostbite Engine
Madness Engine
Unreal Engine

Unity Engine 5.x.x
Support for unity game engine files using lz4 algorithm.

Usage
Place dll near xtool.exe (only during encoding)

Available codecs
Unity

Узнать больше об этом ресурсе...

urban · 13 Авг 2019

xtool сам подтянет плагин unity если нужно где-то прописать? резор не упоминул об этом(

L-e-o-N · 15 Авг 2019

urban написал(а):
xtool сам подтянет плагин unity если нужно где-то прописать? резор не упоминул об этом(

Сам подтяивает, если dll-ка лежит рядом

urban · 15 Авг 2019

L-e-o-N написал(а):
Сам подтяивает, если dll-ка лежит рядом

кстати плагин на новинках не работает. Изменили название. На старых релизах до 18 года все ок работает.

L-e-o-N · 16 Авг 2019

urban написал(а):
кстати плагин на новинках не работает. Изменили название. На старых релизах до 18 года все ок работает.

Не работает? Я изучал xtool 2019 через программу procmon, он загружает библиотеку

SBalykov · 16 Авг 2019

М-да...
Скоро эти плагины, утилиты, библиотеки будут занимать место на диске больше чем упаковываемые (распаковываемые) ими файлы, при минимальном КПД и постоянными походами в аптеку для устранения головной боли от мысли правильного их применения ...

urban · 17 Авг 2019

L-e-o-N написал(а):
Не работает? Я изучал xtool 2019 через программу procmon, он загружает библиотеку

плагин сам работает, просто сам плагин не может прочитать lz4 в последний новинках игр из-за того что заголовки файлов изменились и он не может там найти lz4

Razor12911 · 17 Авг 2019

SBalykov написал(а):
М-да...
Скоро эти плагины, утилиты, библиотеки будут занимать место на диске больше чем упаковываемые (распаковываемые) ими файлы, при минимальном КПД и постоянными походами в аптеку для устранения головной боли от мысли правильного их применения ...

Uh not really, you can only use 1 plugin at a time, you are not required to use all plugins therefore your statement which says that the plugins/tools/utilities and etc will end up using more disk space than what you are trying to save is false. The reason I chose to do it this way is because if I were to put all of this code in the main project, I'll have issues maintaining the project because there will be code everywhere I had this problem maintaining old xtool, plus I can't do separate projects that contain repeated code from other projects that do basically the same thing, there will be a lot of issues so I decided to adopt the plugin apporoach because 95% of the code is already written in the main program, 5% accounts to plugins which I had planned to release sources for people to also be able to make contributions towards the project themselves. For example, I have uploaded the source that was required just to add Unity support, as you can see. It's very small and simple, anyone can just pick up the source and make changes if they need to. Because I won't be always around to make updates.

Edit: This approach of plugin is also faster in terms of performance than putting everything in main program because if you were only seeking unity streams, other code that would exist is never executed.

Also, I mentioned "Place dll near xtool.exe (only during encoding)"
The plugin only serves to help the main program locate streams it cannot find after that, during decoding, it serves no purpose so there isn't a need to also include the plugins in repacks, the main exe and liblz4.dll will be enough.

Код:

library xtool.unity;
{$IF CompilerVersion < 23.0}
{$Message Fatal 'Compiler outdated'}
{$IFEND}
{$R *.res}

uses
  System.SysUtils,
  System.Classes,
  System.StrUtils,
  System.IOUtils,
  MemoryModule in '..\base\MemoryModule.pas',
  PluginUtils in '..\base\PluginUtils.pas',
  LZ4DLL in '..\imports\LZ4DLL.pas';

const
  STATUS_NONE = 0;
  STATUS_PREDICTED = 1;
  STATUS_PROCESSED = 2;
  STATUS_VERIFIED = 3;

type
  PEncodeSI = ^TEncodeSI;

  TEncodeSI = record
    ActualPosition, StorePosition: Cardinal;
    OriginalSize, UnpackedSize: Cardinal;
    InfoPosition, InfoSize: Cardinal;
    Checksum: Cardinal;
    ID, Codec: Byte;
    Option: Byte;
    Status: Byte;
    Reserved: Cardinal;
  end;

  PDecodeSI = ^TDecodeSI;

  TDecodeSI = record
    Position: Cardinal;
    Size: Cardinal;
    ID, Codec: Byte;
    Option: Byte;
  end;

  PFutureSI = ^TFutureSI;

  TFutureSI = record
    Position: Int64;
    OriginalSize, UnpackedSize: Cardinal;
    ID, Codec: Byte;
    Option: Byte;
    Status: Byte;
  end;

var
  XInfo: TArray<TArray<String>>;

function GetID(Method: String): Integer;
var
  I, J: Integer;
begin
  Result := 0;
  for I := Low(XInfo) to High(XInfo) do
    for J := Low(XInfo[I]) to High(XInfo[I]) do
      if XInfo[I][J] = Method then
      begin
        Result := I;
        break
      end;
end;

function GetCodec(Method: String): Integer;
var
  I, J: Integer;
begin
  Result := 0;
  for I := Low(XInfo) to High(XInfo) do
    for J := Low(XInfo[I]) to High(XInfo[I]) do
      if XInfo[I][J] = Method then
      begin
        Result := J;
        break
      end;
end;

type
  TPluginCtx = record

  end;

  PPluginCtx = ^TPluginCtx;

const
  CodecNames: TArray<String> = ['unity'];
  UNITY_CODEC = 0;

var
  CodecInit: TArray<Boolean>;

function XTool_Init(Methods: String; AInfo: TArray < TArray < String >> )
  : PPluginCtx;
var
  I: Integer;
  B: Boolean;
begin
  XInfo := AInfo;
  SetLength(CodecInit, Length(CodecNames));
  for I := Low(CodecNames) to High(CodecNames) do
  begin
    B := AnsiIndexText(CodecNames[I], DecodeStr(Methods, ',')) >= 0;
    case I of
      UNITY_CODEC:
        CodecInit[I] := B and LZ4DLL.DLLLoaded;
    end;
  end;
end;

procedure XTool_Free(ctx: PPluginCtx);
begin

end;

function XTool_Codecs: TArray<String>;
begin
  Result := CodecNames;
end;

procedure XTool_Scan1(ctx: PPluginCtx; MemInput: TMemoryStream; MemPos: Int64;
  AvailSize, TotalSize: Cardinal; Output, Temp: TMemoryStream;
  Info1, Info2: TInfoStore);
var
  Ptr: PByte;
  Pos: Cardinal;
  I: Integer;
  P1, P2: Integer;
  Flags: Integer;
  Count: Integer;
  X, Y: Integer;
  S: String;
  OutPos: Int64;
  StreamInfo1: TEncodeSI;
  StreamInfo2: TFutureSI;
begin
  if BoolArray(CodecInit, False) then
    exit;
  Ptr := MemInput.Memory;
  Pos := 0;
  while Pos < AvailSize do
  begin
    if CodecInit[UNITY_CODEC] and (PInteger(Ptr + Pos)^ = $74696E55) then
    begin
      P1 := GetStr((Ptr + Pos), 10, S) + 1; // ident (UnityFS)
      if S = 'UnityFS' then
      begin
        Inc(P1, Integer.Size); // file version (6)
        Inc(P1, GetStr((Ptr + Pos + P1), 10, S) + 1); // 5.x.x
        Inc(P1, GetStr((Ptr + Pos + P1), 10, S) + 1); // engine version
        Inc(P1, Int64.Size); // file size
        X := EndianSwap(PInteger(Ptr + Pos + P1)^);
        Inc(P1, Integer.Size); // compressed structure size
        Y := EndianSwap(PInteger(Ptr + Pos + P1)^);
        Inc(P1, Integer.Size); // decompressed structure size
        Flags := EndianSwap(PInteger(Ptr + Pos + P1)^);
        Inc(P1, Integer.Size); // flags
        if Temp.Size < Y then
          Temp.Size := Y;
        // decompress the structure to get info
        if (Flags and $3F) = 3 then
          if LZ4_decompress_safe((Ptr + Pos + P1), Temp.Memory, X, Y) = Y then
          begin
            if Y > X then
            begin
              OutPos := Output.Position;
              Output.WriteBuffer(Temp.Memory^, Y);
              StreamInfo1.ActualPosition := Pos + P1;
              StreamInfo1.StorePosition := OutPos;
              StreamInfo1.OriginalSize := X;
              StreamInfo1.UnpackedSize := Y;
              StreamInfo1.InfoPosition := 0;
              StreamInfo1.InfoSize := 0;
              StreamInfo1.Checksum := crc32(0, (Ptr + Pos + P1), X);
              StreamInfo1.ID := GetID('lz4hc');
              StreamInfo1.Codec := GetCodec('lz4hc');
              StreamInfo1.Option := 10;
              StreamInfo1.Status := STATUS_PREDICTED;
              Info1.Add(StreamInfo1);
            end; // optional
            Inc(P1, X);
            P2 := 16;
            Count := EndianSwap(PInteger(PByte(Temp.Memory) + P2)^);
            Inc(P2, Integer.Size); // stream count
            for I := 0 to Count - 1 do
            begin
              Y := EndianSwap(PInteger(PByte(Temp.Memory) + P2)^);
              Inc(P2, Integer.Size); // decompressed stream size
              X := EndianSwap(PInteger(PByte(Temp.Memory) + P2)^);
              Inc(P2, Integer.Size); // compressed stream size
              Inc(P2, SmallInt.Size); // flag
              if Y > X then
              begin
                StreamInfo2.Position := MemPos + Pos + P1;
                StreamInfo2.OriginalSize := X;
                StreamInfo2.UnpackedSize := Y;
                StreamInfo2.ID := GetID('lz4hc');
                StreamInfo2.Codec := GetCodec('lz4hc');
                StreamInfo2.Option := 10;
                StreamInfo2.Status := STATUS_PREDICTED;
                Info2.Add(StreamInfo2);
                // lz4hc, level 10 (predicted)
              end;
              Inc(P1, X); // seek to next stream
            end;
            Inc(Pos, P1);
            continue;
          end;
      end;
    end;
    Inc(Pos);
  end;
end;

procedure XTool_Scan2(ctx: PPluginCtx; MemInput: TMemoryStream; MemPos: Int64;
  AvailSize, TotalSize: Cardinal; Output, Temp: TMemoryStream;
  Info1, Info2: TInfoStore);
begin

end;

procedure XTool_Process(ctx: PPluginCtx; MemInput1, MemInput2: TMemoryStream;
  Temp: TMemoryStream; var StreamInfo: TEncodeSI);
begin

end;

procedure XTool_Restore(ctx: PPluginCtx; Output: TStream;
  MemInput: TMemoryStream; Temp: TMemoryStream; var StreamInfo: TDecodeSI);
begin

end;

exports XTool_Init, XTool_Free, XTool_Codecs, XTool_Scan1, XTool_Scan2,
  XTool_Process, XTool_Restore;

begin

end.

SBalykov · 17 Авг 2019

Razor12911
Thanks, of course, for the code. But I had a completely different when I wrote my somewhat exaggerated message.
Simply, in the pursuit of data compression, recently so many plugins have been released lately that it becomes difficult to understand when and which one can be used and how they differ from each other. That is, in my opinion, it is much more important to develop existing plugins, rather than produce new ones.
This is my personal opinion and is not related either to you personally or to anyone else ...

Razor12911 · 18 Авг 2019

SBalykov

I guess you are right but even I wouldn't be doing this either if lz4 was such a problematic algorithm to make a precompressor for.

xatab · 18 Авг 2019

Razor12911 написал(а):
SBalykov

I guess you are right but even I wouldn't be doing this either if lz4 was such a problematic algorithm to make a precompressor for.

Thank you very much for Your work. Good luck in all your endeavors

Mickey1s · 23 Авг 2019

Haemimont Engine
Support for haemimont game engine files using lz4 algorithm.

Games
Victor Vran, Tropico 5 and many others

Usage
Place dll near xtool.exe (only during encoding)

Available codecs
hpk

Mickey1s · 30 Авг 2019

Frostbite Engine
Support for frostbite 3 game engine files using zlib, lz4, zstd and kraken algorithm.

Games
Battlefield, Need For Speed, FIFA, Mirror's Edge Catalyst and etc

Usage
Place dll near xtool.exe (if fb3.fifa19 codec is used, dll is required when decoding as well)

Available codecs
fb3, fb3.fifa19

Mickey1s · 9 Янв 2021

Unreal Engine

Supported games
Encrypted unreal engine 4 games

Methods available

ue4

Код:

m = method (1 = games that use borderlands3, street figher v... format, 2 = games that use rune ii format)
k = key in hexadecimal.

Usage
Place unreal.dll near xtool.exe and use -mue4:m1:kAAF9AF1F40225A1241C120533277804B09A56CE78 CBF100C131423788DB0E34F

Notes

The library is required both encoding and decoding.

This tool only decrypts like ue4dt but since xtool also does precompress data so if a game for example is zlib compressed and also encrypted. You can use -mzlib+ue4:m1:kAAF9AF1F40225A1241C120533277804B09A56CE78 CBF100C131423788DB0E34F

The order of the method does not matter, you can write it as -mzlib+ue4:m1:kAAF9A.. or mue4:m1:kAAF9A...+zlib, it will still work.

The game will be decrypted and precompressed in one go but you have to use -d1 parameter

Example

Код:

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+ue4:m1:kAAF9AF1F40225A1241C120533277804B09A56CE78CBF100C131423788DB0E34F -c32mb -t100p -d1 --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Benchmarks

ue4dt (side project)

Код:

Compressed 1 file, 244,260,523 => 244,260,523 bytes. Ratio 100.00%
Compression time: cpu 0.23 sec/real 17.06 sec = 1%. Speed 14.32 mB/s

Extracted 1 file, 244,260,523 => 244,260,523 bytes. Ratio 100.00%
Extraction time: cpu 0.42 sec/real 17.09 sec = 2%. Speed 14.29 mB/s

ue4 (plugin version)

Код:

Compressed 1 file, 244,260,523 => 244,271,444 bytes. Ratio 100.00%
Compression time: cpu 0.22 sec/real 1.59 sec = 14%. Speed 153.56 mB/s

Extracted 1 file, 244,271,444 => 244,260,523 bytes. Ratio 100.00%
Extraction time: cpu 0.59 sec/real 1.07 sec = 55%. Speed 228.27 mB/s

Plugin updated

Changes

Added Unreal Engine 3 support to Unreal Engine plugin for lzna compressed games (Mortal Kombat 11)

Tests

GEARASSETS_CYR_ScriptAssets.PSF

Код:

Compressed 1 file, 47,922,080 => 145,786,020 bytes. Ratio 304.21%
Compression time: cpu 0.05 sec/real 14.56 sec = 0%. Speed 3.29 mB/s

Tested 1 file, 145,786,020 => 47,922,080 bytes. Ratio 304.21%
Testing time: cpu 0.06 sec/real 13.97 sec = 0%. Speed 3.43 mB/s

Код:

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mue3 -c32mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

oo2core_5_win64.dll required from game.

Mickey1s · 14 Янв 2021

Ubisoft Dunia 2 Engine

Supported games
Far Cry Primal (requires liblz4.dll, lz4dunia.dll)
Far Cry 5 (requires liblz4.dll)
The Crew 2 (requires oo2core_5_win64.dll)
Watch Dogs 2 (requires liblz4.dll, lz4dunia.dll)
Watch Dogs Legion (requires liblz4.dll, lz4dunia.dll)

To be added
Far Cry 3
Far Cry 4
Far Cry New Dawn
The Crew
Watch Dogs

Usage
Place database (*.xtl file) near xtool.exe along with the required library. Use -mfc5 as an example

Mickey1s · 22 Янв 2021

Hitman

Supported games
Hitman 3

Usage
Place hitman.dll near xtool.exe along with required libraries and use -mhitman3

Notes
The library is required both encoding and decoding.

This game is encrypted and Hitman 2 key was used, I have no idea if its the correct key because Hitman 1 and Hitman 2 used the same key. I ran a test on a sample I was provided (I don't have game).

The data before and after decryption still looks gibberish so I don't know if it worked or not but a 9MB was saved after compression so if you have more information about this game or the hitman series let me know.

Edit:

Information useful for plugin's completion is taken from here:
https://zenhax.com/viewtopic.php?f=9&t=14768

Mickey1s · 4 Фев 2021

Bethesda

Supported games
The Elder Scrolls V: Skyrim (requires liblz4.dll)

Usage
Place bsa.dll near xtool.exe along with required libraries and use -mskyrim

Notes

The library is required both encoding and decoding.

Some results
Skyrim - Textures0.bsa

Код:

Compressed 1 file, 655,189,115 => 1,436,500,795 bytes. Ratio 219.25%
Compression time: cpu 0.69 sec/real 15.58 sec = 4%. Speed 42.06 mB/s

Tested 1 file, 1,436,500,795 => 655,189,115 bytes. Ratio 219.25%
Testing time: cpu 0.69 sec/real 7.27 sec = 9%. Speed 90.11 mB/s

XTool 2019 (Plugins) 2019-08-13

Ветеран

Старожил

Старожил

Старожил

Старожил

Старожил

Старожил

Новичок

Старожил

Новичок

Новичок

Ветеран

Вложения

Ветеран

Вложения

Ветеран

Вложения

Ветеран

Вложения

Ветеран

Вложения

Ветеран

Вложения