packJPG Multi-threaded

[YadeWira]

Hey everyone! Here's my fork of packJPG. One of its new features is multi-threading, along with a few other things you can check out on my GitHub. I look forward to your feedback and ideas!!!

GitHub - YadeWira/packJPG: A compression program for further compressing JPEG image files

A compression program for further compressing JPEG image files - YadeWira/packJPG

github.com

Version 3.0 is still in the testing phase; please report any results or errors. Please do not use it on important files.

packJPG

--> packJPG v3.0 (03/25/2026) by Yade Bravo <--
Copyright 2006-2026 Yade Bravo & Matthias Stirner
All rights reserved


packJPG -- lossless JPEG compression. Typical reduction: ~20%.
Compresses JPEG files to PJG format and decompresses them back,
with bit-for-bit identical reconstruction.

Website: https://github.com/YadeWira/packJPG
Email  : packjpg (at) matthiasstirner.com

Usage: packjpg <subcommand> [switches] [filename(s)]

Subcommands:
 a         compress only: process JPG files, skip PJG
 x         decompress only: process PJG files, skip JPG
 mix       mixed mode: auto-detect (warns if both directions used)
 list      list PJG file info without decompressing

Switches:

 [-ver]   verify files after processing
 [-v?]    set level of verbosity (max: 2) (def: 0)
 [-np]    no pause after processing files
 [-o]     overwrite existing files
 [-sfth]  use 3 cores for single-file compression (pre-pack stages)
 [-th?]   set number of threads (0=auto, def: 1)
 [-r]     recurse into subdirectories
 [-list]  list PJG file info without decompressing
 [-dry]   dry run: simulate without writing output files
 [-module] machine-friendly output: OK/ERROR + time only
 [-od<p>] write output files to directory <p>
 [-p]     proceed on warnings
 [-d]     discard meta-info

Examples: "packjpg -v1 -o baboon.pjg"
          "packjpg -p *.jpg"

Key Features

Multi-threaded mode (-th)

The "-th<n>" switch enables parallel batch processing using n worker
threads. Use "-th0" to auto-detect the number of CPU cores (on x86
builds the auto limit is 2 to prevent out-of-memory errors with large
images; on x64 there is no cap).

In multi-threaded mode, verification is always enabled automatically:
each file is compressed and immediately decompressed and compared
bit-for-bit before the output is written. This ensures no silent
corruption can occur even under heavy parallel load.

Pressing Ctrl+C during multi-threaded processing stops all workers
cleanly, removes any partial output files, and prints a summary of
how many files were completed before the interrupt.

Single-threaded mode (default, no -th flag) behaves exactly as in
previous versions.

Single-file parallel mode (-sfth)

Standard packJPG processes the components of a JPEG (Y, Cb, Cr)
sequentially — one after another in a single thread. The -sfth switch
changes this by encoding all three components simultaneously, each in
its own thread. This is fundamentally different from -th, which
parallelizes across files, not within a single file.

To make the distinction clear:

 -th<n>   : runs N files at the same time, each file uses 1 thread
 -sfth    : runs 1 file at a time, but splits it into 3 parallel threads
 -th<n> -sfth : runs N files at the same time, each using 3 threads

This means -sfth is useful even when processing a single file, while
-th only helps when processing multiple files in a batch.

Benchmark on a single file (Intel Xeon E5-2697 v4):

 without -sfth :  0.23 sec  1.81 MB/s  ratio 67.29%
 with    -sfth :  0.16 sec  2.54 MB/s  ratio 67.30%

The ratio difference (0.01%) comes from the fact that each component
uses its own independent arithmetic coder context. This is the expected
and documented behavior — files are still valid and fully lossless.

Important notes:
 - Files compressed with -sfth use a new PJG format (0x01 marker).
   They require packJPG v3.0 or later to decompress. Attempting to
   decompress them with v2.x will produce a clean error message.
 - Files compressed without -sfth remain fully compatible with v2.x.
 - A warning is shown if -sfth is used with fewer than 3 detected cores.

Optimal usage for machines with N threads:

 "packJPG a -th<N/3> -sfth -o -np *.jpg"

This fills all N cores: N/3 files in parallel, each using 3 threads.
Example on an 18-core machine: -th6 -sfth = 6 x 3 = 18 threads.

For single-file compression: -sfth alone is sufficient.
For batch-only parallelism without intra-file: use -th<n> without -sfth.

FreeArc integration (not tested, report results)

packJPG can be used as an external compressor in FreeArc, acting as a
JPEG preprocessor before FreeArc applies its own compression on top.
Since FreeArc processes one file at a time in this mode, -sfth is the
right flag to use here — -th would have no effect.

Add the following to your arc.ini:

[External compressor:jpg]
packcmd   = packjpg a -sfth -module -np -o $$arcdatafile$$.jpg
unpackcmd = packjpg x -sfth -module -np -o $$arcdatafile$$.pjg
datafile   = $$arcdatafile$$.jpg
packedfile = $$arcdatafile$$.pjg
solid = 0

Known Limitations

packJPG is a compression program for JPEG files only. Other file types
are silently skipped.

packJPG has low error tolerance. JPEG files might not work with packJPG
even if they work perfectly with other image processing software. This
happens because packJPG needs to understand the internal structure of a
JPEG deeply enough to re-compress the DCT coefficients — it is more
strict than a regular image viewer that just renders the pixels.

Common causes of warnings or errors that -p can work around:

 - Inefficient Huffman coding: some encoders generate Huffman tables
   where the last AC coefficient in a block is zero. Technically valid,
   but packJPG cannot reconstruct it bit-for-bit without -p.
 - Incorrect RST markers: restart markers inserted at wrong positions
   or with wrong counters. Other decoders ignore them; packJPG counts
   and validates them.
 - Inconsistent padding bits: bits used to fill the last byte of a
   Huffman scan. The spec requires 1-bits; some encoders write 0-bits.
 - Garbage data after EOI: some files have extra bytes after the end-
   of-image marker. packJPG preserves them but may warn on edge cases.

With -p, packJPG accepts all these cases and compresses them anyway.
The reconstructed image will be visually identical but may not be
bit-for-bit equal to the original. For this reason, -p should never
be combined with -ver.

If you try to drag and drop too many files at once on Windows, there
might be a windowed error message about missing privileges. In that
case try again with fewer files or use the command prompt instead.

Compressed PJG files are not compatible between different packJPG
versions. You will get an error message if you try to decompress PJG
files with a different version than the one used for compression. You
may download older versions of packJPG from:
https://github.com/packjpg/packJPG

On 32-bit Windows builds (x86), the ratio and speed summary may display
as 0.00% and 0.00 MB/s. This is a display-only cosmetic issue — the
compressed files are valid and can be decompressed normally. It affects
the statistics output only and is caused by integer size limitations in
32-bit builds. Use -th2 maximum on 32-bit to avoid out-of-memory errors
with large images.