Важно! ldmf не работает с -mtt1, более того, он крайне бесполезен для -mtt0 с числом тредов больше 1.
So I understand that with mtt1, ldmf is basically like disabled no matter what you do and with mtt0 effect will be almost negligible if -mt > 1, therefore even worse than srep?
-mt1 -mtt0 are thus the only reasonable options for benefiting ldmf and for everything else srep is better?
EDIT: I made 1 quick test with 3 PS1 iso's -> all together 2gb. I only tried -mt4 and with -mtt0 and -mtt1. It really seem to be not working well with -mt4 in either case and giving pretty much same size as with -ldmf0. I get better result with srep+lolz:ldmf0.
From what it looks like to me lolz do not process all data in 1 sweep aka "srep" and then parse it all to matchfinder\cores, it seems rather to work as "per block" or "dictionary", each separately doing its own ldmf in memory? Something like this perhaps?:
data ->mt1 -> lmdf -> lolz1
->mt2 -> lmdf -> lolz2
->mt3 -> lmdf -> lolz3
->mt4 -> lmdf -> lolz4
If so then it would make sense why -mt1 -mtt0 is needed. But this is not be the best solution because lolz is already slow enough and reverting to single core for a feature is really too much with minimal gain against srep.
If I may suggest few ideas:
- much better approach would be to first parse all data at once just like srep does and then compress it. This would make ldmf much more useful regardless of rest of lolz settings. Its basically same behavior as: srep - - <stdin><stdout>.
- rather than using arbitrary numbers for ldc(and maybe ldl), better way would be to directly specify size in b/mb/gb, for both compression and decompression, including chunk size - again similarly to srep. Because user then knows exactly what he get and what data and limits is (s)he working with. Something like -ldl512b(ytes) -ldc4gb(cmp max mem usage) -ldd4gb(decmp. max mem usage) {with ldc and ldd rest would go to swap if limit crossed, for example...}.
-in terms of gaining best speed and maximum possible ratio, ldmf should work exactly like srep with one important exception:
during the "srep" sweep process it would be gentle toward specific data patterns that are known to be better processed by lolz only(aka textures), thus skipping such chunks for example or rearranging them as needed. That is the main thing missing in srep and what was ldmf trying to accomplish in the first place.
In any case, thank you ProFrager. You clearly worked hard during Christmas instead of wasting time drinking and for than you deserve praise. Btw new lolz is still useful contribution, I found it slightly quicker while also consuming less memory on same settings, so thank you again.
EDIT: I have to apologize to ProFrager, I made some tests yesterday and came to better understanding of lolz inner working especially regarding MT/ldmf, at this point I think its current implementation is not so bad at all(You can read about it here:
https://fileforums.com/showthread.php?t=102617 ). Only some questions I have left:
- could you describe more in details how different -lde implementations work?
- if -ldc is specified as "ratio", does it literally mean that ldc1 := mem=file_size(1:1), ldc2:=mem=file_size/2, ldc5:=1/5 of mem usage vs file size etc..?
- ldl8 = 256bytes according to specified calculation correct? Thus ~equal to srep -l256, yet memory usage from the mentioned math (src_size/((2 ^ [ldl]) / 16)) mean that 100gb file and ldl8 would only consume ~6.25gb for ldmf? That would be significantly less than srep, is the reason because of tight implementation with compressor?
- if used srep instead of ldmf, would be better to set -dm00 instead of leaving auto detection?
- how is the math "simplified" in fbb vs fba? Also do I understand that they trigger on values > than specified? Example would fba=256, fbb=1024 mean hardest calculations up to 256, simpler between 256-1024 and simplest above >1024?
Thank you
флуд однозначно