Whoops, since when can zstdcat decompress #zstd, #gzip AND #xz files? Very useful!

Ah, apparently it can be compiled in. On #Manjaro it's linked against gzip, xz and lz4, on #NixOS (by default?) it can only decompress zstd.

Does anyone know of a tool which will inspect a ZStandard dictionary file? #zstd

#zstd 1.5.7 is out and it's honestly quite amazing.

Highlights for me are:

- ~10% faster at small block sizes common in databases(=filesystems?)
- Limited multi-threading by default. (You could already enable it manually; this only concerns the defaults.)
- A new --max flag that inches zstd closer to #lzma/#xz. We'll need to see more testing on how close exactly but it's impressive they managed to broaden the range this far in a single format.

https://github.com/facebook/zstd/releases/tag/v1.5.7

Tein vähän #pakkaus-kokeiluja törmättyäni taannoin uuteen pakkaimeen, #Bzip3:een. Ainakin minun tiedostojeni #varmuuskopiointi-pakkaamisessa se hävisi pakkausteholtaan selvästi #XZ:lle, jota olen varmuuskopiointiin käyttänyt, ja pakkausnopeudeltaan selvästi #ZStd:lle, johon siirtymistä olin aprikoinut. #atkjuttuja

El cliente #Moshidon (por cierto, muy malo, a nivel técnico) no se lleva bien con la compresión #zstd, pero con #brotli no tiene problemas

I love playing around with #compression

In this case, it's all text-based data in csv and xml formats.

Size:

32,696,320 202411.tar
 4,384,020 202411.tar.bz2
 4,015,912 202411.tar.zst
 3,878,583 202411.tar.bz3
 3,730,416 202411.tar.xz

zstd was invoked using zstd --ultra -22
xz was invoked using xz -9e
bzip2 was invoked using bzip2 -9
bzip3 has no compression level options

Speed:

zstd    54.31user 0.25system 0:54.60elapsed 99%CPU
xz      53.80user 0.06system 0:53.93elapsed 99%CPU
bzip2    5.33user 0.01system 0:05.35elapsed 99%CPU
bzip3    3.98user 0.02system 0:04.01elapsed 99%CPU

Maximum memory usage (RSS):

zstd    706,312
xz      300,480
bzip3    75,996
bzip2     7,680

*RSS sampled up to ten times per second during execution of the commands in question

#bzip3 is freaking amazing, yo.

#DataCompression #bzip #bz3 #zstd #zst #zstandard #xz #lzma
#CouldaBeenABlost ;)

#komprimace #gnu_linux
#Zstd je neskutečné dobrý #opensource komprimační algoritmus, měl by se víc propagovat (a používat).
Složky s webovým obsahem (HTML, CSS, obrázky atd.), celkem 220 Mb dat.
Jednovláknová komprimace do .tar.gz za 8 sekund a výsledek: 147 Mb.
Vícevláknová komprimace do .tar.gz za 1 sekundu (a taky 147 Mb).
Vícevláknová komprimace do .tar.zst za 1 sekundu a výsledek: 15 Mb (!!!)
GNU #gzip víc než 1 CPU nepodporuje, ale #pigz ano, který jsem použil pro porovnání rychlosti.

#RStats hottake

I'm pretty tired of the slow, inefficient compression utilities available by default.

gzip is fast but not great compression
xz is great compression but an absolute dog in terms of speed
bzip2 is somewhere inbetween the extremes.

Why not include #Zstd ?

Lol, I forgot how much better #Brotli is than gzip & bz2… I tried to bzip a 80GB file and it was taking forever. I'll need to do a benchmark, but on a small file it's:

gz: down to 32%, 6s
bz2: 17%, 2:20
brotli: 2% (!!), 2:40
brotli -6: 3%, 10s
(unbrotli was also like <10s)

How does #zstd compare? (I haven't tried it yet)

Glad to finally have #Zstd compression enabled for #AlpineLinux's linux-firmware package exactly half a year (minus a few hours) after originally opening the merge request during our post-Fosdem #postmarketOS hackathon!

If you're running Alpine (or postmarketOS) and have all linux-firmware packages installed, the on-disk space will go down from around a gigabyte to around half a gigabyte or so, which I think is pretty decent.

TIL: #ZFS mit #zstd komprimiert eine #prometheus Datenbank mit Faktor 2,5!

#Docker support for #zstd is a mess

#Linux Weekly Roundup for April 7th, 2024: #FFmpeg 7.0, #LinuxMint 22 adopts #PipeWire and packs #Thunderbird as DEB, Linux 6.7 EOL, #Nitrux switches to #Debian repos, #fwupd adopts #zstd, #Kodi 21, new Linux laptop from #System76, explicit sync to boost #Wayland's adoption, and more https://9to5linux.com/9to5linux-weekly-roundup-april-7th-2024

#Linux Firmware Updater #Fwupd 1.9.16 Adopts #Zstd for Compressing Metadata, Brings Support for #Acer T34 & U33 Docks and #Qualcomm Series 5 Devices https://9to5linux.com/fwupd-1-9-16-brings-support-for-acer-t34-and-u33-docks-qualcomm-series-5-devices

After the recent xz-utils attack, guess what was the response some developers thought of?
9to5 Linux, Phoronix
Instead of helping Lasse Collin, the xz-utils maintainer who was tricked and mentally abused, they jumped ship, because the new solution is "more dependable". Wow, I applaud for this stupidity.
Let's shift our gaze somewhere else a bit. netfilter, the management framework of network operations on Linux that's used by virtually every Linux distribution, effectively only has Pablo Neira Ayuso left to maintain the project after Florian Westphal quit the core team. strace only has Dmitry V. Levin there to keep the cogs running. tcpdump and libpcap have only very few people to maintain the lights. And Bash should probably get abandoned with the few people there to keep everything up. The list goes on and on, because this is the freaking norm!
With the current mindset, support of any form, be it encouragement, financial support or contributing, is way too expensive for anyone to give out. I suggest just don't offer the maintainers the love and help they deserve, and speed up the downfall of the current landscape.
To every FOSS developer out there who has been thanklessly maintaining projects, please accept my deepest gratitude. However, to those who either shifted the blame to the xz-utils project and Lasse Collin, or jumped ship because xz-utils is deemed "unsafe" by you, I have two words most suitable for you:
FUCK YOU.

#xz #xzutils #zstd #fwupd #Linux #opensource #FOSS #FLOSS #ffmpeg

#Linux Firmware Update Utility #Fwupd Will Use #Zstd Compression for Future Releases https://9to5linux.com/linux-firmware-update-utility-fwupd-will-use-zstd-compression-for-future-releases