Steam is working on adding zstd compression for game chunks (every game file is split into 1MB), it currently uses LZMA.
Wonder how much of an overall improvement it will be.
“Better” doesn’t always mean “smaller”, especially in this example. LZMA’s strength is that it compresses very small but its weakness is that it’s extremely CPU-intensive to decompress. Switching to ZSTD will actually result in larger downloads, but the massively reduced CPU load of decompressing ZSTD will mean it’s faster for most users. Instead of just counting the time it takes for the data to transfer, this is factoring in download time + decompression time. Even though ZSTD is somewhat less efficient in terms of compression ratio, it’s far more efficient computationally.
Bet that’ll save Valve on some server costs too. Storage is much cheaper than compute (though I imagine they’ll probably keep LZMA around for clients on slow connections).
Hmm true. I was thinking that steam has a lot of games and respective builds it has to compress, even if the decompression benefits are clientside only.
Each new game update would also be compressed too - I have no idea how Steam handles the update to work out what files need replacing on their end though, which might involve decompressing the files to analyse them.
“Better” doesn’t always mean “smaller”, especially in this example. LZMA’s strength is that it compresses very small but its weakness is that it’s extremely CPU-intensive to decompress. Switching to ZSTD will actually result in larger downloads, but the massively reduced CPU load of decompressing ZSTD will mean it’s faster for most users. Instead of just counting the time it takes for the data to transfer, this is factoring in download time + decompression time. Even though ZSTD is somewhat less efficient in terms of compression ratio, it’s far more efficient computationally.
Bet that’ll save Valve on some server costs too. Storage is much cheaper than compute (though I imagine they’ll probably keep LZMA around for clients on slow connections).
Doesn’t decompression only happen client-side? I don’t imagine them compressing the files multiple times.
Hmm true. I was thinking that steam has a lot of games and respective builds it has to compress, even if the decompression benefits are clientside only.
Each new game update would also be compressed too - I have no idea how Steam handles the update to work out what files need replacing on their end though, which might involve decompressing the files to analyse them.
“most users” have gigabit pipes to the internet?