I've dropped Kernel 3.18 into my slacko 5.3.3 and compiled/installed lz4, lz4cat, and versions of mksquashfs and unsquashfs that support gzip, xz, lzo and lz4.
Created a initrd.lz4 that boots my pup just fine. I can make lz4 squashed file systems, view and unsquash them - but I can't get sfs_load to load those sfs's.
The kernel only supports booting of legacy lz4, so when creating the initrd.lz4 I used the -l parameter (legacy switch), but mksquashfs/unsquashfs don't have that legacy version as a command line parameter/option
Any idea as to how to resolve this and get puppy correctly creating/loading legacy lz4 compressed sfs's.
TIA
Where to find LZ4 support for my modified Slacko 5.3.3?
I originally posted this thread in cutting edge, but it was moved presumably by a moderator without any advisement.
Likely doomed to fall down through the pages unnoticed here ... but no matter. I wasn't going to add any further to this thread however kernel support of LZ4 kernel compression is relatively new, not implemented in any (that I know of) Linux systems. Puppy could have been a leader on that front, but as I said now likely set to just be ignored.
Just in case anyone thinking about implementing LZ4 kernel compression searches and finds this thread - some observations.
The current 3.18 kernel implementation of lz4 support is based on the legacy version. Whilst lz4 is noted to have around 3 times faster decompression rates than lzo, and lzo is around 3x quicker than gzip (so lz4 conceptually close to 10x quicker than gzip), that's based on the later streaming based lz4.
After implementing a lz4 based initrd with pup sfs built into that, I didn't see much difference between that and a lzo based initrd of the same puppy. If anything when running and extracting just parts of the larger compressed image, lzo appears to be faster. That's potentially due to legacy lz4 using relatively small 8K sized block size.
As a quick test I took Vanguard Carolina's 518MB ISO file and compressed using both lz4 legacy and lzop level 1 and lz4 was faster, taking 33 seconds compared to lzo 48.5 seconds. For decompression lz4 took 32.8 seconds compared to lzo's 35 seconds. In the way of comparison, gzip level 1 took 1m4s to compress, 33.37 seconds to decompress. These figures are all for a old single core machine. In my book that suggests there all equal on decompression speed, whilst lz4 was quicker on the compression front.
Moving to a multi-core (4) machine and applying tests to a already compressed file of around 200MB, gzip level 1 compression (fastest) had gzip taking 11.28 seconds, lz4 legacy 0.67 seconds, lz4 standard 1.03 seconds and lzo 5.53 seconds. For decompression gzip 2.89 seconds, lz4 legacy 0.35 seconds, lz4 standard 0.69 seconds, lzo 0.5 seconds.
Just a quick and far from a exhaustive or scientific standard test, my own personal conclusions however are that whilst both lz4 and lzo are faster than gzip, the benefits of lz4 legacy over lzo (lzop in Puppy) are not wide enough on the decompression side to warrant migrating from lzo to lz4 but might be appropriate/viable where faster compression speeds are desirable. In compress-once, decompress-many application such as creating/using sfs's, the benefits of lz4 over lzo are small. In real world utilisation there is also a question of lzo potentially being faster at extracting blocks out of the compressed files than lz4, but I've not tested that - just a observation of general running that is subject to many other possible factors.
It would have been nice to be able to sfs load the sfs's I create with a lz4 supported mksquashfs that I'm currently using, but as per my first posting I can't do that. That would have enabled better testing of running applications within a large sfs and testing the differences of lzo versus lz4 in grabbing out and executing those sub parts of the larger sfs. For now however I'll just put the 3.18 lz4 supporting kernel/puppy on the back burner to see how others go on to implement lz4 based kernels.
Likely doomed to fall down through the pages unnoticed here ... but no matter. I wasn't going to add any further to this thread however kernel support of LZ4 kernel compression is relatively new, not implemented in any (that I know of) Linux systems. Puppy could have been a leader on that front, but as I said now likely set to just be ignored.
Just in case anyone thinking about implementing LZ4 kernel compression searches and finds this thread - some observations.
The current 3.18 kernel implementation of lz4 support is based on the legacy version. Whilst lz4 is noted to have around 3 times faster decompression rates than lzo, and lzo is around 3x quicker than gzip (so lz4 conceptually close to 10x quicker than gzip), that's based on the later streaming based lz4.
After implementing a lz4 based initrd with pup sfs built into that, I didn't see much difference between that and a lzo based initrd of the same puppy. If anything when running and extracting just parts of the larger compressed image, lzo appears to be faster. That's potentially due to legacy lz4 using relatively small 8K sized block size.
As a quick test I took Vanguard Carolina's 518MB ISO file and compressed using both lz4 legacy and lzop level 1 and lz4 was faster, taking 33 seconds compared to lzo 48.5 seconds. For decompression lz4 took 32.8 seconds compared to lzo's 35 seconds. In the way of comparison, gzip level 1 took 1m4s to compress, 33.37 seconds to decompress. These figures are all for a old single core machine. In my book that suggests there all equal on decompression speed, whilst lz4 was quicker on the compression front.
Moving to a multi-core (4) machine and applying tests to a already compressed file of around 200MB, gzip level 1 compression (fastest) had gzip taking 11.28 seconds, lz4 legacy 0.67 seconds, lz4 standard 1.03 seconds and lzo 5.53 seconds. For decompression gzip 2.89 seconds, lz4 legacy 0.35 seconds, lz4 standard 0.69 seconds, lzo 0.5 seconds.
Just a quick and far from a exhaustive or scientific standard test, my own personal conclusions however are that whilst both lz4 and lzo are faster than gzip, the benefits of lz4 legacy over lzo (lzop in Puppy) are not wide enough on the decompression side to warrant migrating from lzo to lz4 but might be appropriate/viable where faster compression speeds are desirable. In compress-once, decompress-many application such as creating/using sfs's, the benefits of lz4 over lzo are small. In real world utilisation there is also a question of lzo potentially being faster at extracting blocks out of the compressed files than lz4, but I've not tested that - just a observation of general running that is subject to many other possible factors.
It would have been nice to be able to sfs load the sfs's I create with a lz4 supported mksquashfs that I'm currently using, but as per my first posting I can't do that. That would have enabled better testing of running applications within a large sfs and testing the differences of lzo versus lz4 in grabbing out and executing those sub parts of the larger sfs. For now however I'll just put the 3.18 lz4 supporting kernel/puppy on the back burner to see how others go on to implement lz4 based kernels.
-
battleshooter
- Posts: 1378
- Joined: Wed 14 May 2008, 05:10
- Location: Australia
This LZ4 stuff is quite interesting. Thank you for bringing it to my attention rufwoof.
I've never heard of it before so I may not know what I'm speaking of, but by the sounds of it, LZ4 is a super efficient compressor that compresses more compactly and is able to compress and decompress faster than common compression types such as gzip? As well as this speed significantly increases with each additional CPU core?
It sounds like a dream come true if it really does all that and a solution to my concerns of Vanguard's growing size.
You seem to have played around with it a bit, do you have further thoughts on how to implement LZ4's advantages in a Puplet?
I've never heard of it before so I may not know what I'm speaking of, but by the sounds of it, LZ4 is a super efficient compressor that compresses more compactly and is able to compress and decompress faster than common compression types such as gzip? As well as this speed significantly increases with each additional CPU core?
It sounds like a dream come true if it really does all that and a solution to my concerns of Vanguard's growing size.
You seem to have played around with it a bit, do you have further thoughts on how to implement LZ4's advantages in a Puplet?
[url=http://www.murga-linux.com/puppy/viewtopic.php?t=94580]LMMS 1.0.2[/url], [url=http://www.murga-linux.com/puppy/viewtopic.php?t=94593]Ardour 3.5.389[/url], [url=http://www.murga-linux.com/puppy/viewtopic.php?t=94629]Kdenlive 0.9.8[/url]
This does sound cool. faster decompression is needed, looks like every one in the boot game (kernel, Grub, Squashfs ) is being supported with in the last two months. Wish MODS would move this back to cutting edge, its not ready for regular users yet.
It may be the why its titled, seems too direct a request, "slacko 5.3.3"
Looks like something best tried in cutting edge projects with newer everything QuirkyApril, FD7, etc.
While would not help you directly I am going to investigate this for FD7b2 and may start a general cutting edge thread, this is a game changer.
It may be the why its titled, seems too direct a request, "slacko 5.3.3"
Looks like something best tried in cutting edge projects with newer everything QuirkyApril, FD7, etc.
While would not help you directly I am going to investigate this for FD7b2 and may start a general cutting edge thread, this is a game changer.
http://fastcompression.blogspot.com/201 ... on-is.html
A variant for distro developers ( very slow compression but savings an additional 20% ) and compatible
A variant for distro developers ( very slow compression but savings an additional 20% ) and compatible
-
stemsee
Intereting. I have been trying to compile and use it. My kernel 4.0.1 su[pposrts it and I can load the modules. I built the tools and I can compress files using 'lz4 arg source dest'. I built squashfs with lz4 support (I thought) but how to implement it? Help shows only gzip and xz. Any pointes appreciated.
- Attachments
-
- lz4-r128-i686.pet
- (154.74 KiB) Downloaded 262 times
-
- squashfs4.3-i686.pet
- (114.09 KiB) Downloaded 273 times
I have mksquashfs5 (and unsquashfs5) in /usr/sbin, mksquashfs and unsquashfs are sym linked to those
I also have liblz4.so.1.5.0 in /usr/lib together with symlinks to that of liblz4.so.1 and liblz4.so
I compiled them some time back and have lost the source code reference
Files attached, all actually gzip'd (need to be uncompressed).
With the above I use (assuming puppy is extracted to a directory called extracted_pup)
mksquashfs extracted_pup puppy_tahr_6.0.5.sfs -comp lz4
I also have liblz4.so.1.5.0 in /usr/lib together with symlinks to that of liblz4.so.1 and liblz4.so
I compiled them some time back and have lost the source code reference
Files attached, all actually gzip'd (need to be uncompressed).
With the above I use (assuming puppy is extracted to a directory called extracted_pup)
mksquashfs extracted_pup puppy_tahr_6.0.5.sfs -comp lz4
- Attachments
-
- mksquashfs5.gz
- (103.47 KiB) Downloaded 237 times
-
- unsquashfs5.gz
- (64.71 KiB) Downloaded 192 times
-
- liblz4.so.1.5.0.gz
- (30.31 KiB) Downloaded 222 times
Hello all.
Latest version as of this writing. (Please see attached.)
Compiled today from the github master source at
-- https://github.com/Cyan4973/lz4 --
on Slim-6, a puplet derivative of Slacko-6.3_32-bit.
Enjoy!
Latest version as of this writing. (Please see attached.)
Compiled today from the github master source at
-- https://github.com/Cyan4973/lz4 --
on Slim-6, a puplet derivative of Slacko-6.3_32-bit.
Enjoy!
- Attachments
-
- lz4-2016-04-18.pet
- (194.54 KiB) Downloaded 210 times
Last edited by musher0 on Thu 21 Apr 2016, 00:06, edited 1 time in total.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)
-
Sailor Enceladus
- Posts: 1543
- Joined: Mon 22 Feb 2016, 19:43
I've always just gunzipped everything. This thread opened my eyes to why mksquashfs gives even better results though, xz compression! It looks like the slackware crew have all moved to xz now too. Did not realize xz was so cool before, decompress speeds are not too shabby compared to gz too: https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
I did play with rufwoof's mksquashfs5 and musher0's liblz4.1.7.1. Pretty neat, though if you really want fast speeds just taring with no compression seems twice as fast. Re-squashing a 500MB filesystem with lz4 took me literally only 1 minute though, whereas with xz it takes more like 25 minutes - heh. The filesize was only about 2x also. Nice to know there are many options.
I did play with rufwoof's mksquashfs5 and musher0's liblz4.1.7.1. Pretty neat, though if you really want fast speeds just taring with no compression seems twice as fast. Re-squashing a 500MB filesystem with lz4 took me literally only 1 minute though, whereas with xz it takes more like 25 minutes - heh. The filesize was only about 2x also. Nice to know there are many options.
