Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 03 Aug 2015, 13:19
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Where to find LZ4 support for my modified Slacko 5.3.3?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [7 Posts]  
Author Message
rufwoof

Joined: 24 Feb 2014
Posts: 644

PostPosted: Fri 23 Jan 2015, 20:04    Post subject:  Where to find LZ4 support for my modified Slacko 5.3.3?  

I've dropped Kernel 3.18 into my slacko 5.3.3 and compiled/installed lz4, lz4cat, and versions of mksquashfs and unsquashfs that support gzip, xz, lzo and lz4.

Created a initrd.lz4 that boots my pup just fine. I can make lz4 squashed file systems, view and unsquash them - but I can't get sfs_load to load those sfs's.

The kernel only supports booting of legacy lz4, so when creating the initrd.lz4 I used the -l parameter (legacy switch), but mksquashfs/unsquashfs don't have that legacy version as a command line parameter/option

Any idea as to how to resolve this and get puppy correctly creating/loading legacy lz4 compressed sfs's.

TIA
Back to top
View user's profile Send private message 
rufwoof

Joined: 24 Feb 2014
Posts: 644

PostPosted: Sat 24 Jan 2015, 09:48    Post subject:  

I originally posted this thread in cutting edge, but it was moved presumably by a moderator without any advisement.

Likely doomed to fall down through the pages unnoticed here ... but no matter. I wasn't going to add any further to this thread however kernel support of LZ4 kernel compression is relatively new, not implemented in any (that I know of) Linux systems. Puppy could have been a leader on that front, but as I said now likely set to just be ignored.

Just in case anyone thinking about implementing LZ4 kernel compression searches and finds this thread - some observations.

The current 3.18 kernel implementation of lz4 support is based on the legacy version. Whilst lz4 is noted to have around 3 times faster decompression rates than lzo, and lzo is around 3x quicker than gzip (so lz4 conceptually close to 10x quicker than gzip), that's based on the later streaming based lz4.

After implementing a lz4 based initrd with pup sfs built into that, I didn't see much difference between that and a lzo based initrd of the same puppy. If anything when running and extracting just parts of the larger compressed image, lzo appears to be faster. That's potentially due to legacy lz4 using relatively small 8K sized block size.

As a quick test I took Vanguard Carolina's 518MB ISO file and compressed using both lz4 legacy and lzop level 1 and lz4 was faster, taking 33 seconds compared to lzo 48.5 seconds. For decompression lz4 took 32.8 seconds compared to lzo's 35 seconds. In the way of comparison, gzip level 1 took 1m4s to compress, 33.37 seconds to decompress. These figures are all for a old single core machine. In my book that suggests there all equal on decompression speed, whilst lz4 was quicker on the compression front.

Moving to a multi-core (4) machine and applying tests to a already compressed file of around 200MB, gzip level 1 compression (fastest) had gzip taking 11.28 seconds, lz4 legacy 0.67 seconds, lz4 standard 1.03 seconds and lzo 5.53 seconds. For decompression gzip 2.89 seconds, lz4 legacy 0.35 seconds, lz4 standard 0.69 seconds, lzo 0.5 seconds.

Just a quick and far from a exhaustive or scientific standard test, my own personal conclusions however are that whilst both lz4 and lzo are faster than gzip, the benefits of lz4 legacy over lzo (lzop in Puppy) are not wide enough on the decompression side to warrant migrating from lzo to lz4 but might be appropriate/viable where faster compression speeds are desirable. In compress-once, decompress-many application such as creating/using sfs's, the benefits of lz4 over lzo are small. In real world utilisation there is also a question of lzo potentially being faster at extracting blocks out of the compressed files than lz4, but I've not tested that - just a observation of general running that is subject to many other possible factors.

It would have been nice to be able to sfs load the sfs's I create with a lz4 supported mksquashfs that I'm currently using, but as per my first posting I can't do that. That would have enabled better testing of running applications within a large sfs and testing the differences of lzo versus lz4 in grabbing out and executing those sub parts of the larger sfs. For now however I'll just put the 3.18 lz4 supporting kernel/puppy on the back burner to see how others go on to implement lz4 based kernels.
Back to top
View user's profile Send private message 
battleshooter


Joined: 14 May 2008
Posts: 1259
Location: Australia

PostPosted: Sun 25 Jan 2015, 09:18    Post subject:  

This LZ4 stuff is quite interesting. Thank you for bringing it to my attention rufwoof.

I've never heard of it before so I may not know what I'm speaking of, but by the sounds of it, LZ4 is a super efficient compressor that compresses more compactly and is able to compress and decompress faster than common compression types such as gzip? As well as this speed significantly increases with each additional CPU core?

It sounds like a dream come true if it really does all that and a solution to my concerns of Vanguard's growing size.

You seem to have played around with it a bit, do you have further thoughts on how to implement LZ4's advantages in a Puplet?

_________________
LMMS 1.0.2, Ardour 3.5.389, Kdenlive 0.9.8
Back to top
View user's profile Send private message 
Ted Dog


Joined: 13 Sep 2005
Posts: 3348
Location: Heart of Texas

PostPosted: Sun 25 Jan 2015, 14:08    Post subject:  

This does sound cool. faster decompression is needed, looks like every one in the boot game (kernel, Grub, Squashfs ) is being supported with in the last two months. Wish MODS would move this back to cutting edge, its not ready for regular users yet.
It may be the why its titled, seems too direct a request, "slacko 5.3.3"

Looks like something best tried in cutting edge projects with newer everything QuirkyApril, FD7, etc.

While would not help you directly I am going to investigate this for FD7b2 and may start a general cutting edge thread, this is a game changer. Wink
Back to top
View user's profile Send private message 
Ted Dog


Joined: 13 Sep 2005
Posts: 3348
Location: Heart of Texas

PostPosted: Sun 25 Jan 2015, 14:12    Post subject:  

http://code.google.com/p/lz4/<<= main project page/source Shocked

wow look at that decompression rate!!!!
Back to top
View user's profile Send private message 
Ted Dog


Joined: 13 Sep 2005
Posts: 3348
Location: Heart of Texas

PostPosted: Sun 25 Jan 2015, 14:18    Post subject:  

http://fastcompression.blogspot.com/2011/09/lz4-hc-high-compression-lz4-version-is.html

A variant for distro developers ( very slow compression but savings an additional 20% ) and compatible
Back to top
View user's profile Send private message 
stemsee

Joined: 27 Jun 2013
Posts: 1255
Location: In The Way

PostPosted: Thu 07 May 2015, 21:42    Post subject:  

Intereting. I have been trying to compile and use it. My kernel 4.0.1 su[pposrts it and I can load the modules. I built the tools and I can compress files using 'lz4 arg source dest'. I built squashfs with lz4 support (I thought) but how to implement it? Help shows only gzip and xz. Any pointes appreciated.
lz4-r128-i686.pet
Description 
pet

 Download 
Filename  lz4-r128-i686.pet 
Filesize  154.74 KB 
Downloaded  33 Time(s) 
squashfs4.3-i686.pet
Description 
pet

 Download 
Filename  squashfs4.3-i686.pet 
Filesize  114.09 KB 
Downloaded  34 Time(s) 
Back to top
View user's profile Send private message MSN Messenger 
Display posts from previous:   Sort by:   
Page 1 of 1 [7 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0691s ][ Queries: 12 (0.0110s) ][ GZIP on ]