Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 14 Feb 2016, 06:27
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
lzop - most efficient compressor for big files
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [15 Posts]  
Author Message
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Mon 13 Apr 2015, 11:03    Post subject:  lzop - most efficient compressor for big files
Subject description: Now installable as a *.pet archive
 

(@Flash: please move this thread if it should go elsewhere. Thanks.)
~~~~~~~~~
Hello,

the subject came up in another thread, but I think the Puppy incarnation
should go here.

Compiled from source, stripped with
Code:
strip --strip-unneeded lzop

Pet file contains man1 and html docs at /usr/share/man and
/usr/share/doc respectively.
Executable in /bin replaces the busybox lzop provided in the Puppy.

The reason for this *.pet archive is that the busybox lzop destroys the
archived file automatically after the lzop archive is created.
The real lzop keeps the original file, and this is what we want when
backing-up pupsave files.

Recommendation: use with SFR's excellent PackIt / UExtract
archiving scripts.


Caveat: erase or rename the OOTB Puppy lzop link in /bin BEFORE
installing the attached *.pet archive.


Try to enjoy sadly. Shocked (Just playing with words...) Smile BFN.

musher0
lzop-1.03-i386.pet
Description  Erase or rename the Puppy lzop link in /bin
BEFORE installing this *.pet archive.
pet

 Download 
Filename  lzop-1.03-i386.pet 
Filesize  53.07 KB 
Downloaded  112 Time(s) 

_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."

Last edited by musher0 on Thu 16 Apr 2015, 17:47; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
SFR


Joined: 26 Oct 2011
Posts: 1400

PostPosted: Mon 13 Apr 2015, 11:55    Post subject:  

Hey Musher0

Just a tip: you can avoid the deletion while using busybox's lzop by enabling '-c' option (Write to stdout) and redirecting the output:
Code:
lzop -c somefile > somefile.lzo

I have used the above method in (recommended by you, thanks! Smile ) PackIt/UExtract, so input files are safe, whether you use busybox's or full lzop.

Greetings!

_________________
[O]bdurate [R]ules [D]estroy [E]nthusiastic [R]ebels => [C]reative [H]umans [A]lways [O]pen [S]ource
Omnia mea mecum porto.
Back to top
View user's profile Send private message 
gcmartin


Joined: 14 Oct 2005
Posts: 5966
Location: Earth

PostPosted: Mon 13 Apr 2015, 17:22    Post subject:  

I kinda see what is shared by Musher0 and alluded to by SFR. When this was originally done, we had 486s, 5-686s, with 64/128 RAM. The size difference versus the benefits (including "most efficient archiver for big files", others mentioned and etc.) is worth a look for today's PUPs.
_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engines or use DogPile
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Mon 13 Apr 2015, 21:51    Post subject:  

@SFR: Thanks for the information.

@all:
The capture below should make clear all the features that the "busybox
lzop" neglected to incorporate from the original lzop.

Don't get me going on busybox: in short, you can always trust busybox to
bring a genuine utility this close to being useless. (The notable exception
being ash.)

BFN.

musher0
lzop-compare_2015-04-13.jpg
 Description   
 Filesize   74.29 KB
 Viewed   274 Time(s)

lzop-compare_2015-04-13.jpg


_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Sun 03 Jan 2016, 09:31    Post subject:  

If you Frugal boot puppy pfix=ram,nocopy then a level 1 lzop compressed puppy, something like (assuming your version of mksquashfs supports lzop)

mksquashfs <puppydir> puppy_tarh_6.0.5.sfs -noappend -comp lzo -Xcompression-level 1

compresses relatively quickly, but not as tightly as other alternative compressors.

However the compression is still reasonable (perhaps halves the file size) and decompresses very quickly. Generally the time it takes to data transfer around half the filesize and decompress that is comparable to data transfering the full non compressed file from disk. i.e. no difference (generally) in times to use either the lzop compressed or non compressed.

If you use the nocopy boot option then parts of puppy are read from disk as and when required, but being relatively small once in memory often remain in memory (paged in) for the entire session. Booting without the nocopy option transfers the entire puppy into ram, but that includes many files (content) that might not be used often. gparted for instance. Booting with nocopy results in only the files you (system) uses - and boots a lot quicker.

The nocopy option however does mean that the HDD needs to remain mounted during a session.

Of the possible choices of frugal booting, I prefer storing puppy sfs on hdd and the concept of using pfix=ram,nocopy booting. I don't use a savefile/folder except when I create one to preserve changes/updates, that I then copy into puppy sfs (and delete the savefolder) i.e. extract (unsquashfs) the puppy, copy -rf the savefolder content into that extracted puppy, and then reform the puppy (mksquashfs). That way I have control over when puppy changes, and at all other times its a read only system (with data/docs (and portable firefox/chrome) etc all stored outside of puppy space).

lzop IMO is the #1 best choice for such frugal booted pups.

Last edited by rufwoof on Sun 03 Jan 2016, 14:28; edited 1 time in total
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Sun 03 Jan 2016, 12:12    Post subject:  

Thanks for sharing your experience notes, rufwoof.
Would it be the same booting from CD/DVD ?

BFN.

_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Sun 03 Jan 2016, 12:59    Post subject:  

I don't know (and I no longer have a working CD/DVD so I can't test it). I suspect that it might be quicker to read around half as much data off CD and decompress in ram (i.e. lzop'd) than read the full amount directly off CD. But it might be better in the case of a CD boot to simply copy the entire pup in one go (at bootup) assuming there's sufficient ram to do so. Otherwise its multiple CD search for file, data transfer ..etc per session (and the CD has to remain in its drive). Once paged in (in ram), re-reads might not occur on a PC with copious amounts of ram, but of course if a file that hasn't already been loaded is invoked then it would have to read that from the CD.

For a HDD based frugal with nocopy, booting is very quick (15 seconds on the 15 year old single core Celeron D that I'm currently using (timed from when initrd starts (after BIOS) to when the puppy gui desktop first shows). More so if you also comment out the Updating section in /etc/rc.d/rc.sysinit. And from my perpective subsequent running isn't noticeably slower than if the pup was fully transferred into ram at bootup.

I'd suggest perhaps using a full copy if CD frugal, nocopy for a HDD frugal. Giving grub/puppy helpful pointers as to locations of files also helps, such as including pmedia=idehd, psubdir, PDEV1. i.e. mine looks like :

title TAHR605-K4.3.2-NV
rootnoverify (hd0,2)
kernel /TAHR605-K4.3.2-NV/vmlinuz pfix=ram,nocopy pmedia=idehd PDEV1=sda3 psubdir=TAHR605-K4.3.2-NV
initrd /TAHR605-K4.3.2-NV/initrd.gz
boot
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Sun 03 Jan 2016, 14:27    Post subject:  

errrh!

Not so good. pfix=ram i.e. with copy works great. With pfix=ram,nocopy things start falling over. NTFS partition getting mounted as read only. After running a big mksquashfs having a desktop that when icons are clicked nothing happens, command prompt indicating gettext errors in xwin ... weird.

Easy fix. Revert to copying puppy sfs totally into ram at bootup and just tolerate the extra 15 second longer boot time (my pup is around 1.5GB non compressed, 700MB lzop -1 compressed - as it includes all firmware, modules, nvidia, Libre, Skype ...etc.).

I guess that's one of the issues with having such a flexible puppy system. Sometimes what works one way wont work another way. Fatdog (and Quirky) seek to address that by having a more focussed choice.

EDIT : DOH! (brain fart)

With the nocopy in effect I remastered to the live running version. When copied (just pfix=ram) that's not a issue because the new version is on disk and the live version is in ram. With pfix=ram,nocopy and remastering to a live sfs ... it (obviously) causes problems. !!!!

Last edited by rufwoof on Tue 05 Jan 2016, 07:27; edited 1 time in total
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Sun 03 Jan 2016, 14:53    Post subject:  

Thanks for the additional info, rufwoof. It's interesting, but I meant the compression part
through mksquashfs. Sorry for having been unclear in my question before.

I'm currently building a pup and would like to try the lzop compression on the sfs to
gain speed at bootup, as you mention.

What version of mksquashfs are you using that allows lzop compression? I looked at
my version, and there are only the gzip and xz choices for compression.

Thanks in advance.

_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Sun 03 Jan 2016, 19:23    Post subject:  

musher0 wrote:
Thanks for the additional info, rufwoof. It's interesting, but I meant the compression part
through mksquashfs. Sorry for having been unclear in my question before.

I'm currently building a pup and would like to try the lzop compression on the sfs to
gain speed at bootup, as you mention.

What version of mksquashfs are you using that allows lzop compression? I looked at
my version, and there are only the gzip and xz choices for compression.

Thanks in advance.

Its one I compiled myself. Specifically wanted lz4 compression support at the time, but used a source with both lz4 and lzop. I just named it mksquashfs5 and symlinked mksquashfs to that. Ditto for unsquashfs. Also compiled/grabbed stand alones of lz4 (can't recall from where) and lzop's (horses mouth : http://www.lzop.org/). IIRC the squashfs source was from here http://sourceforge.net/projects/squashfs/files/squashfs/squashfs4.3/.

That's why I also went for emsee's 4.3.2 kernel as the standard Tahr kernel is limited (can only boot/sfs load gz or xz sfs's).

Size limits posting a single sfs, attached are /usr and /lib separately. I think that's all the files/libs required ... and maybe more (don't overwrite the libs you already have)
lib.sfs.gz
Description  fake .gz
gz

 Download 
Filename  lib.sfs.gz 
Filesize  124 KB 
Downloaded  21 Time(s) 
usr.sfs.gz
Description  fake .gz
gz

 Download 
Filename  usr.sfs.gz 
Filesize  204 KB 
Downloaded  23 Time(s) 
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Sun 03 Jan 2016, 20:09    Post subject:  

Hi.

Thanks for the files.

You wrote:
> That's why I also went for emsee's 4.3.2 kernel as the standard Tahr
kernel is limited (can only boot/sfs load gz or xz sfs's).


Same for udf, as I discovered.

Ahah. The cat is coming out of the bag, is she? Smile That line sounds like a
bit much for my old head just now. Can you elaborate a bit?

It's beginning to sound like some great Linux contents editors in high
places are deciding what us regular mortal Linuxians can use and not use.

Thanks.

BFN.

_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Sun 03 Jan 2016, 20:24    Post subject:  

BTW, my remake of pemasu's UpupPrecise-3.8.3.1 is described
here and here.

_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Sun 03 Jan 2016, 21:15    Post subject:  

musher0 wrote:
You wrote:
> That's why I also went for emsee's 4.3.2 kernel as the standard Tahr
kernel is limited (can only boot/sfs load gz or xz sfs's).


Same for udf, as I discovered.

Ahah. The cat is coming out of the bag, is she? Smile That line sounds like a
bit much for my old head just now. Can you elaborate a bit?

Don't fully understand it myself. In Tahr standard with those mksquashfs and unsquashfs you can make/squash and extract/unsquash using lzo or lz4, however Tahr isn't able to view the content/install (sfs load) those sfs's i.e. the kernel was compiled with a different configuration than the configuration used to compile the 4.3.2 kernel (as made by Stemsee) that can view/install those sfs's - that I'm using with Puppy Tahr 6.0.5.

All to do with how the DOTConfig (kernel compile) was configured when the kernel was compiled. You can see each pup's kernel compile configuration (DOTConfig) in /etc/modules/DOTconfig-xxxx In mine (emsee 4.3.2) there is for instance some lines containing

CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y

i.e. all set to y (yes) for those choices. Other DOTConfig's (pups) might have some of those as m (modules) or n (no) or even commented out. I don't understand such configuation other than a very casual awareness.

To fully use lzo or lz4 ... or whatever, you need mksquashfs and unsquashfs that supports those choices of compression AND a kernel that's compiled to support those choices of compressed squashed file systems (SFS's). mksquashfs creates a filesystem that's compressed - but the operating system (kernel) can only read/load that filesystem if it has been built to do so.

At least that's my interpretation - but that could be wrong!
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 6863
Location: Gatineau (Qc), Canada

PostPosted: Mon 04 Jan 2016, 00:40    Post subject:  

Excellent explanation, rufwoof. It's much clearer in my mind now. Many thanks.
_________________
musher0
/ Torture, according to Dostoievsky: "To only have enough intelligence to know that you're lacking in it."
/ La torture, d'après Dostoïevski : "Être juste assez intelligent pour savoir qu'on ne l'est pas assez."
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 1250
Location: UK

PostPosted: Tue 05 Jan 2016, 07:44    Post subject:  

Having used frugal (to HDD) of Tahr 6.0.5 with emsee's 4.3.2 kernel and lz4 compressed initrd, puppy, zdrv and adrv sfs's (I store my 'savefolder' as a read only version in adrv), and booting pfix=ram,nocopy for a while now, its been running like lightning. Boots and runs exceptionally quick. Loading other sfs's created using lz4 compression and they load/run quick also.

Daft when a 15+ year old single core runs quicker than the latest Windows running on a modern multi-core. And having a read only based pup, configured as you like it, means that its more resilient.

The main factor for me is that whilst lzop -1 and lz4 high compression are somewhat comparable for filesize. lz4 has the edge on the decompression side, and its scalable - more cores - faster. Whilst lzop I believe is tied to a single core. Apparently multi-core lz4 can run at speeds approaching ram speeds, so if its 2:1 compressed data being decompressed that's somewhat like virtually doubling up the amount of ram.

Oddly, not reading all of puppy into ram at bootup (booting pfix=ram,nocopy) seems to generally run much quicker (entire session) than if puppy is read into ram at bootup (just pfix=ram alone). Perhaps only copying files into ram as/when needed means that only small parts of puppy are actually being loaded into ram as there's a lot in each puppy that more often doesn't actually get used each session (gparted for instance) - leaving more available for what actually is being run.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [15 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Filesystem
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0920s ][ Queries: 12 (0.0159s) ][ GZIP on ]