Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 23 May 2018, 03:09
All times are UTC - 4
 Forum index » House Training » HOWTO ( Solutions )
How to stuff 7% more material into your sfs's.
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 2 [30 Posts]   Goto page: Previous 1, 2
Author Message
rufwoof

Joined: 24 Feb 2014
Posts: 2323

PostPosted: Mon 27 Mar 2017, 16:09    Post subject: Re: How to stuff 7% more material into your sfs's.  

musher0 wrote:
a bit longer to create, but are perhaps a bit snappier to
unpack to RAM when you load your Puppy

LZ4 compresses less tightly (larger size), but decompresses much quicker than both gzip and xz. It's a trade off between disk IO speed and processing (decompression) and ram transfer speed. LZ4 when using multi-cores can throw out data at ram transfer speeds (extremely fast).

Reading more disk IO/decompress quickly can be quicker than less disk IO (tighter compression)/slower output (decompression).

Also bear in mind that once a program has been read in once during a session, most of that will remain memory bound (cached), such that subsequent re-runs of the same program/libs will already be in memory (cache). Especially in the case of puppy that might be say 500MB uncompressed in total, running on a PC that perhaps has 2GB or ram. Rather than loading the entire pup into ram at startup (slower bootup), just loading files as and when required will load less (gparted for instance might never even be used during a session). Slower to initially load a program (such as gparted if invoked during a session) than having it already available in memory, but no difference for subsequent re-runs of the program.

Generally not loading puppy into ram at startup, using lz4 compressed sfs's is the better overall choice IMO. Maximising xz compression will tend to see a slower system by comparison whilst IO is more predominant, but thereafter the two tend to compare anyway (once cached).

As a example I run Debian frugally with a save partition (rather than a save file/folder) and comparing having all of that inside a sfs to having all of the filesystem in the save partition (non compressed) sees very little difference between the two. For me, having all of the data in the save partition leaving a empty main sfs means that I can also boot that as though it was a full install. There is a noticeable difference between booting full versus frugal however as with frugal all changes are being recorded in memory, not back on disk (I personally like frugal as you can just reboot without saving to undo all changes made during the session i.e. I like to keep the main system pristine/factory-fresh - I only ever tend to boot as though full when I apply updates, and then immediately reboot back to frugal again afterwards).
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4998
Location: Republic of Novo Zelande

PostPosted: Tue 28 Mar 2017, 02:10    Post subject: Re: How to stuff 7% more material into your sfs's.  

rufwoof wrote:
LZ4 compresses less tightly (larger size), but decompresses much quicker than both gzip and xz.
Any chance you could suggest a good default syntax to build an sfs from a directory using LZ4? (and do you have any idea if LZ4 is understood by Slacko?)
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2351
Location: Cradle of Humankind

PostPosted: Mon 03 Apr 2017, 07:20    Post subject:  

Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Results compressing Racy base sfs:
gzip = 142MB
-comp xz (default setting) = 121MB
-comp xz -b 1048576 -Xdict-size 100% = 109MB
-comp xz -b 1048576 -Xbcj x86 = 105MB

Compression slowish, decompression time not affected. Good if you are only going to compress once or seldomly.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12192
Location: Gatineau (Qc), Canada

PostPosted: Mon 03 Apr 2017, 12:09    Post subject:  

nic007 wrote:
Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Results compressing Racy base sfs:
gzip = 142MB
-comp xz (default setting) = 121MB
-comp xz -b 1048576 -Xdict-size 100% = 109MB
-comp xz -b 1048576 -Xbcj x86 = 105MB

Compression slowish, decompression time not affected. Good if you are only going to compress once or seldomly.
Hi nic007.

Your memory came back, I see!!!

"Oh, sh?t." That's a 35% space savings over gz. Wow.

Bravo nic007.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4998
Location: Republic of Novo Zelande

PostPosted: Mon 03 Apr 2017, 12:52    Post subject:  

nic007 wrote:
Some more testing. Best xz compression achieved with: -comp xz -b 1048576 -Xbcj x86

Thanks for the parameters. Using this gave me the following comparison for my base sfs:

454MB using gz method

351MB using Mushero xz syntax

335MB using nic007 xz syntax

Unfortunately both of the xz methods create an issue for me when the system tries to run my firewall setup script in /root/Startup so I assume I must have a delay in unpacking something at a critical moment.

Not a big issue and probably something I can find a way around (I think it is due to the fact I use xdotool to generate a keyboard action during the firewall script. Bad practice really).

The other issue for me is that the xz methods both take 30 seconds longer than gz to boot my machine but then i am working with quite a large sfs so maybe not a problem for others.

The significant space saving does make the xz syntaxes useful for making best use of storage space.
Back to top
View user's profile Send private message 
rufwoof

Joined: 24 Feb 2014
Posts: 2323

PostPosted: Mon 03 Apr 2017, 15:26    Post subject: Re: How to stuff 7% more material into your sfs's.  

greengeek wrote:
rufwoof wrote:
LZ4 compresses less tightly (larger size), but decompresses much quicker than both gzip and xz.
Any chance you could suggest a good default syntax to build an sfs from a directory using LZ4? (and do you have any idea if LZ4 is understood by Slacko?)

I suspect its not in Slacko.

It has to be built into the kernel as well as having updated squashfs-tools ... in which mksquashfs will typically have just -comp lz4 (basic choice) or -comp lz4 -Xhc (higher compression choice (decompression speed isn't any slower, but in decompressing less due to higher compression will throughput quicker)). Compression ratio isn't good. Higher speed comes with less complexity (less compression).

mksquashfs --help will show if the squashfs-tools version you have supports lz4, but the kernel might not necessarily also support it.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4998
Location: Republic of Novo Zelande

PostPosted: Tue 04 Apr 2017, 04:34    Post subject: Re: How to stuff 7% more material into your sfs's.  

rufwoof wrote:
mksquashfs --help will show if the squashfs-tools version you have supports lz4, but the kernel might not necessarily also support it.
Thanks - looks like only gzip and xz available. cheers!
Back to top
View user's profile Send private message 
OvrMaHd


Joined: 10 Nov 2017
Posts: 14

PostPosted: Thu 29 Mar 2018, 14:44    Post subject:  

I would like to thank all of the contributors of this thread. Has helped tremendously, to creating functioning Make SfS, from a plethora of obscure options.

Ovr
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 885

PostPosted: Mon 02 Apr 2018, 00:11    Post subject:  

Hey people,

Anyone knows any way to know how a given sfs was made?

Any command or mksquashfs option to get details about the block size, xdict size, compression used, of a given sfs.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12192
Location: Gatineau (Qc), Canada

PostPosted: Mon 02 Apr 2018, 01:04    Post subject:  

Hi drunkjedi.

Good question. Why do you want to know? As long as it does the job...

If you try to open a sfs file with the real less in binary mode, perhaps you can see a
kind of header? (Please see attached capture.) Now, to interpret it...

You are probably thinking of something like the zipinfo utility for the zip compressor.
I searched a bit on the Web and I have not found anything like it for squash files.

IHTH
squashed-archive-bytes.jpg
 Description   
 Filesize   170.51 KB
 Viewed   158 Time(s)

squashed-archive-bytes.jpg


_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
hamoudoudou


Joined: 24 Jul 2014
Posts: 863
Location: rabat

PostPosted: Mon 02 Apr 2018, 01:34    Post subject: compression tools  

compression tools
To open ISOs squash files i use UEextract..
Just to remember.
User can choose low, fast, that will give different size in resulting files.
My Opinion :compression is useful for very big files, compression makes processors work and slow down install.
A Puppy ISO compressed xz in not lighter than compressed gz, the weight would be the same, compared to one kg of feathers and one kg of iron. For old computers with only one processor, that is important not to give it work to do. (inflate the air mattress)
packit.png
 Description   menu LXpup Xenial
 Filesize   37.6 KB
 Viewed   152 Time(s)

packit.png


Last edited by hamoudoudou on Mon 02 Apr 2018, 04:17; edited 1 time in total
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2351
Location: Cradle of Humankind

PostPosted: Mon 02 Apr 2018, 02:34    Post subject:  

xz compression will ALWAYS produce a smaller compressed file size than gzip compression, if not, you haver done something seriously wrong somewhere (unless you are trying to compress something that has already been compressed, like an ISO with already compressed content).
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 885

PostPosted: Mon 02 Apr 2018, 02:53    Post subject:  

musher0 wrote:
Why do you want to know? As long as it does the job...
Ohh it's just me being lazy.
I have too many sfs laying around. I don't know how they were compressed, or if there's any chance to further save space.
Quote:
I searched a bit on the Web and I have not found anything like it for squash files.
I did search the web before posting, didn't find any way.

Only easy way would be to make a script that will unsquash and then re-squash given bunch of SFS with those options.
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2351
Location: Cradle of Humankind

PostPosted: Mon 02 Apr 2018, 03:32    Post subject:  

drunkjedi wrote:
musher0 wrote:
Why do you want to know? As long as it does the job...
Ohh it's just me being lazy.
I have too many sfs laying around. I don't know how they were compressed, or if there's any chance to further save space.
Quote:
I searched a bit on the Web and I have not found anything like it for squash files.
I did search the web before posting, didn't find any way.

Only easy way would be to make a script that will unsquash and then re-squash given bunch of SFS with those options.

Right-clicking on the sfs and choosing properties from the menu will give you some description of the sfs. A block size of 1 byte (if possible) will generally indicate the highest compression. I've checked my sfs's which were compressed at maximum compression and they all report block sizes of either 1 or 2 bytes.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12192
Location: Gatineau (Qc), Canada

PostPosted: Mon 02 Apr 2018, 16:56    Post subject:  

@drunkjedi:

Looking for something else, I chanced on these methods
to get info about a squashed or sfs file.

The easy way is to open a console in the directory where your sfs file is, and type:
Code:
unsquash -s file.sfs
Another involves the use of dd, and another still uses
Code:
strings file.sfs | head
to get "some" info.

None of these methods seem able to uncover the additional parameters used to
obtain greater compression when squashing.

IHTH.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 2 [30 Posts]   Goto page: Previous 1, 2
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » HOWTO ( Solutions )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0749s ][ Queries: 13 (0.0140s) ][ GZIP on ]