Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 21 Jun 2018, 15:51
All times are UTC - 4
 Forum index » House Training » HOWTO ( Solutions )
How to stuff 7% more material into your sfs's.
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [30 Posts]   Goto page: 1, 2 Next
Author Message
musher0


Joined: 04 Jan 2009
Posts: 12321
Location: Gatineau (Qc), Canada

PostPosted: Sun 26 Mar 2017, 07:34    Post subject:  How to stuff 7% more material into your sfs's.  

Hello all.

This mksquashfs trick is not well know, but it should, IMO.

By comparison with the usual line,
Code:
mksquashfs Directory/ Directory.sfs -noappend -all-root -comp xz
we compress about 7% more if we use this one:
Code:
mksquashfs Directory/ Directory.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%
# Source:
https://techoverflow.net/blog/2013/01/11/create-highly-compressed-squashfs-from-folder

I tried it on the various sfs's of the DpupStretch7. For ex., the main puppy
sfs went down from 190 Mb to 176 Mb by compressing with the new parms.

AFAICT, they take a bit longer to create, but are perhaps a bit snappier to
unpack to RAM when you load your Puppy.

BFN.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2406
Location: Cradle of Humankind

PostPosted: Sun 26 Mar 2017, 08:11    Post subject:  

Is that the maximum compression? Takes longer to compress but does seem a bit quicker to uncompress. Suppose it may be wothwhile if you are only going to compress once. I got my Racy base sfs down to 109MB instead of 116MB (I think this was with another parameter I used before). Normal xz compression is 121MB, so about a ten percent difference. Not bad
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12321
Location: Gatineau (Qc), Canada

PostPosted: Sun 26 Mar 2017, 09:02    Post subject:  

nic007 wrote:
Is that the maximum compression? (...)

I don't know! Worth investigating, certainly.
BFN.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2406
Location: Cradle of Humankind

PostPosted: Sun 26 Mar 2017, 10:16    Post subject:  

musher0 wrote:
nic007 wrote:
Is that the maximum compression? (...)

I don't know! Worth investigating, certainly.
BFN.


It's actually not the best. Somehow I got the compression of my tahr base sfs down to 174MB (sometime ago, can't remember what I did). Your method makes it 179MB.
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 897

PostPosted: Sun 26 Mar 2017, 11:10    Post subject:  

Do xz compressed sfs work on all pups? Older pups?
Just asking....
I generally don't use any options with mksquashfs.
But my sfs are not distributed widely, mostly my personal use.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12321
Location: Gatineau (Qc), Canada

PostPosted: Sun 26 Mar 2017, 12:57    Post subject:  

drunkjedi wrote:
Do xz compressed sfs work on all pups? Older pups?
Just asking....
I generally don't use any options with mksquashfs.
But my sfs are not distributed widely, mostly my personal use.

Only with pups that have xz, I suppose.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2406
Location: Cradle of Humankind

PostPosted: Sun 26 Mar 2017, 14:44    Post subject:  

All puppys with kernel 3 and upwards. Don't know if it's possible to install those modules in older puppys but it definitely does not work by default for older puppys.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5054
Location: Republic of Novo Zelande

PostPosted: Mon 27 Mar 2017, 02:10    Post subject:  

Thanks musher0 - won't get a chance to try this just yet but well worth knowing about. I use manual mksquashfs frequently. If there is a space saving to be had I am keen to learn it.

cheers!
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5054
Location: Republic of Novo Zelande

PostPosted: Mon 27 Mar 2017, 05:28    Post subject:  

Well, i decided i just had to try this so here is some comparison information:

I normally use the following command to compress my personal sfs directory:
Code:
mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs
and I compared the performance of the resulting sfs against the output of the following code using your new syntax:
Code:
mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%

I have attached a pic comparing the size comparison of the two methods. Your method gives a size of 351MB where my normal method gives a size of 454MB.

That is a huge saving.

The question for me is how does that affect boot times? Is it better or worse? The comparison for me is that my normal method boots (from end of bootloader to functional desktop) in 1 minute exactly. Your version boots in 1 minute 21 seconds. Hmmm - I was hoping for a miracle but they are hard to find. At least there is some advantage for me in using your method - I can fit my special puppy on a smaller storage device.

However, maybe there is some variation of your method that might work well for me to reduce my boot times?

My results will not be typical - these tests are done with my non-standard derivative of Slacko 5.6 where I have reversed the normal puppy sfs and the zdrv sfs (which is my personal sfs).

Others may have different results with a standard pup.

I am quite excited by the prospect of shrinking both size and boot times. Maybe Slacko lacks the utilities to make the best of this method. I will have to do more experimentation.
sfsbuild_musher_versus_standard.jpg
 Description   
 Filesize   98.69 KB
 Viewed   376 Time(s)

sfsbuild_musher_versus_standard.jpg

Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 2406
Location: Cradle of Humankind

PostPosted: Mon 27 Mar 2017, 06:08    Post subject:  

greengeek wrote:
Well, i decided i just had to try this so here is some comparison information:

I normally use the following command to compress my personal sfs directory:
Code:
mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs
and I compared the performance of the resulting sfs against the output of the following code using your new syntax:
Code:
mksquashfs squashfs-root/. puppy_b3libre_banksy3.sfs -noappend -all-root -b 1048576 -comp xz -Xdict-size 100%

I have attached a pic comparing the size comparison of the two methods. Your method gives a size of 351MB where my normal method gives a size of 454MB.

That is a huge saving.

The question for me is how does that affect boot times? Is it better or worse? The comparison for me is that my normal method boots (from end of bootloader to functional desktop) in 1 minute exactly. Your version boots in 1 minute 21 seconds. Hmmm - I was hoping for a miracle but they are hard to find. At least there is some advantage for me in using your method - I can fit my special puppy on a smaller storage device.

However, maybe there is some variation of your method that might work well for me to reduce my boot times?

My results will not be typical - these tests are done with my non-standard derivative of Slacko 5.6 where I have reversed the normal puppy sfs and the zdrv sfs (which is my personal sfs).

Others may have different results with a standard pup.

I am quite excited by the prospect of shrinking both size and boot times. Maybe Slacko lacks the utilities to make the best of this method. I will have to do more experimentation.

Your first compression method is gzip, second is xz, so completely different compression tools. This thread is about using the best xz compression setting. As for zip vs xz. Zip is faster (for compression not so much for uncompression) but results in a much bigger file as you have noticed. If you use the xz default setting, compression will be faster than the xz compression with the extra parameters resulting in a bigger file(10% at most) but will still result in a massive savings from gzip. compression.
Back to top
View user's profile Send private message 
perdido


Joined: 09 Dec 2013
Posts: 799
Location: ¿Altair IV , Just north of Eeyore Junction.?

PostPosted: Mon 27 Mar 2017, 09:42    Post subject:  

Would the block size used (in the OP's command line it is 1MB) be affected by kernel limitations imposed by physical memory?

In other words, if I have a computer with 768MB of RAM is that going to limit my maximum block size request when using mksquashfs
to something under 768MB since the kmalloc maximum size is limited to physical memory?

I don't think kmalloc limitations imposed by physical memory can be increased using a swap file, or can it?

EDIT LOL - Just reread my post, please disregard gotta go drink some coffee!

.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12321
Location: Gatineau (Qc), Canada

PostPosted: Mon 27 Mar 2017, 10:23    Post subject:  

Thanks for the methodical test, greengeek.
_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
8Geee


Joined: 12 May 2008
Posts: 1570
Location: N.E. USA

PostPosted: Mon 27 Mar 2017, 13:15    Post subject:  

This is rather interesting stuff.
My questions:

1.) What do those parameter deltas actually do?
if something is not root (is that by priviledge or location?)... what happens?


2.) Is block-size dependant upon file-size? I notice my 152Mb sfs shows a block-size of 9 bytes.

Regards
8Geee

_________________
Linux user #498913

Some people need to reimagine their thinking.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5054
Location: Republic of Novo Zelande

PostPosted: Mon 27 Mar 2017, 14:11    Post subject:  

nic007 wrote:
Your first compression method is gzip, second is xz, so completely different compression tools.
Thanks for the heads up. I haven't followed the increasing usage of xz so this is new to me.
Quote:
This thread is about using the best xz compression setting.
Well maybe not only xz? The title potentially suggests any method for stuffing more data into a smaller sfs.
Quote:
If you use the xz default setting, compression will be faster than the xz compression with the extra parameters resulting in a bigger file(10% at most) but will still result in a massive savings from gzip. compression.
Thanks - i will try to do some testing with the default xz syntax.
Back to top
View user's profile Send private message 
musher0


Joined: 04 Jan 2009
Posts: 12321
Location: Gatineau (Qc), Canada

PostPosted: Mon 27 Mar 2017, 15:14    Post subject:  

8Geee wrote:
This is rather interesting stuff.
My questions:

1.) What do those parameter deltas actually do?
if something is not root (is that by priviledge or location?)... what happens?


2.) Is block-size dependant upon file-size? I notice my 152Mb sfs shows a block-size of 9 bytes.

Regards
8Geee
Hi, 8Geee.

Good to hear from you.

I do not have the answers to your questions. I must confess I am only a
parrot repeating some info, not a program analyst! Wink

I am sure someone knowledgeable on this board can provide answers.

BFN.

_________________
musher0
~~~~~~~~~~
"Logical entities must not be multiplied beyond necessity." | |
« Il ne faut pas multiplier les entités logiques sans nécessité. » (Ockham)
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [30 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » HOWTO ( Solutions )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0778s ][ Queries: 12 (0.0172s) ][ GZIP on ]