Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 14 Nov 2019, 23:51
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
I wonder why SaveFiles are NOT compress
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [14 Posts]  
Author Message
mikeslr


Joined: 16 Jun 2008
Posts: 3473
Location: 500 seconds from Sol

PostPosted: Fri 19 Apr 2019, 23:20    Post subject:  I wonder why SaveFiles are NOT compress  

See this post: http://www.murga-linux.com/puppy/viewtopic.php?p=1025779#1025779

Various compression algorithms (e.g. xz, gz) can be used to compress Application SFSes and IIRC even the Puppy_version_#.sfs. Despite that, Puppies have no problem reading their contents and copying such contents into RAM.

Is there a reason why the application which creates a SaveFile doesn't employ any compression algorithm?
Back to top
View user's profile Send private message 
nic007


Joined: 13 Nov 2011
Posts: 3120
Location: Cradle of Humankind

PostPosted: Sat 20 Apr 2019, 00:22    Post subject:  

The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.
Back to top
View user's profile Send private message 
6502coder


Joined: 23 Mar 2009
Posts: 648
Location: Western United States

PostPosted: Sat 20 Apr 2019, 00:31    Post subject:  

Right. It would only be feasible if it were possible to directly read and write the file system contained inside the compressed savefile. But I don't believe that is possible. The savefile would have to uncompressed for the contents to be accessed. So besides the overhead of the uncompression/recompression you'd have the extra disk or RAM space consumed by the uncompressed copy of the savefile.
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2144

PostPosted: Sat 20 Apr 2019, 00:32    Post subject:  

nic007 wrote:
The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.


One might be able to pull it off with sfs files using the multi-session cd/dvd approach. Only save changes as sfs files during shutdown. Maybe combine sfs files in the background while the system is running.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2144

PostPosted: Sat 20 Apr 2019, 00:33    Post subject:  

6502coder wrote:
Right. It would only be feasible if it were possible to directly read and write the file system contained inside the compressed savefile. But I don't believe that is possible. The savefile would have to uncompressed for the contents to be accessed. So besides the overhead of the uncompression/recompression you'd have the extra disk or RAM space consumed by the uncompressed copy of the savefile.


Mounting sfs files seems to be pretty fast. Perhaps the general principle is the files can be compressed but not the file-system.....or maybe for large files separately compress chunks of the file. Then one only needs to uncompress the files (and/or chunks) as needed.
Back to top
View user's profile Send private message Visit poster's website 
nic007


Joined: 13 Nov 2011
Posts: 3120
Location: Cradle of Humankind

PostPosted: Sat 20 Apr 2019, 01:05    Post subject:  

s243a wrote:
nic007 wrote:
The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.


One might be able to pull it off with sfs files using the multi-session cd/dvd approach. Only save changes as sfs files during shutdown. Maybe combine sfs files in the background while the system is running.

Not only limited tothe multi-session method. I use an sfs instead of savefile updating whenever I like and only save small configuration changes (my updating sfs is about 2MB and takes 3 seconds to update any system changes). But the thing is like I said before, most people work with huge savefiles which will make any uncompressing/re-compressing and updating impractical.

Bottom-line: whether you use a savefile or sfs for updating instead, keeping it small will be practical for decompressing/compressing purposes otherwise not.
Back to top
View user's profile Send private message 
6502coder


Joined: 23 Mar 2009
Posts: 648
Location: Western United States

PostPosted: Sat 20 Apr 2019, 01:26    Post subject:  

My understanding is that read/writes to a savefile are direct-access operations; if you modify a file, only the parts that were modified get rewritten in the savefile, not the entire file. If you try to throw compression on top of that, I don't think it still works--isn't compression an operation that intrinsically works on whole files? By that I mean, I don't think it's possible to separately and independently compress parts of a single file. Maybe by breaking the file into "chunks" as you say, but that would be essentially the same as splitting the original file into many small files. But compression schemes are more efficient on large files and less efficient on small files.
Back to top
View user's profile Send private message 
Mike Walsh


Joined: 28 Jun 2014
Posts: 5547
Location: King's Lynn, UK.

PostPosted: Sat 20 Apr 2019, 11:08    Post subject:  

Forgive the inquiries of a 'noob' in this area. But isn't the save-file 'uncompressed' all the time anyway during the life of a session.....only re-compressing at the end of that session?

That's how I understand it to work. Correct me if I'm wrong, please; always willing to learn more about this kinda stuff.


Mike. Wink

_________________
MY 'PUPPY' PACKAGES


Last edited by Mike Walsh on Sat 20 Apr 2019, 15:21; edited 1 time in total
Back to top
View user's profile Send private message 
jafadmin

Joined: 19 Mar 2009
Posts: 976

PostPosted: Sat 20 Apr 2019, 11:45    Post subject:  

It helps to understand how compression really works. It is impossible to compress a word. It is virtually useless to try to compress a sentence.

Compression is an algorithm applied to streams. If anything within the stream changes, the entire stream must be re-compressed. Otherwise it will not de-compress properly.

So in other words, if you change one word within a file, the entire file must be re-compressed if you are using compression on a file level.

If you are using compression on a partition level, changing a single word in a single file requires you to re-compress the entire partition.
Back to top
View user's profile Send private message 
williams2

Joined: 14 Dec 2018
Posts: 190

PostPosted: Sat 20 Apr 2019, 12:12    Post subject:  

File systems can be compressed. As far a I know, Puppy savefiles have never been compressed.

1. I don't think there is a practical, reliable compressed file system available at this time. There is an ext3 compressed file system at https://sourceforge.net/projects/e3compr/ but there have been no updates since 2008.

2. A compressed file system would probably be slower, especially on older hardware. Compressing is usually slower than uncompressing.

dos 6.2 used to have the ability to compress drives. Puppy save files are not like dos compressed drives. I think it is very important to make sure that people know save files are not compressed, because some people think of dos compressed drives when they think of save files, and sometimes that is why they chose a legacy full install.

As I have said before, a Puppy save file is for all practical purposes byte for byte identical to a partition that is formatted to the same file system.
Back to top
View user's profile Send private message 
williams2

Joined: 14 Dec 2018
Posts: 190

PostPosted: Sat 20 Apr 2019, 12:30    Post subject:  

Another misconception that some people may have, is that they think that the contents of a file system are copied somewhere when a file system is mounted.

Whether it is a partition or an SFS file or a save file, if it is mounted to some dir, the files in that file system are now available, one by one. The files in the file system are not copied en masse to ram, or anywhere else.

Executing a file copies that file into ram, of course. But mounting a file system does not automatically copy any of the files in the mounted file system anywhere.

If the mounted file system is in an SFS file, and that SFS file was copied to a tmpfs file system in ram, then the files in that SFS file system are all in ram, and accessing them is very fast. Uncompressing each file is slower than an uncompressed file system would be, but on modern hardware it would be negligible. The second time a file is accessed it should probably be in a ram cache anyway, uncompressed.
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2144

PostPosted: Sat 20 Apr 2019, 12:53    Post subject:  

jafadmin wrote:
It helps to understand how compression really works. It is impossible to compress a word. It is virtually useless to try to compress a sentence.

Compression is an algorithm applied to streams. If anything within the stream changes, the entire stream must be re-compressed. Otherwise it will not de-compress properly.

So in other words, if you change one word within a file, the entire file must be re-compressed if you are using compression on a file level.

If you are using compression on a partition level, changing a single word in a single file requires you to re-compress the entire partition.


One alternative to compressing a file system is to use a sparce file to represent the file system.
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3617

PostPosted: Sat 20 Apr 2019, 13:31    Post subject:  

Thanks. Never used a sparse file myself, had to look it up.
https://wiki.archlinux.org/index.php/Sparse_file

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3617

PostPosted: Sat 20 Apr 2019, 13:44    Post subject:  

What I've done in FatDog8, frugal install, is to have the save folder preserved as a compressed tar file (.tgz). In the init (boot process) I've added code to extract that (used tar as unsquashfs wasn't in initrd), which creates/populates the fd64save folder (save folder). Similarly when I click the desktop save icon I've added code to create a new .tgz of that savefolder.

In addition to that I also preserve the md5 checksum of the .tgz file after its created, and use that at startup to ensure the .tgz matches that md5 checksum, so if its been tampered with it gets flagged (intrusion detection). I also check the mbr, grldr, vmlinuz, fd64.sfs (main sfs) as part of bootup to ensure none of those have been tampered with.

Presently my fd64save.tgz is just 9MB, so the tar xvzf fd64save.tgz extraction (and tar cvzf fd64save.tgz fd64save ... creation) flashes through in a instant. Obviously for larger save folder sizes it would take longer.

To me that is as good as if booted using a CD/DVD i.e. the integrity is relatively certain. Could be more so if the checksums were stored on a USB that was only inserted for saving/booting purposes.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [14 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0862s ][ Queries: 11 (0.0099s) ][ GZIP on ]