I wonder why SaveFiles are NOT compress

[mikeslr]

See this post: http://www.murga-linux.com/puppy/viewto ... 79#1025779

Various compression algorithms (e.g. xz, gz) can be used to compress Application SFSes and IIRC even the Puppy_version_#.sfs. Despite that, Puppies have no problem reading their contents and copying such contents into RAM.

Is there a reason why the application which creates a SaveFile doesn't employ any compression algorithm?

[nic007]

The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.

[6502coder]

Right. It would only be feasible if it were possible to directly read and write the file system contained inside the compressed savefile. But I don't believe that is possible. The savefile would have to uncompressed for the contents to be accessed. So besides the overhead of the uncompression/recompression you'd have the extra disk or RAM space consumed by the uncompressed copy of the savefile.

[s243a]

The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.

One might be able to pull it off with sfs files using the multi-session cd/dvd approach. Only save changes as sfs files during shutdown. Maybe combine sfs files in the background while the system is running.

[s243a]

Right. It would only be feasible if it were possible to directly read and write the file system contained inside the compressed savefile. But I don't believe that is possible. The savefile would have to uncompressed for the contents to be accessed. So besides the overhead of the uncompression/recompression you'd have the extra disk or RAM space consumed by the uncompressed copy of the savefile.

Mounting sfs files seems to be pretty fast. Perhaps the general principle is the files can be compressed but not the file-system.....or maybe for large files separately compress chunks of the file. Then one only needs to uncompress the files (and/or chunks) as needed.

[nic007]

The savefile needs to be read/write enabled whenever updates to the system are made. The way most people seem to use a savefile (huge with everything installed) will make it impractical to uncompress and compress everytime when accessed for updating. In real-time mode uncompressing and compressing won't work.

One might be able to pull it off with sfs files using the multi-session cd/dvd approach. Only save changes as sfs files during shutdown. Maybe combine sfs files in the background while the system is running.

Not only limited tothe multi-session method. I use an sfs instead of savefile updating whenever I like and only save small configuration changes (my updating sfs is about 2MB and takes 3 seconds to update any system changes). But the thing is like I said before, most people work with huge savefiles which will make any uncompressing/re-compressing and updating impractical.

Bottom-line: whether you use a savefile or sfs for updating instead, keeping it small will be practical for decompressing/compressing purposes otherwise not.

[6502coder]

My understanding is that read/writes to a savefile are direct-access operations; if you modify a file, only the parts that were modified get rewritten in the savefile, not the entire file. If you try to throw compression on top of that, I don't think it still works--isn't compression an operation that intrinsically works on whole files? By that I mean, I don't think it's possible to separately and independently compress parts of a single file. Maybe by breaking the file into "chunks" as you say, but that would be essentially the same as splitting the original file into many small files. But compression schemes are more efficient on large files and less efficient on small files.

[Mike Walsh]

Forgive the inquiries of a 'noob' in this area. But isn't the save-file 'uncompressed' all the time anyway during the life of a session.....only re-compressing at the end of that session?

That's how I understand it to work. Correct me if I'm wrong, please; always willing to learn more about this kinda stuff.


Mike.

[jafadmin]

It helps to understand how compression really works. It is impossible to compress a word. It is virtually useless to try to compress a sentence.

Compression is an algorithm applied to streams. If anything within the stream changes, the entire stream must be re-compressed. Otherwise it will not de-compress properly.

So in other words, if you change one word within a file, the entire file must be re-compressed if you are using compression on a file level.

If you are using compression on a partition level, changing a single word in a single file requires you to re-compress the entire partition.

[williams2]

File systems can be compressed. As far a I know, Puppy savefiles have never been compressed.

1. I don't think there is a practical, reliable compressed file system available at this time. There is an ext3 compressed file system at https://sourceforge.net/projects/e3compr/ but there have been no updates since 2008.

2. A compressed file system would probably be slower, especially on older hardware. Compressing is usually slower than uncompressing.

dos 6.2 used to have the ability to compress drives. Puppy save files are not like dos compressed drives. I think it is very important to make sure that people know save files are not compressed, because some people think of dos compressed drives when they think of save files, and sometimes that is why they chose a legacy full install.

As I have said before, a Puppy save file is for all practical purposes byte for byte identical to a partition that is formatted to the same file system.

[williams2]

Another misconception that some people may have, is that they think that the contents of a file system are copied somewhere when a file system is mounted.

Whether it is a partition or an SFS file or a save file, if it is mounted to some dir, the files in that file system are now available, one by one. The files in the file system are not copied en masse to ram, or anywhere else.

Executing a file copies that file into ram, of course. But mounting a file system does not automatically copy any of the files in the mounted file system anywhere.

If the mounted file system is in an SFS file, and that SFS file was copied to a tmpfs file system in ram, then the files in that SFS file system are all in ram, and accessing them is very fast. Uncompressing each file is slower than an uncompressed file system would be, but on modern hardware it would be negligible. The second time a file is accessed it should probably be in a ram cache anyway, uncompressed.

[s243a]

It helps to understand how compression really works. It is impossible to compress a word. It is virtually useless to try to compress a sentence.

Compression is an algorithm applied to streams. If anything within the stream changes, the entire stream must be re-compressed. Otherwise it will not de-compress properly.

So in other words, if you change one word within a file, the entire file must be re-compressed if you are using compression on a file level.

If you are using compression on a partition level, changing a single word in a single file requires you to re-compress the entire partition.

One alternative to compressing a file system is to use a sparce file to represent the file system.

[rufwoof]

Thanks. Never used a sparse file myself, had to look it up.
https://wiki.archlinux.org/index.php/Sparse_file

[rufwoof]

What I've done in FatDog8, frugal install, is to have the save folder preserved as a compressed tar file (.tgz). In the init (boot process) I've added code to extract that (used tar as unsquashfs wasn't in initrd), which creates/populates the fd64save folder (save folder). Similarly when I click the desktop save icon I've added code to create a new .tgz of that savefolder.

In addition to that I also preserve the md5 checksum of the .tgz file after its created, and use that at startup to ensure the .tgz matches that md5 checksum, so if its been tampered with it gets flagged (intrusion detection). I also check the mbr, grldr, vmlinuz, fd64.sfs (main sfs) as part of bootup to ensure none of those have been tampered with.

Presently my fd64save.tgz is just 9MB, so the tar xvzf fd64save.tgz extraction (and tar cvzf fd64save.tgz fd64save ... creation) flashes through in a instant. Obviously for larger save folder sizes it would take longer.

To me that is as good as if booted using a CD/DVD i.e. the integrity is relatively certain. Could be more so if the checksums were stored on a USB that was only inserted for saving/booting purposes.