Hello,
I apologize if this isn't the appropriate forum section for such a request. Please move it to a more appropriate section if this is the case. I'm just trying to get the best visibility I can for it among the regulars.
I work for a major manufacturer of computer hardware and we're seriously considering Puppy as a candidate for testing our flashdrive-embedded OS proof-of-concept project. Without going into too much initial detail, we're trying to determine the extent of chipset bottleneck in computer boot speeds.
In order to demonstrate our solution, we need to show the difference between booting an OS that is decompressed at boot time and booting an OS that is not decompressed at boot time (but is still in archival format). The two OS platforms must otherwise be identical.
Really this is just a long way of asking someone here who has a properly configured Puppy host-compiler environment to please make available a pup_201.sfs file that has been created with mksquashfs using the -noD -noF and -noI command line parameters. These parameters of course cause squashfs to create the .sfs file without any compression.
If someone here (and I realize this would most likely be Barry, since his host system is the source of the distro) could make available an uncompressed pup_201.sfs, we would be eternally grateful.
We're willing to attempt the feat ourselves, but we haven't managed to set up a proper host-compile environment and will probably need some guidance building one that would provide what we're looking for.
-Shawn
P.S. We're using Puppy in the office here more and more. We love it!
Uncompressed squashfs
-
Locodarwin
- Posts: 4
- Joined: Fri 07 Jul 2006, 15:59
Hi Locodarwin; Here's a complete set of tests with Ubuntu & DSL comparing most of the compressed fs.
http://kerneltrap.org/files/PERFORMANCE.README.txt
Basically it's what I'd conjectured for years... that newer CPUs decompress faster than the HD reads.
This means that with newer PCs, ALL the compression methods are faster than raw files on the HD.
AND... in looking into cloop, squashfs, & e2comp, it looks like Barry made a good choice in Squash.
http://kerneltrap.org/files/PERFORMANCE.README.txt
Basically it's what I'd conjectured for years... that newer CPUs decompress faster than the HD reads.
This means that with newer PCs, ALL the compression methods are faster than raw files on the HD.
AND... in looking into cloop, squashfs, & e2comp, it looks like Barry made a good choice in Squash.
-
Locodarwin
- Posts: 4
- Joined: Fri 07 Jul 2006, 15:59
Thanks for your reply, sunburnt. I apologize for missing it earlier.
The hard drive read speed is indeed the bottleneck in such an application. Modern CPUs have no problem beating the pants off the hard drive when it comes to compressed vs. uncompressed data. You can demonstrate this easily by ghosting a system with Norton Ghost, first creating a "highly" compressed image and then an uncompressed image of the same partition or disk. The highly compressed image will re-ghost leaps and bounds quicker.
But what if that HD bottleneck is removed?
Our tests set the preliminary stage for eventually showing the performance/boot-time difference between systems when how fast it takes to get the information into memory isn't a factor. The goal is to create systems that, after post, boot up to the desktop at the fastest possible speeds.
Near-future media drives will no longer be a bottleneck - hence our decision to look for other boot duration solutions.
Puppy decompresses rapidly no matter what, as the squashfs file isn't very large. That would suggest it might not be the perfect candidate for illustrating our point. However, one cannot simply zip up, say, the entire Windows XP OS and then see how long it takes to boot.
And systems like Ubuntu aren't as easy as Puppy to design this test around, either.
Puppy's decompression speed, while barely interesting to us humans, is perfectly significant for our testing, because in the lab our two flavors of Puppy (now 2.01 compressed .sfs and uncompressed .sfs, as well as a gunzipped init.rd) load off of our proprietary test drives within 0.14 seconds of each other. That's awfully close. It is actually possible, therefore, to measure the (unrelated) decompression time between the two flavors - while simultaneously suppressing unwanted ulterior factors - using just a simple bus signal trace.
The difference is significant, at least on the scale we're considering.
As the amount of data that needs to be decompressed increases, though, the CPU speed then becomes the determining factor of bootup duration. What can we do to diminish the significance of processor speed?
Our question then became...is it possible/practical to offload the work of decompression to the system chipset as it is being streamed to memory from the drive, thereby eliminating the need for the processor to decompress the data during or after the fact?
The conclusion we've drawn is yes, it will be possible and practical, given as-yet-unreleased advances in firmware and flash memory, to decompress data into system memory prior to the involvement of the processor - using an algorithm on a virtual machine inserted into the North bridge portion of the chipset. Basically, we pre-process Puppy's entire compressed OS on the fly as it loads, leaving the processor free to start chewing on the parts as they stream into memory.
We can simulate it using FPGAs at this point. We'll be looking into a hardware demo soon.
It might not sound like much of a time-saver at this point, and in truth it isn't - with current hardware. When you consider the size of Vista and other trends towards gigantic, all-inclusive operating systems, it's not hard to see how important boot-time compression will become, and how significant pre-processed decompression will also therefore become.
I can't speak of further particulars on this forum, as I'm contractually bound by non-disclosure agreement. Just be assured that Puppy is helping us make the case, and in the process is getting some good internal exposure with a pretty significant client base.
We're grateful to Puppy's creator and community for their efforts on a fine, worthy operating system. Puppy has helped to justify my existence at my workplace. How's that for a user testimonial?
-S
The hard drive read speed is indeed the bottleneck in such an application. Modern CPUs have no problem beating the pants off the hard drive when it comes to compressed vs. uncompressed data. You can demonstrate this easily by ghosting a system with Norton Ghost, first creating a "highly" compressed image and then an uncompressed image of the same partition or disk. The highly compressed image will re-ghost leaps and bounds quicker.
But what if that HD bottleneck is removed?
Our tests set the preliminary stage for eventually showing the performance/boot-time difference between systems when how fast it takes to get the information into memory isn't a factor. The goal is to create systems that, after post, boot up to the desktop at the fastest possible speeds.
Near-future media drives will no longer be a bottleneck - hence our decision to look for other boot duration solutions.
Puppy decompresses rapidly no matter what, as the squashfs file isn't very large. That would suggest it might not be the perfect candidate for illustrating our point. However, one cannot simply zip up, say, the entire Windows XP OS and then see how long it takes to boot.
And systems like Ubuntu aren't as easy as Puppy to design this test around, either.Puppy's decompression speed, while barely interesting to us humans, is perfectly significant for our testing, because in the lab our two flavors of Puppy (now 2.01 compressed .sfs and uncompressed .sfs, as well as a gunzipped init.rd) load off of our proprietary test drives within 0.14 seconds of each other. That's awfully close. It is actually possible, therefore, to measure the (unrelated) decompression time between the two flavors - while simultaneously suppressing unwanted ulterior factors - using just a simple bus signal trace.
The difference is significant, at least on the scale we're considering.
As the amount of data that needs to be decompressed increases, though, the CPU speed then becomes the determining factor of bootup duration. What can we do to diminish the significance of processor speed?
Our question then became...is it possible/practical to offload the work of decompression to the system chipset as it is being streamed to memory from the drive, thereby eliminating the need for the processor to decompress the data during or after the fact?
The conclusion we've drawn is yes, it will be possible and practical, given as-yet-unreleased advances in firmware and flash memory, to decompress data into system memory prior to the involvement of the processor - using an algorithm on a virtual machine inserted into the North bridge portion of the chipset. Basically, we pre-process Puppy's entire compressed OS on the fly as it loads, leaving the processor free to start chewing on the parts as they stream into memory.
We can simulate it using FPGAs at this point. We'll be looking into a hardware demo soon.
It might not sound like much of a time-saver at this point, and in truth it isn't - with current hardware. When you consider the size of Vista and other trends towards gigantic, all-inclusive operating systems, it's not hard to see how important boot-time compression will become, and how significant pre-processed decompression will also therefore become.
I can't speak of further particulars on this forum, as I'm contractually bound by non-disclosure agreement. Just be assured that Puppy is helping us make the case, and in the process is getting some good internal exposure with a pretty significant client base.
We're grateful to Puppy's creator and community for their efforts on a fine, worthy operating system. Puppy has helped to justify my existence at my workplace. How's that for a user testimonial?
-S
Locodarwin; Just what I suspected for the HD tests, flash would show less difference (acess times).
Running Puppy in memory isn't going to run any faster compressed or not.
BUT the apps. in a squash file taking up 2.5 times less space in memory still makes it well worth while.
Especially considering that the Puppy boot portion (vmlinuz & initrd.gz) is < 4 MB (initrd.gz decompressed).
The squash file is 67 MB & expands to 175 MB, saving 113 MB of memory, WOW!
Using the MB chipset to do the decompress of the squash file is innovative!
I'd always said: What if the compression was a built in native instruction in the cpu?
Then ALL data storage & communication could be compressed FAST, memory included.
Puppy's squash file loaded into memory is a current method of memory compression.
But the compressor is still a kernel module, & so software, so it's slower.
Imagine the speed if the cpu or another piece of hardware did the work!
Running Puppy in memory isn't going to run any faster compressed or not.
BUT the apps. in a squash file taking up 2.5 times less space in memory still makes it well worth while.
Especially considering that the Puppy boot portion (vmlinuz & initrd.gz) is < 4 MB (initrd.gz decompressed).
The squash file is 67 MB & expands to 175 MB, saving 113 MB of memory, WOW!
Using the MB chipset to do the decompress of the squash file is innovative!
I'd always said: What if the compression was a built in native instruction in the cpu?
Then ALL data storage & communication could be compressed FAST, memory included.
Puppy's squash file loaded into memory is a current method of memory compression.
But the compressor is still a kernel module, & so software, so it's slower.
Imagine the speed if the cpu or another piece of hardware did the work!
Locodarwin,Locodarwin wrote: Puppy decompresses rapidly no matter what, as the squashfs file isn't very large. That would suggest it might not be the perfect candidate for illustrating our point. However, one cannot simply zip up, say, the entire Windows XP OS and then see how long it takes to boot.
-S
I am far from an expert on this topic, in fact I can only follow about half of what you said. When reading this paragraph I quoted however, I thought well why doesn't he just use Puppy unleashed to make the most bloated Puppy ever. Then use the same commands to create an uncompressed version. Puppy Should be quite large if every single package were installed. That might make it easier to demonstrate your point, to whoever your audience is.
CAn8v