Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 25 Jul 2014, 23:16
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Uncompressed squashfs
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [8 Posts]  
Author Message
Locodarwin

Joined: 07 Jul 2006
Posts: 4

PostPosted: Fri 07 Jul 2006, 12:44    Post subject:  Uncompressed squashfs  

Hello,

I apologize if this isn't the appropriate forum section for such a request. Please move it to a more appropriate section if this is the case. I'm just trying to get the best visibility I can for it among the regulars.

I work for a major manufacturer of computer hardware and we're seriously considering Puppy as a candidate for testing our flashdrive-embedded OS proof-of-concept project. Without going into too much initial detail, we're trying to determine the extent of chipset bottleneck in computer boot speeds.

In order to demonstrate our solution, we need to show the difference between booting an OS that is decompressed at boot time and booting an OS that is not decompressed at boot time (but is still in archival format). The two OS platforms must otherwise be identical.

Really this is just a long way of asking someone here who has a properly configured Puppy host-compiler environment to please make available a pup_201.sfs file that has been created with mksquashfs using the -noD -noF and -noI command line parameters. These parameters of course cause squashfs to create the .sfs file without any compression.

If someone here (and I realize this would most likely be Barry, since his host system is the source of the distro) could make available an uncompressed pup_201.sfs, we would be eternally grateful.

We're willing to attempt the feat ourselves, but we haven't managed to set up a proper host-compile environment and will probably need some guidance building one that would provide what we're looking for.

-Shawn

P.S. We're using Puppy in the office here more and more. We love it!
Back to top
View user's profile Send private message 
MU


Joined: 24 Aug 2005
Posts: 13642
Location: Karlsruhe, Germany

PostPosted: Fri 07 Jul 2006, 12:54    Post subject:  

you easily could create it youself.

The folder is
/initrd/pup_ro2

so you create a uncompressed like this:

cd /root
mksquashfs /initrd/pup_ro2 pup_201.sfs -noD -noF -noI

This creates a new pup_201.sfs in 10 seconds, it is 187 MB in size.
Mark
Back to top
View user's profile Send private message Visit poster's website 
Locodarwin

Joined: 07 Jul 2006
Posts: 4

PostPosted: Fri 07 Jul 2006, 13:16    Post subject:  

Mark,

Thanks for your (slightly embarrassing but nonetheless) excellent solution. It is exactly what we need, and as a result Puppy will be our testing OS of choice for future endeavors as well.

-S
Back to top
View user's profile Send private message 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7047
Location: Perth, Western Australia

PostPosted: Sat 08 Jul 2006, 11:02    Post subject:  

I'm just now reading this thread. Thanks for giving the answer Mark!
We would like to know the result of your tests!
Back to top
View user's profile Send private message Visit poster's website 
sunburnt


Joined: 08 Jun 2005
Posts: 5009
Location: Arizona, U.S.A.

PostPosted: Tue 11 Jul 2006, 12:24    Post subject:  

Hi Locodarwin; Here's a complete set of tests with Ubuntu & DSL comparing most of the compressed fs.

http://kerneltrap.org/files/PERFORMANCE.README.txt

Basically it's what I'd conjectured for years... that newer CPUs decompress faster than the HD reads.
This means that with newer PCs, ALL the compression methods are faster than raw files on the HD.
AND... in looking into cloop, squashfs, & e2comp, it looks like Barry made a good choice in Squash.
Back to top
View user's profile Send private message 
Locodarwin

Joined: 07 Jul 2006
Posts: 4

PostPosted: Fri 04 Aug 2006, 16:27    Post subject:  

Thanks for your reply, sunburnt. I apologize for missing it earlier.

The hard drive read speed is indeed the bottleneck in such an application. Modern CPUs have no problem beating the pants off the hard drive when it comes to compressed vs. uncompressed data. You can demonstrate this easily by ghosting a system with Norton Ghost, first creating a "highly" compressed image and then an uncompressed image of the same partition or disk. The highly compressed image will re-ghost leaps and bounds quicker.

But what if that HD bottleneck is removed?

Our tests set the preliminary stage for eventually showing the performance/boot-time difference between systems when how fast it takes to get the information into memory isn't a factor. The goal is to create systems that, after post, boot up to the desktop at the fastest possible speeds.

Near-future media drives will no longer be a bottleneck - hence our decision to look for other boot duration solutions.

Puppy decompresses rapidly no matter what, as the squashfs file isn't very large. That would suggest it might not be the perfect candidate for illustrating our point. However, one cannot simply zip up, say, the entire Windows XP OS and then see how long it takes to boot. Wink And systems like Ubuntu aren't as easy as Puppy to design this test around, either.

Puppy's decompression speed, while barely interesting to us humans, is perfectly significant for our testing, because in the lab our two flavors of Puppy (now 2.01 compressed .sfs and uncompressed .sfs, as well as a gunzipped init.rd) load off of our proprietary test drives within 0.14 seconds of each other. That's awfully close. It is actually possible, therefore, to measure the (unrelated) decompression time between the two flavors - while simultaneously suppressing unwanted ulterior factors - using just a simple bus signal trace.

The difference is significant, at least on the scale we're considering.

As the amount of data that needs to be decompressed increases, though, the CPU speed then becomes the determining factor of bootup duration. What can we do to diminish the significance of processor speed?

Our question then became...is it possible/practical to offload the work of decompression to the system chipset as it is being streamed to memory from the drive, thereby eliminating the need for the processor to decompress the data during or after the fact?

The conclusion we've drawn is yes, it will be possible and practical, given as-yet-unreleased advances in firmware and flash memory, to decompress data into system memory prior to the involvement of the processor - using an algorithm on a virtual machine inserted into the North bridge portion of the chipset. Basically, we pre-process Puppy's entire compressed OS on the fly as it loads, leaving the processor free to start chewing on the parts as they stream into memory.

We can simulate it using FPGAs at this point. We'll be looking into a hardware demo soon.

It might not sound like much of a time-saver at this point, and in truth it isn't - with current hardware. When you consider the size of Vista and other trends towards gigantic, all-inclusive operating systems, it's not hard to see how important boot-time compression will become, and how significant pre-processed decompression will also therefore become.

I can't speak of further particulars on this forum, as I'm contractually bound by non-disclosure agreement. Just be assured that Puppy is helping us make the case, and in the process is getting some good internal exposure with a pretty significant client base. Smile

We're grateful to Puppy's creator and community for their efforts on a fine, worthy operating system. Puppy has helped to justify my existence at my workplace. How's that for a user testimonial?

-S
Back to top
View user's profile Send private message 
sunburnt


Joined: 08 Jun 2005
Posts: 5009
Location: Arizona, U.S.A.

PostPosted: Fri 04 Aug 2006, 17:57    Post subject:  

Locodarwin; Just what I suspected for the HD tests, flash would show less difference (acess times).
Running Puppy in memory isn't going to run any faster compressed or not.
BUT the apps. in a squash file taking up 2.5 times less space in memory still makes it well worth while.
Especially considering that the Puppy boot portion (vmlinuz & initrd.gz) is < 4 MB (initrd.gz decompressed).
The squash file is 67 MB & expands to 175 MB, saving 113 MB of memory, WOW!

Using the MB chipset to do the decompress of the squash file is innovative!
I'd always said: What if the compression was a built in native instruction in the cpu?
Then ALL data storage & communication could be compressed FAST, memory included.
Puppy's squash file loaded into memory is a current method of memory compression.
But the compressor is still a kernel module, & so software, so it's slower.
Imagine the speed if the cpu or another piece of hardware did the work!
Back to top
View user's profile Send private message 
can8v

Joined: 15 Jul 2006
Posts: 584
Location: Yuba City, CA

PostPosted: Mon 14 Aug 2006, 21:51    Post subject:  

Locodarwin wrote:

Puppy decompresses rapidly no matter what, as the squashfs file isn't very large. That would suggest it might not be the perfect candidate for illustrating our point. However, one cannot simply zip up, say, the entire Windows XP OS and then see how long it takes to boot. Wink And systems like Ubuntu aren't as easy as Puppy to design this test around, either.
-S

Locodarwin,
I am far from an expert on this topic, in fact I can only follow about half of what you said. When reading this paragraph I quoted however, I thought well why doesn't he just use Puppy unleashed to make the most bloated Puppy ever. Then use the same commands to create an uncompressed version. Puppy Should be quite large if every single package were installed. That might make it easier to demonstrate your point, to whoever your audience is.
CAn8v
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 1 [8 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0683s ][ Queries: 12 (0.0036s) ][ GZIP on ]