upx, the compressor for executables

Stuff that has yet to be sorted into a category.
Post Reply
Message
Author
User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

upx, the compressor for executables

#1 Post by MU »

Hi,

somewhere in this board I read, someone uses Opera 5 statically compiled with QT.

The result should be a huge executable.

To minimize size, you might try upx, the compressor for executables: http://upx.sf.net

I use it to compress the wxbasic-interpreter.

example:
Consoleversion uncompressed: 150 kb, compressed: 50 kb
Gtk-version uncompressed: 5.8 MB, compressed: 1.8 MB

Mark

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#2 Post by Lobster »

Very good advice - I have found upx very useful - the programs run faster when compressed, being smaller on the load into memory (but this I have never noticed so it is a small increase)

As a side question, Puppy uses a compression system (I am sure this has been thought out) would someone like to explain this?
Last edited by Lobster on Thu 25 Aug 2005, 02:46, edited 1 time in total.
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

Bruce B

#3 Post by Bruce B »

According to my understanding a UPX compressed file uncompresses itself prior to loading in memory. It uses just as much memory as the uncompressed version.

The main advantage I know of off hand to compressing an executable binary file with UPX is it occupies less disk space. Or RAM space if that's where you are storing it.

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#4 Post by Lobster »

?

Are you sure - I have only used the Windows version - this actually compresses the executibles. :?:
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

Bruce B

#5 Post by Bruce B »

> Are you sure - I have only used the Windows version - this actually compresses the executibles.

It compresses them quite effectively. It has to be decompressed to run. The decompression is seamless and you don't see it. Am I sure?

98% sure.

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#6 Post by Flash »

If I understand correctly, here's how works in Knoppix: the info on the CD is compressed (personally, I think "condensed" is more accurate than "compressed," but I'll use compressed if it keeps everyone happy.) For the CPU to execute it, the info must be expanded (reconstituted) into the RAM, which is done (by the CPU, on the fly) as it is read from the CD. I've read that, in the case of Knoppix at least, the time saved by reading the info from the CD in compressed form more than makes up for the time it takes for the CPU to expand the compressed info. I have only the vaguest idea how a compression/expansion algorithm might work.

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#7 Post by Lobster »

Thanks Bruce

As usual I have not a clue wot I am on about - so thought it good to ask . . .
Does UPX put a header program in the executable that decompreses the (in effect a zipped type file)?

If that is an oversimplification - good - dumb it down further I say . . .

:)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

doopdoop
Posts: 48
Joined: Thu 28 Jul 2005, 08:38
Location: Magdeburg, Germany

#8 Post by doopdoop »

Puppy does basically the same things. The usr-filesystem from CD is a is a compressed read-only system (name squashfs, I think Knoppix uses the same). The algorithm is optimized for fast decompression with small overhead under these conditions. squashfs is implemented as an kernel module, the same way as reiserfs, ext2, iso9660 or msdos.
Compressing the executables on this fs does not reduce the real file size further( or not much, depending on the efficency of the algorithm), because they are already compressed. There are information thereotical limits for that.
But if you are really tight in the pup001 (which is an ext(2/3?), uncompressed), it can help.

Bruce B

#9 Post by Bruce B »

> in the case of Knoppix at least, the time saved by reading the info from the CD in compressed form more than makes up for the time it takes for the CPU to expand the compressed info. I have only the vaguest idea how a compression/expansion algorithm might work.

CD-ROM drives are not very fast compared to today's hard drives. Like way slow.

The decompression happens in RAM which is much faster than even a hard drive.

I'm not familiar with Knoppix much but the statement appears sound to me.

-----------------------

Back to Puppy, suppose some one wants to compress the binary exectables on the hard disk. I bet you would never notice the delay in the time it takes to decompress.

In theory the read heads could pick it up quicker because it is smaller. The decompression is very fast.

I've not noticed one ounce of difference in running a compressed exectable as compared to an uncompressed executable.

Also UPX can decompress the file you compressed to its orginal state. From this perspective one has nothing to lose by experimenting. The changes one makes don't have to be permanent.

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

Squashfs

#10 Post by Flash »

doopdoop wrote:Puppy does basically the same things. The usr-filesystem from CD is a is a compressed read-only system (name squashfs, I think Knoppix uses the same). The algorithm is optimized for fast decompression with small overhead under these conditions. squashfs is implemented as an kernel module, the same way as reiserfs, ext2, iso9660 or msdos.
Compressing the executables on this fs does not reduce the real file size further( or not much, depending on the efficency of the algorithm), because they are already compressed. There are information thereotical limits for that.
But if you are really tight in the pup001 (which is an ext(2/3?), uncompressed), it can help.
Thanks! I wondered what squashfs is, but figured it would take a long hunt through many websites before the light began to dawn. Your explanation, that it is in essence a compressed filesystem implemented as a kernel module, is very helpful to understanding what it is used for and how it is used.

Is that about right?

The initial Linux kernel which loads into RAM from the CD to begin the boot process obviously cannot be compressed on the CD, because the computer does not contain any program able to expand it. (Though, in theory, the BIOS could contain something similar to squashfs.) That initial kernel is what contains the squasfs module, which expands the subsequent compressed info that is read from the CD as the boot process continues.

doopdoop
Posts: 48
Joined: Thu 28 Jul 2005, 08:38
Location: Magdeburg, Germany

#11 Post by doopdoop »

Well, yes, partially right. The Bootloader (grunb, lilo, whatever) starts a part of the kernel, which can expand an initial ram disk (initrd.gz). This ram disk contains the modules (like squashfs, ext3 or SCSI, whatever). So, in fact the modules are compressed too, but with a different system.

Bruce B

#12 Post by Bruce B »

I'd say the rule for an option 1 install is do not compress anything in /usr

It will backfire on you. /usr is a merged file system with /root/.usr

Gnumeric for example is in /usr and not in /root/.usr by default. If you compress gnumeric, it (the compressed version) will show up in /root/.usr as a new file that was not there before. Suppose the compressed size is 1.5 mb, you have justed added 1.5 mb to pup001.

If you want to save space in /usr which is where a lot of executable files exist, make sure that the file exists in /root/.usr

If it already exists in /root/.usr then you will save space in pup001 by compressing it. Otherwise you will add space.

As far as my-roxapps and my-applications/bin are concerned you can save space by compressing them. This post applies only to /usr and the merged file system /root/.usr.

Post Reply