Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 23 Jul 2014, 20:39
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Unnamed Remastering Tool
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [9 Posts]  
Author Message
Iguleder


Joined: 11 Aug 2009
Posts: 1867
Location: Israel, somewhere in the beautiful desert

PostPosted: Fri 09 Sep 2011, 02:09    Post subject:  Unnamed Remastering Tool
Subject description: A next-gen remastering tool that builds packages
 

Unnamed Remastering tool

Lately, I've been seriously involved in the development of automated package building infrastructures. I wrote a set of scripts which built, trimmed and packaged pretty much anything. I even made a magical script which takes a Puppy ISO and the matching devx module and uses them to build packages, so you can build packages for a Puppy version different from the one you use.

Some time ago I had a brilliant idea: if we have remastering tools and the means for automated package building on any Puppy for any Puppy, why don't we combine the three to form the ultimate remastering tool?

I decide to implement this idea in a tool, which takes a Puppy, builds your favorite packages inside it (so they're 100% compatible with it), optimizes it (strips all binary files, optimizes all images, re-compresses all archives, etc') and produces an optimized puplet with lower memory requirements and small size.

It is in alpha quality; use with care. Any damage done to your system is under your responsibility; if you don't agree with this, don't use it.

The tool is not meant for novice users; use it only if you have a certain degree of experience with Puppy, package building, package creation and remastering.

How It Does Whatever It Does

It's simple. In the UNIX family of operating systems (which includes Puppy), there is a cool utility called "chroot", which runs a command and changes the file system root it sees. For example, if you run this:
Code:
chroot /path/to/directory /path/to/executable a b c

your executable will run with the given command-line arguments and see the specified directory as its file system root (/), while your system is unaffected.

Now, imagine a Puppy is extracted to a large partition with the devx module loaded on top of it, as in a running Puppy. If you use chroot with such setup, you can use this environment to build packages; the only thing you need is an automated way to build packages; that's where Builder fits in.

Builder is a simple tool I wrote for building packages. It receives one command-line argument, which is the path to a script which defines how to build a certain package.

Here's an example for a script skeleton:
Code:
#!/bin/sh

PKG_NAME="mplayer"
PKG_VER="svn$(date +%d%m%Y)"
PKG_REV="1"
PKG_DESC="Multimedia player"
PKG_CAT="Multimedia"
PKG_DEPS="+yad"

download() {
    ...
}

build() {
    ...
}

package() {
    ...
}


The script defines the package name, version, description, category and dependencies. Then, it is split into three parts, which contain the instructions for downloading the package, building it and turning it into a Puppy PET package.

Builder prepares a build directory and builds each package according to the magic recipe contained in its build script. Once it's done, Builder trims the package, splits it and compresses it. It is 100% automatic.

In short, the remastering tool extracts the Puppy you give it, then uses chroot with Builder to build your desired packages inside it. Then, it trims and optimizes it; unneeded documentation and some useless files are removed to reduce its size.

Finally, the puplet is created from the extracted Puppy.

System Requirements

- squashfs-tools 4.1 or above, with support for at least one compression method. Make sure it supports the "-comp" option; if it doesn't, install a newer version. This one should be just fine.
- Wary, Slacko, dpup or any Puppy built with a recent Woof.
- Some build scripts require xz-utils, Git and Subversion.

Limitations

In its current implementation, the remastering tool has several limitations:
- It doesn't deal with dependencies; you have to build the packages in the right order if they depend on each other.
- It doesn't have a name :p

Included Build Scripts
- Develpment: AdvanceCOMP, OptiPNG.
- Fun: DOSBox, ScummVM and OpenTTD.
- Utilities: Galculator, gHasher, TrayFreq, Xarchiver, rxvt-unicode, emelFM2.
- Documents: Leafpad, Zathura.
- System tools: LXTask.
- Multimedia: Aumix, DeaDBeeF, FFmpeg, MPlayer, TiMidity++, freepats.
- Misc. stuff: file, Squashfs, syslinux, dhcpcd, Yad.
- Graphics: mtPaint, XSane, Viewnior.
- Desktop: LXAppearance.
- Fonts: Liberation, Ubuntu, Libertine, DejaVu, font-bh-ttf, FreeFont.
- Internet: Seamonkey, youtube-dl.

Desktop stuff:
- Window Maker: Window Maker, AlsaMixer.app, asmon, wmclock, wmsystemtray.

The included build scripts allow you to transform a Puppy into something very similar to Next Puppy, a minimalistic dpup puplet with Window Maker.

Notes

- The trimming script assumes you have Builder and all its dependencies available on the host distribution used to run the remastering tool. If you don't, simply copy the contents of the Builder directory to / and install both AdvanceCOMP and OptiPNG if you want to optimize your puplet. Bear in mind that puplets created using the remastering tool are "self hosting" and should have all those built-in, so they can be used to create the next versions of your puplet without any additional effort.
- If you read the list of included build scripts, the tool is able to build DOSBox and ScummVM, which are great if you're into retro DOS gaming. I also wrote build scripts for TiMidity++ and freepats (which contains sound samples), which provide MIDI emulation through software. It is useful since most modern sound cards lack the MIDI support that was popular back at the DOS era; most DOS use MIDI for music and sound effects.
- The MPlayer package built using the script can be built statically, without an external FFmpeg. This saves lots of space and some RAM.
- The MPlayer build script also installs a script called "mplayer_shell", which includes a file chooser that makes it easier to use the "mplayer" binary when GNOME MPlayer or any other GUI frontend is missing. It depends on Yad, so make sure you have it, too.
- The Seamonkey package is built from the static, official binary packages provided by Mozilla and not compiled, since it takes ages and because it updates so often it makes no sense to compile it on your own. Also, when you build it yourself, the updater is missing. Moreover - the official binaries are optimized for size, so there's very little (if any) difference in the result package size.
- The font packages are huge, especially the DejaVu fonts; use with care. For the best balance between small size and looks, go for the DejaVu and Liberation fonts.
- At the moment, the tool does not support selection of the default wallpaper, GTK theme, window manager theme, etc'.
- At the moment, the tool deletes all wallpapers in /usr/share/backgrounds, because they waste lots of space. I can change this behavior to make it keep only the default wallpaper.

How to Use It

Using the tool is very simple. Start by editing the configuration file.
Code:
defaulttexteditor config


Make sure you give it the right paths to the Puppy ISO image and the matching devx module. Also, make sure the kernel your puplet uses supports the Squashfs compression you chose.

Once the configuration file is ready, run the extraction script to fully extract the Puppy you are remastering.
Code:
./0extract


If you ever want to remove up the extracted files, run it with the "clean" option:
Code:
./0extract clean


If the extraction succeeded, you'll see no errors. Proceed to the build environment preparation script:
Code:
./1prepare


Bear in mind that this script mounts file systems and does some very dangerous stuff. Use it with care; never call 0extract with the "clean" option before you make sure the file systems mounted by 1prepare are unmounted.

As with 0extract, you can always clean up the directories created by 1prepare using the "clean" option:
Code:
./1prepare clean


This will also unmount all file systems mounted by the script and make sure it's safe to call 0extract with the "clean" option. I'll say it again: never ever call 0extract with the "clean" option without doing this first.

Once the build environment is ready, you can proceed to the package building stage:
Code:
./2build package


This script will build and install a single package. The PET packages built through it will be available in the "packages-pet" directory and the sources will be available under "sources". This is the heaviest part of the remastering tool.

Regarding the packages you can build, you can find all included build scripts under builder/var/builder. Of course, you can add your own build scripts to that directory. Just take a look at the existing ones to see how to write one. It's very easy.

In order to produce smaller and faster packages, build AdvanceCOMP and OptiPNG and make sure you build the former twice. AdvanceCOMP is used for optimizing packages (or more percisely, to compress them better), so you'll need to build it again to make it optimize its own package.

Of course, you can always clean up the working directory used for building packages using the "clean" option:
Code:
./2build clean


Once you built all your packages, run 1prepare with the "clean" option to unmount all mounted file systems:
Code:
./1prepare clean


This ensures you don't run into any synchronization or permission problems because of mounted file systems when you proceed to the next step.

Then, you can call the trimming script:
Code:
./3strip


This script will optimize your puplet. It takes ages to run in some cases, but it's definitely worth it in my opinion. If you remaster a truly barebones puplet it won't take long.

Finally, call the puplet building script:
Code:
./4builddistro


This script will take a long time, since it compresses lots of files. Once the script is done, the build output (your puplet's ISO image and a matching devx module) will be available in the "dist" directory.

You can easily clean the output of 4builddistro using the "clean" option:
Code:
./4builddistro clean


FAQ

Does it work with Lucid Puppy?

I don't know, since Lucid Puppy was built using an older Woof. I think it won't work. The only thing I can say for sure is that Lucid Puppy supports only gzip compression.

Regarding Puppy 4.3.1 and older - I'm confident it won't work.

Which compression should I use?

Traditionally, Puppy uses gzip compression, which provides a good balance between performance and the puplet size. It is the sane default you should use if unsure.

You can also use XZ compression, which provides much better compression at the cost of time and speed. It requires Linux 2.6.38 and above; it is good for USB installations, since the bottleneck in many cases is the slow reading speed - if the puplet is smaller, less time is spent on reading it from the slow flash drive.

Also, there's LZO compression, my personal favorite - it is supported on Linux 2.6.36 and above and provides insanely fast decompression. It makes your puplet way faster, but much bigger. I think it's nice to have a LZO-compressed flavor for a puplet, for those who use it for hard drive installations and don't care much about size.

If you're still unsure after this explanation - try all. You'll find out that LZO is very fast and comfortable to work with, XZ produces smaller puplets at the cost of very long compression times and some loss of performance, while gzip is well-balanced.

Can I use it to optimize my Puppy?

Yes, you can use this tool just to optimize a Puppy, without building any packages.

Which packages should I choose?

As mentioned earlier: I suggest you to always include AdvanceCOMP and OptiPNG and build the former twice.

Regarding the rest, the answer is simple: listen to your heart. Laughing

How long does it take to build a puplet?

It depends on how fast your machine is, the compression you use, whether you run the 3strip script and your choice of packages.

Does it remaster the devx?

Yes, it does.

Can I use it with a Puppy that has a zdrv module?

Maybe; this feature exists but I haven't tested it.
remasterer-001.tar.bz2
Description 
bz2

 Download 
Filename  remasterer-001.tar.bz2 
Filesize  17.22 KB 
Downloaded  192 Time(s) 
screeny.png
Description  Screenshot of dpup remaster with Window Maker
png

 Download 
Filename  screeny.png 
Filesize  133.68 KB 
Downloaded  229 Time(s) 

_________________
My homepage
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
ICQ Number 
puppyluvr


Joined: 06 Jan 2008
Posts: 3183
Location: Chickasha Oklahoma

PostPosted: Fri 09 Sep 2011, 17:09    Post subject:  

Very Happy Hello,
Downloaded, but not yet checked out...
Brilliant concept...
The talent pool around here is getting really deep...
I love being a small fish in an ever expanding pond...
So much learning to do.. Cool

I`ll get back after a bit of studying, and then a bit more...

_________________
Close the Windows, and open your eyes, to a whole new world
http://puppylinuxstuff.meownplanet.net/puppyluvr/
Puppy Linux Users Group on Facebook

Puppy since 2.15CE...
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4277

PostPosted: Fri 09 Sep 2011, 17:58    Post subject:  

I have one that I did a couple years ago that handled any number of sfs files (including the main one) and would then do installed pets and any number of additional pets/sfs...never really felt like "marketing" it though. It's pre-woof so it may be useless now, but I can send an older version if it will be helpful.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
ldolse

Joined: 23 Oct 2009
Posts: 366

PostPosted: Mon 26 Sep 2011, 13:02    Post subject:  

I've been working quite a bit with the remastering tool, and although the latest version is in the Next thread I figure this is probably a more appropriate place to talk about it in general.

Firstly - unti I read this thread I didn't really comprehend how awesome this tool is, or what it's doing under the hood - I'm still building all the packages for my first real remaster using it so I haven't stressed everything yet, but what I've used is looking really good.

One issue that I'm seeing is that the tool seems to assume that strippkg is in the sfs or the devx that's being targeted for remastering. They weren't there, and until I understood the deal with chroot I couldn't figure out why I kept getting file not found errors. Once I stuck them in the sfs the issue went away. Not sure if splitpkg and buildpkg are similarly affected.

I've got a couple feature requests - I suspect these might actually be quite simple based on the way aufs works:
  • Target packages for installation into the devx - e.g. I didn't realize I need cmake for a package, and I need to install it to the chrooted environment in order to use it, but I don't want it on the final sfs. Targeting to the devx would be perfect. Otherwise I need to figure out some manual install to the devx folder (and unfortunately installwatch doesn't support cmake), or worst case re-build the iso/devx from woof.
  • Target the packages for a third sfs folder - this would allow you to create pets/sfs files for projects that are typically difficult because installwatch doesn't support the installation methods (e.g. perl/python/cmake installers). As an example I'd like to build a Squeezebox sfs file as that project is huge, has lots of packages, and is mostly perl based. Targeting a different sfs would be perfect for a scenario like that.
I think because these would wind up in a different folder in the UnionFS it would be trivial to control which folder the files finally ended up when everything gets cleaned up. Let me know if I'm misunderstanding how it works.


I'm also seeing one issue with a little package - this package doesn't have a proper configure script, just a Makefile that's pre-made as it's a simple package. When you run make install it's executing these lines:
Code:
install: all
   @echo Installing...
   $(Q)install -m 644 $(SND_PCM_BIN) ${DESTDIR}/usr/lib/alsa-lib/
   $(Q)install -m 644 $(SND_CTL_BIN) ${DESTDIR}/usr/lib/alsa-lib/

${DESTDIR} is mapping to the actual build directory when called from ./1build, and since usr/lib/alsa-lib/ doesn't reside inside the build directory the install fails. Do you think I just need to remove ${DESTDIR}? Working on an alternative install, but apparently Next is missing the alsa-lib development headers (a common issue on several puppy variants). I figured while I was fiddling with that I'd ask if there was a simple way to fix the Makefile.
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2217

PostPosted: Mon 26 Sep 2011, 13:12    Post subject:  

DESTDIR is null unless you give it a value. To use it you need to run:
make DESTDIR=/some/dir install
or sometime before executing make install:
export DESTDIR=/some/dir
...
make install
Back to top
View user's profile Send private message 
Iguleder


Joined: 11 Aug 2009
Posts: 1867
Location: Israel, somewhere in the beautiful desert

PostPosted: Mon 26 Sep 2011, 13:17    Post subject:  

The first is something I wanted to implement in the long term, I think I'll get around to it soon, it's pretty easy.

Regarding the second feature request - it's something very hard to implement, since there's no generic way to do this. I think I won't implement that.

_________________
My homepage
Back to top
View user's profile Send private message Visit poster's website MSN Messenger 
ICQ Number 
ldolse

Joined: 23 Oct 2009
Posts: 366

PostPosted: Mon 26 Sep 2011, 13:42    Post subject:  

Iguleder wrote:
Regarding the second feature request - it's something very hard to implement, since there's no generic way to do this. I think I won't implement that.


If it's hard to implement then no big deal, but I guess I'm not understanding part of how it works. I thought it was working something like this:
  • Set up a temp layered file system with 3 layers, devx, sfs, and new content layer
  • Chroot to this new filesystem, build packages
  • Move the files from the 3rd layer for new content to the sfs layer


If I'm understanding it correctly couldn't you then choose to move the new content to either the devx, sfs, or a third custom folder? I guess if you're doing this with multiple packages and dependencies you need to copy to both the sfs and the third custom folder for each new package, but that wouldn't be a big deal.

If I'm misunderstanding and it's just two layers then I see why it would be difficult.
Back to top
View user's profile Send private message 
ldolse

Joined: 23 Oct 2009
Posts: 366

PostPosted: Mon 26 Sep 2011, 13:44    Post subject:  

amigo wrote:
DESTDIR is null unless you give it a value. To use it you need to run:
make DESTDIR=/some/dir install
or sometime before executing make install:
export DESTDIR=/some/dir
...
make install


Somehow it's getting a value from the build scripts, and the value is the root of the source folder - maybe it's just a variable collision. It sounds like I can delete it from the Makefile and I should be fine then.

Edit - I see where I can tweak the build script to get rid of it now, thanks for the tip!
Back to top
View user's profile Send private message 
noryb009

Joined: 20 Mar 2010
Posts: 537

PostPosted: Mon 26 Sep 2011, 22:40    Post subject:  

It would be great if you make the variable and function names interchangeable with another build system's (like arch's PKGBUILD) variable and function names. It would at least give users an optional starting point to making Builder scripts.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [9 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1171s ][ Queries: 12 (0.0036s) ][ GZIP on ]