CLOSED : Underdog/Hybrid Debian Jessie (boot frugal or full)

A home for all kinds of Puppy related projects
Message
Author
User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

CLOSED : Underdog/Hybrid Debian Jessie (boot frugal or full)

#1 Post by rufwoof »

THREAD CLOSED

Barry outlined UnderDog in http://barryk.org/puppylinux/developmen ... works.html ... extracts of which follow :
An exciting alternative to the squashfs extensions is to use an existing installed Linux distro as the bottom layer:
Image
What the above diagram is intended to convey is that the bottom layer is a partition, not the "underdog.lnx" file itself. File underdog.lnx is just a text file, containing the name of a partition, for example "hda1".
At bootup Puppy will read underdog.lnx and will mount the partition as the bottom layer. If that partition happens to have a Linux distro installed in it, then the entire distro filesystem will "show through" on the top layer of Puppy's unionfs.
But noted in http://barryk.org/blog/?viewDetailed=02668
There are possible problems, mostly to do with mismatched libraries. The Puppy libraries would be in a higher layer, which might mean that an application that you want to run in the Underdog Linux layer might not work. With some care though, it should be ok, for example with Wary/Racy, use an older version of Ubuntu not the latest as the Underdog. Or, better, use a matching distro, example Slacko and Slackware 13.37.
The image file available here is a form of Debian Jessie LXDE/Openbox (64 bit) UnderDog (of sorts, there's no underdog.lnx file involved as initrd and vmlinuz are as provided by Debian).
Image
Its a xz compressed image file (1.3GB that expands to a 16GB USB image) so you'll need a USB of a least 16GB and a means to install the image to that USB http://barryk.org/news/?viewDetailed=00330 and have your PC set up so that it can boot that USB http://barryk.org/quirky/uefi.htm. For Linux users, you can use dd to write the image to USB :

Be careful, write to entire drive, for example 'sdf', not to a partition, example 'sdf1'.
Make sure that the drive is unmounted, as some Linux distros will auto-mount it as soon as it is plugged in.
Also be extra careful that it is the desired USB drive you are writing to, not your main hard drive!

Code: Select all

xz --decompress --stdout jessie-usb.img.xz | dd of=/dev/sdf bs=4M conv=fsync
Be patient, it might take a hour or more to write to USB.

BOOT CHOICES

1.

Boot loader → vmlinuz → initrd → Debian Jessie bottom layer + main sfs (empty by default) + additional sfs's/.dir/.ext layers if added.
Changes stored in ram, which can be flushed to the bottom (persistent) layer using flush2disk (2nd icon on panel), or are lost at shutdown/reboot.
Layering achieved by using persistence persistence-read-only boot parameters

2.

Boot loader → Debian Boot loader → vmlinuz → initrd → Debian Jessie
Changes stored as and when they occur. Comparable to a Debian full install.


Same vmlinuz and initrd in both cases, as supplied by Debian

Benefits of type 1 is that it largely circumvents that USB's are slow to write to and their life can be reduced by excessive/repeated writes. Another benefit is that changes can be 'undone' simply by rebooting without saving (handy for trying out things). Disadvantage is that as changes are stored in memory (with option to flush those changes to disk upon demand) which tends to be more limited compared to free disk space.

Type 2 in writing changes to disk as and when they occur is constrained to the number of changes only by available free disk space. Disadvantage is that it runs slower than type 1 that records changes in memory and that all changes are preserved no matter what.


By default auto logs into desktop as user

Passwords :

user = live
root = me

Note that pcmanfm (file manager) has a right click option to open a window as root (right click on folder, select "SU file manager" and that window opens under root userid).

If you only install programs from Debian, using apt-get (text) or Synaptic (gui), then you partake in a large central system of stable programs that work well with each other. Security issues/updates from Debian tend to be quickly addressed/applied/distributed, and provided you update periodically then over time you'll automatically be migrated onto later releases/versions.

The img does contain non-free software/firmware, so please inspect and confirm whether or not it is ok for you to use such software in your country/region. Downloading and usage is on the basis that you acknowledge, respect and honour each individual components Copyright and Conditions of Use.

With thanks and greatest appreciation going out to Barry without whom such projects would never have seen light. And of course to Linus Torvalds. And also to the extensive Debian team who tirelessly develop and maintain Debian. Thanks also to all those who helped to develop, maintain and enhance snapmergepuppy/save2flash (and to Fred who helped me fork that for this particular project). Thanks also go out to all those others such as the LXDE and Openbox teams ... amongst many others, who have helped make Linux what it is today.
Attachments
sc.gif
(55.92 KiB) Downloaded 836 times
Last edited by rufwoof on Sat 15 Oct 2016, 20:40, edited 11 times in total.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#2 Post by rufwoof »

Technical background - How Built :

Debian Live CD /live and /boot folders copied out of the ISO onto HDD and booted frugally using grub4dos. Updates/additions (programs)/cosmetically changed and saved where the save 'folder' is the same partition as where grub4dos and /live /boot were installed (note uses a save partition, not a save file or save folder). Largely turning a livecd version (as on the livecd) into a more full install version (/etc/apt/sources.list extended from the limited livecd repository set to a full set). Set to boot using SystemD using persistence persistence-read-only boot parameters for the frugal boot choice, or as a more conventional full install for the second boot choice (that chains to Debian boot menu that offers three boot choices).

All of the filesystem.squashfs (main squashed file system) was then extracted and merged into the save 'file' (partition) ... in effect moving everything into the save file, leaving no need for the main filesystem.squashfs. A near empty filesystem.squashfs has been left however as that better enables loading of sfs's if so desired. sfs's can be loaded by dropping them into the /live folder and listing them in /live/filesystem.module (they are loaded/layered in the order specified in that file). Alternatively if filesystem.module is deleted (or renamed) then all files with a .squashfs suffix are loaded in alphanumeric order.

To facilitate saving changes recorded in memory during a frugal boot, flush2disk script (amended version of save2flash) was added.

The HDD installed version (partition) was then replicated on a USB, and that usb was then used to form the img.xz (after zero'ing free space to aid better compression).

The initrd.img file as supplied by debian is a .gz compressed file. To extract the content make a copy of that file with a .gz suffix and then uncompress that file. Make a folder beneath where that file is, cd to that folder and run cpio -id <../initrd (or whatever filename you decompressed to). To reform use find | cpio -o -H newc >../initrdnew and then cd .. and compress that file using gzip (gzip initrdnew) and finally add a img suffix (mv initrdnew initrdnew.img).

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#3 Post by rufwoof »

Operationally the default (frugal style) boot can be as quick as a HDD installed version. Writes to disk (usb) however are the speed killer, so any saves or booting full mode (that writes all changes to usb as they occur) are slow. Faster USB 3 would help in that respect. As such applying updates (apt-get update; apt-get upgrade), and even saves, are best applied whilst you have other tasks to do.

So best run in frugal mode, saving docs/files on another partition/disk. Or perhaps creating a direct link to the lowest layer for document storage :

su (and enter 'me' password)
cd /home/user
mkdir /lib/live/mount/persistence/sdf1/Documents
ln -s /lib/live/mount/persistence/sdf1/Documents Documents-persistent

type command (changing sdf1 accordingly), which for 'user' will create a Documents-persistent folder (symbolic link) in the home folder that writes directly to the usb (preserved across reboots, even if no save is made).

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

HDD installation

#4 Post by rufwoof »

If you decide to install to HDD you can't just copy the files from usb to hdd (there are symbolic links and permissions that just wont copy correctly and you'll just have a broken system). Instead use a means that preserves those links/permissions.

Personally I tend to create a sfs of the files/folders and then extract that sfs, but as you can't extract a sfs to the root folder I use a temporary folder and then use a file manager to drag/move all of the contents of that folder up to the root level.

If for instance the usb is mounted as /mnt/sdf1 then change directory to the root of a newly created ext3 or ext4 HDD partition and

mksquashfs /mnt/sdf1 jessie.sfs

once that completes and you have /jessie.sfs on the HDD then

unsquashfs jessie.sfs

which creates a /squashfs-root folder on the HDD with all of the files/folders. Then use filemanager to open up that squashfs-root folder, select all files (including the 'hidden' files) and drag those up to the / level and select the 'move' option. The empty squashfs-root folder can then be deleted.

That will still require grub4dos to be installed (unless you have it installed already), so run that, but don't create a menu.lst file (or make a backup of the existing menu.lst and replace the new one with that backup after running grub4dos installation). Or if you already have grub4dos installed then edit the existing /menu.lst file content to include the entries contained within the jessie-usb menu.lst file.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

Firewall (UFW)

#5 Post by rufwoof »

A note about ufw (firewall). When you click on the panel shield icon for ufw the values are all greyed, and you have to click unlock in order to see the actual current status of the firewall (by default its sets ON, even though the greyed out initial window looks as though its set OFF). That's just the way, for whatever reason, ufw works.

backi
Posts: 1922
Joined: Sun 27 Feb 2011, 22:00
Location: GERMANY

#6 Post by backi »

Hi rufwoof aka captain Kirk !

Phoenix/Sirius rising ...
Maybe call it--- Sirius Puppy-The Greater Dog .

Or Debian-Dog-Star .

" Sirius – in the constellation Canis Major the Greater Dog – is the sky’s brightest star. It’s very easy to spot on winter and spring evenings. "

http://earthsky.org/brightest-stars/sir ... htest-star
https://en.wikipedia.org/wiki/Sirius

There are also options for names Sirius A........or Sirius B .

"What the naked eye perceives as a single star is actually a binary star system, consisting of a white main-sequence star of spectral type A1V, termed Sirius A, and a faint white dwarf companion of spectral type DA2, called Sirius B.

This is advanced Puppielogie .

Greetings from Mother Earth to captain Kirk aka rufwoof on Star Ship Enterprise in Galaxy Linux .

May the Force be with you

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#7 Post by rufwoof »

My fine fellow, follically challenge as I am would equate me more to Picard

VY Canis has a nice canine ring to it and perhaps the better choice :) In size, dwarfs the earth bound pup's.

[That video is dated BTW, Canis is no longer the top dog].

/usr/share/locale is nigh on 400MB alone!
Loads of firmware ready to go
Bluedevil (bluetooth)
Libre office suite
Skype
Master PDF editor
Firefox-esr
Chromium with PepperFlash
Openshot (video editor) together with Blender for 3D animated titles, full inkscape
Audacity for sound editing
Fred's - Dog Radio (Dance Groove is my favourite background music channel)
Pulse Audio
Record my desktop
mtpaint/gimp ..... etc.

All updated to the latest Debian Stable version at the time of uploading.

There's also a full documentation set, for instance if in a terminal you type
man live-boot
then you'll see the manual entry for live-boot.

If you boot full (no sfs's loaded) and run
apt-get update
apt-get upgrade
you'll be updated with the latest security fixes (and program updates). Which might include kernel updates also.
Image
(full image here)

Providing you stick with Debian repository of stable programs that all work well with each other/together, then installing/removing programs is relatively easy using Synaptic gui pack manager. And being able to boot read only means you can try programs out (install them), and if no good, a reboot undoes all of that trial.

Given Debian's Toy Story naming, Wheezy -> Jessie -> Stretch, I guess Kirk/Picard is out of the window (airlock).
Last edited by rufwoof on Sat 01 Oct 2016, 13:51, edited 3 times in total.

backi
Posts: 1922
Joined: Sun 27 Feb 2011, 22:00
Location: GERMANY

#8 Post by backi »

Seems i am a bit outdated (excuse me.... living on the countryside )
It looks like you`re quite well informed ....... .... me impressed again :D
By the way.... do you refer to ......Jacques or his father Auguste Piccard ?...... :D :D :D just kidding :) :)


O.K maybe call it Debian Super Dog .....or Pluto the Dog or...or..or.....

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#9 Post by rufwoof »

Mouse into the bottom left corner and it might (depending upon your graphics card/setup), morph a desktop full of overlapping windows ... into a nicely spaced out set of windows where you can use left, middle (or depress scrollwheel), right mouse actions to focus a windows, minimise or close windows.
Attachments
a.jpg
(58.76 KiB) Downloaded 678 times
b.jpg
(29.6 KiB) Downloaded 695 times

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

Executing a script at startup

#10 Post by rufwoof »

LXDE is a bit 'odd' for adding startup scripts

Open file manager (pcmanfm) and navigate to /usr/local, then right click on bin and select the SU file manager choice so that you open up /usr/local/bin in a su file manager window (pcmanfm puts a little icon near the top right to show its running as root, in contrast the likes of Thunar put up a total window wide orange banner, which personally I find a bit OTT).

Create a script in /usr/local/bin and make it executable. Then navigate to /home/user/.config/lxsession/LXDE and in autostart at the name of that /usr/local/bin script with a @ prefix, i.e. the default set looks like

@lxpanel --profile LXDE
@pcmanfm --desktop --profile LXDE
@xscreensaver -no-splash
@remount-rw
@brightside
@python-preload.py
@libreoffice --quickstart
@pre-save

At reboot those are run with root authority, so there's no need to have su checks in the /usr/local/bin script.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

SystemD service

#11 Post by rufwoof »

jessie-usb runs under systemD so you can run commands (as root) such as
systemd-analyze
that will show boot up timing or
systemd-analyze blame
that shows more detailed timings of individual services

To plug something into systemD, let's say run a test for a file in /tmp and if present do some action, you create a script in /usr/local/bin

Code: Select all

	
#!/bin/bash
if [ -f /tmp/dosomethingatshutdown ]; then
  echo hello >/tmp/hello
fi
(Obviously with the code you actually want to run)

and then create a systemD service to link that into systemD i.e. create a .service file such as

/lib/systemd/system/dosomethingatshutdown.service

Code: Select all

[Unit]
Description=do something at shutdown if /tmp/dosomethingatshutdown file exists
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/dosomethingatshutdown

[Install]
WantedBy=halt.target reboot.target shutdown.target
... and enable it

systemctl enable dosomethingatshutdown.service

you can check its status is enabled by scanning through

systemctl list-unit-files

that shows all files, or just look for the single file by name

systemctl list-unit-files dosomethingatshutdown.service
that typically reports back something like :
UNIT FILE STATE
dosomethingatshutdown.service enabled

Or you can disable it
systemctl disable dosomethingatshutdown.service

Of course all of the above should be using root (su).

If you can read quick enough, you might even see that service being checked during shutdown amongst all of the other textual messages logged to screen.

One thing I forgot in the uploaded version was to re-enable the ModemManager service. I have no modems so I disabled that bootup service, but if you do have a modem then you may need to re-enable it

systemctl enable ModemManager.service

The tricky part, at least for me, is the .service file and knowing how to code it to slot the execution into the right place, Before, Wanted By, After type values to enter. I guess to be more proficient in that you have to have a better understanding of all of the other .services that are running/available. I simply got the basics for the above example from browsing around/searching for examples.

backi
Posts: 1922
Joined: Sun 27 Feb 2011, 22:00
Location: GERMANY

#12 Post by backi »

Me thinks..... you Captain Kirk did press the Hyper-Drive Button ......or are you even in Warp Mode ?

Oh... where are ? .....Ground Control to Major Tom ......

Seems " Major Elvis Kirk Piccard Tom " has left the building . ooohps... i meant this three Dimensional Universe .

" And If I don't meet you no more in this world ........I'll meet ya on the next one .... don't be late...
Don't be late "

Jimi Hendrix ....Voodoo Child

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#13 Post by rufwoof »

Time warped ... and back again. Seen the future, where sfs's are less necessary, storage and transfer rates much quicker. Where even GB's are lost in available ram space :)

Got-to-go, need to see a doctor ... whatshisname - forget, I remember the place though, near Gallifrey.
" And If I don't meet you no more in this world ........I'll meet ya on the next one .... don't be late...
Don't be late "

Jimi Hendrix ....Voodoo Child

backi
Posts: 1922
Joined: Sun 27 Feb 2011, 22:00
Location: GERMANY

#14 Post by backi »

Wwooowww ........ :roll: :roll: :roll:

Let me call it Cyber-Shamanism .

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

.dir .ext loading at bootup

#15 Post by rufwoof »

There's a undocumented feature that Toni (Saintless) noted by inspection of /bin/boot/9990-misk-helpers.sh i.e. within initrd) where if you create a folder with a .dir suffix in /lib/live/mount/persistence/sdf1/live (replacing sdf1 with whatever your setup uses), then at bootup that folder will be 'mounted' under /

Similar to loading a sfs, except a conventional directory content layered in.

Any changes made to the / version of that, for example if I create /lib/live/mount/persistence/sdf1/live/test.dir/abc.txt and reboot then that will appear as /test.dir/abc.txt but changes wont be preserved unless you save the session (flush2disk, second icon in the panel). Or you could edit it directly as /lib/live/mount/persistence/sdf1/live/test.dir/abc.txt so that changes are preserved even without saving the session.

The other undocumented cases is that it will also load .ext2, .ext3 or .ext4 i.e. complete filesystems.

Unlike sfs's that have a maximum of 7 coded in by Debian, you can load as many .ext or .dir as you like. I haven't really experimented with that but looks like you might be able to create something similar to read/writable sfs type 'things' to load at bootup (at least in the case of .dir)

A observation however is that whilst if a filesystem.module file exists in the live folder then only squashfs files listed in that file are loaded at bootup, in the order specified in that file; If filesystem.module doesn't exist then all .squashfs files in the live folder are loaded at bootup in alphanumeric sorted order; Even if a filesystem.module exists however and doesn't mention a .dir folder, then that .dir folder still gets loaded at bootup.

Just as a reminder, you can create file filesystems using something like
dd if=/dev/zero of=somefile.ext4 bs=1M count=1024
mkfs.ext4 somefile.ext4

and then mount it using something like
mkdir mountpoint
mount -o loop somefile.ext4 mountpoint

and don't forget to umount it when done
umount mountpoint

In addition to .dir and .ext2/3/4 you can apparently also use .jffs2 (which I know nothing about). i.e. /bin/boot/9990-misk-helpers.sh 'looks for' *.squashfs|*.ext2|*.ext3|*.ext4|*.jffs2 when scanning for things to 'load'.

Being undocumented, a concern is however that Debian could upgrade the kernel/initrd in a manner that excludes that code.
Last edited by rufwoof on Sat 01 Oct 2016, 19:04, edited 1 time in total.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

USB Portability

#16 Post by rufwoof »

If you use the USB in more than one PC/laptop, then it would be wise to delete/rename

/etc/udev/rules.d/70-persistent*
/home/user/.config/autostart/LXRandR autostart (if exists)
and maybe /etc/X11/xorg.conf (if exists)

files. The first is a record of the network connection last used so that a bootup will look/use that if present, which on a different PC might not work leaving you with no network connection.
The next two are graphics records, which again on a different PC likely will need to be different.

By deleting those the system will deduce those files/content for itself at bootup i.e. pick up the correct network and graphics to use.

You could edit/change the /usr/local/bin/flush2disk script to add in those deletions automatically as part of the 'save', so that the system always deduced for itself the network/graphics to use at each bootup.

backi
Posts: 1922
Joined: Sun 27 Feb 2011, 22:00
Location: GERMANY

#17 Post by backi »

Hi ruf (also known as the famous leader of -The Intergalactic Research Orchestra -) .
For me , being just an ordinary Earthling ,i`ll need some more incarnations to grok it .

So if i won`t grok it no more in this world , i will grok it in the next one .........maybe...

"It`s all too much "

https://www.youtube.com/watch?v=E7pgawaKtpc

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

read write sfs

#18 Post by rufwoof »

rufwoof wrote:...In addition to .dir and .ext2/3/4 you can apparently also use .jffs2 (which I know nothing about). i.e. /bin/boot/9990-misk-helpers.sh 'looks for' *.squashfs|*.ext2|*.ext3|*.ext4|*.jffs2 when scanning for things to 'load'.
Mounting a .jffs2 file filesystem looks interesting as jffs2 supports compression. So conceptually you might have a file filesystem that's read/write and compressed ... in other words a read/write SFS !!!

You need to
apt-get update
apt-get install mtd-utils
to install mkfs.jffs2

Listing the supported compression methods and my favourite (lzo) is disabled :( [I suspect its relatively easy to enable it though :)]
mkfs.jffs2 -L
lzo priority:80 disabled
zlib priority:60 enabled
rtime priority:50 enabled

I tried making a file filesystem out of /var
cd /var
mkfs.jffs2 -o ../test.img.jffs2
and afterwards it showed that file filesystem size to be
-rw-r--r-- 1 root root 163128016 Oct 1 21:49 test.img.jffs2
compared to pcmanfm showing folder properties for /var of
365,957,064 total size of files (10214 files)
399,179,776 size on disk
... so not bad, better than 50% compression.

I got stuck however mounting it
modprobe loop
losetup /dev/loop1 test.img.jffs2
modprobe block2mtd
modprobe jffs2
mknod /tmp/mtdblock0 b 31 0
echo "/dev/loop1,128KiB" > /sys/module/block2mtd/parameters/block2mtd
mkdir mountpoint
mount -t jffs2 /tmp/mtdblock0 ./mountpoint

... produced
mount: /tmp/mtdblock0 is write-protected, mounting read-only
mount: cannot mount /tmp/mtdblock0 read-only

:(

Useful links http://www.linuxcertif.com/man/1/mkfs.jffs2/149744/
http://wiki.emacinc.com/wiki/Mounting_J ... a_Linux_PC

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

usb format for faster writes

#19 Post by rufwoof »

USB's are read quicker than they are written to. This web page provides some guidance as to a good choice of format settings that at least for me helped improve usb write speed. http://linux-howto-guide.blogspot.co.uk ... speed.html

# fdisk -H 224 -S 56 /dev/sdb

Now turn on expert mode with fdisk and force the partition to begin on 128k alignment. In my case I have set new beginning of data to 256. Create as many partitions as you need (I created only one - /dev/sdb1).
Do not forget to save changes and write new layout to flash drive (all data on the flash disk will be lost)
Now it's time to create the filesystem. I used ext4 because there is a way to tell it to specify a strip width to keep your filesystem aligned:

# mke2fs -t ext4 -E stripe-width=32 -m 0 /dev/sdb1

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#20 Post by rufwoof »

Experimented with a usb stick formatted to btrfs that supports lzo compression.

Installed btrfs-tools from the repository (so gparted has mkfs.btrfs format activated), installed jessie-usb files and booted via chained grub4dos to grub2 (small ext2 partition created on the USB for grub4dos as that can't directly access btrfs whereas grub2 can).

I had been hopeful that compression/decompression overhead at the benefit of around half the IO (compression) might be a improvement for read and in particular write (being slower than read on USB's) speeds. Didn't however see any apparent improvement, and some jutter where things would be quick then stall then quick and stall again.... type interaction.

Once the btrfs partition is created and formatted you need to mount it with the compression set

mkdir /mnt/usb
mount -o compress-force=lzo /dev/sdf2 /mnt/usb

otherwise files wont be compressed, just stored as-is when written. Apparently running defrag of the partition is one means to convert any non compressed files in the partition to being compressed (never tested that aspect myself however, as I just wrote all files in after setting compression active).

Post Reply