Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 17 Oct 2019, 04:00
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
FirstRib default WeeDog Linux build system
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 46 of 47 [701 Posts]   Goto page: Previous 1, 2, 3, ..., 44, 45, 46, 47 Next
Author Message
rufwoof


Joined: 24 Feb 2014
Posts: 3546

PostPosted: Sat 05 Oct 2019, 14:01    Post subject:  

rufwoof wrote:
I also prefer mc (midnight commander) for file management over rox, but I do use rox (and geany) at times. mc is available through F1 (tilda), and where typically that is the first thing I load up in one of tilda's tabs (multiple terminals).

Dumped rox altogether. Was only using as for its large icon/display thumbnails image display option, but as thumbnails can't be set on by default, has to be configured for directories I've ejected it and installed gthumb for image viewing.

In mc, F3 or clicking view on a image file fires up gthumb. F4/Edit launches mtpaint. As does just pressing Enter.

No rox pinboard (no desktop icons) and that's nicer for playing mp4's directly to the root window (mplayer -rootwin <mp4file.mp4>) ... nice 'animated' wallpaper Smile

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sat 05 Oct 2019, 21:11    Post subject:  

rufwoof wrote:
Here's the core part (snippet) of my /etc/rc.shutdown for saving (or not)
I'm just maintaining a single 02changes.sfs now, and whenever save is selected that is renamed to changes.sfs.bak (renaming it to 02changes.sfs.bak and it will get loaded into ram if copy2ram is used at the next bootup !!!). That renaming automatically revises its active loop, so after renaming it (mv) we have to use the new name when layering it with upper_changes i.e. so that all of the current 02changes.sfs content is layered beneath upper_changes content and where the new 02changes.sfs is created from that merged pair. I boot with changes=RAM, so upper_changes is mounted to /mnt/layers/RAM/upper_changes.
...
if 02changes.sfs doesn't already exits then it just runs mksquashfs against upper_changes alone to create the new 02changes.sfs.


I'll have to get round to trying out some of your save scheme rufwoof. It looks very interesting and useful. I'm not able to do much on computer for a few days because my partner is on abroad business trip and I'm entertaining the kids and up late in conversations with partner, so basically too occupied. But please keep reporting your alternative refined save mechanisms with code so I can easily try them out when time permits.

Anything user-ready you want me to put link to in User Contributions post, please just let me know and I'll do that any time.

Much appreciated.

Cheers, wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3546

PostPosted: Sat 05 Oct 2019, 21:28    Post subject:  

Good luck with the parenting alone experience wiak.

When it comes to user ready scripting ... not me. I can read fluently, but when it comes to writing I'm dyslexic and admire/envy those like yourself who can write fluently.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sat 05 Oct 2019, 21:49    Post subject:  

rufwoof wrote:
Good luck with the parenting alone experience wiak.

When it comes to user ready scripting ... not me. I can read fluently, but when it comes to writing I'm dyslexic and admire/envy those like yourself who can write fluently.


Ok rufwoof, thanks, I'll take time later to examine the snippets you've posted and see what we can do. Always open for interesting ideas.

cheers, wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
seaside

Joined: 11 Apr 2007
Posts: 935

PostPosted: Sun 06 Oct 2019, 14:19    Post subject:  

rufwoof wrote:
Here's the core part (snippet) of my /etc/rc.shutdown for saving (or not)
I'm just maintaining a single 02changes.sfs now, and whenever save is selected that is renamed to changes.sfs.bak (renaming it to 02changes.sfs.bak and it will get loaded into ram if copy2ram is used at the next bootup !!!). That renaming automatically revises its active loop, so after renaming it (mv) we have to use the new name when layering it with upper_changes i.e. so that all of the current 02changes.sfs content is layered beneath upper_changes content and where the new 02changes.sfs is created from that merged pair. I boot with changes=RAM, so upper_changes is mounted to /mnt/layers/RAM/upper_changes.
Code:
.
.....
        mv 02changes.sfs changes.sfs.bak
        # renaming the sfs on disk has losetup auto renamed as well so changes.sfs.bak now.
        CL=`losetup | grep changes.sfs.bak | awk '{print $1}'`
        mkdir /mnt/CL
        mkdir /mnt/TOP
        mount -t squashfs -o loop $CL /mnt/CL
        mount -t overlay overlay -o lowerdir=/mnt/layers/RAM/upper_changes:/mnt/CL /mnt/TOP
        sync
        mksquashfs /mnt/TOP 02changes.sfs -comp lz4 -Xhc
.
.

if 02changes.sfs doesn't already exits then it just runs mksquashfs against upper_changes alone to create the new 02changes.sfs.


rufwoof.

Ah yes. I was running up "NNchanges.sfs" files and when It came to around 30, I said to myself "why am I doing this" and "wouldn't it be better to do a "puppy-like" single save file?

And you've nailed it here, thanks.

I haven't had a chance to actually try your code and one item that caught my eye was "# renaming the sfs on disk has losetup auto renamed as well so changes.sfs.bak now."

So the actual mountpoint got automatically changed when the filename was changed?

Cheers and thanks for your insights,
s
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sun 06 Oct 2019, 16:05    Post subject:  

rufwoof wrote:
I boot with changes=RAM, so upper_changes is mounted to /mnt/layers/RAM/upper_changes.
Yes I also prefer to boot with changes=RAM since I don't always want to save what I'm doing (in fact I often don't). So that for me is the mode I'd use for saving out the, in that case, RAM-based upper_changes. I've been too busy myself on the core build script functionality to try saving out these changes. But I have glanced at the various mechanisms rufwoof has been using, and they all look good to me. So thus far that includes a general utility for simply merging together numbered upper_changes layers, or a simple compressed tar archive which gets untarred back in again (different, but a bit like tinycore Linux save archive), or rufwoof's latest .bak file idea, which is quite novel by the look of it.
Sorry, half asleep just now and typing on android phone so probably typos or autocorrect errors.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3546

PostPosted: Sun 06 Oct 2019, 17:54    Post subject:  

This is the full version of my /etc/rc.shutdown code. Note how I use lsblk to list available partitions as a aid to identifying the usb, or record the lsblk list before, then prompt to attach a usb, and then lsblk again and pick out the differences as a means to 'automatically' identify the usb. That's because device names are dynamic (random) i.e. a usb on /dev/sdb1 one time might be /dev/sdc1 the next.
Code:
# Default rc.shutdown for void; add your custom commands here.
#
# This is run by runit in stage 3 after the services are stopped
# (see /etc/runit/3).
# Default rc.shutdown for void; add your custom commands here.
#
# This is run by runit in stage 3 after the services are stopped
# (see /etc/runit/3).

clear
echo -n "Do you want to save changes y/n "
read -n1 n
echo
if [ "$n" == "y" ]; then
    bootfrom=$(grep bootfrom /proc/cmdline | awk -F= '{print $2}' | awk '{print $1'})
    cd $bootfrom
    if [ -f 02changes.sfs ]; then
   if [ -f changes.sfs.bak ]; then
       rm changes.sfs.bak
   fi
   mv 02changes.sfs changes.sfs.bak
   # renaming the sfs on disk has losetup auto renamed as well so changes.sfs.bak now.
   CL=`losetup | grep changes.sfs.bak | awk '{print $1}'`
   mkdir /mnt/CL
   mkdir /mnt/TOP
   mount -t squashfs -o loop $CL /mnt/CL
   mount -t overlay overlay -o lowerdir=/mnt/layers/RAM/upper_changes:/mnt/CL /mnt/TOP
   sync
   mksquashfs /mnt/TOP 02changes.sfs -comp lz4 -Xhc
   sync
   umount -lf /mnt/TOP
   umount /mnt/CL
   sync
   rmdir /mnt/TOP
   rmdir /mnt/CL
    else # no 02changes.sfs being used so we don't have to merge upper_changes with that
   mksquashfs /mnt/layers/RAM/upper_changes 02changes.sfs -comp lz4 -Xhc
    fi
    clear
    while :; do
   echo Here is a list of devices
   lsblk -l | grep part | grep -Ev loop | awk '{print $1}' >/tmp/l1
   cat /tmp/l1
   echo -n "Enter the usb device name i.e sdb1 or attach the usb now and press Enter "
   read D
   if [ -z "$D" ]; then
       sleep 1
       lsblk -l | grep part | grep -Ev loop | awk '{print $1}' >/tmp/l2
       D=$(diff --side-by-side --suppress-common-lines \
             /tmp/l1 /tmp/l2 | awk -F'>' '{print $2}' | sed 's/ //g' | sed $'s/\t//g')
   fi
   [ -f /tmp/l1 ] && rm /tmp/l1
   [ -f /tmp/l1 ] && rm /tmp/l2

   # validate
   [ ! -d /mnt/$D ] && mkdir /mnt/$D
   mount /dev/$D /mnt/$D >/dev/null 2>&1
   if [ -f /mnt/$D/VOID6/initramfs05.gz ]; then
       break
   else
       rmdir /mnt/$D
       clear
       echo "That doesn't look like a valid boot (save) usb ... trying again"
   fi
    done
    # update [md5sum filesize] recording on usb for intrusion detection
    store02md5 $bootfrom /mnt/$D/VOID6/initramfs05.gz
fi


store02md5 looks like the following. Basically when a new save is made (new 02changes.sfs created) it just writes the concatenated md5sum and filesize for 02changes.sfs to the usb's initramfs (inside), so my bootup process (rdsh1.plug) can measure the current md5sum and filesize checksum for the hdd stored 02changes.sfs and compare it with the one recorded inside initramfs (/02.md5) and flag if that differs (potential intrusion detected). I also have a store01md5 that is very similar, but records the 01firstrib_rootfs.sfs and rdsh1.plug files md5sum/filesize(s) for checking at bootup.

You have to store/compare both the md5sum and filesize as otherwise its relatively trivial to just add on bytes at the end of a modified file until its md5sum is the same as the original files md5sum.
Code:
#!/bin/bash

# $1 where hdd 02changes.sfs is stored i.e. /mnt/sda1/VOID6
# $2 pre mounted location of usb's initramfs05.gz i.e. /mnt/sdb1/VOID6/initramfs05.sfs

_usage () {
    echo Requires two parameters
    echo "$1 where hdd 02changes.sfs is stored i.e. /mnt/sda1/VOID6"
    echo "$2 pre mounted location of usb's initramfs05.gz i.e. /mnt/sdb1/VOID6/initramfs05.gz"
    exit
}

if [ -z $2 ]; then
   _usage
fi

if [ ! -f $2 ]; then
    _usage
fi

if [ ! -f $1/02changes.sfs ]; then
    echo $1/02changes.sfs not found
    _usage
fi

CD=`pwd`
cd /tmp
mkdir extracted
cd extracted
echo Extracting usb initramfs
zcat $2 | cpio -id
cd $1
echo Recording md5sum
# use the same method as in initramfs (rdsh1.plug)
M=`busybox md5sum 02changes.sfs`
S=`busybox stat -c %s 02changes.sfs`
echo -n "$M $S" > /tmp/extracted/02.md5
# { md5sum 02changes.sfs; stat --printf="%s" 02changes.sfs; } | tr "\n" " " >/tmp/extracted/02.md5
cd /tmp/extracted
echo Recreating the initramfs
find . | cpio -o -H newc | gzip >$2
cd ..
echo "Syncing (flushing)"
sync
rm -rf extracted
echo "Done. Don't forget to umount and unplug the usb"
cd $CD


This is pretty much my rdsh1.plug content, except I use a ansi art voidlinu logo in the logon function

Code:
_logo () {
   echo voidlinux
}

SFSLOC=`env | grep bootfrom | awk -F= '{print $2}'`
CD=`pwd`
cd $SFSLOC

_logo
echo "Running Intrusion Detection checks ... please wait "
if [ ! -f 01firstrib_rootfs.sfs ]; then
    echo
    echo No 01firstrib_rootfs.sfs file found
    echo -n "Press Enter "
    read n
else # Don't bother wasting time if there's no recorded md5
    if [ -f /01.md5 ]; then
   M=`md5sum 01firstrib_rootfs.sfs`
   S=`stat -c %s 01firstrib_rootfs.sfs`
   echo -n "$M $S" > /tmp/01.md5
    fi
fi
if [ -f 02changes.sfs ]; then
    if [ -f /02.md5 ]; then
   M=`md5sum 02changes.sfs`
   S=`stat -c %s 02changes.sfs`
   echo -n "$M $S" > /tmp/02.md5
    fi
fi
if [ -f rdsh1.plug ]; then
    if [ -f /rdsh1.plug.md5 ]; then
   M=`md5sum rdsh1.plug`
   S=`stat -c %s rdsh1.plug`
   echo -n "$M $S" > /tmp/rdsh1.plug.md5
    fi
fi

_logo
if [ -f /01.md5 ] && [ -f 01firstrib_rootfs.sfs ]; then
    D01=`diff /01.md5 /tmp/01.md5`
    if [ ! -z "$D01" ]; then
   echo "01firstrib_rootfs.sfs md5 checksum or size does not match"
   echo "That could mean the sfs on HDD may have been tampered with"
   echo -n "Power down and investigate, or Press Enter to continue "
   read n
    else
   echo "01firstrib_rootfs.sfs validated as OK"
    fi
else
    echo "No checksum recording found for 01firstrib_rootfs.sfs"
    echo "Unable to validate, integrity checks not fully operational"
fi
if [ -f /02.md5 ] && [ -f 02changes.sfs ]; then
    D01=`diff /02.md5 /tmp/02.md5`
    if [ ! -z "$D01" ]; then
   echo "02changes.sfs md5 checksum or size does not match"
   echo "That could mean the sfs on HDD may have been tampered with"
   echo -n "Power down and investigate, or Press Enter to continue "
   read n
    else
   echo "02changes.sfs validated as OK"
    fi
else
    if [ -f 02changes.sfs ]; then
   echo "No checksum recording found for 02changes.sfs"
   echo "Unable to validate, integrity checks not fully operational"
    fi
fi

if [ ! -f /01.md5 ]; then
    _logo
    echo "No pre-recorded checksum for 01firstrib_rootfs.sfs"
fi

# check our rdsh1.plug hasn't been tampered with
if [ -f /rdsh1.plug.md5 ] && [ -f rdsh1.plug ]; then
    D01=`diff /rdsh1.plug.md5 /tmp/rdsh1.plug.md5`
    if [ ! -z "$D01" ]; then
   echo "rdsh1.plug md5 checksum or size does not match"
   echo "That could mean the sfs on HDD may have been tampered with"
   echo -n "Power down and investigate, or Press Enter to continue "
   read n
    else
   echo "rdsh1.plug validated as OK"
    fi
else
    if [ -f rdsh1.plug ]; then
   echo "No checksum recording found for rdsh1.plug.sfs"
   echo "Unable to validate, integrity checks not fully operational"
    fi
fi

if [ -f /tmp/rdsh1.plug.md5 ]; then
    rm /tmp/rdsh1.plug.md5
fi
if [ -f /tmp/01.md5 ]; then
   rm /tmp/01.md5
fi
if [ -f /tmp/02.md5 ]; then
   rm /tmp/02.md5
fi
cd $CD
echo
echo -n "Remove USB and Press Enter "
read n
clear


I boot from usb, so mbr, bootloader, vmlinux, initramfs are all on usb, and part way through bootup it prompts to remove the usb. 01firstrib_rootfs.sfs, 02changes.sfs and rdsh1.plug are on hdd, not copied into ram just run from where they are, so only what is actually needed actually gets read (and typically stored in cache), using lz4 decompression (which is very fast). And where those files are validated (md5sum/filesize as above) against the 'key' (recording of md5sum and filesize) stored on the usb.

Really I should move 02changes.sfs up to a higher number filename in case of wanting to load sfs's below that at bootup, perhaps rename it 10changes.sfs instead.

Another benefit of having changes=RAM is that any changes actually made are lost at shutdown/reboot. If upper_changes (changes) were stored on disk then even if deleted/not saved at shutdown, they're still there, just flagged as 'available' blocks ... that could be read (all your browser activities/cache, opened documents content ...etc.). I also load a swap file at bootup, but use encryption for that as well ... for similar reason. Its encrypted with a random session key at each bootup that not even I know, so the content is in effect unreadable without that key.

Presently I set it to use a encrypted swap that is 50% of the size of memory as typically when it does get used it tends to use relatively little unless its a runaway process in which case all of it will tend to get used no matter how much is available. I invoke it in /etc/rc.local with a line of /usr/local/bin/swap-encrypted.sh sda1 (i.e. the script is called swap-encrypted.sh and is stored in /usr/local/bin)
Code:
# Requires xbps-install cryptsetup

T=`swapon -s | grep partition | cut -d' ' -f1`
if [ ! -z $T ]; then
   clear
   echo "Looks like you already have a swap partition activated"
   sleep 3
   exit
fi

# swap file device partition i.e. sdc1 passed on kernel boot cmd line
SF=`cat /proc/cmdline | awk -F'swappartition=' '{print $2}' | awk '{print $1}'`
if [ -z ${SF} ]; then  # if no kernel boot parameter for swappartition= then
   SF=$1          # maybe being run as a script and passed $1 argument
fi
if [ ! -z ${SF} ]; then
   if [ ! -d /mnt/${SF} ]; then
          mkdir -p /mnt/${SF}
   fi
   mount /dev/${SF} /mnt/${SF}
   C=$?
   if [ $C -eq 0 ] || [ $C -eq 32 ]; then  # 32 if already mounted
      cd /mnt/${SF}
      if [ ! -f swapfile.crypt ]; then
         MEM=$(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE) / (1024 * 2048)))
            echo "creating $MEM MB swapfile /mnt/${SF}/swapfile.crypt ... please wait"
            echo "Once created once it can be reused in later sessions"
            dd if=/dev/urandom of=swapfile.crypt bs=1024k count=$MEM
      fi
      # Reuse prior loop if had already been created
      loop=`losetup | grep swapfile.crypt | cut -d' ' -f1 | head -1`
      if [ -z $loop ]; then   # otherwise add new loop
          loop=$(losetup -f)
          losetup ${loop} swapfile.crypt
      fi
      cryptsetup open --type plain --key-file /dev/urandom ${loop} swapfile
      mkswap /dev/mapper/swapfile
      swapon -f /dev/mapper/swapfile
   else
      echo "Error locating device /dev/${SF} and mounting to /mnt/${SF}"
      echo "Either not specified as a kernel boot parameter i.e. swappartition=sda1"
      echo "or this script run without a parameter i.e. $0 sda1"
      echo "or in mc, highlight the script and run the mc command line"
      echo "sh %f sda1 (or whatever alternative location to sda1)"
      sleep 4
   fi
else
   clear
   echo "no swappartition=sda1 (or similar) swap partition kernel boot parameter"
   echo "or no sda1 (or similar) command line parameter for swap"
   echo
   echo "If you're running this script from within mc then highlight it and run"
   echo "if from the mc command line using something like ... sh %f sda1"
   sleep 5
fi


If you store data outside of the OS region then there's little need to use 'save'. And when you do want to reconfigure things just boot, reconfigure and save - so the changes are applied and stored from a known 'clean' version. Otherwise just boot, use, shutdown without saving.

I don't save bookmarks in the browser as the above way means a clean browser after each reboot. Instead I have a text file in data space that I load up in a tilda terminal window that is available via F1 (that toggles show/hide of tilda) and where tilda supports clickable links so text of http://murga-linux.com/puppy in that text file for instance will underline when the mouse is hovered over it, and activate chromium to that link if clicked. A little annoying to go to a regular web site and have it pop up with all the first time visitor popups, but after a while you just accept that and click the OK to accept the terms/conditions.

It's way better to be booting a known clean system each and every time than it is to try and keep nasty stuff out 24/7 and where even just a single slip at some point in the past might have seen the system turn from clean to dirty. Yet as the above scripts outline, its relatively simple to set up a reasonably secure setup/system (data and online security are separate matters).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3546

PostPosted: Sun 06 Oct 2019, 18:05    Post subject:  

As per the other day, I guess if rdsh1.plug, main sfs and changes sfs can all be stored/run-from a ntfs folder, then so also might a encrypted swap file. If you're also booting from usb then a Windows HDD might remain pretty much as-is (best to just create a folder such as c:\voidlinux and store the sfs's, .plug and swapfile in that directory). Other than that, just need to get the Windows box to boot from usb before hdd so whenever the usb is attached at bootup (or selected) it boots into voidlinux instead of Windows. Whilst the intrusion detection mechanism ensures that even if the Windows box is infected, then the linux boot will detect anything 'unusual'.

When I build a voidlinux setup I record copies of the build scripts within that (including the firstrib plug (build script)) i.e. on usb, so worse come to worse, 20 minutes of rebuild time or whatever using those scripts could have the system back up and running again as before.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3546

PostPosted: Sun 06 Oct 2019, 20:38    Post subject:  

md5sum is a lag factor when the file being checked is large. So on my two core laptop I did some timing trials of using straight md5sum compared to splitting it into two halves (two md5sum's) running concurrently and each assigned to its own core (nice ... timeset). A crude test with some overlap of the two halves. Overall the comparison of straight md5sum was much the same as the split based execution timing Sad
Code:
#!/bin/sh

# Might overlap slightly, but I'm looking at creating two md5sum's instead of
# just one as potentially quicker for applying to large files

SZ=`wc -c $1 | awk '{print $1}'`
echo size is $SZ
HALF=`expr $SZ / 2`
echo half is $HALF

nice -n -20 taskset 1 dd if=$1 bs=$HALF count=1 | md5sum &
nice -n -20 taskset 2 dd if=$1 bs=$HALF skip=1 | md5sum &

wait # wait for both background tasks to complete

Busybox in initramfs does have nice, but I don't know whether it actually supports taskset (binding to a particular cpu core) anyway, so even had it been quicker in user space not so sure it would have worked in initramfs.

As a reminder taskset is a bitwise operator 2 = 10 binary = core 1, 1 = 01 binary = core 0 (3 = 11 binary = both core 0 and 1 ..etc.). Nice -n -20 is allocate the highest priority possible (negative 20) ...down to least favourable for the process of 19 (positive 19).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sun 06 Oct 2019, 22:44    Post subject:  

I've had trouble with 'nice' command in the pass not actually changing process priority, but so long ago I can't remember the details. Something to do with Linux scheduler not using it. I'll have to google and see if I can remember why it didn't work the way its manage suggested. Or maybe I'm dreaming that issue since distant past attempts anyway.

I meant to look into that USB random partition boot matter. I've never had that on my machine but had meant to use a flag file for safety and had thought of the code but then forgot to implement.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Mon 07 Oct 2019, 03:07    Post subject:  

rufwoof wrote:
This is the full version of my /etc/rc.shutdown code.
I put link to your post from the User Contributed HowTo's thread (I put it under the Utilities section, thanks):

http://www.murga-linux.com/puppy/viewtopic.php?p=1029029#1029029

I've promised myself to try some save changes on exit mechanisms soon as I have some time back to see what will work well for me. I probably won't have any complex system (e.g. I don't use encrypted swap or filesystems) but will experiment and try out some parts of your variations too rufwoof.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
rockedge


Joined: 11 Apr 2012
Posts: 1262
Location: Connecticut, United States

PostPosted: Wed 09 Oct 2019, 18:16    Post subject:  

I am wondering how the kernel upgrades are used in WeeDog.
with xbps-install -Su the updates roll along but I am never using the newer upgraded version it seems.

Do I have to re-run the build_weedog_initramfs05_s205.sh script and build a new vmlinuz ?

I've run
Code:
vkpurge  list
vkpurge rm all

and have installed xtools and looked at
Code:
xcheckrestart list


_
Back to top
View user's profile Send private message Visit poster's website 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Wed 09 Oct 2019, 22:33    Post subject:  

rockedge wrote:
I am wondering how the kernel upgrades are used in WeeDog.
_


Good question. The thought had crossed my mind too.
Yes, you certainly need to get the appropriate modules into 01firstrib_rootfs.sfs and also to rebuild the initramfs05.gz so these modules are also available in there (unless using 00firstrib_firmware_modules.sfs arrangement, in which case you'd want to rebuild that, albeit only with modules inside, not firmware - the sfs name was used because previoulsy used for Bionic zdrv firmware_modules, but with Void it only needs to contain modules if using it). However, you won't lose any of your previous work via the above procedures since that remains untouched.

EDIT: At the moment what I'd do is (untested right now):

EDIT2: Currently working on a script and testing it (current code printed towards end of this post FYI but, as I say, not guaranteed yet since just under test)

EDIT3: yes, this is tricky... There are issues with vkpurge - doesn't really work as expected in chroot, and though you 'could' upgrade kernel (new one ending up, for example, in upper_changes), the initramfs would also have to be upgraded in sync with that (since needs new kernel modules); I'm continuing to look into this less-than-trivial issue. Below is not a complete answer at all I think.

After CHECKING you really want to proceed (especially since about to delete an existing initramfs05.gz)... remove any existing:

firstrib_rootfs_for_initramfs_sNNN
uncompressed firstrib_rootfs
initramfs05.gz
vmlinuzNNNNN

Though I'd suggest making backups first, either manually, or using code such as:

Code:
[ -d squashfs-root ] && rm -rf squashfs-root
[ -d firstrib_rootfs_for_initramfs_sNNN ] && mv firstrib_rootfs_for_initramfs_sNNN BKUPfirstrib_rootfs_for_initramfs_sNNN
[ -d firstrib_rootfs ] && mv firstrib_rootfs BKUPfirstrib_rootfs
[ -f 01firstrib_rootfs.sfs ] && mv 01firstrib_rootfs.sfs BKUP01firstrib_rootfs.sfs
[ -f initramfs05.gz ] && mv initramfs05.gz BKUPinitramfs05.gz


I suggest putting BKUP at front of name as above, rather than at end, so has effect also of NNlayer disabled. Also alphabetically groups all the backups together in file list.

Then:

1. unsquash the 01filesystem.sfs (or its backup) and make sure the resultant squashfs-root folder is renamed to firstrib_rootfs. That is:

Code:
unsquashfs BKUP01firstrib_rootfs.sfs
mv squashfs-root firstrib_rootfs


EDIT NOTE WELL: Don't do WRONG 2 and WRONG 3 immediately below... you instead need to remove the current kernel using vkpurge, inside a chroot, as I detail later (just done a quick try)
WRONG: 2. rm -rf firstrib_rootfs/usr/lib/modules # DON'T DO THIS
WRONG: 3. rm -rf firstrib_rootfs/boot/vmlinuz* # DON'T DO THIS

CORRECT (I think... but still testing...):

2.
Code:
./mount_chrootver007.sh

to chroot into firstrib_rootfs, and from there:

3. Now you could check current kernel(s) installed using, say (-l is small ell). However, since going to use vkpurge in next step, this check is probably unnecessary in practice:

Code:
xbps-query -l | grep linux


EDIT3: Better is:

Code:
xbps-query --regex -s '^linux[0-9.]+-[0-9._]+'


You need to examine the resulting list for the kernel name (you can try the meta-package/template name but maybe need main kernel version name; I tested but forget).

and then (untested but I think it is correct - thanks rockedge):

Code:
vkpurge rm all

(not sure if you also need to remove previous firmware at this stage - I expect you don't need to).

4. followed by installing, for example, the current latest kernel with:

Code:
xbps-install -Suy linux

which should install the new kernel and its related modules
Follow by:
5.
Code:
exit
, and then
Code:
./umount_chrootver007.sh

to clean up the chroot mounts, and lastly:
6.
Code:
./build_weedog_initramfs05_XXX.sh void [options]

to rebuild the initramfs05.gz with the new modules inside it, which also outputs the new kernel for booting via grub menu.lst frugal arrangement, and a new 01firstrib_rootfs.sfs as usual

I guess you could write a simple script to automate the above (using for example a cat heredocument, similar to that used in build firstrib rootfs, to automate the xbps-install -Suy linux part done inside the chroot). Easy enough to do above manually though, and sometimes its best not to over-automate in case something odd happens...

I'm currently trying the following script - I'll post later if successful

EDIT: I think the following needs more work... There are issues with vkpurge I briefly mention in following posts. vkpurge is a shell script in /usr/bin that was designed to be used on running Void system (so not from say a chroot running on other Linux host).

Code:
#!/bin/sh
# rebuild_weedog_void_frugal_001 with current kernel/modules
# Revision 0.0.1 Date: 10 Oct 2019
# Copyright wiak 10 Oct 2019+; Licence MIT (aka X11 license)

#### variables-used-in-script:
# script commandline arguments $1 $2 $3 $4
# "$1" is distro (e.g. void); "$2" is optional mksquashfs compression (or "default");
# "$3" is optional "huge" initramfs (or "default); $4 is optional busybox url to use

kernel="$1"
case "$1" in
   '-v'|'--version') echo "Rebuild WeeDog Void Frugal Revision 0.0.1";exit;;
   '-h'|'--help'|'-?') printf '
Usage:
./rebuild_weedog_void_frugal_001.sh void [OPTIONS]

-v --version       display version information and exit
-h --help -?       display this help and exit

First argument void is required to undertake rebuild
Optional second argument is mksquashfs compression (or "default")
Optional third argument is "huge" (or "default") initramfs
"huge" includes 01firstrib_rootfs.sfs inside initramfs/boot/initramfsNN
Optional fourth argument is busybox_url (in case, e.g. arm64 required)
fourth argument can optionally also be "default"

EXAMPLES:
./rebuild_weedog_void_frugal_NNN.sh void
./rebuild_weedog_void_frugal_NNN.sh void "-comp lz4 -Xhc"
./rebuild_weedog_void_frugal_NNN.sh void default huge

For more details read the attached README and/or
visit: https://github.com/firstrib/firstrib
';exit;;
   '') echo 'First argument must be distro name: void';exit;;
esac

[ -d squashfs-root ] && rm -rf squashfs-root
[ -d firstrib_rootfs_for_initramfs_sNNN ] && mv firstrib_rootfs_for_initramfs_sNNN BKUPfirstrib_rootfs_for_initramfs_sNNN
[ -d firstrib_rootfs ] && mv firstrib_rootfs BKUPfirstrib_rootfs
[ -f 01firstrib_rootfs.sfs ] && mv 01firstrib_rootfs.sfs BKUP01firstrib_rootfs.sfs
[ -f initramfs05.gz ] && mv initramfs05.gz BKUPinitramfs05.gz
echo 'Unsquashing BKUP01firstrib_rootfs.sfs ... Please wait ...'
unsquashfs BKUP01firstrib_rootfs.sfs
mv squashfs-root firstrib_rootfs

# bind mount host virtual filesystes required for chroot into firstrib_rootfs to work on host system
# as in mount_chrootXXX.sh
mkdir -p firstrib_rootfs/run/udev
mount --bind /proc firstrib_rootfs/proc && mount --bind /sys firstrib_rootfs/sys && mount --bind /dev firstrib_rootfs/dev && mount -t devpts devpts firstrib_rootfs/dev/pts && mount --bind /tmp firstrib_rootfs/tmp && mount --bind /run/udev firstrib_rootfs/run/udev && cp /etc/resolv.conf firstrib_rootfs/etc/resolv.conf

# The following commands gets done inside firstrib_rootfs (via here_document pipe to chroot):
cat << INSIDE_CHROOT | chroot firstrib_rootfs sh
# Remove all old kernel/modules
vkpurge rm all
echo 'Installing current kernel and modules ... Please wait ...'
xbps-install -Suy linux
exit
INSIDE_CHROOT

# Clean up the bind mounts that were used for the chroot
# as in umount_chrootXXX.sh:
umount -l firstrib_rootfs/proc && umount -l firstrib_rootfs/sys && umount -l firstrib_rootfs/dev/pts && umount -l firstrib_rootfs/dev && umount -l firstrib_rootfs/tmp && umount -l firstrib_rootfs/run/udev

printf "\nAt this stage you can manually modify firstrib_rootfs if you wish
Once you are ready to complete the rebuild press Enter to continue
"
read go

# Rebuild initramfs05.gz, 01firstrib_rootfs.sfs and output new vmlinuzXXX
echo "Executing $0 $@ ... Please wait ..."
exec ./build_weedog_initramfs05_s204.sh "$@"
exit

-----------------------------------------------

As far as my current efforts are concerned, Arch pacman reminds me of xbps a bit but pacman static isn't independent (it requires other code bits and pieces to be made available for it) like xbps static so I'm probably going to firstrib include it using archbootstrap or whatever it is called.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130

Last edited by wiak on Sat 12 Oct 2019, 08:30; edited 9 times in total
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sat 12 Oct 2019, 03:53    Post subject:  

@rockedge: You are correct, I should be using vkpurge. When I wrote my immediately above post I hadn't read:

https://docs.voidlinux.org/config/kernel.html

EDIT: I'm now editing my immediately above post with script I'm currently testing. I'll update later (and post script as file) once that script code is ready for use.
EDIT2: However, in my current tests, 'vkpurge list' isn't finding the old kernels I've installed (using xbps-install) for the test, so 'vkpurge rm all' isn't removing them. I'm looking into solution.

Also can check what already installed locally with:

Code:
xbps-query --regex -s '^linux[0-9.]+-[0-9._]+'


I'll see if I can modify my above script accordingly to update kernel automatically.

Also, according to Void Linux wiki, to get a list of all packages installed without version numbers, you can use:

An example command line to get a list of all installed package names without their version is:

https://wiki.voidlinux.org/Frequently_Asked_Questions#How_do_I_get_a_list_of_installed_packages.3F

Code:
xbps-query -l | awk '{ print $2 }' | xargs -n1 xbps-uhelper getpkgname | fmt


wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1808
Location: not Bulgaria

PostPosted: Sat 12 Oct 2019, 06:40    Post subject:  

Apparently, 'vkpurge rm all' safely remove all old kernel/modules that were installed via updates (but leaves the most recent kernel installed). However, it does not remove explicity installed kernels; I guess you need to use xbps-remove linux-NNNetc for that.

I can't say that the Void documentation on vkpurge is particularly clear on all of this.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 46 of 47 [701 Posts]   Goto page: Previous 1, 2, 3, ..., 44, 45, 46, 47 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1469s ][ Queries: 13 (0.0426s) ][ GZIP on ]