Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 15 Dec 2019, 23:29
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
FirstRib default WeeDog Linux build system
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 27 of 49 [722 Posts]   Goto page: Previous 1, 2, 3, ..., 25, 26, 27, 28, 29, ..., 47, 48, 49 Next
Author Message
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Sat 31 Aug 2019, 00:47    Post subject:  

tallboy wrote:
Hi wiak, I'll play with the new tools, but please, please don't use yellow text! Confused


Hi tallboy,

(I'm tempted to write this in bold orange text in case you still have your snow sunglasses on...)

Once you have done a quick default build, by running the two build scripts one after the other (best/being from an empty directory you intend to boot from - be that on hard disk or usb stick doesn't matter) let us know. If you have any trouble getting that to boot, just tell us what files you have and what your grub or grub4dos configuration is (i.e. the menu.lst details you are using). When you are new to a distro, getting it to boot is trickiest on the first occasion - once you have that sorted out it becomes easy thereafter of course. I'd advise you just try to build and then boot the default

Code:
./build_firstrib_rootfs_100.sh void rolling amd64 firstrib00.plug_void_default_anyarch


(or use i386 instead of amd64 for 32bit system build)

Then, once that first stage finished (can take a wee while - have a coffee and be patient), simply run second script with command:

Code:
./build_weedog_initramfs05_s100.sh void


The result (takes a wee while too) should be the following files available that grub uses to boot:

01firstrib_rootfs.sfs, initramfs05.gz, vmlinuzXXXXXXX

These three are all that menu.lst needs to boot (though I normally rename vmlinuzXXXXXXX simply to vmlinuz).

My menu.lst might then be (if say using Linux formatted, say ext2, usb stick at /mnt/sdb1 with the above three files stored in directory /weedog on that):

Code:
title WeeDog Linux
root (hd1,0)
kernel /weedog/vmlinuzX.XX usbwait=12 bootfrom=/mnt/sdb1/weedog
initrd /weedog/initramfs05.gz


EDIT: there is a chance above won't work if system doesn't see the usb stick as /mnt/sdb1, of course, in which case, you best to use usb stick's uuid, which you find with blkid command, instead of using: root (hd1,0) - if you don't know about uuid and blikid we can explain about it later should you have trouble getting WeeDog to boot.

Assuming your grub is working okay, that should boot up fine to the commandline. If so, you are home and dry. Obviously though you then need to learn the commands/method to get on the Internet (I always just use wifi myself but ethernet even easier) and learn how to add new apps (such as X windows - or do a second build later with a firstrib00.plug file by someone else, like rufwoof or rockedge, that has X etc all done for you). Main thing is that at any of these stages we can easily help out since been playing with this for months now so know all the tricks to get it Internet connected and so on (very simple commands to do that actually).

Good thing about using this system, is that Void Linux repos and native package manager (xbps) uses makes it easy to try out lots of desktop managers and so (JWM, or openbox, or MATE, LXDE and so on). And the resultant desktop is amazingly efficient, according to our tests, in terms of RAM and CPU used.

Finally WeeDog initramfs has some nice tricks in terms of save persistence and copy2ram and so on, and more to come - easy to learn and easy to use - worth persevering with I'd say (and not because I'm biased).

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Sat 31 Aug 2019, 08:35    Post subject:  

I made some further progress today, without adding any bloat, just fixes and a couple of facilities discussed on the thread (including option to set tmpfs size and also the rockedge discovered udhcpc default script fix). I haven't quite finished coding the small changes yet, and still have to test them, but expect to publish the new build scripts tomorrow since the changes are so small. As I say, nothing major but a couple of useful new options.

The updated scripts won't however upset any existing firstrib00.plug plugins, existing or being developed; these will work just fine as before.

Once I've released that update I'm hoping to get on with a few actual builds of my own since I've yet to prepare a desktop for my own daily use, and am keen to do so. Just for fun I might make a few ubuntu or debian builds too (extended firstrib00.plug_ubuntu/deb plugins - that should be pretty easy). I also want to play with rufwoof's merge-changes.sh script and maybe make a little make iso script since it might be worth producing a small iso for those who may never enjoy the true pleasure of Void xbps package management otherwise...

I suppose I should try and script produce a stripped down iso (for size) but seems like a lot of effort, though would be useful for copy2ram case on older computers - I'm not sure I will get round to that though since it has low priority for me.

Thereafter, learning package production via xbps-src would probably a good idea.

For anyone new to this who keeps forgetting whether the Void package manager name is xpbs or xbps (or was it just me that had that problem?); I remember the ps comes at the end because it is like the ps command used to display "snapshot of current processes" though xbps really has nothing to do with ps command of course Wink

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
tallboy


Joined: 21 Sep 2010
Posts: 1572
Location: Drøbak, Norway

PostPosted: Sat 31 Aug 2019, 11:12    Post subject:  

Thank you for the guidance, wiak. Very Happy
_________________
True freedom is a live Puppy on a multisession CD/DVD.
Back to top
View user's profile Send private message 
rockedge


Joined: 11 Apr 2012
Posts: 1367
Location: Connecticut, United States

PostPosted: Sat 31 Aug 2019, 11:59    Post subject:  

Quote:
learning package production via xbps-src would probably a good idea.


you and me both! I really would like to manage to make a Void package of zoneminder...once I have the steps / script worked out.

the ZM build script is doing some things that are unexpected occasionally and I am working on that. ZM as a Void package would really compact the set up as well eliminating all the development libraries need to compile
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Sat 31 Aug 2019, 14:27    Post subject:  

wiak wrote:
I suppose I should try and script produce a stripped down iso (for size) but seems like a lot of effort, though would be useful for copy2ram case on older computers - I'm not sure I will get round to that though since it has low priority for me.

In addition to set the default mksquashfs to lzo -Xhc, in build_weedog_initramfs05_s100.sh I modify the code to (snippet) ...

Code:
# Create tmpfs in RAM should grub kernel line request copy2ram or changes=RAM
if [ \$copy2ram -eq 0 ] || [ "\$changes" == "RAM" ]; then
        mkdir -p /mnt/layers/RAM
        mount -o mode=1777,nosuid,nodev,size=100% -n -t tmpfs inram /mnt/layers/RAM # might prefe
r size limit
        # Rufwoof
        swapon /dev/sda3
        mount -o remount,size=13G /mnt/layers/RAM
        # /Rufwoof
fi

where sda3 is my swap partition. So conceptually I could load up to 13GB into ram (plus swap).

Using extreme mksquashfs compression I can get the main sfs down to 1GB, that's with kdenlive, chromium, full libre office suite, audacity ...etc. Not sure how that might fair if loaded into perhaps 1GB of actual ram and say 4GB of swap partition, suspect it would still work, but you'd have to be patient at times.

And of course you could use a swap file as a alternative to a dedicated swap partition. Difficult to test on my 4GB system, as even if I load up a 8GB sfs to load at bootup (copy2ram), the system would tend to rapidly swap out the stuff that wasn't being used and keep only the most commonly used stuff in actual ram.

One aspect of using swap is that shutdown's can be slowed by the swapoff process that's triggered as part of shutdown. If you're running solely in ram however you could just hit the power off button, but that could corrupt swap such that it wasn't loaded at the next reboot. Similarly forced/immediate reboot/power off (that runs without running swapoff) might do similar. I guess however that a startup process to clean swap could be added.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Sat 31 Aug 2019, 14:40    Post subject:  

As I already run all in ram/swap, and as I have 4GB of ram + 10GB of swap, and as my firstrib_rootfs folder content is 3.6GB in total, out of interest I'm making a sfs of that with no compression (-noX -noI -noD -noF mksquashfs parameters). Doing that now.

Conceptually that will be slower than using lz4 -Xhc compression that reduces the sfs filesize down to around 1.4GB, as lz4 runs multi-core when compressing/decompressing and at speeds that can on larger systems (4 cores+) yield throughput that compares to ram speeds (less physical I/O and instead throwing decompressed data out at lighting speeds).

Purely out of interest to see if that will load and run, and if so how well it runs. Will report back later.

EDIT: Well boots ok. After first boot to cli and logging in as root, free -m indicates
Code:
              total        used        free      shared  buff/cache   available
Mem:           3408         169         159        2949        3079         110
Swap:         10239         418        9821

After starting up (which also starts chromium)
Code:
              total        used        free      shared  buff/cache   available
Mem:           3408         381         137        2151        2889         643
Swap:         10239        1235        9004

... and after starting up kdenlive and loading a 1 hour video for editing, as well as also starting LibreOffice calc
Code:
              total        used        free      shared  buff/cache   available
Mem:           3408         867         389        1278        2151        1029
Swap:         10239        2152        8087

Operationally OK, there was a slight jitter initially on playing the kdenlive video, but that rapidly disappeared. That 2GB of swap utilisation would take some time to clear on (swapoff) shutdown but that could be circumvented.

That's with a 01firstrib_rootfs.sfs filesize of
Code:
# ls -lart /mnt/sda1/VOID2/01*
-rw-r--r-- 1 root root 3529768960 Aug 31 19:39 /mnt/sda1/VOID2/01firstrib_rootfs.sfs


Initially I was directly loading that 1 hour video into kdenlive for editing from hdd. After copying that to /root so I could umount the hdd and with kdenlive editing that /root based copy of the video
Code:
# df -h
Filesystem      Size  Used Avail Use% Mounted on
inram            13G  3.8G  9.3G  29% /mnt/layers/RAM
overlay_result   13G  3.8G  9.3G  29% /
run             1.7G  688K  1.7G   1% /run
dev             1.7G  8.0K  1.7G   1% /dev
shm             1.7G   23M  1.7G   2% /dev/shm
cgroup          1.7G     0  1.7G   0% /sys/fs/cgroup
tmpfs           1.7G  369M  1.4G  22% /tmp

where the .mp4 file being edited shows a size of 403746084 bytes.

For reboot I stopped kdenlive, removed the /root copy of the .mp4 file, closed Chromium and Libre Calc and as man reboot indicated
Quote:
-n Don't sync before reboot or halt. Note that the kernel and
storage drivers may still sync.
and in expectation of a long wait for shutdown as swapoff might otherwise be run, I tried that reboot -n option (swap was showing 2.7GB being used at that time) ... and reboot was as good as normal (fast) Smile

Personally I think that ram+swap option, at least for voidlinux style boots is great, as your 'ram' is just limited to the amount of free disk space. Yes on ram challenged systems having too much in swap, too little actual ram would tend to see sluggishness, but conceptually it would still run 'correctly'.

I suspect that loading into ram+swap in initrd doesn't get released and continues to remain available after the switch-root as switch-root only closes any free files (in unix everything is a file), so as swap isn't closed, it continues to remain available/used. Again at least as far as voidlinux is kernel is concerned.

Yes that is somewhat cheating as its like using hdd, but where the 'storage' area is within the swap partition instead of on a ext2/3/4/whatever partition. Would be nice if encrypted swap could be utilised, however I suspect that isn't something that could be passed over from initrd to the main system (at least I couldn't get that to work last time I tried some time back).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rockedge


Joined: 11 Apr 2012
Posts: 1367
Location: Connecticut, United States

PostPosted: Sat 31 Aug 2019, 16:58    Post subject:  

just want to add I borrowed retrovol-0.10-x86_64.pet opened it up with xarchiver and extracted and installed manually. works really well on my WeeDog (Void).

http://www.murga-linux.com/puppy/viewtopic.php?t=69486
2019-08-31-171746_600px.png
 Description   
 Filesize   57.16 KB
 Viewed   482 Time(s)

2019-08-31-171746_600px.png

Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Sun 01 Sep 2019, 05:27    Post subject:  

http://murga-linux.com/puppy/viewtopic.php?p=1035078#1035078 updated to use functions style layout so its easier to comment out whole blocks of code/script such as if the latest version of mc has config files that don't match what is being modified within firstrib00.plug. As part of that I've also extended the script to build some source code. As isomaster isn't in the voidlinux repo and is simple enough to compile that seemed like a good starter candidate and here's the firstrib00.plug code section I've used to do that
Code:
###########################################
# ISOMASTER
# Work in progress ... as a test, download isomaster source

_isomaster () {

  # Requires : xbps-install gcc make gtk+-devel pkg-config gettext

  cd /usr/src
  wget -c http://littlesvr.ca/isomaster/releases/isomaster-1.3.14.tar.bz2
  tar xvfj isomaster-1.3.14.tar.bz2

  # Simple build process that only requires us to run 'make'
  cd isomaster-1.3.14
  make
  make install

}

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Sun 01 Sep 2019, 07:23    Post subject:  

rufwoof wrote:
http://murga-linux.com/puppy/viewtopic.php?p=1035078#1035078 updated to use functions style layout so its easier to comment out whole blocks of code/script such as if the latest version of mc has config files that don't match what is being modified within firstrib00.plug. As part of that I've also extended the script to build some source code.


Interesting plugin. Keep up the good work rufwoof.

WeeDog is creeping up the Distrowatch_murga rankings (a one of snapshot just for fun sample average thread views/day count), despite its short/limited exposure to the general public and taking into consideration its slow start:

http://www.murga-linux.com/puppy/viewtopic.php?p=1035827

wiak

EDIT: I've completed the build script updates (including inram tmpfs size option and udhcpc default.script fix) but still have to do testing after which I'll upload tomorrow or whenever I'm satisfied all seems to be working. I've added plugin source location so you can also mount your swap via suitable code plugin (and I can also provide plugin for zram swap I think). That way you can add instructions/plugin for doing so on your related firstrib00.plug contribution pages if you wish.

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Sun 01 Sep 2019, 18:43    Post subject:  

Void Linux de repo seems to be down at the moment (or maybe too many people building a Firstrib Void root filesystem??? I hope not...), so I am making build rootfs with us repo (via using attached firstrib.repo plugin placed in same folder as build_firstrib_rootfs_NNN.sh script).

EDIT: It's odd, so I'm delaying upload of new versions of build scripts till sorted out. Is anyone else having an issue. In particular, are you able to access repo link (from your browser)? I can't.:

https://alpha.de.repo.voidlinux.org/static

EDIT2: Okay, it's back now

wiak
firstrib.repo.tar
Description  place in build scripts dir to use us repo instead of de one
remove dummy tar
tar

 Download 
Filename  firstrib.repo.tar 
Filesize  42 Bytes 
Downloaded  43 Time(s) 

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Sun 01 Sep 2019, 21:03    Post subject:  

Die hard grub4dos user, today from a voidlinux boot I installed grub. Formatted a usb to ext3 format (single partition) and set its boot flag on. Once applied/saved I mounted it (fdisk -l indicated sdb1 in my case)
mkdir /mnt/usb
mount /dev/sdb1 /mnt/usb
and then ran
grub-install --no-floppy --root-directory=/mnt/usb /dev/sdb
to install grub.
Created a /boot/grub/grub.conf file (/mnt/usb/boot/grub/grub.conf) with content
Code:
set default=0
set timeout=3
menuentry "VOID2" {
      echo    Loading VoidLinux ...
      linux   /VOID2/vmlinuz-4.19.69_1 bootfrom=/mnt/sda1/VOID2 changes=RAM
      echo    Loading initial ramdisk ...
      initrd  /VOID2/initramfs05.gz
  }

and copied the initrd and vmlinuz over to a /VOID2 sub folder on the usb.

That's booting where the MBR, bootloader, vmlinuz and initrd are all on the USB, which can be unplugged once booted to physically isolate those files, and where it uses the main sfs on HDD along with a swap partition also on hdd. Mostly I just boot and use the main read only sfs, storing changes in ram and just losing those changes at shutdown.

Looking around I haven't seen voidlinux using anything like OpenBSD's kernel randomisation so along with fixed MBR and bootloader, those files/data are potential crack targets (crack a session and with root privileges modify each/any of those to 'own' the system). So its nice having the option to physically isolate them (unplug usb) immediately after the system has booted.

I'm using a usb3 sandisk stick (16GB) plugging that into a usb2 port for booting (as I was finding it runs really hot when used in the USB3 port), and it stays nice and cool.

Make a md5 of (part of) the main sfs, storing its value on usb, and immediately after boot you can validate that the main sfs hasn't been tampered with (unlikely anyway).

So far its been booting up nice and quick and even without any usbwait=5 or whatever kernel boot parameter delay command being required.

That did highlight a problem with my save.sh and merge-changes.sh files determination of the boot-from= kernel boot parameter so I changed them to use
Code:
cd `cat /proc/cmdline | awk -F'bootfrom=' '{print $2}' | awk '{print $1}'`
instead (and I've updated the firstrib00.plug in http://murga-linux.com/puppy/viewtopic.php?p=1035078#1035078)
_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Mon 02 Sep 2019, 07:10    Post subject: New/revised build scripts uploaded  

New/revised build scripts uploaded

build_firstrib_rootfs_101.sh and build_weedog_initramfs05_101.sh uploaded to here:

http://www.murga-linux.com/puppy/viewtopic.php?p=1035524#1035524

build_firstrib_rootfs CHANGES:

fix to udhcpc default.script (thanks rockedge).

build_weedog_initramfs CHANGES:

1. can specify busybox_url (as second commandline argument) if you want to install different one (e.g. for arm64 initramfs).

2. new grub kernel line optional parameter: inram_sz=size
where size could be in bytes, M, G or % (of total RAM)
(inram being the tmpfs used by /mnt/layers/RAM when either changes=RAM, or copy2ram being specified)
NOTE WELL: inram (/mnt/layers/RAM) size can be as large as VIRTUAL RAM (i.e. physical RAM + and swap space), so can be more than 100% physical RAM. e.g. With enough swap space (be it zram swap or conventional swap) you could use, say, inram_sz=200% (thanks rufwoof for the solution to that idea)

Optional new plugin: inram00.plug, which, if present in bootfrom directory can be used to mount swap (as rufwoof has done in his modified initramfs/init versions) and/or to activate zram swap.

You should view the script code to see location of where inram00.plug gets loaded (it will happen each time system is booted) (and also the new load positions for rdsh0, rdsh1, rdsh2, rdsh3, rdsh4 and rdsh5). None of these plugins need to be used of course - without them build initramfs behaviour is exactly as previous version.

I have created an inram00.plug I use on my 4GB physical RAM system.

Kernel docs suggest not to use more than 2 x total physical RAM for zram, so I have experimented with 8GB zram swap in this plugin (a bit less maybe more sensible). Amazing to have so much swap space and all in RAM! However, note that even unused zram does use 0.1% x its created size of physical RAM, even when zram itself empty (so that will make free RAM figure not look so good when using large zram swapsize), but overall benefit is huge in terms of fast virtual RAM:

Code:
              total        used        free      shared  buff/cache   available
Mem:        3874884       99020     3084004      328876      691860     3250748
Swap:       8388604           0     8388604


Note (rufwoof et al) that you could alternatively activate normal swap, either in inram00.plug. Again, you are best to check main script to see how that works. Such a plug could contain, for example:

Code:
# For rufwoof's set up, could contain the following:
swapon /dev/sda3
mount -o remount,size=13G /mnt/layers/RAM



As for zram alternative (or addition) here is the exemplar of the inram00.plug I am using during boot of my system:

NOTE WELL, that if you want to use an inram00.plug file you must make sure it is placed in the /mnt/bootpartition/bootfrom directory at boot time or it won't of course be found.


Code:
#inram00.plug
# examplar of setting up zram (or conventional) swap
# wiak 09Sep2019

# Refer: https://www.kernel.org/doc/html/latest/admin-guide/blockdev/zram.html

# NOTE: maximum physical RAM size on my system is: 4 GB
# Modify the following according to your own system RAM capacity:
# so 1024 x (1024x1024) is 1 GiByte (or half of total physical RAM)

# In Below the total max zram created is twice physical RAM size (e.g. 2 x 4 GB on my system)
# Kernel docs recommended to use no more than that (can be less)
# since typical zram compression is 2 times
# Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful

[b]EDIT: The following is out of date (and may need modified) since rdsh plug positions have moved:[/b]

# NOTE that an alternative for achieving the same ends would be to put
# the following into a separate rdsh2.plug (then inram00.plug shouldn't be used)

# Create /dev/zram{num_devices}; was 2 cores on my Core2Duo system
modprobe zram num_devices=`nproc`  # here using same number as CPU cores

# Select zram compression algorithm to use (default is lzo)
echo lz4 > /sys/block/zram0/comp_algorithm

# zram device size; tho 8GB maybe slightly too much for my 4GB RAM system
echo  8G > /sys/block/zram0/disksize  # zram0 size 8GiB

# Using zram0 for swap
mkswap -L zram0 /dev/zram0
swapon -p 100 /dev/zram0
# mkswap /dev/sda6
# swapon /dev/sda6


Note that I'm not suggesting best options how to use these new inram/plugin facilities in terms of optimum size values (I don't know since just experimenting with different combinations myself).

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130

Last edited by wiak on Thu 03 Oct 2019, 21:36; edited 11 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Mon 02 Sep 2019, 09:35    Post subject:  

That's great wiak. Thanks.

My thinking around loading into ram/swap is that swap can be set to any size (up to the limit of actual free disk space). That's the same as just using hdd as storage/work space except where instead of being on a ext3 or whatever partition its a swap style formatted partition/file. Like hdd based ram+swap also means that the main sfs + save can be pretty much any size (at least that can fit into ram + size of allocated swap file/partition).

Booting with say main sfs stored on hdd, changes stored on disk versus changes stored in ram/swap will tend to be faster for ram/swap as even if its actually being written to disk/swap that is (much faster) direct writing/reading rather than going through (slower) file I/O processes. With hdd based storage upper_changes remains on disk, so you either have to delete that content or make a 02changes.sfs (03changes.sfs...etc) copy of it and delete the content of upper_changes in readiness for the next boot. With ram+swap based upper_changes you either make the 02changes.sfs copy of that, or just reboot -n (avoiding having to delete upper_changes). And from what I've generally seen, reboot -n (without running swapoff) is as quick as standard reboots where no swap is involved. I have also noticed that running a save.sh when using ram+swap is lightning fast, especially when using lz4 mksquashfs compression that multi-cores and processes at potentially ram like speeds; More often you don't even get to see the mksquashfs ==== progress line it flashes through so quickly.

When the cpu has swap space available and something needs to be ejected out of cache to make way for other data to be read in, then the cached page being ejected might be copied directly into swap as-is (fast direct write). Later that might be pulled back into cache again when needed (fast direct read). When no swap is available then the cached page is just deleted, so if later needed again its pulled back in from disk, which in the case of sfs's means reading in the compressed content using standard (slower) file io and then decompressing that (around 1.5 times the size of the actual data memory space being required), a much slower process compared to fast direct read/writes from swap, and more cpu usage compared to just swapping pages in/out swap space.

That all said, if you have 4GB of ram like you and I then its pretty much mute as after a relatively short while after booting the entire system is likely sitting and running all in ram anyway.

Trying a build of my firstrib00.plug now and a copy of your inram.plug, but set to 6GB instead of 8GB for zram (my laptop's much the same as yours, 2 core, 4GB). For menu.lst I'm using a entry of
Code:
title VOID3
   root (hd0,0)
   kernel /VOID3/vmlinuz-4.19.69_1 bootfrom=/mnt/sda1/VOID3 inram=100% changes=RAM copy2ram
   initrd /VOID3/initramfs05.gz

i.e. with the inram=100% boot parameter. Also for the build_wee.... script I've set the mksquashfs compression to be just -comp lz4 (without the -Xhc compression switch), on the assumption that as zram also is set to use lz4 having the same might be quicker to move into zram (is the kernel clever enough to spot that the same compression is being used and just do a direct copy rather than decompressing lz4 compressed source and compress it again using lz4 into ram ??? (suspect not)).

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3681

PostPosted: Mon 02 Sep 2019, 10:33    Post subject:  

After building the cpio formation failed. mksquashfs ran through OK and then it threw out some errors. I copied your

Code:
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 5401
Number of ids (unique uids + gids) 5
Number of uids 3
   root (0)
   user (1000)
   unknown (1003)
Number of gids 5
   root (0)
   lp (10)
   user (1000)
   xbuilder (101)
   unknown (1003)
./build_weedog_initramfs05_s101.sh: line 381: 8: command not found
./build_weedog_initramfs05_s101.sh: line 392: initramfs05.gz: command not found
./build_weedog_initramfs05_s101.sh: line 393: unexpected EOF while looking for matching `''
./build_weedog_initramfs05_s101.sh: line 419: syntax error: unexpected end of file
#

Just making a note here before I reboot and lose those messages, as I'm going to build the cpio by hand and reboot with that.

Edit: Looking like it didn't pick up on the inram=100% kernel parameter. Showing 1.7GB instead of 3.4GB. A 1.4GB main sfs boots OK (gzip'd), but lzo compressed sfs doesn't (1.9GB)

Tried hard coding a resize of /mnt/layers/RAM to 4G size at the end of the copy2ram code section within build_wee... script, but again didn't boot the lzo version. Off to try out hard coding the inram=100% value directly within build_wee....

... that worked OK (hard coded size=100% within build_wee... Need to also resize /mnt/layers/RAM, I put mine in just after the rdsh2

_rdsh rdsh2 # rdsh2.plug will be sourced here
mount -o remount,size=3G /mnt/layers/RAM

I'm rebuilding now with that set to 6G. Under 3G /mnt/layers/RAM I was able to boot a 1.8GB main sfs (lzo compressed) with copy2ram boot parameter, and then dd a file of /dev/urandom of 1GB in size within root, and then another run of that had it produce a 0.2GB dd file before it ended and the system slowed. urandom data is not very compressible (if at all). That pretty much filled up the 3GB of /mnt/layers/RAM so increasing that to 6GB might see even more potential filesize (will report back once I've tested that (takes a while as I rebuild freshly each time)).

...

First of 1GB bigfile1 creation with /mnt/layers/RAM set to 6G (posting now as I suspect the next bigfile2 creation may result in a lockup)
Code:
# free -m
              total        used        free      shared  buff/cache   available
Mem:           3408         331         542        1864        2534         985
Swap:          6143           0        6143
# df -h
Filesystem      Size  Used Avail Use% Mounted on
inram           6.0G  1.9G  4.2G  31% /mnt/layers/RAM
overlay_result  6.0G  1.9G  4.2G  31% /
run             1.7G  696K  1.7G   1% /run
dev             1.7G  4.0K  1.7G   1% /dev
shm             1.7G   11M  1.7G   1% /dev/shm
cgroup          1.7G     0  1.7G   0% /sys/fs/cgroup
tmpfs           1.7G  8.0K  1.7G   1% /tmp
# pwd
/root
# time dd if=/dev/urandom of=bigfile1 bs=1024k count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.0954 s, 71.1 MB/s

real   0m15.099s
user   0m0.007s
sys   0m13.699s


.. just about to press enter on
Code:
# free -m
              total        used        free      shared  buff/cache   available
Mem:           3408        3114         112          55         181          61
Swap:          6143        3059        3084
# df -h
Filesystem      Size  Used Avail Use% Mounted on
inram           6.0G  2.9G  3.2G  48% /mnt/layers/RAM
overlay_result  6.0G  2.9G  3.2G  48% /
run             1.7G  696K  1.7G   1% /run
dev             1.7G  4.0K  1.7G   1% /dev
shm             1.7G   12M  1.7G   1% /dev/shm
cgroup          1.7G     0  1.7G   0% /sys/fs/cgroup
tmpfs           1.7G  8.0K  1.7G   1% /tmp
# pwd
/root
# time dd if=/dev/urandom of=bigfile2 bs=1024K count=512

... yep, that totally trashed the system (X crashed, console locked up).

So my 4GB system with 100% size has around 3.4GB of ram (after radeon also reserves its space). Resizing /mnt/layers/RAM to 3GB and even when fully loaded with compressed data (1.8GB of main sfs lzo compressed loaded into ram (copy2ram), and around 1.2GB of urandom (comparable to compressed) dd filesize, it still continues to work. Much above that and its ouch-time. 3GB of compressed data in ram compares to 6GB of non compressed, assuming a 2:1 average compression ratio. More than enough for the average 'Pup' style OS.

Off now to see if I can spot the issue with the inram=100% kernel parameter problem.

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Mon 02 Sep 2019, 16:33    Post subject:  

rufwoof wrote:
Edit: Looking like it didn't pick up on the inram=100% kernel parameter.


Oh, I thought I'd fixed that. I did have that same problem previously. I wonder if I uploaded the version before I fixed it? Going out for an hour but will also check once I'm back.

EDIT: did you put inram_sz=100% on grub menu.lst kernel line? i.e. with the % at the end?

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 27 of 49 [722 Posts]   Goto page: Previous 1, 2, 3, ..., 25, 26, 27, 28, 29, ..., 47, 48, 49 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1244s ][ Queries: 13 (0.0318s) ][ GZIP on ]