Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 19 Sep 2014, 04:13
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Problem with new Remaster script
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [13 Posts]  
Author Message
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Thu 21 Aug 2014, 04:47    Post subject:  Problem with new Remaster script
Subject description: sfsdirect - 3 minute remaster in ram!
 

I have arrived at a new remaster script which simply squashes the system running in ram with extra sfs and savefile included. The problem I now experience is that the reulting sfs causes kernel panic at switchroot. I tried changing permissions to proc tmp mnt sys but didn't help. So I am appealing to those more knowledgeable than myself to point out what is going wrong!

Code:
#!/bin/sh
#sfsdirect remastering suite by stemsee copyright Marcos Contant 2014

. /etc/DISTRO_SPECS
echo "SfS direct is a live remaster system which works in either ram or device.
Make sure that your system has enough ram or uses a swap for the remaster, or
specify path and drive. e.g. mnt sda2.

Do you want to work in ram? y or n"
read ram
case $ram in
    y) if [ -d /tmp/quaddirs ]; then
   rm -r /tmp/quaddirs
   rm /tmp/*.sfs
   mkdir -p /tmp/quaddirs/{mnt,tmp,sys,proc}
   else
   mkdir -p /tmp/quaddirs/{mnt,tmp,sys,proc}
   fi
   chmod 777  /tmp/quaddirs/tmp
   chmod 755  /tmp/quaddirs/sys
   chmod 755  /tmp/quaddirs/mnt
   chmod 755  /tmp/quaddirs/proc
   if [ -d /aufs ]; then
   ln -fnrsv /tmp/quaddirs/tmp /tmp/quaddirs/run
   fi
   mksquashfs  /* /tmp/quaddirs/* /tmp/$DISTRO_NAME.sfs -e /run /proc /tmp /mnt /initrd /sys /aufs /archive /root/.cache /root/.Trash/* /root/Downloads/* /root/.history /root/.bash_history /root/.Skype /etc/resolv.conf /root/.XLOADED /etc/.XLOADED /etc/blkid-cache /etc/reslove.conf /lib/modules /lib/firmware -noappend
   echo "'$DISTRO_NAME.sfs has been created in /tmp. Where do you want to save it to?
   Enter the path e.g. /mnt/sda2/"
   read savepart
   mv -f /tmp/$DISTRO_NAME.sfs $savepart
   echo "All done!" && exit;;

   n)  if [ -d /tmp/quaddirs ]; then
   rm -r /tmp/quaddirs
   mkdir -p /tmp/quaddirs/{mnt,tmp,sys,proc}
   else
   mkdir -p /tmp/quaddirs/{mnt,tmp,sys,proc}
   fi
   chmod 777  /tmp/quaddirs/tmp
   chmod 755  /tmp/quaddirs/sys
   chmod 755  /tmp/quaddirs/mnt
   chmod 755  /tmp/quaddirs/proc
   if [ -d /aufs ]; then
   ln -fnrsv /tmp/quaddirs/tmp /tmp/quaddirs/run
   else
   echo "Input path and drive and directory e.g. mnt sda2"
   read path drive
   mount /dev/$drive /$path/$drive
   mksquashfs /* /tmp/quaddirs/* /$path/$drive/$DISTRO_NAME.sfs -e /proc /tmp /mnt /initrd /proc /sys /aufs /archive /root/.cache /root/.Trash /root/Downloads/* /root/.history /root/.bash_history /root/.Skype /etc/resolv.conf /root/.XLOADED /etc/.XLOADED /etc/blkid-cache /etc/reslove.conf /lib/modules /lib/firmware  -noappend
   echo "All done!"
   fi && exit;;
   *) echo "You didn't enter anything relevant! Exiting script." && exit;;
esac



Previous effort
Code:
#!/bin/bash
#Unattended-Remaster script by StemSee with additions from Fredx181!

timelimit=10
dir1=apupsys
dir2=bpupiso
niso=remiso.iso

if [ -d /live-rw-backing ]; then
path=media
else
path=mnt
fi

echo -e "                      INTRODUCTION
This is an unattended remastering script.  All questions throughout the
script are optional. There is no need to answer any of them, as default fallback
answers are provided. However to make an iso please input the path to your boot
directory so that the initrd and vmlinuz and zdrv (kernel-modules.sfs) can be
easily located and copied to '$dir2' directory.
Mount the boot medium/device/partition now.

                       SPECIAL FUNCTIONS
1 ... If you just want to quickly mksquashfs an existing '$dir1' directory
enter 'mksquashfs' at the 'name' prompt for a gz compressed '$DISTRO_PUPPYSFS'
'mskquashfsfd' for fatdog '$DISTRO_NAME'and 'mksquashfsdd' for DebianDog.

2 ... If you just want to re/make an iso enter 'iso' for the 'name' and for the
third question enter the partition of the existing '$dir2' directory.

                       WORKDIR & BOOT PATH
Two optional questions to set up 'workdir' and get current OS 'boot' path
(where/what you booted this running OS from) for vmlinuz and initrd for iso.
The script will default to 'sda2' for workdirectory and '$DISTRO_PUPPYSFS' for
'name'.

Enter 'name' for special functions or wait for default. $timelimit seconds\n: \c"
sleep 10
name=""
read -t $timelimit name
if [ ! -z "$name" ]; then
echo -e "\n You entered '$name'"
else
echo -e "\n TIME OUT\n Assume 'default'."
name=distro
fi

echo
echo -e "Enter partition name of ext 2/3/4 partition for remastering folder
      eg 'sda3' or 'home'
or press enter for '/' in ram or wait for default 'sda2' $timelimit seconds\n: \c"

drive=""
read -t $timelimit drive
if [ ! -z "$drive" ]; then
   echo -e "\n You entered '$drive'"
else
   echo -e "\n TIME OUT\n Assume default."
   drive=sda2
fi

echo
if [ ! -d /$path/$drive ]; then
   mkdir /$path/$drive
else
   mount /dev/$drive /$path/$drive
fi

echo
echo -e "Enter current path for boot directory to find vmlinuz and initrd.gz

   eg 'mnt/sda3/boot' full install you must put initrd.gz in /boot
      'home/slacko-5.7' from frugal install folder
      'media/sr0'  from cd/dvd.
            
no leading slash '/' and no final slash '/' as they will be provided.
Other needed boot files will be copied from /boot. Waiting $timelimit seconds\n: \c"

boot=""
read -t $timelimit boot
if [ ! -z "$boot" ]; then
   echo -e "\n You entered '$boot' as current boot drive."
else
   echo -e "\n TIME OUT\n Assume default."
   boot=boot
fi

case $name in
   iso) for file in vmlinuz initrd initrd.gz kernel-modules.sfs
   do
   cp -a /$boot/$file /$path/$drive/$dir2/
   done
   cp -a /boot/* /$path/$drive/$dir2/
   cd /$path/$drive
   mkisofs -b isolinux.bin -c boot.cat -D -l -R -v -no-emul-boot -boot-load-size 4 -boot-info-table -o "$niso" /$path/$drive/$dir2
   echo "ISO made." && exit;;
   mksquashfs) . /$path/$drive/$dir1/etc/DISTRO_SPECS
   mksquashfs /$path/$drive/$dir1 /$path/$drive/$DISTRO_PUPPYSFS -comp xz -noappend
   mv -f /$path/$drive/"$DISTRO_PUPPYSFS" /$path/$drive/$dir2/
   echo "squashed using gz compression and is in /'$path'/'$drive'/'dir2'" && exit;;
   mksquashfsfd) . /$path/$drive/$dir1/etc/DISTRO_SPECS
   mksquashfs /$path/$drive/$dir1 /$path/$drive/$DISTRO_NAME -comp xz -noappend
   mv -f /$path/$drive/$DISTRO_NAME /$path/$drive/$dir2/
   echo "squashed using gz compression and is in /'$path'/'$drive'/'dir2'" && exit;;
   mksquashfsdd) mksquashfs /$path/$drive/$dir1 /$path/$drive/01-filesystem.squashfs -comp xz -noappend
   mv -f /$path/$drive/01-filesystem.squashfs /$path/$drive/$dir2/
   echo "squashed xz compression and is in /'$path'/'$drive'/'dir2'" && exit;;
   distro) echo " remastered sfs will be created and an iso will be made.";;
esac

echo
echo "Proceeding with remaster on '$drive'."
sleep 2
echo
echo "Checking if previous remaster directories exist."
echo
if [ -d /$path/$drive/$dir1 ]; then
   echo "Previous remaster directories on '$drive' exist.
   They will be overwritten unless you press Ctrl + c to stop this script now
   and backup the directories '$dir1' '$dir2' which are on '$drive'."
   sleep 8
   rm -rf /$path/$drive/{$dir1,$dir2}
   else
   echo "Deleted!!"
fi

echo
echo "Checking for previous '$niso'."
if [ ! -e /$path/$drive/$niso ]; then
   echo "No previous '$niso' found on '$drive'."
else
   echo "About to delete previous '$niso' on '$drive'.
Press 'Ctrl + c' now to stop and manually save it.
Or wait 6 seconds to proceed."
   sleep 6
   echo
   rm -f /$path/$drive/$niso
   echo "Deleted previous '$niso'"
fi
echo
mkdir -p /$path/$drive/{$dir1,$dir2}
echo "Start copying from your running system.
Press Ctrl+c anytime to stop!"
echo
echo "rsync '/'"
rsync -urlptgoD --delete --progress /* /$path/$drive/$dir1/ --exclude=/{run,media,live,/lib/live/mount,cdrom,aufs,dev,proc,mnt,sys,tmp,initrd,archive,audit,/var/cache/apt,/var/cache/slapt,var/lib/apt/lists,var/lib/slapt/lists}
for subdir in {run,media,cdrom,live,/lib/live/mount,aufs,dev,proc,mnt,sys,tmp,archive,audit,live-rw-backing}
do
   if [[ -e /"$subdir" ]]; then
      mkdir -p /$path/$drive/$dir1/$subdir
      echo "'$subdir' created."
   else
      echo "'$subdir' directory not needed for this distro."
   fi
done

echo ""
echo "Entering remaster directory '$dir1'."
cd /$path/$drive/$dir1

echo "
Cleaning caches and .bak, .wh files and history."
rm -rf tmp/* audit/* lost+found/*
cd root
rm -rf .bash_history .history .Trash/* .cache/chrome/default/* .cache/mozilla/firefox/*
rm -rf .cache/mozilla/seamonkey/* .cache/dillo/* Downloads/* .minecraft/* .steam/*
rm -rf .wine/drive_c/users/root/Cookies/* .wine/drive_c/users/root/AppData/XBMC/userdata/addon_data/*
rm -rf .wine/drive_c/users/root/AppData/Tencent/Users/* .wine/drive_c/users/root/AppData/Tencent/QQ/*
rm -rf .wine/drive_c/windows/temp/* .wine/drive_c/users/root/Temp/* .Skype/*

echo "
Leaving '$dir1' directory."

cd ../../../../
echo -n "More cleaning..."
rm -f /$path/$drive/$dir1/var/lib/alsa/asound.state
rm -f /$path/$drive/$dir1/root/.bash_history
rm -f /$path/$drive/$dir1/root/.xsession-errors
rm -rf /$path/$drive/$dir1/root/.cache
rm -rf /$path/$drive/$dir1/root/.thumbnails
rm -f /$path/$drive/$dir1/etc/blkid-cache
rm -f /$path/$drive/$dir1/etc/resolv.conf
rm -rf /$path/$drive/$dir1/etc/udev/rules.d/70-persistent*
rm -f /$path/$drive/$dir1/var/lib/dhcp/dhclient.eth0.leases
rm -f /$path/$drive/$dir1/var/lib/dhcpcd/*.lease
rm -f /$path/$drive/$dir1/root/.XLOADED
rm -f /$path/$drive/$dir1/etc/.XLOADED
chroot /$path/$drive/$dir1/ find . -type f -name '.wh.*' -print -exec rm -rf {} \;  #remove .wh. files
chroot /$path/$drive/$dir1/ find . -type f -name '*.bak' -print -exec rm -rf {} \;  #remove *.bak files
echo
if [[ -f /$path/$drive/$dir1/etc/DISTRO_SPECS ]]; then
. /$path/$drive/$dir1/etc/DISTRO_SPECS
fi
echo -e "
Making the main.sfs squashfs filesystem choose mksquashfs code.
Key:       For normal puppy '$DISTRO_PUPPYSFS': enter 'pp'
                For fatdog '$DISTRO_NAME': enter 'fd'
                For debbiandog '01-filesystem.squashfs': enter 'dd'
      Or wait for default or to enter an sfs name.
     
      Enter mksquashfs code within $timelimit seconds\n: \c"

comp=""
   read -t $timelimit comp
if [ ! -z "$comp" ]; then
   echo -e "
   \n You entered '$comp'.
Using  '$comp' compression for mksquashfs."
else
   echo -e "
   \n TIME OUT\n Assume default gz."
   if [[ -d /aufs ]]; then
      comp=fd
      elif [[ "$path" = media ]]; then
      comp=dd
      elif [[ "$path" = mnt ]]; then
      comp=pp
      else
      echo "Enter name for sfs. '01-filesystem.squashfs' 'puppy.sfs' etc."
      read comp
   fi
fi
echo "Checking zdrv for modules!"
if [ -d /initrd/pup_z/lib/modules ]; then
   echo "Removing modules."
   rm -r /$path/$drive/$dir1/lib/modules
   echo "Removing firmware."
   rm -r /$path/$drive/$dir1/lib/firmware
   echo "done."
else
   echo "Huge detected, removing modules and firmware."
fi
case $comp in
   pp) mksquashfs  /$path/$drive/$dir1 /$path/$drive/$DISTRO_PUPPYSFS -comp xz -noappend;;
   fd) mksquashfs  /$path/$drive/$dir1 /$path/$drive/$DISTRO_NAME -comp xz -noappend;;
   dd) cp -a /dev/console /$path/$drive/$dir1/dev/
   chmod a=rwx,o+t /$path/$drive/$dir1/tmp
   mksquashfs /$path/$drive/$dir1 /$path/$drive/01-filesystem.squashfs -noappend -comp xz -b 1024k -Xbcj x86;;
   *) mksquashfs  /$path/$drive/$dir1 /$path/$drive/$comp -noappend;;
esac

echo "
If you want to make an iso file then these 8 boot files will be supplied
   isolinux.bin,
   isolinuz.cfg,
   grldr,
   boot.msg,
   help.msg,
   help2.msg,
   logo.16,
   README.HTM
copied from /boot to /'$path'/'$drive'/'$dir2'. Or supply your own."
echo
   echo -e "
   Type 'y' to create iso or 'n' to NOT create iso then press enter.
   Defult is 'y'. You have $timelimit seconds\n: \c"
   answer=""
   read -t $timelimit answer
if [ ! -z "$answer" ]; then
   echo -e "
   \n You entered '$answer'"
else
   echo -e "
   \n TIME OUT\n Assume yes"
   answer=y
fi

case $answer in
   y) echo "Copying boot files for iso."
    if [ ! -f /$path/$drive/$dir2/vmlinuz ]; then
      for file in vmlinuz initrd initrd.gz kernel-modules.sfs
      do
      cp -a /$boot/$file /$path/$drive/$dir2/
      done
      cp -a /boot/* /$path/$drive/$dir2/
      cd /$path/$drive
      mkisofs -b isolinux.bin -c boot.cat -D -l -R -v -no-emul-boot -boot-load-size 4 -boot-info-table -o "$niso" /$path/$drive/$dir2
      echo "'$niso' is on drive /'$path'/'$drive'."
    else
      echo "All the needed boot files are not in /'$path'/'$drive'/'$dir2'."
    fi
    exit;;
   
   n)  for file in vmlinuz initrd initrd.gz kernel-modules.sfs
      do
      cp -a /$boot/$file /$path/$drive/$dir2/
      done
      cp -a /boot/* /$path/$drive/$dir2/
      echo "'$dir2' directory left in existence.
   Can be deleted for you if no longer needed.";;
   
   *) echo "Sorry, unexpected input.'$dir1' and '$dir2' directories on '$drive'
   left in existence.  Exiting script! Have a nice day!" && exit;;
esac

echo -e "
Enter 'k' to keep or 'd' to delete /'$path/'$drive'/'$dir2'.
You have $timelimit seconds\n: \c"

   choice=""
   read -t $timelimit choice
if [ ! -z "$choice" ]; then
   echo -e "
   \n  You entered '$choice'."
else
   echo -e "
   \n TIME OUT\n Assume keep"
   choice=k
fi

case $choice in
   k) echo "
   '$dir2' directory left in existence.";;

   d) echo "
   Deleting /'$path'/'$drive'/'$dir2'"
   rm /$path/$drive/$dir2 -r
   echo "
   DELETED!";;
   
   *) echo "
   Unexpected input. Not deleting!";;
esac

echo
sleep 3

if [ ! -f /$path/$drive/$niso ]; then
   echo "'$drive'/'$dir2' directory left in existence."
   sleep 3
else
   echo "'$niso' has been created on '$drive' so deleting '$dir2' directory."
   sleep 3
   rm -rf /$path/$drive/$dir2
   echo "
   All done!"
fi

exit 0
##THE END##

Last edited by stemsee on Yesterday, at 17:07; edited 4 times in total
Back to top
View user's profile Send private message MSN Messenger 
mavrothal


Joined: 24 Aug 2009
Posts: 1617

PostPosted: Thu 21 Aug 2014, 05:17    Post subject:  

/proc and /sys get 555 permissions *after* the kernel is done writing in them. 555 makes the dirs not writable hence the kernel panic (I think)
Change them to 755 and see if this solves it.

PS: you may also need the /run directory that depending on your booting process may cause panic as udev needs it to make devices.
BTW, how the directories on the remastered sfs compare to the original sfs?

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Thu 21 Aug 2014, 10:30    Post subject:  

How do they compare? In terms of permissions? I started listing all permissions in running system to compare with remastered directory, but rightclick->properties-> shows a graphic status of permissions I do not know how they are translated numerically! Except by experimenting on a NewDir in Downloads, which I was doing until I got distracted somehow and stopped. Is there a table showing puppy dir permissions in number format?

Also I thought that /run was not a used dir in puppy linux! DebianDog yes, fatdog 700 has a symlink /run linked to /tmp . But i wont hurt to have it there!
Back to top
View user's profile Send private message MSN Messenger 
saintless


Joined: 11 Jun 2011
Posts: 2445
Location: Bulgaria

PostPosted: Thu 21 Aug 2014, 10:56    Post subject:  

stemsee wrote:
Is there a table showing puppy dir permissions in number format?

Try this one:
Code:
ls -l | awk '{k=0;for(i=0;i<=8;i++)k+=((substr($1,i+2,1)~/[rwx]/) \
             *2^(8-i));if(k)printf("%0o ",k);print}'

Or this:
Code:
stat -c '%a %n' /tmp /root

_________________
Farewell, Nooby, you will be missed...
Back to top
View user's profile Send private message MSN Messenger 
mavrothal


Joined: 24 Aug 2009
Posts: 1617

PostPosted: Thu 21 Aug 2014, 11:02    Post subject:  

There plenty of articles for linux permissions like this one or this one
Running
Code:
ls -l /initrd/pup_ro2

in a running puppy, will show the default directories and permissions in a puppy sfs.

_________________
Kids all over the world go around with an XO laptop. They deserve one puppy (or many) too Very Happy
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Thu 21 Aug 2014, 11:24    Post subject:  

ok....thanks guys! Another lesson to learn and apply ...

Saintless's code lists with numerically represented permissions ie tmp=777 etc!

Perfect.
Back to top
View user's profile Send private message MSN Messenger 
Flash
Official Dog Handler


Joined: 04 May 2005
Posts: 11066
Location: Arizona USA

PostPosted: Thu 21 Aug 2014, 16:24    Post subject:  

Stemsee, is your problem solved? Does the remastered iso work now?
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Fri 22 Aug 2014, 03:28    Post subject:  

@Flash

It didn't solve the issue! But i have to look at mksquashfs -e (exclude) option to make sure tmp mnt sys proc are excluded from the running system, and only those from /tmp/quaddirs/* are added to the sfs.

But try it ... maybe it will work on your distro!
Back to top
View user's profile Send private message MSN Messenger 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Fri 22 Aug 2014, 06:09    Post subject:  

I have an idea what might be the error. Namely, when creating new mnt in /tmp/quaddirs/ must also create subdirs for mount points - sda1-8 sr0-3 home and ram1, just remastering now. Also experimenting with -read-queue and -write-queue to speed things up.
Back to top
View user's profile Send private message MSN Messenger 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Fri 22 Aug 2014, 13:20    Post subject:  

Nah - that didn't make any difference either. I wonder if it has to do with uuid or guid?

So i was trying to void using an inermeiary directory that you copy into first but now I will use one in ram --- see if that works.
Back to top
View user's profile Send private message MSN Messenger 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Fri 22 Aug 2014, 13:38    Post subject:  

I think you need to investigate the 'standard' perms/ownerships used by other distros in order to get a clear idea of what is needed. And, the same to find out what should not be included in a clone of a system. There should be nothing copied from kernel-gernerated content, that is, /proc /sys and /run should never be copied over. And, /mnt should also not be copied if it contains non-system mounts.

Trying to clone a puppy system is quite difficult to figure out because best-practices are completely ignored or contravened.

Normally, you should be able to just cd into / and:
Code:
cp -a list-of-dirs destination

where list-of-dirs excludes the above mentioned dirs. This is the list I always use -I often clone a running system:
bin lib opt root var etc home sbin srv usr
These:
dev media mnt tmp boot proc run sys
should be handled with care. dev should only be copied if static device files are being used. boot is often mounted on another filesysetm so your clone may or may not need that content. mnt and media should be empty dirs or empty dir structures. proc sys tmp and run should be empty dirs on the cloned system

Normal best-practices usage of the filesystem hierarchy *would* make this possible also under puppy.

I have written two tools which convert perms from octal to human and human to octal which I can provide here if needed in the end. But, the problems booting are owning to improper perms and content, I'm afraid.
Back to top
View user's profile Send private message 
stemsee


Joined: 27 Jun 2013
Posts: 363
Location: London

PostPosted: Fri 22 Aug 2014, 14:07    Post subject:  

Thanks amigo

That makes sense! I believe I have figured out the right way to do it! Avoiding the pitfalls you mention. As i am sure it is kernel-generated content. So I will use the mksquashfs command specifying several more folders for source, namely /initrd/pup_ro2 /initrd/pup_rw /initrd/pup_* (any mounted sfs greater than zero) It must work I think!! And no intermediary directory just more careful excludes
Back to top
View user's profile Send private message MSN Messenger 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Sat 23 Aug 2014, 03:17    Post subject:  

The contents of /initrd should also not be copied as they are setup during boot.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [13 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1240s ][ Queries: 11 (0.0073s) ][ GZIP on ]