Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 26 May 2017, 15:09
All times are UTC - 4

 Forum index » Advanced Topics » Cutting edge
populating /dev from /sys
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 4 [54 Posts]   Goto page: 1, 2, 3, 4 Next
Author Message
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sat 11 May 2013, 17:20    Post subject:  populating /dev from /sys  
kernel's shell example (~16.0 seconds)
Code:
#!/bin/bash

  # Populate block devices

  for i in /sys/block/*/dev /sys/block/*/*/dev
  do
    if [ -f $i ]
    then
      MAJOR=$(sed 's/:.*//' < $i)
      MINOR=$(sed 's/.*://' < $i)
      DEVNAME=$(echo $i | sed -e 's@/dev@@' -e 's@.*/@@')
      mknod /dev/$DEVNAME b $MAJOR $MINOR
    fi
  done

  # Populate char devices

  for i in /sys/bus/*/devices/*/dev /sys/class/*/*/dev
  do
    if [ -f $i ]
    then
      MAJOR=$(sed 's/:.*//' < $i)
      MINOR=$(sed 's/.*://' < $i)
      DEVNAME=$(echo $i | sed -e 's@/dev@@' -e 's@.*/@@')
      mknod /dev/$DEVNAME c $MAJOR $MINOR
    fi
  done
Of course it is slow - they are calling sed 3 times for every entry
my reworking
Code:
#!/bin/sh
IFS=":"
DEV=/dev

#char devices
for i in /sys/bus/*/devices/*/dev /sys/class/*/*/dev;  do
   read MAJOR MINOR < "$i" && i=${i%/dev} && NAME=${i##*/} && \
   mknod "$DEV/$NAME" c $MAJOR $MINOR 2>/dev/null
done

#block devices
for i in /sys/block/*/dev /sys/block/*/*/dev;  do
   read MAJOR MINOR < "$i" && i=${i%/dev} && NAME=${i##*/} && \
   mknod "$DEV/$NAME" b $MAJOR $MINOR 2>/dev/null
done

my version runs in ~2 seconds with bash 4
pretty good improvement, but ash reduces it to under 1 second
I then tried it with busybox built to:
1)prefer applets, 2)run as a standalone shell and 3)run nofork applets directly
This got it down to just over 1/4 second
but out of curiousity I took a look at mknod and found out that it is only listed as a noexec applet, so I modified include/applets.src.h to change mknode from NOEXEC to NOFORK and that reduced the execution time to just over 1/10th of a second

... great now I can quickly create my containerized $container/dev to use with lxc (which is why I didn't hardcode /dev btw) - kiss my ass udev/systemd - don't need your slow, bloated crap
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sat 11 May 2013, 18:02    Post subject:  
Next up: handling hotplug events

Process to figure out what to do:
1) need to know environment variables and args that are passed
--->use env >>somefile in a dummy script
--->echo /root/dummy /proc/sys/kernel/hotplug
2)use this data to analyze what we may need to do.

attaching a gzip output from removing then reinserting a usb flash drive
hotplug.events.gz
Description 
gz

 Download 
Filename  hotplug.events.gz 
Filesize  747 Bytes 
Downloaded  644 Time(s) 

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Sat 11 May 2013, 21:13    Post subject:  
Very nice as function for rc.sysinit !

time dyn_mknod
Code:


#!/bin/ash
#technosaurus, May 2013
#http://murga-linux.com/puppy/viewtopic.php?t=86125

function_dyn_mknod(){

alias mknod='busybox mknod'
alias rm='busybox rm'
alias mkdir='busybox mkdir'

ERR=/dev/stderr
OLDIFS="$IFS"
IFS=":"
DEV=/dev
mkdir -p "$DEV"

#char devices
for i in /sys/bus/*/devices/*/dev /sys/class/*/*/dev;  do
   MAJOR='';MINOR='';NAME=''
   read MAJOR MINOR < "$i" && i=${i%/dev*} || continue
   NAME=${i##*/}
   echo "'$NAME' c '$MAJOR' '$MINOR'"
   [ "$MAJOR" -a "$MINOR" -a "$NAME" ] || continue
   #rm -f "$DEV"/"$NAME" || continue
   rm ${DEV}/${NAME} || continue
   #mknod "$DEV"/"$NAME" c $MAJOR $MINOR 2>$ERR
   mknod ${DEV}/${NAME} c $MAJOR $MINOR
done

#block devices
for i in /sys/block/*/dev /sys/block/*/*/dev;  do
   MAJOR='';MINOR='';NAME=''
   read MAJOR MINOR < "$i" && i=${i%/dev*} || continue
   NAME=${i##*/}
   echo "'$NAME' b '$MAJOR' '$MINOR'"
   [ "$MAJOR" -a "$MINOR" -a "$NAME" ] || continue
   #rm -f "$DEV"/"$NAME" || continue
   rm ${DEV}/${NAME} || continue
   #mknod "$DEV"/"$NAME" b $MAJOR $MINOR 2>$ERR
   mknod ${DEV}/${NAME} b $MAJOR $MINOR
done

IFS="$OLDIFS"
}

function_dyn_mknod



[ .. huge snip .. ]
'sda9' '8' '9'
'sdb1' '8' '17'

real 0m3.964s
user 0m0.128s
sys 0m0.412s

Had strange messages for using double quotes in the
mknod "$DEV"/"$NAME" b $MAJOR $MINOR 2>$ERR line >
'sdb1' '8' '17'
/root/my-applications/bin/dyn_mknod: line 44: can't create : nonexistent directory
Shocked Never had such before ..
BB is Lupu BusyBox v1.16.2 (2010-06-19 18:02:46 GMT-8 )
running a luma preview by plazdayz from 2010 ..
Funny the system still runs without the nodes Very Happy

Anyway nice code . Have such similar in my /etc/init.d/alsa, but much more heavy:
Code:
  D=`find /sys -type d -name "*sound*"`
   POOL=''
    for i in $D ; do
     ##echo $i ;
     DEV=`find $i -type f -name "dev" -exec echo -n {}@ \; -exec cat {} \; | tr ':' '@' | tr ' ' '@' -exec echo \; ` ;
     POOL="$POOL $DEV"
    done
   POOL=`echo $POOL | rev`
    for i in $POOL ; do
     NODE=`echo $i | cut -f 3 -d '@'`
     NODE=`echo $NODE | rev`
     NODE=`basename $(dirname $(echo $NODE) )`
     MAJ=`echo $i | cut -f 2 -d '@'`
     MAJ=`echo $MAJ | rev`
     Min=`echo $i | cut -f 1 -d '@'`
     Min=`echo $Min | rev`
     echo "$NODE" "$MAJ" "$Min"
     rm /dev/$NODE
     mknod /dev/$NODE c $MAJ $Min
     rm /dev/snd/$NODE
     mknod /dev/snd/$NODE c $MAJ $Min
    done 
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sat 11 May 2013, 22:11    Post subject:  
there is no need to prepend busybox ... (if busybox has it ash will use it)
and ERR-... should be ERR=... (probably the error source)
and what was the point of adding the rm ... ?
its usually better to use [ ! -f "$VAR" ] && command "$VAR"
... as it is, your rework will remove all existing nodes that are found and not recreate it (bad when Puppy uses a static /dev), but rerunning mknod on an existing device is quick and fails harmlessly (thus the 2>/dev/null)
keep in mind this script should only run once, so there shouldn't be any preexisting nodes (aside from the static ones that could be wrong)
also unnecessary echoes in loops really slow down scripts and should only be used for debugging - you can use OUTPUT="$OUTPUT
$NEWDATA" in the loop and then echo $OUTPUT afterwards

but Puppy's busybox is pretty conservatively built, if you're building it for an initrd, the following should be enabled 1)prefer applets, 2)standalone shell and 3)run nofork applets directly
This alone makes a 5x speed improvement.

Another related tip
"$VAR1/$VAR2" makes the whole thing one string because all variables are expanded inside double quotes (but not in single quotes) - using "$A"/"$B" is unnecessary and just slows it down

FYI - this makes it possible to construct a large command with many arguments (including quoted ones) - especially useful with programs like dialog Xdialog

Code:
[ "$DISPLAY" ] && DIALOG=Xdialog || DIALOG=dialog
MSG="hello world"
mycommand="$DIALOG --msgbox '$MSG' 0 0"
eval $mycommand
#this technique is useful for iteratively building a command in a loop

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Sat 11 May 2013, 22:46    Post subject:  
Quote:
and ERR-... should be ERR=... (probably the error source)

was the failure =- typo . Thanks .
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sat 11 May 2013, 22:58    Post subject:  
start of a hotplug script:
Code:
#!/bin/ash
DEV=/dev

#todo? maybe modify to "$ACTION$SUBSYSTEM" ???
add(){
SUBDIR=${DEVNAME%/*}
[ "$MAJOR" ] && [ "$MINOR" ] || return
[ "$SUBDIR" ] && [ ! -d "$SUBDIR" ] && mkdir -p "$SUBDIR"
case $SUBSYSTEM in
   block)mknod "$DEV/$DEVNAME" b $MAJOR $MINOR;;
   *)mknod "$DEV/$DEVNAME" c $MAJOR $MINOR;;
esac
}

remove(){
[ "$DEVNAME" ] && rm  "$DEV/$DEVNAME"
}

$ACTION

#notes
#ACTION=add,remove,???
#SEQNUM=integer
#DEVPATH=subdirectory of /sys
#SUBSYSTEM=scsi,bsg,...
#MODALIAS=scsi:t-0x00
#DEVTYPE=scsi_device (same as $1 ??)
#DEVNAME=bsg/8:0:0:0 (@ /dev/... ??)
#ACTION=add
#MAJOR=integer
#MINOR=integer

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Sat 11 May 2013, 23:03    Post subject:  
For bsg device node path I could only think of reading the uevent files
like :
Code:

#!/bin/ash

ERR=/dev/null

CHAR_DEVICES=`sed -n '/Character devices:/,/Block devices:/p' /proc/devices`
BLOK_DEVICES=`sed -n '/Block devices:/,//p' /proc/devices`

OLDIFS="$IFS"

find /sys -type f -name uevent |
while read uev_file
do
 MAJ='';Min='';DEV=''
 IFS='='
  while read name val
  do
   case $name in
    MAJOR)   MAJ=$val;;
    MINOR)   Min=$val;;
    DEVNAME) DEV=$val;;
    *) :;;
   esac
  done<$uev_file
 IFS="$OLDIFS"
 [ "$MAJ" -a "$Min" -a "$DEV" ] || continue
 KINDC=`echo "$CHAR_DEVICES" | grep -w "$MAJ"`
 [ "$KINDC" ] && KIND=c || {
 KINDB=`echo "$BLOK_DEVICES" | grep -w "$MAJ"`
 [ "$KINDB" ] && KIND=b ; }
 [ "$KIND" ] || continue
 #echo "'$DEV' '$KIND' '$MAJ' '$Min'"
 #rm /dev/$DEV
 mknod /dev/$DEV $KIND $MAJ $Min
done 2>$ERR

but that takes 10 seconds here .
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sat 11 May 2013, 23:57    Post subject:  
I used the uevent files in one of my jwm tools, but all you have to do is
. pathto/uevent
and it will source the variables in it

find will recurse the giant barrage of symlinks many levels, plus find is external, so if the path can be narrowed down to:

/sys/something/something/*/whatever
you can use a
Code:
for x in /sys/something/something/*/whatever; do
something_with "$x"
done


what are the results of find?
looks like /sys/class/bsg/<devicename>
with a symlink @
/sys/block/<devname>/queue/bsg

WTF is it a char or block device?
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Sun 12 May 2013, 11:08    Post subject:  
Don't know .. had to read https://en.wikipedia.org/wiki/Device_file

What seems to be that there are some doublettes there for both

Character devices:
1 mem
128 ptm
254 bsg


Block devices:
1 ramdisk
128 sd
254 mdp
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Sun 12 May 2013, 12:46    Post subject:  
Quote:
find will recurse the giant barrage of symlinks many levels

from man find :

-P Never follow symbolic links. This is the default behaviour.

Think that find in /sys is quiet fast .
But of course finds a lot of unallowed uevent files .

-readable
Matches files which are readable.
-perm mode
File's permission bits

-readable did not work for me but
Code:
find -P /sys -type f -name uevent -perm /u+r

find itself is fast but processing of the uevent files is slow .

Version 2 for events ie pup_event_frontend_d detects sdc in /sys/block
Code:

#!/bin/ash

[ "$*" ] && WANTED="${*// /|}"
WANTED="${WANTED#|}"
WANTED="${WANTED%|}"

ERR=/dev/null;QUIET='-q'
#ERR=/dev/stderr
mount -t proc none /proc &>$ERR
mount -t sysfs none /sys &>$ERR
[ -L /dev/fd ] || ln -sf /proc/self/fd /dev/fd
ln -sf /dev/fd/0 /dev/stdin
ln -sf /dev/fd/1 /dev/stdout
ln -sf /dev/fd/2 /dev/stderr

CHAR_DEVICES=`sed -n '/Character devices:/,/Block devices:/p' /proc/devices`
BLOK_DEVICES=`sed -n '/Block devices:/,$ p' /proc/devices`

OLDIFS="$IFS"

function_read_uev(){
while read uev_file
do
 MAJ='';Min='';DEV='';KINDB='';KINDC='';KIND=''
 IFS='='
  while read name val
  do
   case $name in
    MAJOR)   MAJ=$val;;
    MINOR)   Min=$val;;
    DEVNAME) DEV=$val;;
    *) :;;
   esac
  done<$uev_file
 IFS="$OLDIFS"
 [ "$MAJ" -a "$Min" -a "$DEV" ] || continue
 Dev="${DEV##*/}"
 KINDC=`echo "$CHAR_DEVICES" | grep -w "${MAJ}" | grep "${Dev:0:2}"`
 [ "$KINDC" ] && KIND=c || {
 KINDB=`echo "$BLOK_DEVICES" | grep -w "${MAJ}" | grep "${Dev:0:2}"`
 [ "$KINDB" -o "$MAJ" = 259 ] && KIND=b ; }
 ## 259 is blkext but nodes are still /dev/sda not /dev/blkexta
 [ "$KIND" ] || continue
 #echo "'$DEV' '$KIND' '$MAJ' '$Min'"
 #rm -f /dev/$DEV || rmdir /dev/$DEV
 DEV="/$DEV"
 mkdir -p "/dev/${DEV%/*}"
 mknod "/dev/$DEV" $KIND $MAJ $Min
done 2>$ERR
}

if [ ! "$WANTED" ] ; then
find -P /sys -type f -name uevent -perm /u+r | function_read_uev
else
 for p in ${WANTED//|/ }
 do
  find -P /sys -path "*${p}*" -type f -name uevent -perm /u+r | function_read_uev
 done
fi

# time sys-ev2 sdc

real 0m1.413s
user 0m1.185s
sys 0m0.460s
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sun 12 May 2013, 15:38    Post subject:  
the while read blocks like that will skip the last line unless you do something like
while read LINE || [ "$LINE" ];do

guess we've been lucky the last thing is not important

in this case the LINE is a uevent file with shell formatted contents that can be sourced as a file with variable names, (but we may need to NULL out previous VARs if it is found that some uevent files do not contain all of MAJOR, MINOR, DEVNAME and DEVTYPE) such as:
Code:
while read LINE || [ "$LINE" ];do
    MAJOR="" MINOR="" DEVNAME="" DEVTYPE=""
    . "$LINE" #this is the full path of the uevent file
    #now we can do stuff with MAJOR, MINOR, DEVNAME and DEVTYPE, no parsing required
done


EDIT: ... wow - I need to look into what kernel version got /sys/dev/{char,block} - that would be so much easier (has symlinks to ALL char and block devices in appropriate subdirectories) I guess the kernel docs are more outdated than I even thought
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sun 12 May 2013, 16:19    Post subject:  
here is my rework using /sys/dev/{char,block}/*/uevent

Code:
#!/bin/ash
IFS=": "
DEV=/dev #change this for testing

#char devices
for i in /sys/dev/char/*/uevent;  do
   . "$i"
   [ "${DEVNAME%/*}" != "${DEVNAME}" ] && [ ! -d "$DEV/${DEVNAME%/*}" ] && mkdir -p "$DEV/${DEVNAME%/*}"
   mknod "$DEV/$DEVNAME" c $MAJOR $MINOR 2>/dev/null
done

#block devices
for i in /sys/dev/char/*/uevent;  do
   . "$i"
   [ "${DEVNAME%/*}" != "${DEVNAME}" ] && [ ! -d "$DEV/${DEVNAME%/*}" ] && mkdir -p "$DEV/${DEVNAME%/*}"
   mknod "$DEV/$DEVNAME" b $MAJOR $MINOR 2>/dev/null
done

#todo create symlinks?

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Sun 12 May 2013, 22:35    Post subject:  
I uncovered a bug ... mknod will only accept major and minor 0..255, but the kernel supports higher values, so the mknod in busybox fails on /dev/bus/usb/004/001 (amongst others) because it has a minor value of 384 ... so I patched busybox mknod to use MINOR&0xff making it have an actual minor value of 128 - IDK if this is OK or not - probably not

musl-libc doesn't account for it, but it looks like uclibc and glibc do (prepends the extra bit in an unsigned int vs using just an unsigned short) - posting to musl-libc list
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 4207
Location: Kiel,Germany
PostPosted: Mon 13 May 2013, 12:09    Post subject:  
The idea of sourceing is really fine !
Would probably come to it after several years due to not using eval gtkdialog or source /etc/rc.d/PUPSTATE much .
Most sourced files are actualy scripts containing functions .

*
Just did a
BusyBox v1.16.2 (2010-06-19 18:02:46 GMT-8 ) multi-call binary.
# busybox mknod /dev/test c 499 499
# ls -l /dev/test
crw-r--r-- 1 root root 499, 499 2013-05-13 17:12 /dev/test
# busybox mknod /dev/testb b 500 500
# ls -l /dev/testb
brw-r--r-- 1 root root 500, 500 2013-05-13 17:15 /dev/testb

seems to work OK.
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4718
PostPosted: Mon 13 May 2013, 13:33    Post subject:  
I figured out a way to allow other packages to provide their own configurations for a shell script. this one is an example for a hotplugger shell script replacement (just the subsystem section)

Code:
subsystem="case $SUBSYSTEM in "
for x in /etc/mdev.d/subsystem.*; do
   subsystem="$subsystem $(< ${x})"
done
subsystem="$subsystem esac"
eval "$subsystem"


so if you wanted to do something for all block subsystems you would
Code:
echo "block)dosomething $some $args ;;" > /etc/mdev.d/subsystem.block
but it could be more complex

I may find some use for this technique in jwm_tools so that packages can provide their owns configs - how do I come up with this crap?
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 4 [54 Posts]   Goto page: 1, 2, 3, 4 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.1621s ][ Queries: 13 (0.0150s) ][ GZIP on ]