'better' usbstoragefunc in the init script

What features/apps/bugfixes needed in a future Puppy
Message
Author
Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

'better' usbstoragefunc in the init script

#1 Post by Jesse »

Hi Barry & co,

I have been poking my nose around the init script and I have found a 'better' way to detect when the usb-storage module has identified the usb-storage devices, so that we can then proceed to mount them.
Basically we scan the dmesg output for the count of devices that are usb-storage, and the count of usbstorage devices that have been scanned, and wait for the number to be the same.
I have split the function into two components, because there is an enforced pause after loading usb-storage, it does a pause so that any device 'just plugged in' has time to 'spin up', in case it is a real hard disk.

To use them:

Code: Select all

usbstoragefunc
sleep 1
usbstoragefunc_completeion
Or to use them and be clever with the waiting period:

Code: Select all

usbstoragefunc
insmod /lib/modules/${KERNVER}/cdrom.ko    #before ide-cd.
insmod /lib/modules/${KERNVER}/ide-cd.ko

#might as well do this too...
insmod /lib/modules/${KERNVER}/sr_mod.ko

#ide zip or ide ls-120 drive?...
[ ! "`dmesg | grep "ATAPI FLOPPY"`" = "" ] && insmod /lib/modules/${KERNVER}/ide-floppy.ko

#I dunno why, a bit further down complains needs nls_cp437 and nls_iso8859-1.
insmod /lib/modules/${KERNVER}/nls_cp437.ko
insmod /lib/modules/${KERNVER}/nls_iso8859-1.ko
#filesystems...
[ -f /lib/modules/${KERNVER}/reiserfs.ko ] && insmod /lib/modules/${KERNVER}/reiserfs.ko
[ -f /lib/modules/${KERNVER}/ntfs.ko ] && insmod /lib/modules/${KERNVER}/ntfs.ko
#...my 2.6.16.7 has reiserfs and ntfs builtin, leave these lines for other kernels.
insmod /lib/modules/${KERNVER}/unionfs.ko
usbstoragefunc_completeion
Here are the functions:

Code: Select all

usbstoragefunc() {
 if [ "$USB" ];then
  #k2.6: annoying, lsmod reports usb-storage.ko as usb_storage if loaded...
  if [ "`lsmod | grep "^usb_storage"`" = "" ];then
   echo "USB-Storage: Loading usb-storage module"
   insmod /lib/modules/${KERNVER}/usb-storage.ko &
   echo "USB-Storage: Detecting connected device(s)..."
  fi
 fi
}

usbstoragefunc_completeion() {
 if [ "$USB" ];then
  #k2.6: annoying, lsmod reports usb-storage.ko as usb_storage if loaded...
  if [ ! "`lsmod | grep "^usb_storage"`" = "" ];then
   SLPCNT="0";
   USBSTORAGES="0";
   USBSTORAGESAVAILABLE="0";
   while true;do
     USBSTORAGES="`/bin/dmesg | grep -o "usb-storage: device found at.*" | wc -l | sed -e 's/ //g'`";
     USBSTORAGESAVAILABLE="`/bin/dmesg | grep -o "usb-storage: device scan complete" | wc -l | sed -e 's/ //g'`";
     if [ $USBSTORAGES -gt 0 ]; then
       if [ $USBSTORAGES -eq $USBSTORAGESAVAILABLE ]; then
         echo "USB-Storage: Device scan complete, found $USBSTORAGESAVAILABLE devices"
         break
       fi
     fi
     if [ $SLPCNT -gt 50 ]; then
       echo "USB-Storage: Device scan time-out $SLPCNT secs, found $USBSTORAGESAVAILABLE of $USBSTORAGES device(s)."
       break
     fi
     sleep 1
     SLPCNT=`expr $SLPCNT + 1`
   done
   echo ""
  fi
 fi
}
I've quite a few reboots on my system and this is working for me.
Would anyone else like to give it a go?

Jesse

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#2 Post by BarryK »

Jesse, that's great, yes I'll test it
...soon, my PC is rather preoccupied right now (see developer news about T2)!

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#3 Post by Jesse »

Hi Barry,

coolies, yes usb still needs a little work, I'm trying out a few ideas for hardware matching too, based on the pci hardware scanning tool I posted a while back, with scripting in init. Looks like it could well be promising.

T2 sounds very interesting, I don't know if I could handle not having my pc for four straight days though! I salute your dedication!

Jesse

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#4 Post by BarryK »

Jesse,
for comparison, you might like to evaluate 'hwscan2' -- I got it out of the
ROCK Linux project.
I can compile it statically with dietlibc if you like.
Source attached...
Attachments
hwscan.tar.gz
(43 KiB) Downloaded 531 times

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#5 Post by Jesse »

Hi Barry,

Thanks, your download contains a binary.
hwscan2 looks quite cool, its doing the same sort of thing that I've been looking into for getting modules loaded from initrd.
Though after going inside and outside the process for detecting USB controllers, it seems to be fairly simple, but as far as I can tell, you need an executable to extract the "class subtype" from the /proc/bus/pci/00/00.0 device files, which contains binary, not text info, so you can't really write a bash script file to get it.

Ok, so how can we make this work for us? well PCI class 0c03 is USB Controller (0c is for "serial", 03 is for "USB"), which is Class category and Class Type, lspci shows us some useful info:

sh-3.00# lspci | grep 0c03
00:02.0 Class 0c03: 10de:005a (rev a2)
00:02.1 Class 0c03: 10de:005b (rev a3)

but to get the class subtype I use my program:

sh-3.00# ./elspci -l | grep -i 0c03
00:02.1 0C0320 10DE:005B <>
00:02.0 0C0310 10DE:005A <ohci_hcd>

The angle brackets show the module bound to that pci device (loaded driver).

According to the /lib/modules/2.6.16.7/modues.pcimap there are just three matches for the USB controller modules

uhci-hcd 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0x000c0300 0xffffffff 0x0
ehci-hcd 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0x000c0320 0xffffffff 0x0
ohci-hcd 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0x000c0310 0xffffffff 0x0

Now that we have "class subtype" we can match exactly which modules are needed to any pc, in the init script.

Code: Select all

  [ ! "`elspci -l | grep '0C0300'`" = "" ] && insmod /lib/modules/${KERNVER}/uhci-hcd.ko
  [ ! "`elspci -l | grep '0C0310'`" = "" ] && insmod /lib/modules/${KERNVER}/ohci-hcd.ko
  [ ! "`elspci -l | grep '0C0320'`" = "" ] && insmod /lib/modules/${KERNVER}/ehci-hcd.ko 
Potentially this means we do not need the /initrd/pci-usb.ids file to correctly identify what type of controllers we have by description string.

sh-3.00# lspci -i /initrd/pci-usb.ids | grep USB
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)

Look they are both identified as "USB" rather than 00:02.1 being "USB 2".
hmmmm, makes you wonder how many id strings are compatible?

So now we can check to see exactly what we have installed:

sh-3.00# modprobe ehci-hcd
sh-3.00# ./elspci -d -l | grep -i 0c03
00:02.1 0C0320 10DE:005B <ehci_hcd>
00:02.0 0C0310 10DE:005A <ohci_hcd>

Coolies! Now I get USB2 speed rather than having to use the "low speed compatibiliy", now my Flash disk boot is about 60 seconds quicker :)

Jesse
Attachments
elspci.tar.gz
(9.59 KiB) Downloaded 532 times

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#6 Post by BarryK »

That's great !!! :D

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#7 Post by Jesse »

Hmm, seems I found a bug on my system when there is no usb devices needed to be identified, i.e. booting from cdrom it would do the 50 second pause.

So here is an update with a few more comments.

Code: Select all

usbstoragefunc() {
 if [ "$USB" ];then
  #k2.6: annoying, lsmod reports usb-storage.ko as usb_storage if loaded...
  if [ "`lsmod | grep "^usb_storage"`" = "" ];then
   echo "USB-Storage: Loading usb-storage module"
   insmod /lib/modules/${KERNVER}/usb-storage.ko &
   echo "USB-Storage: Detecting connected device(s)..."
  fi
 fi
}

usbstoragefunc_completeion() {
 if [ "$USB" ];then
  #k2.6: annoying, lsmod reports usb-storage.ko as usb_storage if loaded...
  if [ ! "`lsmod | grep "^usb_storage"`" = "" ];then
   SLPCNT="0";
   USBSTORAGES="0";   # number of storages that are pending device identification.
   USBSTORAGESAVAILABLE="0";   # number of storages that are up and running
   USBSTORAGES_NEEDED="0";    # Do we need these things for boot process?
   if [ "`echo -n "$PMEDIA" | grep --extended-regexp "ide|sata|scsi"`" = "" ];then
    # booting off a usb drive... so we need to wait till we have found them
    USBSTORAGES_NEEDED="1";
   fi
   while true;do
     #What can we find out?
     USBSTORAGES="`/bin/dmesg | grep -o "usb-storage: device found at.*" | wc -l | sed -e 's/ //g'`";
     USBSTORAGESAVAILABLE="`/bin/dmesg | grep -o "usb-storage: device scan complete" | wc -l | sed -e 's/ //g'`";
     #have we got everything needed and found?
     if [ $USBSTORAGES_NEEDED -eq 1 ]; then
       if [ $USBSTORAGES -eq $USBSTORAGESAVAILABLE ]; then
         echo "USB-Storage: Device scan complete, found $USBSTORAGESAVAILABLE devices"
         break
       fi
     fi
     # Do we really need to hang about? early break out.
     if [ $USBSTORAGES_NEEDED -eq 0 ]; then
       if [ $SLPCNT -ge 2 ]; then
         echo "USB-Storage: Minimal boot delay, found $USBSTORAGES usb storage devices"
         break
       fi
     fi
     #have we timed out?
     if [ $SLPCNT -gt 50 ]; then
       echo "USB-Storage: Device scan time-out $SLPCNT secs, found $USBSTORAGESAVAILABLE of $USBSTORAGES device(s)."
       break
     fi
     sleep 1
     SLPCNT=`expr $SLPCNT + 1`
   done
   echo ""
  fi
 fi
}
Jesse

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#8 Post by BarryK »

Jesse,
I've announced your work on the Developer News page.
Attached is a modified init script, and elspci compiled statically with dietlibc.
-- it has to be compiled statically to work in the 2.02 initial ramdisk, and it's
only 14K compared with 6.5K for the shared version.

I modified the init script before seeing your last post. Perhaps you would like
to peruse my script and make any mods to it such as the above improvement.
I integrated your suggestions into the init script, with careful (paranoid)
checking that modules do load okay, also note that usbstoragefunc gets
called from two different places, so you will also find a call to
usbstoragecompletionfunc in two places.

Note, I haven't tested it yet.

The init script is for Puppy 2.02, not 2.01.
Attachments
elspci-stuff.tar.gz
(21.91 KiB) Downloaded 453 times

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#9 Post by Jesse »

Hi Barry,

I noticed that you only coded for one usb driver, they all used the same variable. I've made minor changes to allow them all to load as necessary.
Looks good, fingers crossed! :)

Jesse
Attachments
init.tar.gz
(13.2 KiB) Downloaded 501 times

User avatar
RobertB
Posts: 145
Joined: Tue 03 Jan 2006, 01:06
Location: Big D
Contact:

#10 Post by RobertB »

As a fan of Eats, Shoots & Leaves, I must humbly point out that the correct spelling of the function would be "completion", not "completeion".
Absolutely meaningless to the execution of the program, of course -- the functions could be called "Fred" and "Wilma", for all it would care.

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#11 Post by Jesse »

Hi RobertB,

Yep, you're completely right :) My ability to spell correctly is directly proportionaly to either my ability to remember, or energy to check things out on dictionary.com.... my 'typo' for sure. Looks like Barry fixed the spelling in the latest incarnation, all should be good unless I have another go at it :roll:

Jesse

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#12 Post by Jesse »

Hi Barry & co,

Boot time checking of is a disk USB or SATA in a bash script.
Prompted by problem reported by losl in thread http://www.murga.org/~puppy/viewtopic.php?p=61994

Still needs some code for getting other SCSI disks into a SCSIDRIVES list.

May need to replace "cat /var/log/messages" with "dmesg"

output example:

SATADRIVES found: sda
USBDRIVES found: sdb sdc sdd sde sdf sdg sdh sdi

Code: Select all

USBDRIVES="";
SATADRIVES="";
SCSIDRIVES="";

SCSI_CTRL="`cat /var/log/messages | grep -o "scsi. : .*" | sed -e 's/SCSI emulation for USB Mass Storage devices/usb_storage/' -e 's/ : /:/' -e 's/ /_/g' `"
#echo SCSI_CTRL found: $SCSI_CTRL

USBCTRLRS="";
SATACTRLRS="";
SCSICTRLRS="";
for ONE in $SCSI_CTRL
do
	SCSI="`echo $ONE | grep -o "scsi[0-9]"`";
	CTRLR="`echo $ONE | grep -o ":.*" | sed -e 's/://' `";
	CTRLRTYPE="SCSI";
	if [ $CTRLR = "usb_storage" ]; then
		CTRLRTYPE="USB";
	fi
	if [ ! "`echo $CTRLR | grep -i sata`" = "" ]; then
		CTRLRTYPE="SATA";
	fi
	CTRLRN="`echo $SCSI | grep -o "[0-9]"`";
	case $CTRLRTYPE in
		USB)	USBCTRLRS="$CTRLRN|$USBCTRLRS";;
		SATA)	SATACTRLRS="$CTRLRN|$SATACTRLRS";;
		SCSI)	SCSICTRLRS="$CTRLRN|$SCSICTRLRS";;
	esac
	echo "Controller $SCSI is type $CTRLRTYPE   ( $CTRLR )"
done
#echo "USB $USBCTRLRS";
#echo "SATA $SATACTRLRS";
#echo "SCSI $SCSICTRLRS";



SATA_CTRL="`cat /var/log/messages | grep -o "scsi. : .*" | grep ata `"
#echo SATA_CTRL found: $SATA_CTRL

#get the list of SATA drives
SATA_SCAN="`cat /var/log/messages | grep "sd .* Attached scsi disk" | grep -o "sd .*" | cut -d ' ' -f 1,2,6 | tr ' ' '.' `";
#looking for:    sd 2:0:0:0: Attached scsi disk sda
#echo SATA_SCAN found: $SATA_SCAN


#get the list of USB drives
for ONE in $SATA_SCAN
do
	DRIVE=`echo $ONE | grep -o "sd[a-z].*"`
	CTRLR=`echo $ONE | cut -d '.' -f 2 | cut -d ':' -f 1`
	#echo $DRIVE on $CTRLR hopefully in $SATACTRLRS
	if [ ! "`echo "|$SATACTRLRS" | grep "|${CTRLR}|"`" = "" ]; then
		SATADRIVES="`echo $SATADRIVES $DRIVE | tr ' ' '\n' | sort -u | tr '\n' ' '`";
	fi
done

echo SATADRIVES found: $SATADRIVES


USB_CTRL="`cat /var/log/messages | grep "scsi. : SCSI emulation for USB Mass Storage devices" | grep -o "scsi."`";
#echo USB_CTRL found: $USB_CTRL

USB_DISKS="";
USB_SCAN="`cat /var/log/messages | grep "sd .* Attached scsi removable disk" | grep -o "sd .*" | cut -d ' ' -f 1,2,7 | tr ' ' '.' `";
# looking for:    sd 4:0:0:0: Attached scsi removable disk sdb
#echo USB_SCAN found: $USB_SCAN

#get the list of USB drives
for ONE in $USB_SCAN
do
	DRIVE=`echo $ONE | grep -o "sd[a-z].*"`
	CTRLR=`echo $ONE | cut -d '.' -f 2 | cut -d ':' -f 1`

	#echo $DRIVE on $CTRLR hopefully in $USBCTRLRS

	if [ ! "`echo "|$USBCTRLRS" | grep "|${CTRLR}|"`" = "" ]; then
		USBDRIVES="`echo $USBDRIVES $DRIVE | tr ' ' '\n' | sort -u | tr '\n' ' '`";
	fi
done

echo USBDRIVES found: $USBDRIVES

Jesse

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#13 Post by Jesse »

Wha hey! this init script stuff is getting easier!!! :D

I've made it print out a list of devices relative to the PMEDIA setting, if it contains sata or usb
You can also get more info on the screen if you set the pfix to include 'sd'.

This update tested lots of times on my pc, and snapshot below:

Image

Attachment includes the init script and the elspci file.

Jesse
Attachments
for_initrd.tar.gz
(23.41 KiB) Downloaded 607 times

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#14 Post by Jesse »

Oh the 'pfix=sd' goes in the syslinux.cfg file in the root directory of the flash drive, if you're using the syslinux (default) bootloader.
If this init script gets it wrong, it would be great to see the output when pfix contains 'sd'.

Also, if you can wingle the PUI (Puppy universal installer) to install puppy to SATA hard disk drive, use a PMEDIA=satahd as this will only look in the sd* devices that are actually sata.
If this init script gets it wrong, it would be great to see the output when pfix contains 'sd'.

This update "for_initrd.tar.gz" is NOT official (just like all the rest in this thread) but this one is 'much better', if you're keen and know how to update and try it out, please have a go, and let us know how it went, good and bad feedback are both great to see, so let us know! :wink:

Jesse

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#15 Post by BarryK »

jesse,
I was wondering about timeout for a usb1 interface.
usb1 can take up to 25 seconds for a usb drive to be "ready" after the usb-storage
module is loaded.
I was wondering just where that delay occurs. This dmesg message:
usb-storage: device found at
...does that appear immediately, or in the first second or so?
then:
usb-storage: device scan complete
...this one appears up to 25 seconds later?

...I guessed that is probably how it works, so modified usbstoragecompletionfunc
accordingly, but it's just a guess.

Have you tested on a usb1 system?

User avatar
edoc
Posts: 4729
Joined: Sun 07 Aug 2005, 20:16
Location: Southeast Georgia, USA
Contact:

#16 Post by edoc »

It is great to read that y'all are chasing the USB problem. On our 2.02 desktop the bug also
jams up any effort to write to a FDD or CD.

I've been struggling with this for weeks in a different thread and only today heard that
anyone even recognized that there was a problem.

Almost began to think that my computer had a demon! :-)

I guess the PC is unuseable until the fix is released, unless I wipe it and reinstall an earlier version, perhaps pre-2.xx?

I do worry that I have been referring folks to Puppy and bragging about the 2.xx release.

Thanks! doc
[b]Thanks! David[/b]
[i]Home page: [/i][url]http://nevils-station.com[/url]
[i]Don't google[/i] [b]Search![/b] [url]http://duckduckgo.com[/url]
TahrPup64 & Lighthouse64-b602 & JL64-603

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#17 Post by Jesse »

Hi Barry,

Yes there is a big delay between these two lines:

usb-storage: device found at
usb-storage: device scan complete

Each usb-storage device on the usb has a pair of those two lines.
The "device found at" means there is a usb chip saying "i do usb-storage" and comes up right away, the "device scan complete" means that the chip has finished identifying its device(s) and told the system about them.

My guess is that I've coded it right for 1.1 as well as 2.0. I'll go test it aye! :)

Jesse

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#18 Post by BarryK »

Okay, I did it.
I fired up a usb1 PC, did 'rmmod usb-storage' and I wrote a little script with a
loop that loads usb-storage then reports what's in dmesg every 1 sec.
I plugged in the usb1 flash drive (into the usb1 interface) then immediately
ran the script.

yes, 'usb-storage: device found at' comes up immediately.
I got 'usb-storage: device scan complete' after 6 seconds.

Same thing for a usb2 flash drive plugged into the usb1 interface.

lookin good!

That delay you want beteen loading usb-storage and running the
usbstoragecompleteionfunc, that's not normally needed is it?
Not for flash drive anyway. I think that you thought it may be needed
for usb hard drives.

Jesse
Posts: 466
Joined: Sun 08 May 2005, 16:07
Location: Auckland, NZ

#19 Post by Jesse »

Hi Barry,

Coolies, yep I thought it would work.
I think that 1 sec sleep might be needed beteen loading usb-storage and running the usbstoragecompleteionfunc, as there is no catch if there are zero "device found at" lines, might have to check that one out.

I've also been having a bit of success with getting SCSI disk to boot, see thread:

http://www.murga.org/~puppy/viewtopic.php?p=62770

with only a few more minor tweaks to the init script, and a few more kb of drivers.

Jesse

User avatar
pakt
Posts: 1157
Joined: Sat 04 Jun 2005, 16:54
Location: Sweden

#20 Post by pakt »

BarryK wrote: yes, 'usb-storage: device found at' comes up immediately.
I got 'usb-storage: device scan complete' after 6 seconds.
Yep, 6 seconds sounds familiar - that's the same figure I came up with when we were getting USB detection to work with the new kernel in 2.00b.

Post Reply