Slacko32/64 700 BETA 2

[gyro]

i placed the code in rc.sysinit ("making the filesystem usable") https://github.com/puppylinux-woof-CE/w ... ff48b77c54

Unfortunately when I tried to boot slacko 6.9.6.1(32) with the latest rc.sysinit from 'rationalise', I couldn't get X to start. Not sure what's going on there.
So, after back tracking to an older rc.sysinit, I can confirm that the non-printing problem is fixed with a

Code: Select all

chmod 1777 /initrd/mnt/tmpfs/tmp

In the absense of the rc.sysinit fix. I added a

Code: Select all

chmod 1777 /mnt/tmpfs/tmp

into init at the point that the directory is created.
This worked. So I have a patch to "init" but I haven't pushed it yet.
I think it's slightly tidier to fix the problem at it's source.
(I also checked, and /tmp is fine in pupmode=13).

gyro

[gyro]

@gcmartin,
There is no reason why you could not have a number of "BOOT_SPECS" files stored on your system, and copy the appropriate one into "initrd.gz" before the next boot, using "file2initrd".
The current "bootspecs" utility is not setup to do that.

gyro

[jlst]

gyro i went ahead and pushed this commit: https://github.com/puppylinux-woof-CE/w ... 2705ec79fe

i noticed /tmp is a symlink to /initrd/mnt/tmpfs/tmp only when there is a pupsave, so i created an empty savefile for a usb to test this... then i was faced with the upgrade option... it didn't do well with an empty savefile.... so i think i'll add a check to prevent upgrades when the old version = ""

[gyro]

i placed the code in rc.sysinit ("making the filesystem usable") https://github.com/puppylinux-woof-CE/w ... ff48b77c54

Unfortunately when I tried to boot slacko 6.9.6.1(32) with the latest rc.sysinit from 'rationalise', I couldn't get X to start. Not sure what's going on there.

Sorry, my bad.
After downloading I forgot to check it had execute permisssions.
Strange how things don't work quite right when rc.sysint can't execute.

Still leaves the question, fix the /initrd/mnt/tmpfs/tmp permissions bug in "init" or "rc.sysinit"?

Edit: well you've patched init anyway.

gyro

[gyro]

it didn't do well with an empty savefile.... so i think i'll add a check to prevent upgrades when the old version = ""

My experience is that puppy has never done well with an empty savefile or savefolder. So it's a wider issue that just upgrades. Should really reject empty savefile/folder erlier that in upgrade code.
Perhaps we could reject any savefile/folder that doesn't contain "/initrd/DISTRO_SPECS"?
In "setup_save_file()", immediately after the "#[ "PUPSAVE" ] && decrypt directory at SAVE_LAYER", line.

gyro

[zigbert]

Major issue in 6.9.6.1 - both 32/64-bit

Piping output of find into grep fails.

pMusic (and probably other scripts) has used this mechanism for years, so it would have been detected if existing in previous Puppies.

# find /mnt/sdb1/musikk/ | wc -l
70326
# find /mnt/sdb1/musikk/ -name *mp3 | wc -l
64658
# find /mnt/sdb1/musikk/ | grep mp3 | wc -l
8506
# find /mnt/sdb1/musikk/ -name *mp3 | grep mp3 | wc -l
64658

The 'red' command ends with the notice (not sent to stderr):
Binary file (standard input) matches

Code: Select all

#find /mnt/sdb1/musikk/ 2> /dev/null | grep mp3 2> /dev/null
[...]
/mnt/sdb1/musikk/mp3/Blink 182 - Point of view.mp3
/mnt/sdb1/musikk/mp3/Johnny Bode - Det är så underbart at...mp3
Binary file (standard input) matches

Any clues?

Using the -a switch forces grep to read binary files (corrupted files) as text. This partly fixes the issue.

Code: Select all

# find /mnt/sdb1/musikk/ | grep -a mp3 | wc -l
65154

The next problem was awk that removes duplicated lines depending on the value in a column by

Code: Select all

awk -F'|' '!a[$1]++'

Now fixed by defining LANG=C

Code: Select all

LANG=C awk -F'|' '!a[$1]++'

I will update pMusic

Next chapter:
Grep is the troublemaker, and it troubles with special chars.

With default settings (Slacko 6.9.6.1):

Code: Select all

# echo "$LANG"
en_US.UTF-8
# grep mp3 ./index_mymusic.bak | wc -l           
14680
#
# LANG=en_US.UTF-8 grep mp3 ./index_mymusic.bak | wc -l
14680

With LANG set to something not UTF

Code: Select all

# LANG=en_US grep mp3 ./index_mymusic.bak | wc -l      
70971
# LANG=C grep mp3 ./index_mymusic.bak | wc -l    
70971

[zigbert]

In Slacko 6.3.2 grep works as expected, but it includes an older version of grep - 2.14.
Grep in Slacko 6.9.6.1 is version 2.25

[01micko]

In Slacko 6.3.2 grep works as expected, but it includes an older version of grep - 2.14.
Grep in Slacko 6.9.6.1 is version 2.25

Did you happen to try busybox grep? You could perhaps use it on the condition of the 'grep -V'. IDK if it would work because I only have one album that has special chars and that seem to work anyway. (Just a note - these alphas ship busybox 1.25.0)

BTW - 'streamripper' package was broken. I'll fix it soon.

[zigbert]

The following text is for grep testing. I have also attached the same text in a file in case copy/pasting converts it (copy/paste worked for me). The gz is fake - it is a plain text file.

Code: Select all

line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3

Here is my result with plain grep and with the -a switch

Code: Select all

# grep mp3 /root/forum      
line 1.mp3
Binary file /root/forum matches
#
# grep -a mp3 /root/forum   
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3

And the result with(out) UTF-8

Code: Select all

# LANG=en_US grep mp3 /root/forum
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3
#
# LANG=en_US.UTF-8 grep mp3 /root/forum
line 1.mp3
Binary file /root/forum matches
# 

How do I know if I use the busybox alternative?

Code: Select all

# grep --version
grep (GNU grep) 2.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

[01micko]

zigbert - My result is same as yours (slacko64-6.9.6.1) but I added busybox grep invocation.. works as expected.

Code: Select all

# grep -V                  
grep (GNU grep) 2.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
# grep mp3 /root/forum     
line 1.mp3
Binary file /root/forum matches
# grep -a mp3 /root/forum  
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3
# LANG=en_US grep mp3 /root/forum
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3
# LANG=en_US.UTF-8 grep mp3 /root/forum
line 1.mp3
Binary file /root/forum matches
# busybox grep mp3 /root/forum         
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3
# echo $LANG
en_AU.UTF-8

[zigbert]

I see... busybox grep works as expected ... but that doesn't really help, does it? I understand that busybox is used during startup, but we use the full_bloated_grep doesn't we?

On the other hand this might not interfere much with the Puppy core since no special chars is used in the scripts itself. Maybe pFind and other scripts handling personal file(names) will suffer ...

[Sailor Enceladus]

@zigbert: What do you suggest? Make busybox grep the default, always use grep -a, or revert all new puppies to grep-2.14?

[01micko]

I see... busybox grep works as expected ... but that doesn't really help, does it? I understand that busybox is used during startup, but we use the full_bloated_grep doesn't we?

'busybox' is twice in the system.. a static version in the initrd.gz and the version compiled with shared libs in the main system. This is how it has always been since puppy-2.x series.

Of course busybox is a multicall binary. It has an enormous amount of symlinks that mimic full versions and depending on the configuration of busybox many more too including 'grep' which is why when you invoke 'busybox grep [some pattern] /some/file' it more or less behaves like real 'grep'.

On the other hand this might not interfere much with the Puppy core since no special chars is used in the scripts itself. Maybe pFind and other scripts handling personal file(names) will suffer ...

Further to this, I have found that encoding of your test file makes a huge difference...

If it is encoded with unicode... (with geany > document > set encoding)

Code: Select all

# file /root/forum
/root/forum: UTF-8 Unicode text
# grep mp3 /root/forum        
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - Surfin´ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3

or ISO-8859

Code: Select all

# file /root/forum
/root/forum: ISO-8859 text
# grep mp3 /root/forum
line 1.mp3
Binary file /root/forum matches

I suspect that grep is not the problem but encoding is and perhaps some setting in slacko.

The plot thickens.

[zigbert]

Further to this, I have found that encoding of your test file makes a huge difference...

[...]

I suspect that grep is not the problem but encoding is and perhaps some setting in slacko.

Good catch!

I made a test to verify - switch the grep in this Slacko alpha and the latest stable:
- Run grep 2.14 in Slacko 6.9.6.1 --> Grep finds all lines in the testfile
- Run grep 2.25 in Slacko 6.3.2 --> Grep halts after line 1

Conclusion:
It is grep that troubles with the encoding. Both Slacko's behave the same with equel versions of grep.

[radky]

Most likely, the current issue of failed 'grep' in Slacko-7 is due to the new binary-detection heuristics initially implemented in grep 2.21 (2014-11-23).

In some locales, the new grep may interpret special characters as encoding errors and, by design, parsing of the input string simply aborts with truncated output.

For scripts that may process special characters, the GNU grep developers suggest setting the system's locale to a unibyte locale where all bytes are valid, or editing the affected scripts to parse input strings as binary data (grep -a).

GNU bug reports for grep >= 2.21

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22838
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=24347

[01micko]

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22838

A rather in depth disscussion there. Thanks radky!

Looks like grep-2.25 *might* fix the issue. Building now; packages soon.

[zigbert]

setting the system's locale to a unibyte locale

What does this mean? How do I do that?

[01micko]

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22838

A rather in depth disscussion there. Thanks radky!

Looks like grep-2.25 *might* fix the issue. Building now; packages soon.

Of course not! That is the version we have

From the conversation in the link they don't look like reverting so going forward we might need to look at a wrapper for grep>=2.21 to preserve backward compatibility.

Surely pFind/pFilesearch will suffer as well as many others.

[Sailor Enceladus]

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22838

A rather in depth disscussion there. Thanks radky!

Looks like grep-2.25 *might* fix the issue. Building now; packages soon.

I tried making a grep-2.25 with the 0001patch in message #64. It just skips the three lines in zigberts post then keeps going now, which is still not ideal. So even though the issue is closed, I don't think the "patch" they posted fixes the issue entirely.

Code: Select all

# grep mp3 /root/forum
line 1.mp3
line 5.mp3
Binary file /root/forum matches
# grep -a mp3 /root/forum
line 1.mp3
Gartnerlosjen - Tøffe tak Tute.mp3
The Beach Boys - SurfinŽ U.S.A.mp3
Mötley Crüe - Primal Scream.mp3
line 5.mp3

[01micko]

setting the system's locale to a unibyte locale

What does this mean? How do I do that?

Seems a non-UTF locale (in our context) is unibyte. UTF-8 is multibyte.

The first problem with the C library API is the support of multibyte locales. According to the locale encoding, in general, every character is represented by one or more bytes (up to 4 bytes in practice — but use MB_LEN_MAX instead of the number 4 in the code). When every character is represented by only 1 byte, we speak of an “unibyte locale