Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 18 Apr 2014, 04:19
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Compiling with musl (an alternate libc)
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 5 [71 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Author Message
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Wed 16 May 2012, 14:28    Post subject:  

jamesbond wrote:
I was thinking along you'd post a build script on github.com/idunham/musl that will download source packages (the ones you've listed) and relevant patches to make it compile with musl; and compile it to some location (/opt/musl-09 is good)

look at github.com/idunham/src-musl
Currently handles linux headers, musl, zlib, ncurses, and libnl.
Need to add openssl, libssh2, libcurl, maybe c-ares or such...
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1875
Location: The Blue Marble

PostPosted: Fri 18 May 2012, 00:31    Post subject:  

Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.

cheers!

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Sat 19 May 2012, 20:42    Post subject:  

jamesbond wrote:
Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.

Thanks for the news!
Mentioned this on IRC, and Rich commented "if you bet that any given piece of software had a security vulnerability when linked to diet, you'd probably come out ahead."
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4134

PostPosted: Sun 03 Jun 2012, 00:23    Post subject:  

so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewtopic.php?p=631399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Sun 03 Jun 2012, 18:00    Post subject:  

technosaurus wrote:
so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewtopic.php?p=631399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?

Yes PLEASE! Very Happy
There aren't many of the musl distros that have X11 really working, and the build approach for Sabotage (the main distro) are fairly similar.

On an unrelated note:
musl 0.9.1 has just been released.
At this point it's anticipated that glibc binaries not using pthreads and from source conforming to POSIX should mostly work with musl (ie, partial LSB ABI support).
There are several regex fixes, which among other things will allow native builds of ncurses with busybox sed...
The ARM port has several stability fixes.
In other words: It's a big improvement over 0.9.0, and is pretty much mandatory if you're using musl on ARM.
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4134

PostPosted: Sun 03 Jun 2012, 21:46    Post subject:  

It's a static only gcc wrapper tool chain if that matters. I only build standalone static binaries or multicall app collections.

I am hoping to get a llvm based toolchain though, so each architecture only needs the command to convert the llvm bitcode into machine code and the main build can be on a heavy duty x86 (anyone have one they'd like to donate Smile ?). Then an ARM/mips/openrisc build would only need an initramfs with that and a multicall bitcode megafile to bootstrap itself. The whole thing gets a lot less complex with a single multicall "bit-ary" or at least static binaries (shared libs and dependencies just complicates things on the user end)

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Mon 04 Jun 2012, 15:08    Post subject:  

technosaurus wrote:
It's a static only gcc wrapper tool chain if that matters. I only build standalone static binaries or multicall app collections.

Not a problem. What I'm particularly curious about though is the tinyX11 part...(especially the source code and whether an X server works)
Quote:
I am hoping to get a llvm based toolchain though, so each architecture only needs the command to convert the llvm bitcode into machine code and the main build can be on a heavy duty x86 (anyone have one they'd like to donate Smile ?). Then an ARM/mips/openrisc build would only need an initramfs with that and a multicall bitcode megafile to bootstrap itself. The whole thing gets a lot less complex with a single multicall "bit-ary" or at least static binaries (shared libs and dependencies just complicates things on the user end)

Just yet, there's only ARM/AMD64/i386 ports of musl.
But you may want to look at http://ellcc.org if you want musl cross-compilers based on llvm.
Do you think that the llvm bytecode interpreter would work as init? I suspect that you'd want a real init, which does an absolute minimum before calling the interpreter.
FYI, ack uses the em bytecode interpreter, so that's a much smaller toolchain/dependency.

Of course, the other approach (maybe not as good?) is to try getting fatelf working...

Last edited by Ibidem on Mon 04 Jun 2012, 21:03; edited 1 time in total
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4134

PostPosted: Mon 04 Jun 2012, 19:57    Post subject:  

Ibidem wrote:
Of course, the other approach (maybe not as good?) is to try getting fatelf working...
would that be code named santa?

I guess my idea is not possible currently
http://llvm.org/docs/FAQ.html#platformindependent
The llvm bitcode would need to store info regarding the preprocessed #if* macros that set a lot of machine independent variables. Its a real shame because, not only would it enable more platforms, users could get extra performance on newer CPUs ... or GPUs for that matter, now that it supports openCL. The sizeof issue could be resolve by reducing them to a named global constant such as arch_sizeof_int instead of just 4, so that it would get the correct value at machine code build time. Oh well, at least it is modular, so maybe, eventually. I am reminded of a little document I read a while back that talked about having multiple functions at the same address space and it executed depending on the instructions available - I wish I could remember more detail.

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Mon 04 Jun 2012, 21:02    Post subject:  

I haven't tried ack + musl, but that could work.
(T)ACK is short for the Amsterdam Compiler Kit, what Minix has historically used. The main compiler is for C, and ACK goes $LANGUAGE > em "assembler" >em bytecode
Full programs, libraries, or object files can be em bytecode.
An em object file can be linked with em libraries or converted to native object code and linked to native libraries.
em can be used as compiler or interpreter.

fatelf http://icculus.org/fatelf/ is a project where 1 file holds multiple binaries for different architectures (like OS/X Universal binaries, but with ELF instead of Mach).
So you can have one huge ELF binary that can run natively on x86, amd64, armhf, armel, mips(el), power, and so on.
You'd probably have to update/port all the patches needed.
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1875
Location: The Blue Marble

PostPosted: Wed 20 Jun 2012, 02:13    Post subject:  

I tried bootstrapping sabotage linux from rofl0r's repo. Works fine, I've ended with 900MB rootfs (for amd64 platform, not including the /src folder) when I installed pkg and xorg. Xorg boots fine but with only twm inside I can't stay for too long there.

I tried building something too but it failed with gm_tmoff or something --- but I can't for the life of me remember what I tried to build, sorry. When I remember I'll let you know.

It's interesting that with musl as the system library, one can avoid the usual 3-stage gcc bootstrap? I see it only builds gcc3, then uses that gcc3 to build the system gcc4.

I tried to cross compile (platform amd64, build i386) but apparently it doesn't work, and since I'm novice in this cross compiling thing I'll try easier stuff first ...

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Thu 21 Jun 2012, 21:46    Post subject:  

Re Sabotage:
the server in xorg is actually Xfbdev.
Most deliberate musl users currently would rather use the shell than X, afaict.
musl seems to be appreciated by the folks at suckless, though it isn't a suckless project. TWM is considered a decent WM there. I don't *enjoy* it, but it is reasonably usable for me. I'd rather use twm than some of the new bloated eyecandy-laden DEs, myself.
Other WMs can be compiled; there's "Snowflake Linux" with xfce4, and in the pupngo thread someone mentioned getting jwm linked against tinyX.
Re gcc:
1. configure with --enable-bootstrap will do a full 3-stage compile of gcc.
2. The 3 stage thing is not so much a hard requirement as an attempt to break the "trusting trust" attack (you may trust the source, but can you trust that the compiler creates the right binary? A hacked compiler could inject code that isn't in the source.)

Cross-compiling: musl or sabotage?
with musl it's ./configure --target=i386 after exporting CC=gcc CFLAGS=-m32
(AFAICT; CC="gcc -m32" seems not to work yet; -m32 _might_ be automatically figured out)

FWIW, you can have a working system using just a 1 MB static build of busybox (not possible with glibc, I know!)
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1875
Location: The Blue Marble

PostPosted: Fri 22 Jun 2012, 09:23    Post subject:  

Thanks Ibidem.

Ibidem wrote:
Re Sabotage:
the server in xorg is actually Xfbdev.
Most deliberate musl users currently would rather use the shell than X, afaict.
musl seems to be appreciated by the folks at suckless, though it isn't a suckless project. TWM is considered a decent WM there. I don't *enjoy* it, but it is reasonably usable for me. I'd rather use twm than some of the new bloated eyecandy-laden DEs, myself.
Other WMs can be compiled; there's "Snowflake Linux" with xfce4, and in the pupngo thread someone mentioned getting jwm linked against tinyX.
I guess I just have long forgotten the twm shortcuts and gestures. I just want to see how far sabotage is up to. Would be interesting to see if it can compile gtk - if it does, I can probably bring openbox online. I am a silent listener on the upupngo too Smile

Quote:

Re gcc:
1. configure with --enable-bootstrap will do a full 3-stage compile of gcc.
No, no, I'm perfectly happy with no bootstrap if I can end up with a working system (especially when cross compiling). I'm just surprised that musl can get away *without* doing it - I thought it is a hard requirement for gcc. Probably it is a hard requirement for glibc.

Quote:
2. The 3 stage thing is not so much a hard requirement as an attempt to break the "trusting trust" attack (you may trust the source, but can you trust that the compiler creates the right binary? A hacked compiler could inject code that isn't in the source.)
I never thought that to be the case, I though it is more to ensure that none of the host system libraries get carried out (as dependency) to the target system.

Quote:
Cross-compiling: musl or sabotage?
Sorry I wasn't clear. It was cross compiling sabotage I was asking. I was trying to get i386 target from my x86_64 host system, but the longer term idea is to get ARM to build from any intel system (x86_64 or i386).

Quote:
with musl it's ./configure --target=i386 after exporting CC=gcc CFLAGS=-m32
(AFAICT; CC="gcc -m32" seems not to work yet; -m32 _might_ be automatically figured out)
Thanks, that works. I managed to cross-compile musl from x86_64 to 32bit using -m32. I think some (broken) apps would need LDFLAGS=-m32 too.

Quote:
FWIW, you can have a working system using just a 1 MB static build of busybox (not possible with glibc, I know!)

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
Ibidem

Joined: 25 May 2010
Posts: 429
Location: State of Jefferson

PostPosted: Sun 01 Jul 2012, 16:27    Post subject:  

I guess cross-compiling starts with building a toolchain, while sabotage starts with building musl...(I might be wrong about how sabotage does it).
Step 1 would be build a cross-binutils (--target=arm-unknown-linux, I think), then a cross-gcc (same target, --with-binutils=$BINUTILS_PREFIX, again IIRC).
Then build arm musl, etc.
If you can get a root fs, try it in qemu. You can build some things under qemu.
This is more-or-less what Aboriginal Linux does.

Also, have you gotten GTK yet? it should be possible, though Snowflake would be the place to look.


Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization. I haven't got X on here yet, though. Just busybox + build + {links2.7, ncurses, libedit, readline, bible (as in Debian's bible-kjv package), cmix, minimp3, libowfat, dietsniff, curl, ncurses, openssl, openssh, libssh2, lynx}
I also have kerberos5 (MIT 1.8.x), I could build vim, mpg123, and several other packages. Probably the heirloom toolchest and vim will be the next ones to add...
Back to top
View user's profile Send private message 
lmemsm

Joined: 27 Jun 2012
Posts: 11

PostPosted: Mon 02 Jul 2012, 07:35    Post subject:  

Ibidem wrote:
Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization. I haven't got X on here yet, though. Just busybox + build + {links2.7, ncurses, libedit, readline, bible (as in Debian's bible-kjv package), cmix, minimp3, libowfat, dietsniff, curl, ncurses, openssl, openssh, libssh2, lynx}
I also have kerberos5 (MIT 1.8.x), I could build vim, mpg123, and several other packages. Probably the heirloom toolchest and vim will be the next ones to add...


If you end up sharing your distribution anywhere, hope you'll post about it. I have an old laptop and I'm always looking for ways to improve performance on it. There is a DeLicate Linux ( http://delicate-linux.net/ ) which uses ulibc in place of glibc. Tried it, but there are a lot of programs that won't compile because of compatiblity issues (or incompatiblity issues). I'm assuming musl would have some of the same issues? One interesting thing I read in a comparison of the different libc alternatives ( http://www.etalabs.net/compare_libcs.html ) was that musl attempts to be backward compatible. That's something I look for in a library.

Also wondering if FLTK 1.3, SDL and wxWidgets (SDL or X based version) would work with musl. I'm working on finding enough alternative programs that between them curses and command line options, I'm hoping to not need any GTK+ or Qt based options. SDL will run in Framebuffer mode outside of X using directfb. There's also a port of FLTK that runs in Framebuffer mode with directfb, but it's not the actively developed version (1.3). I've also read about the FLTK 1.3 version working on top of nano-x. Might be very interesting to see just how many GUI applications would run okay outside of X.

Look forward to reading about your progress using musl.
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 1875
Location: The Blue Marble

PostPosted: Mon 02 Jul 2012, 09:26    Post subject:  

Ibidem wrote:
I guess cross-compiling starts with building a toolchain, while sabotage starts with building musl...(I might be wrong about how sabotage does it).
Step 1 would be build a cross-binutils (--target=arm-unknown-linux, I think), then a cross-gcc (same target, --with-binutils=$BINUTILS_PREFIX, again IIRC).
Then build arm musl, etc.
That's what I read in LFS and CLFS. Sabotage (rofl0r fork at least) builds "butch" first (butch is a cross between package manager and autobuild system), and then tried to build gcc3 directly, which bombed out. I was trying for the simpler case first with x86_64 -> i386. I added "--target i386-sabotage-linux-gnu" - gcc3 complained of assembler opcodes, so I added MULTILIB_CFLAGS="-Wa,--32" -- and then it complained again about missing headers whose name sound like kernel headers. So I stopped there.

If gcc3 build is successful, next build would be musl and then binutils; so I think sabotage get it backwards Laughing It works though if host platform == target platform.

Quote:
Also, have you gotten GTK yet? it should be possible, though Snowflake would be the place to look.
I have not, too busy trying the cross-compile to work. Anyway, I think I should try that using native sabotage build (which works atm).

Quote:
Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization.
Yeah, when I managed to get sabotage x86_64 rootfs, it won't boot the system from SD card unless I gave the rootwait / rootdelay parameter ... Rolling Eyes Fast kernel, slow usb devices Laughing
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 5 [71 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0960s ][ Queries: 13 (0.0058s) ][ GZIP on ]