Compiling with musl (an alternate libc)

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Message
Author
Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#16 Post by Ibidem »

jamesbond wrote:I was thinking along you'd post a build script on github.com/idunham/musl that will download source packages (the ones you've listed) and relevant patches to make it compile with musl; and compile it to some location (/opt/musl-09 is good)
look at github.com/idunham/src-musl
Currently handles linux headers, musl, zlib, ncurses, and libnl.
Need to add openssl, libssh2, libcurl, maybe c-ares or such...

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#17 Post by jamesbond »

Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#18 Post by Ibidem »

jamesbond wrote:Got it, thanks. I will play with it. BTW - musl (plain musl without the additional libs) will be in Fatdog devx, replacing dietlibc.
Thanks for the news!
Mentioned this on IRC, and Rich commented "if you bet that any given piece of software had a security vulnerability when linked to diet, you'd probably come out ahead."

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#19 Post by technosaurus »

so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewto ... 399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#20 Post by Ibidem »

technosaurus wrote:so, I used the latest git to build mupdf
http://www.murga-linux.com/puppy/viewto ... 399#631399

the difference between a static build with glibc and X11 vs musl and tinyx11 is huge

would anyone like me to try and package up the toolchain?
Yes PLEASE! :D
There aren't many of the musl distros that have X11 really working, and the build approach for Sabotage (the main distro) are fairly similar.

On an unrelated note:
musl 0.9.1 has just been released.
At this point it's anticipated that glibc binaries not using pthreads and from source conforming to POSIX should mostly work with musl (ie, partial LSB ABI support).
There are several regex fixes, which among other things will allow native builds of ncurses with busybox sed...
The ARM port has several stability fixes.
In other words: It's a big improvement over 0.9.0, and is pretty much mandatory if you're using musl on ARM.

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#21 Post by technosaurus »

It's a static only gcc wrapper tool chain if that matters. I only build standalone static binaries or multicall app collections.

I am hoping to get a llvm based toolchain though, so each architecture only needs the command to convert the llvm bitcode into machine code and the main build can be on a heavy duty x86 (anyone have one they'd like to donate :-) ?). Then an ARM/mips/openrisc build would only need an initramfs with that and a multicall bitcode megafile to bootstrap itself. The whole thing gets a lot less complex with a single multicall "bit-ary" or at least static binaries (shared libs and dependencies just complicates things on the user end)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#22 Post by Ibidem »

technosaurus wrote:It's a static only gcc wrapper tool chain if that matters. I only build standalone static binaries or multicall app collections.
Not a problem. What I'm particularly curious about though is the tinyX11 part...(especially the source code and whether an X server works)
I am hoping to get a llvm based toolchain though, so each architecture only needs the command to convert the llvm bitcode into machine code and the main build can be on a heavy duty x86 (anyone have one they'd like to donate :-) ?). Then an ARM/mips/openrisc build would only need an initramfs with that and a multicall bitcode megafile to bootstrap itself. The whole thing gets a lot less complex with a single multicall "bit-ary" or at least static binaries (shared libs and dependencies just complicates things on the user end)
Just yet, there's only ARM/AMD64/i386 ports of musl.
But you may want to look at http://ellcc.org if you want musl cross-compilers based on llvm.
Do you think that the llvm bytecode interpreter would work as init? I suspect that you'd want a real init, which does an absolute minimum before calling the interpreter.
FYI, ack uses the em bytecode interpreter, so that's a much smaller toolchain/dependency.

Of course, the other approach (maybe not as good?) is to try getting fatelf working...
Last edited by Ibidem on Tue 05 Jun 2012, 01:03, edited 1 time in total.

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#23 Post by technosaurus »

Ibidem wrote:Of course, the other approach (maybe not as good?) is to try getting fatelf working...
would that be code named santa?

I guess my idea is not possible currently
http://llvm.org/docs/FAQ.html#platformindependent
The llvm bitcode would need to store info regarding the preprocessed #if* macros that set a lot of machine independent variables. Its a real shame because, not only would it enable more platforms, users could get extra performance on newer CPUs ... or GPUs for that matter, now that it supports openCL. The sizeof issue could be resolve by reducing them to a named global constant such as arch_sizeof_int instead of just 4, so that it would get the correct value at machine code build time. Oh well, at least it is modular, so maybe, eventually. I am reminded of a little document I read a while back that talked about having multiple functions at the same address space and it executed depending on the instructions available - I wish I could remember more detail.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#24 Post by Ibidem »

I haven't tried ack + musl, but that could work.
(T)ACK is short for the Amsterdam Compiler Kit, what Minix has historically used. The main compiler is for C, and ACK goes $LANGUAGE > em "assembler" >em bytecode
Full programs, libraries, or object files can be em bytecode.
An em object file can be linked with em libraries or converted to native object code and linked to native libraries.
em can be used as compiler or interpreter.

fatelf http://icculus.org/fatelf/ is a project where 1 file holds multiple binaries for different architectures (like OS/X Universal binaries, but with ELF instead of Mach).
So you can have one huge ELF binary that can run natively on x86, amd64, armhf, armel, mips(el), power, and so on.
You'd probably have to update/port all the patches needed.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#25 Post by jamesbond »

I tried bootstrapping sabotage linux from rofl0r's repo. Works fine, I've ended with 900MB rootfs (for amd64 platform, not including the /src folder) when I installed pkg and xorg. Xorg boots fine but with only twm inside I can't stay for too long there.

I tried building something too but it failed with gm_tmoff or something --- but I can't for the life of me remember what I tried to build, sorry. When I remember I'll let you know.

It's interesting that with musl as the system library, one can avoid the usual 3-stage gcc bootstrap? I see it only builds gcc3, then uses that gcc3 to build the system gcc4.

I tried to cross compile (platform amd64, build i386) but apparently it doesn't work, and since I'm novice in this cross compiling thing I'll try easier stuff first ...
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#26 Post by Ibidem »

Re Sabotage:
the server in xorg is actually Xfbdev.
Most deliberate musl users currently would rather use the shell than X, afaict.
musl seems to be appreciated by the folks at suckless, though it isn't a suckless project. TWM is considered a decent WM there. I don't *enjoy* it, but it is reasonably usable for me. I'd rather use twm than some of the new bloated eyecandy-laden DEs, myself.
Other WMs can be compiled; there's "Snowflake Linux" with xfce4, and in the pupngo thread someone mentioned getting jwm linked against tinyX.
Re gcc:
1. configure with --enable-bootstrap will do a full 3-stage compile of gcc.
2. The 3 stage thing is not so much a hard requirement as an attempt to break the "trusting trust" attack (you may trust the source, but can you trust that the compiler creates the right binary? A hacked compiler could inject code that isn't in the source.)

Cross-compiling: musl or sabotage?
with musl it's ./configure --target=i386 after exporting CC=gcc CFLAGS=-m32
(AFAICT; CC="gcc -m32" seems not to work yet; -m32 _might_ be automatically figured out)

FWIW, you can have a working system using just a 1 MB static build of busybox (not possible with glibc, I know!)

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#27 Post by jamesbond »

Thanks Ibidem.
Ibidem wrote:Re Sabotage:
the server in xorg is actually Xfbdev.
Most deliberate musl users currently would rather use the shell than X, afaict.
musl seems to be appreciated by the folks at suckless, though it isn't a suckless project. TWM is considered a decent WM there. I don't *enjoy* it, but it is reasonably usable for me. I'd rather use twm than some of the new bloated eyecandy-laden DEs, myself.
Other WMs can be compiled; there's "Snowflake Linux" with xfce4, and in the pupngo thread someone mentioned getting jwm linked against tinyX.
I guess I just have long forgotten the twm shortcuts and gestures. I just want to see how far sabotage is up to. Would be interesting to see if it can compile gtk - if it does, I can probably bring openbox online. I am a silent listener on the upupngo too :)
Re gcc:
1. configure with --enable-bootstrap will do a full 3-stage compile of gcc.
No, no, I'm perfectly happy with no bootstrap if I can end up with a working system (especially when cross compiling). I'm just surprised that musl can get away *without* doing it - I thought it is a hard requirement for gcc. Probably it is a hard requirement for glibc.
2. The 3 stage thing is not so much a hard requirement as an attempt to break the "trusting trust" attack (you may trust the source, but can you trust that the compiler creates the right binary? A hacked compiler could inject code that isn't in the source.)
I never thought that to be the case, I though it is more to ensure that none of the host system libraries get carried out (as dependency) to the target system.
Cross-compiling: musl or sabotage?
Sorry I wasn't clear. It was cross compiling sabotage I was asking. I was trying to get i386 target from my x86_64 host system, but the longer term idea is to get ARM to build from any intel system (x86_64 or i386).
with musl it's ./configure --target=i386 after exporting CC=gcc CFLAGS=-m32
(AFAICT; CC="gcc -m32" seems not to work yet; -m32 _might_ be automatically figured out)
Thanks, that works. I managed to cross-compile musl from x86_64 to 32bit using -m32. I think some (broken) apps would need LDFLAGS=-m32 too.
FWIW, you can have a working system using just a 1 MB static build of busybox (not possible with glibc, I know!)
Image
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#28 Post by Ibidem »

I guess cross-compiling starts with building a toolchain, while sabotage starts with building musl...(I might be wrong about how sabotage does it).
Step 1 would be build a cross-binutils (--target=arm-unknown-linux, I think), then a cross-gcc (same target, --with-binutils=$BINUTILS_PREFIX, again IIRC).
Then build arm musl, etc.
If you can get a root fs, try it in qemu. You can build some things under qemu.
This is more-or-less what Aboriginal Linux does.

Also, have you gotten GTK yet? it should be possible, though Snowflake would be the place to look.


Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization. I haven't got X on here yet, though. Just busybox + build + {links2.7, ncurses, libedit, readline, bible (as in Debian's bible-kjv package), cmix, minimp3, libowfat, dietsniff, curl, ncurses, openssl, openssh, libssh2, lynx}
I also have kerberos5 (MIT 1.8.x), I could build vim, mpg123, and several other packages. Probably the heirloom toolchest and vim will be the next ones to add...

lmemsm
Posts: 51
Joined: Wed 27 Jun 2012, 15:01

#29 Post by lmemsm »

Ibidem wrote:Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization. I haven't got X on here yet, though. Just busybox + build + {links2.7, ncurses, libedit, readline, bible (as in Debian's bible-kjv package), cmix, minimp3, libowfat, dietsniff, curl, ncurses, openssl, openssh, libssh2, lynx}
I also have kerberos5 (MIT 1.8.x), I could build vim, mpg123, and several other packages. Probably the heirloom toolchest and vim will be the next ones to add...
If you end up sharing your distribution anywhere, hope you'll post about it. I have an old laptop and I'm always looking for ways to improve performance on it. There is a DeLicate Linux ( http://delicate-linux.net/ ) which uses ulibc in place of glibc. Tried it, but there are a lot of programs that won't compile because of compatiblity issues (or incompatiblity issues). I'm assuming musl would have some of the same issues? One interesting thing I read in a comparison of the different libc alternatives ( http://www.etalabs.net/compare_libcs.html ) was that musl attempts to be backward compatible. That's something I look for in a library.

Also wondering if FLTK 1.3, SDL and wxWidgets (SDL or X based version) would work with musl. I'm working on finding enough alternative programs that between them curses and command line options, I'm hoping to not need any GTK+ or Qt based options. SDL will run in Framebuffer mode outside of X using directfb. There's also a port of FLTK that runs in Framebuffer mode with directfb, but it's not the actively developed version (1.3). I've also read about the FLTK 1.3 version working on top of nano-x. Might be very interesting to see just how many GUI applications would run okay outside of X.

Look forward to reading about your progress using musl.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#30 Post by jamesbond »

Ibidem wrote:I guess cross-compiling starts with building a toolchain, while sabotage starts with building musl...(I might be wrong about how sabotage does it).
Step 1 would be build a cross-binutils (--target=arm-unknown-linux, I think), then a cross-gcc (same target, --with-binutils=$BINUTILS_PREFIX, again IIRC).
Then build arm musl, etc.
That's what I read in LFS and CLFS. Sabotage (rofl0r fork at least) builds "butch" first (butch is a cross between package manager and autobuild system), and then tried to build gcc3 directly, which bombed out. I was trying for the simpler case first with x86_64 -> i386. I added "--target i386-sabotage-linux-gnu" - gcc3 complained of assembler opcodes, so I added MULTILIB_CFLAGS="-Wa,--32" -- and then it complained again about missing headers whose name sound like kernel headers. So I stopped there.

If gcc3 build is successful, next build would be musl and then binutils; so I think sabotage get it backwards :lol: It works though if host platform == target platform.
Also, have you gotten GTK yet? it should be possible, though Snowflake would be the place to look.
I have not, too busy trying the cross-compile to work. Anyway, I think I should try that using native sabotage build (which works atm).
Right now I'm using muslin, my own distro (if you can call it that yet!). Most of the boot time is kernel USB initialization.
Yeah, when I managed to get sabotage x86_64 rootfs, it won't boot the system from SD card unless I gave the rootwait / rootdelay parameter ... :roll: Fast kernel, slow usb devices :lol:
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#31 Post by Ibidem »

1-Apparently cross-compiling is not supported--rofl0r says he doesn't want to support it, it's practically impossible/full of pitfalls, and "if you want to cross-compile, use Aboriginal".
2-A lot of the time, #musl on freenode is the quickest way to get ahold of someone who's using sabotage. You might also try the sabotage mailinglist.
3-Tried doing it in an x86 chroot with setarch to fake i386?

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#32 Post by jamesbond »

Ibidem wrote:1-Apparently cross-compiling is not supported--rofl0r says he doesn't want to support it, it's practically impossible/full of pitfalls, and "if you want to cross-compile, use Aboriginal".
But Aboriginal doesn't support creating apps using musl as libc - or does it? Agreed, it is tough if you're doing from scratch. I persisted for a while - trying to build gcc 3.4.6 cross compiler (that was used in stage0 sabotage before rofl0r upgraded it to gcc4 a while back). Got there, sort of - made a cross-gcc that produces 32-bit assembly by default (I have forgotten that this is all that gcc outputs - an assembly language file). Because I haven't made cross-binutils yet, I coaxed my existing 64-bit binutils to work with that (--32 and -m elf-i386) and link with 32-bit musl. It works.

EDIT: I shouldn't have wasted my time, this and this build musl cross-compiler and the scripts are simple enough to understand.
2-A lot of the time, #musl on freenode is the quickest way to get ahold of someone who's using sabotage. You might also try the sabotage mailinglist.
Thank you, yes I should do that.
3-Tried doing it in an x86 chroot with setarch to fake i386?
No, not even aware that "setarch" command exists, I should read more about it. Anyway I don't have any 32-bit development libraries (but could be installed I suppose). This won't work for Intel --> ARM cross, though.

Sorry this is becoming out of topic, we should be discussing musl instead of cross-compiling here. Just tell me so if you don't wish me to continue here.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#33 Post by Ibidem »

Aboriginal: you'd have to hijack the toolchain. Possible, though difficult.

I wouldn't bother cross-compiling sabotage, I think. bootstrap-linux would be better for that.
More-or-less build chain I'd try (as make takes it):

Code: Select all

step1: cross-binutils cross-gcc

#If you don't have perl, apply a couple patches so linux builds with  BB sed, dc.
linux-headers: step1

linux: step1

musl: step1

make: musl

busybox: musl linux-headers

native-binutils: musl linux-headers #not sure about the latter

native-gcc: native-binutils
This should give you what you need by way of binaries; I don't know about a cross-built bootable rootfs...

Talking about cross-compiling is fine, as far as I'm concerned...as long as it isn't totally irrelevant, like uclibc-, glibc-, or diet- specific issues.

@lmemsm:
There are plenty of (in)compatability issues, but they seem to be disappearing fairly fast. Someone just managed to patch musl to the point where they had glxgears running on the proprietary nvidia driver.
Generally,
-try no CFLAGS (ANSI)
-try -D_XOPEN_SOURCE (POSIX/ UNIX2008)
-try -D_BSD_SOURCE (BSD-like API)
The "big hammer" is -D_GNU_SOURCE.

SDL can be built, from what I saw of the sabotage package list.
FLTK hasn't been tried _yet_, AFAICT. It theoretically should work, if C++ has been enabled and the X libs built (all possible, I just got to the point where I could build openmotif).

What sort of packages would you want? I can think of a couple spreadsheets, a wordprocessor or two, half a dozen web browsers (dillo, w3m, retawq, lynx, links, elinks...), a framebuffer image viewer (also supports PS/PDF), and some other things besides.

Ibidem
Posts: 549
Joined: Wed 26 May 2010, 03:31
Location: State of Jefferson

#34 Post by Ibidem »

jamesbond wrote:I persisted for a while - trying to build gcc 3.4.6 cross compiler (that was used in stage0 sabotage before rofl0r upgraded it to gcc4 a while back). Got there, sort of - made a cross-gcc that produces 32-bit assembly by default (I have forgotten that this is all that gcc outputs - an assembly language file). Because I haven't made cross-binutils yet, I coaxed my existing 64-bit binutils to work with that (--32 and -m elf-i386) and link with 32-bit musl. It works.

EDIT: I shouldn't have wasted my time...this build musl cross-compiler and the scripts are simple enough to understand.
That's a git mirror of http://bitbucket.org/GregorR/musl-cross; just today, Gregor added cross-compiler downloads for armeb, arm(el), x86-64 (amd64), and i686 (this toolchain may or may not work with i486).

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#35 Post by jamesbond »

Thanks. Didn't know the main repository is in bitbucket.org. The one in github seems to more than a day late.

I've got i386 (actually its i686) rootfs up and running using bootstrap. Thank you for the recipe, bootstrap seems to follow that closely. Seems that making gcc4 cross-compile is a lot less hassle than gcc3 (though you need to have mpfr, gmp, mpc available ...) - but may be it's just me :) I'll try to coax bootstrap to build arm rootfs too - looks like that all I need to do is change the --target to the appropriate ones; but we'll see how it goes :D

EDIT: Got arm rootfs to boot, using bootstrap linux. I used bootstrap from git (the same as the one I used for i686 cross above).
A few notes:
- (bootstrap) not enough just to specify the arch as "arm", I need to sed "linux-musl" to "linux-musleabi" (the provided patch for gcc is only for arm-eabi).
- (bootstrap) Linux kernel build didn't work, I need to use my own Kconfig instead of the generic linux.config provided. I'm using versatile emulation.
- (musl) musl 0.9.2 doesn't work (busybox segfaults), I need to get from latest git.

bootstrap linux rootfs consist of these:
- busybox (seems to work)
- make (seems to work)
- native binutils - still segfaults
- native gcc - many still segfaults.
In the i686 version all the above works flawlessly :?
Since everything is statically compiled, I would say the failure happens during during musl cross compilation for the cross compiler.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

Post Reply