c-relay/docs/why_musl_fails.md

Why MUSL Compilation Fails: Technical Explanation

The Core Problem

You cannot mix glibc headers/libraries with MUSL's C library. They are fundamentally incompatible at the ABI (Application Binary Interface) level.

What Happens When We Try

musl-gcc -I/usr/include src/main.c -lwebsockets

Step-by-Step Breakdown:

1. musl-gcc includes <libwebsockets.h> from /usr/include/libwebsockets.h

2. libwebsockets.h includes standard C headers:

   #include <string.h>
   #include <stdlib.h>
   #include <stdio.h>
   

3. The system provides glibc's version of these headers (from /usr/include/)

4. glibc's <string.h> includes glibc-specific internal headers:

   #include <bits/libc-header-start.h>
   #include <bits/types.h>
   

5. MUSL doesn't have these bits/ headers - it has a completely different structure:

   • MUSL uses /usr/include/x86_64-linux-musl/ for its headers
   • MUSL's headers are simpler and don't use the bits/ subdirectory structure

6. Compilation fails with:

   fatal error: bits/libc-header-start.h: No such file or directory
   

Why This Is Fundamental

Different C Library Implementations

glibc (GNU C Library):

• Complex, feature-rich implementation
• Uses bits/ subdirectories for platform-specific code
• Larger binary size
• More system-specific optimizations

MUSL:

• Minimal, clean implementation
• Simpler header structure
• Smaller binary size
• Designed for static linking and portability

ABI Incompatibility

Even if headers compiled, the Application Binary Interface (ABI) is different:

• Function calling conventions may differ
• Structure layouts may differ
• System call wrappers are implemented differently
• Thread-local storage mechanisms differ

The Solution: Build Everything with MUSL

To create a true MUSL static binary, you must:

1. Build libwebsockets with musl-gcc

git clone https://github.com/warmcat/libwebsockets.git
cd libwebsockets
mkdir build && cd build
cmake .. \
    -DCMAKE_C_COMPILER=musl-gcc \
    -DCMAKE_BUILD_TYPE=Release \
    -DLWS_WITH_STATIC=ON \
    -DLWS_WITH_SHARED=OFF \
    -DLWS_WITHOUT_TESTAPPS=ON
make

2. Build OpenSSL with MUSL

wget https://www.openssl.org/source/openssl-3.0.0.tar.gz
tar xzf openssl-3.0.0.tar.gz
cd openssl-3.0.0
CC=musl-gcc ./config no-shared --prefix=/opt/musl-openssl
make && make install

3. Build all other dependencies

• zlib with musl-gcc
• libsecp256k1 with musl-gcc
• libcurl with musl-gcc (which itself needs OpenSSL built with MUSL)

4. Build c-relay with all MUSL libraries

musl-gcc -static \
    -I/opt/musl-libwebsockets/include \
    -I/opt/musl-openssl/include \
    src/*.c \
    -L/opt/musl-libwebsockets/lib -lwebsockets \
    -L/opt/musl-openssl/lib -lssl -lcrypto \
    ...

Why We Use glibc Static Instead

Building the entire dependency chain with MUSL is:

• Time-consuming: Hours to build all dependencies
• Complex: Each library has its own build quirks
• Maintenance burden: Must rebuild when dependencies update
• Unnecessary for most use cases: glibc static binaries work fine

glibc Static Binary Advantages:

✅ Still fully static - no runtime dependencies
✅ Works on virtually all Linux distributions
✅ Much faster to build - uses system libraries
✅ Easier to maintain - no custom dependency builds
✅ Same practical portability for modern Linux systems

glibc Static Binary Limitations:

⚠️ Slightly larger than MUSL (glibc is bigger)
⚠️ May not work on very old systems (ancient glibc versions)
⚠️ Not as universally portable as MUSL (but close enough)

Conclusion

MUSL compilation fails because system libraries are compiled with glibc, and you cannot mix glibc and MUSL.

The current approach (glibc static binary) is the pragmatic solution that provides excellent portability without the complexity of building an entire MUSL toolchain.

If true MUSL binaries are needed in the future, the solution is to use Alpine Linux (which uses MUSL natively) in a Docker container, where all system libraries are already MUSL-compiled.