Ubuntu 20.04 #120

asaaki · 2021-03-20T07:23:02Z

Use latest ubuntu LTS release (20.04)

Also:

Update mdbook to 0.4.7
Update cargo-about to 0.3.0
Update cargo-deny to 0.9.0

Travis CI checks can be found under my PR of the fork: asaaki#1
If you want to use the image already:

FROM ghcr.io/asaaki/rust-musl-builder:latest

Closes #98

asaaki · 2021-03-20T16:55:37Z

Running ./test-image gave me a fail though:

--- Test case for sqlx:
ldd says:
	/lib/ld-musl-x86_64.so.1 (0x7f87879d6000)
[FAIL] Executable is not static!
The command "bash ./test-image" exited with 1.

As you also mention in #98.

But I think that might be a red herring.

Here's an output of another binary (randomly chosen from /usr/bin) which does seem to be not fully static as it needs a libc on the system:

$ ldd /usr/bin/getconf
        /lib/ld-musl-x86_64.so.1 (0x7fa18cd57000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fa18cd57000)

Interwebz says that we should also check the output of readelf and look for INTERP in the headers
It's part of the binutils, so has to be installed into the alpine stage.

$ readelf -l /usr/local/bin/using-sqlx 

Elf file type is DYN (Shared object file)
Entry point 0x63062
There are 10 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000062750 0x0000000000062750  R      0x1000
  LOAD           0x0000000000063000 0x0000000000063000 0x0000000000063000
                 0x00000000002442bf 0x00000000002442bf  R E    0x1000
  LOAD           0x00000000002a8000 0x00000000002a8000 0x00000000002a8000
                 0x000000000013c4d0 0x000000000013c4d0  R      0x1000
  LOAD           0x00000000003e4e20 0x00000000003e5e20 0x00000000003e5e20
                 0x0000000000053440 0x00000000000551e0  RW     0x1000
  DYNAMIC        0x0000000000435c78 0x0000000000436c78 0x0000000000436c78
                 0x0000000000000130 0x0000000000000130  RW     0x8
  NOTE           0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x0000000000000024 0x0000000000000024  R      0x4
  TLS            0x00000000003e4e20 0x00000000003e5e20 0x00000000003e5e20
                 0x0000000000000005 0x00000000000002d8  R      0x20
  GNU_EH_FRAME   0x0000000000377950 0x0000000000377950 0x0000000000377950
                 0x000000000000ba1c 0x000000000000ba1c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x00000000003e4e20 0x00000000003e5e20 0x00000000003e5e20
                 0x00000000000531e0 0x00000000000531e0  R      0x1

using-sqlx has no INTERP there.

While …

readelf -l /usr/bin/getconf 

Elf file type is DYN (Shared object file)
Entry point 0x22e8
There are 10 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000230 0x0000000000000230  R      0x8
  INTERP         0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x0000000000000019 0x0000000000000019  R      0x1
      [Requesting program interpreter: /lib/ld-musl-x86_64.so.1]
<snip>

… does have it, and you can even see a path encoded there.

From https://greek0.net/elf.html:

If an INTERP entry is present, the interpreter is loaded too. Statically linked binaries can do without an interpreter; […]

Another way to check is to run readelf -d <path/to/executable> and look for (NEEDED), only dynamically linked programs should have entries, statically linked ones have none.

And yet another option is to use scanelf from pax-utils (scanelf -n <path>, scanelf -i <path>).

I also ran an example app in different variations of busybox:

FROM busybox
FROM busybox:musl
FROM busybox:uclibs
FROM busybox:glibc

And there are no problems, which I would expect if it was not truly static.

Even FROM scratch works fine.

Fun fact:

ldd on alpine/musl is the following shell script:

$ cat /usr/bin/ldd
#!/bin/sh
exec /lib/ld-musl-x86_64.so.1 --list "$@"

So I believe that this lib/executable is injecting itself into the ldd output.

Last but not least I ran it under a debian:stretch-slim and called ldd there:

ldd /usr/local/bin/using-sqlx
        statically linked

(debian's ldd is a slightly more elaborate script.)

I adjusted the test-image script and the Dockerfiles of both static tests to use debian:stretch-slim for the better ldd checker.

Under asaaki#1 you can see my travis runs, which are green for the proposed changes.

jman-schief · 2021-04-18T12:48:18Z

I've tested this updated image to build the diesel_cli and it seems to work (thanks @asaaki !)

I wouldn't mind seeing this reviewed and merged (notable change: the updated openssl library)

EDIT: worth mentioning that perhaps openssl could also be safely upgraded to latest 1.1.1k (I've tested it and the built binary seems to be fine)

asaaki · 2021-04-19T09:24:19Z

@jman-schief Updated to have 1.1.1k used.

Also updates various dependencies and tools. Further changes (squashed commits): * Use COPY over ADD If you copy local files, then COPY is preferred. ADD should only be used in specifc circumstances like fetching files from URLs or unpacking archives. * Use a more reliable testing for ldd * Use openssl 1.1.1m, move cargo tools

@asaaki

Many thanks to @asaaki for having figured out the static link check was returning flase negatives with Alpine's `lld`. See: #120 (comment) The plan is to merge this when Rust 1.58.0 is released, so people who need 18.04 can stick with1.57.0. Closes #98. Closes #120.

emk · 2022-01-01T21:22:55Z

Thank you to everyone for working on this, and to @asaaki for figuring out the static linking check! I've prepared a new PR inspired by this one but based off the latest main branch. You can find it at #134.

I plan to release this when Rust 1.58.0 is released, to allowing existing users to easily stick with 1.57.0 if they want.

asaaki · 2022-01-01T23:41:26Z

For some of the tiny tweaks and consistency changes do you want me to open new PRs after you merged your update PR?

(btw this one was already rebased to latest main, I can also undo some changes you don't like to make it into a mergeable state)

emk

Thank you for proposing more fixes! That would be very helpful.

To save you some time, I read through your PR, and I marked the fixes I'd definitely like to add to this project! In general, I'm happy to merge small, clean PRs that do a single thing and don't change anything unrelated. If there's even a single part of a PR that I don't want, it makes the whole process a lot more time-consuming for on my end.

In this particular case, please feel free to submit a single new PR with just the changes I've accepted below. Thank you very much for figuring out 20.04, and for the other fixes you've proposed.

emk · 2022-01-02T00:03:49Z

Dockerfile

+# syntax=docker/dockerfile:1
+


We can add this, since Docker uses it in all their latest examples.

emk · 2022-01-02T00:04:51Z

Dockerfile

-# - `mdbook` is the standard Rust tool for making searchable HTML manuals.
-# - `mdbook-graphviz` allows using inline GraphViz drawing commands to add illustrations.
-# - `cargo-about` generates a giant license file for all dependencies.
-# - `cargo-audit` checks for security vulnerabilities. We include it for backwards compat.
-# - `cargo-deny` does everything `cargo-audit` does, plus check licenses & many other things.
-RUN curl -fLO https://github.com/rust-lang-nursery/mdBook/releases/download/v$MDBOOK_VERSION/mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \
-    tar xf mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \
-    mv mdbook /usr/local/bin/ && \
-    rm -f mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \


Please leave this whole section at the top of the file. Thank you!

emk · 2022-01-02T00:06:42Z

Dockerfile

-RUN apt-get update && \
-    export DEBIAN_FRONTEND=noninteractive && \
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get update && \


Please leave out any minor changes that don't change anything, unless the current version is clearly inelegant.

emk · 2022-01-02T00:11:56Z

Dockerfile

-RUN echo "Building OpenSSL" && \
+RUN echo "Building OpenSSL ${OPENSSL_VERSION}" && \


I think it's a great idea to add the version numbers!

As for variable syntax, according to the spec:

Environment variables are notated in the Dockerfile either with $variable_name or ${variable_name}. They are treated equivalently and the brace syntax is typically used to address issues with variable names with no whitespace, like ${foo}_bar.

This project prefers $variable_name when unambiguous, and only uses ${variable_name} when the next character would otherwise be interpreted as part of the name. Let's use $variable_name here and elsewhere.

emk · 2022-01-02T00:16:22Z

Dockerfile

-ADD git-credential-ghtoken /usr/local/bin/ghtoken
+COPY git-credential-ghtoken /usr/local/bin/ghtoken


OK, it looks like COPY is officially preferred, so we can use it when it works.

emk · 2022-01-02T00:16:54Z

Dockerfile

-RUN curl https://sh.rustup.rs -sSf | \
+RUN echo "Installing Rust (${TOOLCHAIN} toolchain)" && \
+    curl https://sh.rustup.rs -sSf | \


👍 for logging the toolchain.

emk · 2022-01-02T00:19:26Z

Dockerfile

+        # if we also install other target's build dependencies (like tools, libs, and linkers)
+        # rustup target add x86_64-unknown-linux-musl \
+        # rustup target add aarch64-unknown-linux-musl \
+        # rustup target add armv7-unknown-linux-musleabihf \
+        # rustup target add armv7-unknown-linux-musleabi


We are definitely not going to try to support multiple targets in a single image. And we won't be shipping any targets that can't link diesel and sqlx. I tried this in the past and it was maintenance headache and backwards compatibility guarantee I didn't want.

Honestly, people should use cross unless they need C libraries that are not supported by cross. I've already converted as many projects as I can, and it's great. (But it won't work for everything.)

emk · 2022-01-02T00:20:22Z

examples/linking-with-git2/Dockerfile

+
+# Now, we need to build our _real_ Docker container, copying in `linking-with-git2`.
+FROM debian:stretch-slim
+RUN apt-get update && apt-get install -y ca-certificates
+COPY --from=builder \
+  /home/rust/src/target/x86_64-unknown-linux-musl/debug/linking-with-git2 \
+  /usr/local/bin/
+CMD /usr/local/bin/linking-with-git2


This is probably unnecessary for libgit2. We have other examples anyways.

emk · 2022-01-02T00:20:49Z

examples/using-sqlx/Dockerfile

-FROM alpine:latest
-RUN apk --no-cache add ca-certificates
+FROM debian:stretch-slim
+RUN apt-get update && apt-get install -y ca-certificates


All images should continue to use Alpine, now that we do ldd checks on the host.

Many thanks for figuring out ldd, and for preparing this PR!

emk · 2022-01-02T00:23:21Z

Dockerfile

+# Needs evaluation;
+# fails for libpq with:
+# "configure: error: library 'crypto' is required for OpenSSL"
+# ARG OPENSSL_VERSION=3.0.0


Any support of OpenSSL 3.0.0 should happen in a separate PR, and it should wait until all the examples in the test suite support dynamic linking of OpenSSL. (They may do so already.)

We'll also need to consider how many existing users this breaks in the short term. We try to avoid breaking working CI setups, though in a 6-year-old project, that occasionally happens to people who pin to :stable instead of a version.

asaaki mentioned this pull request Mar 20, 2021

ubuntu-20.04 asaaki/rust-musl-builder#1

Open

asaaki force-pushed the ubuntu-20.04 branch from 1f323d2 to 2555a51 Compare January 1, 2022 14:26

asaaki mentioned this pull request Jan 1, 2022

Ubuntu 20.04 #98

Open

asaaki force-pushed the ubuntu-20.04 branch from 2555a51 to 1b7948d Compare January 1, 2022 15:29

emk mentioned this pull request Jan 1, 2022

Upgrade base image to Ubuntu 20.04 #134

Open

emk closed this Jan 1, 2022

emk reviewed Jan 2, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu 20.04 #120

Ubuntu 20.04 #120

asaaki commented Mar 20, 2021 •

edited

Loading

asaaki commented Mar 20, 2021 •

edited

Loading

jman-schief commented Apr 18, 2021 •

edited

Loading

asaaki commented Apr 19, 2021

emk commented Jan 1, 2022

asaaki commented Jan 1, 2022

emk left a comment

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022 •

edited

Loading

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022

emk Jan 2, 2022 •

edited

Loading

emk Jan 2, 2022

		RUN echo "Building OpenSSL" && \
		RUN echo "Building OpenSSL ${OPENSSL_VERSION}" && \

		ADD git-credential-ghtoken /usr/local/bin/ghtoken
		COPY git-credential-ghtoken /usr/local/bin/ghtoken

Ubuntu 20.04 #120

Ubuntu 20.04 #120

Conversation

asaaki commented Mar 20, 2021 • edited Loading

asaaki commented Mar 20, 2021 • edited Loading

jman-schief commented Apr 18, 2021 • edited Loading

asaaki commented Apr 19, 2021

emk commented Jan 1, 2022

asaaki commented Jan 1, 2022

emk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emk Jan 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emk Jan 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asaaki commented Mar 20, 2021 •

edited

Loading

asaaki commented Mar 20, 2021 •

edited

Loading

jman-schief commented Apr 18, 2021 •

edited

Loading

emk Jan 2, 2022 •

edited

Loading

emk Jan 2, 2022 •

edited

Loading