Can't run Complement for current Conduit #409

New issue

Closed

opened 2023-12-01 08:06:23 +00:00 by _ZN3val · 5 comments

_ZN3val commented

2023-12-01 08:06:23 +00:00

(Migrated from gitlab.com)

Currently, it fails to run because of Rust version in the base image:

error: package `time-macros v0.2.9` cannot be built because it requires rustc 1.65.0 or newer, while the currently active rustc version is 1.64.0
Error: building at STEP "RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release": while running runtime: exit status 101

Replacing the base builder image with docker.io/rust:latest fixes this issue, but it then fails to build rocksdb-sys:

error: failed to run custom build command for `librocksdb-sys v0.11.0+8.1.1`

Caused by:
  process didn't exit successfully: `/workdir/target/release/build/librocksdb-sys-32e82f4d44ddc71e/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.65.1/lib.rs:603:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
Error: building at STEP "RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release": while running runtime: exit status 101

Then I tried replacing the builder altogether and using an externally build Conduit, which allowed Conduit to run after applying #561:

diff --git a/.dockerignore b/.dockerignore
index c78ddba..3703344 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -1,5 +1,9 @@
 # Local build and dev artifacts
-target
+#target
+target/debug/build
+target/debug/deps
+target/release/build
+target/release/deps
 tests
 
 # Docker files
diff --git a/complement/Dockerfile b/complement/Dockerfile
index 50173a1..774afcc 100644
--- a/complement/Dockerfile
+++ b/complement/Dockerfile
@@ -1,22 +1,5 @@
-# For use in our CI only. This requires a build artifact created by a previous run pipline stage to be placed in cached_target/release/conduit
-FROM registry.gitlab.com/jfowl/conduit-containers/rust-with-tools:commit-16a08e9b as builder
-#FROM rust:latest as builder
-
-WORKDIR /workdir
-
-ARG RUSTC_WRAPPER
-ARG AWS_ACCESS_KEY_ID
-ARG AWS_SECRET_ACCESS_KEY
-ARG SCCACHE_BUCKET
-ARG SCCACHE_ENDPOINT
-ARG SCCACHE_S3_USE_SSL
-
-COPY . .
-RUN mkdir -p target/release
-RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release
-
-## Actual image
-FROM debian:bullseye
+### Actual image
+FROM debian:bookworm
 WORKDIR /workdir
 
 # Install caddy
@@ -29,15 +12,13 @@ ENV SERVER_NAME=localhost
 ENV CONDUIT_CONFIG=/workdir/conduit.toml
 
 RUN sed -i "s/port = 6167/port = 8008/g" conduit.toml
-RUN echo "allow_federation = true" >> conduit.toml
-RUN echo "allow_check_for_updates = true" >> conduit.toml
 RUN echo "allow_encryption = true" >> conduit.toml
-RUN echo "allow_registration = true" >> conduit.toml
 RUN echo "log = \"warn,_=off,sled=off\"" >> conduit.toml
 RUN sed -i "s/address = \"127.0.0.1\"/address = \"0.0.0.0\"/g" conduit.toml
 
-COPY --from=builder /workdir/target/release/conduit /workdir/conduit
-RUN chmod +x /workdir/conduit
+#COPY --from=builder /workdir/target/release/conduit /workdir/conduit
+#RUN chmod +x /workdir/conduit
+COPY target/debug/conduit /workdir/conduit
 
 EXPOSE 8008 8448

However, Complement still fails to run with Conduit:

2023/12/01 09:00:34 config: &{BaseImageURI:complement-conduit:dev DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix: SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:fed CACertificate:0xc000dce000 CAPrivateKey:0xc00003a960 BestEffort:false HostnameRunningComplement:host.containers.internal EnableDirtyRuns:false HSPortBindingIP:127.0.0.1 PostTestScript:}
=== RUN   TestWriteMDirectAccountData
2023/12/01 09:01:05 fed.1_servers.hs1 : failed to deployBaseImage: fed.1_servers.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container f02199030d7a9a820e00868c4c5cb7ef7e9f57a4b4fcf5eb7f34d1b6d11d4e4c => health: 
2023/12/01 09:01:05 ============================================


2023/12/01 09:01:05 fed.1_servers.hs1 : Server logs:
Linux f02199030d7a 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
{"level":"info","ts":1701417635.4956849,"msg":"using provided configuration","config_file":"caddy.json","config_adapter":""}
{"level":"warn","ts":1701417635.5098195,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"/complement/ca/ca.crt"}
{"level":"info","ts":1701417635.5099268,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1701417635.5099335,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1701417636.005045,"msg":"certificate installed properly in linux trusts"}
{"level":"info","ts":1701417636.0052943,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
2023-12-01T08:00:39.705153Z  WARN trust_dns_resolver::hosts: could not parse an IP from hosts file
2023-12-01T08:00:39.721780Z  WARN conduit::database: Created new rocksdb database with version 13
Hello, Timo here, Thanks for updating your Conduit. Starting now, you will receive notifications in this room for important Conduit news, like when a new release comes out or a new security issue was discovered. This works by sending a simple GET request to conduit.rs once per hour. You can opt out of this by setting allow_check_for_updates = false in your config. Have a nice day!
2023/12/01 09:01:05 ============== fed.1_servers.hs1 : END LOGS ==============


    direct_messaging_test.go:25: Deploy: Failed to construct blueprint: ConstructBlueprintIfNotExist(1_servers): failed to ConstructBlueprint: errors whilst constructing blueprint 1_servers: [fed.1_servers.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container f02199030d7a9a820e00868c4c5cb7ef7e9f57a4b4fcf5eb7f34d1b6d11d4e4c => health: ]
--- FAIL: TestWriteMDirectAccountData (32.03s)

Currently, it fails to run because of Rust version in the base image: ``` error: package `time-macros v0.2.9` cannot be built because it requires rustc 1.65.0 or newer, while the currently active rustc version is 1.64.0 Error: building at STEP "RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release": while running runtime: exit status 101 ``` Replacing the base builder image with `docker.io/rust:latest` fixes this issue, but it then fails to build rocksdb-sys: ``` error: failed to run custom build command for `librocksdb-sys v0.11.0+8.1.1` Caused by: process didn't exit successfully: `/workdir/target/release/build/librocksdb-sys-32e82f4d44ddc71e/build-script-build` (exit status: 101) --- stderr thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.65.1/lib.rs:603:31: Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])" note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace warning: build failed, waiting for other jobs to finish... Error: building at STEP "RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release": while running runtime: exit status 101 ``` Then I tried replacing the builder altogether and using an externally build Conduit, which allowed Conduit to run after applying #561: ```diff diff --git a/.dockerignore b/.dockerignore index c78ddba..3703344 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,5 +1,9 @@ # Local build and dev artifacts -target +#target +target/debug/build +target/debug/deps +target/release/build +target/release/deps tests # Docker files diff --git a/complement/Dockerfile b/complement/Dockerfile index 50173a1..774afcc 100644 --- a/complement/Dockerfile +++ b/complement/Dockerfile @@ -1,22 +1,5 @@ -# For use in our CI only. This requires a build artifact created by a previous run pipline stage to be placed in cached_target/release/conduit -FROM registry.gitlab.com/jfowl/conduit-containers/rust-with-tools:commit-16a08e9b as builder -#FROM rust:latest as builder - -WORKDIR /workdir - -ARG RUSTC_WRAPPER -ARG AWS_ACCESS_KEY_ID -ARG AWS_SECRET_ACCESS_KEY -ARG SCCACHE_BUCKET -ARG SCCACHE_ENDPOINT -ARG SCCACHE_S3_USE_SSL - -COPY . . -RUN mkdir -p target/release -RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release - -## Actual image -FROM debian:bullseye +### Actual image +FROM debian:bookworm WORKDIR /workdir # Install caddy @@ -29,15 +12,13 @@ ENV SERVER_NAME=localhost ENV CONDUIT_CONFIG=/workdir/conduit.toml RUN sed -i "s/port = 6167/port = 8008/g" conduit.toml -RUN echo "allow_federation = true" >> conduit.toml -RUN echo "allow_check_for_updates = true" >> conduit.toml RUN echo "allow_encryption = true" >> conduit.toml -RUN echo "allow_registration = true" >> conduit.toml RUN echo "log = \"warn,_=off,sled=off\"" >> conduit.toml RUN sed -i "s/address = \"127.0.0.1\"/address = \"0.0.0.0\"/g" conduit.toml -COPY --from=builder /workdir/target/release/conduit /workdir/conduit -RUN chmod +x /workdir/conduit +#COPY --from=builder /workdir/target/release/conduit /workdir/conduit +#RUN chmod +x /workdir/conduit +COPY target/debug/conduit /workdir/conduit EXPOSE 8008 8448 ``` However, Complement still fails to run with Conduit: ``` 2023/12/01 09:00:34 config: &{BaseImageURI:complement-conduit:dev DebugLoggingEnabled:false AlwaysPrintServerLogs:false EnvVarsPropagatePrefix: SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] BaseImageURIs:map[] PackageNamespace:fed CACertificate:0xc000dce000 CAPrivateKey:0xc00003a960 BestEffort:false HostnameRunningComplement:host.containers.internal EnableDirtyRuns:false HSPortBindingIP:127.0.0.1 PostTestScript:} === RUN TestWriteMDirectAccountData 2023/12/01 09:01:05 fed.1_servers.hs1 : failed to deployBaseImage: fed.1_servers.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container f02199030d7a9a820e00868c4c5cb7ef7e9f57a4b4fcf5eb7f34d1b6d11d4e4c => health: 2023/12/01 09:01:05 ============================================ 2023/12/01 09:01:05 fed.1_servers.hs1 : Server logs: Linux f02199030d7a 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux {"level":"info","ts":1701417635.4956849,"msg":"using provided configuration","config_file":"caddy.json","config_adapter":""} {"level":"warn","ts":1701417635.5098195,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"/complement/ca/ca.crt"} {"level":"info","ts":1701417635.5099268,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"} {"level":"info","ts":1701417635.5099335,"msg":"define JAVA_HOME environment variable to use the Java trust"} {"level":"info","ts":1701417636.005045,"msg":"certificate installed properly in linux trusts"} {"level":"info","ts":1701417636.0052943,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."} 2023-12-01T08:00:39.705153Z WARN trust_dns_resolver::hosts: could not parse an IP from hosts file 2023-12-01T08:00:39.721780Z WARN conduit::database: Created new rocksdb database with version 13 Hello, Timo here, Thanks for updating your Conduit. Starting now, you will receive notifications in this room for important Conduit news, like when a new release comes out or a new security issue was discovered. This works by sending a simple GET request to conduit.rs once per hour. You can opt out of this by setting allow_check_for_updates = false in your config. Have a nice day! 2023/12/01 09:01:05 ============== fed.1_servers.hs1 : END LOGS ============== direct_messaging_test.go:25: Deploy: Failed to construct blueprint: ConstructBlueprintIfNotExist(1_servers): failed to ConstructBlueprint: errors whilst constructing blueprint 1_servers: [fed.1_servers.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container f02199030d7a9a820e00868c4c5cb7ef7e9f57a4b4fcf5eb7f34d1b6d11d4e4c => health: ] --- FAIL: TestWriteMDirectAccountData (32.03s) ```

_ZN3val commented

2023-12-01 08:06:40 +00:00

(Migrated from gitlab.com)

changed the description

_ZN3val commented

2023-12-01 08:23:37 +00:00

(Migrated from gitlab.com)

changed the description

_ZN3val commented

2023-12-01 09:38:47 +00:00

(Migrated from gitlab.com)

A workaround for that last issue is to disable's Complement's container healthcheck check:

diff --git a/internal/docker/deployer.go b/internal/docker/deployer.go
index 18f0312..332d50a 100644
--- a/internal/docker/deployer.go
+++ b/internal/docker/deployer.go
@@ -528,31 +528,6 @@ func waitForPorts(ctx context.Context, docker *client.Client, containerID string
 func waitForContainer(ctx context.Context, docker *client.Client, hsDep *HomeserverDeployment, stopTime time.Time) (iterCount int, lastErr error) {
        iterCount = 0
 
-       // If the container has a healthcheck, wait for it first
-       for {
-               iterCount += 1
-               if time.Now().After(stopTime) {
-                       lastErr = fmt.Errorf("timed out checking for homeserver to be up: %s", lastErr)
-                       return
-               }
-               inspect, err := docker.ContainerInspect(ctx, hsDep.ContainerID)
-               if err != nil {
-                       lastErr = fmt.Errorf("inspect container %s => error: %s", hsDep.ContainerID, err)
-                       time.Sleep(50 * time.Millisecond)
-                       continue
-               }
-               if inspect.State.Health != nil &&
-                       inspect.State.Health.Status != "healthy" {
-                       lastErr = fmt.Errorf("inspect container %s => health: %s", hsDep.ContainerID, inspect.State.Health.Status)
-                       time.Sleep(50 * time.Millisecond)
-                       continue
-               }
-
-               // The container is healthy or has no health check.
-               lastErr = nil
-               break
-       }
-
        // Having optionally waited for container to self-report healthy
        // hit /versions to check it is actually responding
        versionsURL := fmt.Sprintf("%s/_matrix/client/versions", hsDep.BaseURL)

A workaround for that last issue is to disable's Complement's container healthcheck check: ```diff diff --git a/internal/docker/deployer.go b/internal/docker/deployer.go index 18f0312..332d50a 100644 --- a/internal/docker/deployer.go +++ b/internal/docker/deployer.go @@ -528,31 +528,6 @@ func waitForPorts(ctx context.Context, docker *client.Client, containerID string func waitForContainer(ctx context.Context, docker *client.Client, hsDep *HomeserverDeployment, stopTime time.Time) (iterCount int, lastErr error) { iterCount = 0 - // If the container has a healthcheck, wait for it first - for { - iterCount += 1 - if time.Now().After(stopTime) { - lastErr = fmt.Errorf("timed out checking for homeserver to be up: %s", lastErr) - return - } - inspect, err := docker.ContainerInspect(ctx, hsDep.ContainerID) - if err != nil { - lastErr = fmt.Errorf("inspect container %s => error: %s", hsDep.ContainerID, err) - time.Sleep(50 * time.Millisecond) - continue - } - if inspect.State.Health != nil && - inspect.State.Health.Status != "healthy" { - lastErr = fmt.Errorf("inspect container %s => health: %s", hsDep.ContainerID, inspect.State.Health.Status) - time.Sleep(50 * time.Millisecond) - continue - } - - // The container is healthy or has no health check. - lastErr = nil - break - } - // Having optionally waited for container to self-report healthy // hit /versions to check it is actually responding versionsURL := fmt.Sprintf("%s/_matrix/client/versions", hsDep.BaseURL) ```

CobaltCause commented

2023-12-29 08:49:51 +00:00

(Migrated from gitlab.com)

Applying this diff on top of ca6219723b makes it work for me:

diff --git a/complement/Dockerfile b/complement/Dockerfile
index 50173a1..33962ac 100644
--- a/complement/Dockerfile
+++ b/complement/Dockerfile
@@ -1,19 +1,12 @@
-# For use in our CI only. This requires a build artifact created by a previous run pipline stage to be placed in cached_target/release/conduit
-FROM registry.gitlab.com/jfowl/conduit-containers/rust-with-tools:commit-16a08e9b as builder
-#FROM rust:latest as builder
+FROM rust:1.70.0 as builder

 WORKDIR /workdir

-ARG RUSTC_WRAPPER
-ARG AWS_ACCESS_KEY_ID
-ARG AWS_SECRET_ACCESS_KEY
-ARG SCCACHE_BUCKET
-ARG SCCACHE_ENDPOINT
-ARG SCCACHE_S3_USE_SSL
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libclang-dev

 COPY . .
-RUN mkdir -p target/release
-RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release
+RUN cargo build --release

 ## Actual image
 FROM debian:bullseye
@@ -29,10 +22,6 @@ ENV SERVER_NAME=localhost
 ENV CONDUIT_CONFIG=/workdir/conduit.toml

 RUN sed -i "s/port = 6167/port = 8008/g" conduit.toml
-RUN echo "allow_federation = true" >> conduit.toml
-RUN echo "allow_check_for_updates = true" >> conduit.toml
-RUN echo "allow_encryption = true" >> conduit.toml
-RUN echo "allow_registration = true" >> conduit.toml
 RUN echo "log = \"warn,_=off,sled=off\"" >> conduit.toml
 RUN sed -i "s/address = \"127.0.0.1\"/address = \"0.0.0.0\"/g" conduit.toml

Applying this diff on top of ca6219723b0d562d86a69824720215f8a1851836 makes it work for me: ```diff diff --git a/complement/Dockerfile b/complement/Dockerfile index 50173a1..33962ac 100644 --- a/complement/Dockerfile +++ b/complement/Dockerfile @@ -1,19 +1,12 @@ -# For use in our CI only. This requires a build artifact created by a previous run pipline stage to be placed in cached_target/release/conduit -FROM registry.gitlab.com/jfowl/conduit-containers/rust-with-tools:commit-16a08e9b as builder -#FROM rust:latest as builder +FROM rust:1.70.0 as builder WORKDIR /workdir -ARG RUSTC_WRAPPER -ARG AWS_ACCESS_KEY_ID -ARG AWS_SECRET_ACCESS_KEY -ARG SCCACHE_BUCKET -ARG SCCACHE_ENDPOINT -ARG SCCACHE_S3_USE_SSL +RUN apt-get update && apt-get install -y --no-install-recommends \ + libclang-dev COPY . . -RUN mkdir -p target/release -RUN test -e cached_target/release/conduit && cp cached_target/release/conduit target/release/conduit || cargo build --release +RUN cargo build --release ## Actual image FROM debian:bullseye @@ -29,10 +22,6 @@ ENV SERVER_NAME=localhost ENV CONDUIT_CONFIG=/workdir/conduit.toml RUN sed -i "s/port = 6167/port = 8008/g" conduit.toml -RUN echo "allow_federation = true" >> conduit.toml -RUN echo "allow_check_for_updates = true" >> conduit.toml -RUN echo "allow_encryption = true" >> conduit.toml -RUN echo "allow_registration = true" >> conduit.toml RUN echo "log = \"warn,_=off,sled=off\"" >> conduit.toml RUN sed -i "s/address = \"127.0.0.1\"/address = \"0.0.0.0\"/g" conduit.toml ```

CobaltCause commented

2024-01-30 01:10:23 +00:00

(Migrated from gitlab.com)

Closing this since it was fixed in https://gitlab.com/famedly/conduit/-/merge_requests/575. I do want to revisit this in the future to make the OCI images built via Nix instead of a Dockerfile for reproducibility, but that'll probably be pretty involved. For the record, I was using this commit of Complement at the time of that MR.

Closing this since it was fixed in <https://gitlab.com/famedly/conduit/-/merge_requests/575>. I do want to revisit this in the future to make the OCI images built via Nix instead of a Dockerfile for reproducibility, but that'll probably be pretty involved. For the record, I was using [this commit of Complement](https://github.com/matrix-org/complement/commit/458bfba5f7f3c22e71aeaf7b181bfb6563e29ba3) at the time of that MR.