2026-05-01 04:44:09 <ikke> dne: thanks, was due to a change, reverted the change now
2026-05-01 06:22:13 <ncopa> I'm rebooting usa2-dev1 (s390x) with newer kernel now
2026-05-01 06:24:39 <ncopa> rebooting nld-bld-3 (x86) with newer kernel
2026-05-01 06:26:11 <ncopa> s390x builder is back. I wonder if I dare upgrade the alpine version
2026-05-01 06:42:38 <ikke> If it doesn't boot, we would need help from Marista, may delay a release
2026-05-01 06:42:50 <ncopa> im upgrading the loongarch64 build servers
2026-05-01 06:42:56 <ikke> Ok
2026-05-01 06:43:03 <ncopa> yeah
2026-05-01 06:43:19 <ncopa> im also upgrading the x86 and x86_64 builders
2026-05-01 06:43:19 <ikke> I was working on getting zabbix agent deployed on them and adding them to zabbix
2026-05-01 06:44:10 <ncopa> on the loongarch64 machines?
2026-05-01 06:44:18 <ncopa> I havent rebooted them yet, just apk upgrade
2026-05-01 06:44:32 <ncopa> im waiting for the qt6-qtwebengine build to finish
2026-05-01 06:45:01 <ikke> Yes, but no problem to reboot them
2026-05-01 06:45:05 <ikke> I can only continue later
2026-05-01 06:45:46 <ncopa> i do have an issue with the build-*-x86 host. i initiated a reboot. lost sshd but its not rebooting
2026-05-01 06:46:15 <ikke> How long did you wait?
2026-05-01 06:46:18 <ncopa> 64 bytes from 185.15.220.27: seq=840 ttl=42 time=30.147 ms
2026-05-01 06:46:18 <ncopa> 64 bytes from 185.15.220.27: seq=841 ttl=42 time=30.098 ms
2026-05-01 06:46:49 <ncopa> i wonder if it is the sccache build
2026-05-01 06:57:27 <ikke> Nld-bld-3 is hanging at stopping lxc containers
2026-05-01 07:01:01 <ikke> Want me to force reboot it?
2026-05-01 07:04:15 <ncopa> yes please
2026-05-01 07:05:30 <ncopa> Im  stopping lxc on nld-bld-4 before doing reboot
2026-05-01 07:27:36 <ikke> Reset that server
2026-05-01 07:30:59 <ikke> ncopa: it's booted again
2026-05-01 07:39:01 <omni> !101684 has the same getopt issue
2026-05-01 08:43:25 <ncopa> ikke: thanks
2026-05-01 08:43:51 <ncopa> the nld[89]-dev1.a.o machines (x86/x86_64) are rebooted
2026-05-01 09:02:16 <ncopa> im starting with usa6-dev1 (ppc64le)
2026-05-01 10:31:26 <Sertonix[m]> It looks like build-edge-riscv64 hasn't uploaded the testing repo since 10 days.
2026-05-01 10:32:19 <omni> makes sense
2026-05-01 10:33:30 <omni> or I just assume that it has been failing with various aports and haven't gotten through them all yet
2026-05-01 10:33:57 <ikke> Yes
2026-05-01 10:34:32 <ikke> exabgp, py3-venusian
2026-05-01 10:34:37 <ikke> Others possibly 
2026-05-01 10:34:43 <Sertonix[m]> Will take a loon
2026-05-01 10:34:47 <Sertonix[m]> *look
2026-05-01 10:45:32 <Sertonix[m]> There was some previous discussion about --prefix ''  but I forgot what the result was. !101743 should fix some build failures
2026-05-01 10:54:05 <ikke> ptrc mentioned something about it
2026-05-01 10:58:19 <ikke> ncopa: I've added the che servers to zabbix now as well, so they should show up in the problem llist
2026-05-01 14:23:06 <ikke> blaboon: It appears our promotion credits have not been applied anymore. We got a ticket about payment issues. Can you check?
2026-05-01 15:06:20 <ikke> blaboon: I've replied to the ticket, someone is looking into it
2026-05-01 16:57:15 <ncopa> i have rebooted the loongarch64 and ppc64le builder too
2026-05-01 16:57:42 <ncopa> I changed loongarch64 builder to v3.23
2026-05-01 19:35:19 <ikke> rebooting aarch64 CI VM to apply kernel updates
2026-05-01 19:49:41 <ikke> rebooting armv7 CI VM to apply kernel updates
2026-05-02 06:32:46 <ikke> Was too late with putting ^ into maintenance. I'm upgrading the host
2026-05-02 07:07:23 <ikke> I wonder where that unexpected ridirect comes from
2026-05-02 07:14:59 <ikke> Ok, interesting: routing issue from ipv4 from zabbix -> deu-t1-1
2026-05-02 07:15:36 <ikke> The http client seems to automatically follow redirects, so the status is 200, normally no warning for unexpected redirects (warning would be expected)
2026-05-02 07:16:15 <ikke> It can reach http://deu-t1-1.a.o, so it sees the redirect, but then connecting to the redirected url (https://...), it times out -> reports about unexpected redirect, but not the timeout
2026-05-02 07:23:45 <ikke> hmm, some strange routing issues
2026-05-02 07:24:02 <ikke> I can reach it from my workstation
2026-05-02 09:18:29 <ncopa> I have upgraded kernel on gitlab-runner-s390x
2026-05-02 10:20:58 <dne> hm now the getopt missing issue happens on all 3.20 runners except x86
2026-05-02 10:34:34 <ncopa> huh
2026-05-02 10:36:50 <dne> from the other day: 1. start with alpinelinux/alpine-gitlab-ci:latest 2. downgrade to 3.20 3. apk add util-linux-misc 4. apk del util-linux-misc 5. /bin/getopt is gone! (and abuild-sign fails)
2026-05-02 10:37:05 <dne> happens on 3.20 ci runners
2026-05-02 10:37:28 <dne> (if util-linux-misc gets installed as deps)
2026-05-02 10:38:32 <dne> as in https://gitlab.alpinelinux.org/dne/aports/-/pipelines/431056
2026-05-02 10:39:23 <ncopa> when apk del util-linux-misc happens, the busybox trigger is supposed to run to restore the /bin/getopt -> /bin/busybox symlink
2026-05-02 10:39:54 <dne> yes but it doesn't seem to happen, after a downgrade
2026-05-02 10:41:18 <dne> (it works fine on 3.20 that hasn't been downgraded from edge)
2026-05-02 12:52:57 <Sertonix[m]> ncopa: The busybox trigger doesn't seem to run. Is /bin a symlink on these systems?
2026-05-02 12:52:59 <Sertonix[m]> Ether way that looks like an apk-tools bug
2026-05-02 12:53:56 <Sertonix[m]> Sooner or later we will need the CI to not downgrade from a newer alpine version. At some point the apkv3 installed db will not be readable by apkv2 anymore
2026-05-02 12:54:12 <ikke> Sertonix[m]: https://tpaste.us/yBVx 
2026-05-02 12:55:01 <ikke> registry.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci before (up/down)grade
2026-05-02 16:37:41 <ikke> bootstrapping ghc on aarch64
2026-05-02 17:42:20 <ikke> done
2026-05-02 19:13:09 <mio> getting "ERROR: Failed to remove network for build" on #204 (Hnyf5AhA) shared-runner che-ci-2 (loongarch64)
2026-05-02 19:13:28 <mio> thanks algitbot, but it's https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2332493
2026-05-02 19:21:40 <mio> ikke: ghc still seems to be looking for a specific bootstrap version https://build.alpinelinux.org/buildlogs/build-3-24-aarch64/community/ghc/ghc-9.10.3-r1.log
2026-05-02 19:34:49 <ikke> mio: the build faild, didn't notice
2026-05-02 19:42:49 <mio> okay
2026-05-02 19:49:32 <ikke> oof
2026-05-02 19:51:09 <ikke> aarch64 ci host fails to boot
2026-05-02 19:51:23 <ikke> 'invalid magic'
2026-05-03 11:17:00 <dne> shared-runner che-ci-2 (loongarch64) is having some docker issue: "ERROR: Preparation failed: Error response from daemon: client version 1.43 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version (docker.go:958:0s)"
2026-05-03 11:30:18 <ikke> dne: hmm, ncopa mentioned a similar issue after upgrading to 3.23
2026-05-03 11:32:46 <ikke> I suppose gitlab-runner is the client in this case
2026-05-03 11:46:26 <ikke> dne: and thanks for reporting
2026-05-03 11:48:03 <dne> no pb at all :) mio reported the same issue last night I believe, but not that specific error message
2026-05-03 12:13:59 <ikke> Updating gitlab-runner fixed it
2026-05-03 12:17:48 <dne> cool!
2026-05-03 15:10:38 <ikke> ncopa: for reference, the docker client version mismatch you ran into with alpine 3.23 means that gitlab-runner needs to be updated
2026-05-03 16:38:48 <aelin> hmmm is it just me or is IPv6 broken on the Alpine mirrors?
2026-05-03 16:39:32 <aelin> dl-1 does not have an AAAA record, for dl-2, dl-3 and dl-4, any attempt to connect over IPv6 just times out 
2026-05-03 16:39:46 <aelin> been like this for a few weeks now
2026-05-03 16:40:13 <aelin> see e.g. curl -6 https://dl-4.alpinelinux.org/alpine/
2026-05-03 16:45:20 <ikke> The dl-<n> mirrors are just aliases nowadays, kept them for backwards compatibility
2026-05-03 16:46:44 <aelin> ah okay, so the recommended URL would be dl-cdn now?
2026-05-03 16:47:15 <ikke> yes, but I do need to figure out why it's not working ipv6
2026-05-03 16:47:39 <ikke> dl-cdn points to fastly, while dl-<n> point directly to our servers
2026-05-03 16:48:50 <aelin> we actually hardcode dl-4 in pmbootstrap for that reason I think
2026-05-03 16:49:25 <aelin> because we want to avoid a case where the CDN may choose an outdated mirror, since it must be deterministic in our source URLs
2026-05-03 16:49:47 <ikke> aelin: It should not make any difference
2026-05-03 16:50:09 <aelin> I'll change that to dl-cdn then, thank you :)
2026-05-03 16:50:37 <ikke> dl-cdn points to exactly the same servers as the dl-<n> ones, but we do use geolocation for dl-<n>
2026-05-03 16:50:55 <ikke> For ipv6 however we only have one server that provides it sadly
2026-05-03 16:55:29 <ikke> aelin: fixed, https was missing from the firewall rules
2026-05-03 16:57:24 <aelin> oh nice, thank you! indeed works now
2026-05-03 16:58:17 <ikke> oh, lol
2026-05-03 17:00:23 <ikke> uhh
2026-05-03 17:03:11 <ikke> Always fun (not) the descrepency between ipv6 and ipv4 in docker
2026-05-03 17:03:45 <ikke> rsync should work again for ipv4, but because in (the installed version of) docker, ipv4 is forwarded, but ipv6 is not
2026-05-03 17:04:22 <ikke> Good that I didn't provide a AAAA record for ipv6 yet for rsync
2026-05-03 17:51:15 <mio> can the following archives be copied to 3.24 distfiles for rebuild please? https://distfiles.alpinelinux.org/distfiles/v3.23/grub-6811f6f09d61996a3acbc4fc0414e45964f0e2d9.tar.gz https://distfiles.alpinelinux.org/distfiles/v3.23/libfakekey-0.3.tar.gz
2026-05-03 17:57:49 <ikke> done
2026-05-03 17:58:16 <mio> thanks!
2026-05-03 18:07:35 <ikke> Interesting, when trying to connect to rsync on deu-t1-1 over ipv6 it manages to send a client initialization packet, but then the connection is terminated. That means the tcp handshake succeeded
2026-05-04 04:26:45 <ikke> meh
2026-05-04 09:26:14 <fcolista> hi friends, can someone help with this aarch64 error? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2333962
2026-05-04 09:27:26 <ikke> Sounds like an outdated gitlab-runner-helper image
2026-05-04 09:48:09 <fcolista> yes
2026-05-04 09:48:28 <fcolista> is it something I can fix?
2026-05-04 09:48:46 <ikke> I started a pipeline to rebuild the image
2026-05-04 09:48:51 <ikke> See if that helps
2026-05-04 10:22:45 <ikke> Hm, no
2026-05-04 10:29:03 <ikke> gitlab-runner in aports is a bit outdated
2026-05-04 10:52:04 <ncopa> maybe I should revert to alpine 3.22
2026-05-04 10:53:44 <ikke> It's working on the loongarch runners
2026-05-04 11:16:51 <ikke> ncopa: can you try `docker pull alpinelinux/gitlab-runner-helper:latest` on that host?
2026-05-04 11:17:31 <ikke> The -timeout flag was added in 18.10.0
2026-05-04 11:44:09 <ncopa> so docker compose pull is not enough?
2026-05-04 11:46:07 <ncopa> seems to work now. thanks!
2026-05-04 11:53:41 <ncopa> hum. not it doesnt. who do I check which version it actually runs?
2026-05-04 11:54:38 <ncopa> it uses registry.alpinelinux.org/alpine/infra/docker/gitlab-runner:latest
2026-05-04 12:03:32 <dne> docker run --rm -it --entrypoint gitlab-runner registry.alpinelinux.org/alpine/infra/docker/gitlab-runner:latest --version
2026-05-04 12:04:07 <dne> version 18.10.0?
2026-05-04 13:10:52 <ncopa> Version:      18.10.0
2026-05-04 13:13:05 <ikke> What about the helper image?
2026-05-04 13:15:24 <ncopa> Version:      17.2.1
2026-05-04 13:15:27 <ncopa> there we go
2026-05-04 13:15:53 <ncopa> happens even after a docker pull
2026-05-04 13:16:17 <ncopa> lima-alpine-gitlab-ci:~/compose/gitlab-runner-alpine-ci# docker run --rm -it --entrypoint gitlab-runner-helper  alpinelinux/gitlab-runner-helper:latest --version | tpaste
2026-05-04 13:16:17 <ncopa> https://tpaste.us/pyjj
2026-05-04 13:16:43 <ikke> Can you check registry.alpinelinux.org/alpine/infra/docker/gitlab-runner-helper:latest?
2026-05-04 13:17:40 <ncopa> https://tpaste.us/WWYY
2026-05-04 13:17:58 <ncopa> so I have pulled it, do I need to change the image somewhere?
2026-05-04 13:18:01 <ikke> Ok, I guess we no longer publish that image to docker hub, but it's still being used
2026-05-04 13:18:08 <ikke> ncopa: in the runner config
2026-05-04 13:18:20 <ikke> It should specify the helper image
2026-05-04 13:19:10 <ncopa> here? https://gitlab.alpinelinux.org/admin/runners/210/edit
2026-05-04 13:20:13 <ikke> No, docker compose exec gitlab-runner vi /etc/gitlab-runner/config.toml
2026-05-04 13:41:43 <ncopa> thanks
2026-05-04 15:52:40 <ncopa> Linode is back
2026-05-04 19:39:04 <blaboon> ikke: apologies for the delay. a bunch of stuff has changed and i unfortunately don't really have much visibility into our marketing/promo stuff anymore so i haven't been able to keep your account in standing. i saw your ticket though and reached out to other folks in marketing and they should be taking care of things now
2026-05-04 19:39:48 <ikke> blaboon: good to hear from you. 
2026-05-04 19:39:51 <ikke> blaboon: and thanks!'
2026-05-04 19:42:57 <ikke> blaboon: One thing we wanted to confirm, not sure if you know, would a suspension like we experienced affect linode dns as well?
2026-05-05 00:04:38 <blaboon> ikke: i believe our nameservers continue serving domains even from a suspended account, so i don't think they would have been affected
2026-05-05 05:01:26 <ikke> blaboon: yeah, that would be our expectation as well
2026-05-05 05:57:02 <ncopa> ikke: FYI: riscv64 kernel (for pioneer) has # CONFIG_CRYPTO_USER_API_AEAD is not set
2026-05-05 05:58:58 <ncopa> thats also the case for 6.6.53-2-spacemit (the bpi f3 kernel)
2026-05-05 06:01:21 <ikke> Ah, good to know, so not vulnerable 
2026-05-05 06:02:03 <ikke> Is the configs module loaded by default?
2026-05-05 06:02:14 <ikke> Could use it to check it with zabbix 
2026-05-05 06:03:47 <ncopa> no, configs module is not loaded by default
2026-05-05 06:04:51 <ncopa> but you can check /boot/config-$(uname -r)
2026-05-05 06:17:53 <ikke> Ok
2026-05-05 07:45:17 <omni> uhm when did build.a.o get a horizontal scrollbar?
2026-05-05 07:47:57 <omni> and why? sliding it kinda just toggles the visibility of the left and right border lines of the tables
2026-05-05 08:57:13 <ncopa> probably since the [online] badge showed up and took pixels in horizontal space
2026-05-05 08:58:15 <ncopa> we added a badge that shows the status of the builder. last time we did stable releases we had a riscv64 builder that had gone offline without anyone noticing
2026-05-05 08:58:54 <ncopa> so I added an mqtt will topic to tell when a builder unexpectedly went missing
2026-05-05 08:59:18 <ncopa> and added a 'offline' state for indicating that a builder was intentionally stopped
2026-05-05 08:59:56 <ncopa> hum... I though I put usa-t1-2.alpinelinux.org in maintenance mode
2026-05-05 09:05:02 <omni> ok, I only noticed when I scrolled to the bottom of the page
2026-05-05 09:30:05 <ncopa> sigh.. the usa-t1-2.a.o did not come back after a reboot
2026-05-05 10:38:45 <ikke> ok, good
2026-05-05 11:01:50 <ncopa> usa-t1-2 should be all back after some drama
2026-05-05 11:02:11 <ncopa> the disks were reordered and md0 did not come up
2026-05-05 11:02:34 <ncopa> the network didnt come up properly either, buts it back now
2026-05-06 04:12:02 <lotheac> ikke: do you think we could sort out a way for me to access the CI kube cluster and create pods there? i want to debug some test failures for python3 which only happen for x86/x86_64 CI, and I can't repro them on a fresh k0s cluster...
2026-05-06 05:27:24 <ikke> lotheac: Yeah, I think that should be possible
2026-05-06 05:29:02 <lotheac> the apiserver lb is on public internet, i can access port 6443... so i just need a way to authenticate to it (and some privileges via a role or clusterrole)
2026-05-06 05:29:30 <lotheac> i think you mentioned using gitlab as openid provider for oidc auth at some point?
2026-05-06 05:29:58 <ikke> Yeah, that would be ideal
2026-05-06 05:30:24 <ikke> But I think for the interrim I can also arrange something
2026-05-06 05:30:57 <lotheac> cheers
2026-05-06 09:14:25 <ncopa> ikke: I'm upgrading kernel on nld-t1-2 now. and then I'll do nld-t1-1
2026-05-06 09:14:59 <ikke> Ok, do note that we do not have console access for those I believe
2026-05-06 09:15:20 <ikke> Oh, we should have for HorizonIQ, but not for Osso
2026-05-06 09:57:16 <ncopa> nld-t1-2 is upgraded to alpine 3.23
2026-05-06 16:49:09 <ncopa> bootstrapping ghc on aarch64 failed
2026-05-06 17:25:57 <ikke> ncopa: documentation issue?
2026-05-06 17:26:14 <ikke> ncopa: mio suggested https://gitlab.haskell.org/ghc/ghc/-/commit/e8f5a45de561ec80c88cd3da2c66502deb32d4c3
2026-05-07 06:01:58 <mio> can matchbox-keyboard tarball be added to 3.24 distfiles for rebuild? thanks! https://distfiles.alpinelinux.org/distfiles/v3.23/matchbox-keyboard-0.1.1.tar.gz
2026-05-07 07:00:55 <ncopa> mio: done
2026-05-07 15:37:12 <mio> ncopa: thanks!
2026-05-07 22:58:53 <omni> I don't see 3.20 builders on build.a.o
2026-05-07 22:59:50 <omni> oh, supposedly not supported since first of april? I thought we'd drop it once 3.24 is out
2026-05-08 03:41:34 <ikke> omni: since msg.a.o got rebooted, it lost the messages for those. Trying those builders will make them reappear 
2026-05-08 07:05:37 <dne> that downgrade issue is somewhat blocking a few security upgrade MRs for 3.20
2026-05-08 07:17:03 <ncopa> it does. I havent been able to reproduce it locally
2026-05-08 07:25:36 <ikke> Ftr: https://gitlab.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci/-/merge_requests/32
2026-05-08 07:25:52 <dne> hm weird, today I can't reproduce it either - the other day I could
2026-05-08 07:38:15 <dne> no sorry, used the wrong image. I can still reproduce: https://paste.dismail.de/?35243bd4dafa5fb7#5SXcaf82CcviaHJfYQgXWiYUwve8PteZmyQB6reYPGYU
2026-05-08 09:02:41 <ncopa> dne: thanks. I can reproduce too now. I think I know what happens. edge runs apk3. when downgrading on v3.20 it installs older busybox with apk3 + older apk 2. the apk2 is not able to find/execute the busybox trigger installed by apk3.
2026-05-08 09:02:54 <ncopa> workaround: apk fix busybox after downgrade
2026-05-08 09:03:01 <ncopa> but I think it applies to all triggers
2026-05-08 09:04:02 <ikke> Merging MR 3w2
2026-05-08 09:04:04 <ikke> Merging MR 32
2026-05-08 09:04:16 <ikke> Still need some changes in aports ci to make it use those images
2026-05-08 09:04:52 <ikke> Though, I could hardcode it in 3.20 for now to see if it works
2026-05-08 09:05:12 <ncopa> upgrade apk to newer 2.x should also work
2026-05-08 09:06:01 <ikke> I believe sertonix mentioned that, once we switch to apkv3 indexes, it would break anyway
2026-05-08 09:12:08 <lotheac> hmph... can't repro the python3 test failures in the ci kube cluster either, with a manual build. ikke: could i ask for read-only access to the namespace gitlab ci uses? i want to see the pod specs of the CI pods, maybe there's something there that affects the behavior
2026-05-08 09:12:50 <lotheac> (although, that's probably for next week, got places to be)
2026-05-08 09:40:49 <ikke> dne: it's working now: https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2340134
2026-05-08 10:10:53 <ncopa> lotheac, ikke: thanks!
2026-05-08 10:21:59 <Sertonix[m]> ikke: * installed db
2026-05-08 10:22:48 <ikke> Sertonix[m]: ah ok
2026-05-08 10:49:42 <dne> ikke: great!
2026-05-08 10:50:23 <ikke> So rebasing should fix any pipelines that failed due to this on 3.20
2026-05-08 10:51:29 <dne> cool, will do
2026-05-08 10:59:50 <ikke> ncopa: I've added an extra page to the kernel dashboard for CVE-2026-43284
2026-05-08 11:00:40 <ikke> Waiting for CVE-2026-43500
2026-05-08 11:17:57 <ncopa> ikke: thanks! I can start upgrading kernel on the builders once the kernel is available
2026-05-08 11:20:21 <ikke> Right
2026-05-08 13:59:45 <ncopa> im rebooting the arm build server
2026-05-08 14:01:49 <ikke> Ok
2026-05-08 14:10:19 <ncopa> im rebooting che-bld-2, the loongarch64 builder
2026-05-08 14:18:00 <ikke> ncopa: has the 2nd CVE already been fixed? I see comments that they are still working on it