2026-05-01 04:44:09 <ikke> dne: thanks, was due to a change, reverted the change now
2026-05-01 06:22:13 <ncopa> I'm rebooting usa2-dev1 (s390x) with newer kernel now
2026-05-01 06:24:39 <ncopa> rebooting nld-bld-3 (x86) with newer kernel
2026-05-01 06:26:11 <ncopa> s390x builder is back. I wonder if I dare upgrade the alpine version
2026-05-01 06:42:38 <ikke> If it doesn't boot, we would need help from Marista, may delay a release
2026-05-01 06:42:50 <ncopa> im upgrading the loongarch64 build servers
2026-05-01 06:42:56 <ikke> Ok
2026-05-01 06:43:03 <ncopa> yeah
2026-05-01 06:43:19 <ncopa> im also upgrading the x86 and x86_64 builders
2026-05-01 06:43:19 <ikke> I was working on getting zabbix agent deployed on them and adding them to zabbix
2026-05-01 06:44:10 <ncopa> on the loongarch64 machines?
2026-05-01 06:44:18 <ncopa> I havent rebooted them yet, just apk upgrade
2026-05-01 06:44:32 <ncopa> im waiting for the qt6-qtwebengine build to finish
2026-05-01 06:45:01 <ikke> Yes, but no problem to reboot them
2026-05-01 06:45:05 <ikke> I can only continue later
2026-05-01 06:45:46 <ncopa> i do have an issue with the build-*-x86 host. i initiated a reboot. lost sshd but its not rebooting
2026-05-01 06:46:15 <ikke> How long did you wait?
2026-05-01 06:46:18 <ncopa> 64 bytes from 185.15.220.27: seq=840 ttl=42 time=30.147 ms
2026-05-01 06:46:18 <ncopa> 64 bytes from 185.15.220.27: seq=841 ttl=42 time=30.098 ms
2026-05-01 06:46:49 <ncopa> i wonder if it is the sccache build
2026-05-01 06:57:27 <ikke> Nld-bld-3 is hanging at stopping lxc containers
2026-05-01 07:01:01 <ikke> Want me to force reboot it?
2026-05-01 07:04:15 <ncopa> yes please
2026-05-01 07:05:30 <ncopa> Im  stopping lxc on nld-bld-4 before doing reboot
2026-05-01 07:27:36 <ikke> Reset that server
2026-05-01 07:30:59 <ikke> ncopa: it's booted again
2026-05-01 07:39:01 <omni> !101684 has the same getopt issue
2026-05-01 08:43:25 <ncopa> ikke: thanks
2026-05-01 08:43:51 <ncopa> the nld[89]-dev1.a.o machines (x86/x86_64) are rebooted
2026-05-01 09:02:16 <ncopa> im starting with usa6-dev1 (ppc64le)
2026-05-01 10:31:26 <Sertonix[m]> It looks like build-edge-riscv64 hasn't uploaded the testing repo since 10 days.
2026-05-01 10:32:19 <omni> makes sense
2026-05-01 10:33:30 <omni> or I just assume that it has been failing with various aports and haven't gotten through them all yet
2026-05-01 10:33:57 <ikke> Yes
2026-05-01 10:34:32 <ikke> exabgp, py3-venusian
2026-05-01 10:34:37 <ikke> Others possibly 
2026-05-01 10:34:43 <Sertonix[m]> Will take a loon
2026-05-01 10:34:47 <Sertonix[m]> *look
2026-05-01 10:45:32 <Sertonix[m]> There was some previous discussion about --prefix ''  but I forgot what the result was. !101743 should fix some build failures
2026-05-01 10:54:05 <ikke> ptrc mentioned something about it
2026-05-01 10:58:19 <ikke> ncopa: I've added the che servers to zabbix now as well, so they should show up in the problem llist
2026-05-01 14:23:06 <ikke> blaboon: It appears our promotion credits have not been applied anymore. We got a ticket about payment issues. Can you check?
2026-05-01 15:06:20 <ikke> blaboon: I've replied to the ticket, someone is looking into it
2026-05-01 16:57:15 <ncopa> i have rebooted the loongarch64 and ppc64le builder too
2026-05-01 16:57:42 <ncopa> I changed loongarch64 builder to v3.23
2026-05-01 19:35:19 <ikke> rebooting aarch64 CI VM to apply kernel updates
2026-05-01 19:49:41 <ikke> rebooting armv7 CI VM to apply kernel updates
2026-05-02 06:32:46 <ikke> Was too late with putting ^ into maintenance. I'm upgrading the host
2026-05-02 07:07:23 <ikke> I wonder where that unexpected ridirect comes from
2026-05-02 07:14:59 <ikke> Ok, interesting: routing issue from ipv4 from zabbix -> deu-t1-1
2026-05-02 07:15:36 <ikke> The http client seems to automatically follow redirects, so the status is 200, normally no warning for unexpected redirects (warning would be expected)
2026-05-02 07:16:15 <ikke> It can reach http://deu-t1-1.a.o, so it sees the redirect, but then connecting to the redirected url (https://...), it times out -> reports about unexpected redirect, but not the timeout
2026-05-02 07:23:45 <ikke> hmm, some strange routing issues
2026-05-02 07:24:02 <ikke> I can reach it from my workstation
2026-05-02 09:18:29 <ncopa> I have upgraded kernel on gitlab-runner-s390x
2026-05-02 10:20:58 <dne> hm now the getopt missing issue happens on all 3.20 runners except x86
2026-05-02 10:34:34 <ncopa> huh
2026-05-02 10:36:50 <dne> from the other day: 1. start with alpinelinux/alpine-gitlab-ci:latest 2. downgrade to 3.20 3. apk add util-linux-misc 4. apk del util-linux-misc 5. /bin/getopt is gone! (and abuild-sign fails)
2026-05-02 10:37:05 <dne> happens on 3.20 ci runners
2026-05-02 10:37:28 <dne> (if util-linux-misc gets installed as deps)
2026-05-02 10:38:32 <dne> as in https://gitlab.alpinelinux.org/dne/aports/-/pipelines/431056
2026-05-02 10:39:23 <ncopa> when apk del util-linux-misc happens, the busybox trigger is supposed to run to restore the /bin/getopt -> /bin/busybox symlink
2026-05-02 10:39:54 <dne> yes but it doesn't seem to happen, after a downgrade
2026-05-02 10:41:18 <dne> (it works fine on 3.20 that hasn't been downgraded from edge)
2026-05-02 12:52:57 <Sertonix[m]> ncopa: The busybox trigger doesn't seem to run. Is /bin a symlink on these systems?
2026-05-02 12:52:59 <Sertonix[m]> Ether way that looks like an apk-tools bug
2026-05-02 12:53:56 <Sertonix[m]> Sooner or later we will need the CI to not downgrade from a newer alpine version. At some point the apkv3 installed db will not be readable by apkv2 anymore
2026-05-02 12:54:12 <ikke> Sertonix[m]: https://tpaste.us/yBVx 
2026-05-02 12:55:01 <ikke> registry.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci before (up/down)grade
2026-05-02 16:37:41 <ikke> bootstrapping ghc on aarch64
2026-05-02 17:42:20 <ikke> done
2026-05-02 19:13:09 <mio> getting "ERROR: Failed to remove network for build" on #204 (Hnyf5AhA) shared-runner che-ci-2 (loongarch64)
2026-05-02 19:13:28 <mio> thanks algitbot, but it's https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2332493
2026-05-02 19:21:40 <mio> ikke: ghc still seems to be looking for a specific bootstrap version https://build.alpinelinux.org/buildlogs/build-3-24-aarch64/community/ghc/ghc-9.10.3-r1.log
2026-05-02 19:34:49 <ikke> mio: the build faild, didn't notice
2026-05-02 19:42:49 <mio> okay
2026-05-02 19:49:32 <ikke> oof
2026-05-02 19:51:09 <ikke> aarch64 ci host fails to boot
2026-05-02 19:51:23 <ikke> 'invalid magic'
2026-05-03 11:17:00 <dne> shared-runner che-ci-2 (loongarch64) is having some docker issue: "ERROR: Preparation failed: Error response from daemon: client version 1.43 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version (docker.go:958:0s)"
2026-05-03 11:30:18 <ikke> dne: hmm, ncopa mentioned a similar issue after upgrading to 3.23
2026-05-03 11:32:46 <ikke> I suppose gitlab-runner is the client in this case
2026-05-03 11:46:26 <ikke> dne: and thanks for reporting
2026-05-03 11:48:03 <dne> no pb at all :) mio reported the same issue last night I believe, but not that specific error message
2026-05-03 12:13:59 <ikke> Updating gitlab-runner fixed it
2026-05-03 12:17:48 <dne> cool!
2026-05-03 15:10:38 <ikke> ncopa: for reference, the docker client version mismatch you ran into with alpine 3.23 means that gitlab-runner needs to be updated
2026-05-03 16:38:48 <aelin> hmmm is it just me or is IPv6 broken on the Alpine mirrors?
2026-05-03 16:39:32 <aelin> dl-1 does not have an AAAA record, for dl-2, dl-3 and dl-4, any attempt to connect over IPv6 just times out 
2026-05-03 16:39:46 <aelin> been like this for a few weeks now
2026-05-03 16:40:13 <aelin> see e.g. curl -6 https://dl-4.alpinelinux.org/alpine/
2026-05-03 16:45:20 <ikke> The dl-<n> mirrors are just aliases nowadays, kept them for backwards compatibility
2026-05-03 16:46:44 <aelin> ah okay, so the recommended URL would be dl-cdn now?
2026-05-03 16:47:15 <ikke> yes, but I do need to figure out why it's not working ipv6
2026-05-03 16:47:39 <ikke> dl-cdn points to fastly, while dl-<n> point directly to our servers
2026-05-03 16:48:50 <aelin> we actually hardcode dl-4 in pmbootstrap for that reason I think
2026-05-03 16:49:25 <aelin> because we want to avoid a case where the CDN may choose an outdated mirror, since it must be deterministic in our source URLs
2026-05-03 16:49:47 <ikke> aelin: It should not make any difference
2026-05-03 16:50:09 <aelin> I'll change that to dl-cdn then, thank you :)
2026-05-03 16:50:37 <ikke> dl-cdn points to exactly the same servers as the dl-<n> ones, but we do use geolocation for dl-<n>
2026-05-03 16:50:55 <ikke> For ipv6 however we only have one server that provides it sadly
2026-05-03 16:55:29 <ikke> aelin: fixed, https was missing from the firewall rules
2026-05-03 16:57:24 <aelin> oh nice, thank you! indeed works now
2026-05-03 16:58:17 <ikke> oh, lol
2026-05-03 17:00:23 <ikke> uhh
2026-05-03 17:03:11 <ikke> Always fun (not) the descrepency between ipv6 and ipv4 in docker
2026-05-03 17:03:45 <ikke> rsync should work again for ipv4, but because in (the installed version of) docker, ipv4 is forwarded, but ipv6 is not
2026-05-03 17:04:22 <ikke> Good that I didn't provide a AAAA record for ipv6 yet for rsync
2026-05-03 17:51:15 <mio> can the following archives be copied to 3.24 distfiles for rebuild please? https://distfiles.alpinelinux.org/distfiles/v3.23/grub-6811f6f09d61996a3acbc4fc0414e45964f0e2d9.tar.gz https://distfiles.alpinelinux.org/distfiles/v3.23/libfakekey-0.3.tar.gz
2026-05-03 17:57:49 <ikke> done
2026-05-03 17:58:16 <mio> thanks!
2026-05-03 18:07:35 <ikke> Interesting, when trying to connect to rsync on deu-t1-1 over ipv6 it manages to send a client initialization packet, but then the connection is terminated. That means the tcp handshake succeeded
2026-05-04 04:26:45 <ikke> meh
2026-05-04 09:26:14 <fcolista> hi friends, can someone help with this aarch64 error? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2333962
2026-05-04 09:27:26 <ikke> Sounds like an outdated gitlab-runner-helper image
2026-05-04 09:48:09 <fcolista> yes
2026-05-04 09:48:28 <fcolista> is it something I can fix?
2026-05-04 09:48:46 <ikke> I started a pipeline to rebuild the image
2026-05-04 09:48:51 <ikke> See if that helps
2026-05-04 10:22:45 <ikke> Hm, no
2026-05-04 10:29:03 <ikke> gitlab-runner in aports is a bit outdated
2026-05-04 10:52:04 <ncopa> maybe I should revert to alpine 3.22
2026-05-04 10:53:44 <ikke> It's working on the loongarch runners
2026-05-04 11:16:51 <ikke> ncopa: can you try `docker pull alpinelinux/gitlab-runner-helper:latest` on that host?
2026-05-04 11:17:31 <ikke> The -timeout flag was added in 18.10.0
2026-05-04 11:44:09 <ncopa> so docker compose pull is not enough?
2026-05-04 11:46:07 <ncopa> seems to work now. thanks!
2026-05-04 11:53:41 <ncopa> hum. not it doesnt. who do I check which version it actually runs?
2026-05-04 11:54:38 <ncopa> it uses registry.alpinelinux.org/alpine/infra/docker/gitlab-runner:latest
2026-05-04 12:03:32 <dne> docker run --rm -it --entrypoint gitlab-runner registry.alpinelinux.org/alpine/infra/docker/gitlab-runner:latest --version
2026-05-04 12:04:07 <dne> version 18.10.0?
2026-05-04 13:10:52 <ncopa> Version:      18.10.0
2026-05-04 13:13:05 <ikke> What about the helper image?
2026-05-04 13:15:24 <ncopa> Version:      17.2.1
2026-05-04 13:15:27 <ncopa> there we go
2026-05-04 13:15:53 <ncopa> happens even after a docker pull
2026-05-04 13:16:17 <ncopa> lima-alpine-gitlab-ci:~/compose/gitlab-runner-alpine-ci# docker run --rm -it --entrypoint gitlab-runner-helper  alpinelinux/gitlab-runner-helper:latest --version | tpaste
2026-05-04 13:16:17 <ncopa> https://tpaste.us/pyjj
2026-05-04 13:16:43 <ikke> Can you check registry.alpinelinux.org/alpine/infra/docker/gitlab-runner-helper:latest?
2026-05-04 13:17:40 <ncopa> https://tpaste.us/WWYY
2026-05-04 13:17:58 <ncopa> so I have pulled it, do I need to change the image somewhere?
2026-05-04 13:18:01 <ikke> Ok, I guess we no longer publish that image to docker hub, but it's still being used
2026-05-04 13:18:08 <ikke> ncopa: in the runner config
2026-05-04 13:18:20 <ikke> It should specify the helper image
2026-05-04 13:19:10 <ncopa> here? https://gitlab.alpinelinux.org/admin/runners/210/edit
2026-05-04 13:20:13 <ikke> No, docker compose exec gitlab-runner vi /etc/gitlab-runner/config.toml
2026-05-04 13:41:43 <ncopa> thanks
2026-05-04 15:52:40 <ncopa> Linode is back
2026-05-04 19:39:04 <blaboon> ikke: apologies for the delay. a bunch of stuff has changed and i unfortunately don't really have much visibility into our marketing/promo stuff anymore so i haven't been able to keep your account in standing. i saw your ticket though and reached out to other folks in marketing and they should be taking care of things now
2026-05-04 19:39:48 <ikke> blaboon: good to hear from you. 
2026-05-04 19:39:51 <ikke> blaboon: and thanks!'
2026-05-04 19:42:57 <ikke> blaboon: One thing we wanted to confirm, not sure if you know, would a suspension like we experienced affect linode dns as well?
2026-05-05 00:04:38 <blaboon> ikke: i believe our nameservers continue serving domains even from a suspended account, so i don't think they would have been affected
2026-05-05 05:01:26 <ikke> blaboon: yeah, that would be our expectation as well
2026-05-05 05:57:02 <ncopa> ikke: FYI: riscv64 kernel (for pioneer) has # CONFIG_CRYPTO_USER_API_AEAD is not set
2026-05-05 05:58:58 <ncopa> thats also the case for 6.6.53-2-spacemit (the bpi f3 kernel)
2026-05-05 06:01:21 <ikke> Ah, good to know, so not vulnerable 
2026-05-05 06:02:03 <ikke> Is the configs module loaded by default?
2026-05-05 06:02:14 <ikke> Could use it to check it with zabbix 
2026-05-05 06:03:47 <ncopa> no, configs module is not loaded by default
2026-05-05 06:04:51 <ncopa> but you can check /boot/config-$(uname -r)
2026-05-05 06:17:53 <ikke> Ok
2026-05-05 07:45:17 <omni> uhm when did build.a.o get a horizontal scrollbar?
2026-05-05 07:47:57 <omni> and why? sliding it kinda just toggles the visibility of the left and right border lines of the tables
2026-05-05 08:57:13 <ncopa> probably since the [online] badge showed up and took pixels in horizontal space
2026-05-05 08:58:15 <ncopa> we added a badge that shows the status of the builder. last time we did stable releases we had a riscv64 builder that had gone offline without anyone noticing
2026-05-05 08:58:54 <ncopa> so I added an mqtt will topic to tell when a builder unexpectedly went missing
2026-05-05 08:59:18 <ncopa> and added a 'offline' state for indicating that a builder was intentionally stopped
2026-05-05 08:59:56 <ncopa> hum... I though I put usa-t1-2.alpinelinux.org in maintenance mode
2026-05-05 09:05:02 <omni> ok, I only noticed when I scrolled to the bottom of the page
2026-05-05 09:30:05 <ncopa> sigh.. the usa-t1-2.a.o did not come back after a reboot
2026-05-05 10:38:45 <ikke> ok, good
2026-05-05 11:01:50 <ncopa> usa-t1-2 should be all back after some drama
2026-05-05 11:02:11 <ncopa> the disks were reordered and md0 did not come up
2026-05-05 11:02:34 <ncopa> the network didnt come up properly either, buts it back now
2026-05-06 04:12:02 <lotheac> ikke: do you think we could sort out a way for me to access the CI kube cluster and create pods there? i want to debug some test failures for python3 which only happen for x86/x86_64 CI, and I can't repro them on a fresh k0s cluster...
2026-05-06 05:27:24 <ikke> lotheac: Yeah, I think that should be possible
2026-05-06 05:29:02 <lotheac> the apiserver lb is on public internet, i can access port 6443... so i just need a way to authenticate to it (and some privileges via a role or clusterrole)
2026-05-06 05:29:30 <lotheac> i think you mentioned using gitlab as openid provider for oidc auth at some point?
2026-05-06 05:29:58 <ikke> Yeah, that would be ideal
2026-05-06 05:30:24 <ikke> But I think for the interrim I can also arrange something
2026-05-06 05:30:57 <lotheac> cheers
2026-05-06 09:14:25 <ncopa> ikke: I'm upgrading kernel on nld-t1-2 now. and then I'll do nld-t1-1
2026-05-06 09:14:59 <ikke> Ok, do note that we do not have console access for those I believe
2026-05-06 09:15:20 <ikke> Oh, we should have for HorizonIQ, but not for Osso
2026-05-06 09:57:16 <ncopa> nld-t1-2 is upgraded to alpine 3.23
2026-05-06 16:49:09 <ncopa> bootstrapping ghc on aarch64 failed
2026-05-06 17:25:57 <ikke> ncopa: documentation issue?
2026-05-06 17:26:14 <ikke> ncopa: mio suggested https://gitlab.haskell.org/ghc/ghc/-/commit/e8f5a45de561ec80c88cd3da2c66502deb32d4c3
2026-05-07 06:01:58 <mio> can matchbox-keyboard tarball be added to 3.24 distfiles for rebuild? thanks! https://distfiles.alpinelinux.org/distfiles/v3.23/matchbox-keyboard-0.1.1.tar.gz
2026-05-07 07:00:55 <ncopa> mio: done
2026-05-07 15:37:12 <mio> ncopa: thanks!
2026-05-07 22:58:53 <omni> I don't see 3.20 builders on build.a.o
2026-05-07 22:59:50 <omni> oh, supposedly not supported since first of april? I thought we'd drop it once 3.24 is out
2026-05-08 03:41:34 <ikke> omni: since msg.a.o got rebooted, it lost the messages for those. Trying those builders will make them reappear 
2026-05-08 07:05:37 <dne> that downgrade issue is somewhat blocking a few security upgrade MRs for 3.20
2026-05-08 07:17:03 <ncopa> it does. I havent been able to reproduce it locally
2026-05-08 07:25:36 <ikke> Ftr: https://gitlab.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci/-/merge_requests/32
2026-05-08 07:25:52 <dne> hm weird, today I can't reproduce it either - the other day I could
2026-05-08 07:38:15 <dne> no sorry, used the wrong image. I can still reproduce: https://paste.dismail.de/?35243bd4dafa5fb7#5SXcaf82CcviaHJfYQgXWiYUwve8PteZmyQB6reYPGYU
2026-05-08 09:02:41 <ncopa> dne: thanks. I can reproduce too now. I think I know what happens. edge runs apk3. when downgrading on v3.20 it installs older busybox with apk3 + older apk 2. the apk2 is not able to find/execute the busybox trigger installed by apk3.
2026-05-08 09:02:54 <ncopa> workaround: apk fix busybox after downgrade
2026-05-08 09:03:01 <ncopa> but I think it applies to all triggers
2026-05-08 09:04:02 <ikke> Merging MR 3w2
2026-05-08 09:04:04 <ikke> Merging MR 32
2026-05-08 09:04:16 <ikke> Still need some changes in aports ci to make it use those images
2026-05-08 09:04:52 <ikke> Though, I could hardcode it in 3.20 for now to see if it works
2026-05-08 09:05:12 <ncopa> upgrade apk to newer 2.x should also work
2026-05-08 09:06:01 <ikke> I believe sertonix mentioned that, once we switch to apkv3 indexes, it would break anyway
2026-05-08 09:12:08 <lotheac> hmph... can't repro the python3 test failures in the ci kube cluster either, with a manual build. ikke: could i ask for read-only access to the namespace gitlab ci uses? i want to see the pod specs of the CI pods, maybe there's something there that affects the behavior
2026-05-08 09:12:50 <lotheac> (although, that's probably for next week, got places to be)
2026-05-08 09:40:49 <ikke> dne: it's working now: https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2340134
2026-05-08 10:10:53 <ncopa> lotheac, ikke: thanks!
2026-05-08 10:21:59 <Sertonix[m]> ikke: * installed db
2026-05-08 10:22:48 <ikke> Sertonix[m]: ah ok
2026-05-08 10:49:42 <dne> ikke: great!
2026-05-08 10:50:23 <ikke> So rebasing should fix any pipelines that failed due to this on 3.20
2026-05-08 10:51:29 <dne> cool, will do
2026-05-08 10:59:50 <ikke> ncopa: I've added an extra page to the kernel dashboard for CVE-2026-43284
2026-05-08 11:00:40 <ikke> Waiting for CVE-2026-43500
2026-05-08 11:17:57 <ncopa> ikke: thanks! I can start upgrading kernel on the builders once the kernel is available
2026-05-08 11:20:21 <ikke> Right
2026-05-08 13:59:45 <ncopa> im rebooting the arm build server
2026-05-08 14:01:49 <ikke> Ok
2026-05-08 14:10:19 <ncopa> im rebooting che-bld-2, the loongarch64 builder
2026-05-08 14:18:00 <ikke> ncopa: has the 2nd CVE already been fixed? I see comments that they are still working on it
2026-05-08 15:37:13 <ncopa> ikke: i dont think so
2026-05-08 15:41:30 <ikke> https://lwn.net/Articles/1071775/
2026-05-08 15:41:35 <ikke> "Kroah-Hartman has confirmed that a second patch is required, but it is still in development and has not yet been merged"
2026-05-08 15:41:45 <ikke> https://lwn.net/ml/all/2026050859-ahead-anchovy-05e2@gregkh/
2026-05-09 20:05:23 <mio> are build-3-24-{x86,x86_64} online? they are not displayed on build.a.o, just checking if it is just my browser having a stale cache
2026-05-09 20:08:13 <mio> tried to clear cache but they have not yet reappeared
2026-05-09 20:19:49 <ikke> they are stopped
2026-05-09 20:20:06 <ikke> at least x86_64
2026-05-09 20:20:46 <ikke> Yeah, only build-3-24-x86_64 was stopped
2026-05-09 20:21:45 <mio> ah okay ... they're back now? thanks
2026-05-09 20:22:03 <mio> seeing them listed now
2026-05-09 20:22:22 <ikke> Yeah, x86 was hanging
2026-05-13 06:26:52 <fcolista> FYI: did you notice that the "LAST DEVELOPMENT" section in alpinelinux.org is stucked to May 4th?
2026-05-13 06:27:14 <fcolista> Maybe this is wanted because May 4th be with you... :D
2026-05-13 06:40:37 <ikke> Fixed, the service crashed
2026-05-13 06:58:33 <fcolista> thx ikke 
2026-05-13 18:35:53 <omni> build-edge-x86_64 lost
2026-05-13 18:37:24 <ikke> omni: I'm testing traefik
2026-05-13 18:38:10 <ikke> but cannot reproduce the failure
2026-05-13 18:42:30 <omni> aha!
2026-05-13 20:15:07 <mio> can build-3-24-armhf and build-3-24-ppc64le builds be restarted please?
2026-05-13 21:02:33 <ncopa> rebooted them
2026-05-13 21:05:40 <mio> thanks!
2026-05-14 12:00:04 <fcolista> Hi friends! Is build-edge-riscv64 builder working?
2026-05-14 12:04:10 <mio> yes? as of 3m ago?
2026-05-14 12:05:31 <fcolista> 3m is 3 min or 3 months?
2026-05-14 12:05:36 <mio> 3 minutes
2026-05-14 12:11:33 <fcolista> oh ok...fiuu
2026-05-14 16:06:43 <ikke> durrendal: Jason from internet archive responded
2026-05-14 16:26:01 <durrendal> This is awesome, really exciting to see they're amenable to this idea!
2026-05-14 16:27:56 <durrendal> It sounds like all we need is a dedicate repo archive account, and to figure out a migration plan
2026-05-14 16:28:19 <ikke> right
2026-05-14 16:28:28 <ikke> would you be open to investigate what is needed?
2026-05-14 17:14:50 <durrendal> yes more than happy to figure out what we need to do to move this forward. :)
2026-05-14 19:41:21 <ikke> Seems to be some kind of routing issue
2026-05-14 19:41:54 <ikke> I can reach it, but from there, I cannot reach most of the alpine infra
2026-05-16 04:58:15 <ikke> ncopa: I'm planning on upgrade gitlab today (including updating kernel). 
2026-05-16 04:58:25 <ikke> Any time that would be convenient for you?
2026-05-16 07:01:54 <ncopa> maybe 13:00?
2026-05-16 07:02:20 <ncopa> i mean, it does not matter that much today
2026-05-16 07:02:40 <ncopa> how long downtime are you expecting?
2026-05-16 07:03:44 <ikke> 15 minutes perhaps
2026-05-16 07:26:48 <ncopa> then you can do it anytime
2026-05-16 07:27:02 <ncopa> im pushing kernel updates to older branches now
2026-05-16 07:28:01 <ikke> ncopa: thanks
2026-05-16 07:28:15 <ikke> I'll also work on updating and rebooting the servers'
2026-05-16 08:01:03 <ikke> rebooting the wireguard hub for upgrades
2026-05-16 08:15:26 <ikke> ncopa: I'm a bit puzzled why in some cases we need to explicitly specify the docker API version for traefik, and in some cases we don't
2026-05-16 08:49:52 <ikke> I see that the scaleway servers use a kernel from scaleway, we cannot upgrade it
2026-05-16 09:35:52 <ikke> rebooting che-bld-1 (aarch64/arm* build server)
2026-05-16 09:41:30 <ikke> It's back
2026-05-16 13:04:33 <ikke> Ugh, boot issues.
2026-05-16 13:08:23 <ikke> Ok, just another reboot
2026-05-16 13:24:50 <ikke> ncopa: on usa6-dev1: ERROR: linux-lts-6.6.139-r0: No space left on device. /boot is only 100M on that host
2026-05-16 13:26:14 <ikke> I moved vmlinux-lts somewhere else so that it had enough space to write the new image
2026-05-16 14:11:36 <ikke> Rebooting x86 build host
2026-05-16 16:06:01 <ncopa> I have usually moved the files to a temp dir, done the apk upgrade and then deleted the temp files
2026-05-16 16:06:23 <ikke> Yes, that worked
2026-05-17 06:06:44 <ncopa> I have moved the build-3-24-riscv64 to my p550 to see if it is faster or not
2026-05-17 06:20:54 <ikke>  Ok
2026-05-17 06:21:21 <ikke> Do you have a way to measure that?
2026-05-17 06:56:00 <ncopa> nope
2026-05-17 06:56:10 <ncopa> just "gut feeling" at this point
2026-05-17 06:57:31 <ncopa> I think we need build-3-24-riscv64 progress 260 packages per day minimum
2026-05-17 07:00:23 <ikke> Yeah, was thinking of a rough idea
2026-05-17 08:24:32 <ikke> Just stating it here to have a record of it. Pushing large docker images to our registry started to fail since upgrade the server. Part of the upgrade was updating traefik to v2.11. It turns out v2.11 started adding timeouts (https://doc.traefik.io/traefik/v2.11/migration/v2/#entrypointtransportrespondingtimeoutsreadtimeout). Adding a larger readTimeout to the entrypoint fixed
2026-05-17 08:24:34 <ikke> the issue
2026-05-17 11:03:34 <ikke> The wiki gets het by requests from many ip addresses
2026-05-18 02:38:27 <huajingyun> gitlab.alpinelinux.org is down?
2026-05-18 03:51:09 <ncopa> apparently it is
2026-05-18 04:13:16 <ncopa> load average: 159.42, 139.09, 131.09
2026-05-18 04:34:36 <ncopa> it is gitaly again
2026-05-18 04:34:59 <ncopa> I am investigating
2026-05-18 04:46:31 <ncopa> im stopping gitlab
2026-05-18 05:09:37 <ncopa> ikke: I have limited the concurrency of gitaly a bit
2026-05-18 05:11:40 <ikke> ncopa: ok, let's see if that helps
2026-05-18 05:32:57 <ncopa> I see that we have a small swap disk attached to it. Could we increase the size to 4G?
2026-05-18 05:39:44 <ikke> ncopa: would that really help?
2026-05-18 05:40:10 <ikke> I think it's disk contention that is the bottle neck 
2026-05-18 06:00:49 <ncopa> there was OOM involved
2026-05-18 06:01:32 <ncopa> lots of disk io so we need the memory for disk cache. swapping out cold data makes sense
2026-05-18 06:02:14 <ncopa> I have added temp zram
2026-05-18 06:02:34 <ncopa> i think a little swap on disk may help
2026-05-18 06:02:48 <ncopa> but the real problem is that server is overloaded
2026-05-18 06:06:23 <ncopa> other idea I had was that we could have the builders git pull from the cgit instance instead of gitlab
2026-05-18 06:07:48 <ncopa> we have 18 builders pulling at the same time
2026-05-18 06:09:34 <ncopa> so sync cgit instance via webhook, once the sync is done, do the mqtt publish. the builders then do git pull from the cgit instance
2026-05-18 06:10:03 <ncopa> would reduce 18 pulls
2026-05-18 06:22:39 <ikke>  I did switch some builders already to use git.a.o
2026-05-18 06:23:08 <ikke> There can be race conditions though where git.a.o is not yet up-to-date
2026-05-18 06:24:13 <ikke> For the record, overall the server is doing fine resource wise. It's just certain moments where it reaches a state that the load compounds
2026-05-18 06:27:21 <ikke> And the processes stacking up leads to increased memory usage as well
2026-05-18 06:50:54 <huajingyun> thanks
2026-05-18 07:00:03 <ikke> ncopa: so yes, having the builders trigger on the git.a.o update would alleviate pressure of our gitlab instance
2026-05-18 18:21:48 <ncopa> I have added a --status-up/down-* feature to mqtt-exec. It can now automatically publish 'online' after a reconnect, so it does not show up as 'lost' til next restart
2026-05-18 18:22:18 <ncopa> I have only updated the config for build-edge-ppc64le where I have verified that it works as expected
2026-05-18 18:27:12 <ikke> Nice
2026-05-18 18:45:30 <mio> is there currently a block on fetching build.a.o logs via busybox wget?
2026-05-18 18:45:57 <mio> can view them from a browser, but not download with wget
2026-05-18 18:46:37 <mio> connection seems to eventually time out
2026-05-18 18:49:25 <mio> `wget: can't connect to remote host (172.105.82.32): Operation timed out`
2026-05-18 18:52:01 <ikke> mio: what url?
2026-05-18 18:54:38 <mio> https://build.alpinelinux.org/buildlogs//build-edge-riscv64/testing/asdf/asdf-0.18.0-r10.log
2026-05-18 18:54:55 <mio> or any other ones on build.a.o
2026-05-18 18:55:41 <ikke> There is nothing in front that would block on user agent or something like that
2026-05-18 18:56:20 <ikke> mio: does curl or gnu wget work?
2026-05-18 18:58:20 <ncopa> works for me
2026-05-18 18:59:35 <mio> just rechecked, it works from one machine and not another, it may have accidentally got caught in a filter maybe?
2026-05-18 19:00:19 <ikke> nothing fancy happening on that server
2026-05-18 19:00:21 <mio> curl doesn't work either on the machine where busybox wget didn't
2026-05-18 19:00:21 <ikke> just plain nginx
2026-05-18 19:01:36 <mio> okay, thanks for checking nonetheless
2026-05-18 19:11:51 <mio> ikke informed me it's likely a transient routing issue. will try again later (in case anyone else is seeing something similar)
2026-05-19 08:40:50 <ikke> mio: curious, is that machine still having issues reaching build.a.o?
2026-05-19 12:06:36 <mio> ikke: it's fine now, thanks for asking :)
2026-05-19 12:06:47 <ikke> Ok, glad to hear
2026-05-20 07:55:41 <ncopa> gitlab is down again
2026-05-20 08:02:50 <ikke> Ugh
2026-05-20 18:09:41 <Sertonix[m]> Has go-away blocked the CI from accessing gitlab.a.o? https://gitlab.alpinelinux.org/dev_urandom/aports/-/jobs/2357758
2026-05-20 18:53:26 <ikke> Sertonix[m]: I did make a change, reverted it again, let me know if it still happens
2026-05-23 02:32:56 <qaqland> Error: access denied: denied by administrative rule 877b4b3acaf5621dcaf110344571fea4/560c674e98e70c71aa1e
2026-05-23 02:37:43 <qaqland> I'm using a proxy network, maybe it's because the quality of the proxy server is not good...
2026-05-23 05:16:38 <ikke> qaqland: do you have a request id (preferably a recent one?
2026-05-23 09:01:00 <qaqland> ikke: I have switched to another proxy node, and it could access now :)
2026-05-23 20:52:32 <omni> build-edge-x86 has been stuck for a while
2026-05-24 10:04:06 <omni> there's nothing stuck with package uploads, is there?
2026-05-24 10:05:28 <omni> I'm just used to them being available fairly soon after build, and I mean both on the fastly mirror and as listed on pkgs.a.o
2026-05-24 12:00:22 <omni> mysql is still the latest available aport
2026-05-24 12:02:24 <omni> mariadb*
2026-05-24 13:15:24 <omni> seems to be catching up now
2026-05-24 13:15:38 <ikke> Ok, I have been away this weekend, will check once I'm back
2026-05-25 10:40:57 <omni> is build-3-23-aarch64 stuck?
2026-05-25 10:55:35 <ikke> omni: seems idle
2026-05-25 10:56:35 <ikke> omni: restarted the build service, and it's now building
2026-05-25 10:56:44 <omni> thanks!
2026-05-25 10:57:36 <omni> since it's chromium, it should be done or fail again before midnight
2026-05-25 10:57:48 <omni> UTC
2026-05-25 20:01:47 <omni> gitlab is having a day today, isn't it?
2026-05-25 20:04:58 <ikke> restarting gitlab
2026-05-25 20:13:18 <mio> had a similar issue as in the pulumi MR, after rebase the MR author's commit/changes disappeared
2026-05-25 20:15:26 <ikke> mio: similar issue as what?
2026-05-25 20:15:59 <omni> !102897
2026-05-25 20:16:30 <omni> very weird
2026-05-25 20:16:42 <mio> that one, yes
2026-05-25 20:16:45 <mio> !103002
2026-05-25 20:17:45 <mio> fetched the branch and the commit/changes are there
2026-05-25 20:18:18 <ikke> gitab computes the mr diff asynchronously
2026-05-25 20:22:07 <ncopa> i think i will move back the build-3-24-riscv64 to the pioneer machine. the p550 has not been fast enough
2026-05-25 20:36:04 <ncopa> I have moved back the build to build-3-24-riscv64
2026-05-25 20:36:23 <ncopa> edge is finished so it should not be as much load as it was before
2026-05-25 20:50:14 <mio> ikke: that makes sense. it would appear the diff lagged a bit behind the rebase as presented in the ui, but eventually it got there
2026-05-26 14:03:34 <Sertonix[m]> Is the x86 CI broken?
2026-05-26 14:16:41 <ikke> Sertonix[m]: I was working on updating those. The provider said they could upgrade them, so I put all of them in maintenance, but it turns out it's better to do it myself.
2026-05-26 14:16:52 <ikke> I'll enable those runners again
2026-05-26 14:17:51 <ikke> Sertonix[m]: can you retry a job and see if it works now?
2026-05-26 14:26:52 <Sertonix[m]> Works, thanks!
2026-05-26 16:03:28 <aelin> ikke: heya, I'm wondering how the builders and CI are set up for armhf. What kind of hardware is used for that?
2026-05-26 16:03:52 <ikke> aelin: we use aarch64 hosts, and run armhf userspace with linux32
2026-05-26 16:04:11 <ikke> (The CPU would need to support 32-bits mode)
2026-05-26 16:04:33 <aelin> any attempts I've had at running armhf userspace on aarch64 so far only resulted in SIGILL when executing mcr instructions
2026-05-26 16:04:42 <aelin> because those changed behavior in armv7 
2026-05-26 16:04:49 <aelin> and the aarch32 mode is only compatible with armv7 and not armv6
2026-05-26 16:05:26 <aelin> maybe there's something I'm missing here
2026-05-26 16:05:38 <ikke> Not sure if related, but we do set abi.cp15_barrier = 2
2026-05-26 16:05:50 <aelin> AH! that sounds related
2026-05-26 16:05:50 <ikke> (sysctl)
2026-05-26 16:06:01 <aelin> in particular the crashing instruction is mcr p15, #0x0, r0, c7, c10, #0x5
2026-05-26 16:08:29 <aelin> yeah that looks like exactly what I was looking for, thank you!
2026-05-26 16:08:42 <ikke> Great
2026-05-26 16:08:51 <ikke> Had to remember where we set that again
2026-05-27 02:07:32 <ptrc> https://ptrc.gay/nWLAGXLT.png
2026-05-27 02:07:32 <ptrc> hm
2026-05-27 02:07:34 <ptrc> that looks odd
2026-05-27 02:08:05 <ptrc> actually.
2026-05-27 02:08:10 <ptrc> https://ptrc.gay/EgHntKMq.png
2026-05-27 02:08:22 <ptrc> this is last 12h
2026-05-27 02:08:45 <ptrc> it looks like something spamming some tmpfs
2026-05-27 02:09:31 <ptrc> the memory usage percentage doesn't really go down at any point
2026-05-27 02:10:58 <ptrc> it's only che-ci-1 though, others are chaotic as ever
2026-05-27 03:13:15 <ikke> If you look at a larger period, you'll see it's normal. The host has 3 vms, each for one of the arches
2026-05-27 03:13:55 <ikke> It's just that the host was rebooted yesterday
2026-05-27 03:21:23 <ptrc> ah, huh
2026-05-27 03:21:34 <ptrc> and i assume the VMs don't free their memory back to the host?
2026-05-27 03:26:02 <ptrc> also, are those VMs also hooked up to Zabbix? i wanted to look at memory usage for the aarch64 CI jobs, but i'm having a hard time figuring out which runner is on which host now :p
2026-05-27 05:23:56 <ikke> ptrc gitlab-runner-*
2026-05-27 05:24:25 <ptrc> makes sense :D
2026-05-27 05:25:01 <ptrc> though i still can't match some runners to Zabbix hostnames, e.g. nor-ci-2
2026-05-27 05:39:33 <ikke> They are not in Zabbix.
2026-05-27 05:39:39 <ikke> Managed by ncopa
2026-05-27 05:46:37 <ptrc> ah!
2026-05-27 05:46:39 <ptrc> ploe
2026-05-27 05:46:40 <ptrc> okie *
2026-05-27 11:35:34 <omni> remote: rpc error: code = Unavailable desc = The git server, Gitaly, is not available at this time. Please contact your administrator.
2026-05-27 11:36:19 <omni> when trying to push to gitlab.a.o
2026-05-27 11:41:27 <omni> and now it works again
2026-05-27 12:03:38 <omni> for some subset of "works"...
2026-05-27 12:04:01 <omni> ikke: this is not complaining about your work! <3
2026-05-27 15:08:55 <ncopa> build-3-24-riscv64 builders has full disk
2026-05-27 15:14:00 <ncopa> we need to clean it up a bit
2026-05-27 15:14:14 <ncopa> not sure how to use the cleanup-builders script
2026-05-27 15:14:46 <ncopa> sh cleanup-builders | xargs -0 rm -r ?
2026-05-27 15:18:04 <ikke> Yes
2026-05-27 15:34:39 <ikke> omni: still issues?
2026-05-28 09:37:44 <omni> ikke: I was away for a bit but I don't experience them now