2025-10-01 06:24:47 <ncopa> those branches are not used often since they are EOL, so their caches etc are probably dead cold
2025-10-01 06:37:36 <ikke> ncopa: i have already updated them recently 
2025-10-01 06:39:31 <ncopa> but I also wonder if we are low on resources on the gitlab machine, if it is our own machines that causes it
2025-10-01 06:40:58 <ikke> It's a pathological case
2025-10-01 06:41:09 <ikke> Hard to get enough resources for that
2025-10-01 07:52:12 <ikke> What we need to fix is the performance cost of those git operations. The memory usage and load caused by the pile up of many of such processes should vanish as a result 
2025-10-01 07:55:57 <lotheac> it's only those builders? fetches done by later releases have different perf characteristics?
2025-10-01 07:57:33 <ikke> Correct 
2025-10-01 07:57:44 <ikke> Possibly git protocol related 
2025-10-01 07:57:55 <lotheac> git over https, i assume?
2025-10-01 07:57:59 <ikke> Yes
2025-10-01 08:11:28 <lotheac> did minor spelunking over the release notes between v3.18 git (2.40.4) and v3.19 git (2.43.7) and came up with this wild guess https://github.com/git/git/commit/eaa0fd658442c2b83dfad918d636bba3ca3b4087
2025-10-01 08:12:14 <lotheac> it may very well be something different though
2025-10-01 10:27:14 <ikke> lotheac: using GIT_TRACE_CURL=1, it shows protocol v2 is used: Send header: Git-Protocol: version=2
2025-10-01 10:51:44 <ikke> Just a single git pull takes >1m
2025-10-01 10:53:27 <ikke> real    2m 18.07s
2025-10-01 10:53:57 <ikke> which brought in just 2 commits
2025-10-01 12:07:48 <ncopa> so separating giatly to a dedicated server, with more memory,cpu and faster storage wouldnt help?
2025-10-01 12:08:21 <ncopa> maybe we should have the builders to pull from the github mirror?
2025-10-01 12:08:48 <ncopa> im just thinking out loud...
2025-10-01 14:14:57 <ncopa> something appears to be wrong with the milk-v pioneer machine
2025-10-01 14:15:09 <ncopa> occationally processes hangs on 99.9% cpu
2025-10-01 14:15:28 <ncopa> first it was in perl process when building openssl in both 3.22-stable and 3.21-stable
2025-10-01 14:15:47 <ncopa> i killed the process and retried and 3.22-stable passed
2025-10-01 14:16:15 <ncopa> now it is a cc1 process that hangs on 3.21-stable
2025-10-01 14:17:21 <ncopa> retrying now
2025-10-01 14:56:38 <lotheac> ikke: are those findings true from both 3.18 and more recent builders? for me the diff looked like it might cause v0 instead of v2 for newer ones
2025-10-01 14:56:48 <lotheac> just fact-finding
2025-10-01 14:57:32 <ikke> I'll check it
2025-10-01 15:05:58 <ikke> lotheac: on edge:  => Send header: Git-Protocol: version=2 
2025-10-01 15:08:52 <ikke> ncopa: we could also try to point them to git.a.o. My worry though is race conditions (builders get a notification and start pulling before the other repo is updated)
2025-10-01 15:09:34 <lotheac> right, so same protocol - that wasn't the problem
2025-10-01 15:10:40 <lotheac> i also thought about giving them a mirror to hit - it would solve the immediate problem but not the real issue
2025-10-01 15:11:14 <lotheac> that said, if nobody is abusing the real issue, good enough, right? :)
2025-10-01 15:16:52 <ncopa> i suppose we could fix the notifications. builders could subscribe to notification that git.a.o is synced
2025-10-01 15:17:50 <ikke> We can first try and see if it's an actual issue, and then adjust if necessary
2025-10-01 15:18:09 <ikke> And the other part is that <=3.18 are EOL anyway, so we do not expect many commits on them
2025-10-01 15:54:01 <ikke> switched the 3.17 builders to git.a.o, and retrying them makes them finsh (before they would just hang on pulling git)
2025-10-01 15:57:33 <ikke> same for 3.18
2025-10-01 16:11:11 <achill> https://build.alpinelinux.org/buildlogs/build-edge-ppc64le/testing/fungw/fungw-1.2.2-r0.log
2025-10-01 16:11:23 <achill> Can't open ".SIGN.RSA./home/buildozer/.abuild/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub" for writing, No such file or directory
2025-10-01 16:11:29 <achill> That looks creepy
2025-10-01 18:41:21 <ikke> achill: not really sure what happened, but I don't see anything out of the ordinary on that builder
2025-10-01 18:41:59 <achill> at least now it seems to build again
2025-10-01 18:42:08 <achill> before it didn't even pick up new pkgs
2025-10-01 23:29:02 <Sertonix[m]> achill: That package pulled in dash-binsh and abuild-sign which failed uses /bin/sh as shebang. And it seems like keyname=${pubkey##*/} didn't work as expected which seems impossible to fail.
2025-10-01 23:32:11 <Sertonix[m]> Would need to re-check the posix spec but this looks like a dash bug: v=foo/bar dash -c 'echo ${v##*/}'
2025-10-02 00:26:58 <Sertonix[m]> Could anybody please backport https://git.kernel.org/pub/scm/utils/dash/dash.git/commit/?id=6dcc007a72f13c3e518a65bffef571795ad6678c
2025-10-02 05:56:55 <mio> can someone maybe check on the 3.22 riscv64 builder progress? it might be stuck
2025-10-02 10:23:36 <ikke> ncopa mentioned that he noticed processes started to spin on that builder
2025-10-02 10:23:45 <ikke> I see rustc process spinning 100% now
2025-10-02 10:29:16 <ikke> I've restarted it
2025-10-02 12:41:25 <mio> thanks!
2025-10-02 13:22:50 <ncopa> build-3-21-riscv64 now get compiler internal errors
2025-10-02 13:23:06 <ncopa> i wonder if we should try do a reboot
2025-10-02 13:24:58 <ikke> We can try
2025-10-02 16:51:45 <ikke> ncopa: should I reboot it?
2025-10-02 17:03:47 <ncopa> ok
2025-10-02 17:13:29 <ikke> done
2025-10-02 17:53:14 <ncopa> thanks
2025-10-02 17:55:16 <ikke> If you have a moment, can you check https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/90971?
2025-10-02 17:55:22 <ikke> This keeps breaking the ppc64le builder
2025-10-04 20:16:01 <ikke> nu_: We lost the default route on the builder
2025-10-04 20:16:10 <ikke> (ipv6)
2025-10-04 20:17:43 <ikke> I can still access it via the OOB network
2025-10-06 08:50:10 <ncopa> so the build-3-22-aarch64 is unavailable now?
2025-10-06 08:52:30 <ikke> Yes, I've forwarded an email about it 
2025-10-07 12:13:59 <ncopa> still no arm64 builder?
2025-10-07 13:27:47 <ikke> Nope 
2025-10-07 15:36:29 <ikke> ncopa: now it is ^
2025-10-07 15:43:39 <nu_> it is back, sorry about it
2025-10-07 15:45:05 <nu_> i should check this channel more often
2025-10-07 15:45:39 <nu_> could i get email notifs when the builder goes down? i remember we you mentioned that it would be a possibility
2025-10-07 16:12:26 <ikke> nu_: right we did, I forgot about it
2025-10-07 16:13:31 <ikke> To your protonmail address?
2025-10-07 16:22:47 <nu_> yup
2025-10-08 06:02:06 <ncopa> nu_: thank you!
2025-10-08 12:37:20 <nu_> welcome!
2025-10-08 18:24:50 <ikke> nu_: I'm triggering an issue to test if you get the alert for it
2025-10-08 19:42:04 <ikke> nu_: Did you receive the email now?
2025-10-08 22:41:17 <nu_> ikke: no:/
2025-10-09 11:09:10 <ikke> clandmeter: network issues ^?
2025-10-09 11:21:04 <ikke> The dmpvn router is back, but the builders not yet. The router was rebooted
2025-10-10 07:01:57 <ikke> clandmeter: thanks
2025-10-10 08:37:17 <clandmeter> looks like the power was down when i was away
2025-10-10 08:39:53 <ncopa> clandmeter: wb
2025-10-11 02:55:02 <lotheac> something's up with the gitlab x86_64 runners: Unschedulable: "0/3 nodes are available: 1 Insufficient cpu, 1 Insufficient memory, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling."
2025-10-11 02:56:31 <lotheac> node.kubernetes.io/unreachable sounds like a network partition between the node controller and those nodes
2025-10-12 03:48:38 <qaqland> s390x runner coredumped when unpacking
2025-10-12 06:55:27 <ikke> qaqland: there's a musl regression affecting s390x
2025-10-14 15:07:22 <ncopa> im bootstrapping 3.23 builders now
2025-10-14 19:34:33 <ptrc> apparently the x86 CI works differently between kubernetes and x86.ci.alpinelinux.org when it comes to 64-bit detection
2025-10-14 19:35:11 <ptrc> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2053544
2025-10-14 19:35:23 <ptrc> here, when it runs on kubernetes, the architecture is detected as x64
2025-10-14 19:35:34 <panekj> I'm curious how the CPU bitness leaks
2025-10-14 19:35:40 <ptrc> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2053531
2025-10-14 19:35:50 <ptrc> here, on the shared runner, it's ia32 (correct)
2025-10-14 19:36:32 <ptrc> panekj: it depends how python's "platform.machine()" works
2025-10-14 19:37:55 <panekj> os.uname()
2025-10-14 19:37:57 <panekj> unless it fails
2025-10-14 19:38:33 <ptrc> could it be that on kubernetes it somehow doesn't do `linux32` / whatever the equivalent would be
2025-10-14 19:39:06 <ptrc> well, let's try that i gues
2025-10-14 19:39:07 <ptrc> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/91519/diffs#9853bba0d23ae23ea74b8cef51bc99a59d0780dd_59_59
2025-10-14 19:39:14 <ptrc> yes thank you algitbot
2025-10-14 19:40:02 <panekj> interesting results
2025-10-14 19:40:03 <ptrc> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2053568
2025-10-14 19:40:04 <ptrc> indeed
2025-10-14 19:43:26 <panekj> how is it setup to run 32-bit mode in k8s?
2025-10-14 19:44:15 <ikke> I wasn't really ready yet to commission x86 on kubernetes yet
2025-10-14 19:45:52 <ptrc> oh, so x86 CI shouldn't even be on k8s?
2025-10-14 19:46:23 <panekj> looking at py source it should be just `uname -p` output?
2025-10-14 19:46:38 <ptrc> you mean -m?
2025-10-14 19:46:48 <panekj> ['uname', '-p'],
2025-10-14 19:47:07 <ptrc> that just prints `unknown` everywhere for me
2025-10-14 19:47:44 <f_> unknown for me too
2025-10-14 19:48:02 <f_> Kind of expected given it's non-portable
2025-10-14 19:48:41 <ptrc> not to mention that cpython doesn't exec `uname` or anything, it does the syscall
2025-10-14 19:48:44 <ptrc> so idk where `-p` came from
2025-10-14 19:49:17 <ikke> The idea is that it should run with linux32
2025-10-14 19:49:55 <panekj> ptrc: in class _Processor
2025-10-14 19:51:37 <ptrc> that's just a fallback and it doesn't execute for me
2025-10-14 19:51:53 <ptrc> and not like it matters, since `machine` is the field that we care about
2025-10-14 19:52:13 <panekj> dunno, all python code is disaster
2025-10-14 19:52:25 <panekj> anyway the issue is known
2025-10-14 19:53:03 <panekj> ikke: so it doesn't run with it now 
2025-10-14 19:53:06 <panekj> ?
2025-10-14 19:53:18 <ikke> I would expect it to
2025-10-14 19:53:38 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/.gitlab-ci.yml?ref_type=heads#L93
2025-10-14 19:53:48 <ikke> But maybe the entrypoint is ignored with the kubernetes executor
2025-10-14 19:55:42 <panekj> do you have FF_KUBERNETES_HONOR_ENTRYPOINT feature flag enabled?
2025-10-14 19:56:10 <ikke> Not explicitly
2025-10-14 19:56:12 <panekj> "When enabled, the Docker entrypoint of an image will be honored if FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY is not set to true"
2025-10-14 19:57:45 <panekj> uh
2025-10-14 19:58:02 <panekj> idk if that applies or not
2025-10-14 20:21:38 <ikke> Me neither
2025-10-14 20:21:57 <ikke> I wanted to verify first before enabling it
2025-10-14 20:28:04 <panekj> well, it's verified now :P
2025-10-16 09:16:41 <nu_> ikke: unfortunately i still didnt get the monitoring mail:/
2025-10-16 09:17:21 <nu_> a wait, i responded to something i already responded, sry, ignore moe :p
2025-10-16 09:44:02 <ncopa> ikke: how can i remove the announcement of holding back merges to master?
2025-10-16 09:48:53 <ikke> Broadcast messages in the admin panel 
2025-10-16 09:57:21 <ncopa> done. thanks!
2025-10-16 13:26:40 <ncopa> I asked around in the #gcc channel about the ICE segafults we have when building clang on build-3-23-riscv64
2025-10-16 13:26:49 <ncopa> they say it is hardware
2025-10-16 13:26:58 <ncopa> so we have unreliable hardware for riscv64
2025-10-16 13:27:39 <ncopa> I am copying over the lxc to the other milkv machine to see if it makes any difference
2025-10-16 13:43:21 <ncopa> i continued the build on both machines and on nld-bld-1 it segfaulted again
2025-10-16 13:43:45 <ncopa> nld-bld-2 it is still running
2025-10-16 13:44:25 <ncopa> i think nld-bld-1 is flaky. not sure what the difference between them are?
2025-10-16 13:58:20 <ncopa> this is the diff: https://tpaste.us/e6wV
2025-10-16 13:58:37 <ncopa> i think we should consider replace the nvme storage
2025-10-16 14:23:10 <ncopa> clandmeter: I wonder if you could order a new storage for nld-bld-1 (172.16.30.2) maybe a samsung evo 990 pro 1TB something
2025-10-16 14:23:49 <ncopa> the samsung 970 evo on the nld-bld-2 appears to have been stable
2025-10-16 16:15:57 <ikke> ncopa: so it's the storage that causes the issue?
2025-10-16 16:56:11 <ncopa> ikke: i dont know. could also be insufficient cooling
2025-10-16 16:56:31 <ncopa> maybe i should have checked the temp sensors if there are any
2025-10-16 16:57:03 <ncopa> but that was the only thing I could see that differs on the hardware https://tpaste.us/e6wV
2025-10-16 16:58:25 <ikke> Both are below 50°
2025-10-17 16:01:30 <clandmeter> ncopa: i have it already
2025-10-17 16:01:37 <clandmeter> just didnt have tiem to copy it over
2025-10-17 16:01:43 <clandmeter> we got it sponsored last time
2025-10-17 16:03:13 <clandmeter> and yes i think its the ssd related
2025-10-19 09:16:48 <ikke> nu_: I'm checking one more time what happens when Zabbix tries to send an email to you
2025-10-19 16:21:59 <nu_> oki
2025-10-19 16:22:07 <nu_> i didnt get a mail:/
2025-10-19 22:03:02 <mio> could someone please check on the 3.23 x86_64, armv7 and aarch64 builders to see if the build is still progressing? they may be having a bit of trouble with the same package
2025-10-20 09:32:12 <achill> 🚪knock knock: ERROR: Job failed: failed to pull image "alpinelinux/gitlab-runner-helper:latest" with specified policies [always]: Error response from daemon: Head "https://registry-1.docker.io/v2/alpinelinux/gitlab-runner-helper/manifests/latest" <https://registry-1.docker.io/v2/alpinelinux/gitlab-runner-helper/manifests/latest">;: received unexpected HTTP status: 503 Service Unavailable (manager.go:250:0s)
2025-10-20 09:40:28 <ikke> achill: known AWS issue 
2025-10-20 09:40:37 <ikke> Seems to be recovering 
2025-10-20 09:40:59 <achill> ahhhhh i see 🙃
2025-10-20 11:33:43 <omni> putting all eggs in one basket as a service
2025-10-20 13:25:58 <ncopa> i have done go boostrap on build-3-23-x86
2025-10-20 13:26:14 <ncopa> doing go bootstrap on build-3-23-ppc64le now
2025-10-20 13:26:21 <ikke> 👍
2025-10-20 13:43:46 <ncopa> also build-3-23-loongarch64
2025-10-20 14:00:09 <ncopa> bootstrapping openjdk8 on build-3-23-armhf
2025-10-20 14:01:06 <ncopa> rm: can't remove '/home/buildozer/aports/community/go/tmp/go-tool-dist-2567746971/fips-v1.0.0-c2097c7c/golang.org/fips140@v1.0.0-c2097c7c/LICENSE': Permission denied
2025-10-20 14:01:14 <ncopa> im getting various of those on loongach64
2025-10-20 14:02:06 <ncopa> it fails to delete tmp/
2025-10-20 14:25:56 <ikke> Yeah, me too
2025-10-20 15:37:25 <ncopa> i have bootstrapped openjdk8 on build-3-23-aarch64
2025-10-20 15:37:31 <achill> thanks meow :3
2025-10-20 19:10:46 <ikke> nu_: I finally was able to grab the logs, I typoed your email addres
2025-10-20 19:22:05 <ikke> nu_: status=sent (250 2.0.0 Ok: queued as 4cr4yv66Fkz3n
2025-10-21 06:21:00 <ncopa> morning! im stopping builders due to trouble building ocaml
2025-10-21 06:28:57 <achill> mornya
2025-10-21 06:29:00 <achill> im still so tired aaaaaa
2025-10-21 08:47:38 <ncopa> ikke: nld-bld-4 appears to be stuck. I cant even kill -9. I think I'm gonna upgrade the kernel and reboot?
2025-10-21 08:48:05 <ncopa> build-3-23-x86_64 is stuck. cannot reboot it
2025-10-21 08:48:09 <ncopa> the container
2025-10-21 08:48:41 <ikke> ncopa: ok
2025-10-21 10:49:29 <ikke> There are multiple processes in disk sleep
2025-10-21 10:51:43 <ikke> rebooting
2025-10-21 11:07:37 <ncopa> hang on a sec
2025-10-21 11:07:59 <ncopa> lets wait for chromium to comlete the build
2025-10-21 11:08:15 <ncopa> so it is somethign with the disks?
2025-10-21 11:26:22 <ikke> not necessarily
2025-10-21 11:26:32 <ikke> But it's an uninterruptable sleep
2025-10-21 11:26:41 <ikke> dmesg does not show anything
2025-10-21 11:27:02 <ikke> in any case, no reboot command seems to work anyway
2025-10-21 11:33:15 <ikke> bootstrapping openjdk8 on armv7
2025-10-21 12:58:21 <nu_> ikke: monitoring received
2025-10-21 12:58:23 <nu_> thanks!
2025-10-21 13:01:45 <ikke> Good
2025-10-21 14:13:37 <ikke> Rebooted ^
2025-10-21 15:22:47 <mio> has gcc ice on stable riscv64 been resolved? getting an ice on u-boot rebuild on 3.20, should the upgrade linked to the rebuild be reverted for now?
2025-10-21 15:23:06 <ikke> 3.20?
2025-10-21 15:23:09 <mio> yeah
2025-10-21 15:23:29 <ikke> Have not heard of one there
2025-10-21 15:23:47 <mio> https://build.alpinelinux.org/buildlogs/build-3-20-riscv64/main/u-boot/u-boot-2024.04-r7.log
2025-10-21 15:25:05 <ikke> Possibly related to the host / disks
2025-10-21 15:25:29 <ikke> At least, I've heard ncopa mention that some error disappeared once he moved it to another host
2025-10-21 15:26:20 <mio> yeah, possibly related
2025-10-21 18:41:38 <ikke> bootstrapping go on riscv64
2025-10-22 03:45:37 <ikke> rebooting, https://tpaste.us/
2025-10-22 05:45:21 <ikke> running a zpool scrub ^
2025-10-22 06:34:56 <ikke> Hmm
2025-10-22 17:39:19 <ikke> rebooting the x86 builder host
2025-10-22 18:02:58 <ikke> ncopa: nld-bld-3 had some issues loading iptables, so I've upgraded it to 3.22 now, and everything is working again
2025-10-22 18:44:25 <ikke> upgrading and rebooting the x86_64 builder host
2025-10-23 06:22:41 <ncopa> nice! thank you!
2025-10-23 10:56:02 <ncopa> I wonder if we should create an email alias moderation@alpinelinux.org or similar for CoC topics
2025-10-23 11:21:36 <ncopa> send and receive emails. maybe change the help@a.o to moderation@a.o
2025-10-23 11:22:05 <ncopa> we should also try on-board moderators, who can help us deal with CoC incidents
2025-10-23 11:29:09 <ikke> Yes, we should
2025-10-23 15:17:43 <ikke> bootstrapping openjdk21 on aarch64
2025-10-23 19:01:28 <ikke> bootstrapping openjdk17 on aarch64
2025-10-24 06:31:35 <lotheac> something wrong with build-edge-x86_64? seems like it's stuck on testing/electron-38.4.0-r0, the MR for which was merged 21h ago !91957
2025-10-24 06:48:35 <ikke> You underestimate chromium 
2025-10-24 06:51:27 <lotheac> well, electron's gitlab CI build on x86_64 was finished in 237 minutes, so i would not really expect 21h :) although i don't really know when the electron build started on the builder
2025-10-24 06:53:49 <lotheac> (also none of the other arches are building that anymore)
2025-10-24 06:58:20 <achill> the builder was busy before building another chromium 🙃
2025-10-24 06:59:19 <lotheac> i suppose that would explain it
2025-10-25 13:07:27 <AnIDFB> if the standard isos can be built for riscv64 (https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/90444), what's preventing them from being added to dl-cdn?
2025-10-25 13:10:40 <ikke> AnIDFB: it would be done at the next release
2025-10-25 13:11:56 <AnIDFB> ikke: 3.23.0 or 3.22.3? (if the latter's even happening)
2025-10-25 13:13:36 <ikke> 3.23.0
2025-10-25 13:14:41 <AnIDFB> ikke: is this planned or just a maybe?
2025-10-25 13:15:40 <ikke> It's part of mkimg.standard, so it would be automatic
2025-10-25 13:16:52 <AnIDFB> ikke: alright, thanks for the info!