2026-02-02 18:57:37 <achill> edge-riscv64 is out of space apparently
2026-02-02 18:57:38 <achill> /home/buildozer/aports/community/qt6-qtbase/src/qtbase-everywhere-src-6.10.2/build/src/network/Network_autogen/include/moc_qdnslookup_p.cpp:124:1: fatal error: error writing to /tmp/ccojifpb.s: No space left on device
2026-02-02 18:59:40 <ikke> cleaning up
2026-02-02 18:59:47 <achill> merci beaucoup
2026-02-03 11:27:54 <achill> someone mind looking at why https://gitlab.alpinelinux.org/alpine/docker-abuild/-/jobs/2200297 fails?
2026-02-03 11:29:51 <ikke> The runner was not assigned to the project
2026-02-03 12:54:09 <Sertonix[m]> I finally had looked at openjdk on 32-bit again, this time with build instructions: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/96922
2026-02-03 12:56:47 <Sertonix[m]> Are the build instructions good enough for somebody to do the bootstrap for the builders?
2026-02-03 13:35:41 <ikke> Sertonix[m]: I'll have to check later. Please remind me if I forget
2026-02-03 21:37:28 <omni> I still think build-3-22-x86_64 is stuck, perhaps also build-3-22-x86
2026-02-04 13:53:41 <Sertonix[m]> Is build-3-22-x86_64 stuck?
2026-02-04 13:53:56 <Sertonix[m]> Ref #17938
2026-02-04 14:32:37 <ikke> There was a job stuck
2026-02-04 14:46:30 <Sertonix[m]> Thanks!
2026-02-04 14:48:09 <ikke> omni: thanks for reporting as well
2026-02-07 13:53:22 <Sertonix[m]> ikke: reminder for https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/96922
2026-02-09 11:03:36 <f_> oh wow riscv64 is slow
2026-02-09 11:03:43 <f_> er wrong chan
2026-02-09 22:48:50 <ncopa> seems like gitlab.a.o is down
2026-02-09 22:55:09 <ikke> checking
2026-02-09 22:55:15 <ikke> there appears to be load
2026-02-09 22:56:52 <ikke> processes that are not commonly showing high cpu
2026-02-09 23:02:02 <ikke> It's recovering
2026-02-09 23:06:04 <ikke> load is back to normal
2026-02-10 21:40:40 <dne> oops, linodeobjects.com is currently blocked by quad9
2026-02-11 06:45:29 <ikke> dne: is our container registry affected?
2026-02-11 07:05:04 <dne> ikke: it was, that’s how I noticed. looks like the block is gone now though 
2026-02-11 09:12:10 <ikke> jvvv: One thing I need to work on is making sure I'm not the only one who understands all the infra we have
2026-02-11 09:14:33 <jvvv> That makes a lot of sense. I spent some time last spring setting up a VM with Xen and k0s. I got it working, didn't do very much with it. At that point I spent about two weeks trying to get my head around terraform.
2026-02-11 09:16:30 <jvvv> I've been trying to get better with yaml since then, but I drifted away from it because I was feeling defeated. I think I should get back to learning more about that kind of thing>
2026-02-11 09:17:06 <ikke> I'm happy to explain certain things when I have time
2026-02-11 09:18:19 <lotheac> yaml is just markup... and not a particularly great markup language, but "it is what it is"
2026-02-11 09:18:55 <lotheac> kyaml is supposed to make it a bit better i guess but i don't think the tooling is yet there
2026-02-11 09:19:36 <ikke> For me it's marginally better
2026-02-11 09:20:20 <lotheac> i don't think there's a huge difference between kyaml and religiously running yamlfmt at save time in your editor
2026-02-11 09:20:21 <ikke> json with slightly less double quotes and comments
2026-02-11 09:20:45 <lotheac> now if yamlfmt did kyaml, that'd be nice
2026-02-11 09:20:59 <lotheac> oh actually it does
2026-02-11 09:21:12 <lotheac> guess i need to migrate
2026-02-11 09:21:39 <jvvv> One of the things that made terraform seem out of reach was that some much of it seems proprietary... I'm not a zealot, it just made it seem out of reach to try some of the things I was trying to learn about.
2026-02-11 09:21:51 <lotheac> opentofu is a thing
2026-02-11 09:21:53 <ikke> yup
2026-02-11 09:22:11 <ikke> For Alpine I've switched to opentofu
2026-02-11 09:23:04 <jvvv> yes, I discovered that. I will dig back into it again starting today. Will be glad I have a place I can ask questions.
2026-02-11 09:23:28 <lotheac> i have some philosophical qualms with terraform though. it's not a good tool at almost any reasonable scale, because config drift and statefiles become huge issues
2026-02-11 09:23:44 <lotheac> but it's not exactly easy to replace
2026-02-11 09:25:09 <ikke> config drift can be handled by periodically running terraform apply
2026-02-11 09:25:14 <ikke> (scheduled)
2026-02-11 09:26:44 <lotheac> the devil's always in the details though
2026-02-11 09:26:48 <lotheac> are we doing that in alpine?
2026-02-11 09:27:04 <ikke> No, not at the moment
2026-02-11 12:57:06 <omni> is build-edge-riscv64 stuck?
2026-02-11 14:08:16 <durrendal> lotheac: any recommendations for things that scale better than terraform? I haven't seen any major issues personally, so I'm curious about your personal experience and where you start to see problems.
2026-02-11 14:09:03 <lotheac> when i say "scale" i mostly mean team size
2026-02-11 14:09:10 <lotheac> terraform gets annoying at n=2
2026-02-11 14:09:45 <lotheac> but no, like i said, it's not easy to replace. i don't have real alternatives... maybe crossplane for some stuff but it's not a good fit for everything
2026-02-11 14:16:36 <lotheac> in larger organizations it seems to happen quite often that there are a gajillion different terraform root modules maintained by separate teams across the engineering org, but there is no machinery that would rerun stuff to run apply. so the terraform code bitrots, and in many cases is not even rerunnable as-is because the modules are not version-constrained properly. at some point someone will install a tool like Atlantis to attempt to combat issu
2026-02-11 14:16:36 <lotheac> es with statefiles being out of sync with the git repo, and then everything becomes extra annoying because you can no longer even do "terraform plan" without creating a pull request and asking the bot to do it for you in a comment
2026-02-11 14:17:33 <lotheac> and then when you do that with your 2-line change, you get a 300 line diff from plan because it's drifted so badly and you just wanted to add a stupid vpc or something
2026-02-11 14:19:37 <ikke> We run terraform purely in CI/CD
2026-02-11 14:19:45 <ikke> No manually terraform apply
2026-02-11 14:19:54 <lotheac> my point is that the concept of "state file" which is maintained separately from the code (usually in a shared object storage bucket or something) plus the "anyone can just apply their changes from their local machine" don't really work together even if you just have one tf root module
2026-02-11 14:20:33 <lotheac> ikke: that does help, but is difficult to enforce if you want to allow *plans* for your code locally
2026-02-11 14:21:00 <ikke> Yes, it does in that case
2026-02-11 14:21:09 <lotheac> and if you don't allow local plans, then developing tf code becomes quite painful
2026-02-11 14:21:57 <lotheac> it's a problem that the "plan" operation requires essentially the privs as applying does in most cases
2026-02-11 14:21:59 <ikke> It depends on what you manage with terraform. At $work we're managing an application, and it's easy to create a local instance
2026-02-11 14:22:15 <ikke> So you test against a local instance, including being able to run and apply plans
2026-02-11 14:22:26 <ikke> But for infrastructure that's more difficult
2026-02-11 14:22:43 <lotheac> usually in orgs that i've seen it's "we manage all of our infra, production and other envs, as IaC"
2026-02-11 14:22:53 <lotheac> and then it becomes hard
2026-02-11 14:27:05 <ikke> But I suppose that counts for any simimlar tool. In some cases, disabling refresh should allow terraform plan to run without having access to the infrastructure
2026-02-11 14:27:40 <lotheac> at $client we don't have terraform at all... right now. but it's a very small organization. a few cloud resources are managed via crossplane, applications/deployments mostly via kube (custom resources/operator for some internal stuff)... and then there's a bunch of ansible for the stuff that's been there forever
2026-02-11 14:27:43 <ikke> But giving access to the statefile should also be considered, it can easily contain secrets
2026-02-11 14:34:27 <lotheac> yeah. i feel like a good tool would have a sort of "happy path" and make it easy to do the right thing/follow best practices
2026-02-11 14:34:47 <lotheac> with terraform it feels like you have to do a ton of things, none of which are obvious, to avoid footguns and still it's kinda iffy
2026-02-11 17:00:02 <durrendal> lotheac: ah that makes sense, I've only used it by myself, with maybe the occasional contributor. In that context having the state file in S3 was enough, but it was infrequent and minor contributions
2026-02-11 20:07:01 <iggy> I don't hate the state opentofu/terraform is in... it's (mostly) simple enough to get started without too much overhead in the "there's 1-2 of us doing the IaC" case, but there's a lot of other tooling (Spacelift, env0, terramate, terragrunt, Atlantis, etc),  you can add around it when you start to scale... I just wish it scaled a little better so you didn't _need_ all that
2026-02-11 20:07:03 <iggy> other tooling
2026-02-11 20:08:16 <iggy> I think the scaling would be better at this point if hashi hadn't been trying to push terraform cloud so hard for so many years... it wasn't really until after opentofu forked that they even considered that people might want to not use tfc
2026-02-14 08:48:51 <ikke> lotheac, nu_, durrendal: An issue in case you wanted to work on something https://gitlab.alpinelinux.org/alpine/infra/docker/exec/-/issues/1
2026-02-14 08:49:02 <ikke> I'm trying to create more issues like this
2026-02-14 08:50:43 <lotheac> i have a working solution at work (on self-hosted github actions runners, which run on kube) but they need to be privileged
2026-02-14 08:51:49 <lotheac> i thought we discussed this the other week
2026-02-14 08:52:30 <lotheac> "exec unit" means a gitlab CI job?
2026-02-14 08:53:23 <ikke> Task oriented docker iamge
2026-02-14 08:53:25 <ikke> image
2026-02-14 08:56:50 <lotheac> alright, i can probably take a stab sometime next week
2026-02-14 08:57:49 <lotheac> the docker-based builder we have (there's also a podman-based one) uses docker:dind in a second container in the same pod
2026-02-14 08:58:09 <lotheac> and the first container talks to that dockerd to build
2026-02-14 08:58:33 <lotheac> might be the easiest way to be compatible
2026-02-14 08:59:09 <ikke> Doesn't dind also require privileges?
2026-02-14 08:59:52 <lotheac> yes, it requires a privileged container but i think there's not really a way around that
2026-02-14 09:00:20 <lotheac> podman/buildah require privileges too for uid namespace stuff
2026-02-14 09:00:51 <lotheac> arguably a privileged container is still better than exposing host docker socket though
2026-02-14 09:03:08 <lotheac> that said i should revisit that cern blog post (and other sources) to recheck if there isn't a way to build without privs
2026-02-14 09:04:32 <lotheac> i think last time we came to the conclusion that .spec.hostUsers=false was not usable on k0s/containerd yet?
2026-02-14 09:05:07 <lotheac> i'll check later. gotta run now
2026-02-14 09:05:21 <ikke> o/
2026-02-14 09:05:28 <lotheac> o/
2026-02-14 09:05:48 <ikke> lotheac: yeah, the PR to upgrade containerd is still being worked on
2026-02-14 09:06:17 <ikke> Apparently it regularly needs to be rebased due to conflicts
2026-02-14 09:06:27 <lotheac> right, then from what i remember we're gonna need privileged until it is
2026-02-14 09:06:44 <ikke> https://github.com/k0sproject/k0s/pull/7011
2026-02-14 09:07:12 <lotheac> i guess i gotta spin up a test k0s cluster again
2026-02-16 11:54:28 <achill> can someone kick edge-armhf, edge-armv7 & edge-s390x? they are deadlocked in the tests of perl-net-curl-promiser
2026-02-16 11:55:27 <achill> also edge-x86_64 seems to run for a long while
2026-02-16 11:56:53 <ikke> Done with the first request
2026-02-16 17:24:37 <achill> i think you can also kick the x86_64 one, the package looks horribly broken currently
2026-02-16 18:17:59 <ikke> done
2026-02-16 20:10:33 <achill> thanks!
2026-02-17 23:54:43 <omni> would this be ok? https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/97658/diffs?commit_id=1d7e458082563ff81f246ce50e08085d19d12557
2026-02-18 05:17:23 <ikke> omni: yes
2026-02-18 13:37:44 <omni> thanks
2026-02-18 13:50:55 <omni> ikke: also backporting to 3.23-stable, that differ a bit, like this? https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/97697/diffs?commit_id=0a01c1f08085305b0b2024c1d16473707ee327d5
2026-02-18 14:03:17 <omni> the arm builders still have issues with IPv6?
2026-02-18 14:04:30 <omni> or perhaps it's an upstream issue and that the builders don't fall back to IPv4?
2026-02-18 14:11:51 <ikke> omni: yeah, the arm servers are primarily ipv6, do have ipv4, and bb wget does not fall back to ipv4 
2026-02-18 14:11:52 <omni> looks like an upstream issue with https://download.libguestfs.org
2026-02-18 14:12:51 <ikke> Arm servers are the only servers where even the build containers have public ipv6 addresses 
2026-02-18 14:13:17 <omni> ok
2026-02-18 14:13:38 <omni> could they get distfiles/edge/libnbd-1.24.1.tar.gz to make the builds pass then?
2026-02-18 14:15:43 <omni> there was a #libguestfs@libera so I mentioned it there, not sure how active that channel is though
2026-02-18 14:28:36 <ikke> omni: it would be helpful to backport all missing commits to 3.23 regarding ci, so that backporting future changes is easier
2026-02-18 14:51:30 <omni> ikke: ok, I'll do that, backport the individual commits instead of adjusting as I did in the MR above, after the CI builds pass
2026-02-19 12:24:21 <dne> it looks like build-edge-s390x is stuck
2026-02-19 13:36:09 <ikke> dne: I've killed it
2026-02-20 21:40:28 <Sertonix[m]> Are the coreutils builds stuck?
2026-02-20 21:51:11 <ikke> no, build services crashed
2026-02-20 21:53:03 <ikke> /usr/bin/mqtt-exec, pid 349, terminated by signal 1
2026-02-20 21:53:21 <ikke> Not sure what's sending SIGHUP
2026-02-20 21:54:44 <ikke> maybe caused by coreutils test suite?
2026-02-21 11:56:47 <ncopa> somethign is weird with coreutils testsuite. the tests appears to fail and the abuild rebuild it again
2026-02-21 12:57:34 <ikke> ncopa: I don't think abuild is involved
2026-02-21 12:57:54 <ikke> Running `abuild -v check` does not indicate it calles make the 2nd time
2026-02-21 13:05:02 <ikke> It seems to first run the coreutils tests, and later the gnulib tests
2026-02-21 16:21:49 <ikke> ..
2026-02-21 16:22:24 <ikke> packetloss from zabbix
2026-02-21 16:24:04 <ikke> Seems like routing issues from gbr to nld
2026-02-22 01:25:44 <omni> are arm and x86 builders stuck on coreutils?
2026-02-22 13:11:37 <achill> omni: the testsuite killed mqtt-exec or something
2026-02-22 13:29:54 <ikke> I'll try the patch from jvvv and see if it fixes things
2026-02-22 19:30:52 <ikke> The builders are back
2026-02-22 21:12:53 <dne> build-3-23-riscv64 appears to be stuck
2026-02-22 21:13:30 <ikke> the builder is gone (unstable hw)
2026-02-22 21:13:38 <ikke> need to wait for it to be restarted
2026-02-22 21:14:16 <dne> ah ok
2026-02-24 06:41:10 <ikke> storage on nld-bld-2 seems very erratic at the moment
2026-02-24 06:41:12 <ikke> 2.43 k requests completed in 31.4 s, 9.49 MiB read, 77 iops, 309.2 KiB/s
2026-02-24 06:41:22 <ikke> min/avg/max/mdev = 79.7 us / 12.9 ms / 31.1 s / 631.6 ms
2026-02-26 09:46:42 <DWwanghao> Has the CI directory layout been changed? It seems the build can’t find the artifacts directory now
2026-02-26 10:16:49 <ikke> Example? Nothing has changed
2026-02-26 10:38:20 <achill> that was likely also due to e48a1565
2026-02-26 10:38:25 <achill> just rebase and see if it'll fix it
2026-02-26 11:39:55 <DWwanghao> achill: After rebasing e48a1565, it can indeed be fixed, thank you
2026-02-26 11:42:50 <DWwanghao> ikke: The failed pipeline for 98077 and 98076 are still there
2026-02-26 11:43:10 <DWwanghao> !98077
2026-02-26 12:36:19 <ikke> DWwanghao: those are just warnings, and don't cause the pipelines to fail
2026-02-26 12:37:06 <ikke> like achill said, the cause is the syntax error in the community repo that existed, which results in ap builddirs returning nothing, and CI not to build any packages
2026-02-26 12:47:57 <DWwanghao> Got it, thanks for the explanation
2026-02-27 02:36:17 <Sertonix[m]> Is it intentional that unprivilidged user namespaces are available on the x86_64 CI?
2026-02-27 05:37:15 <ikke> Sertonix[m]: it's not something we have explicitly enabled or anything.
2026-02-27 06:55:04 <huajingyun> dolphin is currently failing tests on both loongarch64 and riscv64 builder. PureTryOut and achill submitted fixes to the apkbuild file yesterday (arches other than loongarch64 and riscv64 were not triggered for rebuild)
2026-02-27 06:55:11 <huajingyun> however, according to the gitlab ci results, all arches actually failed the test: https://gitlab.alpinelinux.org/huajingyun01/aports/-/pipelines/409927
2026-02-27 11:36:44 <ikke> It's recovering already
2026-02-27 11:37:03 <ikke> although load is still quite high
2026-02-27 11:44:16 <ikke> Something is scraping many repos again using git
2026-02-27 11:48:23 <jvvv> The LLM's have found a way around the counter measures?
2026-02-27 11:52:54 <ikke> It's harder to distinguish between legitimate traffic and scrapers
2026-02-27 11:55:35 <jvvv> I see a time coming when opensource orgs are going to have to switch to using git-ssh more out of necessity.
2026-02-27 12:47:20 <qaqland> is gitlab down?
2026-02-27 17:36:12 <ikke> oh no
2026-02-28 12:07:44 <lotheac> riscv64 CI might be having some issues? I got "fatal error: unlock of unlocked lock" on a go build on fra-ci-1 and "SIGILL: illegal instruction" on the next three reruns (at different points in the build, with different stacktraces), all three on nor-ci-1
2026-02-28 12:08:05 <lotheac> https://gitlab.alpinelinux.org/alpine/aports/-/pipelines/410222/builds
2026-02-28 12:24:18 <lotheac> hm, maybe not actually infra problem after all... 3/4 SIGILLs happened in internal/bytealg/indexbyte_riscv64.s:82, and go 1.26.0 has https://github.com/golang/go/commit/3406a617d9643902e932ca99c6f276fa7b19e030 
2026-02-28 12:25:03 <lotheac> which maybe means that go1.26 for riscv64 was compiled with vector support detected but the CI doesn't?
2026-02-28 12:34:01 <ikke> lotheac: we have various runners for rv64 CI
2026-02-28 12:34:31 <lotheac> right, this makes me think that go1.26 was compiled in a way that is not supported by all the runners
2026-02-28 12:34:46 <lotheac> doing some research on that now
2026-02-28 12:41:35 <lotheac> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/98166 this might help but i don't have a way to verify it
2026-02-28 12:41:46 <lotheac> outside of merging and seeing if it does :p 
2026-02-28 13:02:57 <lotheac> ... well, bit of a problem there since it's using the previously built go version on the CI runner to build itself now and fails to do that because of the same issue
2026-02-28 13:03:09 <lotheac> achill: any good ideas?
2026-02-28 13:04:04 <achill> i guess we can merge it
2026-02-28 13:04:10 <achill> and see what happens
2026-02-28 13:04:22 <achill> i dont have any rv64 hardware that i could use for thta
2026-02-28 13:04:27 <lotheac> yeah me neither
2026-02-28 13:04:43 <lotheac> ok, let's merge then. yolo : )
2026-02-28 13:16:15 <lotheac> looks like it built on builders at least
2026-02-28 13:25:18 <lotheac> but it didn't seem to help. https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2238201
2026-02-28 13:25:55 <lotheac> ikke: would it be possible to get access to a rv64 container to investigate this?
2026-02-28 13:26:31 <lotheac> preferably on hardware similar to (siLj_NXx) shared-runner nor-ci-1 (riscv64)
2026-02-28 13:27:32 <lotheac> although i guess i don't really have time today
2026-02-28 13:58:13 <lotheac> don't really know what's going on there... reading the go code rva20u64 is already the default, and in arm_riscv64.h hasV is only defined #ifdef GORISCV64_rva23u64
2026-02-28 14:00:12 <lotheac> ah hmm, but it actually only branches to the non-vector impl if internal∕cpu·RISCV64+const_offsetRISCV64HasV(SB) returned zero
2026-02-28 14:01:47 <lotheac> s/returned/is//