2026-03-01 00:48:14 <lotheac> i added some debugging code to my pipeline... on the (siLj_NXx) shared-runner nor-ci-1, which is crashing with SIGILL on a vector instruction, cpuinfo does claim vector support: model name: Spacemit(R) X60 isa: rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt
2026-03-01 00:50:35 <lotheac> and, even if go is compiled with GORISCV64=rva20u64 (which is also the default), it will check at runtime whether the CPU supports RVV 1.0 (vector extensions). and it seems this CPU does, but regardless crashes with SIGILL at internal/bytealg/indexbyte_riscv64.s:83
2026-03-01 00:54:21 <lotheac> i am thinking, maybe we patch out the runtime check and always use the non-vector impls (which is what go1.25 does) and report this to the go people to let them sort it out? i'm not familiar with riscv64 cpu extensions
2026-03-01 06:54:50 <lotheac> the weird thing here is that the commits which implement the vector support in go specifically mention Spacemit X60, which is ostensibly the same CPU as is failing here
2026-03-01 07:29:33 <lotheac> ikke: you around? i could really use a rv64 shell for this instead of trying to hope my CI jobs get scheduled on the correct box :p
2026-03-01 07:32:22 <ikke> Only on milk-v pioneer hw. nor-ci-1 is different
2026-03-01 07:32:53 <lotheac> that doesn't help unfortunately, shared-runner pioneer1 cpu does not support the vector insns and thus does not exhibit the issue
2026-03-01 07:34:56 <lotheac> is nor-ci-1 ncopa's perchance?
2026-03-01 07:37:13 <ikke> Yes
2026-03-01 07:38:02 <lotheac> so i guess i need to talk to him unless i get lucky with CI scheduling
2026-03-01 07:38:46 <lotheac> ncopa: would you mind giving me a temp shell on nor-ci-1 for debugging a vector instruction issue on that specific cpu?
2026-03-01 22:45:11 <aelin> the s390x runner is idle... but has not picked up any of the spirv rebuilds yet. maybe it's broken?
2026-03-03 10:23:26 <aelin> gentle poke, what's the state of the s390x builder?
2026-03-03 10:26:29 <ikke> Will check later.
2026-03-03 10:48:23 <ncopa> something happened with the s390x builder
2026-03-03 10:49:18 <ncopa> it rebooted 3 days ago and it appears that /var was not mounted, so no lxc
2026-03-03 10:52:00 <ncopa> vgdisplay shows scary messages
2026-03-03 10:52:06 <ncopa> vg1 is gone apparently
2026-03-03 10:53:39 <ncopa>   WARNING: VG vg1 is missing PV fKw74P-69tj-NEHO-jWLi-KTCg-ycFF-mfykG6 (last written to /dev/dasdg1).
2026-03-03 10:53:39 <ncopa>   WARNING: VG vg1 is missing PV jHErDq-RYC7-w9u1-iaQo-lCfR-P4mP-WLCra3 (last written to /dev/dasdh1)
2026-03-03 11:17:17 <ncopa> i think this effectively means that we lost the data
2026-03-03 11:23:35 <ncopa> I think I will reboot it again and see if the disks comes back
2026-03-03 11:24:02 <ncopa> usa2-dev1 [~]# uptime
2026-03-03 11:24:02 <ncopa>  11:23:54 up 3 days, 18:32,  0 users,  load average: 0.00, 0.00, 0.00
2026-03-03 11:25:00 <ncopa> nope
2026-03-03 11:25:50 <ikke> :(
2026-03-03 11:26:26 <ncopa> i think we lost  /dev/dasdg1 and /dev/dasdh1 for some reason
2026-03-03 11:26:52 <ncopa> I dont know who we should ask
2026-03-03 11:27:59 <ikke> Martha
2026-03-03 11:28:35 <ncopa> I think it was rebooted Friday 27 February and after that we lost those two disks
2026-03-03 11:34:15 <ikke> Sent an email
2026-03-03 11:38:08 <ncopa> Thanks!
2026-03-03 16:06:31 <ncopa> lotheac: Sorry I can give you a login to nor-ci-1, but not right now. Please ping me tomorrow
2026-03-03 16:44:34 <ikke> ncopa: ping
2026-03-03 16:45:28 <ikke> I got reaction from Marist. They services and rebooted the machine. They expect the new disk configuration was not made persistent. I found that the disks are listed in /etc/zipl.conf, and added the suggested disk numbers there
2026-03-03 16:48:52 <ikke> A ran `zipl`. Rebooting now and see if it comes back
2026-03-03 16:51:50 <ikke> ok, it's back, including the volumes and containers.
2026-03-04 06:14:15 <ikke> aelin: thanks for reporting btw
2026-03-04 06:14:53 <lotheac> ncopa: good morning. ping re: nor-ci-1
2026-03-04 07:35:52 <ncopa> lotheac: good morning! do you have access to the dmvpn network? can you route to 172.21.1.3?
2026-03-04 07:43:07 <lotheac> i do not
2026-03-04 07:43:47 <ncopa> ok. let me try set up some portforward after i eat breakfast
2026-03-04 07:43:51 <lotheac> cheers
2026-03-04 16:35:40 <ikke> ^ is expected
2026-03-05 11:51:41 <ikke> Tweaked this trigger to wait slightly longer
2026-03-06 06:01:48 <huajingyun> jujutsu build has been stuck on build-edge-riscv64 for at least 6 hours, or even longer
2026-03-06 06:03:39 <huajingyun> ikke: do you have time to take a look? thanks
2026-03-06 06:09:41 <ikke> huajingyun: I've killed the build
2026-03-06 06:16:29 <huajingyun> ikke: Okay, thank you. hope it will pass this time
2026-03-08 23:09:12 <omni> looks like build-edge-riscv64 is stuck on jujutsu again
2026-03-09 01:27:17 <huajingyun> omni: yeah, the second build also got stuck. mio has already disabled checks for riscv64, so it might still require ikke to manually kill the build that has entered the 3rd build of jujutsu
2026-03-09 06:50:23 <ikke> huajingyun: killed it again
2026-03-09 06:52:18 <huajingyun> ikke: thanks a lot
2026-03-11 03:17:02 <durrendal> ikke: I was curious about the package mirror disc usage, put together a little analysis on it here: https://lambdacreate.com/paste/alpine-mirror-analysis.txt
2026-03-11 03:18:18 <durrendal> It looks like we're using ~5tb of space based on what I can see from rsync, not sure if that's the absolute total. Regardless I was kind of surprised how much space all the EOL releases take up and how much the distro has grown
2026-03-11 03:18:46 <durrendal> edge is like 40x the size of 3.3 which is wild.
2026-03-11 03:19:28 <invoked> good stuff man!
2026-03-11 03:19:38 <durrendal> also the addition of disc images for AWS, Azure, GCP is non-trivial
2026-03-11 03:20:43 <durrendal> invoked: thank you! lots of discussion going on around this stuff lately, but it's hard to actually discuss it without data
2026-03-11 03:21:26 <invoked> very helpful, thank you
2026-03-11 03:21:58 <durrendal> anyways, I think long term maybe the answer is we reach out to archive.org and ask if they can provide an archive.alpinelinux.org mirror for us like they do for Debian, Fedora, etc. That'd probably need TSC or Council involvement I guess, someone to represent the distro and broker the connection.
2026-03-11 03:22:13 <durrendal> or well, maybe not THE answer, but a possible solution. 
2026-03-11 08:00:46 <clandmeter> durrendal: nice
2026-03-11 08:01:17 <clandmeter> i think thats a good idea reg archive.org
2026-03-11 08:02:24 <clandmeter> would you mind checking this out and let us know what to do? 
2026-03-11 09:29:11 <ikke> I did look at archive.org before
2026-03-11 09:31:06 <ikke> It would probably take an announcement from us that we are deleting those files before they take it u 
2026-03-11 09:31:08 <ikke> Up
2026-03-11 09:32:21 <ikke> Archlinux does have some arrangement though
2026-03-11 12:49:45 <durrendal> clandmeter: yeah I don't mind at all, our best bet is to probably get it on Jason Scott's radar, he's the curator for those archives. I'll see if I can find an email for him and then reach out. 
2026-03-11 12:50:21 <durrendal> Any issues if I cc you both on that?
2026-03-11 12:52:52 <durrendal> my sense is it probably will require an announcement at some point, but they do this for most of the larger distros so I don't think we'd be asking too much
2026-03-11 13:00:48 <clandmeter> durrendal: go ahead, we appricate the support.
2026-03-11 13:10:07 <ikke> No problem for me
2026-03-11 14:01:58 <durrendal> awesome :) I just sent, used both of your @alpinelinux.org emails for the CC, felt more official that way
2026-03-11 14:02:12 <durrendal> fingers crossed we get a good response!
2026-03-11 15:26:40 <ikke> Durrendal: Thanks, though 3.14-3.19 are not EOL yet. The official designation is "on request"
2026-03-11 15:36:45 <durrendal> Drat, I forgot about that distinction. Still if they can take everything from 3.0 to 3.13 that still gives us some breathing room and a transition plan
2026-03-11 15:37:30 <durrendal> how much space do we have total? Is it enough to just trim space consistently?
2026-03-11 15:39:40 <ikke> With each release increasing in size, it would  not
2026-03-11 15:40:14 <ikke> How does archlinux handle redirects in case of 3rd-party mirrors
2026-03-11 15:40:41 <ikke> durrendal: we have several components with differents amount of storage
2026-03-11 15:58:27 <durrendal> that was my sense, and the real space saving would include the larger more recent releases.
2026-03-11 15:58:31 <durrendal> https://wiki.archlinux.org/title/Arch_Linux_Archive
2026-03-11 15:59:01 <durrendal> My understanding of how Arch does it is the historical archive has a redirect, so for a user perspective it's transparent just a little slower
2026-03-11 15:59:31 <ikke> They are a rolling release, so less comparible
2026-03-11 16:00:29 <durrendal> yes unfortunately I haven't found any references to how it works for Debian etc but I believe they have this sort of arrangement and are closer to our use case
2026-03-11 16:10:31 <ikke> For our CDN, we could probably implement the redirects, but the 3rd-party mirrors won't implement that and the files would get removed
2026-03-11 16:11:33 <ikke> Another thing worth looking at is retention for cloud images
2026-03-11 16:12:28 <ikke> like you pointed out, they also take up quite some space
2026-03-11 16:23:30 <invoked> naive question: is changing the builders to share noarch content (as Sertonix suggested) a complex change?
2026-03-11 16:25:20 <ikke> invoked: one builder would need to be the provider for these packages
2026-03-11 16:25:31 <ikke> which means that builder becomes a depedency for other builders
2026-03-11 16:26:14 <ikke> Harder to get consistent builds
2026-03-11 16:27:11 <invoked> hm. on the other hand that seems like it could be the most impactful thing we could do
2026-03-11 16:27:19 <ikke> if builder B provides the noarch packages, and builder A finishes before builder B, which finished before builder C, it could be that builder A was built against version N of a package, while builder C builds against version N+1
2026-03-11 16:27:55 <invoked> ah, so it's racy
2026-03-11 16:28:17 <ikke> Yes
2026-03-11 16:28:53 <ikke> The current consistency is achieved by having a single builder which has a full view and sole custody over it's repositories
2026-03-11 16:29:20 <ikke> Distributing that, and you get all the fun and joy from distributed systems
2026-03-11 16:29:38 <invoked> :(
2026-03-11 16:29:51 <invoked> thanks for the explanation
2026-03-11 16:32:10 <ikke> Changing the build architecture would need to find a solution for this problem
2026-03-11 16:32:20 <ikke> (or see how others have fixed it)
2026-03-11 16:33:21 <jvvv> Wow, I would not have foreseen that builder race condition, but it makes sense to me now that I have read that a few times.
2026-03-11 16:36:59 <ikke> These problems are not impossible to solve, just requires a well-thought out architecture
2026-03-11 16:37:24 <ikke> ("
2026-03-11 16:37:32 <ikke> ("just" carrying a lot of weight here :P)
2026-03-11 16:39:01 <Saijin_Naib[m]> Was gonna say, so you mean it is impossible to solve, haha
2026-03-11 16:41:28 <ikke> The power of the current architecture, with all its limitations, is its simplicity
2026-03-11 16:41:54 <ikke> Changing that would probably introduce more points of failure / instability
2026-03-11 17:28:27 <Sertonix[m]> ikke: My idea for noarch would be to do that post-builds: indexing a directory, checking for duplicate package ids, repack file with combined signatures, sym-/hardlink file.
2026-03-11 17:30:35 <Sertonix[m]> The checking of package id/hash is important since there are many noarch  packages that are not the same across arches
2026-03-11 17:32:48 <ikke> If I understand it correctly, that would only save space on the mirrors, right? Each builder would still have all the packages (including noarch)
2026-03-11 17:35:38 <ikke> (not dismissing the idea, just trying to understand it)
2026-03-11 17:36:14 <Sertonix[m]> If packages have to be local, yes
2026-03-11 17:37:29 <ikke> Right now, they have to be
2026-03-11 18:20:04 <durrendal> ikke: so it's not enough to address the storage problem on the mirrors, it must also be addressed on the builders due to their current design?
2026-03-11 18:20:23 <ikke> durrendal: More or less, yes
2026-03-11 18:20:49 <ikke> Though, technically we could prune more on the builders
2026-03-11 18:32:40 <durrendal> hmm a bit of a rock and a hard place. It's one thing to argue funneling money into expanding mirror storage, another if you also have to scale every other system that is related to packaging and release.
2026-03-11 18:33:11 <ikke> Even mirror storage involves multiple hosts due to geolocation
2026-03-11 18:34:06 <durrendal> I assumed as much from your previous comment. And throwing money at a problem is a bandaid fix, not a long term solution.
2026-03-11 18:35:14 <durrendal> can we release 3.24 as is without addressing the storage problem? 3.25? trying to get a sense for how much runway we have
2026-03-11 18:36:09 <durrendal> also, I wasn't around, I assumed we purged 0.x through 2.0 since we only have 3.x on the mirrors?
2026-03-11 18:37:37 <ikke> Yes, I recall we used to have an old.a.o, but it got dropped somewhere among several infra changes
2026-03-11 18:43:49 <durrendal> haha well 3.24 could be 4.0, cut it and start over.
2026-03-11 18:43:57 <durrendal> jk of course
2026-03-11 18:44:44 <ikke> In any case, how it's now, 3.24 would not fit
2026-03-11 18:44:48 <ikke> on the master mirror
2026-03-11 18:44:57 <ikke> Note that they have always provided us more space when we asked for it
2026-03-11 18:45:33 <ikke> But, I don't know how much more we expect their generosity to extend
2026-03-11 18:50:18 <durrendal> That's a sobering statement, and explains where the discussions around package acceptance policy have been coming from. If we grew less quickly, we'd have more runway.
2026-03-11 18:52:48 <durrendal> Is the policy on which versions get maintenance by request set in stone? If we dropped 3.14 - 3.18 for example that's 1196.2GB of space, leaving 3.19 as still at request?
2026-03-11 18:53:04 <durrendal> that last ? should be a .
2026-03-11 18:54:22 <ikke> It's also that those releases are more likely to still have active users, and dropping them has a much larger impact
2026-03-11 18:55:28 <ikke> You'd underestimate how many docker images / pipelines there are that do `FROM alpine:<old-version>; RUN apk add ....`
2026-03-11 18:56:58 <durrendal> that's a valid argument, we don't want to break user space. But if we publicly announced a policy change, in tandem migrated what we could to archive (if they'll help), and requested additional space on the master mirror then perhaps that change in policy would be less impactful
2026-03-11 18:57:16 <durrendal> I'm assuming a lot, we can't affect the technical decisions of downstream consumers.
2026-03-11 18:57:20 <ikke> yes, but that would probably long-term, not short-term
2026-03-11 18:58:21 <durrendal> agreed, and if the only thing we get is a short term fix we'll be talking about not being able to release 3.25+ or similar, which is what I worry will happen.
2026-03-11 18:59:25 <durrendal> there's likely not silver bullet or magic solution, but if each snapshot of edge is going to take up 0.6TB+ going forward then policy has to change somewhere. 
2026-03-11 19:00:18 <durrendal> I guess what's unclear to me, and probably other contributors, what is the greater good? preserving the old version history as semi-warm, or mass pruning packages from the repos to refocus the distro?
2026-03-11 19:00:49 <durrendal> or something else, I haven't sat with the problem long enough to definitively state these are our options/
2026-03-11 19:05:20 <achill> is the storage the one that is expensive, or traffic? storage is dirt cheap itself, e.g. you get 1tb for 3.2 euro/month, or 20tb for ~40 euros/month. traffic on the other hand is massively expensive especially with our scale
2026-03-11 19:05:35 <achill> (that is on hetzner as a example)
2026-03-11 19:05:41 <ikke> For us, it's the opposite
2026-03-11 19:05:46 <ikke> We got plenty of traffic
2026-03-11 19:05:50 <ikke> But storage is limited
2026-03-11 19:06:51 <achill> interesting, i wouldve expected not such a tight storage limitation. really makes me wonder
2026-03-11 19:06:57 <ikke> And for the record, at the moment, everything is sponsored
2026-03-11 19:07:04 <achill> yeah
2026-03-11 19:07:13 <ikke> achill: what we get is mostly bare-metal
2026-03-11 19:07:44 <ikke> Not VMs with network-backed storage that can infinitely expand
2026-03-11 19:08:07 <achill> hm
2026-03-11 19:08:44 <durrendal> but if it's bare metal could we funnel some of the donations into additional storage then?
2026-03-11 19:08:49 <achill> but also a few tbs hdd storage are not that expensive
2026-03-11 19:08:51 <ikke> durrendal: possibly
2026-03-11 19:09:04 <ikke> achill: we need more than a few TB
2026-03-11 19:09:05 <achill> durrendal: that would be good yea
2026-03-11 19:09:23 <durrendal> I'd be happy to know that my donations bought storage that allowed us to release 3.24 for what it's worth. I imagine many would feel the same
2026-03-11 19:09:41 <achill> the donations are made for exactly this purpose in mind afaik
2026-03-11 19:09:47 <ikke> Yes
2026-03-11 19:09:57 <achill> but i dont know how to funnel this money into the actual storage servers
2026-03-11 19:10:26 <ikke> ack, that's the tricky situation. We would at least need to discuss that with the sponsors
2026-03-11 19:10:35 <achill> yeah yeah makes sense
2026-03-11 19:10:43 <achill> thank you for being on top of this stuff! :3
2026-03-11 19:10:58 <achill> this is really great
2026-03-11 19:11:13 <durrendal> indeed, will second that, and for taking the time to discuss it with those of us that are interested in understanding the problem and trying to help solve it
2026-03-11 19:11:39 <ikke> I also appreciate you brainstorming along
2026-03-11 19:11:46 <ikke> This has been in my mind for quite some time already
2026-03-11 19:12:16 <durrendal> you got me thinking about it the last time we talked, I've just had bandwidth issues on my end due to $work
2026-03-11 19:12:26 <ikke> yeah, we're all busy
2026-03-11 19:13:08 <durrendal> ain't that the truth, which makes all of this all the more meaningful to keep going, we all care enough to do it despite that
2026-03-11 19:15:48 <jvvv> I don't have the level of expertise of most of you, but one thing I have acquired is time. I am now a pensioner. If some don't mind doing some occasional hand holding, I am willing and interested in helping.
2026-03-11 19:17:13 <pj> ikke: i don't suppose there is a document with current infra design?
2026-03-11 19:17:13 <ikke> jvvv: I'm certainly willing to do
2026-03-11 19:17:17 <pj> or is it fine to pester you for answers :>
2026-03-11 19:17:54 <ikke> There'se a lot I have to document..
2026-03-11 19:17:59 <ikke> Things changing all the time does not help
2026-03-11 19:18:15 <ikke> pj: anything in particular you wonder about?
2026-03-11 19:18:22 <pj> I know that too well
2026-03-11 19:19:43 <pj> I'm not sure, at least not yet. I have like some ideas: i.e noarch could be scheduled on the most powerful arch builders
2026-03-11 19:22:02 <pj> about the racing stuff, I'm feeling that there should be some kind of locking/cancelling
2026-03-11 19:22:31 <ikke> That requires some orchestrating / centralized services
2026-03-11 19:22:36 <pj> I have to write this down in my notes and then do some scribbles to see how that would work
2026-03-11 19:22:41 <ikke> each builder is now completely independent
2026-03-11 19:23:11 <ikke> (read, each arch)
2026-03-11 19:23:38 <ikke> When a new commit is pushed, each builder can start building it right away
2026-03-11 19:23:41 <pj> right but there is a centralised pipe of information for builds, iirc mqtt
2026-03-11 19:24:01 <pj> is it mqtt that schedules builds or each builder looks at git
2026-03-11 19:24:16 <ikke> each git push is anounced over mqtt
2026-03-11 19:24:27 <ikke> and then the builders pick that up and start building
2026-03-11 19:24:58 <ikke> So triggering would be a better word
2026-03-11 19:26:41 <pj> I'll see if I'm awake enough to ponder about this later
2026-03-11 19:27:42 <ikke> https://gitlab.alpinelinux.org/alpine/tsc/-/issues/82
2026-03-11 19:29:30 <pj> I know about this issue but I remember it didn't went anywhere and I'm hoping to not redesign anything
2026-03-11 19:29:33 <pj> just do small patch :>
2026-03-11 19:30:32 <pj> > The builders need to have all packages locally, resulting in storage issues. Sometimes it's possible to get extra disk storage, but that's not the case for all architectures
2026-03-11 19:31:02 <pj> I think that could be solved with rootbld?
2026-03-11 19:31:15 <pj> but I guess rootbld hasn't been tested (and I doubt it works flawlessly)
2026-03-11 19:31:16 <ikke> No, rootbld solves a different problem
2026-03-11 19:31:35 <pj> I mean rootbld would pull the packages on demand no?
2026-03-11 19:31:45 <ikke> The builders could do that as well
2026-03-11 19:31:56 <ikke> without rootbld*
2026-03-11 19:32:11 <ikke> The builders are now the canonical source of the repo
2026-03-11 19:32:29 <ikke> they rsync it to the master mirror, which then gets distributed
2026-03-11 19:32:40 <pj> I guess that's true but it isn't really explained why it needs all packages
2026-03-11 19:32:50 <pj> or is it because builders just churn through everything and upload on end?
2026-03-11 19:32:57 <ikke> Yes
2026-03-11 19:33:06 <ikke> and rsync --delete to the master mirror
2026-03-11 19:33:15 <ikke> So they need a full snapshot of the repo
2026-03-11 19:33:24 <pj> ok, i see now
2026-03-11 19:34:54 <ikke> But rootbld on lxc needs some special config
2026-03-11 19:35:19 <ikke> What is solves is some persistent state problems that the builders now have
2026-03-11 19:38:21 <pj> have you explored already the existing solutions then?
2026-03-11 19:38:58 <ikke> Not really yet
2026-03-11 19:39:34 <ikke> Someone making a list (with references) would be helpful
2026-03-11 19:40:14 <pj> i meant the solutions listed in tsc issue
2026-03-11 19:41:00 <ikke> I did experiment with buildbot in the past
2026-03-11 19:41:48 <ikke> The rest are very generic and would require some additional layers to glue everything together
2026-03-11 19:41:52 <pj> like we would need to run actual PoC and try something out to see if it works well
2026-03-11 19:42:45 <pj> I'm not even looking at NATS, it's too complex
2026-03-11 19:42:45 <ikke> We need some orchestration that ensures the built packages remain consistent (if we do distributed builds)
2026-03-11 19:44:13 <ikke> if I push an update to a package, and realize I forgot to bump pkgrels, so push that 1 minute later, those rebuilds should happen only after the first package has been built, and the rebuilds should have access to the first package
2026-03-11 19:44:26 <ikke> (just an made up scenario)
2026-03-11 19:45:16 <ikke> Right now, because there is just a single builder, it will first finish the building of the package, then start the rebuilds, and because the built package is local, the rebuilds immediately have access to that package
2026-03-11 19:46:16 <ikke> So, I think that's the first question we need to answer, before looking at software that we may use to implement it
2026-03-11 19:48:01 <pj> I think that can influence the solution to choose but I also think we don't need to concern ourselves with distributed building (unless you want noarch to be distributed?)
2026-03-11 19:48:03 <pj> for now
2026-03-11 19:48:37 <durrendal> sounds like we need a state machine
2026-03-11 19:49:12 <pj> I'm type of person that always wants perfect world but if we try hunting for perfect thing, we might never solve any of the issues :> 
2026-03-11 19:49:21 <ikke> yeah
2026-03-11 19:49:52 <ikke> If we don't want distributed builds, it would probably be best to stay with the current architecture and try to improve the painpoints
2026-03-11 19:50:15 <ikke> But for architectures like riscv64, there's a lot to win there
2026-03-11 19:50:23 <ikke> (But also a high price to pay)
2026-03-11 19:51:08 <pj> Is it because of smaller builders?
2026-03-11 19:51:13 <pj> or do you have just multiple same machines
2026-03-11 19:51:22 <ikke> combination of both
2026-03-11 19:51:30 <ikke> the current hardware is slow
2026-03-11 19:51:51 <ikke> so distributing it over multiple machines could potentially improve things
2026-03-11 19:51:56 <pj> one concern is that we generally shouldn't schedule things like LLVM on weakest machines
2026-03-11 19:52:34 <pj> mmmm, distcc would go nice
2026-03-11 19:52:56 <ikke> but that would only help with packages built with c(++), right?
2026-03-11 19:53:08 <pj> (it wasn't serious suggestion)
2026-03-11 19:53:08 <ikke> go, rust etc would not benefit from it?
2026-03-11 19:53:14 <ikke> heh
2026-03-11 19:53:41 <ikke> For CI, I already have a system to add an additional tag to make sure the projects are only schduled on larger instances
2026-03-11 19:53:50 <pj> well, if there isn't any clash with CPU flash across those boards
2026-03-11 19:53:51 <pj> it could technically be used
2026-03-11 19:53:58 <pj> s/flash/flags/
2026-03-11 19:54:36 <pj> right but that is per gitlab project?
2026-03-11 19:54:54 <ikke> Yes, just for aports
2026-03-11 19:55:02 <pj> or is it done as a pre-job to figure out which package is built and uses dynamic pipeline
2026-03-11 19:55:10 <ikke> that, yes
2026-03-11 19:55:31 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/.gitlab/job-config.yaml
2026-03-11 19:56:04 <pj> nice
2026-03-11 19:56:36 <pj> ok, I have to eat something and get some juices for the brain to function
2026-03-11 19:56:48 <ikke> o/
2026-03-11 19:59:26 <ikke> So one thing that APKv3 was mentioned to help with is that it could do remote signing
2026-03-11 20:29:45 <ikke> Number of requests per repo on a single mirror server over a period of ~24h https://tpaste.us/xD01 
2026-03-11 20:30:28 <ikke> And that's not even taking cdn caching into account
2026-03-11 21:17:11 <ikke> The alternative is manually backporting the security fixes, but that's also not without risk
2026-03-11 21:17:16 <ikke> wrong channel
2026-03-11 21:34:29 <durrendal> Wow half as much volume on 3.14 as latest stable is baffling, not at all what I expected. I thought it'd be 25% at most by comparison
2026-03-11 21:35:01 <ikke> Like I said, you would understimate it :P
2026-03-11 21:38:21 <durrendal> Hahaha you weren't wrong!
2026-03-12 16:58:05 <jvvv> The build for py3-lmdb on the loongarch64 builder segfaulted in check(). AFAICT it hasn't tried to rebuild it. Could it have crashed the builder?
2026-03-12 17:00:11 <ikke> jvvv: it tried it over and over and over and over
2026-03-12 17:00:52 <ikke> excerpt: https://tpaste.us/vo0o 
2026-03-12 17:12:41 <jvvv> Oh, wow. I think I better ask for advice on the loongarch64 channen
2026-03-12 17:12:48 <jvvv> *channel
2026-03-12 17:14:47 <ikke> That would be helpful
2026-03-12 18:05:10 <jvvv> I've pushed !98926 to address that test. If I can find a fix, can add a bump. but at least this should help it to not backup  the builder
2026-03-12 18:11:42 <ncopa> jvvv: thanks!
2026-03-12 18:14:07 <jvvv> Welcome
2026-03-12 18:16:04 <ikke> Packetloss / flapping
2026-03-13 10:33:35 <dne> can't get this to pass w/o getting "OOMKilled": https://gitlab.alpinelinux.org/dne/aports/-/jobs/2257614
2026-03-13 10:33:58 <dne> (and on 32-bit x86 the threading test never completes…)
2026-03-14 03:54:49 <UnknownArtistBot> Hi all! check this out, An Unknown Artist - https://www.anunknownartist.com/
2026-03-14 09:01:18 <lotheac> i'm confused about build-edge-riscv64 uploads. go-1.26.1-r1 was built there on march 10th according to the logs timestamp but not available on mirrors or visible on pkgs.alpinelinux.org
2026-03-14 12:05:23 <ikke> lotheac: it only uploads once the entire repository is finished, which has not happened yet
2026-03-14 12:12:18 <lotheac> ah i see
2026-03-14 12:12:31 <lotheac> so build failures can block uploads of other stuff?
2026-03-14 12:13:14 <ikke> yes
2026-03-14 12:13:48 <lotheac> thanks for the explanation
2026-03-15 05:44:31 <lotheac> ikke: ERROR: Job failed (system failure): error dialing backend: dial tcp 192.168.40.103:10250: i/o timeout https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2260358
2026-03-15 05:45:00 <lotheac> 10250 is the kubelet port (used for eg. getting logs or starting exec's on pods)
2026-03-15 05:50:23 <ikke> lotheac: the node itself appears to be healthy
2026-03-15 05:50:44 <lotheac> can you connect to 10250 on it from another node?
2026-03-15 05:51:55 <ikke> 192.168.40.103 (192.168.40.103:10250) open
2026-03-15 05:52:28 <ikke> curl -k https://192.168.40.103:10250
2026-03-15 05:52:30 <ikke> 404 page not found
2026-03-15 05:52:34 <ikke> So, yes
2026-03-15 05:53:16 <lotheac> well that's weird
2026-03-15 05:54:42 <lotheac> clearly the gitlab executor is able to connect to the api server (otherwise it would not be able to get statuses of pods or create them), but then it fails execing into the pod (which would be a tcp connection initiated from the api server host to the target kubelet)
2026-03-15 05:55:44 <lotheac> so... does opening a tcp connection to that address work from all control plane nodes?
2026-03-15 05:55:59 <ikke> Not to that address
2026-03-15 05:57:12 <ikke> That's a private vlan only reachable from the nodes within that network
2026-03-15 05:57:28 <lotheac> if that's the case then "kubelet logs pod/whatever" should also fail (for a pod running on that host)
2026-03-15 05:57:58 <lotheac> and it would mean that the host's internal node ip is wrong if that address is not connectable from the rest of the cluster
2026-03-15 05:59:07 <ikke> Interestingly enough it does work on the other nodes, which use the same network as internalIP
2026-03-15 05:59:39 <lotheac> really? sounds like it should not :D
2026-03-15 06:00:47 <ikke> https://tpaste.us/wa0X 
2026-03-15 06:02:03 <lotheac> so .102 and .104 tcp/10250 *are* connectable from cplane nodes? or?
2026-03-15 06:03:08 <ikke> No, it would not
2026-03-15 06:03:21 <ikke> I mean, the nodes are able to accept jobs
2026-03-15 06:03:23 <ikke> and they pass
2026-03-15 06:03:26 <lotheac> ah, i see
2026-03-15 06:03:51 <lotheac> that doesn't make much sense either though... afaiu the gitlab kube executor always uses exec
2026-03-15 06:04:07 <lotheac> and connectivity to internalIP:10250 is required for exec as well as kubectl logs
2026-03-15 06:05:06 <ikke> I should've probably explicitly specified what interface to use
2026-03-15 06:05:17 <ikke> But strange that it works
2026-03-15 06:05:48 <lotheac> yeah, i can't explain why it would work :D
2026-03-15 06:06:24 <ikke> I have a wg tunnel to each of the nodes
2026-03-15 06:06:53 <lotheac> from?
2026-03-15 06:07:10 <ikke> dmvpn <-> wg concentrator <-> nodes
2026-03-15 06:07:44 <lotheac> routed using those same addresses?
2026-03-15 06:07:49 <lotheac> ah i gotta go afk for a bit anyway
2026-03-15 06:07:56 <ikke> not the internal ip obviously
2026-03-15 06:08:04 <ikke> internal lan subnet
2026-03-15 06:08:32 <ikke> lotheac: alright, thanks
2026-03-15 06:25:12 <lotheac> anyway i think the correct thing to do is rejoin those nodes to the cluster with internalIPs that are reachable from the control plane (or even all other nodes)
2026-03-15 06:27:51 <lotheac> maybe it's not a requirement that we understand why .102 and .104 are working without that :D
2026-03-15 06:34:25 <lotheac> i guess it's this option https://github.com/k0sproject/k0sctl?tab=readme-ov-file#spechostsprivateinterface-string-optional-default-
2026-03-15 06:36:10 <ikke> yes
2026-03-15 06:36:17 <ikke> I've already used it before
2026-03-15 06:36:58 <ikke> Let me start with node3
2026-03-15 06:40:37 <lotheac> you could kubectl drain them before you make changes if you want to minimize disruption i guess, assuming that the gitlab executor pods have appropriate disruption budgets that prevent running jobs from being evicted
2026-03-15 06:47:08 <ikke> I already cordonned the node
2026-03-15 06:47:27 <lotheac> alright
2026-03-15 06:48:29 <ikke> usa11-ci-3.alpinelinux.org   Ready,SchedulingDisabled   <none>          25d    v1.33.3+k0s   172.16.252.37
2026-03-15 06:49:05 <ikke> Hmm, apparently it can update it
2026-03-15 06:49:31 <lotheac> ah, cool, i had assumed it would be immutable for the node obj
2026-03-15 06:49:46 <lotheac> not that it's much of a difference to have to delete and rejoin it anyways though
2026-03-15 06:52:37 <ikke> It does reinstall k0s as it calls it, but the running pods are not interrupted
2026-03-15 06:52:47 <ikke> (the daemonset pods were and are still running on the node)
2026-03-15 06:53:05 <lotheac> yeah, those are handled by the container runtime
2026-03-15 06:53:16 <ikke> right
2026-03-15 06:53:30 <ikke> But it did not do a complete reset or anything like that
2026-03-15 06:53:50 <lotheac> right, it might have recreated the local containers after rejoining
2026-03-15 06:55:36 <ikke> tarted:      Thu, 12 Mar 2026 00:06:15 +0100
2026-03-15 06:55:48 <ikke> Container has kept running
2026-03-15 06:56:16 <lotheac> i believe you, i meant to say "it might have ... [but clearly didn't]"
2026-03-15 06:56:54 <lotheac> thanks for handling it
2026-03-15 07:16:00 <ikke> I've done 2 nodes, the last one is building a long-running job that I don't want to interrupt
2026-03-15 07:16:38 <lotheac> you could leave it cordoned and do it later
2026-03-15 07:20:40 <ikke> yes
2026-03-15 07:20:49 <ikke> That's exactly what I did :)
2026-03-15 07:21:34 <lotheac> _b
2026-03-15 07:29:52 <ikke> I have a node where calico is not working correctly after reboot. ip -d link show dev vxlan.calico shows that it uses the wrong external interface
2026-03-15 07:32:13 <lotheac> i don't use calico on my clusters, but i image it would be configurable in some fashion to help it pick the right interfaces. in cilium that can be done with a crd CiliumNodeConfig
2026-03-15 07:33:35 <lotheac> maybe the Node object https://docs.tigera.io/calico/latest/networking/ipam/ip-autodetection#autodetecting-node-ip-address-and-subnet
2026-03-15 08:00:51 <ikke> Trying ipAutodetectionMethod: "kubernetes-internal-ip" in the calico config
2026-03-15 11:47:21 <omni> is CI having DNS isssues?
2026-03-15 12:25:38 <lotheac> why? did you get an error about resolving something?
2026-03-15 12:27:21 <ikke> I may have broken something 😔
2026-03-15 12:54:32 <omni> fatal: unable to access 'https://gitlab.alpinelinux.org/alpine/aports.git/': Could not resolve host: gitlab.alpinelinux.org (Timeout while contacting DNS servers)
2026-03-15 13:24:45 <omni> during generate-build-jobs
2026-03-15 13:42:06 <omni> gitlab also feels extra sluggish, but it could be my connection
2026-03-15 17:06:46 <tomalok> omni - i'm getting the same
2026-03-15 19:49:06 <ikke> I think it should be fixed now
2026-03-15 19:49:10 <ikke> Let me know if you still face issues
2026-03-15 19:58:23 <ikke> lotheac: I had to make sure every node consistently used the internal ip address, which I did now
2026-03-15 20:16:09 <ikke> I see the latest go package for riscv64 is now also available
2026-03-15 20:19:00 <mio> in lint runner: ERROR: Preparation failed: starting pod watcher: not synced: *v1.Pod
2026-03-15 20:19:25 <mio> should the pipeline be restarted? only retried just lint
2026-03-15 20:19:36 <mio> the rest of the pipeline was fine
2026-03-15 20:19:59 <mio> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2260904
2026-03-15 20:25:57 <ikke> retried the job, and it works. Not sure what the problem is/was
2026-03-15 20:26:26 <mio> okay, thanks!
2026-03-16 00:43:31 <lotheac> ikke: thanks
2026-03-16 11:10:44 <Sertonix[m]> ikke: reminder for https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/96922
2026-03-16 20:22:58 <ikke> Sertonix[m]: Do we think we can keep maintaining it even though upstream is dropping 32-bit support?
2026-03-16 20:25:53 <Sertonix[m]> The only thing dropped by upstream was optimizations on x86 for openjdk25. Before that I don't think there was anything dropped by upstream, they just never tested musl libc.
2026-03-16 20:26:24 <Sertonix[m]> *never test musl libc on 32-bit systems
2026-03-17 09:11:57 <omni> is build-edge-x86_64 stuck?
2026-03-17 09:26:31 <ikke> No
2026-03-17 09:45:16 <omni> oh, ok, it just hasn't uploaded anything the past few hours, perhaps the duckdb build (with failure) takes a lot of time
2026-03-17 09:49:48 <ikke> It's building rust atm
2026-03-17 09:50:03 <ikke> And build.a.o shows that
2026-03-17 09:56:33 <clandmeter> ikke: i need to kick bld2?
2026-03-17 09:57:20 <ikke> Yes please
2026-03-17 10:19:19 <omni> ikke: it was building duckdb when I asked
2026-03-17 10:19:42 <ikke> Understood, it just started building rust when I looked
2026-03-17 10:20:05 <omni> yeah, and it then failed with OOM =(
2026-03-17 10:20:29 <omni> probably because I thought I could go back to default number of jobs, as it worked in CI
2026-03-17 10:20:31 <ikke> Somethings not healthy
2026-03-17 10:21:28 <omni> it's building rustc_driver in stage2 that it seems to have the same issue aarch64 had, before I switched to the default thing-local lto for it
2026-03-17 10:21:43 <ikke> Is it stack exhaustion?
2026-03-17 10:23:14 <omni> how much memory per core (or thread) is available on the aarch64 and x86_64 builders?
2026-03-17 10:24:44 <ikke> Roughly 3G and 5G respectively
2026-03-17 10:42:14 <omni> how is it on loongarch64, ppc64le and s390x? where we seem to not have the issue
2026-03-17 10:44:47 <ikke> On ppc64le it's 2G per core
2026-03-17 10:45:17 <ikke> Similar on s390x
2026-03-17 10:45:45 <ikke> Loongarch is 8GB
2026-03-17 10:45:54 <ikke> But I have not seen these machines run low on memory
2026-03-17 10:48:37 <omni> no.. =/
2026-03-17 10:49:24 <omni> anyway, !99162 if it continues to fail
2026-03-17 10:50:27 <omni> thin-local is the default lto but I think we want to use thin if we can
2026-03-17 10:50:38 <omni> https://nnethercote.github.io/perf-book/build-configuration.html#link-time-optimization
2026-03-17 10:53:17 <omni> the alternative seems to be to lower the number of build jobs
2026-03-17 10:55:29 <ikke> But why is it not an issue for s390x and ppc64le, which have the least amount of memory available per job
2026-03-17 11:02:51 <omni> probably releated to the uefi target, only enabled on aarch64 & x86_64, the comment for lowering number of jobs said as much
2026-03-17 11:03:24 <omni> but I went with thin-local lto for aarch64 instead of lowering the number of jobs even more
2026-03-17 11:04:02 <omni> I haven't tried to build arm* with thin lto, perhaps that's no longer an issue
2026-03-17 11:45:12 <ikke> clandmeter: thanks
2026-03-17 21:48:05 <Sertonix[m]> Is something broken with the ipv6 config of irclogs.alpinelinux.org? 'curl -6 https://irclogs.alpinelinux.org' never returns
2026-03-17 21:52:43 <Sertonix[m]> *timeouts
2026-03-18 08:00:30 <ikke> Sertonix[m]: I'll check
2026-03-18 11:34:10 <ikke> Sertonix[m]: it points to the wrong address, I'll fix it
2026-03-18 11:55:49 <ikke> Sertonix[m]: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/117
2026-03-18 12:29:20 <ikke> Sertonix[m]: should be fixed now (once everything has propagated)
2026-03-18 12:30:21 <Sertonix[m]> Yes, thanks!
2026-03-18 16:17:39 <ikke> load is high
2026-03-18 19:47:34 <ikke> Sertonix[m]: trying to bootstrap openjdk11 on x86. The abuild-setup-cross script fails because CBUILDROOT is not set. It would require $CHOST != $CTARGET, but when function.sh is sourced, that's not the case
2026-03-18 19:48:04 <ikke> It's apparently overwritten
2026-03-18 19:49:16 <ikke> Am I missing something?
2026-03-18 19:49:36 <ikke> I'm running ash `abuild-setup-cross x86 ./aports` like mentioned in the MR
2026-03-18 19:51:32 <ikke> ah, my bad
2026-03-18 19:52:41 <Sertonix[m]> I didn't run the commands exactly like written in the MR so mistakes are possible
2026-03-18 19:53:21 <ikke> I missed one crucial detail :P
2026-03-18 20:06:54 <ikke> Ok, build finished
2026-03-18 20:08:08 <Sertonix[m]> \o/
2026-03-18 20:10:59 <ikke> I guess the next step is to use https://dev.alpinelinux.org/~kevin/openjdk11-bootstrap/community/x86/ to bootstrap it on the builder?
2026-03-18 20:14:01 <ikke> Would it make sense to merge !96922 and then use the built packages to bootstrap it?
2026-03-18 20:15:28 <Sertonix[m]> Sounds reasonable. I don't know what is possible on the builders
2026-03-18 20:15:42 <ikke> Similar to how we bootstrap go and others
2026-03-18 20:17:26 <ikke> abuild-apk add -X https://dev.alpinelinux.org/~kevin/openjdk11-bootstrap/community openjdk11-bootstrap
2026-03-18 20:17:46 <ikke> That installs the -bootstrap package, then openjkd11 itself should be able to be built
2026-03-18 20:18:32 <ikke> Sertonix[m]: it's building now :)
2026-03-18 20:22:59 <ikke> Sertonix[m]: and finished
2026-03-18 20:42:57 <Sertonix[m]> Thanks!
2026-03-18 21:08:08 <l0rd_hex> hi friends, I was wondering about the Alpine packaging infrastructure, are the packages built via cross compiling for different archs?
2026-03-18 21:35:08 <ikke> no, native hosts
2026-03-18 21:43:41 <l0rd_hex> ah, interesting 
2026-03-18 21:43:57 <ikke> For 32-bits, we use 64-bits host in 32-bit mode
2026-03-18 21:45:35 <l0rd_hex> I'm new to package maintaining and was looking at submitting one that's not currently available, is the testing on different arches done post-submission in the testing repo?
2026-03-18 21:45:48 <l0rd_hex> I should check it's not in testing actually 
2026-03-18 21:46:26 <ikke> When you create a merge request, it's already built in CI on all arches
2026-03-19 08:34:37 <ncopa> lotheac: I finally rebooted the bananapi runner
2026-03-19 13:28:41 <lotheac> thanks! i will test my theory and commit if it works
2026-03-19 13:31:14 <lotheac> .. tomorrow, after gitlab is no longer down
2026-03-19 13:32:04 <ikke> It should be back in a sec
2026-03-19 13:32:34 <lotheac> you're the best ikke <3
2026-03-19 13:32:46 <ikke> It's a planned upgrade :)
2026-03-19 13:33:01 <lotheac> doesn't make you any less than the best
2026-03-19 13:33:59 <ikke> :_
2026-03-19 13:34:01 <ikke> :)
2026-03-19 13:34:03 <ikke> It's booting again
2026-03-19 13:35:20 <lotheac> it's alive!
2026-03-19 13:35:21 <lotheac> thank you
2026-03-19 13:53:44 <lotheac> i guess it will have to be tomorrow though, because i'm not sober enough for this: 
2026-03-19 13:53:45 <lotheac> cmd/dist
2026-03-19 13:53:45 <lotheac> Building Go cmd/dist using /usr/lib/go. (go1.26.1 linux/riscv64)
2026-03-19 13:53:45 <lotheac> go tool dist: unknown $GOARCH unsupported
2026-03-19 13:53:45 <lotheac> >>> ERROR: go: build failed
2026-03-19 13:55:09 <lotheac> (on nor-ci-1, manual build in manual container)
2026-03-19 13:56:10 <lotheac> well, problems for future me. thanks ncopa :)
2026-03-19 16:01:23 <pj> can i give alpine matrix rooms alpine logo
2026-03-19 16:01:51 <pj> i sometimes lose them between all other IRC rooms :>
2026-03-19 16:22:26 <ikke> Funny, just as I upgrade gitlab today, they release a new version
2026-03-19 16:22:32 <ikke> pj: I think so
2026-03-19 19:13:14 <panekj> ikke: is the build.sh used in CI in the CI image?
2026-03-19 19:13:22 <ikke> yes
2026-03-19 19:13:41 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci/-/blob/master/overlay/usr/local/bin/build.sh
2026-03-19 20:22:44 <invoked> anarcat shared how to fix the "work items" situation in the new gitlab. https://gitlab.com/gitlab-org/gitlab/-/work_items/590689#note_3175494866
2026-03-19 20:22:58 <invoked> that change is going over like a lead balloon
2026-03-19 21:20:05 <ikke> It mentions gitlab-ctl reconfigure, but we're not using the omnibus installation
2026-03-26 16:22:23 <ikke> lotheac: I'll discuss with carlo if I can add you as a developer to the infra group, then you no longer need to use forks
2026-03-26 16:22:30 <lotheac> cheers
2026-03-26 16:22:49 <ikke> We don't have any instance runners for building docker images
2026-03-26 16:22:58 <ikke> (on purpose)
2026-03-26 16:23:06 <lotheac> that makes sense
2026-03-26 16:23:11 <ikke> so I need to explicitly assign runners to the fork
2026-03-26 16:23:19 <ikke> Which I did
2026-03-26 16:23:48 <ikke> But by using $CI_REGISTRY_IMAGE, it would require the image to build the images to be present in the fork
2026-03-26 16:24:30 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/exec/-/jobs/2276008
2026-03-26 16:24:47 <lotheac> right. i guess it could have been bootstrapped in the fork separately but that doesn't feel productive :p
2026-03-26 16:25:25 <ikke> No
2026-03-26 16:25:47 <ikke> Would be a lot of busy work :)
2026-03-26 16:26:42 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/exec/container_registry/59
2026-03-26 16:26:52 <lotheac> it's alive :)
2026-03-26 16:27:48 <ikke> Thanks, I'll try to test it out when I have time
2026-03-26 16:27:56 <lotheac> np, thanks for merging
2026-03-26 16:28:25 <lotheac> i didn't really do any perf testing, it's possible it's slower than docker
2026-03-26 16:28:41 <lotheac> but the tradeoff is less privs
2026-03-26 16:28:46 <ikke> yeah
2026-03-26 16:28:58 <ikke> The only case where it matters somewhat is the gitlab image
2026-03-26 16:30:48 <lotheac> if it does become an issue it *is* possible to run docker in kube too. might be slightly annoying to set up though
2026-03-26 16:31:05 <lotheac> like this https://gitlab.alpinelinux.org/alpine/infra/docker/exec/-/issues/1#note_585276
2026-03-26 16:32:08 <ikke> In general I just push it and wait for it to be finished, but if there is some urgen thing to fix (security or bug) it can be anoying if it takes too loiing
2026-03-28 13:00:43 <mio> could the build-edge-ppc64le and build-edge-s390x builds be restarted please? unfortunately the test is hanging
2026-03-28 13:01:57 <ikke> done
2026-03-28 13:03:48 <mio> thanks! sorry, x86 as well?
2026-03-28 13:07:21 <ikke> done
2026-03-28 13:07:21 <mio> thanks
2026-03-28 13:07:46 <mio> thank you, should be able to pass on retry
2026-03-29 21:01:36 <clandmeter> ikke: ^
2026-03-29 21:01:47 <ikke> clandmeter: yes, I noticed, Thanks!
2026-03-30 14:50:40 <mio> can someone restart the build on build-edge-x86 please?
2026-03-30 15:11:01 <mio> thanks
2026-03-30 15:11:01 <ikke> done
2026-03-30 15:11:07 <mio> thanks!
2026-03-31 07:29:31 <omni> is it an infrastructure issue that !98913 is failing on x86* ?
2026-03-31 07:52:47 <ikke> Technically it's a project using excessive memory :P
2026-03-31 13:22:51 <omni> so, large build hosts for that too?
2026-03-31 13:59:55 <ikke> clandmeter: thanks