2026-07-01 12:42:38 hello, what can we do to unlock build-edge-riscv64 ? 2026-07-01 12:47:14 Find stable, fast and available rv64 hw 2026-07-01 12:47:24 oh, and affordable 2026-07-01 12:49:21 The builders are locking up / freezing 2026-07-01 13:04:03 i have ordered k3 machine(s). not sure if/when they arrive 2026-07-01 13:04:21 I guess you want to own a server, not renting one at providers like Scaleway, right? 2026-07-01 13:04:47 how many rv64 sbcs did you order so far? 2026-07-01 13:15:50 raspbeguy: we have 2 servers on scaleway, but they are used for CI 2026-07-01 13:16:01 ok 2026-07-01 13:21:05 2x K3. one from sipeed.com and one milk-v jupiter 2 2026-07-01 13:23:11 i also have a banapi bpi-f3 machine on my desk here, as CI, and a hifive pioneer p550 which is the current build-edge-riscv64 and build-3-20-riscv64. I think CI is disabled on the p550 for now 2026-07-01 13:25:20 i also have a stafive sifive 2 as a router, and an orangepi rv2 that is not set up yet 2026-07-01 13:25:50 oh also a starpro64, which also is waiting for a kernel 2026-07-02 05:01:30 great 2026-07-02 20:52:27 smth is borked with gitlab ci aarch64: https://gitlab.alpinelinux.org/dne/aports/-/jobs/2419937 2026-07-02 21:20:52 dne: I added a new runner to help with the load, but it needs to be tuned a abit 2026-07-02 21:44:29 ikke: Is there anything more I should do with https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/merge_requests/13? 2026-07-02 21:45:39 No, looks good 2026-07-02 21:45:50 I've added the mirror to our monitoring to see how stable it was 2026-07-02 21:47:19 dne: I've paused the runner for now 2026-07-02 21:47:48 Thanks. Appreciated. 2026-07-03 05:06:16 fyi, I've enabled the runner again, it should work better now, but please let me know if it still has issues 2026-07-03 07:01:58 ikke: I see, seems better now 2026-07-06 08:37:08 Hey! Quick question: Is it possible that the riscv64 builder skipped the latest build of 'baikal' from 2026-06-23? 2026-07-06 08:38:27 I can see that e.g. 'rust' has been built for 'risc' yesterday, 2026-07-06 (triggered for all architectures originally on 2026-07-03), so rust was triggered after 'baikal' has been 2026-07-06 08:38:54 Are perhaps 'community' and 'main' aports prefered over 'testing' aports or has there been a skip? 2026-07-06 08:39:08 see https://pkgs.alpinelinux.org/packages?name=rust&branch=edge&repo=&arch=&origin=&flagged=&maintainer= 2026-07-06 08:39:22 sorry, that's https://pkgs.alpinelinux.org/packages?name=baikal&branch=edge&repo=&arch=&origin=&flagged=&maintainer= 2026-07-06 08:40:43 Yeah the builder build main, push the repo, community, push the repo, and testing and push the repo. currently its on its way building community (see https://build.alpinelinux.org/), so it didnt got to baikal yet, but rust is already pushed 2026-07-06 08:41:58 I see :) Then just patience is the way. Thanks for confirming! 2026-07-06 09:37:02 we have a problem with the riscv64 builder. new builds come in faster than the builder manages to build 2026-07-06 09:37:29 maybe I should move it back to the pioneer machine, now that 3.24 is done 2026-07-06 10:02:40 I think that would help 2026-07-06 10:26:33 BTW, thought I'd mention, I sometimes get in an infinite loop with go-away 2026-07-06 10:30:07 f_: hmm, on gitlab? 2026-07-06 10:31:04 yeah 2026-07-06 10:31:27 Not sure it's something you can do much about, this isn't exclusive to alpine gitlab 2026-07-06 10:31:45 (e.g. I sometimes get that behavior on fdo..) 2026-07-07 07:19:58 Hmm 2026-07-07 07:20:28 Server is slighly overloaded 2026-07-07 07:21:29 Yeah very much noticeable 😅 2026-07-07 07:48:31 Ok, should be better now 2026-07-07 07:48:51 lots of requests from python-requests/2.34.2 from amazon 2026-07-07 08:04:45 Smells like LLM training 2026-07-07 14:38:37 ikke: just block that 2026-07-07 14:38:46 many websites block it for these reasons 2026-07-07 14:39:02 Legitimate software will set their UA correctly anyway 2026-07-07 14:39:18 f_: I have 2026-07-07 14:40:02 good 2026-07-07 14:40:15 blocked up to 80 r/s at some point 2026-07-07 14:40:20 woah 2026-07-08 09:22:35 More crawlers... 2026-07-08 09:22:42 Need to send abuse mails to AWS 2026-07-08 09:28:31 maybe send an invoice instead ;) 2026-07-08 09:28:40 If only 2026-07-08 10:47:35 It's so anoying, the load is quite low and easy to handle under normal circumstances. We are forced to make everything a lot more complicated just to handle this abuse 2026-07-08 10:48:31 Server load is below 5 now. While under load, it was 150+ 2026-07-08 15:14:21 and here we go again 2026-07-08 15:17:50 I could use some help with improving analysis 2026-07-08 15:17:52 I have an idea 2026-07-08 15:18:23 I have most components already, but need them glued together 2026-07-08 15:22:56 ikke: For what it's worth, we've been suffering from the same thing on the Arch Linux side for some time 😕 Pretty annoying indeed. 2026-07-08 15:23:43 Antiz1: Yeah, can imagine 2026-07-08 15:23:56 Some traffic is easy to identify, sometimes it's harder 2026-07-08 15:23:58 ikke: For what it's worth, crawlers are presenting themselves with a Windows NT or Macintosh User agent 🤷 2026-07-08 15:24:02 On our side that is. 2026-07-08 15:24:06 https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/1217/diffs 2026-07-08 15:24:39 The above seems to work for reducing the load on our side. 2026-07-08 15:24:43 Yeah, I've seen that coming around 2026-07-08 15:24:54 Ok, will apply a similar filter on our go-away setup, thanks! 2026-07-08 15:25:08 We had to do the same for the Arch Wiki as well. 2026-07-08 15:25:16 Yeah, similar here 2026-07-08 15:25:22 ikke: Alright, hopefully it'll help 🤞 2026-07-08 15:25:27 You're welcome! :) 2026-07-08 15:26:10 That looks like a smart way to bump the filter 2026-07-08 15:26:48 ikke: I used https://gitlab.com/anarcat/asncounter/-/blob/main/asncounter.1.md on openmw.org to ban problematic ASN 2026-07-08 15:27:37 I know that Ubuntu is using the https://dustri.org/b/trivial-anti-crawler-with-caddy.html trick for some of its infra 2026-07-08 15:27:42 ^ https://github.com/canonical/ubuntu-autopkgtest-operators/pull/127/changes 2026-07-08 15:27:45 jvoisin: interesting, I have something similar to that, didn't know about asncounter 2026-07-08 15:27:57 at least, what I have was just meant to augment asn data 2026-07-08 15:29:16 we don't have it yet in alpine 2026-07-08 16:04:58 oof, 27 armv7 jobs pending 2026-07-09 04:50:56 wow, if the numbers are correct, it had a small peak of 400+ r/s yesterday 2026-07-09 04:51:01 or tonight 2026-07-09 04:51:17 how are you measuring the numbers? 2026-07-09 04:51:59 I'm reading out go-away metrics 2026-07-09 04:52:19 nginx itself received a peak of 50/s 2026-07-09 04:54:33 https://paste.pictures/fGtMkREsMG.png 2026-07-09 04:55:19 that's something :) 2026-07-09 04:56:44 lotheac: btw, found this yesterday: https://devopsbeast.com/blog/oomkilled-wrong-pod-qos 2026-07-09 04:56:52 And saw that the runner pods have no requests or limits 2026-07-09 04:57:04 So they are the first ones to get evicted 2026-07-09 04:57:22 re: improving analysis that you mentioned... could be a good idea to have horizontally scalable reverse proxies that log all requests (including select headers) and sprinkle some visualization on top. my stack is usually traefik -> victorialogs, grafana 2026-07-09 04:57:25 ikke: heh 2026-07-09 04:57:59 i'm a bit surprised that they do not have requests/limits, i thought we did set those 2026-07-09 04:58:13 I thought so too, but not on the runners themselves 2026-07-09 04:58:21 the job pods have 2026-07-09 04:58:41 Which explains why they are killed so frequently 2026-07-09 04:58:51 right 2026-07-09 04:59:05 i guess i was confused about which gitlab options control which pods 2026-07-09 04:59:46 https://gitlab.alpinelinux.org/alpine/infra/k8s/ci-cplane-1/-/blob/master/kustomize/apps/gitlab-runner/base/small-x86/config.toml like... runners.cpu_request does actually not add cpu reqs to the runner pod? 2026-07-09 05:00:38 No, the config.toml is consumed by the runner 2026-07-09 05:00:50 so it can only apply to jobs it spawns 2026-07-09 05:01:18 The operator controls the runners 2026-07-09 05:02:30 It would have to be set in runner.yaml I believe 2026-07-09 05:02:42 ah, i see what you're saying 2026-07-09 05:03:06 okay, yes, that makes sense 2026-07-09 05:04:47 generally speaking, the idea is that if you have memory requests and limits set on every pod, then it should not be possible for the oom killer to kill random processes (because none of the nodes should have been able to be overprovisioned wrt memory) 2026-07-09 05:05:01 We deployed an aarch64 runner, but because the runner gets killed, it looses track of the build pods, which then linger around, preventing new pods to get scheduled 2026-07-09 05:05:33 Yeah, but the nature of build jobs is that you have to over-provision 2026-07-09 05:05:49 not if you have autoscaling 2026-07-09 05:06:21 but if you do overprovision, then it might be a good idea to dedicate the build jobs to specific nodes 2026-07-09 05:06:31 as in, prevent other pods from scheduling there 2026-07-09 05:06:47 To be honest, with the limits / requests in kubernetes we have more issues then without it on docker 2026-07-09 05:07:37 yeah, you do kinda have to apply the same logic to everything in order for it to work 2026-07-09 05:08:01 On docker, we just let everything do it's thing and it somehow works out 2026-07-09 05:08:27 Only very occasianally with extreme large builds, running 6 times in parallel, it overwhelems one host 2026-07-09 05:08:43 (oh, and that is not even CI) 2026-07-09 05:09:01 if you add a taint to the nodes (say dedicated=build), and a label of the same name, and then add a toleration plus nodeselector correspondingly to the build jobs, then they will only run on those specifid nodes (and nothing else will) 2026-07-09 05:09:29 All our nodes in the CI cluster except for the control plane are build hosts 2026-07-09 05:10:25 well, certainly the infrastructure for creating the build pods are different though :) 2026-07-09 05:10:43 you could also have them be scheduled on the cplane nodes i suppose 2026-07-09 05:11:25 for x86_64 and x86 things seem to be stable right now 2026-07-09 05:11:48 But due to the lack of requests/limits for the runner pods, they easily get OOM killed if something heavy is running 2026-07-09 05:12:10 if the runner pod is only responsible for creating the job pods and interacting with them, that in itself does not need to run on the target architecture, right? 2026-07-09 05:12:24 although i didn't check how exactly gitlab does this... 2026-07-09 05:12:29 correct 2026-07-09 05:13:01 well, then it might make sense to run them on cplane nodes (which probably have some spare memory/cpu anyway) 2026-07-09 05:13:07 not a lot 2026-07-09 05:13:18 I picked the smallest instances available 2026-07-09 05:13:50 i suppose adding request/limit to the runner pods should be the simplest solution. do we have metrics on their usage? 2026-07-09 05:15:36 they look pretty light 2026-07-09 05:15:51 https://tpaste.us/moeg 2026-07-09 05:16:04 Yeah, I don't expect them to use much 2026-07-09 05:16:38 I was trying to find if gitlab recommended anything 2026-07-09 05:17:14 heh Error from server (Forbidden): nodes is forbidden: User "lotheac" cannot list resource "nodes" in API group "" at the cluster scope 2026-07-09 05:18:13 https://tpaste.us/R9oo 2026-07-09 05:18:37 yeah that's not too much spare memory 2026-07-09 05:20:11 On the docker nodes, we have no issues with the runners running on them 2026-07-09 05:21:05 at one of my clients -- though github, not glab -- the idea is that there's very little spare capacity for build CI in the cluster. but, the job pods have tactical nodeselectors and taints on them which allow them only to be scheduled on a specific type of VM. there are normally 0 in the cluster, but if there's a pending build job, cluster-autoscaler will talk to the cloud provider API to create a new node and cloud-init it into the cluster 2026-07-09 05:21:26 and then cluster-autoscaler will delete autoscaled nodes after 10 mins of not having any job run on them 2026-07-09 05:21:42 which is cost-efficient, but a bit complex 2026-07-09 05:22:58 We have a fixed set of nodes that are always-on 2026-07-09 05:23:01 i understand your frustration. kube has many footguns 2026-07-09 05:24:51 how do the docker runners handle scheduling many concurrent jobs? 2026-07-09 05:25:19 We limit the runners to max 2 concurrent builds 2026-07-09 05:25:24 generally with kube one should set mem limit == mem request 2026-07-09 05:26:12 with the 48Gi limit but 24Gi request, the scheduler will happily run as many jobs on the same node at the same time as fit in there, calculated with 24G per job 2026-07-09 05:26:38 yes, that's the idea 2026-07-09 05:26:57 build jobs are not fixed size memory workloads 2026-07-09 05:27:44 if you did the same with docker, i don't see how you would avoid oomkills either 2026-07-09 05:28:23 Sometimes build jobs would get oomkilled 2026-07-09 05:28:35 but not frequently 2026-07-09 05:28:44 and what would prevent dockerd from getting killed? 2026-07-09 05:29:02 Nothing, but the OOM killer would always target gcc or something like that 2026-07-09 05:30:01 But because kubernetes activelly adjusts the oom score based on limits/requests, now the runner becomes #1 target 2026-07-09 05:30:42 right 2026-07-09 05:31:09 adding a priorityclass to the runner would probably help too 2026-07-09 05:32:18 Only against eviction apparently 2026-07-09 05:33:18 yeah. the other part of the story is adding requests/limits 2026-07-09 05:34:44 and also adding monitoring for pods that do get oomkilled... 2026-07-09 05:38:09 One reason we need to set requests so low is to make sure the cluster can actually accept jobs that the runner picks 2026-07-09 05:38:23 if the runner says it can accept a job, but the cluster says there is no space, then the job fails 2026-07-09 05:39:00 right. and it wouldn't know before it tried to schedule the pod 2026-07-09 05:39:06 correct 2026-07-09 05:39:52 So requests has been set to the minimum value to allow 2 jobs, but not 3 jobs 2026-07-09 05:40:17 (per node targetted by a specific runner) 2026-07-09 05:40:49 this might be helpful if you are looking for other pods in the cluster that should probably get limits/requests added: kubectl get pod -A -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName 2026-07-09 05:41:41 quite some BestEffort pods 2026-07-09 05:41:52 Most all kube-system 2026-07-09 05:42:12 fun :) 2026-07-09 05:43:24 but those most likely have priorityClassName set though 2026-07-09 05:43:29 add PRIO:.spec.priorityClassName 2026-07-09 05:44:07 yes 2026-07-09 05:44:27 system-node|cluster-critical 2026-07-09 05:44:55 yeah, that's probably how *they* avoided being killed 2026-07-09 05:45:05 The problem with the runner getting killed is that it looses track of the pods 2026-07-09 05:45:18 if it would gracefully recover, it would not even be an issue in the first place 2026-07-09 05:45:24 that sounds like an implementation issue 2026-07-09 05:45:31 yes, there is a gitlab issue about it 2026-07-09 05:47:13 if the runner pods supported multiple replicas that would help too 2026-07-09 05:47:21 but i suspect that they do not 2026-07-09 05:49:30 each would act as a separate runner accepting jobs 2026-07-09 05:49:41 right, so no 2026-07-09 05:50:35 Easiest fix is setting reqs+limits 2026-07-09 05:50:53 i suppose we could try to hack together some cleanup thing that runs in an initcontainer for the runner pod or something that looks for orphaned jobs and deletes them :p 2026-07-09 05:51:15 yeah, that helps for getting targeted by OOM killer, but does not help for any other situation in which the runner exits or gets killed 2026-07-09 05:51:19 crashes or exceeding its own mem limit 2026-07-09 05:51:45 https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/27333 2026-07-09 05:52:15 old issue :X 2026-07-09 05:52:24 i'll take a proper look later, gotta run to a meeting now 2026-07-09 05:53:19 Have to go as well, talk to you later 2026-07-09 05:53:23 later! 2026-07-09 13:25:37 The aarch64 builder seems to need a bump 2026-07-09 14:16:54 lotheac: I've now set requests+limits on both the container as the init container, it's now has a QOS for guaranteed 2026-07-09 14:17:03 Sertonix[m]: ftr, I bumped it 2026-07-10 01:03:59 ikke: nice :) 2026-07-10 05:32:02 Wow, received spikes of over 100r/s last night 2026-07-10 05:38:25 Even with the modifications you put in the go-away setup? That's got to be frustrating 2026-07-10 05:38:40 Most have been blocked, no impact 2026-07-10 05:38:49 Oh, nice! 2026-07-10 05:42:24 I'm working on some python scripts to help me try to track mirrors that are not responding, hugely out of date, and eventually to verify that certain files are valid, like any APKINDEX's they have. 2026-07-10 05:43:20 Ok, we do track which mirrors our out of date in our monitoring 2026-07-10 05:44:01 When I have time, I'll give you access to that 2026-07-10 05:44:34 Yeah, I'm just looking to put the results in to a db so I can look it heuristics 2026-07-10 05:45:07 No hurry. You are always so busy 2026-07-10 05:45:28 yes, but I keep being busy if I don't delegate things :-) 2026-07-10 05:46:07 Well, if you think of something else for me, don't hesitate to ask 2026-07-10 05:47:34 Anyway, even if I don't really need to do the scripts, I am improving my python foo 2026-07-10 05:48:22 Class inheritance is much easier for me to understand in python that it ever has been for c++ 2026-07-10 05:48:26 Heh 2026-07-10 05:48:50 Having something to practice with really helps 2026-07-10 05:49:12 That is very true 2026-07-10 05:49:54 I will eventually try to tackle go at some point, but for now python is nice because it is so quick to get moving in 2026-07-10 10:08:53 Is build-edge-ppc64le stuck? 2026-07-10 17:51:26 Sertonix[m]: it was 2026-07-11 06:03:58 lotheac: aarch64 runner seems stable now 2026-07-11 06:31:53 great :) 2026-07-13 09:31:37 meh 2026-07-13 14:16:55 Is buils-x86_64-edge stuck? 2026-07-13 15:14:31 Sertonix[m]: no 2026-07-13 15:14:40 but gcc is failing to build 2026-07-13 15:17:37 I think we need to patch gcc to not include linux/scc.h 2026-07-13 15:17:59 I suppose 2026-07-13 15:26:09 oh, nobody did that yet? 2026-07-13 15:27:29 no 2026-07-13 15:29:51 ok, give me a sec 2026-07-13 15:31:00 fyi: https://gitlab.alpinelinux.org/alpine/alpine-gcc-patches 2026-07-13 15:31:25 although, not sure that's still in use 2026-07-13 15:34:31 from what i can tell there are newer patches in aports that are not in any branch there 2026-07-13 15:35:21 Yeah, and I don't see that repo referenced anymore except in a comment 2026-07-13 15:42:04 https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/105266 fingers crossed 2026-07-13 15:48:07 ah, an existing patch 2026-07-13 15:48:34 Not sure a pkgrel bump is needed 2026-07-13 15:48:49 it already fails on x86_64, and since it does not affect the other arches, no rebuild is required 2026-07-13 15:50:48 hm, want me to try without? i do feel it's kinda safer to rebuild since by definition newer kernel headers will fail the build if installed 2026-07-13 15:52:07 and i'm not sure why other arches would not be failing 2026-07-13 15:58:37 Others probably do not fail due to not having the address sanitizer enabled 2026-07-13 15:59:43 ah, could be 2026-07-13 16:01:03 yeah, it's only enabled on x86_64. that makes me curious but it isn't the rabbit hole i want to jump into now :) 2026-07-13 16:08:32 i think if it passes ci i'll just merge it with the bump, even if that wasn't strictly necessary 2026-07-13 16:09:03 or is that a bad idea for reasons i'm not realizing right now 2026-07-13 16:10:18 It would trigger a rebuild on all arches instead of just x86_64. 2026-07-13 16:10:48 rebuild of just gcc or more? 2026-07-13 16:13:57 i think it's a good thing if gcc is rebuilt when its patches change, even if the patch "should" only affect a subset of architectures. but i might have a different opinion if it causes a ton of rebuilds 2026-07-13 17:18:27 lotheac: it would only rebuild gcc 2026-07-13 17:27:08 that's what i thought. good enough for me :) 2026-07-13 18:12:20 \o/ 2026-07-13 18:57:42 6.6G O_o 2026-07-13 19:00:49 durrendal: what is the status of IA? 2026-07-13 19:05:24 py3-internetarchive just got merged into testing a bit ago, and in my testing all we need to do is bundle the contents of the repository into a zip file and upload it to IA. From there it's usable as is as a repository. So close. 2026-07-13 19:06:56 Ok, nice 2026-07-13 19:07:12 If I make an rsync clone of the repos we're migrating I could then build and upload them. Happy to continue to do that under my own account, but it probably makes more sense for us to have an IA account tied to an @alpinelinux.org account 2026-07-13 19:07:24 yes, makes sense 2026-07-13 19:07:56 I'll get everything into a repo and share it with you, I think it currently all lives in an incus container in my homelab 2026-07-13 19:08:01 right 2026-07-13 19:08:28 I suppose it's something we occasionally invoke 2026-07-13 19:10:00 I think we would also need to create a post on www.a.o to anounce this 2026-07-13 19:10:48 yeah rarely, as we choose to release support, unlike Arch's implementation this is just a frozen repo so it's much simpler. 2026-07-13 19:11:18 happy to take a crack at an announcement post, I actually wanted to write something about it for my blog too since it was a fun problem to solve :) 2026-07-13 19:17:12 what are you doing? archiving entire alpine releases in the internet archive? 2026-07-13 19:17:19 (i can also just wait for your post) 2026-07-13 19:17:32 Habbie: yes, that's the idea 2026-07-13 19:17:35 very nice 2026-07-13 19:17:56 what scale? bunch of images? all main aports for a few arches? 2026-07-13 19:18:18 Entire branches 2026-07-13 19:18:40 oh, just git stuff? 2026-07-13 19:18:48 No, repo branches 2026-07-13 19:18:53 ah 2026-07-13 19:18:56 v3.1-v3.13 initially 2026-07-13 19:19:21 We can try to add redirects so that existing clients / links may not even break 2026-07-13 19:19:27 ohh. old stuff we don't want to keep carrying around ourselves 2026-07-13 19:19:31 exactly 2026-07-13 19:19:35 perfect 2026-07-13 19:24:08 Yeah the IA was very accomodating of the idea actually. 2026-07-13 19:25:25 interesting. Testing out rdfind on a mirror, and it seems to be able to hardlink quite some files 2026-07-13 19:25:35 (DRYRUN MODE) Making 73685 links. 2026-07-13 19:25:39 Didn't expect that 2026-07-13 19:26:13 wow 2026-07-13 19:26:29 seems to be quite some overlap between 3.24 and edge 2026-07-13 19:26:58 which seems to imply that the files are reproducable 2026-07-13 19:28:58 This is between 3.23, 3.24 and edge 2026-07-13 19:32:13 probably just static files? 2026-07-13 19:32:32 probably, yes 2026-07-13 19:32:34 like I imagine that -doc or something like -py would be identical between arch or versions 2026-07-13 19:32:44 although not exclusively 2026-07-13 19:32:51 edge/community/x86_64/gitea-1.25.4-r3.apk to v3.24/community/x86_64/gitea-1.25.4-r3.apk 2026-07-13 19:33:01 edge/main/loongarch64/mariadb-11.8.8-r0.apk to v3.24/main/loongarch64/mariadb-11.8.8-r0.apk 2026-07-13 19:33:26 go produces same bitcode and doesn't rely on libc at all so that's not surprising 2026-07-13 19:33:33 mariadb as wlel 2026-07-13 19:33:45 that seems like same version 2026-07-13 19:33:50 yeah, probably a meta package 2026-07-13 19:34:23 it's nice to see that some stuff is reproducible 2026-07-13 19:34:25 unless it's empty 2026-07-13 19:35:49 mariadb is actual package 2026-07-13 19:35:59 with whole sw inside 2026-07-13 19:36:53 I wanted to revive alpine in reproducible builds but got no answer ever 2026-07-13 19:38:32 yeah reproducible builds would be quite cool. In due time, once we solve for our space issues there'll be a little more breathing room, hopefully 2026-07-13 20:40:27 gained about 35G through hardlinking 2026-07-13 20:40:35 between 3 releases 2026-07-13 20:51:08 czkawka_cli serves me well for just throwing it at directories with data and hardlinking 2026-07-13 20:51:55 especially since it keeps cache/history of hashed files 2026-07-13 20:52:10 and uses blake3 by default 2026-07-13 20:52:23 never heard of that before 2026-07-13 20:52:44 https://github.com/qarmin/czkawka/ 2026-07-13 20:52:49 it started as gui tool 2026-07-13 20:53:02 now it has multiple frontends like cli, qt, tauri, etc. 2026-07-13 20:53:59 s/qt/gtk/ 2026-07-14 03:43:33 why 2026-07-15 08:48:39 I have moved build-edge-riscv64 back to the pioneer box. Hopefully it will speed up the riscv64 builds a bit 2026-07-15 09:22:30 \o/ 2026-07-15 09:22:42 And hopefully it will not lock up too much 2026-07-15 12:49:34 Could this file be deleted? https://distfiles.alpinelinux.org/distfiles/edge/ocaml-containers-3.7.tar.gz 2026-07-15 12:50:06 There has been hacky workaround in community/ocaml-containers but I would line to get rid of it. 2026-07-15 12:50:23 *due to checksum mismatch 2026-07-15 13:20:18 Renamed it 2026-07-15 14:36:21 ikke: https://krei.lambdacreate.com/Alpine/arkivi <- I have the ia tooling together 2026-07-15 14:36:31 and a partial test collection here -> https://archive.org/details/alpine-linux-v3.0 2026-07-15 14:38:31 You can pull files directly out of the zip files inside the ia collection, so a simple functionality test looks like: curl -sL -o /tmp/APKINDEX.tar.gz 'https://archive.org/download/alpine-linux-v3.0/v3.0-main-x86_64.zip/main/x86_64/APKINDEX.tar.gz' 2026-07-15 14:38:42 lambdacreate:~$ tar tzf /tmp/APKINDEX.tar.gz 2026-07-15 14:38:44 .SIGN.RSA.alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub 2026-07-15 14:38:46 DESCRIPTION 2026-07-15 14:38:48 APKINDEX 2026-07-15 14:39:08 entirely transparent to the user aside from the funky link essentially 2026-07-15 15:07:21 I still need to spin up a v3.0 VM and validate I can pull from the apk repo, but the fact that I can fetch the index is a good indication it should just work like it did with my manual tests. 2026-07-15 15:07:45 durrendal: thanks! I'll look at it when I have some time 2026-07-15 15:08:06 where / how do we create an IA account? 2026-07-15 15:08:26 https://archive.org/signup 2026-07-15 15:08:40 I just went through the signup page, that's what Jason directed me to do. 2026-07-15 15:09:34 ok 2026-07-15 15:10:41 once you have an account created apk add py3-internetarchive and then run ia configure. 2026-07-15 15:11:48 it'll run you through a login process using the account credentials and stash a token in ~/.config/ia.ini (I think, checking that path on that, set it up months ago) 2026-07-15 15:15:05 it's actually ~/.config/internetarchive/ia.ini, but yeah it drops an pair of s3 access keys and a session cookie into that, after which arkivi can use it as well 2026-07-15 15:16:09 just let me know when you get some time to look at it, happy to answer questions/help etc. 2026-07-15 15:17:03 for your situational awareness I'll be on a work trip tomorrow and the day after, likely very slow to respond during that period. 2026-07-15 15:17:27 Ok, I'll try to see how far I get. I'll probably have time to look at it tomorrow 2026-07-15 15:19:38 sounds good, if you /msg me or tag me I'll get a notification on my pebble and will try to step aside when I get a chance. 2026-07-15 16:02:42 durrendal: The archive repo works for installing packages. Is it viable to have the URL end in /v3.0/main/x86_64/APKINDEX.tar.gz to allow it being used like other mirrors? 2026-07-15 16:05:08 so yes and no. If we produced a single zip file named v3.0 that contained all subdirectories then that would work. 2026-07-15 16:06:17 but we have to work within the limitations of what the internet archive provides, so any one of those zip files has a size and file limit we'd hit if we lumped everything together. splitting them across multiple files and collections lets us work within those limits 2026-07-15 16:07:08 long term we'd need to implement a 301 or similar redirect from our endpoints to the path in the collection, for hte user it'll be mostly transparent 2026-07-15 16:07:17 You mean that /v3.0/main couldn't start with the same url as /v3.1/main? 2026-07-15 16:07:56 (that would be fine I think) 2026-07-15 16:08:41 precisely, we'd hit the max collection limit rapidly if we had a single alpine-linx-repo collection instead of per frozen version ones. 2026-07-15 16:09:27 long term a call to https://dl-cdn.alpinelinux.org/v3.0/main/x86_64/APKINDEX.tar.gz 301 redirects to https://archive.org/download/alpine-linux-v3.0/v3.0-main-x86_64.zip/main/x86_64/APKINDEX.tar.gz directly 2026-07-15 16:09:53 so you could point it right at IA if you wanted to, but anyone who's running 3.0 in 2026 just runs it as normal :) 2026-07-15 16:10:28 also thank you for testing! 2026-07-15 16:10:40 Do you know if putting /v3.0/main and /v3.0/community in the same zip file is viable? 2026-07-15 16:11:44 entirely depends on the size of the repository, for these older branches yes, it technically wouldn't matter. 2026-07-15 16:12:06 https://lambdacreate.com/paste/alpine-mirror-analysis.txt <- once you get to about v3.6 it would start to though. 2026-07-15 16:13:52 there's a couple of different problems we have to work around here though, the IA collection limit is one, but the lack of total available space to produce archives non-destructively is the bigger one 2026-07-15 16:14:43 by sharding the collection across multiple discrete zips we can build, upload, validate, then reclaim space in a loop that only requires ~6gb of working space to start with. Technically less since the smallest v3.0 directory is about 1-2gb 2026-07-15 16:15:22 I do have some hosts which have spare disk space, so we could use those for mirroring it to IA 2026-07-15 16:16:00 I kind of figured we could, or I could lend some, the v3.0 collection I shared is based off an rsync clone I made locally 2026-07-15 16:21:59 ikke: for what it's worth there's a --reclaim function in arkivi that can be used to remove the files used to build the zip that gets uploaded to IA https://krei.lambdacreate.com/Alpine/arkivi/src/branch/trunk/src/arkivi/core.py#L382 2026-07-15 16:22:31 https://krei.lambdacreate.com/Alpine/arkivi/src/branch/trunk/src/arkivi/cli.py#L255 <- which is only invoked if you pass --reclaim and it passes a set of criteria 2026-07-15 16:24:40 I know you haven't gotten a chance to look, but mentioning it while it's fresh. If you rsync v3.0 or similar to a separate box you can run through the full process including the --reclaim to see what would happen before ever touching the primary mirrors (which I'm positive is what you already planned to do) 2026-07-15 16:50:29 durrendal: packages with a + in the URL are not installable (it seems to be converted to spaces), it works when percent encoded but apk-tools doesn't do that. 2026-07-15 17:05:10 Sertonix[m]: good finding, do you have a package or link I can reference? 2026-07-15 17:06:09 wait c++ is an obvious choice isn't it 2026-07-15 17:06:39 https://archive.org/download/alpine-linux-v3.0/v3.0-main-x86_64.zip/main/x86_64/g++-4.8.2-r10.apk 2026-07-15 17:07:28 https://archive.org/download/alpine-linux-v3.0/v3.0-main-x86_64.zip/main%2Fx86_64%2Fg%2B%2B-4.8.2-r10.apk <- yeah the link on IAs end escapes it. really good find 2026-07-16 09:55:18 Gitlab is 500'ing for me 2026-07-16 11:31:25 seems to be working for me 2026-07-16 11:31:27 URL? 2026-07-16 11:33:38 Oh, that was 1.5h ago 2026-07-16 11:33:58 But I don't see anything particular in our monitoring 2026-07-16 11:36:44 Working on upgrading gitlab today 2026-07-16 11:37:19 Pushed 19.0.4 to https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/tree/19.0-stable yesterday 2026-07-16 13:20:37 Ah yeah, this upgrade also requires us to update to a new version of postgres 2026-07-16 13:20:53 https://gitlab.alpinelinux.org/alpine/infra/compose/pg-upgrade 2026-07-16 13:31:25 https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab-pg-upgrade 2026-07-16 13:40:06 Upgrading postgres on the gitlab-test environment 2026-07-16 20:56:12 i wonder if can do something with ght slow gitlab instance. I wonder if we should ask gitlab.com if they could host alpine for us 2026-07-16 21:00:44 pmos migrated away from gitlab.com due to limitations 2026-07-16 21:04:30 the main issue we had was that accounts require a credit card to be added nowadays iirc 2026-07-17 11:23:28 ugh 2026-07-17 19:27:47 We need more armv7 CI servers 2026-07-18 00:07:39 Uhm, how is irclogs.a.o (or the matrix to irc conversion) made? I noticed that a recent multiline matrix message ended up in a mixed up order at irclogs.a.o 2026-07-18 01:25:52 It would be interesting if matrix users had to see what their multiline messages looked like to irc users. Those links are atrocious 2026-07-18 01:26:19 Separate topic sorry for barging in 2026-07-18 01:27:27 It wasn't a link, this time. 2026-07-18 01:28:34 But yes I wish for the matrix mirror to reject and delete messages (or reactions) that do not translate (well). 2026-07-19 11:31:37 rv64 builder is having issues, waiting for it to be rebooted :( 2026-07-19 11:36:20 Sertonix[m]: irclogs are kept by algitbot 2026-07-19 11:36:32 (sircbot) 2026-07-19 20:10:11 Sertonix[m]: what message is it? 2026-07-19 20:11:12 ncopa: yes we did, gitlab.com is a nightmare 2026-07-19 20:11:41 ncopa: if you want to outsource gitlab to someone else I'd recommend OSUOSL 2026-07-19 20:12:25 they're hosting the pmOS gitlab and have been very professi... oh wait, you already know OSUOSL and how they are ;) 2026-07-19 20:14:10 In #alpine-linux around: 2026-07-17 21:51:37 idk how irc works but here we go 2026-07-19 20:15:44 https://ircb.dersco.re/uploads/funderscore/b70e411a-f_-paste.txt 2026-07-19 20:16:01 seems in order to me on my irc client 2026-07-19 20:16:56 Maybe it was not on whatever server algitbot is connected to 2026-07-20 14:22:49 !105676 2026-07-20 14:23:40 Any specific reason this is mentioned in #-infra? 2026-07-20 14:24:19 was about to say, perhaps the issue is infra related? since tests pass on x86 in CI 2026-07-20 14:25:21 and we'd rather want to find out what is going on than to disable tests 2026-07-20 14:26:03 Guidance on what to check for would be appreciated then 2026-07-20 14:57:15