2026-10-01 12:32:03 Available here 2026-10-02 09:26:28 build-edge-x86 may be stuck 2026-10-02 10:21:35 kicked it 2026-10-02 10:39:16 <3 2026-10-02 16:07:19 could someone abort the rust build on riscv64? 2026-10-02 16:08:03 done 2026-10-02 16:09:30 <3 2026-10-02 16:09:42 please give me a stale mirror... 2026-10-02 16:11:17 https://ftp.lysator.liu.se/pub/alpine/edge/main/ seems to be one.. 2026-10-03 07:38:01 I broke deu7 (our wireguard hub) 2026-10-03 07:38:05 Have to fix it later: ( 2026-10-03 11:12:34 is the CI infra on pause? 2026-10-03 11:51:08 It's fixed again 2026-10-03 11:51:37 for some reason, /dev/null had wrong permissions (0660) 2026-10-03 12:22:58 uhm.. that's odd 2026-10-03 13:49:59 yes 2026-10-03 13:51:10 (after upgrading from 3.23 to 3.24) 2026-10-03 14:01:03 could it be related to #18460 ? 2026-10-03 14:23:14 Not sure 2026-10-03 14:44:17 /dev should be excluded from the apkovl 2026-10-03 14:58:20 Another server where the permissions are incorrect 2026-10-03 14:59:32 At least, where I notice it 2026-10-03 15:42:55 hmm 2026-10-03 15:43:08 :/ 2026-10-03 16:27:33 x86* CI pipelines are struggling? 2026-10-03 16:34:00 I did update the k8s ci nodes 2026-10-03 22:29:46 there appears to be a problem 2026-10-03 22:29:58 also x86 pipelines appear to be stuck 2026-10-03 22:55:46 yes =( 2026-10-03 22:58:24 it doesn't even generate-build-jobs 2026-10-04 06:28:56 ikke: https://tpaste.us/VnmN seems i can't do much without dmvpn :) but those pods are crashing 2026-10-04 06:31:02 charon (one of the dmvpn services) keeps crashing since upgrading 2026-10-04 06:31:42 i guess that would explain it 2026-10-04 06:45:02 konnectivity-agent on nld13-* nodes is failing, it cannot resolve dns 2026-10-04 06:46:13 and runners on those nodes are affected as well 2026-10-04 06:50:23 apiserver<->kubelet connections are supposed to go over dmvpn, so if dmvpn is not operational, that's the root cause 2026-10-04 06:51:37 But it is operational right now 2026-10-04 06:51:59 ok, i don't have visibility into that 2026-10-04 06:52:18 and i can't access any pod logs due to not being on dmvpn myself 2026-10-04 06:54:07 dns traffic should go to the kube-system/coredns service, and calico is responsible for routing that traffic 2026-10-04 06:54:36 could try kubectl -n kube-system rollout restart ds/calico-node 2026-10-04 06:55:50 which is what I just did 2026-10-04 06:56:29 it does a rolling restart one pod at a time, so it will take a bit 2026-10-04 06:56:37 Yup, watching it (using k9s) 2026-10-04 07:00:31 restarting kube-proxy might help too 2026-10-04 07:01:23 The affected nodes are the ones who use wireguard to connect to dmvpn, the other nodes are fine 2026-10-04 07:01:38 So there may be some issue there after rebooting / upgrading the hub 2026-10-04 07:01:43 i see 2026-10-04 07:01:49 the wg links are up and pingable? 2026-10-04 07:02:08 Yes 2026-10-04 07:02:18 I can ping the cplane nodes through the vpn 2026-10-04 07:04:37 i would exec into a pod on an affected node and try to reach a service on the cplane side - for example 10.96.0.10 (ie. coredns), with tcpdump running on the src and dst nodes 2026-10-04 07:04:48 and trying to figure out where the traffic stops 2026-10-04 07:12:04 I have to go, will need to continue later 2026-10-04 07:16:04 me too. later 2026-10-04 10:41:51 darn name system 2026-10-04 11:34:43 lotheac: problem is calico used the public IP addres again instead of internal ip address 2026-10-04 11:50:16 lotheac: https://gitlab.alpinelinux.org/alpine/infra/k8s/ci-cplane-1/-/commit/7d251e36f344d6cf10c9091a4f065d5dedfeb42e 2026-10-04 12:02:29 nice 2026-10-04 12:02:35 `ip -d show dev vxlan.calico` is what showed it 2026-10-04 12:02:47 (the -d flag is necessary to show the local address) 2026-10-04 12:04:52 If you generate wg keypair and share the public key, I can give you dmvpn access: https://gitlab.alpinelinux.org/alpine/infra/infra/-/wikis/Alpine-wireguard-VPN 2026-10-04 12:09:22 thanks. iโ€™ll do that once iโ€™m at keyboard again, maybe about 5h 2026-10-04 12:09:28 ack, no hurry 2026-10-04 14:55:40 pod problems 2026-10-04 15:51:19 .. 2026-10-04 15:53:24 dmvpn is unhealthy :( 2026-10-04 16:01:02 :( 2026-10-04 16:01:23 kunkku: are you able to help? 2026-10-04 16:03:34 parsed IKE_AUTH response 1 [ N(AUTH_FAILED) ] 2026-10-04 16:22:38 Ok, found the issue. The crl could not be updated because of an issue with dmvpn-ca on newer alpine versions 2026-10-04 17:59:26 Going to reboot gbr-app-1, which hosts things like a.o, wiki 2026-10-04 18:02:41 also msg.a.o 2026-10-04 18:25:35 ikke: my wg pubkey is UkrONV8g9Mzy88z2/laJvGbMxTP7yu6K6yeEIN1j/Cc= 2026-10-04 18:28:43 Ok, I assigned 172.16.252.27/32 to you 2026-10-04 18:29:22 You can set `AllowedIPs = 172.16.0.0/16`, and route that prefix to the wg interface to reach everything 2026-10-04 18:30:00 works 2026-10-04 18:30:03 cool 2026-10-04 18:30:20 kubectl logs work too 2026-10-04 18:30:55 thanks 2026-10-04 18:32:04 yw 2026-10-04 20:29:22 ikke: do we need some fixes to the dmvpn package? 2026-10-04 20:59:14 what's this? 2026-10-04 20:59:15 libfakeroot internal error: payload not recognized! 2026-10-04 20:59:49 just saw it skimming through CI logs 2026-10-04 22:27:11 It happens often without breaking stuff, that is for sure. In the code it looks like the fake stat struct reply could not be found. 2026-10-05 05:29:26 kunkku: probably yes. 1) dmvpn-ca it's broken on the Alpine CA on Alpine versions >= 3.21. 2) Charon seems to periodically crash, but I don't have any further details there. 2026-10-05 18:32:29 it is me who is working on nld-bld-1. trying to get a mainline kernel work there. we are making progress. 2026-10-05 18:32:30 nld-bld-1 [~]# uname -a 2026-10-05 18:32:30 Linux nld-bld-1 7.1.5-1-stable #2-Alpine SMP PREEMPT_DYNAMIC 2026-10-05 15:05:48 riscv64 Linux 2026-10-05 18:33:21 we can now also powercycle nld-bld-1 remotely 2026-10-05 19:28:09 ncopa: yeah, I figured 2026-10-06 18:23:41 riscv64 builder/CI seems to have disk space issues 2026-10-06 19:07:20 yeah, that's nld-bld-2 that's being alerted on 2026-10-06 19:33:49 im doing a docker system prune 2026-10-06 19:34:15 Ok, I was also cleaning up the builders 2026-10-06 19:34:23 But it's taking ages 2026-10-06 19:51:24 rebooting deu-dev1, which hosts mirrors, secdb, algitbot, cgit, and some other services. 2026-10-06 19:58:56 it's back 2026-10-07 12:37:13 im working on the rv builders 2026-10-07 19:37:28 could someone add https//znc.in/releases/archive/znc-1.10.3.tar.gz to distfils? the builders are failing because it is not available over IPv4 2026-10-07 19:42:11 OpenSSL SSL_read: OpenSSL/3.5.9: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2026-10-07 19:43:53 it is fetchable over IPv6 2026-10-07 19:44:13 I've mentioned it in #znc@libera 2026-10-07 19:44:42 4+ hours ago 2026-10-07 19:46:32 oh, misunderstood 2026-10-07 19:47:26 done 2026-10-07 19:47:35 <3 2026-10-07 19:54:45 it may also need to be available on 3.24-stable for !109387 2026-10-08 08:05:22 FYI, im updating directory permissions on the d builders 2026-10-08 08:05:44 what is the disk usage status on builders? I'm about to start up the 3.25 builders 2026-10-08 08:36:21 The disks get fuller with each release, requiring more frequent pruning 2026-10-08 09:23:42 could someone kick build-edge-aarch64 and -x86_64? I got a change in earlier that skips the rustnet test that I think is sometimes getting stuck and guessed they got stuck on 2026-10-08 10:25:15 done 2026-10-08 10:27:30 thanks! 2026-10-08 11:17:29 which machines have most problems with disk? maybe we could buy bigger disks? 2026-10-08 11:27:28 ppc64le (usa6-dev1), s390x (usa2-dev1) 2026-10-08 12:17:14 lotheac: I have created an OIDC application in gitlab. 2026-10-08 20:45:59 >>> musl: Fetching https://musl.libc.org/releases/musl-1.2.6.tar.gz 2026-10-08 20:45:59 ssl_client: SSL_connect 2026-10-08 20:45:59 Connecting to musl.libc.org ([2001:19f0:4009:4061:5400:ff:fe11:6da2]:443) 2026-10-08 20:45:59 wget: error getting response: Connection reset by peer 2026-10-08 20:46:18 i dont know if it is musl that is broken or our network 2026-10-08 20:52:59 have you tried -4 2026-10-08 20:53:05 I could download with curl (http2) but it's quite slow (scrapers?) 2026-10-08 20:58:13 i copied edge/musl-1.2.6.tar.gz to v3.25/ on distfiles 2026-10-08 20:58:41 im thinking we should have a smarter distfiles 2026-10-08 21:01:22 we could store files by hash 2026-10-08 21:02:29 s3 storage or something 2026-10-08 23:28:55 hasn't build-3-24-riscv64 been at it for a while now? 2026-10-09 01:56:17 ikke: thanks! what is the client id? 2026-10-09 13:27:09 I have moved build-3-20-riscv64 back from the alpine-p550 (hifive premiere under my desk) to the milk-v pioneer machine, nld-bld-1, since it is stable now 2026-10-09 13:33:35 and started docker there so the nor-ci-3 is back 2026-10-09 15:02:24 i think the arm server has problems with ipv6 2026-10-09 15:02:55 maybe not 2026-10-09 15:03:14 seems to only be site for rsync that was down for a sec there 2026-10-10 06:27:27 build-3-25-riscv64 is online. I moved it to bld-nld-1, the pioneer with 64GB ram. This is so edge and 3.25 are on separate hosts 2026-10-10 06:28:17 I also stopped docker on it so the 3.25 builds gets priority 2026-10-12 20:53:21 I am looking at the /tmp/pytest-of-buildozer issue again and I think it starts 3 pytest invocation (that meet specific criteria, I think) after running the py3-python-discovery test suite due to it creating a directory with mode 0. 2026-10-12 20:55:12 *and only if the test suite happens to convert warnings into errors 2026-10-12 20:57:35 All that to say that it's not caused by the builders; the usage pattern just makes it more frequent there 2026-10-12 22:04:15 I opened https://github.com/pytest-dev/pytest/issues/15169 2026-10-13 05:38:57 Sertonix[m]: thanks 2026-10-13 06:45:08 gitlab ci seems to be having some issues, reporting errors like "ERROR: Preparation failed: starting pod watcher: not synced: *v1.Pod" 2026-10-13 07:19:31 Yep, CI has been borked since last night (europe) at least. I suspect the "Service charon not running" may have something to do with it? 2026-10-13 09:50:38 that helped! 2026-10-13 14:50:16 ugh 2026-10-13 15:22:08 Can https://distfiles.alpinelinux.org/distfiles/v3.24/etckeeper-1.18.23.tar.gz be copied to the v3.25 distfiles? 2026-10-13 15:34:04 done 2026-10-13 15:36:50 Thanks! 2026-10-13 15:54:42 can build-3-25-{x86,x86_64} builds be restarted, please? seems to be stuck 2026-10-13 16:33:03 done 2026-10-13 16:40:52 thanks! 2026-10-13 17:59:21 Does nodejs on riscv64 fail due to OOM or something else? 2026-10-13 18:00:59 Sertonix[m]: OOM 2026-10-13 19:59:02 so 64MB ram may not be enough. maybe we should move it back to the other machine. or get more RAM for it 2026-10-13 19:59:13 I mean GB. now I feel old 2026-10-13 21:13:40 generating CI build jobs issues 2026-10-13 21:25:59 64MB ram ought to be enough for anybody -- bill gates 2026-10-14 03:44:30 looks like something is up with the CI: ERROR: Job failed (system failure): starting pod watcher: not synced: *v1.Pod 2026-10-14 03:50:47 ikke: i'm starting to think we need to run charon under a supervisor that restarts it... or to rebuild the cluster on addressing that is not on dmvpn 2026-10-14 03:51:09 especially as long as you're the only one that can fix it :) 2026-10-14 03:52:20 lotheac: we should investigate why it's crashing in the first place. 2026-10-14 03:52:49 Also, I was expecting k8s to be able to cope with one nod 2026-10-14 03:53:00 control node being unavailable 2026-10-14 03:53:23 the problem is that the lb for the apiserver does not know one of the nodes is network-split from the others 2026-10-14 03:53:58 so you get a like 33% chance of each request going to an apiserver that is not able to do anything 2026-10-14 03:58:14 and sure i'm all for investigating why it crashes, but mitigating the issue in other ways is usually not a bad idea to do first 2026-10-14 04:00:59 Ftr, I was contemplating switching it to using a supervisor, but then the service started to be stable again for a while 2026-10-14 04:01:40 another thing that could help is changing the apiserver lb's health probe to use http /readyz instead of just tcp, that would theoretically remove nodes from the lb targets when they become unhealthy 2026-10-14 04:01:53 But it also masks the issue 2026-10-14 04:02:03 lotheac: yes, indeed 2026-10-14 04:03:23 the problem there is that apparently k0s does not allow GET /readyz without authentication 2026-10-14 04:03:59 curl -kv https://ci-cplane-1-c2.alpinelinux.org:6443/readyz -> 401 2026-10-14 04:32:56 apparently, k0s needs an extra authentication-config file for that. i'll whip something up 2026-10-14 04:34:36 ah, except it won't help because we're on k0s 1.34 and https://github.com/k0sproject/k0s/issues/7961 is not fixed there 2026-10-14 04:35:25 or it is fixed, and i was blind: https://github.com/k0sproject/k0s/releases/tag/v1.34.11%2Bk0s.0 2026-10-14 04:41:53 https://dev.to/azalio/securing-kubernetes-api-server-health-checks-without-anonymous-access-31f9 2026-10-14 04:42:16 yeah, k8s upstream default is allowing anonymous auth, but k0s flips that to false 2026-10-14 04:54:41 urgh... but linode nodebalancers do not allow https protocol checks on tcp backends 2026-10-14 04:55:21 which defeats the entire point 2026-10-14 04:56:53 the lb protocol must stay tcp; tls has to be terminated at the kube components themselves. but there is no configuration for protocol="tcp" in linode_nodebalancer_config that performs https checks. there's check="http" but it requires plain http 2026-10-14 04:59:19 so... no can do :/ 2026-10-14 05:08:33 or i guess we could run some proxy for /readyz only on a different port but urgh 2026-10-14 05:18:21 In the console, I can set the health check protocol separate from the loadbalancer protocol 2026-10-14 05:18:42 but can the health check protocol be https (not http)? 2026-10-14 05:19:45 I suppose not 2026-10-14 05:27:38 i'm starting to think we should've built the cluster network differently. in the absence of that however i suppose we should figure out what's up with charon 2026-10-14 05:28:29 which i can't really help with since i have no account on the nodes in question 2026-10-14 05:47:30 clandmeter: I think we need more RAM in the nld-bld-1. 2026-10-14 10:41:20 lotheac: I've enabled charon logging now on the cplane nodes, see if that turns up anything. I had already enabled it before on deu7, but it remained stable after that 2026-10-14 15:36:40 Nothing in the logs :( 2026-10-14 15:38:28 ncopa: any idea how we can find out why charon is crashing? 2026-10-14 19:26:05 ncopa: did you check the memory on bld2? 2026-10-14 19:26:25 I think i can take out the other stick and run memtest locally in some system 2026-10-14 19:26:35 but I am limited in time :( 2026-10-15 05:25:57 https://tpaste.us/v5jO 2026-10-15 05:29:52 11 is SEGV 2026-10-15 05:30:43 would be nice to get a backtrace of the actual segfault... perhaps easier said than done 2026-10-15 05:34:20 do we have debug build? 2026-10-15 05:37:53 there is a dbg subpackage 2026-10-15 05:42:06 I've enabled coredumps (ulimit -c unlimited) 2026-10-15 05:42:39 it might not dump... since it's actually handling segv 2026-10-15 05:51:09 And backtrace support requires libunwind... 2026-10-15 05:56:33 And libbacktrace, which is not supported on musl 2026-10-15 14:54:37 FYI. I am doing manual "sharcing" of the riscv64 buidls. I have a clone of the build-3-25-riscv64 builder on my spacemit k3 and I build packages there which I then rsync to the milk-v pioneer builder 2026-10-15 15:26:36 can the build-3-25-armhf build be restarted please? seems to be stuck 2026-10-15 17:41:14 I dont have login to ci-cplane-1-c2 2026-10-15 17:53:02 now you do 2026-10-15 18:01:05 No coredumps :-( 2026-10-15 21:07:56 hi, i am looking for a remote admin job 2026-10-15 21:09:09 today i setup a new mirror in ffm ftp.de.tux.re 2026-10-16 04:17:34 ^ was emergency maintenance 2026-10-16 11:22:41 build job generation issues again 2026-10-16 11:56:33 Should be fixed now 2026-10-16 12:02:56 is there anything I can do to fix MRs that don't think there are any changes? 2026-10-16 12:03:08 it happened again with !109889 2026-10-16 12:04:21 amend the commit and force push 2026-10-16 12:06:39 right... 2026-10-16 12:06:57 my force push has been hanging there for a bit now though... 2026-10-16 12:07:33 It's overloaded :-( 2026-10-16 12:08:44 yeah.. =( 2026-10-16 12:08:57 and looks like build jobs still can't be generated 2026-10-16 12:09:37 now with: 2026-10-16 12:09:38 err: failed to prepare repo: failed to fetch 'master' from 'https://gitlab.alpinelinux.org/alpine/aports': unexpected EOF 2026-10-16 13:27:14 that looks like a rate limit 2026-10-16 13:27:37 these days unexpected EOFs often are 2026-10-16 13:27:48 if it's not some ip stack thing. 2026-10-16 13:28:37 (i dunno if the infra has any rate limiting.) 2026-10-16 13:34:21 invoked: I killed some git processes to get the server to quiet down again 2026-10-16 15:19:09 gbr-app-1 will receive emergency maintenance. That's hosting www.a.o among other things 2026-10-16 15:19:11 wiki 2026-10-16 15:19:31 msg.a.o 2026-10-16 15:57:48 I've installed strongswan from edge on the cplane nodes, see if that fixes the stability issues 2026-10-16 17:40:31 I've changed the init.d file to use supervisor-daemon. Currently it's still running without. I want to see if 6.1.0 still crashes as well 2026-10-16 18:04:58 77 2026-10-16 19:00:26 this should fix the riscv64 builders' "lost" status. https://gitlab.alpinelinux.org/alpine/infra/build-server-status/-/merge_requests/33 2026-10-16 19:02:14 Hmm, I explicitly added SetCleanSession(false) to fix an issue 2026-10-16 19:02:55 https://gitlab.alpinelinux.org/alpine/infra/build-server-status/-/commit/a52e9c620d03dfb2465e030dff588d5e2f69f5b8 2026-10-16 19:06:14 Ok, so it now subscribes on connect 2026-10-16 19:06:41 yeah, currently the riscv64 are showing "lost" but the mqtt backend says its online 2026-10-16 19:07:40 i think it lost connection when it switched to "online" again and kept the "lost" status on reconnect 2026-10-16 19:07:45 its weekend. i need to go 2026-10-16 19:07:48 will continue other day 2026-10-16 19:20:37 ok, so far for that theory 2026-10-16 22:42:28 I'm having issues creating merg requests where GitLab says that the source branch does not exist 2026-10-16 22:50:21 ok, not any more 2026-10-16 22:52:21 well, again... 2026-10-16 23:10:00 yeah, impossible 2026-10-16 23:32:22 things are comming back a bit.. 2026-10-16 23:32:36 (hope I don't jinx it!) 2026-10-17 03:00:38 I also encountered an issue when creating a merge request โ€” GitLab said the source branch does not exist 2026-10-17 03:00:48 Also, it seems itโ€™s now returning a 500 directly 2026-10-17 03:01:34 Looks like the machines are reminding us that we should go take a weekend break:-D 2026-10-17 03:12:48 The form contains the following error: 2026-10-17 03:12:48 Source branch does not exist 2026-10-17 03:12:53 i met this 2026-10-17 14:00:29 build-3-25-s390x: crypto/rc4: mkdir /home/buildozer/aports/community/adguardhome/tmp/go-build3820523482/b148/: no space left on device 2026-10-17 14:25:53 Cleaned up some space 2026-10-17 14:27:35 thanks! 2026-10-17 16:05:08 Are the runner issues due to high load? 2026-10-17 18:21:52 jvvv: not that I'm aware of. Anything in particular? 2026-10-17 18:29:02 I'm trying to find a recent failure I can link to. Right now I got two that are pending that previously had the errors I was seeing, so maybe right now the pending jobs are just backed up. Most of them have been like https://gitlab.alpinelinux.org/jvvv/aports/-/jobs/2557148, but that was from last night. 2026-10-17 18:32:45 There are some pending jobs indeewd 2026-10-17 18:34:46 I should have spoke up earlier 2026-10-17 18:36:52 With all the bot scraping, it's hard to know when issues are created by that or not 2026-10-17 18:37:41 possibly, some runners were not able to obtain jobs 2026-10-17 18:38:08 https://gitlab.alpinelinux.org/jvvv/aports/-/jobs/2558429 2026-10-17 18:38:52 That looks like resource exhaustion, which I would likely attribute to bots or the high load caused by them 2026-10-17 18:40:22 Bots don't cause high load on CI 2026-10-17 18:41:12 This is purely a scheduling thing, where we allocate an amount of resources per job, and if the runner that picks it up tries to schedule it somewhere that does not have the resources, it eventually fails 2026-10-17 18:41:34 Normally this should not happen, but there was a job (pod) stuck that didn't get removed, so it kept the resources claimed 2026-10-17 18:42:45 Ah, when you explain it, it makes sense... I couldn't picture that 2026-10-17 19:12:27 ikke: Is that maybe related to abuild CI not picking up any jobs? https://gitlab.alpinelinux.org/sertonix/abuild/-/jobs/2558398 2026-10-17 23:44:21 https://s.lnl.gay/zY5oMN30yRFHrFpJh4cRWtTWDHl0MntL.png 2026-10-17 23:44:52 Agentic software engineering for your team ๐Ÿ‘ 2026-10-18 11:04:22 What was your plan providing http access to the loadbalancer, since the readyz endpoints now require authentication? 2026-10-18 11:04:27 lotheac: ^ 2026-10-18 11:05:07 i had something like this https://tpaste.us/1gNk 2026-10-18 11:05:47 apiserver authentication-config is not compatible with the oidc-* flags though so those would have to move into the config file too 2026-10-18 11:09:09 are you thinking of some non-nodebalancer solution that involves plain http health checks? 2026-10-18 11:09:54 Something in between that checks the readyz enpoint and closes the listener if the check fails 2026-10-18 11:10:03 That would allow us to keep the tcp balancer 2026-10-18 11:11:38 right, right. could work. if it only needs to be hit from localhost, we could also give it a service account that can hit the healthcheck endpoints, instead of allowing anonymous access 2026-10-18 11:11:46 but either one works 2026-10-18 11:11:54 Yeah, was thinking about that as well 2026-10-18 11:12:30 maybe allowing anonymous access is a little bit less likely to break than certificates which can expire 2026-10-18 11:13:40 What about https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens? 2026-10-18 11:15:18 regardless of the type of authentication, i think for a health check of the api server itself, requiring auth adds failure modes that would not exist otherwise 2026-10-18 11:15:53 we can get into situations where all nodes are healthy and everything works but something in the auth token borks and then all of the servers render themselves unavailable 2026-10-18 11:16:02 yup 2026-10-18 14:16:04 seem to be having trouble generating build job /mouse disable 2026-10-18 14:16:13 https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2559793 2026-10-18 14:16:23 yes 2026-10-18 14:16:47 Should be fixed now 2026-10-18 14:16:56 (sorry, please ignore the /mouse part, wrong buffer) 2026-10-18 14:17:00 thanks! 2026-10-18 15:13:35 So the current problem is that even though openrc restarts the charon service, it does not invoke the post_start function, meaning the the config is never loaded (it runs swanctl --load-all) 2026-10-19 00:14:46 are a few of the riscv64 builders taken out for the moment? 2026-10-19 05:13:42 omni: nld-bld-2 is unreachable, need to wait for carlo to see what's going on 2026-10-19 05:40:44 fyi, ncopa pushed a fix to strongswan which should fix the instability we have been facing with charon 2026-10-19 05:45:28 oh awesome 2026-10-19 07:21:12 lotheac: Are you able to test proxyz a bit? I have not tested it against an actual cluster yet 2026-10-19 07:21:19 Otherwise I'll try to test it today 2026-10-19 10:15:35 i don't have time today, sorry 2026-10-19 10:16:27 i pushed the fix to 3.24-stable as well 2026-10-19 10:22:56 ck 2026-10-19 10:22:58 ack 2026-10-19 10:57:56 lotheac: tested in on a local k0s install. Found one small issue, but for the rest it's working :) 2026-10-19 11:14:51 nice! 2026-10-19 11:14:56 well done 2026-10-19 11:15:13 i would have been too lazy to implement that :D 2026-10-19 11:19:37 ofcourse, the proof is in the pudding, we have to verify it's stable as well 2026-10-19 14:08:27 ikke: you can now reboot yourself 2026-10-19 14:08:40 clandmeter: we tried, but it did not boot 2026-10-19 14:08:58 hmm 2026-10-19 14:09:12 ncopa checked the output, and it got stuck somewhere booting the kernel 2026-10-19 14:09:16 oh 2026-10-19 14:09:26 i think we updated bld1 right 2026-10-19 14:09:37 This is bld2 2026-10-19 14:09:39 and i adjusted bld2 2026-10-19 14:09:46 to be ready for new bsp 2026-10-19 14:09:51 could be it boots wrong kernel 2026-10-19 14:10:05 lets try new bsp 2026-10-19 14:13:31 yeah, i think it boots wrong kernel or something similar. we can reboot it and we see linuxboot on serial, but it does not come back up 2026-10-19 14:14:27 is the serial open? 2026-10-19 14:14:43 i may be attached in a tmux session 2026-10-19 14:15:16 doesnt look like it 2026-10-19 14:16:02 nld-bld-1 [~]# tio /dev/ttyUSB0 # connects to BMC 2026-10-19 14:16:29 nld-bld-1 [~]# tio --log --log-file boot.log /dev/ttyUSB1 # connects to serial console 2026-10-19 14:32:19 great 2026-10-19 14:37:11 weird 2026-10-19 14:37:15 both bsp are same 2026-10-19 14:37:17 but one boots 2026-10-19 14:37:23 other does nothing 2026-10-19 14:43:06 w00t 2026-10-19 14:43:31 interesting obervation 2026-10-19 14:43:40 do tell 2026-10-19 14:43:48 the old 2G sd card does not work as BSP 2026-10-19 14:43:57 but it works when booted 2026-10-19 14:44:19 i put the bld1 card in bld2 2026-10-19 14:44:21 and boom 2026-10-19 14:45:10 but now bld1 has no bsp hehe 2026-10-19 14:45:34 as i dont have a card reader/writer 2026-10-19 14:45:44 somebody borrowed my notebook 2026-10-19 14:46:25 and ill be gone this week 2026-10-19 14:46:43 let me use the card reader in bld1 to prepare another bsd 2026-10-19 14:48:39 ncopa: bld1 has too little memory? 2026-10-19 14:50:40 yeah. nodejs OOM'ed 2026-10-19 14:50:54 building nodejs OOM'ed 2026-10-19 14:51:10 it runs lots of stuff in parallel, and ran out of mem 2026-10-19 14:51:27 we could ofc limit the parallelism but that will slow down things 2026-10-19 14:51:43 i dont think its urgent though 2026-10-19 14:54:40 ncopa: slow is better than failing, though :-p 2026-10-19 14:55:11 well, it only failed twice so far and has 7000 packages to build 2026-10-19 15:00:34 rebooting bld2 with new sd 2026-10-19 15:00:59 yup 2026-10-19 15:01:01 that works 2026-10-19 15:01:14 ncopa: did we fix the auto boot issue? 2026-10-19 15:02:24 ok both builders are now on new bsp and stable kernel 2026-10-19 15:03:27 awesome! thanks! 2026-10-19 15:04:23 it seesm it does not auto boot the specified kernel 2026-10-19 15:04:25 after 5 sec 2026-10-19 15:06:55 also looks like efi does not mount anymore 2026-10-19 15:07:05 ncopa: did you add some mount ops? 2026-10-19 15:10:29 just replced it with defaults 2026-10-19 15:40:52 sorry was out 2026-10-19 16:28:25 ncopa: clandmeter ^ 2026-10-19 17:43:26 It was hanging in the bootloader 2026-10-19 19:01:26 Linux nld-bld-2 7.2.9-2-stable #3-Alpine SMP PREEMPT_DYNAMIC 2026-10-06 20:49:35 riscv64 Linux 2026-10-19 19:01:28 \o/ 2026-10-19 19:54:13 lotheac: https://gitlab.alpinelinux.org/alpine/infra/k8s/ci-cplane-1/-/merge_requests/37 should enable anonymous access 2026-10-19 23:13:29 s390x builders are out of space 2026-10-20 01:53:51 ikke: 3.25 loongarch64 builder seems lost. Its last error report in #alpine-commits was already more than 24 hours ago, and it also seems to no longer automatically pull updated aports commits 2026-10-20 05:08:21 huajingyun: thanks. The service was running, but I restarted it and now it is continuing again 2026-10-20 05:35:53 omni: cleaned up some space, but it's dire 2026-10-20 05:43:58 lotheac: k0sctl apply fails for some reason: https://tpaste.us/wvD5 2026-10-20 05:44:40 hmm.. what is it execing 2026-10-20 05:44:46 i'll check it 2026-10-20 05:46:38 it does not say what command 2026-10-20 05:46:50 phase prepare hosts 2026-10-20 05:47:05 i'm trying to run it with --trace but i didn't get very far because i can't access most hosts 2026-10-20 05:47:35 but maybe you can :) 2026-10-20 05:48:13 yes, just did :-) 2026-10-20 05:49:44 Not sure: https://tpaste.us/JLnv 2026-10-20 05:50:19 /run/lock is empty 2026-10-20 05:50:57 can we run execsnoop on one of the hosts while that is happening 2026-10-20 05:51:45 how would one run execsnoop? 2026-10-20 05:51:46 apk add bpftrace-tools && /usr/share/bpftrace/tools/execsnoop.bt 2026-10-20 05:51:49 ah 2026-10-20 05:52:39 although it doesn't print exit statuses so 2026-10-20 05:52:44 still needs some guessing 2026-10-20 05:53:43 ah 2026-10-20 05:53:54 just a sec 2026-10-20 05:53:58 we can whip up a custom thing 2026-10-20 05:54:04 I found it 2026-10-20 05:54:07 It was apk 2026-10-20 05:54:14 oh :) what was it doing 2026-10-20 05:54:16 I left a package with a repo pin in world 2026-10-20 05:54:29 apk add --update findutils coreutils 2026-10-20 05:54:44 today i learned that k0sctl will try to run apk, i guess 2026-10-20 05:55:48 Yeah, it makes sure all dependencies are available 2026-10-20 05:59:17 Works :) 2026-10-20 05:59:43 It's available on :6444 2026-10-20 06:00:02 curl -k https://localhost:6444/readyz 2026-10-20 06:00:46 nice 2026-10-20 06:01:41 wait, but i thought we wanted it to listen on http, not s 2026-10-20 06:01:50 maybe i'm mixing things up 2026-10-20 06:06:07 ah right yeah, i was 2026-10-20 06:06:20 it's just a tcp proxy 2026-10-20 06:15:08 not sure why i wanted to finish writing it, but something like this is a quick hack that prints pid/cmdline for all execs and pid/exitcode for all exits: bpftrace -e 'tracepoint:syscalls:sys_enter_exec* { printf("> %d ", pid); join(args.argv); } tracepoint:syscalls:sys_enter_exit* { printf("< %d %d\n", pid, args->error_code); }' 2026-10-20 06:34:32 Thanks ikke 2026-10-20 09:26:27 hmm... gitlab emails on the !105464 thread are ending up in my spam folder. rspamd reports SPAM_FLAG=5.000 as the omst significant score increase 2026-10-20 09:27:57 it's hard to tell what the original original headers were because the action taken by rspamd is to add spam headers, but something seems off 2026-10-20 10:25:05 lotheac: do you have a message id? 2026-10-20 10:25:53 Looking at our rspamd instance, I don't see anything being flagged as spam 2026-10-20 10:47:02 cloudon infra has low IO performance atm, causes high load 2026-10-20 11:05:50 ikke: both and had the issue 2026-10-20 11:06:22 it's not impossible that it's a "me" problem, but i don't see this on any other legit mail i received today 2026-10-20 11:15:28 lotheac: the score I see from rspamd is 1.54, "no action" 2026-10-20 11:16:24 lotheac: I've added port 6444 now to the balancer as test 2026-10-20 11:16:25 hmm. i wonder what's going on. the path is gitlab.alpinelinux.org -> gbr-app-1.alpinelinux.org -> mail.hacktheplanet.fi with nothing else in between 2026-10-20 11:17:00 all with opportunistic tls too so wth is adding the flag before my rspamd processes it :) 2026-10-20 11:17:11 thanks for checking 2026-10-20 11:18:38 I do see "X-Spam: Yes" as well, but not sure what is adding it 2026-10-20 11:19:15 ok, that explains it 2026-10-20 11:19:24 explains what? 2026-10-20 11:19:49 well, ok, maybe i should first clarify: at which point in the path do you see that? 2026-10-20 11:19:49 I see my own rspamd instance adding it 2026-10-20 11:20:12 ah 2026-10-20 11:20:35 my hypothesis was that my rspamd sees a x-spam flag in the *original* message that it receives, and adds 5 score because of that 2026-10-20 11:20:58 https://tpaste.us/84Ky because of SPAM_FLAG=5.000 2026-10-20 11:21:22 i failed at copypaste, meant to include X-Spam: yes as well 2026-10-20 11:21:40 does your own rspamd report X-Spam-Status? 2026-10-20 11:21:55 No 2026-10-20 11:22:18 But I do see the classification in the webif 2026-10-20 11:22:36 also... Symbol: PHISHING (2.00)[codingstyle.md->alpinelinux.org] *sigh* 2026-10-20 11:22:44 action: add header; score: 7.14/14 2026-10-20 11:22:50 lotheac: yes, I do see that one 2026-10-20 11:23:15 yeah that one is because there is a html link with text "codingstyle.md" which points to domain "alpinelinux.org" instead of "codingstyle.md" 2026-10-20 11:23:36 but if that were the only problem i don't think SPAM_FLAG=5.000 would be there 2026-10-20 11:24:54 maybe i'll just add some logging for a while and look again later 2026-10-20 11:25:11 You don't have the rspamd interface? 2026-10-20 11:25:38 i guess i don't, but what i actually wanted to see was the original mail without the modifications from my local rspamd 2026-10-20 11:25:50 Right, I don't have that 2026-10-20 11:26:05 i've never used the webif, didn't know that existed :D 2026-10-20 11:29:17 anyway... on that other subject: we might be able to do a controlled experiment where we deliberately stop dmvpn on one node and see what happens to the apiserver readyz responses / linode nodebalancer status 2026-10-20 11:52:26 lotheac: I've stopped charon on one node, the proxy detected the readyz returning 500, and linode showed the backend as down :-) 2026-10-20 11:52:40 awesome 2026-10-20 11:54:50 And now it recovered again 2026-10-20 11:57:03 i set up filter-archive on my smtpd to store incoming messages at different points in my filter chain, maybe i can figure out what's going on next time that thing happens 2026-10-20 11:59:36 oh. maybe i already know. i have filter-senderscore -junkBelow 26 and gbr-app-1 has had senderscore: link-connect addr=213.219.36.190 score=26 in my logs 2026-10-20 12:00:02 -junkBelow 30, i meant 2026-10-20 12:00:48 and that's exactly documented to add X-Spam: yes 2026-10-20 12:02:14 so the issue seems to be gbr-app-1.alpinelinux.org's senderscore :\ 2026-10-20 15:28:23 arm server was moved, but its behaving strangely 2026-10-20 15:29:56 nu_: strangely how? 2026-10-20 15:30:49 i think that was a known behavior that its unresponsive for power button press for about 3 minutes after connecting it to power 2026-10-20 15:30:58 ok finally 2026-10-20 15:31:33 at the previous attempt it seemed to have started the boot but there was no VGA output nor reaching boot after ~10 mins 2026-10-20 15:31:56 booted 2026-10-20 15:35:31 What is the IPv4 address? I don't have anything on record 2026-10-20 15:38:04 no public v4, only v6 2026-10-20 15:38:25 It's not responding on ipv6 2026-10-20 15:42:03 opened a pr 2026-10-20 15:42:17 !127 2026-10-20 15:42:30 not that one :p 2026-10-20 15:43:02 thanks 2026-10-20 15:43:45 nu_: You change che-ci-1, don't think that's intended 2026-10-20 15:44:16 aaa 2026-10-20 15:44:48 i dont know what happened 2026-10-20 15:45:08 i pretty sure just changed che-bld-1 in the webui 2026-10-20 15:45:17 now there are many other changes 2026-10-20 15:50:24 nu_: I can make the MR 2026-10-20 15:51:07 please 2026-10-20 15:51:13 I can ping it, but ssh is not responding 2026-10-20 15:51:23 ah, now it is 2026-10-20 15:52:33 should there still be NAT for ipv4? 2026-10-20 15:53:49 it should be the same setting as before, not sure if it works now 2026-10-20 15:54:01 is there connectivity? 2026-10-20 15:54:18 I can ping the gw, but not further 2026-10-20 15:54:26 ok, checking 2026-10-20 15:55:43 now? 2026-10-20 15:56:12 Still not 2026-10-20 16:06:53 I'm AFK 2026-10-20 16:08:46 ok, im testing 2026-10-20 16:21:56 should work now 2026-10-20 16:25:08 the strange MR happened because gitlab automagically tried to use my forked branch as a base from my change but that branch was 108 commits behind 2026-10-20 19:24:15 nu_: Seem to have lost ssh access now 2026-10-20 19:24:19 ping is working 2026-10-20 19:27:26 hmm 2026-10-20 19:28:15 ssh over v6? 2026-10-20 19:28:20 tesa 2026-10-20 19:28:22 yes 2026-10-20 19:28:39 ssh: connect to host che-bld-1.alpinelinux.org port 22: Connection timed out 2026-10-20 19:31:24 back 2026-10-20 19:31:25 fixed 2026-10-20 19:31:47 sorry, the temp location is hacked together 2026-10-20 19:35:48 just to be sure, the temp location doesn't have a larger prefix available, right? 2026-10-20 19:35:59 Right now the containers have a public IP and try to connect via ipv6, which fails 2026-10-20 19:36:41 I stopped the corerad service so that it doesn't give out a router advertisement 2026-10-20 19:37:04 a /64 could be routed 2026-10-20 19:38:19 So I can use 2001:1620:49f4:fa::/64 as prefix? 2026-10-20 19:39:05 try 2001:1620:49f4:f9::/64 2026-10-20 19:39:31 i need 1 sec to conf it 2026-10-20 19:40:03 done 2026-10-20 19:41:44 afk a bit 2026-10-20 19:45:15 che-bld-1 [~]# uptime 2026-10-20 19:45:15 22:07:32 up 4:34, 0 users, load average: 0.18, 0.08, 0.04 2026-10-20 19:45:19 im in 2026-10-20 19:45:24 awesome! 2026-10-20 19:46:53 yes 2026-10-20 19:48:22 ncopa: I stopped the CoreRAD service to prevent the containers from getting an ipv6 address, which is not working yet 2026-10-20 19:49:08 ok 2026-10-20 19:49:15 'I am trying to strat containers 2026-10-20 19:49:16 but 2026-10-20 19:49:22 need to set the clock first 2026-10-20 19:49:46 im stopping all containers 2026-10-20 19:50:33 I fixed the clock 2026-10-20 19:50:46 chronyc makestep 2026-10-20 19:51:31 thank you sir 2026-10-20 19:54:36 i am strating the containers 2026-10-20 19:55:21 They're on the loose 2026-10-20 19:56:12 nu_: ftr, I'm not able to reach the prefix yet. I've added 2001:1620:49f4:f9::/64 to a bridge (similar to the previous prefix), but I can't ping it yet 2026-10-20 20:06:18 i slightly worried the gitlab instance will not handle the extra load due to people being interested in status 2026-10-20 20:06:57 do we have any graphs on load, requests, etc? 2026-10-20 20:28:28 No space left on device. 2026-10-20 20:28:31 great 2026-10-20 20:28:39 on the arm machine 2026-10-20 20:29:30 which script do I use, clean-builders or clean-builders-new? 2026-10-20 20:32:08 clean-builders-new 2026-10-20 20:39:35 is gitlab struggling? 2026-10-20 20:41:15 asking because !110139 (through ..43) 2026-10-20 20:53:59 prefix should be properly routed now ikke 2026-10-20 21:02:23 i am starting up the builder containers one by one to not overload the machine 2026-10-20 21:02:37 nu_: thank you very much for you help with this! 2026-10-20 21:03:04 i have stopped the edge and 3.25 builders for now. we need to prio the stable builders for now 2026-10-20 21:11:45 ncopa welcome! sorry for the sudden outage, i hope it wasnt too bad 2026-10-21 04:58:28 things are moving! \o/ 2026-10-21 04:59:34 ah, great 2026-10-21 04:59:39 morning 2026-10-21 09:44:18 maybe we should turn off wiki.alpinelinux.org til php has been fixed 2026-10-21 09:46:10 Was thinking about wiki indeed when reading about the vuln 2026-10-21 09:46:32 let just turn it off for now 2026-10-21 09:46:47 Don't forget the staging / testing instance 2026-10-21 09:47:08 although, that one may only be available internally 2026-10-21 09:47:38 who can help stop the wiki.a.o instance? 2026-10-21 10:25:39 I can 2026-10-21 10:29:17 thank you 2026-10-21 10:38:09 done, I've added a temporary redirect 2026-10-21 10:43:22 Stopped the zabbix web frontend as well 2026-10-21 10:43:32 thank you <3 2026-10-21 12:53:48 another reboot? 2026-10-21 12:55:29 there it is again 2026-10-21 13:30:06 No, nothing from my side 2026-10-21 13:30:20 And I can't check monitoring atm (php vulns :/) 2026-10-21 13:35:12 no internal access from inside network? 2026-10-21 13:35:56 Yes, but requires internal dns, not easy on mobile devices 2026-10-21 13:36:05 ah, rip 2026-10-21 14:31:49 Now I did restart things again 2026-10-21 14:39:59 idk for some reason, i cant reach gitlab at all now 2026-10-21 14:46:44 ugh 2026-10-21 14:55:42 heh now its 404 2026-10-21 14:56:09 i guess youre rebooting it or something like that 2026-10-21 15:03:05 Trying to figure out what is going on 2026-10-21 15:18:29 Things are timing out, but I have no idea why 2026-10-21 15:19:04 achill: I suppose you are directly pushing to git now? 2026-10-21 15:19:08 yes 2026-10-21 15:19:16 i would not mind if we blocked all IP address unless to those who works on the sec fixes at this point 2026-10-21 15:19:17 So at least that's still working 2026-10-21 15:19:19 that seems to work instead of mrs 2026-10-21 15:20:47 ncopa: Need to have those IPs though 2026-10-21 15:23:52 I can share mine, maybe achill, omni, panekj can share those with you as well? 2026-10-21 15:27:14 i have some docker-related security updates on my to-do list, but i don't think those are critical to getting out new releases? 2026-10-21 15:27:23 no 2026-10-21 15:29:40 81.173.233.54 or 2a0a:a54a:69e7:fc:81a2:cf8:6c8c:179d are mine 2026-10-21 15:30:48 ikek mentionend it would also block the CI and stuff 2026-10-21 15:30:52 *ikke 2026-10-21 15:38:25 allright, maybe we should just block it all and then open access to the devs working on it, and open for the CI and builders 2026-10-21 15:39:00 or we point the builders to git.a.o and push directly to there 2026-10-21 15:39:22 if you push directly to gitlab it also works if you wait a few moments 2026-10-21 15:42:05 ok, gitlab seems to be reachable for me again 2026-10-21 15:42:38 php85 upgrade is failing on the builders 2026-10-21 15:43:09 yes 2026-10-21 15:43:16 probably makes sense to disable the specific tests i guess? 2026-10-21 15:43:40 can we try to do it through CI if it is up again? 2026-10-21 15:44:10 what if the failing tests are trying to tell us something 2026-10-21 15:46:22 or we let them complete on one stable branch before we try on another? 2026-10-21 15:46:29 I haven't looked into what is failing yet 2026-10-21 15:46:55 first log I checked just two tests were failing 2026-10-21 15:50:48 Ok, most traffic is now blocked 2026-10-21 15:50:53 Let me know if you need access 2026-10-21 15:53:45 The amount of traffic really has ramped up 2026-10-21 15:54:28 oh looks im also blocked? 2026-10-21 15:54:55 I did add your IPs 2026-10-21 15:59:52 I'm still logged in so I guess you had mine too? 2026-10-21 16:01:09 thank you! 2026-10-21 16:02:07 omni: yes, I grabbed it from gitlab 2026-10-21 16:02:33 ncopa-3-21-x86-64:~/aports/main/linux-lts$ git push 2026-10-21 16:02:33 fatal: Could not read from remote repository. 2026-10-21 16:02:33 ssh: connect to host gitlab.alpinelinux.org port 22: Network unreachable 2026-10-21 16:02:35 hm 2026-10-21 16:03:14 what IP? 2026-10-21 16:03:30 wget -O- icanhazip.net 2026-10-21 16:03:37 .com* 2026-10-21 16:04:03 2001:4646:fb05:0:2d17:d7b8:15f8:9002 2026-10-21 16:04:56 85.167.109.114 2026-10-21 16:07:53 shorter: curl ip.wtf :) 2026-10-21 16:08:05 i like ip.achill.org :p 2026-10-21 16:09:57 i think its my lxc container thats broke i have a workaround 2026-10-21 16:17:20 is 148.100.88.57 allow-listed? its s390x.alpinelinux.org 2026-10-21 16:17:33 yes, it is 2026-10-21 16:17:38 but not sure why it's not working 2026-10-21 16:18:04 its failing to git pull 2026-10-21 16:18:23 the armv7 builder fails as well to grab ca-certificates 2026-10-21 16:18:37 also not syncing https://git.alpinelinux.org/aports/log/?h=3.21-stable 2026-10-21 16:23:59 ikke: did you also keep access for andypost? 2026-10-21 16:24:05 the builders seem not to be able to pull aports 2026-10-21 16:24:12 or at least it looks stuck 2026-10-21 16:25:08 yeah they are not able to pull 2026-10-21 16:28:32 ikke: can we try disable the blocking for 10 mins so builders can pull? 2026-10-21 16:28:48 ok 2026-10-21 16:29:19 Done 2026-10-21 16:31:02 wow 2026-10-21 16:31:08 it just went down immediately 2026-10-21 16:31:19 wtf 2026-10-21 16:31:27 thats crazy 2026-10-21 16:31:36 ok, blocked again 2026-10-21 16:31:39 thanks 2026-10-21 16:31:43 may be intentional 2026-10-21 16:31:51 delay the fixes to get out 2026-10-21 16:31:52 yes 2026-10-21 16:32:18 its seriously timed so right that this might be true 2026-10-21 16:33:02 Trying to figure out why it works for us but not for the builders 2026-10-21 16:33:05 the mechanism is the same 2026-10-21 16:33:13 I have an ipset with allowed addresses / prefixes 2026-10-21 16:33:24 and a single iptables rule to accept it as first rule 2026-10-21 16:33:49 achill: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/110201#note_653680 2026-10-21 16:34:22 ah 2026-10-21 16:37:54 Im taking care of 3.24-stable https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/110201/diffs 2026-10-21 16:38:43 pushed the skip tests on 3.24-stable 2026-10-21 16:39:48 okay 2026-10-21 16:39:49 thx 2026-10-21 16:39:55 andy didn't reply to my DM earlier, perhaps we'll have to communicate with them through gitlab for now? 2026-10-21 16:40:11 would be really good if they'd be here 2026-10-21 16:40:19 the gitlab isnt the most reliable rn :/ 2026-10-21 16:40:48 I agree 2026-10-21 16:41:33 but we'll have to work with what we have 2026-10-21 16:41:52 or perhaps there was some issue with irc->matrix 2026-10-21 16:42:14 hm i could try to reach them matrix->matrix 2026-10-21 16:43:48 ok. let us do the patching work in #alpine-security and the work keeping gitlab runnin in this channel. Sorry that I have been mixing it 2026-10-21 16:44:17 I think we all mix it because we're a bit worked up right now 2026-10-21 16:44:44 achill: ๐Ÿ‘ 2026-10-21 16:45:35 yeah, tell me about it :) 2026-10-21 16:46:04 ikke what can we do to get the builders be able to pull from gitlab? 2026-10-21 16:46:37 is it all builders or just some builders? 2026-10-21 16:47:01 good question 2026-10-21 16:48:40 i think its all builders 2026-10-21 16:49:06 build-3-21-x86_64 uses https://git.alpinelinux.org/aports.git 2026-10-21 16:49:21 and git.alpinelinux.org cannot sync atm 2026-10-21 16:49:29 ah 2026-10-21 16:49:43 so i guess you need to whitelist git.a.o? 2026-10-21 16:49:54 build-3-20-armv7 can pull 2026-10-21 16:50:03 ah, git.a.o 2026-10-21 16:50:21 i was also not able to pull from my lxc container locally 2026-10-21 16:50:24 which is weird 2026-10-21 16:50:37 i was able to pull from the host 2026-10-21 16:50:42 incus container 2026-10-21 16:50:57 achill: I did already 2026-10-21 16:51:41 which is the gitlab host? 2026-10-21 16:51:49 deu2-dev1 port 2222 2026-10-21 16:52:01 How I implemented it: 2026-10-21 16:52:09 2 iptables rules in the FORWARD chain 2026-10-21 16:52:16 first one uses ipset net_allow 2026-10-21 16:52:21 2nd one uses ipset net_block 2026-10-21 16:52:35 first rule -j ACCEPT, 2nd -j DROP 2026-10-21 16:53:40 let me try an explicit rule on the ip address 2026-10-21 16:53:59 ok. im gonna go out for 30 mins meanwhile 2026-10-21 16:54:06 brb 2026-10-21 16:54:11 I have to go for a bit soon as well. 2026-10-21 16:56:34 should we abort the build on build-3-24-riscv64 ? 2026-10-21 16:56:49 as I assume it is building without the two failing tests disabled 2026-10-21 16:58:34 i guess yea? 2026-10-21 16:58:53 i should also get some fresh air 2026-10-21 17:24:05 ikke: can I be allowlisted too? 2026-10-21 17:24:25 wanted to help out a bit with the stable releases 2026-10-21 17:31:21 well I guess it's mostly down to getting php to build at this point.. 2026-10-21 17:36:31 done 2026-10-21 17:37:50 hmm I guess you looked at gitlab to get my IP? 2026-10-21 17:38:04 I was connected to gitlab from both Tor and my home IP (confusingly) 2026-10-21 17:38:11 I will PM you my IP 2026-10-21 17:41:01 ikke: what can it be that is blocking the builders then? 2026-10-21 17:56:41 i think it is ipv4 that fails 2026-10-21 17:56:44 im back 2026-10-21 17:58:24 ipv6 is not blocked atm 2026-10-21 18:02:25 ok, i think pipelines are working now 2026-10-21 18:02:28 are you getting further with git.a.o? 2026-10-21 18:03:18 Thanks ikke, it works now 2026-10-21 18:08:13 git.a.o should work as well now. we need to trigger an event 2026-10-21 18:11:14 is anyone looking into the php test failures? 2026-10-21 18:11:19 oh wrong channel 2026-10-21 18:21:47 do you have to kill the existing git pukks? 2026-10-21 18:22:30 oh nice 3.24 works 2026-10-21 18:24:40 ikke: can you help trigger the correct MQTT event so https://git.alpinelinux.org/aports/ gets synced? 2026-10-21 18:43:07 ikke: https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/#cve-2026-110264---integer-overflow-issue-in-regular-expression-compiler-impacts-gitlab-ceee 2026-10-21 18:43:41 the day cant get any better 2026-10-21 18:43:44 got the notification 2026-10-21 18:43:50 sorry :( 2026-10-21 18:49:47 lol 2026-10-21 18:49:54 sorry my firends 2026-10-21 18:50:04 this is getting ridiculous 2026-10-21 18:51:27 fr 2026-10-21 18:51:57 tomorrow my first lecture is at 12:30 finally good news 2026-10-21 18:55:32 congrats 2026-10-21 18:55:37 thank you for sticking around 2026-10-21 18:55:50 i think the builders struggle with git pull still 2026-10-21 18:56:08 i suspect gitlab does not handle well 9 builders pulling at the same time 2026-10-21 18:56:33 Might be best to move them all to git.a.o to reduce load? 2026-10-21 18:57:05 could be yes 2026-10-21 19:10:29 hi -- i assume you are aware of https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/ ? 2026-10-21 19:10:40 yes 2026-10-21 19:10:47 right. thanks for your work either way <3 2026-10-21 19:19:09 yes, many thanks, as always 2026-10-21 19:25:50 can we purge the cdn cache for https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86/alpine-netboot-3.23.7-x86.tar.gz 2026-10-21 19:26:07 i triggered a rebuild when I didnt need to 2026-10-21 19:26:17 if it's just that file then send it with curl -XPURGE 2026-10-21 19:27:18 to be clear - gitlab is still only allowlisted right? 2026-10-21 19:27:23 yes 2026-10-21 19:27:31 though, ipv6 is allowed 2026-10-21 19:27:34 ack 2026-10-21 19:27:53 yes, it works over ipv6 but not ipv4 from my vps 2026-10-21 19:28:04 the container registry returns 503s over ipv6 right now 2026-10-21 19:28:29 which breaks loongarch64 CI jobs amongst other things 2026-10-21 19:29:18 hmm that's a different issue to the allowlist stuff (as I understand, the blocks are done on iptables side not webserver side) 2026-10-21 19:29:31 yeah it's unreachable over ipv4 as expected :D 2026-10-21 20:13:32 Do I dare to lift the ip ban? 2026-10-21 20:14:19 not yet please 2026-10-21 20:14:22 ok 2026-10-21 20:14:38 can you wait til the 3.24 is released? 2026-10-21 20:14:55 Will probably be tomorrow 2026-10-21 20:15:13 i think thats fine 2026-10-21 20:15:40 looks like the 3.24 release may be delayed 2026-10-21 20:15:51 due to builders chewing on php 2026-10-21 20:16:28 yeah, it will take a while on riscv64 even when ppc64le is sorted 2026-10-21 20:18:13 wait, I thought andypost was here too 2026-10-21 20:18:55 ncopa: they mentioned !109726 in #alpine-commits 2026-10-21 20:42:22 aelin: pulling docker images should work again 2026-10-21 20:43:25 ikke: yup, works again, thank you very much! 2026-10-21 20:47:41 if it's not much trouble, otherwise I can just wait out stuff to be fixed, 91.192.199.55 2026-10-21 20:50:30 added 2026-10-21 20:56:33 durrendal: would you have time to look at the wiki? 2026-10-21 21:28:29 From our monitoring it appears that at the peak, gitlab received ~50r/s, but it probably was even higher 2026-10-21 21:33:01 ikke: yeah happy to, php patches right? 2026-10-21 21:33:20 Yes, but also alpine upgrade 2026-10-21 21:35:51 yeah I can do that, I was waiting until mediawiki got moved to community, but I'll pin it to the version in testing and we can just deal with it all later, after the CDN has space. 2026-10-21 21:36:17 I think I only have access to the test instance currently 2026-10-21 21:36:25 right 2026-10-21 21:38:56 not a problem, just flagging it. I'll get testing done tonight. Heading home from work now and should have time shortly after 2026-10-21 21:47:22 I've added your key to the wiki instance itself as well now: 172.16.6.19 2026-10-21 21:47:51 I've changed the config to show a maintenance page here: /etc/nginx/http.d/wiki.alpinelinux.org.conf 2026-10-21 21:53:01 maybe we should keep the firewall blocked til we have updated gitlab 2026-10-21 22:16:23 whee we just managed to get the 3.24 release out before the arm server disappeared 2026-10-21 22:18:05 ikke: thank you, I'll get both of them taken care of tonight. 2026-10-21 22:18:53 thank you everyone for the hard work done today! this team is amazing! 2026-10-21 23:41:29 sure was a day 2026-10-21 23:42:03 I guess we can drop the gitlab banner again 2026-10-22 00:35:46 looks like i missed all the โ€funโ€. sorry. great job everyone! 2026-10-22 01:07:13 ikke: i'll do apk ugprade on ci-cplane-1 controlplane nodes 2026-10-22 01:08:53 assuming i can find creds that allow me to drain them first :-) 2026-10-22 01:11:05 k0sctl kubeconfig helped 2026-10-22 01:25:53 all three controlplane nodes upgraded & rebooted into new kernel now 2026-10-22 01:26:58 i guess i could technically escalate my privs to access the worker nodes too but maybe i'll not do that just yet :p 2026-10-22 06:41:46 ikke: I was able to get stage updated to v3.24, Mediawiki to 1.46.2 on php83. 2026-10-22 06:42:33 it was more time intensive than I had hoped so I didn't get to the prod server yet, but should be able to get to it tomorrow. about 3a here so I'm off to bed 2026-10-22 06:42:48 Alright, thanks so far! 2026-10-22 06:43:45 no problem :) I think prod will take less time now that I've worked out the kinks on staging. 2026-10-22 06:46:22 durrendal: thanks alot! 2026-10-22 07:41:23 is gitlab coming back on ipv4? 2026-10-22 07:44:54 Eventually, yes 2026-10-22 08:06:44 let me know if anyone needs ipv4 access to gitlab to help with security issues. I can allow-list IP addresses 2026-10-22 08:17:17 if we've got any security issues left that i can help with, sure. 153.165.53.116 2026-10-22 08:18:07 if not then i don't need it : ) 2026-10-22 08:19:16 added it anyway :-) 2026-10-22 08:19:52 thanks 2026-10-22 11:42:08 I have started up build-3-25-a* again. edge has catched up more or less 2026-10-22 11:42:32 didnt start them initially to avoid multiple chromium builds in parallel 2026-10-22 14:08:50 I have been slowly decreasing the amount of blocked IPs, so far so good 2026-10-22 16:11:47 lotheac: me and ikke has discussed a bit what we can do to scale up the gitlab instance and I think it would be nice to move it to kubernetes. it woudl allow us to separate the gitaly storage to dedicated nodes 2026-10-22 16:12:39 lotheac: do you think we should set up a separate cluster for that or we just use the same as the runners? 2026-10-22 16:13:55 we can maybe start with 2 nodes, one for gitaly and one for the rest, and maybe use linodes hosted database? alternatively set up a postgres cluster as well if we want to self host that too 2026-10-22 16:14:41 im thinking we could switch to gitlabs official images so we save ourselvs some work creating images every new release 2026-10-22 16:14:44 i'm not super satisfied with how the cluster network is set up on the runner cluster and there have been some issues... but i don't think we want to have a completely separate cluster either, so probably reasonable to put it in the ci/runner cluster for now 2026-10-22 16:15:15 maybe we should fix the network first? 2026-10-22 16:15:40 i have been thinking about a larger design on how to set up clusters and what kind of networking they should use but it isn't exactly something done in a weekend 2026-10-22 16:15:55 ok 2026-10-22 16:16:04 lotheac: what is your idea regarding network? 2026-10-22 16:17:04 my rough idea was that there would be three kinds of cluster networks, chosen based on the requirements of the cluster 2026-10-22 16:18:07 type 1: nodes all need public ip's, cluster network is overlayed with a wg mesh; cplane nodes' set up manually, cilium wg for worker nodes 2026-10-22 16:18:38 type 2: some nodes behind nat/no public ip; dmvpn or similar as the cluster network, like in ci-cplane-1 2026-10-22 16:19:01 type 3: all nodes in the same cloud provider network - no public addressing needed at all for nodes 2026-10-22 16:20:45 Challenge is that we do not always get the choice. Setting up a new k8s cluster is relatively expensive (3 VMs minimum). 2026-10-22 16:21:03 yeah, i'm aware 2026-10-22 16:21:13 this idea may get refined as i think about it further 2026-10-22 16:21:22 it's not exactly ready for implementation 2026-10-22 16:21:50 One risk is that some nodes us wireguard, which depends on a single hub 2026-10-22 16:21:59 use* 2026-10-22 16:22:00 so... as for the original question, i think it's fine if we use the ci cluster at least for now 2026-10-22 16:22:16 The first thing we need to solve is backups 2026-10-22 16:22:28 The current backup is linode taking a snapshot of the VM 2026-10-22 16:22:39 Once we distribute things, that no longer works 2026-10-22 16:22:45 does gitlab need other kind of storage than pg? 2026-10-22 16:23:01 i guess i should do some research 2026-10-22 16:23:32 pg, redis, disk (or object) storage for artifacts / logs 2026-10-22 16:23:34 storage for repositories 2026-10-22 16:23:56 object storage is usually better for horizontal scaling if we can get it 2026-10-22 16:24:01 Yes 2026-10-22 16:26:40 and probably linode's managed pg is ok for now anyway. we _can_ selfhost both pg (with cnpg) and object storage (with something like garage) but i don't think we need to do everything at once 2026-10-22 16:27:40 gitaly is I think the main bottleneck atm 2026-10-22 16:27:56 $CLIENT had some compliance requirements so i did need to selfhost everything there, but alpine may not need to :p 2026-10-22 16:29:19 i think we're also going to want a metrics and logs platform pretty soon after we put gitlab stuff on kube 2026-10-22 16:29:24 Not sure if just moving gitaly out of the VM is enough to handle the load or if we need a cluster for balancing 2026-10-22 16:31:08 https://docs.gitlab.com/charts/charts/gitlab/praefect/ "This experimental version is not yet suitable for production use" 2026-10-22 16:31:27 i wonder 2026-10-22 16:34:13 https://docs.gitlab.com/administration/reference_architectures/cloud_native/ maybe this though 2026-10-22 16:35:10 has exactly 3 pods for gitaly, i guess it's better than one :p 2026-10-22 16:35:29 But sharded 2026-10-22 16:36:01 which is possibly not that helpful for us (the pod hosting alpine/aports) would still get the bulk of the load 2026-10-22 16:36:10 we could try the cluster one, i just wonder about the experimental label 2026-10-22 16:36:20 ah it's sharded per repo 2026-10-22 16:36:24 well yeah then it won't help 2026-10-22 16:37:22 So they still recommend deploying gitaly on regular VMs in that case 2026-10-22 16:37:48 i'm gonna take a look at the chart tomorrow and what it's eaten 2026-10-22 16:38:02 now is bedtime though 2026-10-22 16:38:08 lotheac: thanks, sleep well 2026-10-22 16:38:21 thanks 2026-10-22 16:38:57 https://gitlab.com/groups/gitlab-org/-/work_items/21774 2026-10-22 16:44:25 I realize that mirror status may not be the highest concern right now, but I am going through the mirrors looking at which ones are not synced up. I was thinking of notifying any that were more than a day out that this a critical time to be in sync. Does this seem appropriate? 2026-10-22 17:09:10 i think sharding per repo may help. workflow is fork -> merge. and we have 1.2k open merge requests. users push to the forks, CI pull from forks. so even if its not optimal, it would still help I believe 2026-10-22 17:10:56 yes, but in the case of all builders pulling from alpine/aports, that's not going to help 2026-10-22 17:18:03 but with all the other offloaded it may handle it? 2026-10-22 17:18:42 perhaps 2026-10-22 17:22:59 Hard to measure before we implement it 2026-10-22 17:24:10 yup 2026-10-22 17:33:27 do we have any kind of metrics about how busy alpine/aports is vs. everyone's forks? (assuming the forks would be sharded differently) 2026-10-22 17:33:44 and is there any control over sharding? 2026-10-22 17:39:12 In the logs, alpine/aports is generally the most referred to project 2026-10-22 17:40:03 probably by several orders or magnitude vs. all the rest combined 2026-10-22 19:19:47 removed the last large-range ip block 2026-10-22 22:29:09 Has there ever been any talk of making the gitlab repositories private (login required)? Then a git.a.o could be the public facing, browsable interface. 2026-10-22 22:29:49 s/the public/the only public/ 2026-10-22 23:00:32 jvvv: well, unfortunately the most recent gitlab vuln was for unauthenticated users able to execute arbitrary code via their bespoke regex parser 2026-10-22 23:00:59 so it's just a lot, these days, no matter what you do 2026-10-22 23:41:02 I'm not talking about insulating gitlab from vulns, but to make it so that it is impossible for llm bot scrapers to scrape the gitlab git repos 2026-10-22 23:44:01 If the only interface the bots have is a separate gitweb or cgit, it would cool down the load on gitlab where the actual work gets done 2026-10-23 00:13:11 jvvv: It would sound good to me but I don't if that is in any way practical. And I think issues/MRs should be public browsable in some way. 2026-10-23 00:34:39 I admit, the practicality of such is debatable, and I agree about visibility of at least issues 2026-10-23 00:35:07 MR's, not so sure, but opinion is rarely on my side 2026-10-23 00:37:00 I guess my point is one that I think all opensource people are going to have to ask themselves... how can we shield ourselves for those who blatantly abuse us 2026-10-23 00:37:31 blatantly and brazenly 2026-10-23 05:45:58 ikke: re: `Setting up a new k8s cluster is relatively expensive (3 VMs minimum)` If it's in a cloud and it's services that don't change super frequently (unlike runners), it's maybe okay to just have 1... If the control plane node reboots or is down for a short amount of time, the pods keep running. The thing you have to worry about is the controller going away for an extended 2026-10-23 05:46:00 time or forever, then frequent etcd backups and config-as-code can get you back up pretty quick... it's not ideal, but an option 2026-10-23 05:48:23 I mean at a large entertainment company I worked for, we had high-availability controllers and still had to restore etcd backups frequently (thanks Rancher) 2026-10-23 05:50:09 iggy: we used to have CI with just one cplane node. it's really bad for availability 2026-10-23 05:50:51 many things end up depending on the apiserver in various ways and a control-plane outage can quickly become a dataplane outage too 2026-10-23 05:51:06 CI is another story... that's frequent changes... but a mini-cluster just running gitlab seems like it'd have relatively infrequent changes 2026-10-23 05:51:39 but I was just throwing it out there 2026-10-23 05:52:47 imho, if you're doing one control plane node, why do kubernetes at all 2026-10-23 05:53:04 my vote is for resetting the aports repo completely 2026-10-23 05:54:40 there's plenty of valid reasons for using k8s with 1 controller, but there are also other options 2026-10-23 05:55:21 i'm curious what those reasons are :) 2026-10-23 05:58:05 being able to use existing application packaging tools (i.e. helm, kustomize, etc) and running multiple worker nodes without dealing with docker's half-baked multi-node support for example 2026-10-23 05:59:43 kind and podman kube play exist for the first part, but ok, fair enough -- if you want multiple worker nodes, sure. i would be wary of that setup though because of the single-node spof there 2026-10-23 06:02:59 it's easy enough to set up in a way that 1 controller doesn't have to be a spof... the pods don't stop running when the controller(s) aren't available. So, worst case scenario, you just can't make changes for a period of time 2026-10-23 06:03:17 that's not my experience with most workloads 2026-10-23 06:03:52 things like leases depend on the apiserver being available all the time 2026-10-23 06:04:06 but clearly you're not a fan, so I'll :homer_disappear: 2026-10-23 06:04:16 sorry :) 2026-10-23 06:09:11 where costs are a constraint, i usually prefer having the cplane nodes run non-cplane workloads instead 2026-10-23 06:09:55 which is a tradeoff in itself and potentially not the best idea, but i still like it better than making the cplane non-HA 2026-10-23 06:13:16 especially for running on bare metal that usually makes sense... a lot of the time bare metal tends to have more capacity than what you need just for the cplane 2026-10-23 06:14:48 lotheac: counterpoint, our nld12 cluster has been running with a single node, quite stable 2026-10-23 06:15:00 singe controlplane node* 2026-10-23 06:15:39 i thought the reason why we made the new cluster was that it wasn't that stable 2026-10-23 06:15:56 but yeah, we moved the failure point from the single node to the underlying network connecting the nodes 2026-10-23 06:16:09 which was maybe not stabler than the single node 2026-10-23 06:16:14 No, not really. I wanted to separate the application cluster from the CI cluster 2026-10-23 06:16:23 ok, fair enough 2026-10-23 06:18:00 I've been running loki and now victoria logs on it 2026-10-23 07:19:39 "Work to take Gitaly Cluster (Praefect) on Kubernetes to general availability is de-prioritized in favor of a new Gitaly storage backend" https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/5488 2026-10-23 07:59:36 this is apparently that new backend https://gitlab.com/groups/gitlab-org/-/work_items/21711 2026-10-23 08:07:01 What's going on with ppc64le builders? 2026-10-23 08:07:51 builders or CI runners? 2026-10-23 08:11:14 Buiders 2026-10-23 08:11:18 CI is fine 2026-10-23 08:12:28 Actually I think just edge 2026-10-23 08:15:05 I forced it into submission 2026-10-23 10:00:21 durrendal: are you still working on wiki.a.o? 2026-10-23 10:01:42 i tried to dig into what the gitaly praefect chart is doing, but got distracted (and also gitlab's helm repositories are a big mess). have to revisit it some other day... likely monday 2026-10-23 10:03:57 i also asked a clanker about what's so special about the gitaly cluster that it won't run on kubernetes, and it quoted a bunch of things at me such as gitaly expecting to upgrade itself by replacing its own binary in-place, which doesn't really fly with containers. didn't have time to verify any of that though, but it didn't sound promising 2026-10-23 10:05:11 and now i gotta go 2026-10-23 10:05:43 have a nice weekend! 2026-10-23 10:12:33 you too! 2026-10-23 10:14:49 i think we need to ask for more disk space for s390x again 2026-10-23 11:31:39 deleting build-3-1*-s390x would free up 189G 2026-10-23 11:32:00 that is build-3-1[0-9]-s390x 2026-10-23 11:32:20 should be enough for getting 3.25 out 2026-10-23 11:33:26 I dont really have 200G spare storage locally conveinently available atm 2026-10-23 11:34:07 ncopa: yeah sorry for the radio silence, also scrambling at work over this. I should be able to finish it today though, I'll keep you posted. 2026-10-23 11:34:19 thank you! 2026-10-23 11:45:08 maybe I should just purchase another nvme for backups 2026-10-23 12:36:06 ikke: I wonder if I can shut down the gitlab instance to resize it to a bigger linode? the test instance appears to be successful, but needed a cold resize 2026-10-23 12:37:10 ncopa: Is the storage space usage of the builders mainly the local repository of all packages? 2026-10-23 12:37:28 + releases 2026-10-23 12:38:17 yes + releases as ikke said 2026-10-23 12:59:08 gitlab is not happy 2026-10-23 13:00:17 i wonder if it would feel better if I powered it off for 30 mins. It looks like it needs a short break 2026-10-23 13:37:48 im shutting down the gitlab server for a while 2026-10-23 13:51:15 lets hope it comes back up 2026-10-23 13:52:13 Wait, why are those unreachable.. 2026-10-23 13:53:03 i hope I resized the correct instance 2026-10-23 13:54:11 deu2-dev1 2026-10-23 13:54:39 ah, charon crashed on gbr-app-1 2026-10-23 13:55:10 maybe needs an update? 2026-10-23 13:55:24 ncopa: yes, just updated it 2026-10-23 13:57:45 it reponds to ping now 2026-10-23 13:59:30 I can access it now 2026-10-23 14:01:37 it is back \o/ 2026-10-24 02:40:57 ikke ncopa the production wiki is fully upgraded and back online! 2026-10-24 02:41:12 https://wiki.alpinelinux.org/wiki/Special:Version 2026-10-24 02:48:22 nice! 2026-10-24 02:57:20 thanks :) and it's using packages instead of the old git submodule install too so hopefully this shouldn't languish ever again 2026-10-24 02:57:57 I just need to open a few MRs now 2026-10-24 05:12:36 thank you! 2026-10-24 07:22:38 durrendal, thank you for upgrading the wiki.a.o. atlast, wiki got proper dark mode support. I see that some templates like Tip, Warning, Note needs to be fixed as they are currently unusable due to background/text color issues in dark mode. 2026-10-24 08:05:03 i also requested for two more changes in the past for wiki..enable numbered headings in our wiki. the current css doesn't 2026-10-24 08:05:06 clearly differentiate between H3,H4,H5 and H6. So it's not easy to 2026-10-24 08:05:19 follow instruction on wiki pages with multiple heading levels. 2026-10-24 08:05:34 Adding extension like ArticleRatings allow wiki users to give rating to wiki pages. This will allow editors to focus on badly rated articles. 2026-10-24 10:51:41 Why is build-3-35-x86_64 still showing the compilation failure of py3-pyzmq from before the upgrade 5 days ago? 2026-10-24 11:33:49 >>> py3-pyzmq: Building community/py3-pyzmq 27.1.0-r1 (using abuild 3.18.0_rc7-r1) started Sun, 18 Oct 2026 14:30:17 +0000 2026-10-24 11:33:57 dont know 2026-10-24 11:46:55 prabu: I just checked, you're right, in dark mode those fields are unreadable. I also didn't forget about your request, but I need to package those extensions first. 2026-10-24 11:47:29 I'll see if I can get both of these fixed this wrekend. 2026-10-24 15:40:24 durrendal, thanks for remembering them. 2026-10-24 15:44:20 ikke, after seeing the effort put by durrendal to update wiki.a.o, i feel bad about badgering you about wiki in the past.. 2026-10-24 15:45:02 No worries 2026-10-26 03:20:08 The 3.25 builders for loongarch64 and x86_64 seem to be stuck or having some other issue, can't see the build progress on build.a.o either 2026-10-26 08:54:48 Will have a look at the 3.25 builders in a couple of hours 2026-10-26 21:41:07 seems like the ppc64le edge builder is stuck 2026-10-26 21:41:17 or at least not listening to MQTT events 2026-10-26 21:53:56 let me check 2026-10-26 21:54:41 i have seen it hit the rate-limit of git pull making it fail, and it does not retry 2026-10-26 21:55:53 whops I forgot to check on the 3.25 builders 2026-10-26 21:56:10 thanks, it's building stuff again now 2026-10-26 22:27:30 ugh 2026-10-26 22:27:44 still having some issues with the gitlab 2026-10-26 22:28:46 maybe it needs another time out, it liked that the other day 2026-10-26 22:46:08 we actually need to lock it down yes 2026-10-26 22:46:41 and i *really* need to go to bed now 2026-10-26 22:47:03 gitlab will be down til tomorrow 2026-10-26 22:47:06 at least 2026-10-26 22:47:14 really sorry for the inconvenience 2026-10-26 22:53:21 makes sense, hopefully it'll stop acting up soon 2026-10-27 05:32:42 good morning. turns out, we need to re-install the gitlab server 2026-10-27 05:43:04 what we didn't understand from https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/ is that it is not enough updating gitlab version. we also need update ruby. https://www.ruby-lang.org/en/news/2026/10/23/regexp-fixes-for-gitlab-cves/ we decided to shut down gitlab til have this sorted out 2026-10-27 07:47:43 ncopa: re-install gitlab? do you mind explaining why upgrade is not enough? 2026-10-27 07:58:07 as a precautionary measure. We have been vulnerable much longer than thought. 2026-10-27 07:58:43 Will write up details but now prio is to get it up again 2026-10-27 08:00:05 thanks 2026-10-27 09:03:55 ncopa: in the meantime, would sending patches to the mailing list be ok? 2026-10-27 09:04:21 hugops btw, that's a very unfun job to do :( 2026-10-27 09:33:16 3.25 builder (loongarch64) source fetch address needs updating from distfiles.a.o/distfiles/v3.24 to v3.25, checked the other arches and all fine, so only loongarch64 needs the update 2026-10-27 09:33:39 ncopa,ikke: sorry to bother you while you're busy, no rush, just a note for whenever you have time 2026-10-27 09:33:42 Thanks for all your work! 2026-10-27 09:34:24 huajingyun: thank you for pointing that out 2026-10-27 09:36:07 updated 2026-10-27 11:04:44 Thanks 2026-10-27 13:17:56 ikke, ncopa: what's going on with the php2shell flooding in #alpine-linux ? 2026-10-27 13:18:44 fabled_: no clue, but oftc just handled it 2026-10-27 13:19:16 I set the channel to moderated and I think they left 2026-10-27 13:20:48 Talking about #_oftc_#alpine-linux:matrix.org ? I'm still receiving tons of spam messages 2026-10-27 13:21:07 the ruby disclosure came out 2 days after the gitlab one. fantastic 2026-10-27 13:24:11 I don't want to jinx it but it seems like it's stopped now. I always wonder what the point of such spam attacks is 2026-10-27 13:24:20 PureTryOut: They switched to random nicks now 2026-10-27 13:24:42 PureTryOut: massive denial of service? Prevent coordination? 2026-10-27 13:24:45 Ah you forbid talking I see 2026-10-27 13:24:53 Yeah I suppose, I wonder what the end game is 2026-10-27 13:24:54 ikke: I was thinking that 2026-10-27 13:24:58 PureTryOut: yeah, that's the only thing that helps 2026-10-27 13:25:09 maybe they are AI bots trying to escape their containers and talk to each other ;) 2026-10-27 13:25:11 Yeah good call 2026-10-27 13:32:38 it'll probably go on a while 2026-10-27 13:32:49 โ€˜ 2026-10-27 13:32:51 (make some coffee) 2026-10-27 13:33:11 be glad it's not all the channels 2026-10-27 13:34:16 It's interesting to see that they're still joining and leaving the channel 2026-10-27 13:34:33 it might be over. i didn't notice whether any ips were repeating 2026-10-27 13:34:53 but some were 2026-10-27 13:35:36 yeah, so, not a botnet i guess. 2026-10-27 14:07:13 lots of fun in #alpine-linux 2026-10-27 14:07:53 someone doesn't like us very much it seems. 2026-10-27 14:08:56 and we are such nice and friendly ppl 2026-10-27 17:08:05 Progress has been 2026-10-27 17:08:23 made with new instance 2026-10-27 17:08:29 But not there yet 2026-10-27 18:58:25 ikke: to be honest it looks like compromised hosts 2026-10-27 20:34:33 ๐Ÿ˜ฎ 2026-10-28 10:59:12 ncopa, ikke: someone reset my gitlab password, was it due to the reinstall, or should I be worried? 2026-10-28 10:59:22 No, expected 2026-10-28 10:59:46 ikke: ok, thanks 2026-10-28 11:00:09 Though, the link won't work yet 2026-10-28 11:12:23 abundance of caution or just a side-effect of upgrade? 2026-10-28 11:14:41 abbundance of caution 2026-10-28 12:29:52 Any chance my home IP could be whitelisted to access Gitlab? 2026-10-28 12:30:09 PureTryOut: It's currently in read-only mode anyway 2026-10-28 12:52:40 PureTryOut: I'm whitelisted but there really is not much to see 2026-10-28 12:54:10 the instance is currently in read-only mode 2026-10-28 12:54:17 so I don' 2026-10-28 12:54:32 so I don't think it will help if you want to push 2026-10-28 12:54:53 yeah just out of curiosity I tried logging in, you can't 2026-10-28 12:55:37 Probably because it cannot write sessions or something like that 2026-10-28 12:56:09 possibly 2026-10-28 12:56:17 it was to be expected anyways 2026-10-28 12:56:55 I guess stuff is currently directly pushed to git.a.o? 2026-10-28 13:01:27 i don think anything is pushed atm 2026-10-28 13:01:55 so it wasn't read-only 3 hours ago? 2026-10-28 13:03:38 Yeah, we switched somewhere this morning 2026-10-28 13:05:30 ack 2026-10-28 14:23:34 currently it's read-only yes but it seems some people were already whitelisted and it would be nice for if/when this happens again ๐Ÿ˜… 2026-10-28 14:24:32 PureTryOut: if you provide the address, I can add it 2026-10-28 15:05:05 82.174.109.54 2026-10-28 15:05:23 added 2026-10-28 15:06:42 Thanks! 2026-10-28 15:07:10 Works, awesome ๐Ÿ‘๏ธ Now I'll patiently wait 2026-10-28 15:17:06 Can I reset my password and such already or will that not work due to being read-only? 2026-10-28 15:17:36 Not yet, once the new instance becomes available 2026-10-28 15:21:09 Ok ๐Ÿ‘๏ธ 2026-10-28 17:29:43 for transparency. durrendal is helping us with the gitlab migration 2026-10-28 18:31:09 gitlab is back up? 2026-10-28 18:36:27 how to add new key to otpclient 2026-10-28 18:45:54 yes, but it's still limited to the allow-list 2026-10-28 18:46:28 we're still working on the last bit of clean up though, more information should be available soon. 2026-10-28 21:05:41 success 2026-10-28 21:07:37 mostly yes, couple of small things broken still, like the mqtt integration. 2026-10-28 21:13:03 looks like mqtt works 2026-10-28 21:14:04 mps: I think he means that push events gitlab are announced 2026-10-28 21:15:02 ah, ok. i misunderstood 2026-10-28 21:15:05 yes sorry, half answering questions probably isn't as helpful as I meant to be. 2026-10-28 21:22:08 https://alpinelinux.org/posts/GitLab-security-incident-CVE-2026-97229.html 2026-10-28 21:22:30 pushing webhooks to mqtt to publish website works at least now 2026-10-28 21:22:38 other things are probably still missing 2026-10-28 21:56:00 good writeup, glad it's been rebuilt. hope you can find some respite 2026-10-28 21:57:24 going to bed now, will continue fixing the rest tomorrow 2026-10-28 21:57:40 i think it is still blocked for everyone except the devs 2026-10-28 21:57:49 we will fix that tomorrow 2026-10-28 22:07:50 invoked: thanks for the kind words. Can agree, I'm glad we rebuilt despite it being the harder option. 2026-10-28 22:15:45 only way to be sure 2026-10-28 22:19:09 Exactly our thinking 2026-10-28 23:00:00 going to guess the jackass who crapflooded our irc channels is probably the same jackass who popped gitlab 2026-10-28 23:01:58 I've seen the same thing happen to a completely different channel, on hackint, so Idk 2026-10-29 10:00:43 oh, also CIs doesn't work 2026-10-29 10:03:12 im on it 2026-10-29 10:04:46 good, thanks 2026-10-29 10:06:53 btw, testing/cotp 1.11.1 works fine with OTP changing account on gitlab.a.o 2026-10-29 10:27:46 pipelines should work 2026-10-29 11:26:13 it works 2026-10-29 12:44:00 i got timeout to reach the gitab. still rebuilding? 2026-10-29 12:52:31 still cleaning up the last bits before we can open it for the rest of the world 2026-10-29 12:54:05 great, looking forward to it! 2026-10-29 14:19:41 im opening gitlab now 2026-10-29 14:20:42 should be open for the world now 2026-10-29 14:24:02 \o/ 2026-10-29 14:26:07 git (over ssh) still doesn't seem to work 2026-10-29 14:26:22 \o/ 2026-10-29 14:26:29 awesome, thank you :) 2026-10-29 14:26:58 \o/ working on my end again :) 2026-10-29 14:27:52 dne: hmm, works for me though 2026-10-29 14:29:01 heads up, it looks like webauthn/passkey auth might be broken still 2026-10-29 14:29:13 I was able to add my yubikey and use it as a second factor, but passkey auth fails 2026-10-29 14:29:15 oh, might just be due to new host keys 2026-10-29 14:30:09 we invalidated all passkeys as part of the migration, so this is a newly registered passkey 2026-10-29 14:31:31 SHA256:0IBWmctjXkWm8rTVLQK+Op0b7XGFRI5pKZInbLNfgKc. 2026-10-29 14:31:31 The fingerprint for the ED25519 key sent by the remote host is 2026-10-29 14:31:41 is that correct? 2026-10-29 14:32:27 Should be correct, yes 2026-10-29 14:32:34 thanks 2026-10-29 14:33:48 works 2026-10-29 14:37:32 this throws me a http status 500: https://gitlab.alpinelinux.org/-/profile/two_factor_auth 2026-10-29 14:38:43 Hmm I'm able to hit that link directly without issue, using firefox atm 2026-10-29 14:39:46 firefox here too 2026-10-29 14:41:50 not sure if it matters, but I used to sign in with my github account (before the rebuild), but now I have set a password on gitlab.a.o 2026-10-29 14:44:53 dne: I changed password, generate new OTP (last evening), and removed ssh fingerprint and acceptet new. all works fine 2026-10-29 14:46:13 can't set 2FA since that page doesn't loadโ€ฆ 2026-10-29 14:50:04 just guessing but maybe you have a previously set-up 2fa method that the page tries to load to show it to you and fails because ? 2026-10-29 14:50:06 dne: does it load for you in incognito? wondering if it's something cached? 2026-10-29 14:50:11 if you login with new password it should work 2026-10-29 14:51:02 lotheac: I doubt that I've set 2FA before since I used to sign in via github 2026-10-29 14:55:59 neither signing in via github nor in a private/incognito browser helped 2026-10-29 14:56:18 i guess i guessed wrong :( 2026-10-29 14:56:52 reference for someone with access to logs, Request ID: 01M642JM51KFA36PQNDQ00F34Y 2026-10-29 14:57:05 (not urgent at all though) 2026-10-29 15:04:32 dne: we'll look into it as soon as we can :) thanks for being patient 2026-10-29 15:05:09 thanks! 2026-10-29 15:50:26 looks like we lost arm server 2026-10-29 15:51:35 CI or builder?' 2026-10-29 15:56:45 builder 2026-10-29 15:57:37 They are still online and building 2026-10-29 16:00:56 dne: can you check again? 2026-10-29 16:01:31 it works now, thanks! 2026-10-29 16:01:53 2fa was disabled, but there was still something set 2026-10-29 16:02:20 aha 2026-10-29 16:06:10 ikke: Thanks for merging the torchbyte mirrors MR 2026-10-29 16:19:15 build-edge-ppc64le seems stuck since last night 2026-10-29 16:28:50 dne: restarted it 2026-10-29 16:28:52 thanks 2026-10-29 16:34:19 mqtt say that build-edge-aarch64 is lost 2026-10-29 16:45:15 i think the arm builder may be down. I havent checked 2026-10-29 16:45:23 ncopa: I have, it's up 2026-10-29 17:01:57 aq, hugo, telegraf and grafana hang on it for 5-6 days 2026-10-29 17:02:09 s/aq/sq/ 2026-10-29 17:03:11 mps: it's being worked on 2026-10-29 17:03:56 i saw MRs on CI 2026-10-29 17:50:44 we are going to lock down the gitlab instance again til we have fixed yet another CVE in gitlab. 2026-10-29 17:53:14 ACTION hugs 2026-10-29 17:54:58 fyi, we have also enabled so new accounts needs to be approved by admins 2026-10-29 17:59:39 lockdown in place 2026-10-29 19:20:59 Removed the lockdown 2026-10-29 19:27:22 thank you! <3 2026-10-29 20:35:28 Thanks and kudos to everyone involved in GitLab incident ๐Ÿค— 2026-10-29 20:36:10 Just to make sure, I assume that the ssh host key change is expected right? 2026-10-29 20:37:55 Ah I see https://alpinelinux.org/posts/GitLab-security-incident-CVE-2026-97229.html mentions that GitLab was fully rebuilt, so that explains it. 2026-10-29 20:42:37 yes it's excepted 2026-10-29 20:43:02 but good sense to double check :) 2026-10-29 20:44:20 https://gitlab.alpinelinux.org/help/instance_configuration#ssh-host-keys-fingerprints 2026-10-29 20:44:24 to verify 2026-10-29 20:44:51 durrendal: Never too cautious I guess :) 2026-10-29 20:44:58 ikke: Right, thanks! 2026-10-29 22:08:29 I was trying to auto cleanup old artifacts but trying to disable "Keep artifacts from most recent successful job" returns an internal server error. On a related note I think it might make sense to force or default to that option being off. 2026-10-29 22:09:07 *on my apk-tools fork gitlab.a.o/sertonix/apk-tools 2026-10-30 07:12:04 morning! right now admins needs to approve new gitlab user accounts. I don't know if we want to keep that? It adds overhead 2026-10-30 07:12:09 maintenance work 2026-10-30 07:19:46 seems reasonable to keep for now imo 2026-10-30 07:19:57 archlinux are also doing something similar 2026-10-30 07:21:21 but i think we need clear criteria on when to approve or not 2026-10-30 07:26:50 it is 1-2 per day, so adds admin work 2026-10-30 07:27:17 algitbot: retry master 2026-10-30 07:27:55 and right now you can bypass it by signing up with gitlab.com or github sign-in 2026-10-30 07:28:22 signups via github are like 4 per day 2026-10-30 07:31:54 algitbot: retry master 2026-10-30 07:34:11 we could also enable the admin approval for new accounts when there are new gitlab security releases, and disable it again after we patched? 2026-10-30 07:37:14 algitbot: throw dice :) 2026-10-30 07:37:22 algitbot: retry master 2026-10-30 07:38:49 ok, lets keep new sign-ups require admin approval 2026-10-30 07:39:49 i think the problem isn't signups in itself, though, right? the issue is that anyone with a fresh account can run CI 2026-10-30 07:39:54 and keep this for(ever) :) 2026-10-30 07:40:05 lotheac: yes 2026-10-30 07:40:19 algitbot: retry master 2026-10-30 07:40:20 but we do have some spam 2026-10-30 07:40:21 so maybe keep the signup-requires-approval thing until we have a better idea on how to limit CI access? 2026-10-30 07:40:35 mps: do you mind to the retries in #alpine-commits ? 2026-10-30 07:41:01 lotheac: yeah I think thats a good idea 2026-10-30 07:41:07 ncopa: hm, maybe good idea 2026-10-30 08:02:47 can someone please review? https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/141 2026-10-30 08:03:20 i would like to refactor and re-design the website. dont know when I 'll get time to do it 2026-10-30 08:03:33 I want move it to hugo 2026-10-30 08:04:48 https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/blob/c0d6d92944c6312688073623cee7fb2343599a9b/posts/GitLab-reopened-after-security-incident.md 2026-10-30 08:05:02 someone give me a thumb up and I'll publish it 2026-10-30 08:07:09 thumbs up, but i did have a minor comment 2026-10-30 08:08:36 looks good to me 2026-10-30 08:15:26 thanks! 2026-10-30 08:20:55 https://wwwtest.alpinelinux.org/posts/GitLab-reopened-after-security-incident.html does not render well, but we'll take it 2026-10-30 08:21:04 will eat breakfast first 2026-10-30 08:28:36 bon apetito 2026-10-30 10:12:33 ^ 2026-10-30 11:29:30 ikke: how do we handle devs getting blocked by go-away? 2026-10-30 11:33:21 ncopa: would be helpful to get the request id to see why 2026-10-30 11:44:36 staceee: can you paste the request id? 2026-10-30 12:06:32 lotheac: limiting ci is a good idea (along with limiting the creation of projects, and etc). layers. unauth rce, auth rce, then ci. 2026-10-30 12:08:38 it'll be more chores but given the number of gitlab vulns over the past year or so, i'm not sure there's a choice 2026-10-30 12:09:32 It'll cause the MR queue to grow even more 2026-10-30 12:15:41 i mean, unless we can work on rebuilding infra services in a fraction of the time, i think those tradeoffs are just unavoidable 2026-10-30 12:16:55 (or switch to hosted services) 2026-10-30 12:27:02 If more hands would help make that less of a burden you have at least a few people who'd like to help. Myself included 2026-10-30 13:40:11 ah right this channel seems better. I don't see any go-away page on browsers 2026-10-30 13:40:31 clearing local storage and cookies does not help 2026-10-30 13:42:59 well.. with another machine this works, I confess am confused 2026-10-30 13:45:51 damn I got it.. 2026-10-30 13:45:59 had an /etc/hosts entry.. 2026-10-30 13:46:31 sorry for the noise, gotta bury myself 2026-10-30 13:51:06 Glad you sorted it out 2026-10-30 15:44:59 23:00 going to guess the jackass who crapflooded our irc channels is probably the same jackass who popped gitlab 2026-10-30 15:45:12 actually no, don't think so, I've seen this happen in other completely unrelated channels 2026-10-30 15:47:25 ncopa, lotheac: regarding CI access, freedesktop had even worse than that (people using CI to mine crypto iirc) and since then new users have restricted access to CI 2026-10-30 15:48:02 people have to open a gitlab issue to get access, a human then looks at it, applies an issue label and a bot kicks in to give perms and close the issue 2026-10-30 15:48:19 I'm mentioning this here in case you want to take inspiration from that? 2026-10-30 15:49:15 https://gitlab.freedesktop.org/freedesktop/freedesktop/-/work_items 2026-10-30 15:49:45 e.g. https://gitlab.freedesktop.org/freedesktop/freedesktop/-/work_items/2400 2026-10-30 18:01:42 I am about to migrate alpine-mksite to hugo now 2026-10-30 18:08:15 good luck 2026-10-30 18:27:01 ncopa: if you need some help lmk ^^ 2026-10-30 18:27:15 I use hugo quite a bunch 2026-10-30 18:31:10 ncopa - any impact to how we handle the cloud image page? 2026-10-30 18:41:47 I still keep the lua scripts for fetching and generating the json/yaml stuff 2026-10-30 18:43:25 i don't think it will make big difference 2026-10-30 18:43:49 hugo reads cloud/releases.yaml as is 2026-10-30 18:44:07 and right now i have kept the push to master and then merge to production 2026-10-30 18:44:47 do you need to change the page itself? 2026-10-30 18:47:02 looks like you dont 2026-10-30 18:48:07 there may eventually need to be adjustments going forward, but i think the biggest switcharoo (support all clouds in each image and providing per format downloads) is in place 2026-10-30 18:48:37 https://wwwtest.alpinelinux.org/ is rendered by hugo 2026-10-30 18:49:38 its weekend here now 2026-10-30 18:49:56 all seems to be functional, thank you 2026-10-30 18:50:12 now go do your weekend. you all have earned it (and then some) 2026-10-30 18:51:38 (meanwhile, i'll get back to tightening up tiny-cloud) 2026-10-31 08:28:00 hi, i can't reset my password. idk what's wrong with me. when i submit the new password, it returns 500 2026-10-31 08:28:01 Request ID: 01M68GZJBDQ7E76PEQ5ZVDGNCA 2026-10-31 08:34:28 algitbot: retry s390x 2026-10-31 08:34:51 algitbot: retry s390 2026-10-31 08:43:48 algitbot: retry master 2026-10-31 08:47:14 yesterday ncopa asked politely to use algitbot to retry build on #alpine-commits 2026-10-31 11:44:05 ACTION still got 500 :( 2026-10-31 11:46:53 qaqland: Will check in a bit 2026-10-31 11:47:35 thanks 2026-10-31 12:21:24 ACTION sti 2026-10-31 12:22:08 qaqland: I see the error, trying to fix it 2026-10-31 12:22:38 soory, the ssh just froze and send the broken message 2026-10-31 12:24:07 Can you try again? 2026-10-31 12:28:12 it works! thx :) 2026-10-31 17:33:52 ncopa: btw, designing websites is my passion (though I'm in no way a professional designer, it's still something I like doing from time to time) so if you need help with that I'd be happy to see what I can do 2026-10-31 17:37:48 I also try to be as universal as possible so my stuff works on as many browsers as possible. I test on firefox but also recently started testing on netsurf (and sometimes dillo, though it's quite limited) too 2026-10-31 17:44:14 f_: so you maybe could help me to rewrite https://arvanta.net/alpine/ to hugo 2026-10-31 17:44:54 oh should be quite easy 2026-10-31 17:45:29 happy to help! 2026-10-31 17:45:39 thank you 2026-10-31 17:45:47 do you have a repository for this 2026-10-31 17:46:12 not git but zola dir 2026-10-31 17:46:36 ah it's zola? 2026-10-31 17:46:42 yes 2026-10-31 17:46:58 alright 2026-10-31 17:47:35 i can put it on web 2026-10-31 17:54:00 f_: https://arvanta.net/zola/ 2026-10-31 18:02:11 mps: you have good stuff there. i haven't read through all, but any that i have read helped me (mainly the qemu related posts) 2026-10-31 18:03:22 jvvv: thanks. pleasure is mine 2026-10-31 18:04:31 hope it will be better if f_ improve it with hugo 2026-10-31 18:04:40 i don't remember sure how i stumbled on the page, might have been web searching for qemu/{arm,riscv64} stuff 2026-10-31 18:05:10 eh, i'm bad writer 2026-10-31 18:05:37 the content was helpful, diction is secondary 2026-10-31 18:05:40 usually write bare minimum 2026-10-31 18:05:49 better than fluff 2026-10-31 18:06:23 i count on the that that reader not lazy to think 2026-10-31 18:07:10 works out, thanks for making it available 2026-10-31 18:08:14 jvvv: to repeat, pleasure is mine