2025-05-01 05:12:58 <cely> Hello, could i be added to the alpinedev group for uploading to dev.a.o/archive? (i gather this is the preferred way of fixing source files not file issues, instead of pointing the source to distfiles.a.o)
2025-05-01 08:35:48 <ncopa> cely: I added you there
2025-05-01 08:55:04 <cely> Thanks :)
2025-05-01 10:42:48 <ncopa> usa-che-1 shared runner is out of disk space https://gitlab.alpinelinux.org/haesbaert/aports/-/jobs/1830933
2025-05-01 10:46:33 <ikke> checkiung
2025-05-01 10:46:35 <ikke> checking
2025-05-01 10:51:05 <ikke> hmm, there is 63G available atm
2025-05-01 12:50:03 <ikke> bootstrapping jdk8 on armhf and aarch64
2025-05-01 12:59:56 <HRio> cc1plus: out of memory allocating... https://build.alpinelinux.org/buildlogs/build-edge-armv7/community/neochat/neochat-25.04.0-r0.log
2025-05-01 13:02:07 <ikke> 32-bits, means it may also have run out of address space
2025-05-01 13:02:22 <HRio> ah OK
2025-05-01 13:03:23 <ikke> The builder has >250G memory
2025-05-01 13:03:27 <HRio> looks like that successful build of nats-server at the same time was not uploaded?
2025-05-01 13:03:41 <ikke> It only uploads once the entire repo finished
2025-05-01 13:03:42 <HRio> https://pkgs.alpinelinux.org/packages?name=nats-server&branch=edge&repo=&arch=&origin=&flagged=&maintainer=
2025-05-01 13:03:51 <HRio> ah explains
2025-05-01 13:04:28 <HRio> do you know why armhf is listed two times in the link?
2025-05-01 13:04:55 <HRio> and with different versions
2025-05-01 13:07:16 <ikke> There were some storage issues, I had to recreate the index. Probably both the old and the new version are in the package repo
2025-05-01 13:07:38 <HRio> ah
2025-05-01 14:19:25 <ikke> bootstrapping openjdk11 on loongarch64
2025-05-01 14:40:45 <cely> Oh wait, i think for openjdk11 you have to bootstrap openjdk11-loongarch
2025-05-01 14:42:00 <cely> Or at least if you didn't bootstrap that, you'll have to bootstrap it also, iirc openjdk11-loongarch is able to build openjdk11 but not vice versa
2025-05-01 14:46:00 <ikke> I suppose we would not get the server variant otherwise:?
2025-05-01 14:46:25 <ikke> It did manage to get built quite quickly
2025-05-01 14:46:56 <cely> I think openjdk11 will just fail to build openjdk11-loongarch, as it lacks some feature
2025-05-01 14:47:18 <ikke> And why is/was openjdk11-loongarch necessary again?
2025-05-01 14:48:19 <cely> Oracle doesn't want to accept the Loongarch patches for the Server variant, and relying on just the Zero variant is very slow
2025-05-01 14:48:53 <ikke> right, what I mentioned above
2025-05-01 14:48:57 <cely> From what i remember, using community/abcl, the difference was between half an hour vs half a day
2025-05-01 14:49:11 <cely> s/using/building/
2025-05-01 14:50:32 <ikke> and would I install -loongarch from edge then instead of -bootstrap
2025-05-01 14:50:34 <ikke> ?
2025-05-01 14:51:43 <cely> Yes, that should do it
2025-05-01 14:51:46 <ikke> ack
2025-05-01 14:52:03 <cely> Thanks
2025-05-01 14:52:39 <cely> Only jdk11 has this issue, others you can bootstrap whichever is convenient, but bootstrapping -loongarch should be faster
2025-05-01 14:57:21 <ikke> Finished
2025-05-01 15:06:59 <cely> Speaking of OpenJDK, could you have a look at the openjdk8-jre-base .apk built for 3.22 x86_64? I'm specifically thinking about the /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so file (this is the path it has in the edge .apk)
2025-05-01 15:08:23 <cely> There's a "libjvm.so: No such file or directory" error in 3.22 Octave build log, that's why i'm asking
2025-05-01 15:10:50 <ikke> file usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
2025-05-01 15:10:55 <ikke> usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fee12924f7c5dccef1515b2ea70412c94fa4e22b, stripped
2025-05-01 15:12:04 <cely> Hmm, ok, not sure why Octave couldn't find it
2025-05-01 15:12:42 <cely> I guess i'll wait for the next retry and see if it still happens
2025-05-01 15:27:55 <cely> Can't reproduce the libjvm.so not found error in edge on the CI
2025-05-01 15:29:00 <cely> and i see the octave APKBUILD has !check for x86 for due to: "x86 libjava.so cannot find libjvm.so"
2025-05-01 15:30:10 <cely> Now i'm wondering if it could be caused by /usr/lib/jvm/default-jvm on the builder pointing to some other version (not java-1.8-openjdk)
2025-05-01 15:31:56 <cely> which could be the case if a higher jdk version was also installed (java-common.trigger just sorts the dirs, and symlinks the highest version available to default-jvm)
2025-05-01 15:37:20 <cely> Hmm, never mind, maybe it will be fine after jdk's higher than 8 are bootstrapped on x86_64
2025-05-01 15:37:50 <ikke> /usr/lib/jvm/default-jvm -> java-8-openjdk
2025-05-01 15:40:22 <cely> Yeah, thanks for looking into it, i'll check Octave again after jdk>8 are available for it to use, i think there's a possibility it will build fine then
2025-05-02 14:30:16 <ikke> Seems like the excessive requests to cgit have finally stopped
2025-05-02 14:32:15 <f_> awesome
2025-05-02 14:33:29 <ikke> I switched from drop (502) to block (403)
2025-05-02 14:44:24 <f_> ikke: yeah generally when those scrapers scrap and then see a 502 they'll just try again later
2025-05-02 14:44:49 <f_> I've heard some AI bots try again when the return code isn't 200
2025-05-02 14:53:36 <invoked> it's interesting to me that none of the countries have prosecuted this (where dos attacks have been litigated)
2025-05-02 15:22:07 <f_> that's because it's AI and AI gets to be immune in the eyes of the law™
2025-05-02 19:12:29 <ikke> bootstrapping ghc on aarch64
2025-05-03 10:57:34 <ikke> bootstrapping openjdk21 on x86_64
2025-05-03 13:44:33 <ptrc> uh, dev.a.o is out of space?
2025-05-03 13:44:45 <ptrc> like, 100% used 0 bytes left
2025-05-03 13:51:20 <ikke> Now there should be plenty of space left
2025-05-03 14:18:40 <ptrc> thanks!
2025-05-05 07:16:13 <mio> ikke: py3-loky might need releasing from edge ppc64le, sorry about that. the package had already rebuilt on other arches, but the tests may be a little fragile
2025-05-05 11:40:56 <ikke> mio: I've killed the build 
2025-05-05 13:59:13 <mio> ikke: thanks!
2025-05-06 13:51:02 <ncopa> im gonna bootstrap openjdk17 on build-3-22-x86_64 now
2025-05-06 15:26:46 <ncopa> someone is sending spam via alpine-devel list https://lists.alpinelinux.org/~alpine/devel/%3C2349dc138a0c4f7cbd8f53ef3aa65ccc%40semperincolumem.com%3E
2025-05-06 15:27:40 <ikke> Yes
2025-05-06 15:28:16 <ncopa> i got a warning from linode that they will block something
2025-05-06 15:34:05 <ikke> Yup, they already have
2025-05-06 15:34:10 <ikke> I need to respond
2025-05-06 16:13:48 <ikke> I responded to them
2025-05-06 16:22:22 <mio> hello, is openjdk21 still bootstrapping on loongarch64?
2025-05-06 16:22:27 <mio> (3.22)
2025-05-06 16:22:44 <ikke> sorry, no
2025-05-06 16:22:52 <ikke> started the builder again
2025-05-06 16:23:10 <mio> great, thanks :)
2025-05-06 20:23:25 <clandmeter> blaboon: ping
2025-05-06 20:23:49 <clandmeter> We have an issue with one of our vps 
2025-05-06 20:24:01 <clandmeter> Port 25 is blocked
2025-05-06 20:24:18 <clandmeter> 24913954
2025-05-06 20:24:26 <clandmeter> Ticket 
2025-05-06 20:45:39 <blaboon> i just removed the outbound block from that instance
2025-05-06 20:46:05 <blaboon> i informed our support staff too, so you'll probably see some followup in the support ticket at some point
2025-05-06 21:17:39 <clandmeter> Thanks
2025-05-07 07:12:48 <ncopa> im bootstrapping openjdk21 on buid-3-22-aarch64
2025-05-08 09:54:20 <fossdd|m> loongarch64 and riscv64 CI fail to resolve x.org: https://gitlab.alpinelinux.org/fossdd/aports/-/pipelines/321644
2025-05-08 10:38:09 <ncopa> on loongarch64:
2025-05-08 10:38:09 <ncopa> >>> pixman: Fetching https://www.x.org/releases/individual/lib/pixman-0.46.0.tar.xz
2025-05-08 10:38:09 <ncopa> wget: bad address 'www.x.org'
2025-05-08 10:50:10 <omni> when, really, x.com is the bad address
2025-05-08 17:05:44 <iggy> ipv4 vs ipv6 thing? www.x.org doesn't have a AAAA record
2025-05-08 17:07:29 <f_> x.com doesn't (need to) exist
2025-05-09 15:12:07 <mio> 3.20 x86_64 ran out of space, `fatal error: error writing to /tmp/ccmmFlpj.s: No space left on device` while building linux-lts
2025-05-09 15:15:33 <mio> build-3-20-x86_64 builder that is
2025-05-09 15:16:27 <mio> ncopa: ^ (sorry, not sure who to notify)
2025-05-09 16:37:37 <ikke> Strange, I've recently expanded the volume 
2025-05-09 16:46:06 <mio> no idea, just reporting the sights ... gcc failing on 3.22 x86_64 with a similar error
2025-05-09 18:21:21 <ncopa> thanks for notifying. i supose it is /tmp on tmpfs
2025-05-09 18:24:17 <ncopa> no. it's not
2025-05-09 18:26:55 <ncopa> ikke: im expanding it
2025-05-09 18:31:15 <ikke> Thanks 
2025-05-09 19:27:16 <mio> ncopa: thanks!
2025-05-09 22:22:50 <sertonix[m]> Is the ppc64le CI down? https://gitlab.alpinelinux.org/sertonix/aports/-/jobs/1845156
2025-05-10 07:48:08 <ncopa> looks like ppc64le CI is down indeed
2025-05-10 08:50:40 <ncopa> im bootstrapping openjdk17 on build-3-22-aarch64
2025-05-10 19:07:12 <ikke> bootstrapping openjdk8 on build-3-22-x86
2025-05-10 19:23:29 <ikke> bootstrapping openjdk21 on build-3-22-s39x
2025-05-11 15:43:18 <ikke> bootstrapping ghc on build-3-22-x86_64
2025-05-11 16:34:45 <ikke> bootstrapping openjdk8 on build-3-22-s90x
2025-05-11 16:42:53 <ikke> bootstrapping openjdk8 on build-3-22-loongarch64
2025-05-11 18:34:03 <ikke> bootstrapping openjdk21 on build-3-22-loongarch64
2025-05-11 18:34:44 <ikke> bootstrapping openjdk11 on build-3-22-aarch64
2025-05-12 05:39:31 <ikke> bootstrappking openjdk17 on build-3-22-s390x
2025-05-12 05:39:40 <ikke> ncopa: can you start ^ again when finished?
2025-05-12 05:47:21 <ikke> n/m, it already finished
2025-05-12 07:30:29 <ncopa> we need to get the ppc64le back online. any idea how? who do we contact?
2025-05-12 09:56:30 <ikke> ncopa: I can open a ticket on the osuosl support page 
2025-05-12 10:18:40 <ncopa> would be great. thanks!
2025-05-12 10:23:55 <ikke> Sent an email to the support addrress
2025-05-12 17:49:42 <ikke> It's back :-)
2025-05-12 17:53:00 <f_> :-)
2025-05-12 18:02:06 <mio> could someone also prod the 3.22 armv7 builder please? thanks
2025-05-12 18:03:50 <ikke> done
2025-05-12 18:04:05 <mio> thanks!
2025-05-13 10:01:05 <Kladky> Repology seems to be failing to update after go-away was deployed on cigt: https://repology.org/repository/alpine_edge
2025-05-13 10:02:03 <Kladky> Also see https://github.com/repology/repology-updater/issues/1495
2025-05-13 10:28:12 <ptrc> armhf/aarch64 edge builders are stuck
2025-05-13 10:29:38 <ncopa> i'll have a look
2025-05-13 10:34:12 <ncopa> re repology and go-away. I wonder if it would make sense to have a static website with the aports tree, that is automatically `git pull`ed from on git push events
2025-05-13 10:34:44 <ncopa> that way we dont waste cgit cpu time to generate the aports tree
2025-05-13 11:12:46 <Kladky> Yeah, that sounds sensible
2025-05-13 13:08:03 <ptrc> we could just set up a small script that listens on mqtt and `wget --mirror`s the actual cgit page, i guess?
2025-05-13 13:10:51 <ncopa> i was thinking a mqtt listener that simply do if [ -d aports ]; then git -C aports pull; else git clone ...; fi
2025-05-13 13:11:11 <ptrc> ah, do they expect just the plaintext files?
2025-05-13 13:11:17 <ncopa> i think so
2025-05-13 13:11:49 <ncopa> https://github.com/repology/repology-updater/blob/master/repos.d/alpine/alpine.yaml#L63
2025-05-13 13:27:37 <ptrc> actually, does repology use a specific user agent?
2025-05-13 13:27:40 <ptrc> we could just whitelist that
2025-05-13 13:27:59 <ptrc> https://github.com/repology/repology-updater/blob/c93777454f35b591822c6cb1161a90a9461ca537/repology/fetchers/http.py#L36
2025-05-13 13:28:00 <ptrc> yeah
2025-05-13 13:28:05 <ptrc> that sounds waaaaay simpler
2025-05-13 13:36:22 <ikke> Nod
2025-05-13 14:17:23 <ncopa> good idea
2025-05-13 14:18:21 <ncopa> ikke: how do we add the user agent to the allowlist?
2025-05-13 14:25:06 <ikke> It's defined in /srv/compose/cgit/config/go-away on deu1-dev1
2025-05-13 14:25:30 <ikke> You could copy and adjust one of the existing desired crawlers / bots
2025-05-13 14:25:43 <ikke> But I can do that in a bit as well 
2025-05-13 14:50:43 <ncopa> I added - 'userAgent.contains("repology-fetcher/0 ")'
2025-05-13 14:51:11 <ncopa> do I need to rebuild or restart the container?
2025-05-13 14:52:04 <clandmeter> why dont they use github repo?
2025-05-13 14:52:47 <ikke> ncopa: you can restart the go-away container 
2025-05-13 14:53:26 <ncopa> clandmeter: excellent question
2025-05-13 14:56:23 <ikke> From the Github issue, it's the link checker that's affected? 
2025-05-13 14:56:30 <ikke> > It's not the updater, it's linkchecker
2025-05-13 15:09:09 <ikke> Apparently there is a different user agent for that
2025-05-13 19:09:04 <ncopa> ikke: I assume you cleaned up the go-away config. Thank you!
2025-05-13 19:10:42 <ikke> ncopa: yes, the link checker was still blocked 
2025-05-13 19:21:22 <ncopa> it works now. https://github.com/repology/repology-updater/issues/1495#issuecomment-2877638457
2025-05-13 20:11:00 <ikke> Great
2025-05-13 20:29:37 <ikke> bootstrappking openjdk11 on build-3-22-x86_64
2025-05-14 05:04:26 <ikke> ncopa: fyi, I've requested an extra ppc64le VM from osuosl which was approved. It can act as a CI host
2025-05-14 05:10:32 <ikke> bootstrappking openjdk11 on build-3-22-s390x
2025-05-14 18:19:51 <ikke> durrendal: We have one server available to setup as a mirror, maybe we can work together on it to automate the process?
2025-05-14 18:56:47 <ikke> bootstrappking openjdk17 on build-3-22-loongarch64
2025-05-14 20:33:47 <ikke> bootstrappking openjdk17 on build-3-22-ppc64le
2025-05-14 22:13:40 <durrendal> ikke: slow reply, hectic day today. I would love to help out though! Do you have a deadline, or specific time you want the server up by?
2025-05-15 16:31:27 <mio> would it be possible to have the tarballs of paper-gtk-theme-2.1.0-r2 and pcc-libs-20230603-r0 copied over to 3.22 distfiles or someplace where the 3.22 builders can access them? the upstream source has been returning 404 for more than 2 weeks, and in the case of paper-gtk-theme the github repo no longer exists
2025-05-15 16:33:47 <ikke> ack
2025-05-15 16:34:06 <ikke> could you give the filenames?
2025-05-15 16:34:30 <mio> https://distfiles.alpinelinux.org/distfiles/v3.21/paper-gtk-theme-2.1.0.tar.gz
2025-05-15 16:34:38 <mio> https://distfiles.alpinelinux.org/distfiles/v3.21/pcc-libs-20230603.tgz
2025-05-15 16:34:45 <mio> thanks!
2025-05-15 16:35:25 <ikke> done
2025-05-15 16:36:04 <mio> thanks, appreciated :)
2025-05-15 17:00:35 <mio> sorry, could you also add pcc as well? just unlocked after pcc-libs rebuilt https://distfiles.alpinelinux.org/distfiles/v3.21/pcc-libs-20230603.tgz
2025-05-15 17:01:00 <mio> from the same upstream as pcc-libs
2025-05-15 17:01:30 <ikke> done
2025-05-15 17:01:52 <mio> thanks!
2025-05-15 17:02:31 <ikke> Maybe would be good to open issues for those packages if upstream is gone
2025-05-15 17:03:34 <mio> sure, on it
2025-05-15 17:15:12 <ikke> durrendal: there is no hard deadline that we know of
2025-05-16 00:17:02 <durrendal> ikke: okay great! Do we have any references on how the existing mirror servers are setup? Would love to read through what we have currently.
2025-05-16 00:18:28 <durrendal> Also let me know what time generally is most convenient for you, I'll try and bend my schedule around it if I can :)
2025-05-16 05:11:12 <ikke> durrendal: https://gitlab.alpinelinux.org/alpine/infra/compose/alpine-mirror-sync is the meat of it, combined with traefik in front
2025-05-16 13:08:03 <durrendal> ikke: thanks a ton! I'll dig through this as soon as I've got some time today. My initial thought just glancing at the compose file is that it should be fairly straight forward to come up with a generic deployment mechanism for anything using docker compose.
2025-05-16 13:08:39 <durrendal> secrets could be handled through ansible-vault initially, though it'd be better to use something like openbao to source those
2025-05-16 13:09:20 <durrendal> I'm certain there's more to the deployment than just pulling in the compose file though
2025-05-16 15:08:28 <ikke> yeah, but just slightly
2025-05-17 21:56:31 <ptrc> > Aparently the production builders have lua-term pre-installed, but its files are either incomplete or in a non-default location. In this way luarocks skips the installation and the compiler doesn't find the files later.
2025-05-17 21:57:16 <ptrc> any idea what could need lua-term on the builders?
2025-05-17 21:57:32 <ptrc> in the dependency graph there's just lua-busted
2025-05-17 21:57:47 <ptrc> but lua-aports doesn't touch either
2025-05-18 03:02:45 <ikke> ptrc: it's not pre-installed
2025-05-18 14:26:16 <ikke> Working on deploying renovate-bot on a kubernetes cluster: https://gitlab.alpinelinux.org/alpine/infra/k8s/cluster-nld12/-/tree/master/deployments/apps/renovate-bot
2025-05-18 14:26:32 <ikke> But deploying it through CI/CD, not manually
2025-05-18 14:27:21 <ikke> And it has been deployed here: https://gitlab.alpinelinux.org/alpine/infra/k8s/cluster-nld12/-/jobs/1857474
2025-05-18 15:51:49 <ncopa> awesome!
2025-05-19 11:54:42 <ncopa> LSP cmd connects to lsp server via rpc.connect using hostname
2025-05-19 11:54:42 <ncopa> test/functional/plugin/lsp_spec.lua:5445: test/functional/plugin/lsp_spec.lua:5452: server must receive `initialize` request
2025-05-19 11:55:09 <ncopa> that error on build-3-22-x86_64 when building neovim happens due to ::1 is missing in /etc/hosts. I have fixed it now
2025-05-19 12:57:36 <ncopa> looks like loongarch64 CI cannot lookup www.x.org? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1858905#L137
2025-05-19 14:00:17 <ncopa> I'm bootstrapping openjdk11 on build-3-22-ppc64le
2025-05-20 16:41:06 <ikke> Yo dawg, I heard you liked kubernetes and renovate, so I put renovate in kubernetes, so that renovate can update renovate running in kubernetes
2025-05-20 16:49:09 <f_|DSR> hah
2025-05-21 13:27:26 <ncopa> OSError: [Errno 28] No space left on device
2025-05-21 13:27:36 <ncopa> How can i clean up the x86_64 ci?
2025-05-21 13:44:13 <ikke> Which runner?
2025-05-21 13:54:30 <ncopa> https://gitlab.alpinelinux.org/admin/runners/235
2025-05-21 13:55:06 <ncopa> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1861967
2025-05-21 14:08:41 <ikke> You can ssh into x86-64.ci.alpinelinux.org
2025-05-21 14:08:53 <ikke> And then run docker system prune -af
2025-05-21 14:23:54 <ncopa> ssh: Could not resolve hostname x86-64.ci.alpinelinux.org: Name does not resolve
2025-05-21 15:09:00 <ikke> sorry, it's with an _
2025-05-21 15:09:04 <ikke> root@x86_64.ci.alpinelinux.org
2025-05-22 04:58:38 <ikke> cz.a.o has 275GB of space left
2025-05-22 14:12:41 <fossdd|m> uh so gitlab is down
2025-05-22 14:16:52 <ikke> oof, checking
2025-05-22 14:17:46 <ikke> apparently it's up again
2025-05-22 14:19:47 <fossdd|m> ah yeah
2025-05-22 14:26:34 <ikke> something caused heavy load
2025-05-22 15:08:40 <iggy> you aren't supposed to use _'s in hostnames
2025-05-22 15:08:52 <ikke> ahuh
2025-05-22 16:38:09 <ncopa> some packages were built with old libprotobuf on the ppc64le builder. (maybe old version was installed by mistake?)
2025-05-22 16:38:17 <ncopa> i have cleaned those up now
2025-05-22 16:39:17 <ikke> ok, thanks
2025-05-22 16:59:21 <ikke> ncopa: just setup a new kubernetes cluster for CI
2025-05-22 16:59:30 <ikke> k0stctl makes that extremely easy :)
2025-05-22 17:03:48 <ncopa> there is also a tool, k0smotron, where you can use kubernertes to manage multiple clusters (eg spin up and take down)
2025-05-22 17:04:10 <ncopa> eg a cluster of clusters
2025-05-22 17:04:40 <ikke> heh
2025-05-22 17:04:42 <ikke> kubeception
2025-05-22 17:04:48 <ncopa> but i think k0smotron may have limits on which infra it supports
2025-05-22 17:05:57 <ncopa> i wanted to run k0s on bare metal, managed by tinkerbell
2025-05-22 17:06:04 <ncopa> but thats a bit offtopic :)
2025-05-22 17:07:54 <ikke> That would be cool
2025-05-22 17:22:04 <ncopa> first architectures are done with 3.22 builds
2025-05-22 17:24:30 <mio> \o/
2025-05-22 17:30:25 <mio> thanks for triaging and fixing the remaining builds!
2025-05-22 17:32:57 <ptrc> \o/
2025-05-23 11:52:53 <ikke> nu_: ^
2025-05-23 17:37:45 <mio> the arm builders are online but seem to be having trouble building, builds fail but no build logs
2025-05-23 17:44:52 <ikke> Apparently no ipv4
2025-05-23 18:04:10 <mio> okay, thanks for checking
2025-05-23 18:37:11 <nu_> checking
2025-05-23 18:40:18 <nu_> should be working now
2025-05-23 18:41:16 <mio> builders are back now, thanks!
2025-05-25 12:47:47 <omni> are all arm* and aarch64 builds on the same machine? I forget
2025-05-25 12:48:33 <omni> nodejs has failed to build a couple of times now on aarch64
2025-05-25 12:48:57 <omni> c++: fatal error: Killed signal terminated program cc1plus
2025-05-25 13:34:42 <ikke> omni: yes, a single host as builder 
2025-05-25 13:37:42 <omni> right, and the builds have passed now
2025-05-25 14:41:17 <mio> could someone check on the 3.22 armv7 and x86_64 builders? they seem to be stuck
2025-05-25 21:01:48 <mio> please disregard previous message, the builders are fine now
2025-05-26 05:02:49 <ikke> mio: I didn't ;-)
2025-05-26 05:03:25 <mio> :)
2025-05-26 09:07:31 <omni> the aarch64 CI builder has storage issues, sometimes runs out of space but I guess is cleaned up after runs?
2025-05-26 10:18:37 <ikke> omni: the builder has 700G+ free
2025-05-26 10:24:31 <omni> huh, ok, that's odd, because several times the past couple of days I've seen aarch64 CI jobs fail due to running out of space
2025-05-26 10:27:45 <ikke> Oh, CI host
2025-05-26 10:27:48 <ikke> that's different
2025-05-26 10:28:25 <ikke> We have multiple CI hosts, so it depends on what runner it takes
2025-05-26 10:39:23 <omni> ah, ok
2025-05-26 10:56:44 <ikke> Should be cleaned up now
2025-05-26 11:12:55 <omni> 👍
2025-05-27 09:36:46 <ncopa> I would like to run `apk dot --errors` and publish the result somewhere visible
2025-05-27 09:37:36 <ncopa> ideally it should be run on builders, or master mirror when new packages are uploaded
2025-05-27 09:42:19 <ncopa> cd 
2025-05-27 09:42:28 <ncopa> https://tpaste.us/kn8Z
2025-05-27 10:38:07 <ncopa> I have opened an issue for the 3.22/main dependency graph: https://gitlab.alpinelinux.org/alpine/aports/-/issues/17191
2025-05-27 10:48:13 <ncopa> sorry that was wrong channel
2025-05-27 11:33:20 <sertonix[m]> ncopa: Be aware that the output of apk dot --errors can (and does) differ between arches
2025-05-27 11:55:29 <ncopa> yeah, i know, i was thinking that every arch should build its own svg graph
2025-05-27 11:55:52 <ncopa> the idea is to make those visible, early
2025-05-27 11:57:35 <ncopa> i wonder if it is possible to filter the apk dot --errors. If we could filter the graph to only include errors were a give list of packages are involved, then we could do it from CI
2025-05-27 11:57:42 <ncopa> after the .apk is generated
2025-05-27 12:00:17 <ncopa> i suppose something like: if apk dot --errors | grep -w  "$pkgname"; then genereate_svg && upload_svg_artifact && exit_with_error; fi
2025-05-28 08:24:03 <lotheac> hi o/ per ncopa's suggestion, i'm volunteering to help out with infra if you guys have anything that needs taking a look at
2025-05-28 08:35:44 <ikke> lotheac: hey, thank you for offering your help. What are you experienced in?
2025-05-28 08:37:21 <lotheac> "everything" :-) i've done sysadmin/sre/etc. stuff as my dayjob since 2008. tcp/ip, observability, kube, etc.
2025-05-28 08:38:01 <lotheac> used to have a hobby OS project https://unleashed.31bits.net/
2025-05-28 08:41:44 <lotheac> most recently at $CLIENT i've redone SSO infra, created mesh-VPN infra (tailscale/headscale), created autoscaling kube clusters & metrics/logging onto them on hetzner cloud
2025-05-28 08:43:32 <lotheac> maybe i should have just linked my cv :)
2025-05-28 08:46:31 <ikke> Heh 
2025-05-28 08:49:05 <lotheac> guess it's not a bad idea anyhow, have some lists of technologies there https://lotheac.fi/cv.html
2025-05-28 10:00:20 <ncopa> looks like the CI is having some sort of problem: https://gitlab.alpinelinux.org/ncopa/alpine-conf/-/jobs/1872144
2025-05-28 10:01:59 <ncopa> passed after 3-4 retries
2025-05-28 10:17:08 <Leso> Hey everyone :)leso-kn here with a quick infra question: I recently submitted the paged-markdown-3-pdf aport which is based on Lua. The Gitlab CI pipeline passed so it was merged, but on the buildozers the build fails due to a missing lua module. I know that there is some lua tooling for the builders, so I was wondering if the required file might have been removed or is installed in a non-standard location?
2025-05-28 10:19:24 <Leso> Apologies for the formatting, just getting started with IRC. The module is lua-term and the missing file which should be provided by it is /usr/share/lua/5.1/term/colors.lua
2025-05-28 10:20:10 <Leso> Apologies for the formatting, just getting started with IRC. The module is lua-term (pre-installed on the builders) and the missing file which should be provided by it is /usr/share/lua/5.1/term/colors.lua
2025-05-28 10:24:51 <ikke> Leso: These are the only lua packages installed on the builders: https://tpaste.us/8B7b
2025-05-28 10:26:00 <ikke> And in CI these: https://tpaste.us/r5Jj
2025-05-28 10:29:59 <Leso> Thanks for the quick reply! Hmm, very interesting, according to the build log the bundler found a pre-installed lua-term v0.7-1 – the version that is packaged for alpine, while the latest available via luarocks would be 0.8-1
2025-05-28 10:31:00 <Leso> > see the require error near the end in https://build.alpinelinux.org/buildlogs/build-edge-aarch64/testing/paged-markdown-3-pdf/paged-markdown-3-pdf-0.1.3-r0.log
2025-05-28 10:40:54 <ikke> I cannot find term/colors.lua anywhere on the builder (not even in the src/.luarocks dir)
2025-05-28 10:42:02 <ikke> So not sure why luarocks says 0.7-1 is installed
2025-05-28 10:47:36 <Leso> Exactly, the compilation error is triggered because the 'lua-term' module is detected as installed, but 'term/colors.lua' is missing from the filesystem. The lua-term module should provide it (see https://pkgs.alpinelinux.org/contents?name=lua5.1-term&branch=edge&repo=main&arch=x86_64)
2025-05-28 10:48:11 <Leso> Do the builders happen to have '/usr/lib/luarocks/rocks-5.1/lua-term/0.7-1/rock_manifest' or '/usr/lib/luarocks/rocks-5.4/lua-term/0.7-1/rock_manifest'? These would mark lua-term as installed
2025-05-28 10:50:13 <ikke> https://tpaste.us/PQaQ
2025-05-28 10:50:24 <ikke> That's without any deps installed
2025-05-28 10:51:37 <ikke> (this is on build-edge-aarch64)
2025-05-28 10:56:13 <Leso> okay, that's very strange. In that case the build should succeed, perhaps there were some leftovers from a different build when it got merged 2 weeks ago. Would you have a moment to trigger the build manually before I submit a MR to re-enable the aport?
2025-05-28 10:56:29 <Leso> testing/paged-markdown-3-pdf
2025-05-28 10:57:49 <ikke> Seems to pass now
2025-05-28 10:58:20 <Leso> Nice :) That's a bit concerning, but cool, I'll submit a MR then
2025-05-28 10:58:38 <ikke> lrc 1.0.1-1 depends on lua-term (not installed)
2025-05-28 10:58:40 <ikke> Installing https://luarocks.org/lua-term-0.8-1.rockspec
2025-05-28 11:01:27 <Leso> Perfect, that's how it should be
2025-05-28 11:25:44 <Leso> pass https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/84874
2025-05-28 11:25:53 <Leso> Thanks a lot for your time :) @ikke
2025-05-28 11:28:28 <ikke> No problem 
2025-05-28 14:03:42 <ncopa> anybody know what this is? https://gitlab.alpinelinux.org/razzydm/razzy-dm
2025-05-28 14:08:06 <ncopa> and this: https://gitlab.alpinelinux.org/XalayanMarie/abdera.amyth
2025-05-28 14:12:32 <ncopa> dunno why, but gitlab feels unusual sluggish
2025-05-28 14:41:37 <omni> it's probably due to "AI"
2025-05-28 14:44:32 <ikke> Average amount of requests has increased over time 
2025-05-28 14:52:45 <lotheac> time to set up countermeasures? :)
2025-05-28 15:10:48 <ikke> I already have some counter measures, but I fear the moment I have to go any further
2025-05-28 15:11:47 <lotheac> how so?
2025-05-28 15:12:50 <ikke> Because it most likely will affect legimimate users as well
2025-05-28 15:13:35 <lotheac> seems like a tradeoff that cannot be helped in the world we live in
2025-05-28 15:17:28 <lotheac> i get the reluctance though, but it seems to me most projects in the publicly-accessible software-forges have already made that tradeoff with anubis et al.
2025-05-28 15:18:03 <ikke> I already deployed something for git.a.o
2025-05-28 15:18:51 <lotheac> cool :)
2025-05-28 15:19:16 <lotheac> did it help?
2025-05-28 15:19:19 <ikke> yes
2025-05-28 15:20:49 <ikke> from ~40r/s to now ~1r/s
2025-05-28 15:22:36 <lotheac> it's an unfortunate development for the internet at large, but 40x resource usage is enough in my book to false-positive some legitimate users
2025-05-28 15:23:28 <lotheac> depending on the value of "some" ;)
2025-05-28 15:35:59 <invoked> fwiw - anarcat said (for tor's GL) he just banned alibaba at the ip level, blocked fb/meta's crawler in robots.txt, and added a crawl-delay to robots.txt
2025-05-28 15:36:26 <invoked> to address their slowdown
2025-05-28 15:51:34 <ikke> invoked: we were at some point stoked by random ips with random user agents
2025-05-28 15:51:44 <ikke> No manual filter would help
2025-05-28 15:52:20 <invoked> random user agents kills the anubis approach
2025-05-28 15:52:30 <invoked> which i didn't think would last long anyway
2025-05-28 15:53:11 <ikke> what do you mean, kills that approach?
2025-05-28 15:54:13 <invoked> 'If the client has a User-Agent that does not contain "Mozilla", the client is allowed through.'
2025-05-28 15:54:29 <invoked> that's the first test in anubis
2025-05-28 15:54:51 <ikke> It's not completly random
2025-05-28 15:55:02 <invoked> ah
2025-05-28 15:55:10 <ikke> mostly old chrome versions
2025-05-28 17:27:46 <omni> is build-edge-aarch64 intentionally not building?
2025-05-28 17:28:04 <fossdd|m> the kubernetes runner are a bit flaky today
2025-05-28 17:28:06 <fossdd|m> https://gitlab.alpinelinux.org/fossdd/aports/-/jobs/1872822
2025-05-28 17:28:12 <fossdd|m> dial tcp 10.96.0.1:443: connect: no route to host
2025-05-28 17:33:36 <omni> ok
2025-05-28 17:35:52 <ncopa> yeah
2025-05-28 17:36:28 <ncopa> ERROR: Job failed (system failure): prepare environment: setting up credentials: Post "https://10.96.0.1:443/api/v1/namespaces/gitlab-ci/secrets": dial tcp 10.96.0.1:443: connect: no route to host. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
2025-05-28 17:49:49 <ikke> Yeah, those hosts have not proven to be very stable
2025-05-28 18:05:35 <ncopa> where are they hosted?
2025-05-28 18:06:45 <ikke> cloudon
2025-05-28 18:10:49 <ncopa> do you mind if I take a look?
2025-05-28 18:12:04 <ikke> no
2025-05-28 18:12:16 <ikke> I'll pm youi
2025-05-28 18:12:26 <ncopa> 👍
2025-05-29 05:02:33 <lotheac> what's the kube control plane topology? are etcd and apiserver set up in a HA configuration?
2025-05-29 05:09:55 <lotheac> i'd be interested in making that stuff more reliable :)
2025-05-29 05:13:53 <iggy> lotheac: I know k0s is in use, don't know the exact config, but it's likely (at least partially) in gitlab
2025-05-29 05:15:04 <lotheac> cheers, i'll try to find it
2025-05-29 05:15:07 <iggy> lotheac: https://gitlab.alpinelinux.org/alpine/infra/k8s 
2025-05-29 05:15:10 <lotheac> thanks!
2025-05-29 05:16:23 <iggy> looks like single controller
2025-05-29 05:16:53 <lotheac> cluster-nld13/k0sctl.yaml has four nodes with  role: controller+worker though
2025-05-29 05:17:04 <iggy> https://gitlab.alpinelinux.org/alpine/infra/k8s/cluster-nld12/-/blob/master/terraform/k0sctl.yaml.tftpl?ref_type=heads#L10 (at least for nld12
2025-05-29 05:17:08 <lotheac> i haven't used k0s but i'm assuming that means a HA cplane
2025-05-29 05:17:44 <iggy> yeah, that's a "HA" workerplane
2025-05-29 05:18:57 <iggy> ("HA" because 4 controllers probably isn't a great idea... I mean it's better than 1, but...)
2025-05-29 05:19:01 <lotheac> a bit unclear to me which cluster the gitlab runners are in, can't find manifests for those
2025-05-29 05:19:40 <iggy> k0s/k0sctl is pretty easy to pick up if you have k8s experience
2025-05-29 05:19:52 <lotheac> i figured as much :)
2025-05-29 05:20:27 <lotheac> kubeadm is what i'm most used to
2025-05-29 05:24:50 <lotheac> i don't suppose it would be overly difficult to add more cplane nodes to that cluster, but i obviously don't have the whole picture here
2025-05-29 05:25:23 <lotheac> i mean there is already a LB there (although dunno if it's the LB for the apiserver or something else), etc
2025-05-29 05:29:19 <iggy> My guess is it's a tradeoff between having to rebuild a controller (which should be easy with the config all in tf/k0s/etc) if there's an outage or burning money on multiple cnotrollers
2025-05-29 05:31:09 <lotheac> well, you'd generally want a HA cplane for things like k8s upgrades anyway to avoid downtime. those machines running the cplane don't generally need to be very beefy
2025-05-29 05:33:06 <lotheac> i have a couple nanopi r6s's under my desk doing that for my personal cluster, and at one of my clients' infra 3 minimal-sized aarch64 cloud nodes per cluster... at 3.79 EUR/apiece/month
2025-05-29 05:35:05 <lotheac> upgrades aside, failure resiliency would be nice when the cloud node fails
2025-05-29 05:35:28 <lotheac> i would say that's worth a few dozen dollars a month :-)
2025-05-29 05:36:01 <ikke> lotheac: In many cases, we have to do with what we get
2025-05-29 05:36:21 <iggy> if I were to guess, nld13 is the ci runners which is the 4 controller+worker nodes, so it should be HA
2025-05-29 05:36:41 <lotheac> if so, i wonder what the failure was
2025-05-29 05:36:49 <ikke> The current one is not yet in gitlab but similar to nld13, except one controller
2025-05-29 05:36:52 <lotheac> ikke: yeah, that's understood of course.
2025-05-29 05:37:03 <ikke> It means no dedicated controllers
2025-05-29 05:37:11 <ikke> no loadbalancer
2025-05-29 05:37:23 <lotheac> why does it mean that?
2025-05-29 05:37:35 <lotheac> cost?
2025-05-29 05:37:44 <iggy> keep in mind that just 3 months ago (or less?) alpine got basically evicted from one of their main infra hosters
2025-05-29 05:37:58 <lotheac> right, i had forgotten about that
2025-05-29 05:38:09 <ikke> lotheac: No, they are not general cloud providers, we cannot just buy random resources
2025-05-29 05:38:34 <lotheac> but they generally are cloud stuff and not bare metal?
2025-05-29 05:38:54 <ikke> It depends, but in many cases bare-metal
2025-05-29 05:39:21 <lotheac> i guess i meant more like who owns the metal
2025-05-29 05:39:36 <lotheac> and the space it's in
2025-05-29 05:39:48 <ikke> The companies that sponsor us the HW
2025-05-29 05:40:51 <lotheac> an external LB is not a requirement for a HA cplane if you have some say in how the hardware is networked
2025-05-29 05:40:55 <ikke> In the case of that CI cluster, it happens to be VMs, but we just receive those VMs
2025-05-29 05:40:59 <lotheac> eg. kube-vip etc.
2025-05-29 05:41:10 <lotheac> (or any other virtual-ip solution)
2025-05-29 05:42:16 <ikke> I just learned the other day k0s supports an internal loadbalancer like you see with nld13
2025-05-29 05:42:34 <lotheac> i learned that just now :)
2025-05-29 05:43:45 <lotheac> so are you saying that there is a certain fixed number of VMs that you receive for that use, and you don't want to dedicate more than one of them to running the cplane, in the interest of having more resources to run the actual workloads?
2025-05-29 05:44:37 <ikke> lotheac: yeah, the VMs itself are beefy (at least in bare specs), and it would be a pitty to dedicate it just as a controller.
2025-05-29 05:46:20 <lotheac> then, how about running the cplane itself somewhere else on less beefy stuff? and connecting the worker nodes over wg or somesuch
2025-05-29 05:46:48 <ikke> For CI, HA is not necessarily important, but it depends on the failure mode
2025-05-29 05:47:22 <ikke> If a runner is unavailable, but we have other runners to pick up the job, that's fine
2025-05-29 05:47:29 <lotheac> less babysitting and manual recovery of nodes is pretty good for qol ;)
2025-05-29 05:48:05 <ikke> but if a runner picks up jobs, but that fails for some reason technical reason, that would be annoying
2025-05-29 05:48:17 <ikke> lotheac: that's why for nld13, I have multiple controllers
2025-05-29 05:49:25 <lotheac> sure. i'm just trying to understand why not for the CI cluster too
2025-05-29 05:49:46 <ikke> I tried initially, but the cluster was not usable
2025-05-29 05:50:02 <ikke> I then learned that for multiple controllers you need a loadbalancer
2025-05-29 05:50:13 <ikke> and later I learned that you can have an internal loadbalancer
2025-05-29 05:50:35 <lotheac> hehe :)
2025-05-29 05:51:10 <ikke> Fairly new to k8s, so I'm learning as I go
2025-05-29 05:51:48 <lotheac> thanks for the description, sounds reasonable enough. i'm offering to help if you want it anytime :)
2025-05-29 05:52:04 <ikke> yes, that's very appreciated
2025-05-29 05:52:36 <ikke> I am commissioning nld13 (the k8s cluster is already running like described in that k0sctl file)
2025-05-29 05:53:45 <lotheac> are you using any config management for the nodes themselves aside from k0sctl?
2025-05-29 05:54:12 <ikke> Not at the moment, was looking a bit into ansible
2025-05-29 05:54:29 <ikke> But, the config of those nodes is also very minimal
2025-05-29 05:54:43 <lotheac> in my experience it's better not to, unless you need that to create the cluster in the first place. because you can use the control-plane itself to manage stuff even if it needs host netns etc
2025-05-29 05:55:11 <ikke> right
2025-05-29 05:55:50 <lotheac> one weird hack i did recently was create static pods for kube-vip (the virtual ip thing for the apiserver address) using a daemonset that runs on cplane nodes only and just writes to /etc/kubernetes/manifests/ when stuff in my kube git repo changes
2025-05-29 05:56:09 <lotheac> it's a bit strange but it is pretty nice to avoid ansible etc.
2025-05-29 05:56:57 <lotheac> of course there is a chicken-and-egg problem there that needs resolving at initial cluster creation time, but adding new cplane nodes needs no such hackery
2025-05-29 05:58:48 <lotheac> another thing i did semi-recently (on a cloud cluster) was spinning up tailscale containers as a daemonset using host netns, which gives me connectivity to new autoscaled nodes etc. without having to ansible
2025-05-29 05:59:48 <lotheac> if circumstances dictate, it'd also be possible to run the cluster internal network on top of a vpn like that and you could have nodes from different providers in the same cluster. of course latency is not optimal, but it might not matter
2025-05-29 05:59:54 <ikke> Do you have experience with having the control plane living somewhere else?
2025-05-29 06:00:23 <lotheac> not in practice, but i've been thinking about that
2025-05-29 06:01:10 <lotheac> just need to establish an internal network first to build the cluster on. that could be tailscale or just plain wg
2025-05-29 06:03:08 <lotheac> semantically the control plane being somewhere else doesn't matter much to the worker nodes that are being added (or removed) into the cluster. they spin up and run the necessary initialization procedure to join the cluster by talking to the apiserver
2025-05-29 06:03:40 <lotheac> so you would just need to make sure they can talk to the apiserver from where you're spinning them up -- potentially by just creating a wg link first
2025-05-29 06:54:06 <ncopa> we could have a control plane in some reliable cloud. 3 (smallish) nodes. the worker nodes can be anywhere, and dont need be able to route to each other. at least not for CI
2025-05-29 06:55:29 <lotheac> yeah
2025-05-29 06:56:40 <lotheac> i would try to make it so that worker nodes _can_ communicate though, there might be assumptions baked into things
2025-05-29 07:49:55 <ikke> Would we have a single cluster for all CI hosts? (Currently we have 2 separate sponsors providing us compute)
2025-05-29 07:52:39 <lotheac> i find fewer clusters tends to be easier to manage but you could do it either way (even multiarch/heterogeneous nodes etc)
2025-05-29 07:55:10 <lotheac> if the usecases for separate sponsors’ nodes are identical i think it makes sense to put them in the same cluster. unless there is some security boundary that prevents that
2025-05-29 07:58:27 <ikke> Having 3 control nodes per cluster would quickly add up
2025-05-29 08:06:22 <lotheac> that’s also a good point in favor of fewer clusters
2025-05-29 08:35:04 <ncopa> lotheac: FYI in case you didnt know. I am also on the k0s team. 
2025-05-29 08:42:40 <lotheac> ah, I did not :) that makes sense
2025-05-29 08:43:20 <ikke> So using k0s makes a lot of sense for us :)
2025-05-29 08:45:30 <lotheac> yeah definitely some synergies
2025-05-29 08:46:21 <ikke> I have no experience with alternatives, but setting up a new cluster with k0sctl is minutes of work
2025-05-29 08:46:24 <lotheac> maybe i should try out k0s then :)
2025-05-29 08:47:09 <lotheac> it’s not hard with kubeadm either… haven’t tried rancher but have been exposed to some cloud vendor managed stuff as well
2025-05-29 08:48:33 <ikke> For now, I have chosen to install / manager kubernetes clusters myself instead of using cloud managed options just to get a better feel of what it entails
2025-05-29 08:49:05 <ikke> k0s still abstracts a lot though
2025-05-29 08:59:42 <lotheac> that’s definitely better anyway. the cloud vendors doing managed kube overcharge for simple stuff
2025-05-29 09:00:44 <lotheac> and generally seek to lock you in :)
2025-05-29 09:02:08 <ikke> nod
2025-05-29 09:04:12 <ikke> Not sure if it's feasible for you to do so, but perhaps you could work on a terraform setup similar to what is done for nld12 for a 3 node controller setup with a loadbalancer?
2025-05-29 09:38:52 <lotheac> sure. i don’t much like terraform but i can do that. which cloud provider would you like to work with?
2025-05-29 09:39:10 <ikke> linode
2025-05-29 09:39:36 <lotheac> alright, i’ll take a stab after i’m done with dinner :)
2025-05-29 11:16:25 <ncopa> i wonder how much it would help to add a cache in front of gitlab.a.o. Would be interesting to see access log stats, which pages are the 20 most visited, or how many pages are visited more than once.
2025-05-29 14:04:47 <Kladky> For me, trying to look at the commit list of a certain file/directory in gitlab always results in an error 500
2025-05-29 14:05:07 <Kladky> Request ID: 01JWE5PXANP44Z3YYXQ9V59S2M
2025-05-29 14:06:07 <ikke> "exception.message": "4:Deadline Exceeded."
2025-05-29 14:06:36 <ikke> apparently gitaly is not able to get the data within 30 seconds
2025-05-29 14:14:08 <Kladky> Oh
2025-05-29 14:19:32 <ikke> It comes up more often, but not sure what we can do about it
2025-05-29 14:19:46 <ikke> It's specifically aports that has this issue
2025-05-29 14:45:05 <qaqland> Kladky: use cgit service: git.a.o
2025-05-29 14:47:14 <qaqland> Is this git command helpful? git commit-graph write --changed-paths
2025-05-29 14:47:20 <Kladky> Thanks, but at that point, I'd be more convenient to just cd aports and git log from there
2025-05-29 14:57:31 <ikke> qaqland: gitlab itself is already generating commit graphs
2025-05-29 14:59:30 <ikke> It may help to use --changed-paths, but I'd need to test it first
2025-05-29 15:38:19 <lotheac> ikke: https://gitlab.alpinelinux.org/lotheac/cluster-ci/
2025-05-29 15:39:12 <lotheac> quite a different approach than cluster-nld12 initialization had tbh
2025-05-29 15:41:07 <ikke> Thanks, I'll take a look later. Note that for communication between the control nodes, we could use a private lan in linode
2025-05-29 15:41:49 <lotheac> my poc doesn't, it uses the tailnet for everything
2025-05-29 15:42:03 <lotheac> (allowing each node, whether cplane or not, to be anywhere)
2025-05-29 15:42:26 <lotheac> well, aside from the lb placing constraints on the cplane nodes having to be in linode
2025-05-29 15:43:23 <ikke> I have to think about it. We do have our own automeshing VPN (dmvpn)
2025-05-29 15:44:09 <ikke> though setting it up is slightly more manual
2025-05-29 15:44:24 <lotheac> i'm also not really fond of having the apiserver and the k0s apis available on public internet at all (but that's what the LB does). some creative routing would help make that internal within the meshvpn
2025-05-29 15:45:33 <lotheac> i haven't used dmvpn but i guess not much reason why it couldn't be usable here as well even if it's a bit manual; the cluster is not going to have much node churn
2025-05-29 18:26:24 <ikke> lotheac: If we want to make the control plane only available over VPN, it does mean we would need to host a loadbalancer ourself, right?
2025-05-29 18:26:49 <ikke> (It's something I have been thinking about)
2025-05-30 00:43:24 <lotheac> ikke: yeah, essentially. k0s gives multiple options there though.
2025-05-30 00:44:38 <lotheac> technically i suppose it would also be possible to have only the k8s api available on a public lb, while having the rest of the control plane only on vpn too
2025-05-30 00:46:03 <lotheac> k0s is a bit different here than what i was used to, since it expects not just the kube apiserver lb, but also konnectivity and the k0s api
2025-05-30 00:46:24 <lotheac> https://docs.k0sproject.io/v1.33.1+k0s.0/networking/#required-ports-and-protocols
2025-05-30 00:46:48 <lotheac> and https://docs.k0sproject.io/v1.33.1+k0s.0/high-availability/#load-balancer
2025-05-30 00:48:29 <lotheac> and the "multiple options" i mentioned is are the builtin loadbalancers you pointed out earlier: https://docs.k0sproject.io/v1.33.1+k0s.0/cplb/ & https://docs.k0sproject.io/v1.33.1+k0s.0/nllb/ -- although there's no reason why we couldn't use another virtual-ip solution on top of the vpn as well (options there depending on what layer that vpn operates on)
2025-05-30 00:54:11 <lotheac> i didn't find an option with linode to make the LB available only on a vpc/private interface; seems their nodebalancers always assume you want to connect to them over the internet. some other providers allow private-only LBs. the reason i mention this is, exposing the k8s api on the internet is attack surface that does not generally need to be there.
2025-05-30 00:55:21 <lotheac> that notwithstanding -- i have had setups where there is a cloud-managed LB, but it has been firewalled or otherwise configured thus that it is only successfully routed to from inside the vpn
2025-05-30 01:50:49 <lotheac> https://lotheac.fi/s/k0s-meshvpn.drawio.svg this might help visualize what i'm talking about
2025-05-30 01:51:30 <lotheac> it's a bit messy but hopefully it helps
2025-05-30 02:11:54 <lotheac> the poc i linked yesterday is essentially "option 2"
2025-05-30 09:03:04 <ncopa> looks like gitlab broke
2025-05-30 09:03:43 <ncopa> I dont appear to have access to that server
2025-05-30 09:05:16 <ncopa> yeah gitlab is on its knees
2025-05-30 09:18:34 <ikke> You should have access, but port 2222
2025-05-30 09:19:34 <ncopa> aha. thanks
2025-05-30 09:21:03 <ikke> Seems to have been a load spike 
2025-05-30 09:21:14 <ikke> But it recovered again 
2025-05-30 09:21:43 <ncopa> load avg is 16-18
2025-05-30 09:21:58 <ncopa> we need 15x more cpu power to handle the current load
2025-05-30 09:22:11 <ikke> Not necessarily an increase in requests 
2025-05-30 09:34:50 <ncopa> how can i run `gitlab-ctl tail`?
2025-05-30 09:57:23 <ikke> We do not use omnibus
2025-05-30 09:57:35 <ikke> Which log do you want to tail?
2025-05-30 09:58:13 <ikke> /srv/compose/gitlab/log
2025-05-30 09:58:41 <ikke> Nginx is in s6/nginx/current 
2025-05-30 10:29:49 <ncopa> wanted to investigate what is eating the CPU
2025-05-30 10:35:10 <ikke> At the moment it seems to be gitally running git gc on a project
2025-05-30 10:35:48 <ikke> But that seems to be at most 2 cores
2025-05-30 10:57:04 <ncopa> question, the x86_64 tag  makes it only run on x86_64? https://gitlab.alpinelinux.org/alpine/alpine-conf/-/blob/master/.gitlab-ci.yml?ref_type=heads#L9
2025-05-30 10:57:22 <ncopa> if I remove it, it will run on any arch with docker-alpine?
2025-05-30 10:58:09 <ncopa> x86_64 is currently busy and I'd prefer to run the tests https://gitlab.alpinelinux.org/alpine/alpine-conf/-/merge_requests/256
2025-05-30 10:58:40 <ncopa> never mind. I pushed it without waiting for CI
2025-05-30 11:42:29 <ncopa> ok i have emerngecy woth ppc64le machine now
2025-05-30 11:42:34 <ncopa> its unresonsive
2025-05-30 11:42:38 <ncopa> we need it up ASAP
2025-05-30 11:43:13 <ncopa> very on fortunate
2025-05-30 11:47:52 <ncopa> who do I have to call to powercycle the ppc64le machine?
2025-05-30 12:07:24 <ncopa> ok will do release without ppc64le
2025-05-30 12:23:35 <ikke> /o\
2025-05-30 12:52:46 <ncopa> so, i have a problem
2025-05-30 12:52:56 <ncopa> I tagged the release
2025-05-30 12:53:10 <ncopa> when ppc64le comes back up, I have a short window
2025-05-30 12:53:49 <ncopa> while its building the release on ppc64le I will have to log in, and git checkout 3.22-stable
2025-05-30 12:54:05 <ncopa> before it starts to build the package that was pushed after the branch was done
2025-05-30 12:54:26 <ncopa> otherwise it will start building those packages and upload them to v3.22
2025-05-30 12:54:40 <ikke> Oof
2025-05-30 12:55:14 <ikke> You could disable the ssh key on dl-master
2025-05-30 13:16:38 <ncopa> good idea
2025-05-30 13:16:42 <ncopa> maybe I should do that
2025-05-30 13:17:36 <ncopa> i disabled ppc64le key on dl-master
2025-05-30 14:14:12 <ncopa> ikke: are there any sponsors we should add/remove from release notes?
2025-05-30 14:14:35 <ikke> I was thinking about that, but probably for the next release 
2025-05-30 14:26:49 <ncopa> ok good thanks
2025-05-30 15:13:27 <ncopa> ikke: can you please help me with "Invalidate /alpine/latest-stable/* on dl-cdn" on https://gitlab.alpinelinux.org/alpine/aports/-/issues/17188
2025-05-30 15:17:30 <ikke> ncopa: ack
2025-05-30 15:18:36 <ikke> It's running
2025-05-30 15:23:10 <ncopa> thanks!
2025-05-30 15:29:57 <ncopa> ikke: can you please also help me post something on mastodon?
2025-05-30 15:34:34 <ikke> ncopa: https://fosstodon.org/deck/@alpinelinux/114597512294140956
2025-05-30 15:39:31 <ncopa> i dont remember who has access to the bluesky account?
2025-05-30 15:39:40 <ikke> you?
2025-05-30 15:41:12 <ncopa> ha! that is what I was afraid of
2025-05-30 15:42:01 <ikke> it's back
2025-05-30 15:44:31 <ncopa> im on it
2025-05-30 15:44:33 <ncopa> thanks!
2025-05-30 16:27:47 <ncopa> I don't think I have the password for x.com/alpinelinux seems it has been changed
2025-05-30 16:48:19 <ncopa> which email did we use for twitter? social at a.o?
2025-05-30 18:48:26 <ncopa> do we need to do anything to sync the pkgs.a.o 3.22 repo? https://pkgs.alpinelinux.org/packages?name=&branch=v3.22&repo=&arch=x86_64&origin=&flagged=&maintainer=
2025-05-30 18:51:22 <fossdd|m> no it should fetch the branches automatically
2025-05-30 18:51:38 <fossdd|m> hmm
2025-05-31 04:56:17 <qaqland> Are there any consistency checking tools for APKINDEX, e.g. check whether all packages in the current APKINDEX are fully synchronized?
2025-05-31 04:57:03 <qaqland> our mirror sometimes loses a few packages
2025-05-31 08:23:16 <fossdd|m> there is https://mirrors.alpinelinux.org/
2025-05-31 11:02:10 <qaqland> fossdd|m: link is not helpful, i met an internal issue with local mirror site :|
2025-05-31 11:25:44 <ikke> qaqland: It's something I could add to repo-tools
2025-05-31 12:02:32 <qaqland> ikke: thx :)
2025-05-31 12:04:44 <ikke> qaqland: do you want to check it through the local filesystem?
2025-05-31 12:27:48 <qaqland> ikke: yes, want to do verification after completing rsync
2025-05-31 13:35:22 <ikke> qaqland: are you able to test https://gitlab.alpinelinux.org/alpine/infra/repo-tools/-/merge_requests/1?
2025-05-31 14:11:57 <qaqland> ikke: I can't but I have contacted two friends from different schools. it may take some time
2025-05-31 14:12:15 <ikke> ok
2025-05-31 14:12:28 <ikke> It very simply just checks if the package file exists, nothing more