2025-11-01 08:24:47 <ncopa> kwidgetsaddons tests was deadlocked on build-3-23-s390x. I restarted it
2025-11-02 12:48:18 <omni> =(
2025-11-02 13:40:35 <omni> \:D/
2025-11-02 13:53:54 <omni> but are things still stuck somehow?
2025-11-02 14:13:31 <ikke> anything I've missed?
2025-11-02 14:16:55 <omni> things are moving again
2025-11-02 14:24:13 <omni> ikke: <3
2025-11-03 06:22:20 <lindsay> Unable to access https://build.alpinelinux.org/buildlogs/ , 404 Not Found
2025-11-03 07:08:01 <qaqland> same
2025-11-03 08:30:31 <clandmeter> we have issues with the server 
2025-11-04 17:17:13 <Ariadne> clandmeter: is there anything i can do to help resolve these problems with build.a.o?
2025-11-04 18:31:32 <vkrishn> and no-one has spoken about it in 24hrs+, and its not even Friday
2025-11-04 19:04:20 <ikke> Ariadne: do you have experience with zfs? We regularly get kernel panics. The data is stored on a linode volume, no idea if that's a single disk that can become corrupt.
2025-11-04 19:04:49 <Ariadne> i do have some experience with zfs
2025-11-04 19:05:25 <Ariadne> do you have an example of the kernel panic?
2025-11-04 19:10:20 <ikke> WARNING: zfs: adding existent segment to range tree (offset=2365157c00 size=800)
2025-11-04 19:12:28 <ikke> zpool status -v showed a bunch of files that could not be restored 
2025-11-04 19:14:10 <Ariadne> in that case, the volume is cooked
2025-11-04 19:14:39 <ikke> Yeah, I figured 
2025-11-04 20:31:40 <omni> ok, I'll say it, why not mirror two volumes?
2025-11-04 20:35:30 <ikke> Cost versus available budget 
2025-11-04 20:36:19 <ikke> Disk space, contrary to popular believe, is not cheap 
2025-11-05 01:27:37 <lotheac> generally i would expect cloud block storage to be resilient against corruption. my point being that if it was corrupted due to software we run and not any external factor, then mirroring would not even have helped
2025-11-05 01:28:48 <lotheac> but i agree with the ariadne. probably best to throw away that volume, salvage the data that is salvageable, and make a new one
2025-11-05 01:29:02 <ikke> working on that as we speak
2025-11-05 01:30:23 <lotheac> cool :)
2025-11-05 23:25:26 <ikke> I'm trying to get as much data as possible copied over to the new volume
2025-11-05 23:32:20 <vkrishn> these are just distfiles?
2025-11-05 23:37:08 <vkrishn> restoring only for edge, then last 1 release should get things started
2025-11-05 23:38:48 <vkrishn> rest can be done later, in passive/pull way
2025-11-05 23:44:13 <ikke> The crashes happen while fetching edge and v3.23
2025-11-05 23:45:00 <ikke> The other data hardly caused issues and has been synced already 
2025-11-05 23:47:19 <vkrishn> ok, build.a.o seems back :)
2025-11-06 00:01:41 <ikke> Yes, I'm keeping an eye on it
2025-11-06 00:21:22 <mio> could someone check on the 3.23 loongarch64, x86 and x86_64 builders?
2025-11-06 00:21:48 <ikke> mio: heh, I was just checking all builders :P
2025-11-06 00:22:03 <mio> great timing then, thanks
2025-11-06 00:23:20 <mio> for polari, the 2nd command in check() is unfortunately causing the build to not exit, still looking into the reason
2025-11-06 00:25:08 <mio> py3-anyio should be okay next retry
2025-11-06 00:25:12 <ikke> ok
2025-11-06 00:30:06 <mio> is it possible to request upload access to dev.a.o please? 1-2 aports have source archive missing/no longer available
2025-11-06 01:02:56 <ikke> I have switched distfiles to use the new volume
2025-11-06 01:07:02 <mio> thanks!
2025-11-06 01:52:14 <ikke> bootstrapping openjdk17 on x86_64
2025-11-06 01:54:37 <ikke> mio: can you try to ssh into mio@dev.alpinelinux.org?
2025-11-06 01:55:49 <mio> thanks! ah, not yet, permission denied
2025-11-06 01:56:14 <mio> sorry, meant just tried to do so and got an error
2025-11-06 01:56:44 <mio> (polari should be okay now as well)
2025-11-06 01:58:28 <ikke> I used the ssh keys from gitlab
2025-11-06 01:58:48 <mio> ah, one moment
2025-11-06 02:00:09 <mio> works, thanks!
2025-11-06 02:00:32 <ikke> There's a symlink in your homedir to the archive directory
2025-11-06 02:01:10 <mio> fantastic, thanks very much!
2025-11-06 02:19:52 <ikke> bootstrapping openjdk11 on aarch64
2025-11-06 02:34:40 <ikke> bootstrapping openjdk8 on s390x
2025-11-06 07:41:08 <omni> has anything changed wrt aarch64 CI in the past few days?
2025-11-06 07:41:52 <omni> I'm trying to run for !92407 and it is slower than before and consistently runnig out of memory
2025-11-06 12:27:18 <ikke> omni: nothing has changed
2025-11-06 12:43:20 <ikke> omni: perhaps another heavy job that was running?
2025-11-06 19:16:03 <omni> I've been retrying now and then the past couple of days
2025-11-06 19:17:23 <omni> last run wasn't as slow, but still running out of memory like it didn't last week
2025-11-06 23:27:45 <vkrishn> build-edge-x86_64 seems down, but alert is not getting triggered
2025-11-06 23:28:12 <mio> probably hanging, not necessarily offline
2025-11-06 23:28:44 <mio> as in hanging build, which may explain no notification
2025-11-06 23:31:35 <vkrishn> maybe a regular interval heartbeat
2025-11-06 23:36:46 <vkrishn> i put 2min cron sending devices thermal data to redis/mqtt
2025-11-06 23:41:35 <vkrishn> nope, currently its just battery status
2025-11-07 13:36:52 <ikke> ..
2025-11-07 13:37:09 <ikke> Still working here 
2025-11-09 00:01:30 <ikke> Upgrading gitlab
2025-11-09 00:45:19 <ikke> bootstrapping ghc on x86_64
2025-11-10 16:44:04 <ncopa> bootstrapping openjdk21 on s390x
2025-11-10 20:26:54 <ncopa> im bootstrapping openjdk21 on build-3-23-riscv64
2025-11-10 20:28:56 <ikke> 👍
2025-11-11 08:12:17 <ncopa> ghc was bootstrapped on aarch64
2025-11-11 08:17:35 <ikke> 👍
2025-11-11 14:25:56 <ncopa> im bootstrapping openjdk21 on build-3-23-x86_64
2025-11-11 14:29:30 <mio> could more space be added to build-3-23-x86 please for rust build? "No space left on device (os error 28)"
2025-11-11 14:34:47 <ncopa> added 200G
2025-11-11 14:40:07 <mio> thanks!
2025-11-12 07:18:56 <angeloverlain[m]> uh, can someone take a look at this? https://gitlab.alpinelinux.org/alpine/infra/apkbrowser/-/merge_requests/22 would be plenty useful to make sure we don't have outdated pkgs in aports
2025-11-12 11:01:57 <omni> is Anitya the only thing that can flag aports today?
2025-11-12 11:32:50 <ikke> yes
2025-11-12 11:34:19 <omni> it also does a lot of false flagging
2025-11-12 14:38:04 <ncopa> im bootstrapping ghc and openjdk11 on build-3-23-x86_64
2025-11-12 16:07:05 <angeloverlain[m]> omni: it does that? maybe the mapping could be fixed for the packages where that happens
2025-11-12 17:07:21 <Sertonix[m]> That's not really possible. Eg. librewolf version format differs.
2025-11-12 17:09:30 <ikke> The nice thing about standards is that there are so many of them
2025-11-12 20:01:23 <omni> one thing that could be nice would be the ability to, per aport, opt out of Anitya flagging
2025-11-12 20:04:24 <omni> angeloverlain[m]: the latest release tag may not correspond to any of the tags in multiple lts branches of a project, for example, but that's just a simple case for automatic false flagging
2025-11-12 20:36:10 <angeloverlain[m]> @sertonix:mozilla.org I think that's the RPM versioning scheme used by librewolf, but may be wrong
2025-11-12 20:36:55 <angeloverlain[m]> @_oftc_omni:matrix.org I thought anitya release monitoring only checks edge (hence latest stable, not lts). but you're right, might be nice to opt out per pkg
2025-11-12 20:37:56 <angeloverlain[m]> also, who might I ping so that the version of mediawiki used on wiki.alpinelinux.org gets updated? it's been outdated for a while now :)
2025-11-12 21:28:09 <fcolista> friends I really need to apologize for having missed the TSC meeting :( I missed it totally while busy in other things...I'm very very sorry. Apologize again
2025-11-13 08:14:53 <ncopa> no worries. it happens
2025-11-13 15:40:43 <ikke> Anyone helping with infra that wants to help maintaining the wiki?
2025-11-13 15:57:01 <durrendal> ikke: I don't mind helping out, just let me know what you need done
2025-11-13 16:10:46 <ikke> durrendal: the wiki is hosted in an LXC container
2025-11-13 16:18:11 <durrendal> that makes sense, it tracks how most everything else is deployed. It's mediawiki, so php + a db. Is it using postgres on the backend, and do you have it behind traefik or something else?
2025-11-13 16:22:06 <ikke> It uses mariadb
2025-11-13 16:22:26 <ikke> There is an nginx proxy in front
2025-11-13 16:22:51 <ikke> The database is in a separate container
2025-11-13 16:26:56 <durrendal> That's not bad, seems like a straight forward setup at first glance. I use mariadb for my zabbix deployments so that's nice and familiar
2025-11-13 16:34:47 <ikke> The current setup uses a combination of git submodules and composer from what I can tell (alice was the last one taking care of it)
2025-11-13 18:28:13 <durrendal> ikke: hmm that's not a terrible way to make sure versions don't change out from under you, I wonder if some of those could be converted to regularly installed packages though.
2025-11-13 18:29:37 <nu_> everything can be turned into a package:)
2025-11-13 18:30:10 <durrendal> just requires someone to put their name on the maintainer line and tend and feed it, which I certainly don't mind doing :)
2025-11-13 18:30:51 <durrendal> ikke: since it's an LXC container, could it be possible to clone it so I could have a sandbox to tinker with?
2025-11-14 05:29:53 <prabu> durrendal, thanks for offering to help out on wiki.. i'd earlier requested for adding Extension:ArticleRatings and enabling numbered headings to our wiki in the #alpine-docs channel. i'll appreciate if you can include these as part of testing..
2025-11-16 12:06:29 <fossdd> hmm is build-edge-aarch64 offline?
2025-11-16 12:08:31 <ikke> Hanging on a build
2025-11-16 12:08:41 <ikke> rio
2025-11-16 12:08:56 <fossdd> then it is stuck because its been in rio since yesterday
2025-11-16 12:09:08 <ikke> I've killed the build
2025-11-16 12:09:14 <fossdd> ah thanks
2025-11-16 21:57:44 <ayakael> Hi! I'm getting a 413 Request Entity Too Large too large error on https://gitlab.alpinelinux.org/ayakael/aports/-/jobs/2098524. I reckon it's the build artifacts that are too large for upload, but it's failing the CI.
2025-11-16 21:59:45 <ikke> ayakael: seems like the logs?
2025-11-16 21:59:50 <ikke> >>> Artifact size 3990180864 larger than max (300000000), skipping uploading them
2025-11-16 22:00:06 <ikke> This prevents the build artifacts from being copied in the directory where gitlab would try to upload them
2025-11-16 22:02:22 <ikke> First time we run into this
2025-11-16 22:05:17 <ikke> we copy the build log per build into artifacts to make it easier for automation to obtain that
2025-11-17 09:41:25 <ikke> ftr, it's ipv6 that's down 
2025-11-18 02:46:03 <invoked> fyi, "Hacker Manifesto" on users@lists.a.o is the same fucko who's been polluting openbsd-misc.  best put in the killfile
2025-11-18 03:14:22 <pj> maybe it would be good to sunset aports ml :>
2025-11-18 12:07:51 <invoked> seems like cloudflare is having problems globally
2025-11-18 12:17:03 <invoked> microsoft has its own infra but the cloudflare outage could still be affecting it indirectly.
2025-11-18 12:17:21 <invoked> oops, wrong window.
2025-11-18 12:20:54 <dne> 🍿
2025-11-18 12:22:59 <panekj> matrix.org is dying because of CF
2025-11-18 14:32:08 <mio> could the edge armv7 builder be restarted please? the build is unfortunately hanging
2025-11-18 15:41:26 <ncopa> im on it
2025-11-18 15:43:54 <mio> thanks!
2025-11-18 18:52:46 <ikke> bootstrapping openjdk11 on s390x
2025-11-18 19:35:24 <achill> uhhh are the CI runners down?
2025-11-18 19:35:36 <achill> i would assume its because of cloudflare
2025-11-18 19:37:41 <ikke> achill: I would not expect the cf outage to affect our runners
2025-11-18 19:38:30 <achill> hmm then dunno, just that my mrs dont get a runner
2025-11-18 19:38:39 <achill> maybe i should actually do my uni assignments lol
2025-11-18 19:39:34 <ikke> Lots of running jobs
2025-11-18 19:40:59 <achill> ah i see
2025-11-18 19:44:00 <ikke> Not sure if the jobs are making progress though
2025-11-18 19:44:19 <ikke> https://gitlab.alpinelinux.org/strophy/aports/-/jobs/2100966
2025-11-18 19:44:38 <ikke> https://gitlab.alpinelinux.org/selfisekai/aports/-/jobs/2101211
2025-11-18 19:44:52 <ikke> https://gitlab.alpinelinux.org/strophy/aports/-/jobs/2101255
2025-11-19 15:04:38 <lotheac> gitlab seems to be sending some emails to me more than once. 2-4 times
2025-11-19 15:05:29 <lotheac> different messageid's and timestamps, but identical content
2025-11-19 16:16:31 <ikke> lotheac: for a longer period, or just intermittent?
2025-11-19 16:46:54 <ptrc> same here, doing this for basically any notification in the past hour or so
2025-11-19 16:47:10 <ptrc> and the "reply-to" email is different for each
2025-11-19 21:18:36 <ncopa> \o/
2025-11-19 21:18:43 <ncopa> i have a reproduce for tevent
2025-11-19 21:22:24 <ikke> yay, congrats
2025-11-19 21:39:01 <ncopa> and I have a workaround
2025-11-19 21:39:23 <ncopa> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/87671
2025-11-19 21:39:48 <ikke> oh, wow
2025-11-19 21:40:13 <ncopa> https://bugzilla.samba.org/show_bug.cgi?id=15952
2025-11-19 21:41:04 <ikke> I'm not sure where the data on stdin comes from..
2025-11-19 21:41:39 <ncopa> no idea. but I guess the CI set up stdin to a tty or something? no idea
2025-11-19 21:44:24 <ncopa> The test uses stdin for poll/epoll to test the loop_once. (0 is stdin)  https://gitlab.com/samba-team/devel/samba/-/blob/master/lib/tevent/tests/test_tevent_trace.c?ref_type=heads#L426
2025-11-19 21:48:50 <ncopa> whoops. that was wrong channel
2025-11-19 21:50:36 <ikke> heh
2025-11-19 23:35:46 <omni> did something just break?
2025-11-19 23:36:55 <omni> like gitlab
2025-11-19 23:37:34 <omni> but also looking at build.a.o and seeing several being stuck at "pulling git"
2025-11-19 23:38:42 <omni> go a 404 on gitlab.a.o but now it seems to be back
2025-11-20 00:59:17 <lotheac> ikke: it seems to have resolved now, but lasted a few hours
2025-11-20 06:18:34 <mio> could edge ppc64le and x86_64 builders be restarted please? test is likely hanging, hopefully a retry with timeout will work
2025-11-20 06:19:46 <ikke> done
2025-11-20 06:20:55 <mio> thanks!
2025-11-20 10:02:19 <omni> could the meli tarball be added to distfiles? arm builders are having issues fetching it
2025-11-20 10:15:10 <ptrc> something's definitely off with gitlab
2025-11-20 10:15:16 <ptrc> i can't fetch anything over ssh
2025-11-20 10:15:55 <ptrc> https://ptrc.gay/ERNGyknh
2025-11-20 10:17:53 <ptrc> actually, http also hangs..?
2025-11-20 10:17:58 <ptrc> https://ptrc.gay/LXkroZAT
2025-11-20 18:24:52 <ncopa> maybe we should point all the builders to https://git.a.o instead of gitlab
2025-11-21 19:54:27 <ikke> durrendal: I have created a copy of the wiki container and database
2025-11-21 19:54:44 <ikke> durrendal: but you need to connect to our vpn to access it
2025-11-21 19:54:59 <ikke> Would you be able to send me a wireguard public key?
2025-11-21 20:30:16 <durrendal> ikke: absolutely, it'll have to wait until I get off work if that's okay
2025-11-21 20:30:46 <ikke> sure, np
2025-11-21 20:30:51 <ikke> There's no hurry
2025-11-21 21:00:43 <ikke> fossdd: I wonder if we could enable OIDC between our gitlab instances somehow
2025-11-21 21:00:47 <ikke> https://docs.gitlab.com/integration/openid_connect_provider/
2025-11-21 21:45:40 <ikke> achill: ^
2025-11-21 22:42:17 <ikke> I think it should be possible. We would need to add an application in the admin backend, and an oauth provider in the config file
2025-11-21 22:42:29 <ikke> Let me know if you're interested
2025-11-23 09:55:20 <angeloverlain[m]> @_oftc_achill:matrix.org did you find time to take a look at apkbrowser!22 ?
2025-11-23 09:56:32 <ikke> https://gitlab.alpinelinux.org/alpine/infra/apkbrowser/-/issues/22
2025-11-24 09:18:20 <ncopa> i think py3-dask tests deadlock on build-3-23-ppc64le
2025-11-24 09:18:28 <ncopa> i have killed it and restarted it
2025-11-24 09:18:38 <ncopa> i think I observed it on CI on x86_64 as well
2025-11-24 09:18:52 <ncopa> might be it is the parallel pytest that does it. I dont know
2025-11-24 09:29:28 <ikke> ncopa: i already killed it twice yesterday 
2025-11-24 09:29:42 <ikke> So yeah, deadlock 
2025-11-24 09:31:16 <ncopa> i'll try to reproduce it in my dev env
2025-11-24 10:57:17 <dne> gitlab ci x86_64 runner is a bit sad: Unschedulable: "0/3 nodes are available..."
2025-11-24 13:34:29 <mio> can build-3-22-armhf be restarted please?
2025-11-24 23:22:21 <achill> dinkdonk gitlab down i think
2025-11-24 23:49:33 <achill> seems to be fixed
2025-11-25 07:00:38 <ncopa> morning! I'm working on build-3-23-ppc64le now. trying to identify the test that deadlocks
2025-11-25 11:48:56 <ncopa> i gave up dask ppc64le and manually worked around it. im bootstrapping openjdk25 on build-3-23-riscv64 now
2025-11-25 13:03:20 <angeloverlain[m]> @_oftc_ikke:matrix.org RE: https://gitlab.alpinelinux.org/alpine/infra/apkbrowser/-/merge_requests/22 the DB doesn't actually need to be cleared. my bad
2025-11-25 16:13:18 <ikke> angeloverlain[m]: yeah, I saw your comment
2025-11-26 01:28:49 <zv> seeing some 500's at https://gitlab.alpinelinux.org/alpine/aports/-/commits/master/testing/py3-tokenizers/APKBUILD e.g.
2025-11-26 07:38:48 <ncopa> the gnutls test ktls failures is probably because something is lacking in kernel config on ppc64le, but not sure what
2025-11-26 07:39:07 <ncopa> it also fails on armhf test-memset_explicit, which I suspect is a complier problem
2025-11-26 07:45:22 <ncopa> i wonder if we should upgrade the kernel on the ppc64le machine to 6.12
2025-11-26 07:54:38 <huajingyun> I'm curious, what is the current kernel version of the ppc64le machine? 
2025-11-26 07:57:01 <ncopa> Linux ncopa-edge-ppc64le 6.6.86-0-lts #1-Alpine SMP Tue, 08 Apr 2025 13:37:13 +0000 ppc64le GNU/Linux
2025-11-26 07:57:52 <ncopa> looks like there are some _CRYPTO_ knobs in kernel config that is enabled in aarch64 but not in ppc64le
2025-11-26 08:02:07 <huajingyun> Hmm, why not? (are there any risks involved in upgrading?)
2025-11-26 08:08:18 <ncopa> alwyas a small risk when rebooting remote machines
2025-11-26 08:14:44 <ncopa> ok. either we disable the ktls tests on ppc64le or we disable ktls on ppc64le altogether 
2025-11-26 08:23:49 <huajingyun> Yeah,any one of them,perhaps we do need to unlock the builder first and then deal with them later (after v3.23)
2025-11-27 07:53:41 <ncopa> clandmeter: morning! can you help reboot one of the pioneer boxes?
2025-11-28 18:54:45 <achill> im getting duplicated mails again by gitlab
2025-11-30 12:22:16 <Sertonix[m]> Is it possible to get a group namespace on gitlab.a.o for an alpine related project? Especially one that doesn't conflict with the alpine permissions.
2025-11-30 14:28:16 <ikke> Sertonix[m]: I think something should be possible, what is your idea? A namespace for ports?
2025-11-30 14:29:30 <Sertonix[m]> For unofficial ports, yes
2025-11-30 14:31:01 <ikke> I think we can create /alpine/ports, or even just /ports
2025-11-30 14:33:12 <Sertonix[m]> I think /ports would be better to keep it a bit seperated
2025-11-30 15:05:32 <Sertonix[m]> Do I need to do something before that is possible?
2025-11-30 15:15:21 <ikke> Can you make an issue on infra/infra with details of how it's intended to be used, and who is responsible for the namespace so that it doesn't come down to us?
2025-11-30 19:30:03 <tomalok> ikke https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/jobs/2120637 -- anything change with gitlab CI since the last time i did a tiny-cloud MR?
2025-11-30 19:34:02 <tomalok> ikke never mind seems to have been something transient, i'm getting a real test failure now after a couple retries
2025-11-30 19:50:39 <tomalok> popped up again, fwiw -- "ERROR: Job failed (system failure): prepare environment: setting up trapping scripts on emptyDir: error dialing backend: dial tcp 172.16.252.38:10250: i/o timeout. Check https://docs.gitlab.com/runner/shells/#shell-profile-loading for more information"
2025-11-30 19:52:54 <ikke> That node has some network connections I need to fix. I've disabled it for scheduling
2025-11-30 19:56:23 <tomalok> 👍
2025-11-30 19:56:34 <ikke> connection issues*