2026-01-01 02:14:36 <lotheac> ikke: nice :)
2026-01-01 02:14:53 <lotheac> and happy new year
2026-01-01 17:18:04 <ikke> (trying to cleanup space)
2026-01-01 18:16:30 <ikke> clandmeter: do you have a moment to check gbr2-dev1? apk-browser takes a lot of space and I haven't been able to reduce it yet
2026-01-01 18:16:44 <clandmeter> sure
2026-01-01 18:18:00 <clandmeter> edge db is a bit big
2026-01-01 18:46:40 <clandmeter> ikke: not sure whats going on
2026-01-01 18:46:53 <clandmeter> i tried to vacuum, but i think the tmp dir is not big enough
2026-01-01 18:55:22 <clandmeter> ah has nothing to do with tmp, disk is just full
2026-01-01 18:59:39 <clandmeter> ikke: can i add a tmp disk to the instnace?
2026-01-01 19:01:25 <clandmeter> i think i can
2026-01-01 19:02:41 <clandmeter> why is it not showing...
2026-01-01 19:22:17 <clandmeter> ikke: the instance has double the space
2026-01-01 19:22:27 <clandmeter> just didnt init on the boot disk
2026-01-01 19:22:48 <clandmeter> so we could increase the disk by 160 to 320
2026-01-01 20:57:54 <clandmeter> ikke: it looks like sqlite is not automatically deleting referenced files and such, growing the table to 230M rows
2026-01-01 21:15:24 <clandmeter> foreign_keys is not set anymore
2026-01-01 21:15:45 <clandmeter> so the on delete cascade is broken
2026-01-02 07:30:08 <ikke> clandmeter: thanks, so we have some cleanup to do
2026-01-02 07:30:30 <clandmeter> im writing a script
2026-01-02 07:30:42 <clandmeter> and will created a pr to fix it
2026-01-02 07:31:14 <clandmeter> pmos gitlab is defuct? 
2026-01-02 07:31:40 <clandmeter> ah today it works
2026-01-02 08:18:37 <ikke> How can we resize the disk? Fdisk does not report that it's 320g
2026-01-02 08:28:43 <clandmeter> i didnt resize it yet
2026-01-02 08:28:46 <clandmeter> i didnt know before
2026-01-02 08:28:49 <clandmeter> i just added a disk 
2026-01-02 08:29:12 <clandmeter> i think when i am finished we can remove this disk and resize the boot disk
2026-01-02 08:31:23 <clandmeter> disk is mounted on /mnt
2026-01-02 08:31:32 <clandmeter> and im running a cleanup script in tmux
2026-01-02 08:31:37 <clandmeter> will take some time
2026-01-02 08:32:18 <clandmeter> the result size will be 4.9G :)
2026-01-02 09:56:37 <ikke> clandmeter: nice, that's 10% of the old size
2026-01-02 10:21:51 <clandmeter> and all other db's have the same issue
2026-01-02 12:07:03 <clandmeter> ikke: going to replace the edge db
2026-01-02 12:09:22 <ikke> clandmeter: alright
2026-01-02 12:09:28 <clandmeter> done
2026-01-02 12:10:07 <clandmeter> will cleanup the others now
2026-01-02 13:31:40 <ikke> clandmeter: thanks!!
2026-01-02 14:06:56 <clandmeter> ikke: done
2026-01-02 14:09:06 <clandmeter> i added a local change to fix the issue
2026-01-02 14:22:14 <clandmeter> achill: ping
2026-01-02 14:22:39 <achill> pong
2026-01-02 14:27:47 <clandmeter> achill: not sure who managed pmos pkgs, but i guess you are also missing a setting.
2026-01-02 14:28:38 <achill> i guess postmarketos' infra team, i can tell them about it
2026-01-02 14:28:43 <clandmeter> foreign_keys
2026-01-02 14:28:55 <clandmeter> i think martijn left it out but did copy the logic
2026-01-02 14:29:15 <clandmeter> this will keep increasing the db over time
2026-01-02 14:29:53 <achill> ah alr
2026-01-02 15:55:22 <ikke> clandmeter: I've been thinking about migrating gitlab artifacts to linode object storage to alleviate disk usage
2026-01-02 15:56:28 <ikke> https://docs.gitlab.com/administration/cicd/job_artifacts/#using-object-storage
2026-01-02 18:09:59 <ikke> I must say I'm happy with our gitlab-test environment. Caught a change in configuration we needed to apply that would've prevented gitlab from starting :-)
2026-01-02 22:41:14 <achill> do the CI runners need some poking again now that gitlab was upgraded again?
2026-01-02 23:54:17 <omni> why is the aarch64 CI builder having issues fetching from busybox.net?
2026-01-03 05:30:22 <ikke> achill: no, bot there are some long running jobs
2026-01-03 05:35:04 <ikke> s/bot/but/
2026-01-03 07:22:13 <ikke> omni: I don't think it's specific to aarch64
2026-01-03 07:52:06 <achill> ah
2026-01-03 14:03:41 <ikke> hmm https://gitlab.alpinelinux.org/samitha.mdml/aports/-/commits/eb87367831330c8196e702948ebac924b7054062
2026-01-03 18:33:29 <omni> 404
2026-01-03 18:34:04 <ikke> It's priate
2026-01-03 18:34:07 <ikke> private
2026-01-03 18:40:54 <omni> ok, just between you and them
2026-01-03 18:48:04 <ikke> They're building some ISO in CI and want to store it in gitlab
2026-01-03 18:49:44 <achill> wtf
2026-01-03 18:51:43 <ikke> https://tpaste.us/ZKBg 
2026-01-03 18:52:49 <ikke> hyperland related
2026-01-03 18:52:55 <ikke> not sure why you need a dedicated iso for that
2026-01-03 19:00:02 <invoked> i dunno either. but the way they look at things in hyprland world i assume they'll try to be their own os at some point. it's like they never dep'd on wlroots before.
2026-01-03 20:00:37 <omni> I think mangowc may be a better hyprland, so I packaged it
2026-01-04 17:26:26 <gheja11> hey all, I am not sure if this is the right place to ask or if it is a known issue but I noticed it today that dl-cdn.alpinelinux.org resolves to 146.75.2.132 (AS54113 - Fastly, Inc.) from my IPs in Hungary and that IP cannot be reached (connection timeout), I noticed it about 2 hours ago and the problem still persist
2026-01-04 17:28:08 <gheja11> well, that was not a question after all :) anyway, I just wanted to let you know, I am using mirrors for now, they work fine
2026-01-04 17:54:57 <ikke> gheja11: it's not a known issue
2026-01-04 17:56:00 <ikke> gheja11: can you try a traceroute?
2026-01-04 18:02:50 <gheja11> sure, there are responses until the third hop - https://pastebin.com/raw/srmKEdyE (expires in 1 hour)
2026-01-04 18:06:15 <gheja11> and I can ping the neighbouring IPs, .131 and .133
2026-01-04 18:06:35 <gheja11> and connect to them over http
2026-01-04 18:09:30 <gheja11> I just checked from a different IP range and it worked from there, both the ping and the download
2026-01-04 18:10:33 <gheja11> so I guess my primary provider's IP range is blocked for some reason
2026-01-04 18:22:40 <ikke> Or perh a routing issue 
2026-01-04 18:22:44 <ikke> perhaps 
2026-01-05 06:19:55 <ikke> clandmeter: nld-bld-1 seems unresponsive again :(
2026-01-05 06:30:06 <ikke> ncopa: can you check the date on `shared-runner nor-ci-1`?
2026-01-05 09:53:20 <ncopa> morning! will do
2026-01-05 10:00:38 <ncopa> it looks like it is building: https://gitlab.alpinelinux.org/jvvv/aports/-/jobs/2161542
2026-01-05 10:55:34 <gheja11> I contacted my provider (regarding the connection timeout for dl-cdn.alpinelinux.org 146.75.2.132), they said they contacted Fastly and solved the issue together - now it works from their IP range too
2026-01-05 10:55:51 <gheja11> thanks for the help, @ikke!
2026-01-05 10:56:09 <ikke> Great, thanks for the update
2026-01-05 10:56:43 <ikke> ncopa: I saw some jobs failing due to an tls verification error
2026-01-05 11:46:09 <ncopa> maybe time was off?
2026-01-05 11:50:13 <ikke> yes, that's why I was asking to check the date
2026-01-05 11:54:19 <ncopa> its correct now
2026-01-05 11:54:29 <ikke> Thanks
2026-01-05 22:01:57 <omni> the arm/aarch64 builders seem to default to ipv6 and not try ipv4 when failing
2026-01-05 22:10:52 <omni> when and how are archives added to distfiles?
2026-01-05 22:13:46 <omni> I've mentioned in #meli that their AAAA address doesn't seem to be reachable
2026-01-06 05:32:07 <ikke> omni: that's because we use bb wget 
2026-01-06 05:32:20 <ikke> Which does not fall back 
2026-01-06 05:54:13 <ikke> omni: Once per day the builders sync their distfiles withdistfiles.a.o
2026-01-06 10:14:15 <omni> ikke: ok, thanks!
2026-01-06 10:17:45 <omni> they have reloaded their firewall and the ipv6 address is reachable now
2026-01-06 10:29:33 <ikke> Great
2026-01-06 10:52:33 <ncopa> i think we have corrupt ext4 filesystem on gitlab-runner-ppc64le
2026-01-06 10:52:45 <ncopa> https://build.alpinelinux.org/buildlogs/build-edge-ppc64le/main/linux-lts/linux-lts-6.18.3-r0.log
2026-01-06 10:53:11 <ncopa> we should probably reboot the ppc64le host and run filesystem check
2026-01-06 10:53:24 <ncopa> or I can remount it read-only and do fsck.ext4
2026-01-06 10:54:07 <ncopa>  /dev/sda4: ********** WARNING: Filesystem still has errors **********
2026-01-06 10:56:11 <ncopa> ikke: do you think we should reboot it, or just try fsck without reboot?
2026-01-06 10:57:38 <ncopa> I suppose it is risky to try apk upgrade with a corrupt fs
2026-01-06 10:58:11 <ncopa> I will try fsck it online
2026-01-06 11:00:45 <ncopa> didnt work. I need to reboot it
2026-01-06 11:00:56 <ncopa>  10:59:55 up 3 days, 15:08,  0 users,  load average: 0.44, 22.51, 38.08
2026-01-06 11:00:56 <ncopa> gitlab-runner-ppc64le [~]# uptime
2026-01-06 11:01:14 <ncopa> apk audit --system also did not detect anything changed
2026-01-06 11:01:19 <ncopa> im rebooting it
2026-01-06 11:16:14 <ncopa> build-edge-ppc64le should be back up now
2026-01-06 11:21:38 <ikke> Thanks!
2026-01-06 11:22:20 <ikke> ncopa: how did you schedule the fsck?
2026-01-06 11:27:29 <ncopa> 1) ensure fsck.ext4 is installed. 2) rc-update add fsck boot 3) reboot
2026-01-06 11:27:57 <ncopa> I suppose we should upgrade it to v3.22 or something
2026-01-06 11:28:39 <ncopa> i noticed that gitlab runners does not work well with alpine v3.23. docker version mismiatch
2026-01-06 11:29:06 <ikke> Hmm, ok
2026-01-06 14:36:48 <ikke> clandmeter: thanks
2026-01-06 14:44:53 <clandmeter> rebooting bld1
2026-01-06 14:50:00 <invoked> mds matrix bridge is having issues
2026-01-07 07:02:51 <ikke> clandmeter: I've increased the mirror volume on deu1-t1-1 we have about 600G left
2026-01-07 07:03:04 <ikke> In the vg I mean
2026-01-07 07:03:09 <clandmeter> ok
2026-01-07 07:03:23 <clandmeter> we should also fix the volume on pkgs
2026-01-07 07:03:31 <ikke> Right
2026-01-07 07:03:48 <ikke> Perhaps we can reduce the storage used for dev containers on that server
2026-01-07 07:04:22 <clandmeter> we can kill the second disk
2026-01-07 07:04:31 <clandmeter> and extend the first by 160G
2026-01-07 07:04:41 <clandmeter> i only used it for backup
2026-01-07 07:04:56 <ikke> I meant on deu-t1-1
2026-01-07 07:05:00 <clandmeter> oh you mean the mirror
2026-01-07 07:06:14 <ikke> Yes
2026-01-07 07:06:29 <ikke> That server also has x86_64 dev containers
2026-01-07 07:06:35 <ikke> The volume for that is 1T now
2026-01-07 07:14:19 <clandmeter> yeah thats the german one i remember now
2026-01-08 09:54:08 <ikke> clandmeter: What do you think about moving distfiles to object storage? I think the type of storage would be ideal, but it does require us to adjust how it's stored and how we expose it
2026-01-08 09:54:46 <ikke> (S3 like storage)
2026-01-08 13:04:18 <clandmeter> ikke: ok, you will need some scripts to support it i guess?
2026-01-08 13:09:22 <ikke> There is a fuse driver, if that works reliably, that could be used with minimal impact, otherwise you indeed need something to translate it
2026-01-08 13:11:40 <ikke> And not sure if those drivers would work with Linode
2026-01-08 13:13:26 <achill> funnily enouh i had that same idea yesterday evenin for our KDE nihtly distfiles in postmarketOS since they are enourmous
2026-01-08 13:14:02 <ikke> I was thinking about it when we had fs corruption
2026-01-08 14:14:42 <ikke> clandmeter: fyi, I'm thinking about doing something similar for gitlab artifacts.
2026-01-08 14:14:51 <ikke> But gitlab supports it natively
2026-01-08 15:33:14 <ikke> clandmeter: ^
2026-01-08 20:40:35 <clandmeter> sorry both rv boxes are offline
2026-01-08 20:40:45 <clandmeter> will fix tomorrow
2026-01-08 20:43:12 <clandmeter> ikke: i wonder if it makes more sense to have our scripts be aware of object storage instead of mimic posix
2026-01-08 20:45:55 <ikke> That would be a better option yes
2026-01-08 20:46:29 <ikke> We would also need something to expose it to a webserver though 
2026-01-08 20:56:02 <clandmeter> https://blog.min.io/filesystem-on-object-store-is-a-bad-idea/
2026-01-08 21:20:46 <ikke> Yeah, it's not ideal
2026-01-08 21:22:09 <ikke> But for distfiles, except for listing a lot of files in a 'directory', it does not require a lot of performance
2026-01-09 16:23:05 <iggy>  Caddy supports s3 backends natively. I'd guess other servers do too. And most object stores have some way to serve files directly from them (aws does, looks like linode object storage does). It looks like there are also some options for serving from s3 to rsync if you need that
2026-01-09 16:27:02 <ikke> iggy: Hmm, interesting, thanks for sharing
2026-01-09 16:27:19 <ikke> The builders currently rsync their files to distfiles.a.o
2026-01-09 16:30:34 <iggy> do they use rsync protocol or rsync+ssh?
2026-01-09 16:32:05 <ikke> rsync + ssh
2026-01-09 16:32:19 <ikke> they upload over ssh
2026-01-09 16:32:47 <ikke> But that's because ssh is used as authentication
2026-01-09 16:48:29 <iggy> yeah, figured, there are less options that way, but still some viable... and worst case scenario, you go the s3fs+regular rsync daemon option for that part of it (but switching from rsync to a standard s3 client is probably a better option)
2026-01-09 17:03:55 <mio> is this atools-go sigsegv in the lint job a temporary error? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2166776
2026-01-09 17:04:41 <ikke> mio: probably not
2026-01-09 17:04:48 <mio> same runner seems to be fine a few hours ago, just checking
2026-01-09 17:05:26 <ikke> mio: probably related to something in the APKBUILD
2026-01-09 17:06:26 <mio> ikke: ah, thanks
2026-01-09 17:27:02 <ikke> mio: ah, I already fixed that segfault, but no new release yet
2026-01-09 17:27:56 <ikke> https://gitlab.alpinelinux.org/alpine/infra/atools-go/-/merge_requests/78
2026-01-09 17:29:15 <ikke> Due to things like this: export "GDAL_DRIVER_${name}_PLUGIN_INSTALLATION_MESSAGE=You may install it with 'apk add $i'"
2026-01-09 17:33:48 <mio> okay, good that's already known and fixed :)
2026-01-09 17:34:52 <ikke> Will release a new version
2026-01-09 17:35:12 <mio> great, thanks
2026-01-09 18:03:26 <ikke> mio: https://gitlab.alpinelinux.org/alpine/infra/docker/apkbuild-lint-tools/-/merge_requests/32
2026-01-09 18:13:34 <ikke> mio: retrying that job, and it now passes
2026-01-09 18:54:14 <mio> looking good :)
2026-01-09 21:13:39 <ncopa> Learned about https://edent.github.io/SuperTinyIcons/ today. I wonder if we can use the alpine logo as our official 
2026-01-09 21:14:59 <ikke> The complete logo also contains the name
2026-01-10 14:38:08 <ncopa> today I learned that the mermaid version used in gitlab does not support block or architecture diagrams
2026-01-10 17:12:31 <omni> looks like the loongarch64 CI builder is out of disk
2026-01-10 17:13:35 <ikke> You say 20G is not enough to build a package? :P
2026-01-10 22:47:33 <omni> I see "No space left on device" in the attempted main/rust build here: https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2167849
2026-01-11 07:38:48 <ikke> omni: right, so rust does need more than 20G :)
2026-01-11 07:38:54 <ikke> omni: for the record, I've cleaned it up
2026-01-11 12:45:19 <omni> ikke: thanks
2026-01-11 12:45:58 <omni> that, or that it built at the same time as something that also used a lot of space?
2026-01-11 12:46:22 <ikke> possibly
2026-01-14 15:40:33 <achill> gitlab 500
2026-01-14 15:40:42 <achill> ah works again i think
2026-01-14 15:41:00 <achill> yeah lol i guess it was gone for complete 3 seconds
2026-01-14 18:12:12 <Sertonix[m]> Will the loongarch64 builder catch up on it's own eventually or is some manual intervention needed?
2026-01-14 18:13:45 <ikke> Sertonix[m]: build-edge-loongarch64: failed to build qt6-qtwebengine: https://build.alpinelinux.org/buildlogs/build-edge-loongarch64/community/qt6-qtwebengine/qt6-qtwebengine-6.10.1-r4.log
2026-01-14 19:28:27 <ikke> Sertonix[m]: that is to say, at least some manual intervention is required
2026-01-14 20:24:45 <ncopa> cycle: freerdp -> webkit2gtk-4.1 -> kwallet -> kconfig -> xwayland-run -> weston -> freerdp
2026-01-14 20:24:45 <ncopa> cycle: ghostscript -> gtk+3.0 -> gtk4.0 -> gst-plugins-bad -> zbar -> imagemagick -> ghostscript
2026-01-14 20:24:45 <ncopa> cycle: py3-jaraco.collections -> py3-jaraco.text -> py3-jaraco.context -> py3-jaraco.test -> py3-jaraco.collections
2026-01-14 20:25:03 <ncopa> wrong channel sorry
2026-01-15 12:08:30 <ikke> lotheac: I did make some changes to the agent config. Most notably: concurrency has to be set via the CR, it gets ignored when it's provided via the runner config itself
2026-01-15 13:30:06 <lotheac> oh, okay
2026-01-15 13:34:09 <lotheac> those changes look reasonable to me
2026-01-15 13:37:09 <ikke> What supprised me a bit (especially after reading https://docs.gitlab.com/runner/configuration/advanced-configuration/#long-polling-issues), is that limit on a runner level get overridden by concurrency)
2026-01-15 13:37:44 <ikke> concurrency = 10, runner limit=4 -> max_builds=10
2026-01-15 13:38:49 <lotheac> huh
2026-01-15 13:38:56 <lotheac> i would have expected the opposite
2026-01-15 13:38:59 <ikke> yeah, me too
2026-01-15 13:39:40 <ikke> The behavior is confusing
2026-01-15 13:40:01 <ikke> But the runner kept accepting jobs, even if there were 4 jobs running already (and then failing due to lack of resources)
2026-01-15 23:41:34 <adrian> loongarch builder could probably use an apk fix, it seems like it's in a broken state
2026-01-16 17:37:30 <ncopa> oh great the pioneer machine is failing again
2026-01-16 19:59:11 <ncopa> I have a user that registered to wiki.a.o in 20 July 2009 and cannot recover the password now
2026-01-16 19:59:20 <ncopa> I cannot find out which email he used
2026-01-16 19:59:41 <ncopa> how do we handle those situations?
2026-01-16 20:05:33 <ikke> Good question, we would need some proof of ownership
2026-01-17 21:23:50 <Sertonix[m]> Can somebody nudge the armv7 builder? podman should pass when rebuild
2026-01-18 12:46:57 <achill> also the armhf, riscv64, x86 builder don't get new rebuilds at all
2026-01-18 15:33:25 <ikke> riscv64 build host is awol
2026-01-18 15:33:34 <ikke> other builders should be operating again
2026-01-18 15:55:37 <achill> merci
2026-01-18 20:42:12 <achill> hmm registry.alpinelinux.org returns 500 for me
2026-01-18 20:42:17 <achill> ➜  ~ podman run --rm -it registry.alpinelinux.org/alpine/infra/docker/build-base:edge-ppc64le
2026-01-18 20:42:17 <achill> Trying to pull registry.alpinelinux.org/alpine/infra/docker/build-base:edge-ppc64le...
2026-01-18 20:42:17 <achill> Error: unable to copy from source docker://registry.alpinelinux.org/alpine/infra/docker/build-base:edge-ppc64le: initializing source docker://registry.alpinelinux.org/alpine/infra/docker/build-base:edge-ppc64le: reading manifest edge-ppc64le in registry.alpinelinux.org/alpine/infra/docker/build-base: received unexpected HTTP status: 500 Internal Server Error
2026-01-18 20:44:42 <ikke> database connection pool exhausted
2026-01-18 20:45:08 <achill> oof
2026-01-18 20:46:01 <ikke> It's working again now
2026-01-18 20:46:17 <achill> that was quick thanks
2026-01-18 20:47:00 <ikke> No fix, just a temporary issue
2026-01-18 21:47:36 <Sertonix[m]> Is the armhf builder stuck again?
2026-01-18 21:48:18 <achill> oof, yeah looks so
2026-01-19 01:48:21 <omni> !96103 loongarch64 configured in a special way or something else?
2026-01-20 12:57:18 <achill> database goes brrrr
2026-01-21 14:16:48 <ikke> Sorry for the noise, trying to figure out what is causing it.
2026-01-21 17:25:29 <mio> could someone please check on the edge s390x builder? looks stuck
2026-01-21 17:26:17 <ikke> done
2026-01-21 17:26:27 <mio> thanks!
2026-01-22 09:05:00 <ikke> When this happens, I do see timeouts happening on gitlab, so it's a real issue
2026-01-23 13:27:41 <eletrotupi> hello!
2026-01-23 13:28:25 <eletrotupi> any mechanism to deal with anitya informing packages that aren't lts?
2026-01-23 13:28:52 <eletrotupi> prometheus is doing a lot of releases lately, but we're locked with the lts version of it's package
2026-01-23 13:29:19 <eletrotupi> which I updated yesterday even, but today they released a new version which anitya already sent me a message
2026-01-23 13:29:40 <eletrotupi> and i think it also appears on the pkgs.a.o page as out-of-date which is not entirely correct
2026-01-23 14:24:37 <ikke> There exists, as far as I know, nothing  for that at the moment
2026-01-23 14:47:59 <mio> is anyone else getting a "secure site not available" or similar error with dup.pw links?
2026-01-23 14:49:37 <mio> e.g. http://dup.pw/alpine/aports/4555a1202b20
2026-01-23 15:17:51 <achill> mio: yeah can reproduce, looks broken
2026-01-23 15:19:57 <mio> okay, thanks for confirming ... it's since switched to a self-signed cert error so maybe it's being looked into
2026-01-23 16:19:30 <ikke> huh
2026-01-23 16:21:00 <ikke> domain expired
2026-01-24 18:01:00 <ikke> lotheac: I was looking at building docker images in kubernetes again. I found this article: https://kubernetes.web.cern.ch/blog/2025/06/19/rootless-container-builds-on-kubernetes/
2026-01-24 18:01:28 <ikke> It relies on setting hostUsers to false to allow user namespaces
2026-01-24 18:01:54 <ikke> Apparently, the containerd that k0s uses does not support user namespaces with the kubernetes version we use
2026-01-24 18:02:05 <ikke> https://github.com/containerd/containerd/blob/main/docs/user-namespaces/README.md#stack-requirements
2026-01-24 18:02:18 <ikke> But they are working on updating containerd to 2.0 in k0s: https://github.com/k0sproject/k0s/pull/7011
2026-01-25 01:39:43 <lotheac> buildah can build images in priv constrained envs, have you considered that?
2026-01-25 01:41:58 <lotheac> other than that there are things like docker:dind which iirc do not require changing hostUsers
2026-01-25 01:42:33 <lotheac> i use that at one of my clients for self-hosted github runners which rely on doing a bunch of docker stuff
2026-01-25 01:44:43 <lotheac> those clusters use cri-o though
2026-01-25 01:49:09 <lotheac> i'm under the impression that buildah can build stuff *without* being privileged: true
2026-01-25 01:49:34 <lotheac> but maybe i need to verify that
2026-01-25 12:44:16 <lotheac> seems i was at least partially mistaken, buildah in kube gives "Error: error writing "0 0 4294967295\n" to /proc/22/uid_map: write /proc/22/uid_map: operation not permitted"
2026-01-25 12:55:08 <lotheac> so privileged: true is actually required and my memory isn't great :)
2026-01-25 12:57:58 <lotheac> ikke: this works though, with or without hostUsers: false. kubectl run --overrides='{"spec":{"volumes":[{"name":"containers","emptyDir":{}}],"containers":[{"name":"foo","image":"docker.io/library/alpine:latest","volumeMounts":[{"name":"containers","mountPath":"/var/lib/containers"}],"securityContext":{"privileged":true},"args":["/bin/sh","-c","apk add buildah netavark && echo FROM alpine:latest > /tmp/Containerfile && buildah build /tmp"]}]}}' --rm
2026-01-25 12:57:58 <lotheac>  --image alpine:latest -i foo
2026-01-25 12:59:03 <lotheac> two things are important: 1) privileged=true container so that it can create uid mappings, 2) /var/lib/containers needs to be a volume (or emptydir) so that it can create overlayfs stuff over it
2026-01-25 12:59:19 <lotheac> (or ~/.local/share/containers if building as non-root)
2026-01-25 13:20:32 <ikke> Right, so I think when user namespaces are possible, we no longer need privileged
2026-01-25 13:20:56 <lotheac> i think privileged is still needed in that case (the article you linked says so as well)
2026-01-25 13:21:17 <lotheac> but it's privileged in a less-privileged userns
2026-01-25 13:43:14 <lotheac> (also i did test hostUsers=false privileged=false on cri-o, but it did not work)
2026-01-25 23:06:47 <omni> this is odd https://gitlab.alpinelinux.org/bratkartoffel/aports/-/jobs/2191522#L8640
2026-01-25 23:07:04 <omni> /bin/bash: fork: retry: Resource temporarily unavailable
2026-01-25 23:07:26 <omni> only for openjdk25 and on x86_64
2026-01-25 23:08:11 <omni> do we dare get !96415 in and see if it passes on the package builder?
2026-01-26 14:24:26 <mio> is there a way to allocate more resources to a ci job that is getting oom killed? e.g. https://gitlab.alpinelinux.org/androw/aports/-/jobs/2190558#L1156 (testing/rio)
2026-01-26 14:25:11 <achill> you can reduce the jobs
2026-01-26 14:26:15 <mio> okay, thanks
2026-01-26 16:11:58 <ikke> clandmeter: seems like it's gone again ^
2026-01-26 16:56:40 <iggy> buildah is slow. I measured it to be about 4x slower than docker on the images I was building for work. Not sure if that matters
2026-01-26 16:57:17 <ikke> There's only one case where it would matter, but there the difference could be painful
2026-01-26 17:00:26 <iggy> I take that back, sorry. It was Kaniko that was super slow. Never tested buildah here. We did use buildah at a previous job which is where I confused them in my head.
2026-01-26 17:00:54 <iggy> (had to go check git history to see which one we used... should have done that first)
2026-01-26 17:06:43 <ikke> iggy: ok, good to know
2026-01-26 21:04:23 <ikke> Lots of requests
2026-01-26 21:14:07 <ikke> Something scraping us from a pool of azure addresses
2026-01-26 21:48:36 <ncopa> what are they scraping? gitlab.a.o?
2026-01-26 21:48:42 <ikke> yes
2026-01-26 21:49:15 <ikke> I'm strongly ratelimiting many microsoft prefixes
2026-01-26 21:50:50 <ncopa> thanks
2026-01-26 21:51:08 <ikke> Traffic has been ramping back already
2026-01-26 21:52:07 <ikke> https://paste.pictures/jm0FIR9o63.png
2026-01-26 21:53:37 <ncopa> it was go-away that did the pg requests?
2026-01-26 21:53:48 <ikke> No
2026-01-26 21:53:57 <ikke> Or at least, maybe partially
2026-01-26 21:54:05 <ikke> go-away is just in front of nginx
2026-01-26 21:54:35 <ikke> but part of validation is that it checks if you are logged in, but that on it's own already is quite a bit of traffic
2026-01-26 21:55:09 <ikke> Yesterday I increased the max connections of postgres, and I think we didn't hit the max this time
2026-01-26 21:55:32 <ikke> max reached was 232, and I have set the max to 250
2026-01-26 21:59:48 <ikke> That is to say, although the load increased, gitlab remained responsive
2026-01-27 13:13:27 <fabled> ikke: the apk-tools x86 CI/CD is not working. did something change with the gitlab runner tags?
2026-01-27 13:14:36 <ikke> Yes, it did change a bit, let me take a look 
2026-01-27 13:17:44 <fabled> i think it broke few weeks ago, and i thought it was temporary glitch. but its still persisting
2026-01-27 13:18:10 <fabled> basically things timeout/are stuck. no runner picks up the ci job for x86
2026-01-27 13:18:15 <fabled> on apk-tools repo
2026-01-27 13:18:43 <ikke> Hmm, the runner with those tags is avaible for apk-tools
2026-01-27 13:20:16 <fabled> the tags I speciy are (for the problem builds): docker-alpine x86
2026-01-27 13:21:21 <fabled> https://gitlab.alpinelinux.org/alpine/apk-tools/-/jobs/2193898
2026-01-27 13:21:26 <fabled> Go to project CI settings
2026-01-27 13:21:26 <fabled> This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch , or no runners that match all of the job's tags: docker-alpine x86
2026-01-27 13:22:30 <fabled> "Turn on instance runners for this project" is set on, and looking at the instances shows no x86
2026-01-27 13:22:42 <ikke> The docker-alpine tag is missing on one of the  runners
2026-01-27 13:32:05 <achill> can somebody please re-open https://gitlab.alpinelinux.org/alpine/council/-/issues/6
2026-01-27 15:16:07 <fabled> ikke: is there a particular reason https://gitlab.alpinelinux.org/alpine/apk-tools/-/runners/246 is missing the docker-alpine tag?
2026-01-27 15:16:33 <fabled> should i remove "docker-alpine" tag and use "ci-build" instead?
2026-01-27 15:26:56 <fabled> seems that might work, at least x86 build passed now.
2026-01-27 15:28:24 <ikke> Temporary 
2026-01-27 15:28:54 <ikke> Has to do with the amount of resources it requests 
2026-01-27 15:50:12 <fabled> ikke: apk build is lean. What is the difference of the tags? I'd like to fix them based on what the tags mean.
2026-01-27 15:52:55 <fabled> Confirmed. Build with "ci-build" tag worked.
2026-01-27 15:58:39 <ikke> fabled: the problem is that a docker-alpine runner for x86 is missing 
2026-01-27 16:05:44 <fabled> ikke: What do the two tags mean? When each should be used?
2026-01-27 16:07:04 <fabled> Looking at aports. It uses ci-build for the builds. And docker-alpine for the job generator job only
2026-01-27 16:07:16 <fabled> So it never uses x86+docker-alpine.
2026-01-27 16:07:40 <fabled> And since ci-build is used for the actual build.... I think it makes sense for me to use it too. Right?
2026-01-27 16:14:23 <ikke> It's a recent change. ci-build is for building aports packages, docker-alpine for general jobs 
2026-01-27 16:15:05 <ikke> For most arches, it will be the same runner, but for x86 and x86_64, they are different 
2026-01-27 16:15:42 <ikke> I created the separate runner for x86_64, but it was not done for x86
2026-01-27 16:15:55 <ikke> I will create one in a bit 
2026-01-27 16:32:25 <fabled> No. I'll use ci-build 
2026-01-27 16:32:44 <fabled> Or is there some reason to not use it?
2026-01-27 16:33:15 <fabled> I'd like to use runners that aports uses too. So we do not end up in this situation without anyone noticing 
2026-01-27 16:33:29 <fabled> And you don't need to create extra runners for me
2026-01-27 16:33:51 <fabled> And really I do use it for "ci build" purpose.
2026-01-27 16:36:14 <fabled> From your explanation: it's more of a build job than "general job"
2026-01-27 16:37:04 <ikke> fabled: reason not to use it: contention with other larger jobs
2026-01-27 16:37:41 <ikke> We limit those jobs to about 2 per node, with 2 nodes for x86 atm
2026-01-27 16:37:54 <ikke> so if there are 4 jobs running, there is no more space
2026-01-27 16:38:02 <ikke> apk-tools does not need that much
2026-01-27 16:39:48 <fabled> ikke: ok. Fair enough. Can you fix the tag today/tomorrow? It's blocking APK tools release currently 
2026-01-27 16:40:43 <ikke> yes, will fix it in a bit
2026-01-27 16:47:17 <fabled> ikke: thanks!
2026-01-27 17:07:30 <ikke> fabled: there is an alternative in-between solution
2026-01-27 17:09:35 <ikke> If you would add something like https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/.gitlab-ci.yml?ref_type=heads#L12-13 to the job, it would allow at least kubernetes to schedule it. The ci-build runner itself is still limited to 4 builds, but the jobs should at least wait until there is a spot free
2026-01-27 17:14:04 <fabled> ikke: I'm confused, how does that help if I use docker-alpine tag? There is no x86 runners with tag 
2026-01-27 17:14:18 <fabled> That tag*
2026-01-27 17:14:24 <fabled> I can test it though 
2026-01-27 17:14:27 <ikke> fabled: That would be with the ci-build tag then
2026-01-27 17:14:38 <fabled> Everything works with ci-build tag 
2026-01-27 17:14:56 <ikke> Yes, but not when things are busy
2026-01-27 17:15:11 <fabled> https://gitlab.alpinelinux.org/alpine/apk-tools/-/merge_requests/388
2026-01-27 17:15:17 <fabled> Oh, I see
2026-01-27 17:16:52 <fabled> I can add those as global variables
2026-01-27 17:18:17 <ikke> fabled: right now, there are 4 x86 jobs running for example
2026-01-27 17:22:19 <ikke> yeah yeah, please be patient :P
2026-01-27 17:25:21 <ikke> fabled: the x86 docker-alpine runner should now be available as well
2026-01-27 17:26:44 <ikke> It should at least have capacity for more smaller jobs
2026-01-27 17:48:22 <ikke> fabled: so 2 options: 1. Keep docker-alpine with a dedicated runner, but one that is not used a lot, 2: switch to ci-build, reduce resource requests, but have to deal with job limits
2026-01-27 17:49:32 <fabled> No strong opinion. As long as stuff works I'm happy. Occasional delays are not a problem; that happens with the other runners like s390, ppc64 and arm 32bits too
2026-01-27 17:50:03 <ikke> Ok, in any case, you can always switch to the other option
2026-01-27 20:26:50 <ncopa> I wonder what is wrong with nld-bld-1 machine (pioneer)
2026-01-27 20:27:17 <ncopa> sometimes gcc segfaults, or it deadlocks
2026-01-27 20:27:37 <ncopa> and it happens randomly, so it is hardware related
2026-01-27 20:28:03 <ncopa> but is it the nvme? RAM? cpu?
2026-01-27 20:28:32 <ncopa> one interesting observation today was that nodejs completed at one point
2026-01-27 20:28:47 <ncopa> and at that point another build was deadlocked
2026-01-27 20:30:07 <ncopa> it makes me think that maybe it is one of the 64 cores that is broken? and the deadlock kept the broken core busy, so nodejs could complete
2026-01-27 20:30:30 <ncopa> or maybe it held the broken memory area allocated
2026-01-27 20:30:52 <ncopa> I wonder if we could disable the broken cpu core if that is the case
2026-01-27 20:31:13 <ncopa> or maybe we could get new RAM - which is not cheap nowadays...
2026-01-27 21:45:20 <ncopa> ikke: should we also merge this while at it? https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/122
2026-01-28 14:20:18 <eletrotupi> ikke: regarding anitya, I wonder if we can patch it, if it an Alpine thingie
2026-01-28 14:20:44 <eletrotupi> because it is hella annoying considering that we probably have a lot of stuff "stuck" as LTS
2026-01-28 14:20:59 <ikke> eletrotupi: we only control the integration into apk-browser 
2026-01-28 14:21:09 <eletrotupi> how does that work?
2026-01-28 14:21:26 <eletrotupi> oh, anitya is a fedora thing
2026-01-28 14:28:17 <ikke> Yes
2026-01-28 14:35:59 <eletrotupi> hm, and where is that code located? I suppose we could patch that instead? I mean, considering that the emails come from Alpine
2026-01-28 14:48:57 <ikke> Yes, but not sure if we want to build a secondary rule engine or hardcode fixes for specific packages 
2026-01-28 15:29:31 <eletrotupi> I've opened a issue on their tracker which they said indeed they don't offer support and have considered. So I guess, that the idea would be to patch anitya to offer the LTS info, and then patch Alpine monitoring to have a list of packages that it's set to LTS instead of the latest release
2026-01-28 20:54:40 <Sertonix[m]> Is build-3-21-armv7 stuck? It seems to be "pulling git" for a long time
2026-01-28 20:56:59 <ikke> It was not stuck, but it seems interrupted while updating the repo
2026-01-31 03:07:30 <omni> is build-3-22-x86_64 stuck?