2026-01-01 02:14:36 <lotheac> ikke: nice :)
2026-01-01 02:14:53 <lotheac> and happy new year
2026-01-01 17:18:04 <ikke> (trying to cleanup space)
2026-01-01 18:16:30 <ikke> clandmeter: do you have a moment to check gbr2-dev1? apk-browser takes a lot of space and I haven't been able to reduce it yet
2026-01-01 18:16:44 <clandmeter> sure
2026-01-01 18:18:00 <clandmeter> edge db is a bit big
2026-01-01 18:46:40 <clandmeter> ikke: not sure whats going on
2026-01-01 18:46:53 <clandmeter> i tried to vacuum, but i think the tmp dir is not big enough
2026-01-01 18:55:22 <clandmeter> ah has nothing to do with tmp, disk is just full
2026-01-01 18:59:39 <clandmeter> ikke: can i add a tmp disk to the instnace?
2026-01-01 19:01:25 <clandmeter> i think i can
2026-01-01 19:02:41 <clandmeter> why is it not showing...
2026-01-01 19:22:17 <clandmeter> ikke: the instance has double the space
2026-01-01 19:22:27 <clandmeter> just didnt init on the boot disk
2026-01-01 19:22:48 <clandmeter> so we could increase the disk by 160 to 320
2026-01-01 20:57:54 <clandmeter> ikke: it looks like sqlite is not automatically deleting referenced files and such, growing the table to 230M rows
2026-01-01 21:15:24 <clandmeter> foreign_keys is not set anymore
2026-01-01 21:15:45 <clandmeter> so the on delete cascade is broken
2026-01-02 07:30:08 <ikke> clandmeter: thanks, so we have some cleanup to do
2026-01-02 07:30:30 <clandmeter> im writing a script
2026-01-02 07:30:42 <clandmeter> and will created a pr to fix it
2026-01-02 07:31:14 <clandmeter> pmos gitlab is defuct? 
2026-01-02 07:31:40 <clandmeter> ah today it works
2026-01-02 08:18:37 <ikke> How can we resize the disk? Fdisk does not report that it's 320g
2026-01-02 08:28:43 <clandmeter> i didnt resize it yet
2026-01-02 08:28:46 <clandmeter> i didnt know before
2026-01-02 08:28:49 <clandmeter> i just added a disk 
2026-01-02 08:29:12 <clandmeter> i think when i am finished we can remove this disk and resize the boot disk
2026-01-02 08:31:23 <clandmeter> disk is mounted on /mnt
2026-01-02 08:31:32 <clandmeter> and im running a cleanup script in tmux
2026-01-02 08:31:37 <clandmeter> will take some time
2026-01-02 08:32:18 <clandmeter> the result size will be 4.9G :)
2026-01-02 09:56:37 <ikke> clandmeter: nice, that's 10% of the old size
2026-01-02 10:21:51 <clandmeter> and all other db's have the same issue
2026-01-02 12:07:03 <clandmeter> ikke: going to replace the edge db
2026-01-02 12:09:22 <ikke> clandmeter: alright
2026-01-02 12:09:28 <clandmeter> done
2026-01-02 12:10:07 <clandmeter> will cleanup the others now
2026-01-02 13:31:40 <ikke> clandmeter: thanks!!
2026-01-02 14:06:56 <clandmeter> ikke: done
2026-01-02 14:09:06 <clandmeter> i added a local change to fix the issue
2026-01-02 14:22:14 <clandmeter> achill: ping
2026-01-02 14:22:39 <achill> pong
2026-01-02 14:27:47 <clandmeter> achill: not sure who managed pmos pkgs, but i guess you are also missing a setting.
2026-01-02 14:28:38 <achill> i guess postmarketos' infra team, i can tell them about it
2026-01-02 14:28:43 <clandmeter> foreign_keys
2026-01-02 14:28:55 <clandmeter> i think martijn left it out but did copy the logic
2026-01-02 14:29:15 <clandmeter> this will keep increasing the db over time
2026-01-02 14:29:53 <achill> ah alr
2026-01-02 15:55:22 <ikke> clandmeter: I've been thinking about migrating gitlab artifacts to linode object storage to alleviate disk usage
2026-01-02 15:56:28 <ikke> https://docs.gitlab.com/administration/cicd/job_artifacts/#using-object-storage
2026-01-02 18:09:59 <ikke> I must say I'm happy with our gitlab-test environment. Caught a change in configuration we needed to apply that would've prevented gitlab from starting :-)
2026-01-02 22:41:14 <achill> do the CI runners need some poking again now that gitlab was upgraded again?
2026-01-02 23:54:17 <omni> why is the aarch64 CI builder having issues fetching from busybox.net?
2026-01-03 05:30:22 <ikke> achill: no, bot there are some long running jobs
2026-01-03 05:35:04 <ikke> s/bot/but/
2026-01-03 07:22:13 <ikke> omni: I don't think it's specific to aarch64
2026-01-03 07:52:06 <achill> ah
2026-01-03 14:03:41 <ikke> hmm https://gitlab.alpinelinux.org/samitha.mdml/aports/-/commits/eb87367831330c8196e702948ebac924b7054062
2026-01-03 18:33:29 <omni> 404
2026-01-03 18:34:04 <ikke> It's priate
2026-01-03 18:34:07 <ikke> private
2026-01-03 18:40:54 <omni> ok, just between you and them
2026-01-03 18:48:04 <ikke> They're building some ISO in CI and want to store it in gitlab
2026-01-03 18:49:44 <achill> wtf
2026-01-03 18:51:43 <ikke> https://tpaste.us/ZKBg 
2026-01-03 18:52:49 <ikke> hyperland related
2026-01-03 18:52:55 <ikke> not sure why you need a dedicated iso for that
2026-01-03 19:00:02 <invoked> i dunno either. but the way they look at things in hyprland world i assume they'll try to be their own os at some point. it's like they never dep'd on wlroots before.
2026-01-03 20:00:37 <omni> I think mangowc may be a better hyprland, so I packaged it
2026-01-04 17:26:26 <gheja11> hey all, I am not sure if this is the right place to ask or if it is a known issue but I noticed it today that dl-cdn.alpinelinux.org resolves to 146.75.2.132 (AS54113 - Fastly, Inc.) from my IPs in Hungary and that IP cannot be reached (connection timeout), I noticed it about 2 hours ago and the problem still persist
2026-01-04 17:28:08 <gheja11> well, that was not a question after all :) anyway, I just wanted to let you know, I am using mirrors for now, they work fine
2026-01-04 17:54:57 <ikke> gheja11: it's not a known issue
2026-01-04 17:56:00 <ikke> gheja11: can you try a traceroute?
2026-01-04 18:02:50 <gheja11> sure, there are responses until the third hop - https://pastebin.com/raw/srmKEdyE (expires in 1 hour)
2026-01-04 18:06:15 <gheja11> and I can ping the neighbouring IPs, .131 and .133
2026-01-04 18:06:35 <gheja11> and connect to them over http
2026-01-04 18:09:30 <gheja11> I just checked from a different IP range and it worked from there, both the ping and the download
2026-01-04 18:10:33 <gheja11> so I guess my primary provider's IP range is blocked for some reason
2026-01-04 18:22:40 <ikke> Or perh a routing issue 
2026-01-04 18:22:44 <ikke> perhaps 
2026-01-05 06:19:55 <ikke> clandmeter: nld-bld-1 seems unresponsive again :(
2026-01-05 06:30:06 <ikke> ncopa: can you check the date on `shared-runner nor-ci-1`?
2026-01-05 09:53:20 <ncopa> morning! will do
2026-01-05 10:00:38 <ncopa> it looks like it is building: https://gitlab.alpinelinux.org/jvvv/aports/-/jobs/2161542
2026-01-05 10:55:34 <gheja11> I contacted my provider (regarding the connection timeout for dl-cdn.alpinelinux.org 146.75.2.132), they said they contacted Fastly and solved the issue together - now it works from their IP range too
2026-01-05 10:55:51 <gheja11> thanks for the help, @ikke!
2026-01-05 10:56:09 <ikke> Great, thanks for the update
2026-01-05 10:56:43 <ikke> ncopa: I saw some jobs failing due to an tls verification error
2026-01-05 11:46:09 <ncopa> maybe time was off?
2026-01-05 11:50:13 <ikke> yes, that's why I was asking to check the date
2026-01-05 11:54:19 <ncopa> its correct now
2026-01-05 11:54:29 <ikke> Thanks
2026-01-05 22:01:57 <omni> the arm/aarch64 builders seem to default to ipv6 and not try ipv4 when failing
2026-01-05 22:10:52 <omni> when and how are archives added to distfiles?
2026-01-05 22:13:46 <omni> I've mentioned in #meli that their AAAA address doesn't seem to be reachable
2026-01-06 05:32:07 <ikke> omni: that's because we use bb wget 
2026-01-06 05:32:20 <ikke> Which does not fall back 
2026-01-06 05:54:13 <ikke> omni: Once per day the builders sync their distfiles withdistfiles.a.o
2026-01-06 10:14:15 <omni> ikke: ok, thanks!
2026-01-06 10:17:45 <omni> they have reloaded their firewall and the ipv6 address is reachable now
2026-01-06 10:29:33 <ikke> Great
2026-01-06 10:52:33 <ncopa> i think we have corrupt ext4 filesystem on gitlab-runner-ppc64le
2026-01-06 10:52:45 <ncopa> https://build.alpinelinux.org/buildlogs/build-edge-ppc64le/main/linux-lts/linux-lts-6.18.3-r0.log
2026-01-06 10:53:11 <ncopa> we should probably reboot the ppc64le host and run filesystem check
2026-01-06 10:53:24 <ncopa> or I can remount it read-only and do fsck.ext4
2026-01-06 10:54:07 <ncopa>  /dev/sda4: ********** WARNING: Filesystem still has errors **********
2026-01-06 10:56:11 <ncopa> ikke: do you think we should reboot it, or just try fsck without reboot?
2026-01-06 10:57:38 <ncopa> I suppose it is risky to try apk upgrade with a corrupt fs
2026-01-06 10:58:11 <ncopa> I will try fsck it online
2026-01-06 11:00:45 <ncopa> didnt work. I need to reboot it
2026-01-06 11:00:56 <ncopa>  10:59:55 up 3 days, 15:08,  0 users,  load average: 0.44, 22.51, 38.08
2026-01-06 11:00:56 <ncopa> gitlab-runner-ppc64le [~]# uptime
2026-01-06 11:01:14 <ncopa> apk audit --system also did not detect anything changed
2026-01-06 11:01:19 <ncopa> im rebooting it
2026-01-06 11:16:14 <ncopa> build-edge-ppc64le should be back up now
2026-01-06 11:21:38 <ikke> Thanks!
2026-01-06 11:22:20 <ikke> ncopa: how did you schedule the fsck?
2026-01-06 11:27:29 <ncopa> 1) ensure fsck.ext4 is installed. 2) rc-update add fsck boot 3) reboot
2026-01-06 11:27:57 <ncopa> I suppose we should upgrade it to v3.22 or something
2026-01-06 11:28:39 <ncopa> i noticed that gitlab runners does not work well with alpine v3.23. docker version mismiatch
2026-01-06 11:29:06 <ikke> Hmm, ok
2026-01-06 14:36:48 <ikke> clandmeter: thanks
2026-01-06 14:44:53 <clandmeter> rebooting bld1
2026-01-06 14:50:00 <invoked> mds matrix bridge is having issues
2026-01-07 07:02:51 <ikke> clandmeter: I've increased the mirror volume on deu1-t1-1 we have about 600G left
2026-01-07 07:03:04 <ikke> In the vg I mean
2026-01-07 07:03:09 <clandmeter> ok
2026-01-07 07:03:23 <clandmeter> we should also fix the volume on pkgs
2026-01-07 07:03:31 <ikke> Right
2026-01-07 07:03:48 <ikke> Perhaps we can reduce the storage used for dev containers on that server
2026-01-07 07:04:22 <clandmeter> we can kill the second disk
2026-01-07 07:04:31 <clandmeter> and extend the first by 160G
2026-01-07 07:04:41 <clandmeter> i only used it for backup
2026-01-07 07:04:56 <ikke> I meant on deu-t1-1
2026-01-07 07:05:00 <clandmeter> oh you mean the mirror
2026-01-07 07:06:14 <ikke> Yes
2026-01-07 07:06:29 <ikke> That server also has x86_64 dev containers
2026-01-07 07:06:35 <ikke> The volume for that is 1T now
2026-01-07 07:14:19 <clandmeter> yeah thats the german one i remember now
2026-01-08 09:54:08 <ikke> clandmeter: What do you think about moving distfiles to object storage? I think the type of storage would be ideal, but it does require us to adjust how it's stored and how we expose it
2026-01-08 09:54:46 <ikke> (S3 like storage)
2026-01-08 13:04:18 <clandmeter> ikke: ok, you will need some scripts to support it i guess?
2026-01-08 13:09:22 <ikke> There is a fuse driver, if that works reliably, that could be used with minimal impact, otherwise you indeed need something to translate it
2026-01-08 13:11:40 <ikke> And not sure if those drivers would work with Linode
2026-01-08 13:13:26 <achill> funnily enouh i had that same idea yesterday evenin for our KDE nihtly distfiles in postmarketOS since they are enourmous
2026-01-08 13:14:02 <ikke> I was thinking about it when we had fs corruption
2026-01-08 14:14:42 <ikke> clandmeter: fyi, I'm thinking about doing something similar for gitlab artifacts.
2026-01-08 14:14:51 <ikke> But gitlab supports it natively
2026-01-08 15:33:14 <ikke> clandmeter: ^
2026-01-08 20:40:35 <clandmeter> sorry both rv boxes are offline
2026-01-08 20:40:45 <clandmeter> will fix tomorrow
2026-01-08 20:43:12 <clandmeter> ikke: i wonder if it makes more sense to have our scripts be aware of object storage instead of mimic posix
2026-01-08 20:45:55 <ikke> That would be a better option yes
2026-01-08 20:46:29 <ikke> We would also need something to expose it to a webserver though 
2026-01-08 20:56:02 <clandmeter> https://blog.min.io/filesystem-on-object-store-is-a-bad-idea/
2026-01-08 21:20:46 <ikke> Yeah, it's not ideal
2026-01-08 21:22:09 <ikke> But for distfiles, except for listing a lot of files in a 'directory', it does not require a lot of performance
2026-01-09 16:23:05 <iggy>  Caddy supports s3 backends natively. I'd guess other servers do too. And most object stores have some way to serve files directly from them (aws does, looks like linode object storage does). It looks like there are also some options for serving from s3 to rsync if you need that
2026-01-09 16:27:02 <ikke> iggy: Hmm, interesting, thanks for sharing
2026-01-09 16:27:19 <ikke> The builders currently rsync their files to distfiles.a.o
2026-01-09 16:30:34 <iggy> do they use rsync protocol or rsync+ssh?
2026-01-09 16:32:05 <ikke> rsync + ssh
2026-01-09 16:32:19 <ikke> they upload over ssh
2026-01-09 16:32:47 <ikke> But that's because ssh is used as authentication
2026-01-09 16:48:29 <iggy> yeah, figured, there are less options that way, but still some viable... and worst case scenario, you go the s3fs+regular rsync daemon option for that part of it (but switching from rsync to a standard s3 client is probably a better option)
2026-01-09 17:03:55 <mio> is this atools-go sigsegv in the lint job a temporary error? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2166776
2026-01-09 17:04:41 <ikke> mio: probably not
2026-01-09 17:04:48 <mio> same runner seems to be fine a few hours ago, just checking
2026-01-09 17:05:26 <ikke> mio: probably related to something in the APKBUILD
2026-01-09 17:06:26 <mio> ikke: ah, thanks
2026-01-09 17:27:02 <ikke> mio: ah, I already fixed that segfault, but no new release yet
2026-01-09 17:27:56 <ikke> https://gitlab.alpinelinux.org/alpine/infra/atools-go/-/merge_requests/78
2026-01-09 17:29:15 <ikke> Due to things like this: export "GDAL_DRIVER_${name}_PLUGIN_INSTALLATION_MESSAGE=You may install it with 'apk add $i'"
2026-01-09 17:33:48 <mio> okay, good that's already known and fixed :)
2026-01-09 17:34:52 <ikke> Will release a new version
2026-01-09 17:35:12 <mio> great, thanks
2026-01-09 18:03:26 <ikke> mio: https://gitlab.alpinelinux.org/alpine/infra/docker/apkbuild-lint-tools/-/merge_requests/32
2026-01-09 18:13:34 <ikke> mio: retrying that job, and it now passes
2026-01-09 18:54:14 <mio> looking good :)
2026-01-09 21:13:39 <ncopa> Learned about https://edent.github.io/SuperTinyIcons/ today. I wonder if we can use the alpine logo as our official 
2026-01-09 21:14:59 <ikke> The complete logo also contains the name
2026-01-10 14:38:08 <ncopa> today I learned that the mermaid version used in gitlab does not support block or architecture diagrams
2026-01-10 17:12:31 <omni> looks like the loongarch64 CI builder is out of disk
2026-01-10 17:13:35 <ikke> You say 20G is not enough to build a package? :P
2026-01-10 22:47:33 <omni> I see "No space left on device" in the attempted main/rust build here: https://gitlab.alpinelinux.org/alpine/aports/-/jobs/2167849
2026-01-11 07:38:48 <ikke> omni: right, so rust does need more than 20G :)
2026-01-11 07:38:54 <ikke> omni: for the record, I've cleaned it up
2026-01-11 12:45:19 <omni> ikke: thanks
2026-01-11 12:45:58 <omni> that, or that it built at the same time as something that also used a lot of space?
2026-01-11 12:46:22 <ikke> possibly
2026-01-14 15:40:33 <achill> gitlab 500
2026-01-14 15:40:42 <achill> ah works again i think
2026-01-14 15:41:00 <achill> yeah lol i guess it was gone for complete 3 seconds
2026-01-14 18:12:12 <Sertonix[m]> Will the loongarch64 builder catch up on it's own eventually or is some manual intervention needed?
2026-01-14 18:13:45 <ikke> Sertonix[m]: build-edge-loongarch64: failed to build qt6-qtwebengine: https://build.alpinelinux.org/buildlogs/build-edge-loongarch64/community/qt6-qtwebengine/qt6-qtwebengine-6.10.1-r4.log
2026-01-14 19:28:27 <ikke> Sertonix[m]: that is to say, at least some manual intervention is required
2026-01-14 20:24:45 <ncopa> cycle: freerdp -> webkit2gtk-4.1 -> kwallet -> kconfig -> xwayland-run -> weston -> freerdp
2026-01-14 20:24:45 <ncopa> cycle: ghostscript -> gtk+3.0 -> gtk4.0 -> gst-plugins-bad -> zbar -> imagemagick -> ghostscript
2026-01-14 20:24:45 <ncopa> cycle: py3-jaraco.collections -> py3-jaraco.text -> py3-jaraco.context -> py3-jaraco.test -> py3-jaraco.collections
2026-01-14 20:25:03 <ncopa> wrong channel sorry
2026-01-15 12:08:30 <ikke> lotheac: I did make some changes to the agent config. Most notably: concurrency has to be set via the CR, it gets ignored when it's provided via the runner config itself
2026-01-15 13:30:06 <lotheac> oh, okay
2026-01-15 13:34:09 <lotheac> those changes look reasonable to me
2026-01-15 13:37:09 <ikke> What supprised me a bit (especially after reading https://docs.gitlab.com/runner/configuration/advanced-configuration/#long-polling-issues), is that limit on a runner level get overridden by concurrency)
2026-01-15 13:37:44 <ikke> concurrency = 10, runner limit=4 -> max_builds=10
2026-01-15 13:38:49 <lotheac> huh
2026-01-15 13:38:56 <lotheac> i would have expected the opposite
2026-01-15 13:38:59 <ikke> yeah, me too
2026-01-15 13:39:40 <ikke> The behavior is confusing
2026-01-15 13:40:01 <ikke> But the runner kept accepting jobs, even if there were 4 jobs running already (and then failing due to lack of resources)
2026-01-15 23:41:34 <adrian> loongarch builder could probably use an apk fix, it seems like it's in a broken state
2026-01-16 17:37:30 <ncopa> oh great the pioneer machine is failing again
2026-01-16 19:59:11 <ncopa> I have a user that registered to wiki.a.o in 20 July 2009 and cannot recover the password now
2026-01-16 19:59:20 <ncopa> I cannot find out which email he used
2026-01-16 19:59:41 <ncopa> how do we handle those situations?
2026-01-16 20:05:33 <ikke> Good question, we would need some proof of ownership
2026-01-17 21:23:50 <Sertonix[m]> Can somebody nudge the armv7 builder? podman should pass when rebuild