2023-04-01 05:43:51 <ikke> Maybe this? https://gitlab.com/gitlab-org/gitlab/-/issues/398569
2023-04-01 06:01:14 <psykose> looks like it
2023-04-01 06:34:50 <ikke> aports-qa-bot for some reason cannot find maintainers
2023-04-01 06:35:08 <ikke> "no maintainers found to assign MR=45603 component="MergeRequestJob processor" project=1 service=AutoMaintainer uuid=9d392336-d6c7-469a-8a8c-4b504a1ba833"
2023-04-01 11:50:10 <psykose> some mrs i press rebase on also don't do anything
2023-04-01 11:50:58 <psykose> or rather the ui doesn't update
2023-04-01 11:53:53 <psykose> worked after another invalidation
2023-04-01 12:24:14 <ikke> Welcome to the world of eventual consistency
2023-04-01 12:25:11 <psykose> yeah it's weird
2023-04-01 12:25:20 <psykose> guess they really broke some stuff in this one :p
2023-04-01 12:26:02 <ikke> Not sure if it has to do with the size of aports
2023-04-01 12:26:09 <ikke> so it takes longer before the repo is updated or something like that
2023-04-01 12:26:44 <psykose> it seems like a ui thing solely but weirdly cached
2023-04-01 12:28:47 <ikke> I do see gitaly having issues finding commits
2023-04-01 12:30:14 <ikke> https://tpaste.us/j44v 
2023-04-01 12:31:47 <psykose> not great
2023-04-01 12:31:55 <psykose> maybe there's even more updates? :p
2023-04-01 12:32:09 <ikke> We could go to 15.10
2023-04-01 12:32:20 <psykose> sounds fun
2023-04-01 14:48:53 <clandmeter> update it until it all goes bonkers
2023-04-03 14:49:26 <nmeum> is 172.16.11.61 still the ppc64le docker machine? I can't seem to reach it via wireguard right now. Am I doing something wrong? https://tpaste.us/kXXz
2023-04-03 14:57:15 <ikke> nmeum: there is some strange routing issue on that box that regularly breaks dmvpn 
2023-04-03 15:01:15 <nmeum> ah ok, I will just do something else until it resolves itself then :)
2023-04-03 15:07:44 <ikke> nmeum: should work now
2023-04-03 15:07:53 <ikke> gets these routes for some reason: throw 172.16.0.254 proto static
2023-04-03 15:08:05 <ikke> And I don't know why
2023-04-03 15:08:11 <ikke> suspect something to do with strongswan
2023-04-03 16:10:44 <nmeum> ikke: I still run into routing issues at the 172.16.0.254 hop https://tpaste.us/O88Y
2023-04-03 17:32:46 <ikke> nmeum: what are you trying to connect to? 
2023-04-03 17:50:21 <ikke> nmeum: we lost the builder a couple of months ago with all the containers 
2023-04-03 17:50:49 <ikke> So the container you are trying to connect to does not exist anymore 
2023-04-03 17:51:04 <nmeum> oh. I previously hat a ppc64le docker on 172.16.11.61
2023-04-03 17:51:06 <nmeum> s/hat/had/
2023-04-03 17:51:28 <nmeum> used that occassionally to debug ppc64le-specific issues with aports
2023-04-03 17:52:00 <ikke> I can create a new container for you
2023-04-03 17:53:24 <nmeum> I managed to do my debugging with qemu-user this time :)
2023-04-03 17:58:28 <ikke> nmeum: 172.16.15.42
2023-04-04 10:04:43 <mps> new u-boot disabled build bmp_logo by default. should we follow this or add patch to enable it for alpine? I think we should not add bmp_logo to u-boot-tools on alpine
2023-04-04 10:05:43 <mps> nmeum: please if you have some time tell what is your stance (I respect your opinions)
2023-04-04 10:18:15 <nmeum> I personally don't find the bmp_logo feature particulary useful
2023-04-04 10:19:21 <mps> nmeum: thank you. and I agree
2023-04-04 10:20:56 <mps> and btw, you have hifive unmatched board iirc? I think we don't need hifive-unmatched-ramdisk.patch now in u-boot
2023-04-04 11:30:55 <mps> what happened with gitlab, tried to add new MR and it gives me info about previous one
2023-04-04 12:23:02 <nmeum> mps: yea, I have an unmatched
2023-04-04 12:24:05 <nmeum> it seems the proper upstream patch for hifive-unmatched-ramdisk.patch has still not been merged, no? https://patchwork.ozlabs.org/project/uboot/list/?series=258110&state=*
2023-04-04 12:24:16 <nmeum> but idk how their patchwork works and if it is still updated
2023-04-04 13:09:58 <mps> I think they don't use patchwork in some time now, but not sure
2023-04-04 13:12:32 <mps> but if anyone hit the bug on hifive we can re-add radmdisk patch
2023-04-05 10:24:57 <mps> rebase doesn't work on gitlab now?
2023-04-05 10:26:22 <ikke> You might need to refresh
2023-04-05 10:30:03 <mps> tried but doesn't help
2023-04-05 10:30:29 <mps> '
2023-04-05 10:30:31 <mps> Merge blocked: the source branch must be rebased onto the target branch.'
2023-04-05 10:30:32 <ikke> What MR?
2023-04-05 10:30:52 <mps> !45703
2023-04-05 10:31:35 <mps> !45702
2023-04-05 14:31:02 <tomalok> ikke experincing the same as mps -- !45711 (and another one for edge soon to land)
2023-04-05 14:37:17 <tomalok> !45712
2023-04-05 14:50:05 <ikke> tomalok: the last gitlab update sadly introduced some issues regarding merge requests 
2023-04-05 20:16:15 <psykose> it works if it falls behind again and gets rebased a second time
2023-04-05 20:16:16 <psykose> usually
2023-04-05 20:16:27 <psykose> but it's quite annoying
2023-04-05 20:16:39 <psykose> it's also purely a visual bug since you can still merge it from elsewhere pretty sure
2023-04-05 20:22:57 <ikke> yup
2023-04-05 20:23:05 <ikke> I want to see if I can reproduce it on -test
2023-04-05 20:23:13 <ikke> and if so, if 15.10 behaves the same
2023-04-05 20:33:04 <psykose> ikke: edge-aarch64 is still stuck :p
2023-04-05 20:36:02 <ikke> hmm, not stuck on building a package
2023-04-05 20:36:17 <ikke> (it was hanging)
2023-04-05 21:07:47 <psykose> ikke: have you heard of this https://gitlab.alpinelinux.org/trinity-labs/trinity-skin/-/tree/main
2023-04-05 21:08:27 <psykose> bunch more stuff like that https://gitlab.alpinelinux.org/trinity-labs/lua5.3-subprocess/-/tree/master
2023-04-05 21:12:47 <psykose> seems to be a mirror of the github repos lol https://github.com/trinity-labs
2023-04-06 03:59:07 <psykose> ikke: also aarch64 got stuck again, which is weird
2023-04-06 04:53:06 <psykose> ikke: could you also look at why it's impossible to run binaries with caps inside lxc? e.g. apk add sway && /usr/bin/sway
2023-04-06 04:58:30 <ikke> lxc.cap.drop = sys_nice
2023-04-06 04:58:38 <ikke> in /usr/share/lxc/config/alpine.common.conf
2023-04-06 04:58:52 <psykose> :)
2023-04-06 04:58:55 <psykose> time to remove it then
2023-04-06 04:59:33 <ikke> need to remove it from lxc-templates-legacy-alpine
2023-04-06 05:16:37 <ikke> Question is if we should just blancly remove it
2023-04-06 05:19:09 <psykose> unless there's a way to neutralise caps, yea
2023-04-06 05:19:16 <psykose> for which i don't know of one which is silly
2023-04-06 05:19:33 <psykose> would be nice if lxc could go "yeah see that cap on the binary? running it anyway, but ignoring all that shit"
2023-04-06 05:19:43 <psykose> instead it either fails or don't
2023-04-06 05:19:49 <ikke> similar with docker
2023-04-06 05:20:41 <psykose> indeed
2023-04-06 05:30:35 <ikke> Is this failing during testing?
2023-04-06 05:32:03 <psykose> it fails if something uses sway on the builders or ci
2023-04-06 05:32:36 <psykose> which is used in https://git.alpinelinux.org/aports/tree/community/foot/APKBUILD#n50 or in an mr somewhere for some tests, etc
2023-04-06 10:24:28 <psykose> ikke: 3.17-aarch64 also got stuck
2023-04-06 10:24:32 <psykose> did you figure out why they do
2023-04-06 10:24:51 <ikke> not  yet
2023-04-06 10:25:16 <ikke> buildrepo spins at 100%
2023-04-06 10:25:29 <psykose> weird
2023-04-06 10:25:53 <ikke> pselect6(7, [6], [], NULL, {tv_sec=0, tv_nsec=0}, {sigmask=NULL, sigsetsize=8}) = 0 (Timeout) <0.000006>
2023-04-06 10:26:30 <ikke> 7 unix    u                                   0  376342334  type=STREAM
2023-04-06 10:28:27 <ikke> unix  3      [ ]         STREAM     CONNECTED     376342334 74453/lua5.2
2023-04-06 10:37:55 <ikke> https://tpaste.us/rjjQ 
2023-04-06 10:38:03 <ikke> It's blocked on itself for some reason
2023-04-06 10:46:46 <psykose> perhaps something spawned it twice and then some stuff goes weird
2023-04-06 10:50:21 <psykose> hmm, no, it's like it always gets stuck on the end of a build for something
2023-04-06 13:53:50 <ikke> problem reproduces on gitlab-test
2023-04-06 13:53:59 <ikke> (empty merge request until refresh)
2023-04-06 15:19:32 <ikke> sadly 15.10 does not fix it :(
2023-04-06 17:02:31 <ikke> Anyone happens to know an open source log collector / indexing system that does not use java?
2023-04-06 17:03:44 <ncopa> lucene++?
2023-04-06 17:05:34 <ikke> I suppose you manually need to build something on top of that?
2023-04-07 02:55:01 <psykose> ikke: there's some grafana thing called loki
2023-04-07 02:55:11 <psykose> but it's like prometheus and pull based
2023-04-07 02:55:33 <psykose> it doesn't do indexing either, you just keep labels
2023-04-07 07:03:21 <clandmeter> ikke: is there an open issue for that mr issue?
2023-04-07 07:03:58 <psykose> https://gitlab.com/gitlab-org/gitlab/-/issues/398569
2023-04-07 07:04:38 <psykose> ah, not sure about the other one though
2023-04-07 07:04:41 <psykose> (where you can't merge)
2023-04-07 07:07:22 <clandmeter> hi psykose, we missed you :)
2023-04-07 07:09:55 <psykose> sowwy :(
2023-04-07 07:27:36 <clandmeter> np it happens
2023-04-07 07:27:59 <clandmeter> we can discus the lang issues next time
2023-04-09 19:29:15 <psykose> ikke: i think some arm runners don't =2 parallelism
2023-04-09 19:29:21 <psykose> at least there's only ever 1 armv7 job
2023-04-09 19:29:34 <ikke> though I fixed that, let me check
2023-04-09 19:29:38 <ikke> thought*
2023-04-09 19:29:38 <psykose> or maybe that's only when the usual dotnet is running
2023-04-09 19:31:53 <ikke> armv7 ci has limit = 2 for the shared runner and concurrent = 4
2023-04-09 19:32:02 <ikke> oh, if only I could spell
2023-04-09 19:32:12 <ikke> oncurrent = 4
2023-04-09 19:34:18 <ikke> other 2 are correct
2023-04-09 19:35:20 <psykose> hm
2023-04-09 19:35:29 <psykose> =5? :p
2023-04-09 19:35:44 <ikke> the runner limit is still 2 :)
2023-04-09 19:35:53 <psykose> yes, but it runs into the 4
2023-04-09 19:36:08 <ikke> no, it was just not set correctly
2023-04-09 19:36:17 <ikke> so it was limited to 1 concurrent job over all runners
2023-04-09 19:36:19 <psykose> ah
2023-04-09 19:36:21 <psykose> right
2023-04-09 20:16:38 <ikke> oberservation: small projects do not have issues with merge request being empty. I have a feeling gitally is lagging due to aports being large
2023-04-09 20:19:27 <psykose> yeah
2023-04-09 20:19:43 <psykose> it also always works on retry (rebase after 1 commit behind, instead of opened up-to-date)
2023-04-11 15:30:12 <ncopa> I think gitlab-runner-ppc64le needs a reboot
2023-04-11 15:30:22 <ncopa> upgrade and reboot
2023-04-11 15:30:31 <ncopa> not sure if I dare...
2023-04-11 15:30:48 <ikke> Maybe we should send an e-mail to lance
2023-04-11 15:34:28 <ncopa> its just apk upgrade and reboot
2023-04-11 15:34:47 <ncopa> dmvpn is down too on ppc64le machine. not sure why
2023-04-11 15:34:55 <ikke> ncopa: check `ip route`
2023-04-11 15:35:01 <ikke> see those throw routes?
2023-04-11 15:35:06 <ncopa> yup?
2023-04-11 15:36:37 <ikke> Not sure what causes them, but they break dmvpn
2023-04-11 15:36:43 <ikke> remove them, and it will work for a while
2023-04-11 15:38:45 <ikke> I have a suspicion they come from strongswan, but not sure why
2023-04-11 15:43:53 <ncopa> ok
2023-04-11 16:12:09 <psykose> the ppc64le builder is gone with whatever you did :p
2023-04-11 16:12:59 <pj> it's for the better good
2023-04-11 16:14:41 <ikke> host is still there
2023-04-11 16:14:44 <ncopa> hum...
2023-04-11 16:14:51 <ncopa> network does not work from the container
2023-04-11 16:15:20 <ncopa> build-edge-ppc64le [~]# ping 172.16.15.1
2023-04-11 16:15:20 <ncopa> PING 172.16.15.1 (172.16.15.1): 56 data bytes
2023-04-11 16:15:25 <ncopa> cannot ping default gw
2023-04-11 16:20:23 <ncopa> i'd like to reboot it
2023-04-11 16:20:43 <psykose> it's broken anyway so go for it
2023-04-11 16:22:20 <ncopa> oh.. build-3-17-ppc64le is working apparently
2023-04-11 16:22:41 <psykose> is it
2023-04-11 16:22:55 <psykose> doesn't show up on build.a.o
2023-04-11 16:23:41 <ncopa> ok. im apk upgrade it and rebooting it
2023-04-11 16:23:46 <ikke> There is no route for 172.16.15.0/24
2023-04-11 16:24:04 <ncopa> ERROR: Failed to create boot/vmlinuz-lts: No space left on device
2023-04-11 16:24:09 <ikke> yeah
2023-04-11 16:24:14 <ikke> the bootloader is too small
2023-04-11 16:24:20 <ikke> I mean, the boot partition
2023-04-11 16:24:39 <ncopa> oh my
2023-04-11 16:25:04 <ikke> remove the old one, and regenerate? :P
2023-04-11 16:25:40 <ikke> It cannot fit 2 images
2023-04-11 16:26:18 <ncopa> i made a /boot.backup first
2023-04-11 16:26:25 <ncopa> should be ok now
2023-04-11 16:26:27 <ncopa> i reboot
2023-04-11 16:27:57 <ikke> restoring network (docker / lxc) might involve throwing away the throw routes and manually adding the actual routes
2023-04-11 16:29:58 <ncopa> gitlab-runner-ppc64le [~]# uname -a
2023-04-11 16:29:58 <ncopa> Linux gitlab-runner-ppc64le 5.15.106-0-lts #1-Alpine SMP Wed, 05 Apr 2023 11:21:57 +0000 ppc64le Linux
2023-04-11 16:32:10 <psykose> lol
2023-04-11 16:33:30 <ncopa> gitlab-runner-ppc64le [~]# ip rule
2023-04-11 16:33:30 <ncopa> Dump terminated
2023-04-11 16:33:30 <ncopa> RTNETLINK answers: Address family not supported by protocol
2023-04-11 16:34:50 <ikke> I have no idea what's going on
2023-04-11 16:34:55 <ikke> tried to restore the network as much as possible
2023-04-11 16:35:02 <ikke> Have to go now
2023-04-11 16:35:23 <ncopa> thanks
2023-04-11 16:35:30 <ncopa> i think something is broken in kernel config
2023-04-11 16:40:16 <ncopa> i found it
2023-04-11 16:40:28 <ncopa> # CONFIG_IP_ADVANCED_ROUTER is not set
2023-04-11 20:57:50 <Ramereth> 
ACTION peers in

2023-04-11 20:58:36 <psykose> 
ACTION peers out

2023-04-11 20:59:34 <Ramereth> ikke ncopa: still need my help?
2023-04-12 07:56:51 <ncopa> i found a local backup of ancient alpine releases. v1.1 -> v2.7
2023-04-12 07:57:03 <ncopa> do we have a suiteable server for storing archive?
2023-04-12 08:29:37 <ikke> How big is the data? 
2023-04-12 08:29:44 <ikke> distfiles perhaps 
2023-04-12 08:33:36 <ncopa> 191G currently
2023-04-12 08:40:07 <ncopa> question is if we want it there
2023-04-12 08:42:23 <ncopa> i deleted ncopa-edge-x86 from nld8-dev1.a.o
2023-04-12 08:51:19 <ncopa> i also deleted tt-dev-x86* after asking fabled
2023-04-12 08:54:11 <ncopa> i backing up build-2-*-x86* and will delete those
2023-04-12 08:55:03 <ikke> We have the t1 servers with plenty of storage 
2023-04-12 08:55:06 <psykose> i cleaned all my containers up
2023-04-12 08:55:58 <ncopa> maybe we could have the ancient stuff on the t1 servers?
2023-04-12 08:56:16 <ncopa> and add a dns name ancient.alpinelinux.org
2023-04-12 08:56:30 <ncopa> or archive.alpinelinux.org
2023-04-12 08:56:36 <psykose> didn't we have something like that already for the old prehistory
2023-04-12 08:56:39 <psykose> i swear i remember this
2023-04-12 08:56:57 <ncopa> we used to have an ancient.alpinelinux.org but it was shut down iirc
2023-04-12 08:57:03 <psykose> aha, that's what i remember
2023-04-12 08:57:12 <ncopa> i have the data
2023-04-12 08:57:23 <ncopa> i could maybe even run it on my home router
2023-04-12 08:57:38 <ncopa> but, er... kinda nice to have it on official alpine infra
2023-04-12 08:57:46 <psykose> back in the day it was
2023-04-12 08:57:47 <psykose> :-)
2023-04-12 08:58:35 <psykose> t1 sounds fine i suppose, wonder if there's a way to shrink the space of them, but i guess as raw packages not really
2023-04-12 08:58:54 <ncopa> how much free space we have there now?
2023-04-12 09:11:38 <psykose> ikke: also 3.17-armhf is stuck
2023-04-12 09:39:04 <ncopa> i deleted build-2-*-x86
2023-04-12 09:39:09 <ncopa> i have local backup here
2023-04-12 09:39:56 <ncopa> i wonder if i should also delete build-3-[0-9]-*
2023-04-12 11:02:47 <ncopa> i have backed up and deleted build-2-*-x86* now
2023-04-13 06:22:59 <ncopa> i think i have a rought plan how to avoid having the entire repository on the builders
2023-04-13 06:23:56 <ncopa> first we need have the builder to fetch the APKINDEX from dl-master, and use that to decide what to build (instead of looking for missing apk files in local repo)
2023-04-13 06:24:56 <ncopa> then after packages are built, use the new apk index --merge feature to update the APKINDEX
2023-04-13 06:27:10 <ncopa> actually, before updating APKINDEX we need get a list of the current files on dl-master to be deleted (to lets say filelist.txt)
2023-04-13 06:27:27 <ncopa> then we add the newly build packages to the filelist.txt
2023-04-13 06:27:34 <ncopa> and APKINDEX
2023-04-13 06:27:56 <ncopa> and use that with rsync --include-from=filelist.txt --exclude='*'
2023-04-13 06:30:22 <ncopa>  rsync --include-from=filelist.txt --exclude='*' --delete-after --delay-updates --remove-source-files $src $dest
2023-04-13 06:31:26 <ncopa> will move the built packages and delete the old on remote
2023-04-13 06:31:49 <ncopa> I think it can work
2023-04-13 06:34:55 <ncopa> we should probably also verify that all files in index actually exists on the remote
2023-04-13 07:20:03 <psykose> maybe i'm just really stupid but this sounds like 500 steps that can easily break :p
2023-04-13 07:25:59 <psykose> the rsync command does look alright however
2023-04-13 07:26:56 <psykose> the entire process is basically converting the current git state into an apkindex
2023-04-13 07:27:07 <psykose> which makes me wonder how we can prune subpackages
2023-04-13 07:27:20 <psykose> because there is no way to get the list of subpackages out of just git, you need to build an aport first
2023-04-13 07:28:08 <psykose> so let's say we have a full index. then we commit something that removes one subpackage from something
2023-04-13 07:28:22 <psykose> how do we get a filelist.txt that just has that removed, or send it anywhere? it's not really possible at all
2023-04-13 07:29:12 <psykose> it would have to be something like 'x aports was rebuilt', we send that to a builder, then we replace the full origin+subpackages of that 'x' with whatever the builder sends back, and prune the difference (no more subpackage in index, so delete .apk)
2023-04-13 07:29:24 <psykose> which is just very complex purely because we have no way to detect the package list from apkbuild alone
2023-04-13 07:31:05 <psykose> if we went back 10 years and had a way to statically-get the full subpackage list this would be a lot easier
2023-04-13 07:32:18 <psykose> not sure if just sourcing the file universally works or nmot
2023-04-13 07:49:57 <ncopa> easily break? yeah. its scary.
2023-04-13 07:50:34 <ncopa> you can get the list of subpackages before building. and we must be able to get the list of subpackages without building
2023-04-13 07:51:37 <ncopa> so basically, get a remote index, compare with current git
2023-04-13 07:55:06 <ncopa> ap apk-list gives you a full list of .apk packages (incl subpackages)
2023-04-13 07:55:19 <ncopa> compare that with index from remote
2023-04-13 07:56:20 <ncopa> by comparing that we can figure out: 1) what should be deleted on remote, 2) what needs to be built locally
2023-04-13 07:57:07 <ncopa> yes, its more complicated than what we do now, but this is needed to save the diskspace on builders
2023-04-13 08:00:29 <psykose> yeah, as long as ap apk-list works it's all good
2023-04-13 08:01:18 <ncopa> would be nice to have it im place before 3.18 release so we dont need full repo on 3.18 builders
2023-04-13 08:01:26 <ncopa> but its a bit scary and needs some work
2023-04-13 08:02:08 <ncopa> i would like to bump lua version to 5.4 also
2023-04-13 08:02:20 <ncopa> but its hopeless without a testsuite
2023-04-13 08:36:33 <ncopa> I have written a summary here: https://gitlab.alpinelinux.org/alpine/lua-aports/-/issues/6
2023-04-13 08:47:27 <psykose> sounds good
2023-04-13 09:08:35 <psykose> is it just me or does  http://dl-cdn.alpinelinux.org/alpine/edge/main give an untrusted-signature apkindex
2023-04-13 09:09:23 <ncopa> i think its just you
2023-04-13 09:09:31 <ncopa> try use https?
2023-04-13 09:09:47 <ncopa> could be something with new apk-tools?
2023-04-13 09:09:58 <psykose> hm
2023-04-13 09:09:59 <psykose> https works
2023-04-13 09:10:02 <psykose> http doesn't
2023-04-13 09:10:07 <psykose> it worked up until the past 5 minutes
2023-04-13 09:10:13 <psykose> i've had new apk-tools since it was tagged
2023-04-13 09:10:21 <psykose> maybe i'm getting mitm'd :p
2023-04-13 09:10:36 <ncopa> yeah, might be possible
2023-04-13 09:11:17 <ncopa> try fetch it with http and https and do a deep compare
2023-04-13 09:11:30 <ncopa> maybe with diffoscope
2023-04-13 09:12:08 <psykose> they're the same
2023-04-13 09:12:10 <psykose> by sha
2023-04-13 09:12:44 <psykose> huh
2023-04-13 09:12:45 <psykose> fixed now
2023-04-13 09:13:15 <psykose> but not in cache: WARNING: opening from cache http://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
2023-04-13 09:13:27 <psykose> i guess something went wrong in some update, but that would be new
2023-04-13 09:13:47 <ncopa> and if you diff the file in cache?
2023-04-13 09:16:04 <psykose> sadly i just deleted it and now everything works x.x
2023-04-13 09:16:08 <psykose> i should've diffed it
2023-04-13 09:16:14 <psykose> lets hope it breaks again
2023-04-13 09:16:19 <ncopa> :)
2023-04-13 15:03:50 <psykose> fixed via some caching
2023-04-13 16:27:49 <ikke> just add another layer of caching
2023-04-13 16:30:49 <psykose> there was 0 before :p
2023-04-13 16:31:03 <psykose> added an nginx one via uwsgi_cache_something or whatever
2023-04-13 16:31:27 <ikke> browsers cache stuff, the os caches stuff :P
2023-04-13 16:31:43 <psykose> the specific issue there was like 100 requests a second for https://git.alpinelinux.org/ca-certificates/tree/certdata.txt
2023-04-13 16:31:51 <psykose> from some java http client, from different ips
2023-04-13 16:31:52 <psykose> lol
2023-04-13 16:31:56 <ikke> lesigh
2023-04-13 16:32:05 <ikke> 429 it :P
2023-04-13 16:32:13 <psykose> the whole path? it's not unique ip :p
2023-04-13 16:32:20 <ikke> user agent
2023-04-13 16:32:20 <psykose> but with the cache that is whatever
2023-04-13 16:32:26 <ikke> not serious
2023-04-13 16:33:01 <psykose> you can now see x-cache: HIT
2023-04-13 16:34:31 <psykose> ppc64le CI is still broken
2023-04-13 16:37:25 <ikke> should be fixed now
2023-04-13 16:37:33 <ikke> needed to fix another route
2023-04-13 16:38:06 <ikke> And now idea how to find out where these throw routes come from
2023-04-13 16:38:12 <psykose> aye
2023-04-13 16:38:22 <psykose> the kernel thing was ifxed
2023-04-13 16:38:24 <psykose> should reboot for that
2023-04-13 16:38:59 <ikke> Does that also fix the throw routes? I recall testing older kernels and still having the issue
2023-04-13 16:39:15 <psykose> nah, just the advanced config for the other thing ncopa found
2023-04-13 16:39:22 <ikke> ip rule show
2023-04-13 16:39:25 <ikke> right?
2023-04-13 16:39:32 <ikke> yeah
2023-04-13 16:39:37 <ikke> RTNETLINK answers: Address family not supported by protocol
2023-04-13 16:40:12 <ikke> The think that changed is that we installed dmvpn on that host
2023-04-13 16:40:32 <ikke> (regarding throw routes
2023-04-13 16:40:36 <ikke> )
2023-04-13 16:41:48 <psykose> ype
2023-04-13 16:59:01 <ncopa> RTNETLINK answers: Address family not supported by protocol
2023-04-13 16:59:10 <ncopa> that is kernel config problem
2023-04-13 16:59:12 <psykose> yeah
2023-04-13 16:59:21 <psykose> can just reboot to fix that if it's edge
2023-04-13 16:59:29 <ncopa> i fixed it in edge, but need to ba kport to stable kernel
2023-04-13 16:59:29 <ikke> But then I have to fix all the network agiana
2023-04-13 16:59:59 <ikke> These throw routes are really anoying
2023-04-13 17:00:12 <ncopa> yeah
2023-04-13 17:00:53 <ikke> I really suspect strongswan, trying to figure out how to debug that
2023-04-13 17:01:33 <ncopa> i suspect strongswan or bgp does something when policy routing fails
2023-04-13 17:01:44 <ncopa> probably bgp server
2023-04-13 17:02:00 <ncopa> i hope that just fix the kernel fixes it all
2023-04-13 17:02:08 <ikke> Yeah, that would be nice
2023-04-13 17:02:24 <ikke> We can try it
2023-04-13 17:03:15 <ikke> Not sure how risky it would be to try the kernel from edge
2023-04-13 17:03:34 <psykose> it would be a good pre-test
2023-04-13 17:03:43 <psykose> since with 3.18 it will be the kernel for edge + a few more patch releases
2023-04-13 17:03:51 <psykose> which is basically the same thing
2023-04-13 17:03:52 <ncopa> i think we need to backport it anyway
2023-04-13 17:03:57 <psykose> yeah
2023-04-13 17:03:59 <ncopa> current 3.16 is broken
2023-04-13 17:04:10 <psykose> the ppc config? yeah
2023-04-13 17:04:17 <ncopa> so i'd rather backport it to 3.17 and 3.16 and then test the 3.16 kernel
2023-04-13 17:04:21 <ikke> ok
2023-04-13 17:04:22 <ncopa> and then upgrade to 3.17
2023-04-13 18:45:51 <ncopa> i pushed kernel to v3.16. I suppose we can upgrade and reboot ppc64le machine
2023-04-13 18:46:00 <ikke> yeah, I can arrange that
2023-04-13 18:46:07 <ncopa> wonderful!
2023-04-13 18:46:16 <ncopa> has been crazy busy here today
2023-04-13 18:46:30 <ncopa> but i think I have accomplished pretty much
2023-04-13 18:46:45 <ncopa> tiny-cloud is in a pretty good shape for 3.18 now
2023-04-13 18:47:57 <ikke> nice
2023-04-13 18:51:25 <ncopa> which means i can continue on getting the 3.18 builders up
2023-04-13 18:53:09 <ikke> usa2 has 68G free
2023-04-13 19:46:00 <ikke> ncopa: new kernel is still not available :(
2023-04-13 19:46:45 <ikke> checking the builder
2023-04-13 19:50:38 <ikke> mqtt-exec failed
2023-04-14 09:24:05 <mps> !45940 build fine and pass check on lxc and local machine, but fails one test on CI
2023-04-14 09:26:10 <mps> what to do, merge it and if don't pass check on builders disable check?
2023-04-14 09:28:14 <ncopa> mps: maybe you can cat testsuite.log from APKBUILD on failure?
2023-04-14 09:28:51 <mps> ncopa: it fails only on gitlab CI
2023-04-14 09:29:10 <ikke> You can add that in the APKBUILD for CI 
2023-04-14 09:30:17 <mps> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1009351#L601
2023-04-14 09:31:19 <ncopa> if you add "cat testsuite.log" on failure in the APKBUILD we'll get the output on CI
2023-04-14 09:31:19 <psykose> make check || { cat ./testsuite.log; return 1 }
2023-04-14 09:31:24 <ncopa> yeah
2023-04-14 09:31:44 <ncopa> so we see the testsuite.log on CI
2023-04-14 09:33:42 <mps> hm, isn't this enough https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1009351
2023-04-14 09:35:09 <ncopa> if its enough for you to find out why it fails on the CI and to fix it, then sure
2023-04-14 10:34:04 <ikke> https://gitlab.com/gitlab-org/gitlab/-/issues/407115
2023-04-14 10:34:11 <ikke> apparently not fixed until 16.0
2023-04-14 10:34:24 <psykose> technically a dupe here
2023-04-14 10:34:37 <psykose> is 16.0 that different from a regular upgrade
2023-04-14 10:34:44 <ikke> There are some deprecations
2023-04-14 10:34:48 <ikke> that are removed from 16.0
2023-04-14 10:34:52 <psykose> what specifically
2023-04-14 10:34:58 <psykose> we don't really use much except for algitbot i guess
2023-04-14 10:35:02 <psykose> well, CI
2023-04-14 10:35:58 <ikke> https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=16.0
2023-04-14 10:36:09 <ikke> and docker
2023-04-14 10:36:14 <ikke> (container registry)
2023-04-14 10:36:17 <ikke> terraform
2023-04-14 10:36:47 <ikke> Not saying we are necessarily affected, just what makes 16.0 more significanty
2023-04-14 10:38:07 <psykose> yeah
2023-04-14 10:38:11 <psykose> guess it's some reading
2023-04-14 10:39:15 <ikke> https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=16.0#graphql-api-runner-status-will-not-return-paused potentially affects monitoring
2023-04-14 10:39:26 <pj> you use terraform backend on gitlab?
2023-04-14 10:39:49 <psykose> this change makes like zero sense
2023-04-14 10:39:59 <ikke> pj: yes
2023-04-14 10:40:04 <psykose> it's like some guy didn't like the horizontal triangle shape of some block of code
2023-04-14 10:40:06 <psykose> and just changed it around
2023-04-14 10:40:13 <psykose> literally what is the benefit
2023-04-14 10:40:25 <psykose> oh yeah you removed some enum values and added a boolean next to it
2023-04-14 10:40:26 <psykose> profound
2023-04-14 10:40:27 <pj> ikke: i heard from a colleague it's quite shit
2023-04-14 10:40:52 <ikke> It probably has limitations, but for us it works fine
2023-04-14 10:41:07 <ikke> Our use is not that heavy
2023-04-14 10:41:10 <pj> hmm, thanks
2023-04-14 10:41:37 <pj> yeah, we are full on terraform so a lot matters to us
2023-04-14 10:42:16 <ikke> Do you have large statefiles?
2023-04-14 10:42:59 <ikke> psykose: oh, we do need to upgrade postgres
2023-04-14 10:43:06 <psykose> that's easy enough
2023-04-14 10:43:38 <ikke> You do need to export and import and make sure extensions are enabled again
2023-04-14 10:46:42 <ikke> maybe slightly more anoying is that they change the ci runner registration token workflow
2023-04-14 10:46:54 <ikke> what I understood is that you need to register a per-runner token in advance
2023-04-14 10:48:54 <psykose> yeah, annoying :/
2023-04-14 16:14:34 <ncopa> Did the ppc64le kernel arrive? Do we dare to upgrade it on a Friday?
2023-04-14 16:15:03 <ikke> yes, it's there
2023-04-14 16:16:10 <ikke> New kernel is installed
2023-04-14 16:17:10 <ncopa> but not rebooted. should I reboot it?
2023-04-14 16:27:40 <ikke> I can reboot it
2023-04-14 16:31:17 <ncopa> Thanks
2023-04-14 16:31:25 <ikke> back
2023-04-14 16:31:32 <ikke> wow
2023-04-14 16:31:34 <ikke> no throw routes
2023-04-14 16:32:19 <ikke> seems that fixed it
2023-04-15 18:17:28 <psykose> ikke: think ppc64le builders are stuck on uploading
2023-04-15 18:21:51 <ikke> psykose: seems to be still doing something, but slow
2023-04-15 18:21:59 <psykose> it's been doing it for a really long time :D
2023-04-15 18:22:03 <psykose> i think over a day
2023-04-15 18:54:22 <psykose> same for 3.17 too
2023-04-15 19:40:01 <ikke> got distracted
2023-04-15 19:40:32 <ikke> didn't touch it, got fixed itself
2023-04-15 22:41:18 <psykose> still kinda gets stuck uploading in ci for some reason
2023-04-15 22:41:22 <psykose> but not installing deps or anything
2023-04-15 22:41:28 <psykose> so, ppc networking still a bit weird
2023-04-16 07:19:57 <ikke> psykose: hasn't it always been?
2023-04-16 07:20:05 <psykose> no
2023-04-16 07:20:09 <psykose> well, weird yes
2023-04-16 07:20:16 <psykose> but not 'hangs for 12 hours at rnadom'
2023-04-16 09:18:39 <mps> anyone have idea why !45982 didn't went to CIs
2023-04-16 09:23:00 <psykose> it did
2023-04-16 09:23:12 <psykose> https://img.ayaya.dev/HtxVTQ2EuU5e
2023-04-16 09:23:22 <psykose> the front page only shows the top one
2023-04-16 09:23:38 <psykose> and rebase+skip-ci will then show skipped for latest, but that's ok
2023-04-16 09:26:46 <mps> merged. thanks
2023-04-17 12:36:12 <ncopa> is there a machine I can use as a ipv6 jump box to reach the arm machine(s)?
2023-04-17 12:36:25 <ncopa> i'd like to start setup the 3.18 builders
2023-04-17 12:41:25 <ikke> deu7
2023-04-17 12:41:50 <ikke> I think you should even be able to setup wg with ipv6 there
2023-04-17 12:44:48 <ncopa> deu7-dev1 [~]# ssh che-bld-1.alpinelinux.org
2023-04-17 12:44:48 <ncopa> ssh: connect to host che-bld-1.alpinelinux.org port 22: Connection refused
2023-04-17 12:54:01 <ncopa> and it times out from my ipv6 wifi net
2023-04-17 13:07:50 <ikke> ncopa: fixed with awall enable adp-ssh-client
2023-04-17 13:30:33 <ncopa> works! thank you very much!
2023-04-18 14:17:44 <HRio> ikke: seems I can not merge gitlab MRs anymore, expected change?
2023-04-18 14:18:06 <HRio> get "Merge blocked: the source branch must be rebased onto the target branch.
2023-04-18 14:18:15 <HRio> but its fully rebased
2023-04-18 14:18:56 <HRio> !46192
2023-04-18 14:22:09 <psykose> it's a bug where if you open an mr that is already up-to-date it's broken
2023-04-18 14:22:15 <psykose> git commit --amend and push 
2023-04-18 14:22:18 <psykose> then it merges
2023-04-18 14:22:39 <psykose> if you open merge requests like 1+ commits behind it also works, etc
2023-04-18 14:24:34 <HRio> lol OK, that explains
2023-04-18 14:25:39 <ikke> I wonder if I should report that as a separate bug. It appears to be different from the empty MR before refresh bug.
2023-04-18 14:25:53 <psykose> it seems completely separate but i bet the root cause is the same
2023-04-18 14:26:01 <psykose> that said, sounds reportable
2023-04-18 14:30:47 <ikke> The empty MR is an issue with asynchronous updates.
2023-04-18 14:56:28 <ncopa> I have updated the build-edge* (except riscv64) to only use usr/share/abuild/default.conf now
2023-04-18 14:57:04 <ncopa> which means we can now push default CFLAGS changes with apk
2023-04-18 14:57:06 <psykose> should do riscv64 too
2023-04-18 14:57:21 <psykose> or else you'll forget :p
2023-04-18 14:57:23 <ncopa> will do as soon I have the last arch bootstrapping abuild
2023-04-18 14:57:29 <ncopa> yeah, you are right
2023-04-18 15:02:14 <ncopa> riscv64 needs to wait til abuild package is updated
2023-04-18 15:03:32 <psykose> could just kill it and restart
2023-04-18 15:03:37 <ikke> 49/125 going to take a while 
2023-04-18 15:14:33 <ncopa> im doing it cowboy style. just git pull and build abuild
2023-04-18 15:14:45 <ncopa> without stopping it
2023-04-18 15:15:18 <ncopa> yeehaa!
2023-04-18 15:15:54 <ncopa> job done :)
2023-04-18 15:16:17 <psykose> :3
2023-04-18 16:09:02 <psykose> probably have cloned amended pushed like 90 times so far to fix gitlab merges
2023-04-18 16:09:03 <psykose> oof
2023-04-19 19:39:03 <mps> it become cumbersome for me to maintain packages. if someone wants to take maintainership of pkgs where I'm maintainer please do
2023-04-19 19:46:10 <ikke> Sad to hear that
2023-04-20 00:53:06 <psykose> ikke: 3.18-x86_64 is stuck
2023-04-20 04:47:31 <ikke> rip, htop not available yet
2023-04-20 04:47:35 <ikke> did it manually
2023-04-20 11:07:07 <ncopa> i wonder if I should have a verbose upload of packages?
2023-04-20 11:07:44 <ncopa> basically post the output of rsync to build.a.o
2023-04-20 11:09:25 <ikke> Wouldn't that be hard to follow
2023-04-20 11:11:02 <ncopa> would make it easier to see if ppc64le rsync is stuck or not
2023-04-20 11:11:27 <psykose> that doesn't sound very useful imo
2023-04-20 11:11:41 <ncopa> ok
2023-04-20 11:18:32 <ncopa> seems like uploading packages from ppc64le machien to dl-master is extremely slow
2023-04-20 11:18:47 <ikke> We noticed that before
2023-04-20 11:32:50 <psykose> ikke: could you killrestart edge-x86_64 so it doesn't waste 6 hours on what will rebuild anyway
2023-04-20 11:34:08 <psykose> thanks
2023-04-20 11:40:35 <ikke> ncopa: is build-3-18-aarch64 stopped on purpose?
2023-04-20 11:58:12 <psykose> was ghc
2023-04-20 11:58:25 <ikke> but the entire container is stopped
2023-04-20 11:58:30 <psykose> ah
2023-04-20 12:45:11 <ncopa> i rebooted it after ghc was done
2023-04-20 12:46:19 <ikke> it didn't start again
2023-04-20 12:46:35 <ncopa> oh, ok
2023-04-20 12:46:35 <ikke> should I start it?
2023-04-20 12:46:45 <ncopa> build-3-18-aarch64 is running?
2023-04-20 12:46:51 <psykose> better catch it
2023-04-20 12:47:00 <ikke> oh
2023-04-20 12:47:10 <ikke> there is a separate container build-3-18-aarch
2023-04-20 12:47:15 <ikke> I guess that was a type
2023-04-20 12:47:18 <ikke> typo
2023-04-20 12:47:30 <ikke> (Somehow I must always misspell typo)
2023-04-20 12:48:07 <ikke> ncopa: can I remove build-3-18-aarch?
2023-04-20 13:13:55 <ncopa> yeah. that was a typo
2023-04-20 13:38:33 <psykose> should probably update the builder hosts to 3.18 when it comes out to take advantage of the mglru thing
2023-04-20 14:16:30 <ncopa> any idea why ppc64le builder is so slow to upload things? is it bandwith limitation? is it lossy network?
2023-04-20 14:17:08 <psykose> latter
2023-04-20 14:17:12 <psykose> bandwidth is fine afaik
2023-04-20 14:17:17 <psykose> it just sometimes gets stuck
2023-04-20 14:17:40 <ncopa> i wonder if it would make sense to add --bwlimit to rsync
2023-04-20 14:17:49 <ikke> high latency does not help
2023-04-20 14:18:00 <psykose> i'm not sure why limiting it would help
2023-04-20 14:18:47 <ncopa> i have seen it in the past. it helps to avoid saturating the link
2023-04-20 14:19:07 <ncopa> if the wan link is saturated it will drop packets
2023-04-20 14:19:17 <psykose> i can see it, but that feels like a very specific issue
2023-04-20 14:19:23 <psykose> generally dropping packets is fine, right
2023-04-20 14:19:31 <psykose> the issue is it gets stuck for like a really long timeout or something after that
2023-04-20 14:19:50 <ikke> is it stuck or just very slowly uploading?
2023-04-20 14:19:54 <psykose> so the 'cure' is  whatever $that is, not necessarily avoiding it
2023-04-20 14:20:05 <psykose> (assuming it's stuck and not exactly just low speed)
2023-04-20 14:20:07 <ncopa> i think dropping like 1% of packets has a significant impact on filetransfer
2023-04-20 14:20:16 <ikke> yes
2023-04-20 14:20:18 <ncopa> i think its just slow
2023-04-20 14:20:21 <psykose> yeah, it's pretty big
2023-04-20 14:20:29 <ncopa> <f+++++++++ ppc64le/clang16-static-16.0.2-r0.apk
2023-04-20 14:20:36 <ikke> tcp uses packetloss as indication that the link is full and decreases it's throughput
2023-04-20 14:21:14 <ikke> ncopa: maybe we can check with iperf how much bandwidth we get
2023-04-20 14:21:30 <ncopa> sounds like a good idea
2023-04-20 14:21:49 <ikke> Pinging to dl-master does not show packetloss though
2023-04-20 14:21:50 <ncopa> i have seen --bwlimit making a huge difference in the past
2023-04-20 14:22:06 <ikke> if the packetloss is caused by a sataurated link that might indeed help
2023-04-20 14:22:48 <ncopa> as it is now, it takes more time to upload the packages than it takes to build them
2023-04-20 14:22:55 <ikke> yeah
2023-04-20 14:23:20 <ncopa> im not sure I want to abort it though, as I dont want to start over
2023-04-20 14:23:41 <psykose> it would have kept it partially anyway
2023-04-20 14:23:49 <psykose> for the rest of the files
2023-04-20 14:23:52 <ncopa> are you sure?
2023-04-20 14:24:02 <psykose> rsync is normally per file
2023-04-20 14:24:06 <ncopa> i think I have --delay-updates
2023-04-20 14:24:08 <psykose> isn't it
2023-04-20 14:24:09 <psykose> ah
2023-04-20 14:24:22 <psykose> nvm then
2023-04-20 14:24:33 <ikke> That should keep them in a .~tmp folder
2023-04-20 14:24:51 <ncopa> so i kill it?
2023-04-20 14:25:22 <psykose> go for it
2023-04-20 14:25:27 <psykose> worst case it takes some more
2023-04-20 14:28:07 <ikke> almost 500 packets, not a single packet dropped
2023-04-20 14:35:45 <ncopa> seems like int continues where it left off
2023-04-20 14:35:53 <ikke> good\
2023-04-20 14:36:21 <ncopa> i have --bwlimit=8M which i think corresponds to ~80Mbit?
2023-04-20 14:36:27 <ikke> I knew about that due to the rsync issue we were facing
2023-04-20 14:36:48 <ikke> 8MB/s would be 64mbit
2023-04-20 14:44:56 <ikke> ncopa: iperf seems to indicate we have ~300-400mbit to europe
2023-04-20 14:45:24 <ikke> One test did return 354 Kbits/sec though
2023-04-20 14:46:37 <ncopa> 611 MiB
2023-04-20 14:46:37 <ncopa> clang16-static-16.0.2-r0 installed size:
2023-04-20 14:46:50 <ncopa> with 8MB/s it would still be 76 seconds
2023-04-20 14:46:55 <ncopa> it takes a lot more
2023-04-20 14:47:16 <ikke> Did you try running rsync interactively?
2023-04-20 14:47:34 <ncopa> no
2023-04-20 14:49:45 <ikke> It should show you how fast it goes
2023-04-20 14:56:42 <ncopa> hum
2023-04-20 14:57:00 <ncopa> when running it interactively i easily get  8MB/s now
2023-04-20 14:58:26 <ncopa> it was pretty fast for a while
2023-04-20 14:59:01 <ncopa> now it got stuck again apparently
2023-04-20 14:59:11 <ikke> It goes up and down I have a feeling
2023-04-20 15:03:44 <ncopa> interactive, without --bwlimit i get ~10MB/s
2023-04-20 15:03:59 <ncopa> havent seen anything over 12MB/s
2023-04-20 15:04:04 <ikke> strange
2023-04-20 15:04:12 <ikke> iperf gets 400mbit
2023-04-20 15:04:23 <ikke> Maybe due to latency rsync is slower?
2023-04-20 15:05:22 <ncopa> or due to ssh layer?
2023-04-20 15:05:25 <ncopa> i dunno
2023-04-20 15:05:42 <ikke> perhaps
2023-04-20 15:05:48 <ncopa> or because files not bit enough to get above 11MB
2023-04-20 15:05:56 <ncopa> all files are done
2023-04-20 15:06:13 <ikke> does rsync start a separate tcp connection per file?
2023-04-20 15:06:31 <ikke> otherwise the tcp window should remain big enough
2023-04-20 15:06:35 <ncopa> no, i dont think it does
2023-04-20 15:06:46 <ikke> I would not expect so
2023-04-20 15:06:57 <ncopa> anyway, its all uploaded now
2023-04-20 15:07:07 <ikke> Something to investigate
2023-04-20 15:07:24 <ikke> The latency to usa.t1.a.o is only 50ms
2023-04-20 15:14:32 <psykose> incidentally i wanted to remove most of the clang/llvm *-static as they're not really used
2023-04-20 15:14:38 <psykose> the build system doesn't make it very easy
2023-04-20 16:21:25 <psykose> is edge armv7 stuck or does openjdk just take that long
2023-04-20 16:26:51 <ikke> CPU is spinning
2023-04-20 16:27:23 <psykose> guess that's ok
2023-04-20 16:27:42 <ikke> I saw one process disappear
2023-04-20 16:27:48 <ikke> looking at the logs
2023-04-20 16:28:40 <ikke> yup, still going
2023-04-20 17:59:41 <psykose> apparently there's a thing called a treeless clone of --filter=tree:0
2023-04-20 17:59:46 <psykose> it's like depth=0 with history
2023-04-20 17:59:58 <psykose> how weird
2023-04-20 18:00:00 <ikke> yes, correct
2023-04-20 18:00:28 <psykose> huge question however
2023-04-20 18:00:34 <psykose> can it make aports clone 5000% faster
2023-04-20 18:01:11 <ikke> maybe
2023-04-20 18:01:30 <ikke> It would still need to download the required objects when you try to access them
2023-04-20 18:01:53 <psykose> yeah
2023-04-20 18:02:00 <psykose> though most builds use next to 0 objects
2023-04-20 18:02:01 <psykose> well
2023-04-20 18:02:14 <psykose> if ap is used it might fullscan anyway
2023-04-20 18:02:24 <psykose> theoretically it's the ideal usecase
2023-04-20 18:02:50 <ikke> for CI it needs the history / trees to find out what changed
2023-04-20 18:03:21 <psykose> yeah, it would have enough history to diff the tree
2023-04-20 18:03:29 <psykose> sounds more efficient than an arbitrary depth
2023-04-20 18:03:36 <psykose> would have to test though
2023-04-20 18:04:34 <ikke> depends on how efficient git can send just those objects
2023-04-20 18:05:02 <psykose> sounds doable with a few testcases one can run anywhere, i.e. just a small script and starting a container and fetching aports or whatever
2023-04-20 18:05:06 <psykose> i can give it a go sometime
2023-04-20 18:05:14 <psykose> and comparing some timing
2023-04-20 19:41:16 <mps> psykose: I'm not against your change CONFIG_EFI_HANDOVER_PROTOCOL=y in linux-edge but I think it is not good idea, linux-edge idea is to 'follow' mainline defaults to test next LTS
2023-04-20 19:41:30 <psykose> linux-lts has the same thing set too
2023-04-20 19:41:36 <psykose> ah you mean the other way
2023-04-20 19:41:51 <psykose> sure, but in this case it's more simple in that it either boots or it doesn't
2023-04-20 19:41:58 <psykose> i'm sure they'll figure out a better solution before then
2023-04-20 19:42:00 <mps> LTS is ok, iirc there it is not marked as 'to be removed in future'
2023-04-20 19:42:53 <mps> to repeat, linux-edge idea is to follow upstream defaults
2023-04-20 23:13:07 <psykose> ikke: edge-aarch64 is stuck
2023-04-22 00:13:12 <psykose> 
ACTION squints

2023-04-22 00:13:17 <psykose> do we flag every personal runner in here
2023-04-22 05:29:47 <ikke> psykose: It discovers all runners at the moment 
2023-04-22 05:29:55 <psykose> aye
2023-04-22 14:26:01 <psykose> ikke: could you kick edge-aarch64
2023-04-22 15:00:36 <ikke> done
2023-04-22 15:01:00 <ikke> gitlab 15.11 released
2023-04-22 15:08:44 <psykose> time to upgrade :D
2023-04-22 15:09:30 <ikke> Well, first gotta build it
2023-04-22 15:10:06 <ikke> https://docs.gitlab.com/ee/update/ 15.11 is not listed here yet
2023-04-22 17:16:05 <psykose> ikke: i think CI arm networking is quite slow again
2023-04-22 17:16:12 <psykose> compared to builders and dev container at least
2023-04-22 19:10:04 <psykose> and arm is predictably out of diskspace
2023-04-22 19:11:30 <ikke> 1.6T ..
2023-04-22 19:12:18 <ikke> ~500G buildlogs
2023-04-22 19:13:38 <psykose> hmm
2023-04-22 19:13:43 <ikke> -rw-r--r--    1 buildoze buildoze  389.0G Apr 22 19:13 nbd-3.24-r0.log
2023-04-22 19:13:49 <psykose> uhhhh
2023-04-22 19:14:51 <psykose> just test logs?
2023-04-22 19:14:56 <ikke> I think so
2023-04-22 19:15:01 <psykose> we should also implement the log gzip thing
2023-04-22 19:15:18 <ikke> removing thata file
2023-04-22 19:16:30 <ikke> what
2023-04-22 19:16:34 <ikke> removing that file and only 30G free
2023-04-22 19:17:11 <psykose> maybe something was pending writes or some weird stuff
2023-04-22 19:17:20 <psykose> also gzipping logs makes them do like
2023-04-22 19:17:30 <psykose> 44Mi demon 2023-04-22 21:16 qt5-qtwebengine-5.15.12-r10.log
2023-04-22 19:17:31 <psykose> 1.1Mi demon 2023-04-22 21:16 qt5-qtwebengine-5.15.12-r10.log.gzip9
2023-04-22 19:17:39 <ikke> yeah
2023-04-22 19:17:40 <psykose> should reeeally do that :p
2023-04-22 19:18:28 <psykose> xz is half that but browsers don't like to view it
2023-04-22 19:18:43 <ikke> did ncopa manage to get nginx to show those file as plain text?
2023-04-22 19:18:58 <psykose> i think there was one header left to fix
2023-04-22 19:19:02 <psykose> but that is very trivial
2023-04-22 19:19:25 <psykose> well, try it
2023-04-22 19:19:30 <psykose> just gzip -9 -k a log
2023-04-22 19:19:32 <psykose> and url it
2023-04-22 19:19:34 <psykose> see if it works
2023-04-22 19:21:21 <ikke> Need to fix uploading / cleaning up distfiles on the new builders
2023-04-22 19:21:24 <ikke> arm builders
2023-04-22 19:21:26 <ikke> forgot that
2023-04-22 19:21:31 <psykose> if it works, find . -type f -name '*.log' -print0 | parallel -0 -j`nproc` gzip -9 {}
2023-04-22 19:21:59 <psykose> save disk space and nuke the server with this one easy trick
2023-04-22 19:22:14 <ikke> that's mostly for distfiles.a.o
2023-04-22 19:22:18 <ikke> deu5
2023-04-22 19:22:22 <psykose> ye
2023-04-22 19:27:37 <psykose> nbd started on 3.18-armv7 i think without the skip, might fill it all up again
2023-04-22 19:31:27 <ikke> rsync is going to take a while
2023-04-22 19:32:17 <psykose> somethin like that
2023-04-22 19:33:27 <ikke> something something electron
2023-04-22 19:37:06 <psykose> the amazing part of electron
2023-04-22 19:37:12 <psykose> it's still smaller than half the golang cloud stuff
2023-04-22 19:55:17 <ikke> dotnet is also fun
2023-04-22 19:56:49 <psykose> not
2023-04-22 20:56:08 <ikke> the rsync script also removes the distfiles once synced, so diskspace is slowly recovered
2023-04-22 21:08:50 <psykose> could you gzip one log so we can test a url
2023-04-22 21:09:22 <ikke> rsync just finished
2023-04-22 21:10:39 <ikke> https://build.alpinelinux.org/buildlogs/build-3-15-x86.log.gz
2023-04-22 21:10:42 <ikke> downloads for me
2023-04-22 21:11:02 <psykose> content-type: application/octet-stream
2023-04-22 21:11:02 <psykose> yeah
2023-04-22 21:11:07 <psykose> you need the header somewhere
2023-04-22 21:12:30 <psykose> Content-Type: text/plain
2023-04-22 21:12:31 <psykose> Content-Encoding: gzip
2023-04-22 21:12:39 <psykose> that should be ok
2023-04-22 21:13:00 <psykose> idk if the darkhttpd had an easy way to do that, and every other httpd makes that 2 lines somewhere iirc
2023-04-22 21:13:20 <psykose> i forgot what served the things
2023-04-22 21:13:45 <ikke> it runs nginx
2023-04-22 21:15:35 <ikke> but only for build.a.o, not for distfiles.a.o
2023-04-22 21:15:43 <ikke> I mean, that we want that
2023-04-22 21:23:16 <psykose> gzip_static on;
2023-04-22 21:23:50 <psykose> well
2023-04-22 21:23:54 <psykose> want always i guess
2023-04-22 21:23:58 <psykose> on is for having both
2023-04-22 21:24:37 <psykose> always; and then also gunzip on; would allow serving both still from just .gz disk
2023-04-22 21:25:17 <psykose> personally i am more of a "if you're fetching these logs you can gunzip them" territory tho
2023-04-22 21:25:30 <psykose> weird clients that take plain only
2023-04-22 21:26:10 <ikke> I keep getting octet-stream though
2023-04-22 21:26:29 <ikke> oh
2023-04-22 21:29:27 <ikke> gzip_static on; does not seem to work
2023-04-22 21:30:02 <psykose> hmm
2023-04-22 21:30:08 <psykose> i think you do still need some mime stuff too then
2023-04-22 21:30:21 <ikke> ok
2023-04-22 21:31:20 <ikke> https://build.alpinelinux.org/buildlogs/build-3-15-x86.log.gz
2023-04-22 21:31:42 <psykose> works
2023-04-22 21:31:54 <ikke> yes
2023-04-22 21:31:55 <psykose> well
2023-04-22 21:32:02 <psykose> the headers are still 'wrong' in a funny way
2023-04-22 21:32:04 <psykose> it's set twice
2023-04-22 21:32:21 <psykose> < content-type: application/octet-stream 
2023-04-22 21:32:21 <psykose> < content-type: text/plain
2023-04-22 21:32:21 <psykose> < content-encoding: gzip
2023-04-22 21:32:33 <psykose> that does work of course but it's humorous
2023-04-22 21:32:33 <ikke> yeah, I see
2023-04-22 21:34:11 <ikke> I can override the types per location
2023-04-22 21:34:49 <psykose> that works yeah
2023-04-22 21:35:00 <ikke> fixed
2023-04-22 21:35:37 <psykose> ah and since parallel is probably something you don't want, just find . -type f -name '*.log' -print0 | xargs -0 -n 1 -P `nproc` gzip -9
2023-04-22 21:35:42 <ikke> Now it downloads again
2023-04-22 21:35:46 <psykose> hmmmm
2023-04-22 21:35:54 <psykose> what did you put
2023-04-22 21:36:06 <psykose> now it works
2023-04-22 21:36:18 <ikke> didn't change anything
2023-04-22 21:36:32 <ikke> I mean, since adding types {}
2023-04-22 21:36:41 <ikke> but yeah, works me now as well
2023-04-22 21:40:28 <ikke> it does mean you always need to add --compressed to curl
2023-04-22 21:47:26 <psykose> without the gunzip module yeah
2023-04-22 21:50:34 <ikke> ok, that works
2023-04-22 21:52:26 <psykose> ready to gzip the world? :)
2023-04-22 21:55:05 <ikke> tomorrow
2023-04-22 21:55:21 <psykose> bedtime
2023-04-22 21:55:39 <ikke> Running rdfind atm
2023-04-22 21:56:09 <ikke> On distfiles, that is
2023-04-22 22:07:32 <psykose> what's it findin
2023-04-22 23:14:22 <psykose> and of course edge-armv7 is stuck
2023-04-23 00:07:23 <psykose> ikke: also ppc seems to be stuck in pulling git on most of the release builders
2023-04-23 05:33:33 <ikke> rdfind: Totally, 457 GiB can be reduced.
2023-04-23 06:30:47 <ikke> cleaned up distfiles for edge to only keep what is currently in source= in aports
2023-04-23 07:20:51 <ikke> psykose: we need to tell nginx to look for <file.log>.gz if <file.log> does not exist to keep existin links working
2023-04-23 07:29:25 <ikke> (try_files)
2023-04-23 12:05:16 <ikke> psykose: I think that gunzip does not work if you directly navigate to the .gz file
2023-04-23 12:05:19 <ikke> with curl
2023-04-23 12:10:53 <ikke> Ok, found something that seems to work in all cases
2023-04-23 14:19:55 <psykose> sweet :)
2023-04-23 14:30:45 <ikke> https://tpaste.us/Wx5L 
2023-04-23 14:31:25 <ikke> gzip_static always works even if there is no .gz file
2023-04-23 15:10:35 <psykose> 🥳
2023-04-23 15:12:38 <ikke> I guess we want to compress new files with a cron job?
2023-04-23 15:19:17 <psykose> they should be written compressed by whatever runs the build
2023-04-23 15:19:21 <psykose> but i guess cron is also ok
2023-04-23 15:19:40 <ikke> otherwise we need to modify buildrepo / aports-build
2023-04-23 15:24:50 <psykose> yeah
2023-04-23 15:24:55 <psykose> edge-armv7 is still stuck
2023-04-23 17:47:58 <psykose> and all the ppc git clones on stable and pushes are stuck too :D
2023-04-23 17:50:56 <ikke> not sure what to do about that
2023-04-23 17:52:04 <psykose> me either
2023-04-23 17:55:23 <ikke> Seems like they are just idle
2023-04-23 18:05:42 <psykose> sweet
2023-04-23 18:07:03 <ikke> I have a feeling gitally has become more slow since the last upgrade
2023-04-23 18:09:43 <psykose> the one ages ago or did you do another
2023-04-23 18:09:47 <psykose> but yeah sounds about right
2023-04-23 18:10:04 <ikke> The one that introduced the MR issues
2023-04-23 18:10:14 <psykose> yeah
2023-04-23 18:10:14 <ikke> Which was the last one I did
2023-04-23 18:10:25 <psykose> just jump 2 versions and hope everything is fixed
2023-04-23 18:10:29 <psykose> /s
2023-04-23 18:10:44 <ikke> The MR is empty issue will not be fixed before 16.0 most likely
2023-04-23 18:19:02 <psykose> good to stay current anyway i guess
2023-04-23 18:20:48 <psykose> i wonder if they actually improved the webui or made it even worse by now
2023-04-23 18:21:48 <ikke> The epic seems to indicate there is no clear pictue how they even want to handle these thins
2023-04-23 18:21:50 <ikke> things
2023-04-23 18:22:09 <psykose> my biggest pet peeve in general is it's impossible to view changes when there is like 100 changed files
2023-04-23 18:22:29 <psykose> you scroll and it's the gray nonrendered screen for like 4 seconds until it loads in
2023-04-23 18:22:39 <psykose> and if you make the mistake of actually clicking a file to jump to it.. nothing happens
2023-04-23 18:22:43 <psykose> until you try and scroll
2023-04-23 18:22:50 <psykose> then it starts jumping around the whole page
2023-04-23 18:23:02 <psykose> or maybe that is just a "i have more than 100% zoom" kind of bug
2023-04-23 18:37:48 <ikke> They dropped support for ruby 2.7 in gitlab 15.10, but they do not support 3.1 yet
2023-04-23 18:37:56 <ikke> fully at least
2023-04-23 18:39:03 <psykose> 3.0 then?
2023-04-23 18:39:43 <ikke> I do have a branch with alpine 3.17 + ruby 3.1
2023-04-23 18:42:27 <ikke> For ruby 3.0 we're stuck on alpine 3.16 (as that's what the ruby image we use provides
2023-04-23 18:45:10 <ikke> Can't wait for this to land: https://gitlab.com/gitlab-org/gitaly/-/issues/4742
2023-04-23 18:46:15 <psykose> i think it already has, hasn't it
2023-04-23 18:46:28 <psykose> i.e. the go gitaly already exists
2023-04-23 18:46:34 <psykose> there's just cleanup mrs pending
2023-04-23 18:48:36 <ikke> gitaly has always been in go
2023-04-23 18:49:04 <ikke> (or at least, since a long time)
2023-04-23 18:49:11 <ikke> there are just many ruby components
2023-04-23 18:49:18 <ikke> that they have been slowly migrating to go code
2023-04-23 18:49:33 <ikke> This is about removing the last ruby code
2023-04-23 18:50:06 <psykose> ah
2023-04-23 18:50:07 <psykose> right
2023-04-23 18:50:09 <ikke> (building / installing all the ruby gems takes longer than building gitaly and git combined
2023-04-23 18:54:01 <psykose> 3.18-aarch64 is stuck on stunnel as is usual
2023-04-23 18:54:04 <psykose> ikke: not surprised tbh
2023-04-23 18:54:16 <psykose> wish they could at least do their infinite configures in parallel
2023-04-23 18:55:50 <ikke> stunnel is unstuck
2023-04-23 18:59:41 <psykose> i don't think so
2023-04-23 19:00:17 <ikke> kicked a bit harder
2023-04-23 19:00:20 <psykose> ah
2023-04-23 19:00:49 <ikke> these processes are all terrified of 9
2023-04-23 19:04:09 <psykose> 9 six'd 15
2023-04-23 20:23:17 <psykose> now it's s390x stunnel that is stuck :D
2023-04-23 20:23:23 <psykose> and armhf py3-loky
2023-04-23 20:26:18 <ikke> p00f
2023-04-23 20:27:15 <psykose> rekt
2023-04-24 15:16:19 <ikke> buildrepo on s390x edge is spinning 100%
2023-04-24 15:19:22 <psykose> not great
2023-04-27 03:34:31 <psykose> ikke: is there a way for me to look at why firefox on riscv64 gets killed after about 65 minutes in a loop repeatedly, until after 30 tries it happens to finish in right under that time
2023-04-27 05:16:10 <psykose> also gonna restart deu1/deu7
2023-04-27 07:03:28 <psykose> ikke: also edge-s390x has been stuck for a while
2023-04-27 07:15:31 <ncopa> im working on s390x
2023-04-27 07:15:37 <ncopa> it had a full disk
2023-04-27 07:15:42 <ncopa> i have deleted distfiles
2023-04-27 07:16:04 <ncopa> and now I am copying out build-3-[6-9]-* so i can delete those
2023-04-27 07:16:11 <ncopa> should be enough to get dotnet built at least
2023-04-27 07:16:28 <psykose> i disabled dotnet already
2023-04-27 07:16:37 <ncopa> oh
2023-04-27 07:16:45 <ncopa> i guess thats fine
2023-04-27 07:16:47 <psykose> (but right before it cleaned itself i guess)
2023-04-27 07:17:00 <psykose> better things to do than nurse something that takes like 90g of disk to build
2023-04-27 07:17:08 <ncopa> heh, yeah
2023-04-27 08:51:17 <kunkku> ikke: I have a couple of questions on IPv6 addresses
2023-04-27 08:53:44 <kunkku> 1) in the CA DB, dmvpn1's GRE address starts with fde7 but /etc/network/interfaces has fde8. Is the latter a mistake?
2023-04-27 08:56:34 <kunkku> 2) if one can derive the subnets like 172.16.xx.0/24 -> fde7:4eeb:852a:xx::/64, which IPv6 subnet should be used by the development VPNc (172.21.x.0/24)?
2023-04-27 08:58:22 <kunkku> I have made a couple of IPv6 fixes to nhrpd and I would like to try them out on my dev vpnc first
2023-04-27 09:03:11 <ikke> kunkku: I'll answer in a bit
2023-04-27 09:06:12 <ncopa> will this allow us to route 172.16 over gre tunnel over ipv6? eg allow us to connect the arm machines over nhrp tunnels over ipv6?
2023-04-27 09:06:31 <ncopa> private ipv4 over public ipv6
2023-04-27 09:08:00 <kunkku> the fixes are for ipv6 over gre
2023-04-27 09:08:41 <kunkku> mgre over ipv4 is broken on kernel level
2023-04-27 09:09:01 <kunkku> I mean mgre over ipv6
2023-04-27 09:10:12 <kunkku> ncopa: is that something we would like to fix?
2023-04-27 09:17:34 <ncopa> would be nice, but i dont think it is a priority
2023-04-27 09:19:16 <ncopa> woudl be nice if I somehow could route to the arm server
2023-04-27 09:29:10 <ncopa> nld9-dev1 runs out of disk space
2023-04-27 09:29:14 <ncopa> im deleting some distfiles
2023-04-27 09:36:19 <ncopa> i have backed up build-3-6-s390x  build-3-7-s390x  build-3-8-s390x  build-3-9-s390x on a local usb disk and are deleteting those now
2023-04-27 09:42:49 <ncopa> hum, we dont wipe /tmp on the builders on reboot
2023-04-27 09:42:58 <ncopa> there are a zillion files there
2023-04-27 09:50:47 <ncopa> I think we should refert the "keep /tmp by default" change https://gitlab.alpinelinux.org/alpine/aports/-/commit/bd6326e312f3bb56ba504c0571db142990af036c
2023-04-27 09:56:55 <ikke> kunkku: the prefix is fde7:4eeb:852a::/48
2023-04-27 09:57:10 <ikke> so /etc/network/interfaces is incorrect
2023-04-27 09:58:37 <ikke> kunkku: the dmvpn prefix I used is a 56
2023-04-27 09:58:41 <ikke> is /56
2023-04-27 10:01:09 <ikke> kunkku: so you can use fde7:4eeb:852a:100/64 to fde7:4eeb:852a:1ff::/64
2023-04-27 10:11:58 <kunkku> your scheme assumes the 3rd octet has 2 digits as there is no decimal to hex conversion
2023-04-27 10:13:42 <kunkku> 172.16.26.0/24 (decimal) maps to fde7:4eeb:852a:26::/64 (hex)
2023-04-27 10:14:29 <kunkku> but maybe this is fine if we are not going to have 100 sites
2023-04-27 10:18:35 <ikke> Yeah, we should actually convert the decimal numbers to hexadecimaal
2023-04-27 10:34:52 <ncopa> i have stopped build-3-1[67]-x86_64 for now
2023-04-27 10:35:06 <ncopa> we need let edge and 3.18 builders build the kernel first
2023-04-27 10:35:40 <ncopa> im also backing up older build- 3-*-x86_64 so we can delete them
2023-04-27 10:35:54 <ncopa> the low diskspace is a serious problem now :-/
2023-04-27 10:36:27 <ikke> yes
2023-04-27 10:36:30 <ikke> I was afraid of that
2023-04-27 10:36:45 <ncopa> im also setting wipe_tmp=YES on some of the builders
2023-04-27 10:36:51 <ncopa> its should be off for all
2023-04-27 10:37:03 <ncopa> i mean should be enabled everywhere
2023-04-27 10:37:36 <ncopa> or maybe we should mount /tmp as tmpfs on the lxc containers?
2023-04-27 10:37:41 <ncopa> thats probably better
2023-04-27 10:37:47 <ikke> I'd say so
2023-04-27 10:40:09 <ncopa> everythign touching disk is slow on the nld9 now
2023-04-27 10:41:06 <ikke> due to you backing things up?
2023-04-27 10:41:12 <ncopa> possibly
2023-04-27 10:41:20 <ncopa> dont know
2023-04-27 10:43:42 <ncopa> or maybe due to kernel compile
2023-04-27 10:52:25 <ncopa> what size should we use for /tmp ?
2023-04-27 10:52:42 <ncopa> default? which is 50% of memory
2023-04-27 10:52:55 <ikke> We rarely have changed the size
2023-04-27 10:53:03 <ikke> It only uses what is stored
2023-04-27 10:53:54 <ikke> kunkku: so fde7:4eeb:852a:26::/64 should be fde7:4eeb:852a:1a::/64
2023-04-27 10:55:32 <ncopa> for i in */config; do echo "lxc.mount.entry=tmpfs tmp tmpfs nodev,nosuid,mode=1777 0 0" >> "$i"; done
2023-04-27 10:56:27 <ikke> looks good
2023-04-27 11:13:51 <ncopa> i have stopped build-edge-x86_&4
2023-04-27 11:14:00 <ncopa> the /tmp there is "argument list too long"
2023-04-27 11:14:06 <ncopa> its just insane amount of files there
2023-04-27 11:14:25 <ncopa> the thing is, /tmp should be deleted each reboot
2023-04-27 11:14:40 <ncopa> it hasn't been done since a couple of years ago
2023-04-27 11:14:44 <ikke> ncopa: we should probably also upgrade nld8/9
2023-04-27 11:15:11 <ncopa> ok, maybe I should do that while at it?
2023-04-27 11:15:27 <ncopa> or you mean new hardware?
2023-04-27 11:15:34 <ikke> no, os
2023-04-27 11:15:55 <ncopa> ok
2023-04-27 11:16:10 <ncopa> i'll build kernel for 3.18 first
2023-04-27 11:16:17 <ncopa> delete stuff
2023-04-27 11:16:28 <ncopa> and maybe build kernel on edge
2023-04-27 11:16:39 <ncopa> and then do host upgrade and reboot it
2023-04-27 11:28:48 <ncopa> its still deleting /tmp of build-edge-x86_64
2023-04-27 11:28:54 <ncopa> been deleting for the last 15 mins
2023-04-27 11:29:14 <ikke> ouch
2023-04-27 11:29:55 <ncopa> good thing is that is it is a very simple way to get back disk space
2023-04-27 11:30:03 <ikke> was it taking so much space?
2023-04-27 11:48:16 <ncopa> finally done deleting
2023-04-27 12:14:47 <ncopa> im upgrading nld9 now
2023-04-27 12:15:02 <ncopa> should I jump directly to 3.18 or do I upgrade it to 3.17?
2023-04-27 12:15:51 <ikke> Good question, do we trust 3.18 already?
2023-04-27 12:16:06 <ncopa> not really :)
2023-04-27 12:16:26 <ncopa> but it should not be worse than current edge
2023-04-27 12:16:40 <ikke> Yeah, but we do not run a lot on edge directly
2023-04-27 12:16:43 <ncopa> lets do 3.17 for now I suppose
2023-04-27 12:46:29 <ncopa> im gonna reboot nld9 once the linux-lts is done building in edge
2023-04-27 12:56:04 <ncopa> im rebooting nld9 now
2023-04-27 12:58:51 <ikke> :)
2023-04-27 13:29:08 <ncopa> im updating nld8 now
2023-04-27 13:31:51 <ncopa> actually, im deleteing the tmp/* first
2023-04-27 13:32:04 <ncopa> its gonna take some time apparently
2023-04-27 13:40:46 <ncopa> deleting all the /tmp is actually freeing some disk space
2023-04-27 13:51:05 <ncopa> i found various aports/*/*/src directories. Im deleting them
2023-04-27 13:51:13 <ikke> nod
2023-04-27 13:51:42 <ikke> did you also clean up /home/abuild/.* and /home/abuild/{go,.cargo}?
2023-04-27 13:51:51 <ikke> s/abuild/buildozer
2023-04-27 13:52:17 <ncopa> nope
2023-04-27 13:52:19 <ncopa> not yet
2023-04-27 13:53:20 <ncopa> ok. loads of stuff there too
2023-04-27 13:53:24 <ikke> yup
2023-04-27 13:57:50 <ncopa> nuking it now
2023-04-27 14:10:01 <ncopa> nld8 is rebooting
2023-04-27 14:10:57 <ikke> ncopa: https://i.imgur.com/zClHZyx.png :)
2023-04-27 14:10:59 <ncopa> should I document somewhere that nld8 and nld9 are alpine 3.17?
2023-04-27 14:11:20 <ikke> I keep track of it in netbox
2023-04-27 14:11:32 <ncopa> cool
2023-04-27 14:11:47 <ikke> Do you want to update it there? It's the platform field of the devive
2023-04-27 14:13:23 <ncopa> i have updated it in netbox
2023-04-27 14:13:25 <ncopa> thanks!
2023-04-27 14:14:50 <ikke> 👍
2023-04-27 14:20:16 <ncopa> im deleting some go and .cargo on x86_64 as well now
2023-04-27 14:22:04 <ncopa> i now declare nld8 and nld9 healty again
2023-04-27 14:47:35 <ncopa> i have remoutned /tmp as tmpfs on s390x.a.o containers and deleted the /tmp/*
2023-04-27 15:09:13 <ncopa> i have cleaned up /tmp on ppc64le machine
2023-04-27 15:11:07 <ikke> nice
2023-04-27 15:24:31 <ncopa> the only thing left now is arm builders and riscv64
2023-04-27 15:24:58 <ncopa> but i think we should be ok for the 3.18 release at least
2023-04-27 15:25:14 <ikke> ncopa: I can look at them
2023-04-29 00:53:12 <ptrc> oh hey, those are my runners; what are they doing here?
2023-04-29 05:25:57 <ikke> ptrc: zabbix automatically discovers all runners 
2023-04-29 05:26:39 <ikke> Need to think of a way to discover only (but all) infra managed runners 
2023-04-29 05:29:19 <psykose> ikke: is there a way to tag runners in a way that only an admin could?
2023-04-29 05:29:37 <psykose> so you would just tag them something specific and use some api for that i guess
2023-04-29 05:29:40 <ikke> Not that I'm aware of
2023-04-29 05:30:27 <ikke> psykose: I already notice people just copying the tags from official runners 
2023-04-29 05:30:40 <ikke> It helps somewhat 
2023-04-29 05:30:45 <psykose> yeah
2023-04-30 01:14:05 <psykose> ikke: could you kick the dotnets
2023-04-30 01:34:03 <psykose> and the stunnel
2023-04-30 11:25:03 <psykose> and a second time i guess
2023-04-30 11:25:04 <psykose> cringe
2023-04-30 11:31:55 <ikke> dotnet on aarch64 is still continuing
2023-04-30 11:44:48 <psykose> same for armv7?
2023-04-30 11:47:11 <ikke> the buildlog implies it's already finished
2023-04-30 11:47:31 <ikke> but the processes imply it's still building
2023-04-30 11:47:45 <ikke> and still continuing
2023-04-30 11:48:20 <ikke> oh, wrong builder logs
2023-04-30 11:48:36 <ikke> But yeah, it's still continuing
2023-04-30 11:52:16 <psykose> cute
2023-04-30 11:52:37 <psykose> ikke: could you look through the riscv64 triggered build chain of stuff magic and see if there's something that waits like an hour and dies
2023-04-30 12:08:47 <ikke> Ah, found it
2023-04-30 12:08:49 <ikke> finally
2023-04-30 12:08:58 <ikke> /etc/periodic/15min
2023-04-30 12:09:08 <psykose> :)
2023-04-30 12:09:13 <psykose> what was it
2023-04-30 12:09:19 <ikke> check-abuild.sh
2023-04-30 12:09:28 <psykose> it has been driving me insane for months :p
2023-04-30 12:09:45 <ikke> I did recall clandmeter implemented osmething, but didn't know how
2023-04-30 12:10:17 <ikke> https://tpaste.us/RBpr
2023-04-30 12:10:40 <psykose> yeah, i just remember we disabled it, etc etc
2023-04-30 12:11:00 <ikke> It does imply nothing gets written to the log?
2023-04-30 12:11:16 <psykose> even the usual kill -9 in there
2023-04-30 12:11:17 <psykose> epic
2023-04-30 13:14:18 <ikke> psykose: I've disabled that cron job now
2023-04-30 13:14:56 <psykose> thanks
2023-04-30 14:40:36 <psykose> and guess what, stunnel again :D
2023-04-30 14:40:37 <psykose> which test is it
2023-04-30 14:40:45 <psykose> p02_require_cert?
2023-04-30 14:45:14 <ikke> 062. Failure test "PSKsecrets" option
2023-04-30 14:46:09 <ikke> funny
2023-04-30 14:46:16 <ikke> strace -> killed by segfault
2023-04-30 14:47:16 <psykose> i just tested it randomly and it didn't hang on test
2023-04-30 14:47:24 <psykose> it hung in maketest.py, the thing that starts them, before a single one ran
2023-04-30 14:47:29 <psykose> and only with malloc-ng
2023-04-30 14:47:59 <psykose> still very random tho
2023-04-30 14:48:09 <psykose> now it's verifyChain
2023-04-30 17:09:49 <psykose> sure are a lot of spammers that make an account and fork one random repo
2023-04-30 17:09:57 <psykose> it's like one every 10 minutes
2023-04-30 17:10:16 <ikke> hm
2023-04-30 17:10:52 <psykose> easy fix: just ban outlook.com lmfao
2023-04-30 17:11:07 <ikke> It's not only outlook.com though
2023-04-30 17:12:00 <psykose> not 100% no
2023-04-30 17:13:54 <psykose> something that is actually 100% is 'require 2fa' but eventually i bet they would just have that too
2023-04-30 19:53:44 <ikke> Either somethings wrong, or someone is spamming flags for pkgs.a.o
2023-04-30 19:54:48 <ikke> I receive e-mail about (almost?) all packages I maintain specifying version 1 is available
2023-04-30 19:56:44 <psykose> yeah
2023-04-30 19:56:49 <psykose> i've gotten quite a few
2023-04-30 19:57:06 <psykose> the amount of manual flags to noise i've seen recently is like 1:100
2023-04-30 19:57:27 <psykose> they're all either things in anitya anyway a few hours earlier or useless bug reports or spam or things intentionally not upgraded
2023-04-30 19:57:35 <psykose> kind of tempted to just disable the feature :p
2023-04-30 19:58:15 <psykose> i think it might ruin our email with it, no
2023-04-30 19:58:25 <psykose> since now our mailserver sent out like a million spam emails at once, via the page
2023-04-30 19:58:36 <ikke> Was thinking the same
2023-04-30 19:59:01 <psykose> it's kind of almost an open proxy since anyone can type shit into the page and email someone else
2023-04-30 19:59:25 <ptrc> flagged captcha when
2023-04-30 19:59:30 <psykose> lol
2023-04-30 19:59:38 <psykose> wouldn't solve the non-spam useless flags
2023-04-30 19:59:46 <ptrc> also, i suppose a simple check "if version above current" would fix some things
2023-04-30 20:00:38 <psykose> ther eis a captcha actually
2023-04-30 20:00:46 <ptrc> yeah i haven't seen it before
2023-04-30 23:10:41 <psykose> i added some domains to the gitlab signup blocklist
2023-04-30 23:12:52 <psykose> should catch like 90% of these new accounts