2023-10-02 06:40:45 <ncopa> i plan to set up the 3.19 builders this or next week
2023-10-02 07:10:35 <clandmeter> the che network is very frustrating
2023-10-02 07:10:50 <clandmeter> my images are not build now due to missing CI
2023-10-02 08:49:34 <ncopa> is che machines still unreachable?
2023-10-02 08:55:35 <ikke> Yes
2023-10-02 09:12:19 <ncopa> for how long have they been unreachable now?
2023-10-02 09:19:09 <ikke> 12 days
2023-10-03 09:29:54 <nmeum> is there some sort of network issue with the s390x CI?
2023-10-03 09:29:56 <nmeum> https://gitlab.alpinelinux.org/Celeste/aports/-/jobs/1132121
2023-10-03 09:30:06 <nmeum> looks like it has been attempting to fetch the APKINDEX for 35 minutes
2023-10-03 09:33:29 <ikke> nmeum: there is some issue yes, but not sure if it's pure network 
2023-10-03 09:33:37 <ikke> apk is hanging on read
2023-10-03 09:33:41 <ikke> It's always apk
2023-10-03 09:35:00 <ikke> Haven't tested if it's only dl-cdn it has issues with, but it seems only specific operations have issues (ie, it always hangs on a few specific points)
2023-10-03 09:35:15 <ikke> apk update/fetch 
2023-10-03 09:36:07 <nmeum> so apk is generally misbehaving on s390x or only on the gitlab ci? the s390x builders work fine, right?
2023-10-03 09:36:24 <ikke> Yes, only on s390x and only on ci
2023-10-03 15:12:50 <ncopa> does apk use http or https?
2023-10-03 15:13:23 <ncopa> fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/s390x/APKINDEX.tar.gz
2023-10-03 15:13:48 <ncopa> i know that PMTU is broken with fastly
2023-10-03 15:27:37 <ikke> Https 
2023-10-04 18:53:04 <ikke> raspbeguy: usrhere fyi, there are a couple of mirror requests that you can handle if you want: https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues
2023-10-04 18:58:03 <usrhere> ikke: thanks, I will check
2023-10-04 18:58:47 <ikke> Basically: 1) A merge request to add the mirror in private.yml, 2) add the mirror to zabbix (easiest to clone an existing mirror)
2023-10-04 18:58:53 <ikke> I'll try to document this
2023-10-04 19:00:38 <usrhere> ikke: #1 step is clear, I will do it in a minute
2023-10-04 19:00:55 <usrhere> zabbix part should be ok too
2023-10-04 19:01:32 <ikke> There are 2 macros that you can adjust, 1 for the mirror path and one to switch between http and https
2023-10-04 19:05:48 <raspbeguy> ikke, usrhere: could you let me do the next one when it occurs please?
2023-10-04 19:06:00 <ikke> There are more then one
2023-10-04 19:06:04 <ikke> so you could divide them
2023-10-04 19:11:24 <raspbeguy> Ok then I'll do the rest tomorrow morning 
2023-10-04 19:11:45 <ikke> usrhere: which ones are you doing?
2023-10-04 19:12:26 <usrhere> https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues/633 so far
2023-10-04 19:12:58 <usrhere> raspbeguy: maybe you could assign some to yourself
2023-10-04 19:14:40 <raspbeguy> Could you do assign some to me please? I don't have any computer nearby
2023-10-04 19:14:41 <usrhere> ikke: https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues/633 what would you name as a sponsor here ? Doesn't look like they have a website
2023-10-04 19:15:03 <usrhere> raspbeguy: I will leave some for you unassigned then
2023-10-04 19:15:10 <raspbeguy> Ok
2023-10-04 19:15:53 <ikke> usrhere: Maybe just mirror.5i.fi
2023-10-04 19:29:03 <ikke> You can respond to the issue to let the requester know that the mirror has been added and close it after the MR has been merged
2023-10-04 19:31:22 <usrhere> ok, will do
2023-10-04 19:37:17 <ikke> I've sent a bunch of e-mails to mirrors that are out of date / not working
2023-10-04 19:37:34 <usrhere> ikke: https://zabbix.alpinelinux.org/zabbix.php?name=&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=name&sortorder=ASC&show_details=0&action=latest.view&hostids%5B%5D=10463
2023-10-04 19:37:47 <usrhere> if you would like to check zabbix addition
2023-10-04 19:38:02 <ikke> Checking it
2023-10-04 19:38:29 <ikke> Looks good
2023-10-04 19:41:42 <usrhere> cool :)
2023-10-04 20:37:15 <usrhere> ikke: I added mirror.jingk.ai to zabbix too
2023-10-04 20:38:26 <ikke> looks fine as well
2023-10-04 20:41:49 <usrhere> I will do more in upcoming days, if raspbeguy will leave something :)
2023-10-04 20:42:48 <usrhere> ikke: thanks for creating those issues, I think that's a good way to integrate us step-by-step
2023-10-04 20:43:03 <usrhere> I will keep an eye on that list
2023-10-04 20:43:44 <ikke> No problem, they are created automatically when we receive e-mail on mirrors@alpinelinux.org
2023-10-04 20:43:56 <ikke> You can also subscribe to new issues if you want
2023-10-04 20:44:10 <usrhere> cool, I will
2023-10-04 20:44:40 <usrhere> would be nice if you could create some more issues that are good for us
2023-10-04 20:45:13 <ikke> Yeah, absolutely
2023-10-04 20:46:07 <usrhere> at least for me it's much easier to operate like this, instead of trying to find something to do by myself - I could just check the issue list
2023-10-04 20:46:47 <usrhere> especially when we're limited by permissions etc
2023-10-04 20:47:11 <usrhere> maybe you could even introduce a label or something for us
2023-10-04 20:47:52 <ikke> Yeah, was thinking about something like that
2023-10-04 20:49:12 <usrhere> great
2023-10-04 20:49:32 <ikke> One thing I'm thinking of is upgrading netbox
2023-10-04 20:49:32 <usrhere> I'm done for today, have a nice evening
2023-10-04 20:49:44 <ikke> you too, I was about to head off as well
2023-10-04 20:50:35 <usrhere> thx, let's chat about netbox next time, I would be interested
2023-10-05 07:08:46 <ikke> ncopa: usa6 is still on alpine 3.15, I was thinking asking IBM support with upgrading it.
2023-10-05 07:09:33 <ncopa> sounds good
2023-10-05 07:19:52 <ncopa> also usa2 is on alpine 3.15
2023-10-05 07:19:55 <ncopa> even older
2023-10-05 07:20:11 <ncopa> i wonder if I should just upgrade it to a newer 5.15 kernel and reboot it
2023-10-05 07:27:53 <ikke> Maybe start with the ci host?
2023-10-05 07:43:01 <ncopa> ok
2023-10-05 07:43:32 <raspbeguy> ikke, how do you usually estimate a mirror bandwidth?
2023-10-05 07:43:45 <ncopa> upgrading gitlab-runner-s390x to latest alpine 3.15
2023-10-05 07:43:49 <ncopa> rebooting it now...
2023-10-05 07:44:17 <ikke> raspbeguy: usually it's provided or we ask them
2023-10-05 07:44:47 <raspbeguy> ok, so I won't try to guess
2023-10-05 07:53:01 <raspbeguy> I guess we could use tome issue templates asking for basic info
2023-10-05 07:53:06 <raspbeguy> some*
2023-10-05 08:20:59 <raspbeguy> ikke, so we should ask them to install zabbix agent?
2023-10-05 08:22:20 <ncopa> gitlab-runner-s390x [~]# uptime
2023-10-05 08:22:20 <ncopa>  08:22:12 up 37 min,  0 users,  load average: 0.18, 0.18, 0.18
2023-10-05 08:22:26 <ncopa> successfully rebooted
2023-10-05 08:22:34 <ncopa> do I dare upgrade it to alpine 3.18?
2023-10-05 08:36:56 <ncopa> gitlab-runner-s390x [~]# du -sh /var/log/docker.log 
2023-10-05 08:36:56 <ncopa> 629.3M	/var/log/docker.log
2023-10-05 08:37:04 <ncopa> we need rotate the docker.log
2023-10-05 08:38:39 <ncopa> time="2023-10-05T08:37:35.731211849Z" level=error msg="failed to mount overlay: no such device" storage-driver=overlay2
2023-10-05 08:38:39 <ncopa> time="2023-10-05T08:37:35.731227444Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
2023-10-05 08:39:26 <ncopa> aw....
2023-10-05 08:39:38 <ncopa> something is very off with the gitlab-runner-s390x
2023-10-05 08:39:49 <ncopa> it did not reboot the kernel
2023-10-05 08:40:44 <ncopa> uptime says it rebooted
2023-10-05 08:40:53 <ncopa> but it booted the old kernel for some reason
2023-10-05 08:42:46 <ncopa> yup
2023-10-05 08:42:56 <ncopa> it did not boot the /boot/vmlinuz-lts kernel
2023-10-05 08:43:01 <ncopa> it booted something else
2023-10-05 08:47:49 <ikke> raspbeguy: no, we just do external monitoring via http(s)
2023-10-05 08:53:14 <ikke> ncopa: so we should send an email to IBM?
2023-10-05 08:53:40 <ncopa> i think we may need to run s390-tool's zipl
2023-10-05 08:53:56 <ncopa> but it got an error, so i updated it on alpine 3.15
2023-10-05 08:54:05 <ncopa> let me test if that solves it
2023-10-05 08:55:58 <ncopa> still gives Arithmetic exception
2023-10-05 08:57:07 <ncopa> bah, need to debug this. maybe we should contact IBM
2023-10-05 09:00:16 <raspbeguy> ikke, when I check zabbix, the hosts are monitored either through zabbix agent or SNMP, so I'm probably missing something
2023-10-05 09:01:55 <raspbeguy> https://zabbix.alpinelinux.org/zabbix.php?name=mirror&ip=&dns=&port=&status=-1&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&maintenance_status=1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=name&sortorder=ASC&show_suppressed=0&action=host.view
2023-10-05 09:02:52 <ikke> raspbeguy: there is no dedicated http interface 
2023-10-05 09:02:59 <ikke> It needs just a interface 
2023-10-05 09:03:42 <ikke> ncopa: I recall there were issues with that before 
2023-10-05 09:04:26 <raspbeguy> ok so the port can be unreachable it won't care?
2023-10-05 09:05:00 <ikke> Without any agent specific items, it won't even try 
2023-10-05 09:05:20 <ikke> ncopa: !11544
2023-10-05 09:06:33 <ikke> 3.15 still has 2.21.0
2023-10-05 09:18:02 <ikke> ncopa: should we backport that to 3.15?
2023-10-05 09:19:31 <ncopa> I backported 2.21.0-r1 to alpine 3.15
2023-10-05 09:19:35 <ncopa> issue still happens
2023-10-05 09:19:53 <ncopa> i looked at the git log and there was a use after free that was fixed in 2.24
2023-10-05 09:20:00 <ncopa> soI wonder if we should upgrade to that
2023-10-05 09:20:48 <ncopa> actually, 2.29 is out
2023-10-05 09:28:21 <ikke>  tmhoang ncopa, ikke: both s390x.a.o and s3
2023-10-05 09:28:23 <ikke> 90x-ci.a.o are running latest kernel now. Upgrading kernel won't work unti
2023-10-05 09:28:25 <ikke> l I fix the s390-tools/zipl bootloader issue. Restart is O
2023-10-05 09:28:55 <ikke> That was 1.5 years ago
2023-10-05 09:29:29 <ncopa> argh
2023-10-05 09:29:46 <ncopa> so the zipl bootloader issue is still not fixed
2023-10-05 09:33:50 <ikke> ncopa: what is the current state of the host?
2023-10-05 09:35:02 <ncopa> current state is that i upgraded the kernel to 5.15.133 and rebooted
2023-10-05 09:35:15 <ncopa> i was not aware that zipl needed to run to update the bootloader
2023-10-05 09:35:35 <ncopa> so it appears that boot loader started the old, 5.15.32 kernel (incl initramfs)
2023-10-05 09:35:38 <ncopa> and booted that
2023-10-05 09:35:49 <ncopa> it found and mounted the rootfs
2023-10-05 09:36:15 <ncopa> but it only has the kernel modules available in initramfs, which means that no other kernel modules has been loaded
2023-10-05 09:36:22 <ncopa> and overlayfs didnt load
2023-10-05 09:36:29 <ncopa> which is needed for docker to funtion
2023-10-05 09:36:33 <ncopa> function*
2023-10-05 09:37:07 <ncopa> so, its running old 5.15.32 kernel "degraded", without access to all modules
2023-10-05 09:37:14 <ikke> Right 
2023-10-05 09:37:19 <ncopa> i tried to run zipl manually but get an error
2023-10-05 09:37:38 <ncopa> Arithmetic exception
2023-10-05 09:38:13 <ncopa> i have a backtrace on it, and it seems like it happens in memcpy
2023-10-05 09:39:08 <ncopa> which is fortify-headers
2023-10-05 09:39:28 <ncopa> it could be a memcpy with overlap (which may work with memmove)
2023-10-05 09:39:52 <ncopa> or it could be something else
2023-10-05 09:42:31 <ncopa> bah, the 2.23 s390-tools does not even build with recent gcc/musl
2023-10-05 10:06:08 <ncopa> SIGFPE	erroneous arithmetic operation such as divide by zero
2023-10-05 10:06:16 <ncopa> thats the exception that zipl gets
2023-10-05 10:06:48 <ncopa> and it appears to happen on the instruction where it tries to call the __builtin_memcpy
2023-10-05 10:07:00 <ncopa> which makes no sense to me
2023-10-05 10:19:07 <ncopa> Preparing boot device: dasda (0200).
2023-10-05 10:19:07 <ncopa> Syncing disks...
2023-10-05 10:19:07 <ncopa> Done.
2023-10-05 10:19:07 <ncopa> OK: 1378 MiB in 231 packages
2023-10-05 10:19:12 <ncopa> i think i fixed it
2023-10-05 10:19:25 <ncopa> lets see if I can reboot now...
2023-10-05 10:20:29 <ncopa> gitlab-runner-s390x [~]# uptime
2023-10-05 10:20:29 <ncopa>  10:20:19 up 0 min,  0 users,  load average: 0.34, 0.08, 0.03
2023-10-05 10:20:30 <ncopa> gitlab-runner-s390x [~]# uname -a
2023-10-05 10:20:30 <ncopa> Linux gitlab-runner-s390x 5.15.133-0-lts #1-Alpine SMP Mon, 25 Sep 2023 08:33:15 +0000 s390x Linux
2023-10-05 10:20:33 <ncopa> \o/
2023-10-05 10:23:50 <ikke> ncopa: nice
2023-10-05 10:24:01 <ikke> how did  you fix it?
2023-10-05 10:26:56 <ncopa> disable _FORTIFY_SOURCE
2023-10-05 10:27:12 <ncopa> eg pass -U_FORTIFY_SOURCE to cflags
2023-10-05 10:27:36 <ncopa> I'm pushing a fix, which I will backport all the way to 3.15
2023-10-05 10:27:57 <ncopa> did we have this in any issue in bugtracker?
2023-10-05 10:28:37 <ncopa> nope
2023-10-05 10:28:48 <ikke> Yeah, couldn't find any reference
2023-10-05 10:31:39 <ncopa> do I dare to upgrade to alpine 3.18?
2023-10-05 10:32:51 <ikke> Good question, we haven't really tested booting s390x >3.15
2023-10-05 10:33:32 <ikke> ncopa: maybe we should ask if someone could be stand-by in case it does not boot?
2023-10-05 10:35:48 <ncopa> i also need to fix s390-tools for the other branches
2023-10-05 10:35:53 <ncopa> i only pushed it to 3.15 for now
2023-10-05 10:36:13 <ncopa> yes, we should probably ask someone to be stand-by in case it fails to boot
2023-10-05 10:37:01 <ikke> I'll send an e-mail
2023-10-05 10:40:39 <ncopa> thanks
2023-10-05 10:40:52 <ncopa> still strugling with s390-tools. now i get: /home/ncopa/aports/main/s390-tools/src/s390-tools-2.23.0/zipl/src/zipl-editenv.c:131:(.text.startup+0x6c4): undefined reference to `lseek64'
2023-10-05 10:41:26 <ikke> lfs64
2023-10-05 10:41:31 <ncopa> yup
2023-10-05 10:46:53 <ikke> raspbeguy: is it clear now?
2023-10-05 10:49:40 <raspbeguy> ikke: yes I think. I created a pull request and added the mirror in zabbix
2023-10-05 11:04:11 <ikke> :-)
2023-10-05 11:12:56 <ikke> ncopa: https://gitlab.alpinelinux.org/firefly-cpp/aports/-/jobs/1134285 :(
2023-10-05 11:13:55 <ikke> seems to be stil hanging
2023-10-05 11:19:52 <ikke> raspbeguy: the dns field is not matching the hostname in zabbix
2023-10-05 11:35:42 <raspbeguy> ikke, fixed. but I thought that wasn't important
2023-10-05 11:36:16 <ikke> https://zabbix.alpinelinux.org/items.php?form=update&hostid=10094&itemid=36766&context=template
2023-10-05 11:36:18 <ikke> check the url field
2023-10-05 12:39:17 <mps> no news from telmich ?
2023-10-05 12:39:40 <ikke> mps: we did hear from him
2023-10-05 12:40:01 <mps> are you heard good news
2023-10-05 12:41:07 <mps> else I have again to set my armv7 machine to test u-boot and some other pkgs
2023-10-05 20:51:24 <raspbeguy> ikke, thanks for merging PR 8, could you merge 7 too please?
2023-10-05 20:51:43 <raspbeguy> shush algitbot
2023-10-05 20:52:26 <ikke> It needs to be rebased (add-add conflict)
2023-10-05 20:55:51 <raspbeguy> ikke, done
2023-10-06 10:53:52 <ikke> clandmeter: update-acme-certs on the gitlab server was broken (syntax error)
2023-10-06 10:54:19 <ikke> if -> fi
2023-10-06 16:46:07 <clandmeter> Ah sorry
2023-10-07 11:37:52 <ikke> Hmm, s390x jobs are passing again at the moment
2023-10-08 19:21:17 <usrhere> ikke: hi, that looks either like a spam or a wrong place for it https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues/634
2023-10-08 19:21:29 <ikke> spam
2023-10-08 19:21:41 <ikke> deleted
2023-10-08 20:06:59 <usrhere> ikke: how do you shorten zabbix urls ? 
2023-10-08 20:08:06 <usrhere> I added that mirror.matnetwrk.net mirror to zabbix
2023-10-08 20:11:07 <ikke> usrhere: dup.pw is a custom redirect service operated by ncopa
2023-10-08 20:13:42 <usrhere> so http://dup.pw/a/<trigger-id>/<event-id> ?
2023-10-08 20:13:48 <ikke> yes
2023-10-08 20:14:11 <usrhere> but I have to craft it manually if I want to share something with you?
2023-10-08 20:14:19 <ikke> indeed
2023-10-08 20:14:27 <usrhere> ok, thx
2023-10-09 08:43:25 <ikke> clandmeter: ^
2023-10-09 08:43:37 <clandmeter> nice!
2023-10-09 08:44:47 <clandmeter> ikke: all is back?
2023-10-09 08:48:42 <clandmeter> or am i reading this wrong? ":)
2023-10-09 08:59:18 <ncopa> looks like it is back! \o/
2023-10-09 09:02:35 <mps> good, now it should find stable place
2023-10-09 10:15:42 <ikke> just in time for the next curl security release
2023-10-09 14:07:38 <telmich> hello
2023-10-09 14:07:45 <ikke> telmich: hi
2023-10-09 14:08:02 <telmich> I am back in the country, if there is something that needs to be checked, I can go to the DC in the next days
2023-10-09 14:08:21 <ikke> telmich: it just so happened to recover today 
2023-10-09 14:08:28 <telmich> err
2023-10-09 14:08:29 <telmich> ok
2023-10-09 14:08:38 <telmich> What was the problem/symptom?
2023-10-09 14:08:58 <ikke> vigir23 and everything behind it was unreachable 
2023-10-09 14:11:37 <telmich> Hmm, that's strange
2023-10-09 14:11:39 <telmich> I'll check the logs on it
2023-10-10 08:59:51 <ncopa> seems like s390x CI still has issues
2023-10-10 09:11:25 <ikke> Yes, but not always 
2023-10-10 09:11:32 <ikke> Some jobs pass successfully 
2023-10-10 12:19:48 <ncopa> i wonder if i should upgrade and reboot the usa2-dev1 (s390x) machine
2023-10-10 12:19:57 <ncopa> to latest alpine 3.15 kernel
2023-10-10 12:20:09 <ncopa> zipl should work now
2023-10-10 12:24:02 <ikke> I thought you already upgraded to the latest. But I suppose you just made it work with the current kernel again.
2023-10-10 12:24:39 <ncopa> i only upgraded the ci machine
2023-10-10 12:25:32 <ncopa> ok, im upgrading it
2023-10-10 12:26:19 <ncopa> Using config file '/etc/zipl.conf'
2023-10-10 12:26:19 <ncopa> Error: No such file or directory
2023-10-10 12:26:24 <ncopa> :-o
2023-10-10 12:35:59 <ncopa> ok, i think i fixed it
2023-10-10 12:36:02 <ncopa> usa2-dev1 [~]# uptime
2023-10-10 12:36:02 <ncopa>  12:35:40 up 0 min,  load average: 1.95, 0.42, 0.13
2023-10-10 12:36:02 <ncopa> usa2-dev1 [~]# uname -a
2023-10-10 12:36:03 <ncopa> Linux usa2-dev1 5.15.134-0-lts #1-Alpine SMP Fri, 06 Oct 2023 16:13:38 +0000 s390x GNU/Linux
2023-10-10 12:36:05 <ncopa> phew...
2023-10-10 12:56:46 <ikke> I have not heard back from tmhoang yet
2023-10-10 14:14:17 <ptrc> ikke: https://ptrc.gay/WIWBTARw
2023-10-10 14:16:34 <ikke> ptrc: ok, will look at that
2023-10-11 16:02:21 <ncopa> ikke: where do we want the arm 3.19 builders? on equinix or vigir23?
2023-10-11 17:57:13 <ikke> ncopa: I don't have a strong preference. che-bld-1 has more space, use-bld-1 more availability
2023-10-12 05:38:26 <celie> Is there something blocking the armv7 builder? (Compare https://pkgs.alpinelinux.org/packages?branch=edge&arch=armv7 with https://pkgs.alpinelinux.org/packages?branch=edge&arch=armhf)
2023-10-12 05:39:10 <ikke> https://build.alpinelinux.org/ it's building dotnet7-build
2023-10-12 05:42:01 <ikke> Seems it's stuck
2023-10-12 05:46:19 <celie> build-3-18-armv7 is probably also stuck on dotnet7-build
2023-10-12 07:30:49 <ncopa> ikke: i think we need ask IBM for help troubleshooting the network issues on gitlab-runner-s390x
2023-10-12 07:37:20 <ncopa> i have a tcpdump
2023-10-12 07:37:22 <ncopa> https://tpaste.us/yYly
2023-10-12 07:46:41 <ncopa> with trshark
2023-10-12 07:46:45 <ncopa> tshark
2023-10-12 07:46:47 <ncopa>     9 3.265552317 148.100.88.62 → 151.101.2.132 TCP 66 37026 → 80 [ACK] Seq=139 Ack=2102 Win=64128 Len=0 TSval=799953253 TSecr=2092793954                                                   
2023-10-12 07:46:47 <ncopa>    12 3.272625349 151.101.2.132 → 148.100.88.62 TCP 2954 [TCP Retransmission] 80 → 37026 [PSH, ACK] Seq=2102 Ack=139 Win=145920 Len=2888 TSval=2092793961 TSecr=799953257                 
2023-10-12 07:46:47 <ncopa>    10 3.268972974 151.101.2.132 → 148.100.88.62 TCP 2954 [TCP Previous segment not captured] 80 → 37026 [ACK] Seq=46866 Ack=139 Win=145920 Len=2888 TSval=2092793957 TSecr=799953253 [TCP segment of a reassembled PDU]                                                
2023-10-12 07:46:47 <ncopa>    11 3.268978449 148.100.88.62 → 151.101.2.132 TCP 78 [TCP Dup ACK 9#1] 37026 → 80 [ACK] Seq=139 Ack=2102 Win=64128 Len=0 TSval=799953257 TSecr=2092793954 SLE=46866 SRE=49754             
2023-10-12 07:47:18 <ncopa> there are there are various of those TCP retransmissions
2023-10-12 07:47:41 <ncopa> the connection has packetloss
2023-10-12 07:50:30 <ncopa>     1231996 segments received
2023-10-12 07:50:30 <ncopa>     1538834 segments send out
2023-10-12 07:50:30 <ncopa>     22261 segments retransmited
2023-10-12 07:50:30 <ncopa>     2349 bad segments received.
2023-10-12 07:51:46 <ncopa> that is 1.5% of the segments sent that got re-transmited
2023-10-12 09:23:02 <ncopa> the s390x builder always hangs on apk, right?
2023-10-12 10:05:01 <ikke> ncopa: the ci host, yes
2023-10-12 13:39:37 <ncopa> i tried to troubleshoot it a bit today. apk hangs "forever" at times
2023-10-12 13:39:41 <ncopa> not always
2023-10-12 13:39:56 <ikke> yeah, exactly, that's what I saw as well
2023-10-12 13:40:06 <ikke> I hangs at read(), where it reads from a network socket
2023-10-12 13:40:15 <ikke> which is connected to dl-cdn
2023-10-12 13:40:30 <ikke> but I didn't see any traffic
2023-10-12 13:40:45 <ikke> (like no retransmissions)
2023-10-12 13:40:47 <ncopa> https://gitlab.alpinelinux.org/alpine/apk-tools/-/commit/dcdc0901b4e849b805c5395142539cc03ecd2193
2023-10-12 13:40:59 <ncopa> i tried to reproduce it with curl, but I was not able to
2023-10-12 13:41:05 <ncopa> so I think it may be a problem in apk
2023-10-12 13:41:16 <ncopa> i looked at freebsd libfetch git log
2023-10-12 13:41:29 <ncopa> but didnt find anything of interest (other than minor cleanups)
2023-10-12 13:41:59 <ncopa> i think i maybe should ask fabled to help troubleshoot this
2023-10-12 13:41:59 <ikke> interestingly enough, it's always some update like operation
2023-10-12 13:42:09 <ikke> it's never while installing the packages itself
2023-10-12 13:42:23 <ncopa> i was able to reproduce with apk fetch linux-lts
2023-10-12 13:42:35 <ikke> yes
2023-10-12 13:42:55 <ncopa> i think it may only happen initially, once it passes initial phase, it works fine
2023-10-12 13:42:56 <ikke> In CI, it either happens at the beginning when it's upgrading the system, or at the end, where it uses apk fetch for checkapk
2023-10-12 13:43:36 <ikke> But it's strange this suddenly started
2023-10-12 13:43:42 <ikke> maybe some change in fastly?
2023-10-12 13:43:48 <ikke> or at ibm
2023-10-12 13:43:52 <ikke> (OSU)
2023-10-12 13:47:42 <ncopa> there are packetloss
2023-10-12 13:48:01 <ncopa> dont know where the loss happens
2023-10-12 13:48:11 <ikke> yeah, that's always hard to identify
2023-10-12 14:25:50 <ncopa> ok. i think I have been able to reproduce the issue with curl
2023-10-12 14:27:46 <ikke> Interesting
2023-10-12 14:28:08 <ikke> Any insight?
2023-10-12 14:34:46 <nu[m]1> might be a moonshot, but we've seen similar hanging behaviour with git which psykose tracked back to the following issue:  https://github.com/curl/curl/issues/11194
2023-10-12 14:35:15 <ikke> nu[m]1: we primarily noticed this in apk, which uses libfetch
2023-10-12 14:35:21 <ikke> so not curl specific
2023-10-12 14:45:59 <nu[m]1> til^^
2023-10-12 15:06:40 <ncopa> i was actually able to reproduce it with curl as well
2023-10-12 15:07:24 <ncopa> some details stored here: https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10947
2023-10-12 15:07:43 <ncopa> i think it may help if we would have a --retry option or similar
2023-10-12 15:08:27 <ncopa> if it just passes the initial phase it completes it
2023-10-12 15:10:27 <ikke> I wonder if tmhoang still works for IBM
2023-10-12 15:14:17 <ncopa> i havent seen him in a while
2023-10-12 15:14:37 <ikke> I sent an e-mail, but no response
2023-10-12 15:24:22 <ptrc> could someone cancel https://gitlab.alpinelinux.org/Celeste/aports/-/jobs/1141534 ? :p
2023-10-12 15:24:33 <ikke> dpme
2023-10-12 15:24:35 <ikke> done
2023-10-12 15:25:05 <ptrc> thanks :)
2023-10-12 15:39:37 <zv> how is your x86 build environment set up, is it running a 32-bit kernel, or 32-bit userspace? setarch linux32?
2023-10-12 15:39:50 <ikke> the latter
2023-10-12 15:40:25 <zv> I am at my wits end trying to build thunderbird in a 32-bit x86 environment. memory exhausted.
2023-10-12 15:40:36 <zv> and I don't want to resort to a cross compile
2023-10-12 15:40:47 <ikke> yeah, that's kind of anoying
2023-10-12 15:40:59 <ikke> did you try to lower the amount of threads?
2023-10-12 15:41:06 <ikke> or is it happening at linking stage?
2023-10-12 15:41:10 <zv> linking
2023-10-12 15:41:42 <zv> here is the whole kitchen sink: http://ix.io/4IQ2 and it still says out of memory
2023-10-12 15:44:22 <ikke> I fixed a similar issue here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15205
2023-10-12 15:44:43 <ikke> Fix was disabling debug generating debug info
2023-10-12 15:45:08 <ikke> But that's project specific
2023-10-12 16:48:26 <zv> thanks. I ended up having to disable debug symbols entirely, not just strip them, and now it builds
2023-10-12 16:49:49 <ikke> indeed
2023-10-12 16:49:53 <ikke> glad it helped
2023-10-12 16:52:54 <zv> another related question, are y'all doing anything special for 'uname' calls? we intercept it with LD_PRELOAD, since e.g. setarch linux32 on aarch64 gives armv8a, while we want armv7l
2023-10-12 16:53:19 <ikke> No, we don't do anything special
2023-10-12 16:53:25 <ikke> note that it returns armv8l for us
2023-10-12 16:53:55 <zv> ok. is that your intended target?
2023-10-12 16:54:38 <zv> typo on my end, we also get armv8l without it
2023-10-12 16:57:19 <ikke> intended target for what?
2023-10-12 16:59:43 <zv> alpine targets aarch64 and armv7, I'd imagine some software will use 
2023-10-12 16:59:55 <zv> 'uname' to enable features/other code paths
2023-10-12 17:00:34 <zv> we've seen a few cases of miscompiles due to this
2023-10-12 17:01:29 <ikke> we have also armhf
2023-10-12 17:03:16 <ikke> We did patch some packages to fix this
2023-10-13 12:10:55 <ncopa> i have fixed the build-*-a* on usa-bld-1 to use a shared /var/cache/distfiles, and set up a daily job to purge distfiles
2023-10-13 12:11:11 <ncopa> i will set up the build-3-19-* there as well
2023-10-13 12:11:36 <ncopa> we could consider move build-3-17-* to che-bld-1
2023-10-13 12:19:12 <ikke> ncopa: I can work on that 
2023-10-13 12:19:30 <ncopa> awesome. thanks!
2023-10-13 12:19:47 <ncopa> maybe also move nmeum-edge-aarch64
2023-10-13 15:06:15 <mps> does anyone here use apple silicon machines with alpine (except ncopa, I know he use it)
2023-10-13 17:27:27 <Saijin_Naib[m]> Is it easy to install and undo? I just got a Mac Mini M2 for work, if you need something tested on that 
2023-10-13 17:32:35 <mps> Saijin_Naib[m]: it is not complicated to install but undo is not so simple
2023-10-13 17:32:49 <mps> and I'm not sure how M2 works
2023-10-13 17:33:56 <Saijin_Naib[m]> Ahhh okay. Well, keep me in mind if it might be helpful for you later. 
2023-10-13 17:35:08 <mps> https://github.com/AsahiLinux/docs/wiki/Feature-Support is a list of supported things
2023-10-13 17:36:17 <mps> Saijin_Naib[m]: here is how to install alpine https://arvanta.net/alpine/install-alpine-m1/
2023-10-13 17:36:49 <Saijin_Naib[m]> Oooh, yeah, very early days for my mini... Not much working yet, it seems 
2023-10-13 17:37:40 <mps> I didn't followed M2 status much so I don't know
2023-10-13 17:37:56 <mps> M1 series works fine
2023-10-13 17:40:36 <mps> I prepared kernel with GPU driver and mesa for it here https://arvanta.net/mps/aports-local/aarch64/ for those who want to test
2023-10-13 17:43:07 <mps> Saijin_Naib[m]: I have to add that I'm using M1 nearly two years now as primary workstation and development machine
2023-10-13 17:43:36 <mps> it is not bad although some quirks exists
2023-10-13 17:45:01 <Saijin_Naib[m]> That's incredible! 
2023-10-13 17:45:13 <Saijin_Naib[m]> Alpine must fly on that 
2023-10-13 17:46:01 <mps> yes, alpine was a first distro used as daily driver on apple silicon
2023-10-13 17:47:04 <mps> actually I simply copied root FS from one of my arm64 chromebook to it and added kernel, that's all
2023-10-13 17:47:41 <mps> and I still have traces of this chromebook on it
2023-10-14 07:31:43 <mps> iiuc someone set dendrite (matrix server) somewhere on alpine infra?
2023-10-14 07:32:16 <ikke> yes, but it's broken
2023-10-14 07:32:54 <ikke> some corruption we cannot fix
2023-10-14 07:34:06 <mps> so not yet ready for stable use cases
2023-10-14 07:34:27 <ikke> We asked someone from the community (nu) to host it for us, he is working on setting it up
2023-10-14 07:35:52 <mps> I set it locally some time ago but didn't put in real usage. wanted to ask is it stable to replace synapse with it
2023-10-14 07:36:20 <mps> synapse is irritating for me to keep running
2023-10-14 08:59:21 <ikke> mps: dendrite does not seem to be an improvement
2023-10-14 09:12:13 <mps> ikke: understand. thanks for info
2023-10-14 10:56:01 <clandmeter> mps: dendrite is more efficient in regards of resources
2023-10-14 10:57:02 <clandmeter> but the default is still synapse, so i would assume also more stable and better support
2023-10-14 10:57:46 <clandmeter> numono has it running for almost 2 years i believe with issues. maybe we are just unlucky
2023-10-14 10:57:58 <clandmeter> without*
2023-10-14 10:59:07 <mps> clandmeter: yes, python is not efficient and 'eat' resources
2023-10-14 11:00:05 <mps> I can't upgrade synapse because it can't read database from previous version
2023-10-14 11:00:40 <mps> had to backport old one for armv7 3.18
2023-10-14 11:00:41 <ikke> No migration?
2023-10-14 11:00:59 <mps> ikke: I didn't found anything
2023-10-14 11:01:41 <mps> expected it will automatically at upgrade but it destroyed db, happily had backup
2023-10-14 11:02:18 <mps> s/happily/luckilly/
2023-10-14 11:03:01 <mps> and ofc didn't found anything to migrate db from synapse to dendrite
2023-10-14 11:04:20 <ikke> afaik, that's not possible
2023-10-14 19:17:07 <mps> algitbot: retry master
2023-10-14 19:17:45 <ikke> will not help a lot
2023-10-14 19:18:28 <mps> well, at least some pkgs will be built
2023-10-14 19:18:43 <ikke> It's rebuilding the same packages over and over atm
2023-10-14 19:18:48 <mps> I know it is not 'fix'
2023-10-15 12:20:37 <mps> algitbot: retry master
2023-10-15 12:30:09 <mps> I think builders should be set to not stop on failed pkgs
2023-10-15 12:30:55 <mps> maintainers/contributors should care for failed pkgs
2023-10-15 12:44:09 <ikke> No, we all should care
2023-10-15 12:47:31 <mps> hm, I can fix some things but I don't want to risk to be accused by some maintainers
2023-10-15 12:47:48 <ikke> Then you make an MR and assign it to the maintainer
2023-10-15 12:48:02 <mps> no, thanks
2023-10-15 12:48:34 <mps> I passed this 'path' and it was unpleasant
2023-10-15 12:48:49 <mps> I guess you remember
2023-10-15 12:49:39 <ikke> I can't remember an instance where making an MR caused you issues
2023-10-15 12:50:01 <mps> hm
2023-10-15 12:51:01 <ikke> And generally everyone is okay if you help with fixing issues
2023-10-15 12:51:35 <mps> generally yes, but not always
2023-10-17 15:02:29 <ncopa> ikke: I'd like to add a build with CC=clang to https://gitlab.alpinelinux.org/alpine/abuild/-/blob/master/.gitlab-ci.yml?ref_type=heads
2023-10-17 15:02:36 <ncopa> any suggestions how to do that?
2023-10-17 15:04:43 <ncopa> I would also like to run a scan-build: https://gitlab.alpinelinux.org/alpine/abuild/-/issues/10102
2023-10-17 15:06:41 <ikke> ncopa: so like a separate job?
2023-10-17 15:06:52 <ncopa> yes, run in parallel if possible
2023-10-17 15:07:31 <ncopa> apk add clang cmd:scan-build; make CC=clang
2023-10-17 15:07:46 <ncopa> scan-build make
2023-10-17 15:08:04 <ncopa> strying to figure out how to make scan-build exit with error on warnings
2023-10-17 15:09:22 <ncopa> scan-build --status-bugs make
2023-10-17 15:13:12 <ikke> You'd add the necessary dependencies here: https://gitlab.alpinelinux.org/alpine/infra/docker/abuild-ci/-/blob/master/Dockerfile
2023-10-17 15:13:57 <ikke> And here you'd add the logic to run scan-build: https://gitlab.alpinelinux.org/alpine/infra/docker/abuild-ci/-/blob/master/overlay/entrypoint
2023-10-17 15:14:53 <ikke> Either by adding a new case (and provide `entrypoint <command>` as script), or by looking at an environment variable (which then gets set in the .gitlab-ci.yml)
2023-10-17 15:25:46 <ncopa> what does this do? script: [pwd]
2023-10-17 15:26:17 <ikke> gitlab requires a script section, but this image has an entrypoint
2023-10-17 15:26:44 <ikke> basically, the entrypoint gets called with sh -c 'pwd'
2023-10-17 15:27:11 <ikke> so the test|sh) case gets triggered
2023-10-17 15:27:22 <ncopa> so its only to make the entrypoint happy
2023-10-17 15:27:36 <ncopa> so its only to make gitlab happy
2023-10-17 15:27:58 <ikke> yes
2023-10-17 15:28:19 <ikke> There is a long standing feature request in gitlab to remove the required script section
2023-10-17 15:28:30 <ikke> remove the requirement of having one
2023-10-17 15:28:58 <ncopa> ok
2023-10-17 15:29:12 <ncopa> and abuild-ci is a docker image, only for abuild project?
2023-10-17 15:29:16 <ikke> yes
2023-10-17 15:29:46 <ikke> My personal preference is to have pre-built images rather than installing everything in every CI job
2023-10-17 15:30:12 <ikke> Though it makes changes a bit more involved
2023-10-17 15:34:02 <ncopa> yeah. makes sense to have the deps cached
2023-10-17 15:59:25 <ncopa> ikke: what do you think about this? https://gitlab.alpinelinux.org/alpine/infra/docker/abuild-ci/-/merge_requests/4
2023-10-17 16:00:49 <ncopa> then we do this in abuild: https://tpaste.us/ggzx
2023-10-17 16:04:46 <ikke> Fine with me, though for a multiline script you probably want to use the block syntax instead of scalar
2023-10-17 16:04:52 <ikke> - make
2023-10-17 16:04:54 <ikke> - make test
2023-10-17 16:07:48 <ncopa> indeed
2023-10-17 16:07:49 <ncopa> thanks!
2023-10-17 16:17:43 <ncopa> yay! its working!
2023-10-17 16:18:54 <ikke> cool
2023-10-17 16:33:49 <ncopa> thanks alot!
2023-10-18 07:34:45 <clandmeter> \o/
2023-10-18 08:29:14 <raspbeguy> |-o-| <- T-fighter
2023-10-19 13:35:39 <ikke> ncopa: seems like the network in s390x improved, at least CI jobs are passing again
2023-10-19 13:44:34 <ncopa> yes, i noticed
2023-10-19 13:44:43 <ncopa> and closed the apk-tools issue i created
2023-10-20 08:19:47 <ncopa> we have a bunch of olde builders running, eg build-3-4-x86_64. would it be an idea to power some of those off?
2023-10-20 08:20:10 <ncopa> im about to start up build-3-19-*
2023-10-20 08:21:11 <ncopa> Load average: 45.62 44.35 51.96 56/787 43650
2023-10-20 08:21:29 <ncopa> the x86* builders are sweating
2023-10-20 09:11:47 <clandmeter> hope they dont short circuit 
2023-10-20 10:07:41 <ncopa> ikke: can you help me purge those packages from cdn cache in a couple of hours? https://tpaste.us/LREl
2023-10-20 10:07:52 <ncopa> once those pakcages are rebuilt with updated gcc
2023-10-20 10:08:05 <ncopa> will probably take an hour or two
2023-10-20 10:23:28 <ikke> Ok
2023-10-20 10:27:38 <ikke> ncopa: repo-tools fastly purge pkg --origin linux-lts xtables-addons zfs-lts 
2023-10-20 10:27:56 <ikke> (you can add -n / --dry-run to see what it would do)
2023-10-20 10:29:25 <ikke> Or repo-tools fastly purge pkg linux-lts linux-lts-dev linux-lts-doc xtables-addons zfs-lts zfs-lts-dev to just purge thost listed packages
2023-10-20 10:46:22 <ncopa> ok. thanks!
2023-10-20 12:44:32 <pj> ikke: could I ask for armhf/armv7 container, I would like to try debug rust 1.73 failure
2023-10-22 16:05:39 <numono> heyho, matrix server is almost ready, the pr for the dns entries is pending^^ 
2023-10-22 16:05:45 <numono> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/74
2023-10-24 07:08:08 <ikke> ncopa: did you see the mail from tmhoang?
2023-10-24 08:37:39 <ncopa> yes i did
2023-10-24 08:43:24 <ikke> Do you see any issue in his proposal?
2023-10-24 08:46:38 <ncopa> not really. i responded now
2023-10-24 09:00:04 <mps> strange error 'ERROR: alsa-lib-dev-1.2.10-r0: package mentioned in index not found (try 'apk update')' when using abuild -r for one package
2023-10-24 09:00:18 <mps> apk add alsa-lib-dev works fine
2023-10-24 14:36:35 <nmeum> ikke: if you have the time, could you give me access to an x86 lxc container? would like to debug the community/go build failure on x86 for which I would like to build older gcc version to figure out when gcc-go based bootstrap stopped working on x86
2023-10-25 11:58:39 <mps> any objection on #15351
2023-10-25 12:16:14 <ikke> Yes, there is a plan to start removing packages from testing after some time 
2023-10-25 12:26:10 <mps> heh, then I have to move all 'my' pkgs to community, and not only 'my' but some which I use regularly
2023-10-25 12:27:15 <mps> and have no idea who come to such 'smart' plan
2023-10-25 12:28:17 <mps> but I don't disagree with plan, maybe it quite good
2023-10-25 12:28:25 <mps> it is*
2023-10-25 12:57:55 <mps> ikke: btw, do you know when these removals are planned to begin
2023-10-25 19:12:17 <ikke> numono: Do you need that dns record to be able to continue?
2023-10-25 19:12:38 <ikke> mps: clandmeter would follow up
2023-10-25 19:12:54 <ikke> and to be honest, I have some packages which I should move out of testing as well, mostly forgot about them
2023-10-25 19:15:07 <mps> ikke: ok, thanks for info
2023-10-25 19:15:22 <ikke> nmeum: root@nmeum-edge-x86.nld-dev-1.alpin.pw
2023-10-25 19:23:15 <nmeum> yep, works. thanks
2023-10-25 19:23:35 <ikke> nice
2023-10-25 19:23:52 <ikke> hope you can find that go issue
2023-10-25 19:24:19 <nmeum> will probably have to bisect gcc a bit
2023-10-25 19:24:33 <nmeum> if you want to workaround it for now you could install the go package from edge
2023-10-25 19:25:08 <ikke> right. I think it can wait a bit, but if you cannot find it, we can always do that
2023-10-25 19:25:49 <nmeum> well see it's just a matter of time I think
2023-10-25 19:25:56 <nmeum> *we will
2023-10-25 19:26:47 <ikke> Just a question whether that's before or after the release :P
2023-10-25 19:26:50 <nmeum> yea
2023-10-26 10:40:30 <ikke> clandmeter: seems like meet.a.o works again. I've restarted it and enabled init for monodb (it was leaking a lot of bash processes)
2023-10-26 10:48:20 <ncopa> re alternative to the google sheet for meeting time poll: https://app.rallly.co/invite/2uwd5gEs1ds0
2023-10-26 10:49:03 <ncopa> we can self-host rallly if we'd like: https://github.com/lukevella/rallly
2023-10-26 10:51:15 <ncopa> there is also framadate: https://framadate.org/abc/en/
2023-10-26 10:52:10 <ikke> ncopa: I can look at rallly
2023-10-26 10:53:20 <ncopa> maybe we can host it under meet.a.o/r/ or similar?
2023-10-26 10:54:48 <ncopa> they also provide dockerfile and docker-compose
2023-10-26 11:27:25 <ikke> ncopa: hosting it on meet.a.o is a bit challenging since bbb is directly listening on 443
2023-10-26 11:27:52 <ncopa> ok
2023-10-26 14:19:28 <nmeum> ikke: with gcc 13.1.1_git20230722 gcc-go also doesn't work on x86 so either the issue is caused by an even older gcc change or it is not related to gcc
2023-10-26 14:23:05 <ncopa> which gcc did we use for alpine 3.18? I suppose it was gcc 12?
2023-10-26 14:44:48 <nmeum> https://build.alpinelinux.org/buildlogs/build-3-19-x86/community/go/go-1.21.3-r1.log
2023-10-26 14:45:04 <nmeum> oops wrong buidllog
2023-10-26 14:45:22 <nmeum> https://build.alpinelinux.org/buildlogs/build-3-18-x86/community/go/go-1.20.3-r0.log
2023-10-26 14:45:25 <nmeum> > (5/7) Installing gcc-go (12.2.1_git20220924-r10)
2023-10-26 14:45:58 <nmeum> but iirc I tested gcc-go bootstrap with gcc 13 as well, it worked at some point
2023-10-26 14:51:09 <nmeum> it worked 4 months ago when I last tested it as part of the Go 1.21 upgrade
2023-10-26 14:55:25 <nmeum> ah, might be related to some of those lfs64 changes
2023-10-26 15:42:48 <ncopa> that makes sense
2023-10-27 15:54:22 <mcrute> I'm seeing memory exhaustion errors only on the x86 builder for a package that used to build. Is there something up with that builder? https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1159735
2023-10-27 15:57:29 <ikke> mcrute: more often than not it's address space exhaustion
2023-10-27 15:57:43 <ikke> meaning it tries to allocate >4G in a single process
2023-10-27 15:58:38 <ikke> given that it's the link tool that fails, that's quite plausible
2023-10-27 15:58:53 <mcrute> I remember that being an issue with a build at some time because it could see that the system had 60+GB of RAM but as soon as it tried to allocate some it would fail
2023-10-27 15:59:35 <mcrute> Is there a way we could limit it down to only 4GB of RAM and see if it succeeds?
2023-10-27 15:59:48 <ikke> mcrute: https://tpaste.us/xyOW
2023-10-27 16:00:27 <ikke> try to disable generating debug symbols
2023-10-27 16:00:53 <ikke> That's usually takes a lot of memory while linking
2023-10-27 16:01:03 <mcrute> okay, I'll give that a try
2023-10-27 16:01:08 <ikke> (with c-based projects anyway)
2023-10-27 16:04:38 <mcrute> this one is go-based but I'll disable debug symbol generation and see if that has any impact, can't hurt to test it I guess
2023-10-27 16:07:50 <mcrute> that did indeed work, thanks @ikke
2023-10-27 16:15:14 <ikke> mcrute: cool
2023-10-27 16:25:40 <ikke> mcrute: you did not update the license field of the vault package
2023-10-27 16:26:05 <ikke> ihttps://github.com/hashicorp/vault/blob/v1.15.1/LICENSE
2023-10-27 16:27:44 <mcrute> fixed
2023-10-27 16:28:30 <mcrute> was there any consensus on what we're doing with BUSL stuff? seemed like not on the issue but maybe I missed an IRC conversation
2023-10-27 16:29:06 <ikke> No concensus yet
2023-10-27 16:30:10 <mcrute> okay
2023-10-27 18:00:37 <clandmeter> ikke: do we have a written definition of the repos we provide?
2023-10-27 18:00:56 <ikke> docker repos?
2023-10-27 18:01:01 <clandmeter> apk repos
2023-10-27 18:01:22 <ikke> Not really sure what you are asking
2023-10-27 18:01:39 <clandmeter> what the diffeence between main community and testing is
2023-10-27 18:02:04 <ikke> There is something written here: https://docs.alpinelinux.org/user-handbook/0.1a/Working/apk.html#_repositories_releases_and_mirrors
2023-10-27 18:02:10 <ikke> but nothing policy wise I guess
2023-10-27 18:02:40 <ikke> also here: https://wiki.alpinelinux.org/wiki/Repositories
2023-10-27 18:04:04 <clandmeter> ok
2023-10-27 18:04:29 <clandmeter> there is nothing about support writen about them
2023-10-27 18:05:06 <clandmeter> the wiki desc is not that bad
2023-10-27 18:06:06 <ikke> clandmeter: is this about removing packages? or more about expected support for packages in testing?
2023-10-27 18:06:44 <clandmeter> my question is triggered by the removing part yes
2023-10-27 18:07:12 <clandmeter> but i think in general we should provide a policy 
2023-10-27 18:07:30 <ikke> Yeah, would be good, but it's hard to argue against removing packages that are not being updated
2023-10-27 18:07:57 <clandmeter> its nice to have a policy for all repos
2023-10-27 18:08:13 <clandmeter> and i think it would make sense to have it inside of aports readme.txt
2023-10-27 18:08:18 <clandmeter> .md
2023-10-27 18:08:19 <ikke> although, some packages are not receiving upstream updates anymore while still being useful
2023-10-27 18:09:33 <clandmeter> what do you think if we would add it to aports/README.md?
2023-10-27 18:10:05 <ikke> Yeah, fine with me
2023-10-27 18:10:18 <clandmeter> ill make an MR and see what other think 
2023-10-27 18:10:28 <clandmeter> so we can fine tune it 
2023-10-27 18:12:30 <ikke> clandmeter: numono is almost finished with his matrix setup he created for us. Any objection in pointing our matrix.a.o records to his servers?
2023-10-27 18:12:44 <clandmeter> nope
2023-10-27 18:16:30 <Saijin_Naib[m]> <ikke> "although, some packages are..." <- There are lots of mature packages that are still useful... If upstream is stale but maintainer keeps it working, can those stay? 
2023-10-27 18:16:52 <ikke> Saijin_Naib[m]: I'd personally would say so
2023-10-27 18:17:12 <ikke> But it's a bit of a per-case situation
2023-10-27 18:21:38 <mps> clandmeter: why not POLICY.md in aports
2023-10-27 18:22:46 <clandmeter> not sure it makes sense to introduce a new file if it can go into readme? 
2023-10-27 18:23:27 <mps> ok
2023-10-27 18:26:07 <mps> Saijin_Naib[m]: also I think useful but not actively maintained pkgs should stay, except if they are too buggy or have security risk
2023-10-27 18:28:27 <ikke> telmich: vigir23 is unreachable again. Did you find out what was the cause last time?
2023-10-27 19:25:00 <telmich> <ikke> "nico🇨🇭: vigir23 is unreachable..." <- No, unfortunately not
2023-10-27 19:26:05 <telmich> ikke: btw, is the dm-vpn also affected before? because vigir23 does its own vpn to our vpn gateway, which should be independent of the dmvpn routing
2023-10-27 19:31:42 <ikke> telmich: the previous time it was, I could not connect to the servers via the vpn
2023-10-27 20:08:04 <clandmeter> ikke: i think we could run https://gitlab.alpinelinux.org/clandmeter/aports-ttl-notify
2023-10-27 20:08:24 <clandmeter> but im not sure about the actual deletion, and if we should automate it.
2023-10-27 20:10:42 <ikke> Not sure if it makes sense, but we could let it make a merge request (though at first we should probably throttle)
2023-10-27 20:11:26 <clandmeter> throttle?
2023-10-27 20:12:01 <clandmeter> MR would be good yes
2023-10-27 20:12:07 <ikke> If it would be an MR per package, we don't want to create tens of MRs at once
2023-10-27 20:12:27 <clandmeter> oh, i think one MR would be ok?
2023-10-27 20:12:53 <clandmeter> but as which user would we do this?
2023-10-27 20:12:59 <ikke> aports-qa-bot?
2023-10-27 20:13:09 <clandmeter> ok
2023-10-27 20:13:16 <clandmeter> and it has no privs?
2023-10-27 20:13:32 <clandmeter> i mean no harmful ones
2023-10-27 20:13:39 <ikke> It has privileges on aports
2023-10-27 20:14:11 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/settings/access_tokens
2023-10-27 20:15:31 <clandmeter> how would i create an MR from such kind of users?
2023-10-27 20:15:49 <clandmeter> some kind of golang client?
2023-10-27 20:16:47 <clandmeter> huh
2023-10-27 20:16:55 <clandmeter> git can create MR's?
2023-10-27 20:17:01 <ikke> You can with push options
2023-10-27 20:17:09 <ikke> https://docs.gitlab.com/ee/user/project/push_options.html
2023-10-27 20:17:14 <clandmeter> git push -o merge_request.create
2023-10-27 20:17:17 <ikke> yes
2023-10-27 20:17:24 <clandmeter> funky
2023-10-27 20:17:34 <ikke> And https remote with the access token
2023-10-27 20:18:15 <clandmeter> so I need a user with a repo and an access token to access that repo
2023-10-27 20:19:22 <clandmeter> the remote has to be https?
2023-10-27 20:20:15 <ikke> If you want to use a token, then yes
2023-10-27 20:22:49 <ikke> Disadvantage of one MR is that it's harder to exclude one or more packages for one reason or another
2023-10-27 20:23:23 <clandmeter> right, but its not that hard to get it back
2023-10-27 20:25:20 <ikke> regarding our own registry, I did have some issues pushing a specific image to the registry: https://gitlab.alpinelinux.org/alpine/infra/docker/golang/-/jobs/1152189
2023-10-27 20:25:51 <ikke> Not sure why, but it appears that linode object storage returns a 403 for some reason
2023-10-27 20:26:06 <ikke> But not for all images
2023-10-27 20:26:29 <clandmeter> anyway to get logs?
2023-10-27 20:27:08 <ikke> There are logs on the registry, but not much details
2023-10-27 20:28:13 <clandmeter> what is the difference of using algitbot or aports-qa-bit?
2023-10-27 20:28:44 <ikke> algitbot is admin
2023-10-27 20:30:29 <clandmeter> im confused about the tokens, i guess i can also setup personal tokens that are locked to specific repos
2023-10-27 20:31:00 <clandmeter> ah that was a project token you link to
2023-10-27 20:32:13 <clandmeter> i need to get used to this new interface of gitlab
2023-10-27 20:32:17 <clandmeter> im lost most of the time
2023-10-27 20:32:48 <clandmeter> like why is admin under "your work"
2023-10-27 20:33:15 <clandmeter> ah ok, got it.
2023-10-27 20:35:00 <clandmeter> interesting, you cannot impersonate a user who cannot login
2023-10-27 20:56:08 <clandmeter> ikke: whats up with the expiration date of a project token? 
2023-10-27 20:57:21 <clandmeter> the one you created for aports-qa-bot expires in 25 years
2023-10-27 20:57:37 <clandmeter> if i create one it can be max 1 year
2023-10-27 21:02:54 <ikke> yeah, changed in gitlab 16, not a lot to do about it
2023-10-27 21:03:08 <ikke> (except for manipulating it in the DB directly)
2023-10-27 21:03:38 <clandmeter> https://docs.gitlab.com/ee/administration/settings/account_and_limit_settings.html#limit-the-lifetime-of-access-tokens
2023-10-27 21:03:45 <clandmeter> its a config option
2023-10-27 21:03:53 <clandmeter> which is not available
2023-10-27 21:04:05 <ikke> requires premium
2023-10-27 21:04:12 <ikke> but even then I think you can only lower it
2023-10-27 21:04:24 <clandmeter> it says self managed?
2023-10-27 21:04:31 <clandmeter> not sure how those things work
2023-10-27 21:04:38 <clandmeter> isnt that what we have? 
2023-10-27 21:05:13 <ikke> premium + self-managed
2023-10-27 21:06:09 <ikke> So only for self-managed, but requires premium
2023-10-27 21:06:20 <clandmeter> where do you see premium?
2023-10-27 21:06:42 <ikke> "Limit the lifetime of access tokens"
2023-10-27 21:06:49 <clandmeter> you mean ultimate?
2023-10-27 21:06:55 <ikke> sorry, yes
2023-10-27 21:07:46 <clandmeter> ok, bit stupid though
2023-10-27 21:07:59 <clandmeter> so you manipulated it from db or it was free before
2023-10-27 21:08:14 <ikke> Before there was no limit
2023-10-27 21:08:19 <ikke> I guess this was grandfathered in
2023-10-27 21:08:59 <clandmeter> to further improve our product we have removed some features
2023-10-27 21:09:16 <ikke> well, these are 2 separate features
2023-10-27 21:09:40 <ikke> For 'security' they enforce time limits for tokens
2023-10-27 21:09:59 <ikke> Only ultimate users can enforce a specific maximum lifetime of tokens
2023-10-27 21:09:59 <clandmeter> yes, very handy for automation
2023-10-27 21:10:29 <clandmeter> ok ill look into it later, need to get some sleep. thx for the help.
2023-10-27 21:10:35 <ikke> nite
2023-10-30 11:23:06 <mps> ikke: anyone tried to contact rubicon@mailo.com maintainer of perl-crypt-argon2
2023-10-30 11:23:39 <mps> regarding !53965
2023-10-30 16:18:30 <tomalok> ncopa ikke : https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10798#note_349452 -- can we get something in place to be able to start offering cloud image downloads for 3.19?
2023-10-30 16:21:10 <ikke> tomalok: I've trying to come up with a good way to distribute these images without having to put them on the builders
2023-10-30 16:21:47 <tomalok> what about an interim step where we just do what we do now, and then push it into CDN somehow?
2023-10-30 16:21:58 <tomalok> we don't have to go full CI right away
2023-10-30 18:23:38 <ikke> tomalok: maybe we can set-up a meeting together with clandmeter to work out how we will do it?
2023-10-30 18:27:07 <tomalok> sure -- next couple days i'm on vacation -- but maybe friday or the weekend?
2023-10-30 18:27:32 <ikke> usually that does not work out for clandmeter 
2023-10-30 18:28:44 <tomalok> might be tricky, family stuff 'til Friday (for the most part) and hten back at DayJob next monday, and that'd about do it for me
2023-10-30 18:31:10 <ikke> Doing it manually is certainly an option, just need to do it in a way that will not be overwritten by the builders
2023-10-30 18:34:37 <tomalok> so interleaving it amongst the existing CDN tree would be problmatic...
2023-10-30 18:39:56 <ikke> Appears like the release dir is not cleaned
2023-10-30 18:40:18 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/aports-build/aports-build#L85-90
2023-10-30 18:43:17 <ikke> tomalok: so that means we should just be able to put things in the release dir on dl-master, and it will be available on the cdn
2023-10-30 18:44:05 <tomalok> CDN ever get cleaned up via some other means?
2023-10-30 18:45:40 <ikke> Only manually
2023-10-30 18:46:09 <ikke> package dirs are synced with the builders, so old packages are automatically removed
2023-10-30 18:48:52 <tomalok> so maybe a script on dev that would rsync ~tomalok/public_html/alpine-cloud-images/ to some target on dl-master -- or the opposite direction -- a pull?
2023-10-30 18:49:44 <ikke> yes, that's what I was thinking of
2023-10-30 18:50:23 <tomalok> a pull might be more controllable -- the stuff on dev is already readable, so i (or someone else) wouldn't have to have any extra authnz
2023-10-30 18:50:37 <ikke> ack
2023-10-30 18:51:13 <ikke> clandmeter: any ideas / opinions? ^
2023-10-30 18:51:38 <tomalok> and maybe it pulls initially to a temporary location, optionally signs new images before putting them into place for CDN
2023-10-30 18:53:03 <ikke> Not sure if we want that much logic on dl-master
2023-10-30 18:53:13 <ikke> right now it's mostly just a 'dumb' storage box 
2023-10-30 18:54:03 <ikke> and gpg signing is done by ncopa
2023-10-30 18:54:47 <tomalok> manually, or via script?
2023-10-30 18:55:16 <ikke> script
2023-10-30 18:56:39 <ikke> But only ncopa has the keys
2023-10-30 18:56:56 <ikke> so that's a challenge to automate
2023-10-30 20:39:28 <tomalok> minimal: checking cloud-init image -- it uses dhclient, not dhcpcd
2023-10-30 20:39:40 <tomalok> (wrong channel, nm)
2023-10-31 07:13:18 <clandmeter> good morning
2023-10-31 07:14:13 <clandmeter> tomalok: friday what time?
2023-10-31 08:28:12 <clandmeter> ikke: i guess we could use exclude on the builders
2023-10-31 12:37:14 <mps> ncopa: linux-asahi p18 is buggy
2023-10-31 12:37:43 <mps> I intentionally didn't upgraded it for some time
2023-10-31 12:37:47 <ncopa> oh, ok
2023-10-31 12:37:59 <ncopa> whats wrong with it?
2023-10-31 12:38:09 <mps> be careful with your speakers
2023-10-31 12:38:50 <mps> or on some models sounds doesn't works at all
2023-10-31 12:38:57 <mps> rest is ok
2023-10-31 12:39:43 <mps> looks like it is time to create speakersafetydaemon pkg for alpine
2023-10-31 12:40:34 <mps> https://github.com/AsahiLinux/speakersafetyd
2023-10-31 13:51:10 <mps> ncopa: just tested linux-asahi build, it is ok because it doesn't enable speakers by default
2023-10-31 13:53:01 <mps> did you wanted to enable asahi GPU with latest upgrade?
2023-10-31 13:53:49 <mps> i.e. DRM_ASAHI
2023-10-31 13:54:46 <ncopa> i thought i did?
2023-10-31 13:56:35 <mps> no
2023-10-31 13:56:49 <mps> I will post you diff later
2023-10-31 13:58:27 <mps> https://tpaste.us/ogBg
2023-10-31 13:58:39 <mps> these changes are needed
2023-10-31 14:00:55 <mps> I'm building it now again and will test when it finish
2023-10-31 14:01:24 <mps> aaaand after lunch
2023-10-31 15:20:56 <mps> ncopa: sorry, config-asahi.aarch changes are not needed
2023-10-31 15:21:13 <mps> you did them right
2023-10-31 15:21:54 <mps> why GPU doesn't work correctly then, have to look
2023-10-31 15:40:01 <tomalok> clandmeter: friday 16:00 utc? (i should be semi-conscious by then)
2023-10-31 15:42:55 <clandmeter> tomalok: i think that might work
2023-10-31 15:43:02 <clandmeter> ikke: what about you?
2023-10-31 15:44:16 <clandmeter> I will be on the road but thats fine
2023-10-31 15:45:43 <ikke> Should work (give or take a few minutes)
2023-10-31 16:11:52 <tomalok> what's the usual meeting substrate used?
2023-10-31 16:12:11 <ikke> meet.a.o, but sometimes also google meet or something like that
2023-10-31 16:12:47 <tomalok> does that have SSO from some other credential store, or do i need to sign up?
2023-10-31 16:12:59 <ikke> meet.a.o does not require signup
2023-10-31 18:02:20 <clandmeter> I would prefer to use google meet as I will be on mobile
2023-10-31 18:02:30 <clandmeter> or any other suitable solution
2023-10-31 18:02:38 <clandmeter> meet.a.o is kind of ...
2023-10-31 18:02:58 <clandmeter> i can send an invite, you should not need to register
2023-10-31 18:10:52 <mps> is the edge aarch64 builder stuck?
2023-10-31 18:11:36 <mps> algitbot: retry master
2023-10-31 19:36:43 <ikke> it's doing a lot, but not sure if it's making progress
2023-10-31 19:38:11 <mps> later I noticed it failed on qt6-qtwebengine
2023-10-31 19:38:22 <ikke> It's building it now
2023-10-31 19:38:49 <ikke> I think it's linking right now
2023-10-31 19:40:19 <mps> probably will fail again but let's hope I'm wrong