2022-10-01 09:36:44 <ikke> psykose: Do you want to help with upgrading the wiki container?
2022-10-01 09:37:06 <ikke> I've copied the container and DB to the new host
2022-10-01 09:37:16 <ikke> We can work on that and when it's finished, we can switch oer
2022-10-01 09:37:29 <psykose> sure, whadya need me to do
2022-10-01 09:39:34 <psykose> seems mediawiki support php8+ now
2022-10-01 09:39:51 <psykose> or well, almost https://phabricator.wikimedia.org/T248925#8249837
2022-10-01 09:41:01 <ikke> I've added your key to wiki.gbr-app-1.alpin.pw
2022-10-01 09:41:59 <psykose> in
2022-10-01 09:42:08 <ikke> 👍
2022-10-01 09:43:38 <ikke> The wiki is located at /var/www/wiki.alpinelinux.org
2022-10-01 09:46:39 <psykose> yep, see that and the nginx configs
2022-10-01 10:23:40 <ikke> before we switch-over, I can sync the DB one last time
2022-10-01 10:23:51 <ikke> not sure how that matters for any db migration though
2022-10-01 10:42:13 <psykose> shrug
2022-10-01 10:42:17 <psykose> we'll see when we get there
2022-10-01 10:42:21 <ikke> nod
2022-10-02 06:33:57 <ikke> Load average: 10025.33 10009.43 9966.77
2022-10-02 06:34:02 <ikke> 🤔
2022-10-02 06:40:04 <psykose> on what host
2022-10-02 06:40:10 <psykose> riscv?
2022-10-02 06:40:18 <psykose> been stuck for forever so makes sense
2022-10-02 11:30:04 <ikke> psykose: rv64 ci host (builder moved to a different server)
2022-10-02 21:38:02 <mps> kernel 6.0 is just released
2022-10-02 21:46:50 <ikke> What will be the next lts kernel?
2022-10-02 21:47:16 <mps> 6.1 I guess
2022-10-02 21:48:17 <mps> 3 months are enough for it to be released before new year
2022-10-02 21:52:27 <ikke> but not for 3.17
2022-10-02 21:53:32 <mps>  /o\
2022-10-02 21:54:37 <mps> 2 weeks of merge window plus 7 or 8 weeks of rc release, so 10-11 weeks from now
2022-10-02 21:55:32 <mps> so, depends on ncopa ;)
2022-10-02 21:57:06 <ikke> officially planned date is 1 november
2022-10-02 21:57:57 <mps> ah, so early
2022-10-02 21:58:33 <mps> I thought it will 1 dec
2022-10-02 21:58:43 <mps> will be*
2022-10-03 19:01:31 <mps> hmm, looks like cogitri left alpine
2022-10-03 19:02:19 <Cogitri> Jop
2022-10-03 19:02:55 <mps> Cogitri: reason?
2022-10-03 19:03:17 <mps> answer in private is also ok
2022-10-03 19:07:40 <mps> anyway, I wish you all the best in future life
2022-10-03 19:34:22 <Cogitri> mps: Thanks :)
2022-10-03 19:34:47 <Cogitri> My main reason is that I currently don’t have enough time to work on Alpine next to my masters, work and other stuff
2022-10-04 21:08:33 <mps> huh https://lore.kernel.org/lkml/1664893073238194@kroah.com/T/
2022-10-04 21:08:59 <mps> people with intel laptops shouldn't use linux 5.19.12
2022-10-04 21:10:24 <mps> also https://lore.kernel.org/all/YzwooNdMECzuI5+h@intel.com/
2022-10-04 21:10:47 <psykose> should update edge in 3.16
2022-10-04 21:11:20 <mps> 'Greg, I recommend immediate revert of this stuff, and new stable release ASAP. Plus a recommendation that no one using laptops with Intel GPUs run 5.19.12'
2022-10-04 21:11:50 <mps> psykose: yes
2022-10-04 21:14:13 <mps> pushed 6.0 to 3.15
2022-10-04 21:14:22 <mps> 3.16*
2022-10-04 21:14:57 <mps> hope it doesn't have this problem
2022-10-05 08:53:43 <clandmeter> ikke: did we add the new server to dns?
2022-10-05 09:21:09 <psykose> ikke: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/issues/28 :)
2022-10-05 10:04:50 <ikke> psykose: p00f
2022-10-05 10:05:04 <psykose> free pdf hosting courtesy of the alpine infra team :) thank ya
2022-10-05 10:21:43 <ikke> clandmeter: I think we haven't yet
2022-10-05 10:21:53 <ikke> I can add it
2022-10-05 10:33:23 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/39
2022-10-05 10:35:14 <ikke> psykose: I think regarding the wiki we can stay on 1.38 / alpine 3.15 for now
2022-10-05 10:35:22 <psykose> sounds fine to me
2022-10-05 10:35:38 <ikke> For the cut-over, we do one final sync / migration?
2022-10-05 10:35:57 <ikke> I can put a maintenance page for the wiki for a bit
2022-10-05 10:36:07 <ikke> so that we can just bring it down until it's done
2022-10-05 10:37:23 <psykose> think it's better to test the other instance a bit more first
2022-10-05 10:37:27 <ikke> ok
2022-10-05 10:37:29 <psykose> and make sure all the little things work
2022-10-05 10:37:35 <psykose> but everything i tried was ok i think
2022-10-05 10:37:39 <psykose> you should go over it too
2022-10-05 10:50:25 <ikke> clandmeter: nld-dev-1.alpinelinux.org
2022-10-05 10:57:56 <clandmeter> ok
2022-10-05 10:58:01 <clandmeter> maybe you want to rename the box?
2022-10-05 10:58:02 <clandmeter> in the panel
2022-10-05 10:58:13 <ikke> makes sense
2022-10-05 10:58:19 <clandmeter> it will also udpaste rdns
2022-10-05 10:58:23 <clandmeter> -p
2022-10-05 10:58:38 <clandmeter> what 
2022-10-05 10:58:40 <clandmeter> update :)
2022-10-05 10:59:56 <ikke> Done
2022-10-06 15:31:18 <ikke> clandmeter: We switched the wiki over to gbr-app-1
2022-10-06 15:34:43 <ncopa> Code of conduct has been updated: https://alpinelinux.org/community/code-of-conduct.html
2022-10-06 15:37:37 <ikke> good
2022-10-06 18:21:51 <psykose> ikke: ppc64le seems to be stuck
2022-10-06 18:22:13 <ikke> It's not doing a lot
2022-10-06 18:22:30 <psykose> yep
2022-10-07 09:43:30 <clandmeter> ikke: ok, thats linode i guess
2022-10-07 09:43:59 <clandmeter> are we still operating within billing limits?
2022-10-07 09:46:15 <ikke> clandmeter: yes
2022-10-07 09:46:33 <ikke> Last bill was less then halve 
2022-10-07 09:47:35 <ikke> And that was the idea right? Move everything from nld3 to a linode host 
2022-10-07 09:57:11 <clandmeter> yes
2022-10-07 09:57:17 <clandmeter> with those limitations in mind ofc
2022-10-07 09:57:28 <clandmeter> just wanted to make sure you are aware of it :)
2022-10-07 10:22:08 <ikke> certainly, I'm checking always checking the bill when we receive it
2022-10-07 10:22:18 <ikke> s/checking always/always/
2022-10-07 10:22:18 <alpine-meetbot> ikke meant to say: certainly, I'm always checking the bill when we receive it
2022-10-07 13:21:15 <mps> ikke: about next lts kernel https://lore.kernel.org/lkml/Yz%2FZWBeNZvKenEVM@kroah.com/
2022-10-07 13:22:59 <ikke> Right 
2022-10-08 12:23:08 <ikke> oh, interesting, appears that the alpinelinux namespace on docker is marked sponsored oss
2022-10-09 10:37:25 <clandmeter> ikke: i bet ncopa applied for it and he didnt notice the email saying it was approved
2022-10-09 11:09:29 <ikke> :D
2022-10-10 06:40:39 <clandmeter> good morning
2022-10-10 10:19:04 <ikke> clandmeter: o/
2022-10-10 11:02:03 <ncopa> hi
2022-10-10 11:02:56 <ncopa> do we now have the docker rate limit incresased/removed?
2022-10-10 11:03:13 <ncopa> i never saw any email. I lokoed for response a couple of weeks ago but didnt find any
2022-10-10 11:53:33 <ikke> ncopa: I suspect so 
2022-10-10 11:57:47 <ncopa> i think i lose some emails
2022-10-10 11:57:54 <ncopa> that never go through
2022-10-10 15:10:16 <clandmeter> ncopa: did you manage to build rust for rv64?
2022-10-10 15:15:54 <ncopa> almost
2022-10-10 15:16:08 <ncopa> tried again today but it appears to have deadlocked again :-(
2022-10-10 15:17:20 <ncopa> maybe qemu-riscv64 binary was not a patched one
2022-10-10 15:20:27 <clandmeter> patched?
2022-10-10 15:20:34 <clandmeter> i thought we didnt need a patched version?/
2022-10-10 15:20:42 <clandmeter> just the export
2022-10-10 15:25:16 <ikke> clandmeter: Do you think we're ready to start moving dns records around to point to t1.a.o?
2022-10-10 15:26:46 <clandmeter> i think so
2022-10-10 15:26:56 <clandmeter> if ncopa agrees :)
2022-10-10 15:27:15 <clandmeter> we set a date to do it, but i think we never managed it
2022-10-10 15:27:24 <ncopa> i have patched glib and rebuilt qemu in edge so export should not be needed. i tried build without the export and it failed. I'm adding it back again
2022-10-10 15:27:44 <ncopa> i am ok to move the dns records
2022-10-10 20:03:16 <mps> are these two somehow related to alpine https://www.youtube.com/watch?v=2W4NiS70bDU
2022-10-10 20:05:34 <ikke> Their names are not familiar to me
2022-10-11 07:39:58 <clandmeter> funny to hear him talk about the history
2022-10-11 08:20:23 <ncopa> no, i dont know them
2022-10-11 08:27:09 <mps> well, it was boring to watch but I managed to endure :)
2022-10-11 09:40:47 <ncopa> was a bit long but was nice
2022-10-11 09:56:51 <mps> yes, it is always nice if someone praise you ;-)
2022-10-11 09:57:16 <mps> and pleasant
2022-10-11 09:58:19 <mps> for me it was boring but anyway pleasant to see they praise alpine
2022-10-11 10:02:26 <mps> ncopa: earlier someone told on other channel that you have presentation/talk about building linux distro with musl, but didn't told me if it is online
2022-10-11 17:33:58 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/41
2022-10-11 17:37:37 <ikke> ncopa: seems like your e-mail bounces
2022-10-11 17:37:47 <ikke> 551 5.7.1 Service unavailable (in
2022-10-11 17:37:50 <ikke>     reply to RCPT TO command
2022-10-11 17:38:28 <ikke> might explain why you think you are missing e-mails
2022-10-11 18:51:02 <ikke> TTL of rsync.a.o lowered
2022-10-11 18:51:12 <ikke> So maybe we can switch tomorrow?
2022-10-12 10:16:51 <ikke> clandmeter: ping
2022-10-12 18:22:17 <ikke> psykose: seems like zabbix alerting is broken in algitbot: https://tpaste.us/KeRg 
2022-10-12 18:23:21 <psykose> i'll look at it in like 30 minutes
2022-10-12 18:25:49 <ikke> seems like a permission issue
2022-10-12 18:28:52 <psykose> where do the user/password come from
2022-10-12 18:29:45 <psykose> hm
2022-10-12 18:30:07 <ikke> connect(6, {sa_family=AF_UNIX, sun_path="/var/run/sircbot/"}, 110) = -1 ECONNREFUSED (Connection refused)
2022-10-12 18:31:19 <ikke> local infrachan  = os.getenv("MQTT_EXEC_INFRA_CHANNEL") or "#alpine-infra"
2022-10-12 18:31:51 <ikke> MQTT_EXEC_INFRA_CHANNEL=
2022-10-12 18:31:53 <ikke> that's the issue
2022-10-12 18:32:08 <psykose> wonder how it worked before
2022-10-12 18:32:20 <ikke> old compose
2022-10-12 18:32:28 <ikke> .env handling changed
2022-10-12 18:32:48 <psykose> there's a env.new with it
2022-10-12 18:32:50 <psykose> what does it do now
2022-10-12 18:33:08 <ikke> now you need to quote values
2022-10-12 18:33:18 <ikke> before it just took everything after =
2022-10-12 18:33:31 <psykose> don't think the freenode in here is accurate
2022-10-12 18:33:35 <ikke> fixedf
2022-10-12 18:34:04 <psykose> quote them where, the env.new?
2022-10-12 18:34:07 <ikke> .env
2022-10-12 18:34:11 <psykose> ah
2022-10-12 18:34:28 <psykose> interesting
2022-10-12 18:34:50 <psykose> maybe we should swap to golang compose too
2022-10-12 18:34:51 <ikke> clandmeter already fixed a similar issue
2022-10-12 18:34:59 <ikke> psykose: we already have
2022-10-12 18:35:08 <ikke> well, at least I
2022-10-12 18:35:53 <psykose> how does one set it given that it's in libexec? maybe we should delete the regular python one too
2022-10-12 18:35:54 <psykose> shrug
2022-10-12 18:36:18 <ikke> psykose: `docker compose` executes it
2022-10-12 18:36:26 <psykose> ah i guess i'm just a boomer then
2022-10-12 18:37:48 <ikke> on most hosts I already removed the python one
2022-10-12 19:53:05 <ikke> hey mick_ibm 
2022-10-12 19:53:21 <mick_ibm> hey whats up
2022-10-12 19:53:36 <ikke> I'm Kevin Daudt
2022-10-12 19:53:43 <ikke> We've been in contact via e-mail
2022-10-12 19:53:48 <mick_ibm> ohh yea, whats up Kevin?!
2022-10-12 19:53:57 <ikke> Doing alright
2022-10-12 19:54:22 <ikke> How about you?
2022-10-12 19:56:21 <mick_ibm> sweet, im doing well thanks. today i introduced some high school students to bash scripting and they were pretty excited so i think there is still hope in humanity, haha. otherwise, things are slowly getting ironed out over at osu osl cause of the unicamp shutdown
2022-10-12 19:56:59 <ikke> Nice! And I can imagine such a migration is quite some project
2022-10-12 19:58:36 <mick_ibm> yea, it is sad unicamp is shutting down :( but working with other distros and projects to make sure we dont miss anything for ppc64le
2022-10-12 19:59:44 <ikke> Regarding upgrading the P9 host, I think it would be nice to try to alreayd upgrade it to 3.16
2022-10-12 20:00:01 <psykose> forgot that wasn't the case already
2022-10-12 20:00:16 <psykose> what special booting setup did we have on it
2022-10-12 20:00:43 <ikke> seems to be grub
2022-10-12 20:00:57 <psykose> yolo it :)
2022-10-12 20:01:00 <ikke> grub-ieee1275
2022-10-12 20:01:14 <ikke> mick_ibm will help us with it
2022-10-12 20:01:25 <ikke> we don't have any oob access, so if it does not come back, we do not have any recourse ourself :)
2022-10-12 20:02:12 <mick_ibm> hahaha, ok we can do 3.16, yup grub... i had a meeting with lab admin and still in contact with him, he is aware of the oob access and seems like he is going to handle the upgrade personally
2022-10-12 20:02:30 <ikke> ok, fine with me
2022-10-12 20:02:51 <mick_ibm> i dont think i will be given that access right now unless its absolutely required but he is pretty kernel savvy. 
2022-10-12 20:03:35 <mick_ibm> so what ill do is maybe send an email to that power-dev request that i mentioned in the email to you, but ill ping him on slack to make sure he can still handle it
2022-10-12 20:03:47 <ikke> alright
2022-10-12 20:03:54 <mick_ibm> ill send the email and cc you and maybe carlo to make sure you guys are in the loop
2022-10-12 20:04:05 <ikke> sounds good
2022-10-12 20:04:14 <ikke> certainly keep carlo in the loop :)(
2022-10-12 20:04:14 <mick_ibm> sweet
2022-10-12 20:04:23 <mick_ibm> will do
2022-10-12 20:05:30 <mick_ibm> oh and stan is another employee at ibm that is more on the management side. i forgot he mentioned he was gonna reach out to you, haha sorry about the DUP
2022-10-12 20:05:40 <ikke> No worry :)
2022-10-12 20:15:02 <mick_ibm> so im assuming upgrade to 3.16.2?
2022-10-12 20:15:42 <ikke> the patch version is less relevant
2022-10-12 20:16:11 <ikke> The repos always have the latest version
2022-10-12 20:16:17 <mick_ibm> oh ok, im just not super familiar with the release schedule so just wanna confirm im not missing somethin important. ok got it
2022-10-12 20:16:40 <ikke> patch releases are mostly done for the iso's and docker images
2022-10-12 20:17:47 <ikke> The upgrade process is basically: change the repos in /etc/apk/repositories to v3.16, upgrade all packages
2022-10-12 20:17:59 <ikke> and then fix config if necessary
2022-10-12 20:22:39 <ikke> mick_ibm: I'm sure we'll be in contact with the lab admin, but in just in case: alpine v3.15 started using new signing keys, so you need to make sure they are installed before we can upgrade to v3.16
2022-10-12 20:24:35 <mick_ibm> hmm ok, im not sure im very familiar with signing keys but im sure Lance would know. Sounds like a sha checksum thing to verify? would he already have a key or is that something we need to obtain now?
2022-10-12 20:25:29 <ikke> mick_ibm: they are rsa keys. It's a package that you need to install (alpine-keys) from a repository (alpine version) that has the new keys
2022-10-12 20:25:50 <mick_ibm> ok thanks
2022-10-12 20:26:06 <mick_ibm> ill include that in email request
2022-10-12 20:26:24 <ikke> https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.15.0#New_package_signing_keys
2022-10-12 20:27:22 <mick_ibm> perfect, thank you!
2022-10-12 21:01:27 <mick_ibm> ohhh perhaps i misunderstood Lance's question in the email... is there a time when this machine should not be offline?
2022-10-13 00:13:41 <psykose> not really, the builders are always "up" and there's no priority time
2022-10-13 00:13:43 <psykose> downtime is downtime
2022-10-13 00:14:04 <psykose> soon we will be bootstrapping the 3.17 release but that's going to take a while, so if it's not 1-2 weeks from now (before) it would be fine
2022-10-13 05:32:16 <ikke> psykose: fyi, this is the CI host, not builder
2022-10-13 05:32:24 <psykose> ah
2022-10-13 05:32:40 <psykose> i had mixed them up backwards and thought the builder was p9 instead
2022-10-13 05:32:41 <psykose> right
2022-10-13 05:32:50 <psykose> in that case the downtime doesn't matter nuke it for 2 weeks if you want
2022-10-13 05:32:51 <psykose> : )
2022-10-13 05:32:58 <ikke> I've replied that via e-mail
2022-10-13 05:33:06 <ikke> well, apart from the 2 weeks :P
2022-10-13 05:33:10 <psykose> haha
2022-10-13 06:18:52 <clandmeter> ikke: pong :)
2022-10-13 06:33:41 <ikke> clandmeter: thinking about switching rsync.a.o today 
2022-10-13 06:35:14 <clandmeter> sounds good
2022-10-13 06:35:22 <clandmeter> but i will be kind of busy
2022-10-13 06:35:49 <clandmeter> but i dont think we need 3ppl to change dns records :D
2022-10-13 06:37:12 <ikke> Nope
2022-10-13 06:39:19 <clandmeter> did we have to make changes to fastly also?
2022-10-13 06:40:12 <ikke> Yes
2022-10-13 06:40:31 <ikke> Point it to t1.a.o
2022-10-13 11:23:25 <ikke> new console on equinix
2022-10-13 11:25:22 <ikke> clandmeter: would be nice if we could ipv6 working as well for the t1 infra
2022-10-13 11:35:44 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/42
2022-10-13 11:41:37 <ikke> https://zabbix.alpinelinux.org/zabbix.php?action=dashboard.view&dashboardid=12
2022-10-13 11:48:41 <ikke> Switched
2022-10-13 11:51:12 <ikke> clandmeter: is there a reason we only sync hourly? Should we sync like every 15 minutes?
2022-10-13 11:51:25 <ikke> s/Should/Shouldn't/
2022-10-13 11:51:26 <alpine-meetbot> ikke meant to say: clandmeter: is there a reason we only sync hourly? Shouldn't we sync like every 15 minutes?
2022-10-13 12:14:52 <ikke> nld already receiving 600 mbit
2022-10-13 12:14:56 <ikke> sgp about 40
2022-10-13 12:18:19 <ikke> We did loose traffic. The 3.5gbps spike is gone while not fully compensated by the new mirrors yet
2022-10-13 13:06:23 <ikke> nld getting 1.75gbps.
2022-10-13 13:06:37 <ikke> clandmeter: I don't see incoming traffic to USA, wondering of anycast is working there
2022-10-13 15:07:15 <ikke> https://i.imgur.com/wxCrgDJ.png
2022-10-13 15:40:48 <clandmeter> hi'
2022-10-13 15:41:07 <clandmeter> hmm interesting
2022-10-13 15:41:29 <clandmeter> ikke: are the ports open on usa?
2022-10-13 15:42:45 <clandmeter> im ok with syncing every 15 min
2022-10-13 15:42:56 <clandmeter> but the sync should be automatic
2022-10-13 15:43:06 <clandmeter> the one hour one is just a backup
2022-10-13 16:29:28 <ikke> Ah ok
2022-10-13 16:29:42 <ikke> Havent checked USA yet 
2022-10-13 18:26:38 <ikke> clandmeter: rsync seems to be accessible from usa.t1.alpinelinux.org
2022-10-13 18:43:25 <ikke> traceroute from newyork seems to end up in amsterdam
2022-10-13 18:48:56 <ikke> maybe we can ask equinix
2022-10-13 18:50:09 <ikke> Ramereth: o/ I'm Kevin
2022-10-13 18:50:31 <mick_ibm> i was just about to make that introduction :) 
2022-10-13 18:50:36 <Ramereth> ikke: greetings!
2022-10-13 18:50:43 <Ramereth> hey mick_ibm !
2022-10-13 18:52:47 <mick_ibm> salutations! 
2022-10-13 19:25:34 <clandmeter> Strange
2022-10-13 19:25:51 <clandmeter> did I read the docs wrong?
2022-10-13 19:26:01 <clandmeter> we do need bgp?
2022-10-13 19:26:38 <ikke> "I don't see BGP turned on for these systems, so I assume that your intent is to have a static binding to these addresses.
2022-10-13 19:26:41 <ikke> "
2022-10-13 19:26:48 <ikke> seems like it's an option to not have it
2022-10-13 19:31:39 <ikke> clandmeter: did you follow the steps in that document?
2022-10-13 19:38:11 <clandmeter> Yes that’s what I thought
2022-10-13 19:38:18 <clandmeter>  and I think I tested it
2022-10-13 19:38:30 <ikke> I've replied
2022-10-13 19:38:37 <clandmeter>  nod
2022-10-13 19:41:04 <ikke> Hitting ~4gbps at peaks on both nld + sgp
2022-10-13 19:57:40 <clandmeter> Sounds like not our problem 🤞
2022-10-13 19:59:53 <ikke> indeed
2022-10-13 20:18:20 <clandmeter> Fix tomorrow
2022-10-14 07:56:03 <clandmeter> ikke: i guess still no fix for usa.t1, looks like no traffic.
2022-10-14 07:57:19 <ikke> They said today us morning 
2022-10-14 07:57:56 <clandmeter> oh ok
2022-10-14 14:32:00 <ptrc_> fwiw, https://git.alpinelinux.org/cgit.css returns 429..?
2022-10-14 14:32:14 <psykose> works on my machine dot png
2022-10-14 15:12:33 <ikke> We are receiving traffic now on usa.t1
2022-10-14 18:50:31 <ikke> 7gbps peak :D
2022-10-14 18:50:37 <ikke> (all 3 combined)
2022-10-15 06:14:42 <ikke> ppc64le builder is MIA
2022-10-15 09:22:44 <mps> for those who use linux-edge it is upgraded to 6.0.2 with latest wifi fixes
2022-10-16 19:19:39 <clandmeter> ikke: i guess you didnt update the bug regarding the dns change?
2022-10-16 19:25:49 <ikke> euhm, no, forgot
2022-10-16 19:27:21 <ikke> done
2022-10-16 19:28:19 <ikke> clandmeter: interesting to see how much spikier the traffic in NLD is
2022-10-16 19:28:35 <ikke> looks like mirrors there have a lot more bandwidth and are done sooner :)
2022-10-16 19:29:34 <clandmeter> heh
2022-10-17 08:46:49 <clandmeter> ikke: did you update fastly also?
2022-10-17 08:51:13 <ikke> No, that's the next step 
2022-10-17 09:07:09 <ikke> We also need to make sure it has certificates for the other domains we want to point at it, right? 
2022-10-17 10:15:51 <clandmeter> we want to point all remaning a.o domains to it?
2022-10-17 10:20:07 <clandmeter> ikke: i only see uk.a.o in the mirrors list?
2022-10-17 10:20:33 <ikke> nl is there
2022-10-17 10:20:49 <ikke> and some older dl-*
2022-10-17 10:21:03 <clandmeter> i dont see them 
2022-10-17 10:21:10 <clandmeter> https://mirrors.alpinelinux.org/
2022-10-17 10:21:15 <ikke> No, they are no longer in that list
2022-10-17 10:21:29 <clandmeter> you mean in dns
2022-10-17 10:21:39 <ikke> yes
2022-10-17 10:21:59 <clandmeter> anything we point to cdn will lose rsync
2022-10-17 10:22:04 <clandmeter> if it was ever available
2022-10-17 10:22:26 <ikke> I don't think so
2022-10-17 10:23:59 <clandmeter> think what?
2022-10-17 10:24:07 <ikke> that they were providing rysnc
2022-10-17 10:24:10 <ikke> rsync
2022-10-17 10:24:28 <ikke> I thought we looked at that last time
2022-10-17 10:25:45 <clandmeter> could be
2022-10-17 10:26:35 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/32
2022-10-17 10:28:05 <ikke> not complete yet
2022-10-17 10:29:45 <ikke> uk.a.o, no.a.o
2022-10-17 10:29:56 <ikke> dl-[0-9].a.o
2022-10-17 10:30:20 <clandmeter> and what do we do with uk?
2022-10-17 10:30:24 <clandmeter> point to t1?
2022-10-17 10:33:45 <ikke> uk does have rsync, right?
2022-10-17 10:34:10 <ikke> I guess it would make sense to point it to t1
2022-10-17 10:37:12 <ikke> no.a.o is defunct, so we can remove that
2022-10-17 10:39:52 <ikke> clandmeter: dl-[2-5] do accept rsync connections
2022-10-17 10:40:12 <clandmeter> i think they all point to the old t1
2022-10-17 10:40:23 <clandmeter> nld3-dev1
2022-10-17 10:40:26 <ikke> yes
2022-10-17 10:41:24 <ikke> we can also point everything to t1.a.o to make it easier
2022-10-17 10:41:38 <ikke> then we do no have to add all these domains to fastly
2022-10-17 10:41:46 <ikke> clandmeter: any preference?
2022-10-17 10:42:03 <clandmeter> lets do that
2022-10-17 10:42:13 <ikke> ok
2022-10-17 10:42:14 <clandmeter> we can clean them up later
2022-10-17 10:42:45 <clandmeter> if we really want to cancel them
2022-10-17 10:43:02 <ikke> At that point they're just dns records, should not be too much effort to keep them
2022-10-17 10:47:08 <ikke> clandmeter: we also have geo.a.o, which I guess is not really relevant anymore?
2022-10-17 10:47:15 <ikke> and ns.geo
2022-10-17 10:47:21 <clandmeter> correct
2022-10-17 10:47:25 <clandmeter> we can cancel it
2022-10-17 10:47:35 <ikke> Will do that later
2022-10-17 10:47:36 <clandmeter> i think we still have the service running
2022-10-18 08:33:45 <ncopa> I'd like to start up the 3.17 builders now, is that ok with infra team?
2022-10-18 08:39:47 <psykose> if you mean finish bootstrap and let it rip, i'd wait till tomorrow
2022-10-18 08:39:52 <psykose> travelling today so i'll be away
2022-10-18 08:40:04 <psykose> but you can do everything preemptive if you want :)
2022-10-18 08:46:09 <ikke> ncopa: fine with me, just a little concerned for diskspace  on some of the builders, but we'll see 
2022-10-18 09:04:43 <ncopa> no, im about to start the bootstrapping
2022-10-18 09:34:58 <clandmeter> fine by me
2022-10-18 09:35:10 <clandmeter> ill be away anyways :)
2022-10-18 09:35:36 <clandmeter> ill be back before 3.18 i promise
2022-10-18 09:39:23 <ikke> :D
2022-10-18 09:41:14 <ncopa> lol
2022-10-18 09:44:29 <psykose> brb extending my holiday 5 weeks
2022-10-18 09:58:13 <ncopa> nice
2022-10-18 09:59:20 <ncopa> whats up with the ppc64le builders?
2022-10-18 09:59:24 <ncopa> they seems all be missing
2022-10-18 10:01:04 <ncopa> ppc64le has been gone for 3 days
2022-10-18 10:19:28 <ikke> I've asked our contact at Unicamp via their slack channel
2022-10-18 10:20:03 <ncopa> thanks
2022-10-18 11:35:18 <ncopa> can i join the slackchannel?
2022-10-18 12:16:25 <clandmeter> slack is so confusing
2022-10-18 12:24:59 <ikke> ncopa: sent you an invite
2022-10-18 12:52:09 <ncopa> seems like i dont get the invite. maybe it is bouncing?
2022-10-18 12:54:56 <ikke> Should I send it to a different address? 
2022-10-18 13:32:48 <ncopa> i wonder if I can fix the email server
2022-10-18 13:59:09 <ncopa> Oct 18 13:29:53 mail postfix/smtpd[9184]: NOQUEUE: milter-reject: RCPT from nld3-dev1.alpinelinux.org[147.75.101.119]: 451 4.7.1 Message quota exceeded;
2022-10-18 14:13:40 <ncopa> ikke: can you please try to resend the invitation
2022-10-18 14:24:23 <clandmeter> im talking to one of the infra ppl
2022-10-18 14:24:31 <clandmeter> but it seems not possible
2022-10-18 14:25:37 <ncopa> i think i have fixed the email setup. apparently there was a per ip rate limit in greylisting, and since everything for alpinelinux.org comes from same server it dropped messages
2022-10-18 14:25:47 <ncopa> i have added smtp.a.o to the while list
2022-10-18 14:27:50 <ncopa> *white
2022-10-18 14:33:39 <ikke> Seems like we won't be getting the keys back as clandmeter said 
2022-10-18 14:34:52 <ncopa> ok. any explanation why we didnt get notified? or did we miss the notification?
2022-10-18 14:35:22 <clandmeter> hi, unfortunately no scalation will be possible. We assume the data there is fully open source and public available elsewhere.
2022-10-18 14:35:30 <clandmeter> heh
2022-10-18 14:35:49 <clandmeter> so we just unplug it and burry it.
2022-10-18 14:36:02 <ncopa> i wonder what to do. it means that currently ppc64le does not get any updates
2022-10-18 14:36:18 <ncopa> are ibm ok with no updates for the current stable releases?
2022-10-18 14:36:45 <clandmeter> i asked if we could escalate the issue
2022-10-18 14:36:51 <clandmeter> he said no
2022-10-18 14:37:09 <clandmeter> maybe we can try to find another contact window 
2022-10-18 14:38:46 <ikke> Check netbox 
2022-10-18 14:39:05 <clandmeter> for what?
2022-10-18 14:39:22 <ikke> Contacts
2022-10-18 14:42:27 <clandmeter> hmm netbox kind of changed for me since the last time Ilogged in
2022-10-18 14:46:28 <clandmeter> ikke: want me to write an email?
2022-10-18 14:53:07 <clandmeter> i think the current contacts we have are for US
2022-10-18 14:53:10 <clandmeter> not brazil
2022-10-18 14:53:28 <clandmeter> and the one I have for brazil is the one im talking to on slack
2022-10-18 14:55:23 <ikke> I have a feeling they're not on friendly terms anymore somehow 
2022-10-18 15:09:56 <clandmeter> email send
2022-10-18 15:10:10 <clandmeter> i included everyone that i find from br.ibm.com, i guess some are gone.
2022-10-18 15:18:28 <ikke> clandmeter: thanks
2022-10-18 15:18:31 <ikke> let's see what happens
2022-10-18 15:46:38 <ncopa> can we set up a new builder on the ppc64le CI machine? which ip or hostname?
2022-10-18 16:12:59 <ikke> ppc64le-ci.alpinelinux.org
2022-10-18 16:13:09 <ikke> It's running docker now, but should be able to run lxc as well
2022-10-18 16:14:48 <ikke> it already has dmvpn
2022-10-18 16:20:34 <ikke> ncopa: I'm setting up our default lxc setup on that machine
2022-10-18 16:40:29 <ikke> ncopa: fyi, we are/were planning upgrading this host
2022-10-18 17:32:59 <ikke> having ARP issues in the container
2022-10-18 18:45:45 <mick_ibm> hey carlo, where was that ppc64le builder machine located that was decommissioned? 
2022-10-18 18:47:05 <mick_ibm> im not sure what slack channel you are referring to from that email 
2022-10-18 20:01:12 <ikke> mick_ibm: it was hosted at unicamp 
2022-10-18 20:01:28 <mick_ibm> hmmm ok
2022-10-18 20:01:55 <mick_ibm> this might be a problem then. lemme sync up with some ibmers to learn more
2022-10-18 20:02:17 <ikke> The slack channel is called power developers exchange 
2022-10-18 20:05:07 <mick_ibm> thanks, ill join shortly. originally unicamp was planned to be shutdown in november but it happened sooner so ill dig into this. who were you talking with in that slack channel? 
2022-10-18 20:05:34 <ikke> Rafael Persene 
2022-10-18 20:06:07 <mick_ibm> oh ok, he is in my dept. ill chat with him today
2022-10-18 20:06:24 <ikke> We generated new (stronger) signing keys for the latest releases and forgot to back them up
2022-10-18 20:06:50 <ikke> The rest we can get back 
2022-10-18 21:05:14 <mick_ibm> alrighty i just chatted with rafael and there is not a way to get those keys back :( 
2022-10-18 21:06:09 <mick_ibm> also i can reply to email thread but this seemed high riorty so thought id share here first in order to let you know asap
2022-10-18 21:08:27 <mick_ibm> so is there something i can help with... what exactly does it mean to have "manual intervention to install new signing keys." ?
2022-10-19 04:32:43 <ikke> everyone with an existing installation will need to install the updated key package, which is untrustued
2022-10-19 04:32:55 <ikke> oh, mick is not her
2022-10-19 04:32:58 <ikke> here*
2022-10-19 06:38:45 <ncopa> morning
2022-10-19 06:38:58 <ncopa> is ppc64le-ci a vm or a real machine?
2022-10-19 07:01:51 <ikke> Afaik bare metal 
2022-10-19 08:38:07 <ncopa> i'd like connect my macbook to alpine infra vpn. not sure which is easiest, openvpn or wireguard?
2022-10-19 08:49:41 <mps> ncopa: not much difference ime
2022-10-19 08:50:24 <mps> you want to connect it as client, or server?
2022-10-19 09:02:19 <ncopa> client
2022-10-19 09:02:53 <ncopa> do we have an wireguard endpoint for alpine infra? where? how do I create a key for my client?
2022-10-19 09:03:01 <ikke> Yes, we do 
2022-10-19 09:03:21 <ikke> Just use wg to create a pair 
2022-10-19 09:04:17 <ikke> https://gitlab.alpinelinux.org/alpine/infra/infra/-/wikis/Alpine-wireguard-VPN
2022-10-19 10:21:57 <ikke> ncopa: you can use 172.16.252.14/24
2022-10-19 10:22:10 <ikke> If you give your public key, I can add it to the hub for you
2022-10-19 10:27:26 <ncopa> pubkey: 8pVfygnPMEs+ILYiyPQTn13hjGTosA3+J5O1jVDNg14=
2022-10-19 10:27:32 <ncopa> im using the macOS client
2022-10-19 10:28:56 <ikke> ok, I've added your key
2022-10-19 10:29:51 <ikke> ncopa: note sure how the mac client works, but you should be able to connect now
2022-10-19 10:30:17 <ikke> You need to manually add a route 172.16.0.0/16 via 172.16.252.1
2022-10-19 10:33:20 <ncopa> am I supposed to be able to ping 172.16.252.1?
2022-10-19 10:33:37 <ikke> 64 bytes from 172.16.252.1: icmp_seq=1 ttl=64 time=11.7 ms
2022-10-19 10:33:39 <ikke> yes
2022-10-19 10:33:55 <mps> ikke: from alpine this is enough '172.16.0.0/16 dev qwg0 scope link' (qwg0 is my interface to infra)
2022-10-19 10:34:30 <ikke> as a route?
2022-10-19 10:34:36 <mps> yes
2022-10-19 10:34:45 <ikke> aha
2022-10-19 10:35:32 <ncopa> it should work from behind NAT right?
2022-10-19 10:35:41 <ikke> yes
2022-10-19 10:36:01 <mps> just logged in gateway and I see you added some servers to wg conf
2022-10-19 10:36:15 <ikke> mps: yes, as a quickfix
2022-10-19 10:36:42 <mps> ncopa: yes, behind NAT and on roaming also
2022-10-19 10:37:00 <mps> ikke: everything ok :)
2022-10-19 10:37:03 <ncopa> status says "active" here in the macOS gui
2022-10-19 10:37:16 <ikke> did you add the hub public key?
2022-10-19 10:37:23 <ncopa> yes
2022-10-19 10:37:31 <ikke> did you set the ip address?
2022-10-19 10:37:45 <ncopa> no
2022-10-19 10:37:55 <ikke> You need to explicitly set it
2022-10-19 10:38:00 <ncopa> [Peer]
2022-10-19 10:38:00 <ncopa> AllowedIPs = 172.16.0.0/16
2022-10-19 10:38:00 <ncopa> Endpoint = wg.alpinelinux.org:41414
2022-10-19 10:38:00 <ncopa> PublicKey = hm9zrdHJU2fJsrKnRqo4xXlqbsdNC0CfTySwOzdBwm8=
2022-10-19 10:38:25 <ikke> that's correct
2022-10-19 10:38:32 <ikke> but you need to set a static IP yourself
2022-10-19 10:39:02 <ikke> The one I provided
2022-10-19 10:39:24 <ncopa> brilliant
2022-10-19 10:39:25 <ncopa> 64 bytes from 172.16.252.1: icmp_seq=5 ttl=64 time=52.254 ms
2022-10-19 10:39:32 <mps> just ncopa I can ping your IP from gateway
2022-10-19 10:39:54 <ikke> good
2022-10-19 10:40:08 <mps> and also from my client
2022-10-19 10:41:57 <ncopa> sweet, it works
2022-10-19 10:42:04 <ikke> very good
2022-10-19 10:42:20 <ikke> the only thing I've noticed myself is that on boot it complains that it cannot resolve wg.a.o
2022-10-19 10:42:46 <ncopa> because it has no network at that time so it cannot resolve it
2022-10-19 10:42:58 <mps> ikke: gateway or your clients?
2022-10-19 10:43:01 <ikke> clients
2022-10-19 10:43:30 <ikke> I just added a static interface in /etc/network/interfaces
2022-10-19 10:43:47 <ncopa> it needs to start after the interface with the default gateway
2022-10-19 10:43:53 <ncopa> so you can do dns resolution
2022-10-19 10:44:01 <mps> maybe we could add DNS to config but not sure how it will work with musl resolver
2022-10-19 10:44:02 <ncopa> I think iproute-ng can do depdencies
2022-10-19 10:45:20 <ikke> https://github.com/ifupdown-ng/ifupdown-ng/blob/34753136b86d5b1d5df2b60b22f1bfac16f28416/doc/ifupdown-executor.scd#phases
2022-10-19 10:45:36 <ikke> But that's for executors
2022-10-19 10:46:52 <ikke> the wireguard executor does not have a depends phase
2022-10-19 10:46:53 <mps> ikke: what is address of DNS server in infra
2022-10-19 10:47:03 <ikke> 172.16.6.3
2022-10-19 10:47:19 <ikke> (old one is 172.16.8.3, but will soon be decommissioned)
2022-10-19 10:48:27 <mps> ohm, I have it already but commented. sorry
2022-10-19 10:48:37 <ikke> ah, requires
2022-10-19 10:49:01 <ncopa> PING ns1.alpin.pw (172.16.8.3): 56 data bytes
2022-10-19 10:49:01 <ncopa> 64 bytes from 172.16.8.3: icmp_seq=0 ttl=62 time=52.697 ms
2022-10-19 10:49:12 <ncopa> got dns working
2022-10-19 10:49:23 <ncopa> macOS makes it relatively simple
2022-10-19 10:49:39 <mps> ahm, yes resolvconf add DNS server from wg conf
2022-10-19 10:49:41 <ncopa>  cat /etc/resolver/alpin.pw | tpaste
2022-10-19 10:49:41 <ncopa> https://tpaste.us/yvje
2022-10-19 10:49:59 <ncopa> sweet!
2022-10-19 10:50:15 <ikke> 8.33 will soon be decomissioned
2022-10-19 10:50:27 <ncopa> ok
2022-10-19 10:50:38 <ncopa> I'll remove it from the config here
2022-10-19 10:50:41 <ikke> 6.33 is it's replacement
2022-10-19 10:50:51 <ikke> so just change 8 to 6
2022-10-19 10:52:42 <mps> here is the image of speed diff openvpn and WG https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/
2022-10-19 10:57:27 <ncopa> ikke: are you setting up lxc on ppc64le-ci? anything I can do?
2022-10-19 10:57:58 <ikke> ncopa: I got stuck a bit
2022-10-19 10:58:09 <ikke> I have a test container running, but it cannot access it's gateway
2022-10-19 10:58:16 <ncopa> im trying to set up a ncopa-edge-ppc64le
2022-10-19 10:58:52 <ncopa> ==> Installing Alpine Linux in /var/lib/lxc/ncopa-edge-ppc64le/rootfs
2022-10-19 10:58:52 <ncopa> ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.16/main: UNTRUSTED signature
2022-10-19 10:58:52 <ncopa> fetch http://dl-cdn.alpinelinux.org/alpine/v3.16/main/ppc64le/APKINDEX.tar.gz
2022-10-19 10:58:57 <ikke> yes
2022-10-19 10:59:00 <ikke> ran into that as well
2022-10-19 10:59:06 <ikke> probably due to old  templates
2022-10-19 10:59:18 <ikke> Latest I could install was 3.14
2022-10-19 11:00:14 <ikke> ncopa: 172.16.15.1 dev eth0  used 6/852/3 probes 3 FAILED
2022-10-19 11:00:36 <ikke> (from the test container)
2022-10-19 11:02:47 <ikke> kernel is linux 5.4
2022-10-19 11:04:16 <ncopa> could be firewall issue?
2022-10-19 11:04:34 <ncopa> I can reproduce it in my ncopa-edge-ppc64le that I just created
2022-10-19 11:05:14 <ikke> ncopa: I did try to add a -j ACCEPT rule as first rule in both OUTPUT and INPUT
2022-10-19 11:05:19 <ikke> but yes, it could be the firewall
2022-10-19 11:05:30 <ncopa> may need NAT
2022-10-19 11:05:37 <ikke> Not to ping your local gateway
2022-10-19 11:05:48 <ikke> this is ARP failing
2022-10-19 11:05:56 <ncopa> true
2022-10-19 11:06:14 <ikke> Let me try to purge all rules
2022-10-19 11:07:46 <ncopa> looks like there is an ip address conflict
2022-10-19 11:07:53 <ikke> oh
2022-10-19 11:07:58 <ncopa> there is both br0 and lxcbr0 with same ip
2022-10-19 11:08:12 <ncopa> maybe we just shut down br0 and use lxcbr0?
2022-10-19 11:08:14 <ikke> ah
2022-10-19 11:08:17 <ikke> yes, makes sense
2022-10-19 11:08:48 <ikke> I think I manually added br0 just for dmvpn
2022-10-19 11:09:05 <ikke> feel free to remove it
2022-10-19 11:09:13 <ncopa> yup. that was it
2022-10-19 11:09:19 <ikke> ok, great
2022-10-19 11:09:27 <ikke> restoring iptables
2022-10-19 11:09:42 <ikke> done
2022-10-19 11:10:03 <ikke> ncopa: nice find
2022-10-19 11:10:33 <ncopa> I removed br0 from interfaces file
2022-10-19 11:11:09 <ikke> 👍
2022-10-19 11:24:39 <ncopa> Im starting on build-edge-ppc64le
2022-10-19 11:27:16 <ikke> ncopa: we should be able to gradually replace packages on edge with the new key, right?
2022-10-19 11:27:49 <ncopa> I believe so yes
2022-10-19 11:28:12 <ncopa> but we still have the old key there?
2022-10-19 11:28:33 <ncopa> I mean the old pub key is still usable?
2022-10-19 11:28:38 <ikke> yes, the public key is
2022-10-19 11:28:48 <ikke> we cannot sign any new packages
2022-10-19 11:28:57 <ncopa> I mean the old old
2022-10-19 11:29:05 <ncopa> the one we have the privilege key to
2022-10-19 11:29:13 <ncopa> priv*
2022-10-19 11:29:18 <ikke> ah, yes
2022-10-19 11:29:20 <ikke> that should still work
2022-10-19 11:29:29 <ncopa> so we can use that for stable branches
2022-10-19 11:29:31 <ikke> I did not remove the old pub keys yet
2022-10-19 11:29:34 <ncopa> and for edge initially
2022-10-19 11:29:44 <ncopa> then it should work without anyone noticing
2022-10-19 11:30:07 <ncopa> maybe we can use that for 3.16
2022-10-19 11:30:32 <ikke> yes
2022-10-19 11:30:50 <ncopa> ok, lets do that
2022-10-19 11:31:00 <ikke> afaik we did not remove any public keys
2022-10-19 12:07:19 <ncopa> im syncing the packages to build-edge-ppc64le now
2022-10-19 12:21:43 <clandmeter> so you have a plan to overcome the builder loss?
2022-10-19 12:21:57 <ncopa> yeah
2022-10-19 12:22:09 <ncopa> just a lot of manual work to set up new builders
2022-10-19 12:26:00 <ikke> clandmeter: the old private keys are still valid
2022-10-19 12:29:32 <clandmeter> aha
2022-10-19 12:29:49 <clandmeter> didnt get an reply from ibm
2022-10-19 12:29:52 <ncopa> so we use the old old key for 3.16 and edge
2022-10-19 12:30:20 <clandmeter> which release did we sign with the new key?
2022-10-19 12:31:05 <ncopa> possibly 3.16
2022-10-19 12:31:14 <ncopa> I don't remember
2022-10-19 12:36:50 <ikke> 3.15 we started 
2022-10-19 12:43:28 <clandmeter> so we need to generate a new 4k key i guess?
2022-10-19 12:44:14 <ikke> Yes
2022-10-19 15:07:32 <mps> I need help about triggers when install/upgrade kernel. how to add it for linux-asahi?
2022-10-19 15:08:30 <mps> on install/upgrade it should run script which update second stage bootloader (m1n1+dtbs+u-boot) before reboot to new kernel
2022-10-19 15:09:20 <mps> there is script for this in u-boot-asahi pkg
2022-10-19 15:12:18 <mps> add post-upgrade script which will run needed script from u-boot-asahi pkg?
2022-10-19 15:24:04 <minimal> mps: woulsn't that be a trigger for the bootloader package (u-boot?) on /boot directory? i.e. same as grub has
2022-10-19 15:25:29 <mps> minimal: not sure, this should be run on every linux-asahi upgrade. and if there is grub it should be also triggered
2022-10-19 15:26:06 <mps> but grub is already triggered on upgrade
2022-10-19 15:26:49 <mps> script from u-boot-asahi is one extra step needed
2022-10-19 15:30:28 <minimal> mps: grub is trigger because the grub package declares a trigger on /boot, so if you added a similar trigger on /boot to u-boot-asahi would that not solve your problem?
2022-10-19 15:30:43 <minimal> #s/is trigger/is triggered/
2022-10-19 15:31:02 <ikke> lol, circumventing the bot :D
2022-10-19 15:31:48 <minimal> brain and fingers working at different speeds lol
2022-10-19 15:31:59 <ikke> oh yes
2022-10-19 15:32:04 <ikke> very familiar
2022-10-19 15:32:07 <mps> minimal: idk, maybe. but not sure would this be proper solution
2022-10-19 15:33:08 <mps> so, I think linux-asahi is where trigger should be, but not sure
2022-10-19 15:33:47 <minimal> mps: why not a proper solution? you want the bootloader to react to changes in kernel and/or initramfs?
2022-10-19 15:34:08 <ikke> Should linux-asahi know about u-boot-asahi? 
2022-10-19 15:35:26 <mps> ikke: yes, it will depend on u-boot-asahi
2022-10-19 15:36:57 <mps> minimal: problem is u-boot-asahi is not real bootloader but something between 'fake bios' and grub
2022-10-19 15:37:30 <mps> m1n1+u-boot just sets ENV for grub
2022-10-19 15:37:45 <mps> and load DTBS
2022-10-19 15:37:56 <ikke> mps: so then you would just add a post-(install|upgrade) install script?
2022-10-19 15:38:30 <mps> I think so, but wanted to ask more experienced ones
2022-10-19 15:38:59 <ikke> The reason why triggers are used is for decoupling
2022-10-19 15:39:16 <ikke> grub needs to know about kernel upgrades without linux-lts being involved
2022-10-19 15:39:25 <ikke> or linux-virt
2022-10-19 15:39:32 <mps> aha, lets build and test
2022-10-19 15:40:40 <mps> thanks both
2022-10-19 15:40:48 <minimal> and likewise on x86/x86_64 another reason for the decoupling is that the bootloader could be grub, or syslinux, or something else
2022-10-19 15:42:30 <ikke> yes
2022-10-19 15:42:47 <mps> oh yes, this give me one idea to check
2022-10-19 16:29:33 <ncopa> https://build.alpinelinux.org/ "PLEASE MAKE YOUR MQTT READ ONLY!! psy0rz"
2022-10-19 16:33:56 <ncopa> build-edge-ppc64le is up
2022-10-19 16:53:39 <ncopa> I have changed the mosquitto config on msg.a.o. due to: https://paste.pics/3438debecafc6f0b87c8bb5ede8bc189
2022-10-19 16:55:05 <ncopa> I changed "topic build/#" to "topic read build/#" under "anonymous topic access" in /etc/mosquitto/aclfile
2022-10-19 16:55:57 <ncopa> I believe this means nothing can publish anything
2022-10-19 16:56:21 <ncopa> which means builders will not pick up git pushes anymore
2022-10-19 16:56:56 <ncopa> well, I think they will pick up git pushes, but we can not monitor it on build.a.o til we have fixed this
2022-10-19 17:02:16 <ncopa> i also upgraded alpine-msg to v3.16 while at it
2022-10-19 17:12:13 <ncopa> new mosquitto changed allow_anonymous=true to false as default
2022-10-19 17:13:07 <ikke> That might break some things 
2022-10-19 17:13:14 <ncopa> oh yes
2022-10-19 17:13:19 <ncopa> it broke everything
2022-10-19 17:14:29 <ncopa> it now allows anonymous but builders can no longer write their status
2022-10-19 17:15:02 <ncopa> and im tired
2022-10-19 17:15:13 <ncopa> im gonna continue tomorrow
2022-10-19 17:17:18 <ikke> Thanks 
2022-10-19 17:17:49 <ncopa> i disabled the readonly mode. it is still possible for everyone to publish on msg.a.o
2022-10-19 17:17:58 <ncopa> we need figure out how to solve it
2022-10-19 17:18:10 <ncopa> we can use password everywhere
2022-10-19 17:18:21 <ncopa> or we can use certificates
2022-10-19 17:18:33 <ncopa> or we can move it to private network only
2022-10-19 17:19:05 <ncopa> not sure what is the easiest thing to do
2022-10-19 18:06:23 <psykose> i did the same meme like two months ago xD
2022-10-19 18:06:37 <psykose> and there's a 4 year old issue about it
2022-10-19 18:06:46 <psykose> but ultimately there's not much it does, just triggers the builders
2022-10-19 18:07:51 <psykose> ikke: seems a test failed due to g+s ownership on homedir on ppc64le
2022-10-19 18:07:55 <psykose> probably have to set that too again
2022-10-19 18:27:20 <clandmeter> didnt we already set a password for the builders?
2022-10-19 19:46:35 <ikke> psykose: probably because the container was recreated
2022-10-19 19:46:40 <ikke> I guess it's part of the template
2022-10-19 19:48:01 <ikke> psykose: removed the g+s
2022-10-19 20:16:15 <psykose> thankies
2022-10-19 20:16:25 <psykose> it's also part of the busybox default for adduser
2022-10-19 20:16:27 <psykose> no idea why
2022-10-19 20:16:34 <ikke> ah ok
2022-10-19 20:16:41 <ikke> maybe that's what added it then
2022-10-19 20:16:52 <ikke> as the buildozer user is created after the container is created
2022-10-19 20:41:04 <ikke> clandmeter: should I reply to Michael that we have a solution?
2022-10-20 10:13:12 <mps> ncopa: could we merge some pkgs in testing now
2022-10-20 11:22:21 <ncopa> "please fix" https://build.alpinelinux.org/ 
2022-10-20 11:23:55 <ikke> childish
2022-10-20 11:24:12 <ikke> it's hacked because you can send a message
2022-10-20 11:24:15 <ikke> whooptidoo
2022-10-20 11:25:06 <ikke> They're also trying all kinds of redirection apparently
2022-10-20 11:25:16 <ikke> shell redirection
2022-10-20 11:26:40 <ncopa> i wonder if we should make it write only from private net and usetunnels
2022-10-20 11:27:00 <ncopa> or use a password for builders
2022-10-20 11:48:13 <ikke> ncopa: how would you limit write access to the vpn only?
2022-10-20 11:58:53 <ncopa> i wonder if it possible to do so  ased on the client ip addr?
2022-10-20 11:59:16 <ikke> Could not find any config in skimming over the config file
2022-10-20 11:59:45 <ncopa> what other options do we have?
2022-10-20 11:59:58 <ikke> making sure writers authenticate
2022-10-20 12:00:07 <ncopa> username and password?
2022-10-20 12:00:14 <ikke> That's the simplest
2022-10-20 12:05:24 <ncopa> ok. will look into it
2022-10-20 12:05:34 <ncopa> will just try bootstrap the 3.17 ppc64le first
2022-10-20 12:05:37 <ikke> yes
2022-10-20 12:06:15 <ncopa> clandmeter: apparently the builders does not connect using user/password to msg.a.o
2022-10-20 12:07:25 <ikke> ncopa: I recall being stranded at providing credentials to mqtt-exec
2022-10-20 12:07:51 <ikke> at least, we didn't want to have the password in the cmdline
2022-10-20 12:10:48 <ikke> But I think we found out the other day that it's possible to provide the password in the environment
2022-10-20 12:11:32 <ikke> https://github.com/ncopa/mqtt-exec/blob/master/mqtt-exec.c#L301
2022-10-20 12:11:44 <ikke> ncopa: Let me handle that, I have some time
2022-10-20 12:24:09 <ncopa> ok. i'll continue with the 3.17 builders then
2022-10-20 12:27:32 <ncopa> i updated the alpine-msg server to alpine 3.16 yesterday
2022-10-20 12:27:40 <ikke> yes, I read that
2022-10-20 12:27:43 <ikke> thanks
2022-10-21 14:17:09 <ncopa> ok looks like i finally managed to get the mqtt setup working with auth
2022-10-21 15:16:04 <ncopa> i think all builders should now use mqtt auth
2022-10-21 15:16:11 <ncopa> seems like its all working now
2022-10-21 15:17:09 <ncopa> all ppc64le builders should be back as well
2022-10-25 09:06:07 <ncopa> mps: I have a question about cd36c615499ac448623e0f8e7d3604c69d9a67de
2022-10-25 09:06:32 <ncopa> "add libax25-dev and libcap-dev to makedepends, needed for this version"
2022-10-25 09:06:43 <ncopa> what is libax25 needed for in iproute2?
2022-10-25 09:07:07 <ncopa> it seems like upstream libax25 is gone, so I wonder if we can remove it?
2022-10-25 09:07:28 <ncopa> iproute2 builds just fine without it, and iproute2 does not seem to link against it
2022-10-25 09:12:45 <ikke> Apparently to do with hamradio 
2022-10-25 09:13:11 <ncopa> i see that there are support for ax25 in iproute2. i suspect the headers was needed at some point
2022-10-25 09:19:14 <ncopa> the pkgver is also incorrect
2022-10-25 09:19:25 <ncopa> lets move it to testing
2022-10-25 09:40:29 <mps> ncopa: libax25 was needed at that time
2022-10-25 09:40:38 <ncopa> i figured
2022-10-25 09:40:48 <ncopa> but only for headers right?
2022-10-25 09:40:57 <mps> iirc yes
2022-10-25 09:41:12 <ncopa> ok good. thanks
2022-10-25 09:41:43 <ncopa> upstream libax25 seems to be gone, so I moved it back to testing to be decided what to do with it
2022-10-25 09:42:01 <ncopa> there are only 2 packages using it and they are both in testing
2022-10-25 09:42:21 <ncopa> i think we either fix those and move them all to community, or we delete them
2022-10-25 09:42:59 <mps> I know, but I think it is most reasonable to keep it in testing
2022-10-25 09:44:03 <mps> but if there reasons to delete them I'm not against
2022-10-25 09:53:40 <ncopa> why do you think its more reasonable to keep it in testing rather than community?
2022-10-25 09:54:19 <ncopa> in either case, the upstream source url needs to be fixed one way or the other
2022-10-25 09:57:11 <mps> because these packages do not have any changes upstream for some (long?) time
2022-10-25 09:57:57 <mps> if you want them in community maybe better option is to use debian as source for them
2022-10-25 10:09:57 <mps> https://salsa.debian.org/debian-hamradio-team/libax25
2022-10-25 10:16:49 <ncopa> mps: do you mind take care of that, since you are the package maintainer?
2022-10-25 10:17:40 <ncopa> also pkgver looks wrong. it looks like it really is 0.0.12_rc5
2022-10-25 10:18:09 <ncopa> thanks!
2022-10-25 10:26:15 <mps> I will look after 3.17 release, does it make sense to have these pkgs in alpine at all. till now no one provide any feedback
2022-10-25 10:26:46 <ikke> That counts for most packages in aports
2022-10-25 10:28:02 <mps> right, and sometimes I want at least counters of downloaded pkgs
2022-10-25 17:44:20 <panekj> 🤔
2022-10-25 17:45:09 <ikke> ?
2022-10-25 17:45:26 <ikke> I
2022-10-25 17:45:33 <ikke> I'm cleaning up internal dns records
2022-10-25 17:45:54 <panekj> nothing, someone complains in alpine-linux channel (on another network) that dl-{4,5} is gone
2022-10-25 17:46:40 <panekj> and it returns 404
2022-10-25 17:46:42 <ikke> oh, I've updated the DNS records to a new host, didn't check if the host actually responded to it :P
2022-10-25 17:48:38 <panekj> perhaps need to add healtcheck for HTTP 2xx :P
2022-10-25 17:49:22 <ikke> We usually have, but apparently not for those
2022-10-25 17:49:48 <panekj> best strategy to find something that is still used: remove and see who complains :D
2022-10-25 17:49:56 <ikke> exactly :D
2022-10-25 17:51:35 <ikke> hmm, interesting, we do have checks for the status for those mirrors, but apparently it gets a 200
2022-10-25 17:54:09 <panekj> i get 404 in browser
2022-10-25 17:54:17 <panekj> (HTTP code)
2022-10-25 17:55:21 <ikke> Yes, somehow it resolves to an internal (vpn) address
2022-10-25 17:55:26 <ikke> trying to figure out where that comes from
2022-10-25 18:02:15 <ikke> mirrors are fixed
2022-10-25 18:06:29 <panekj> yup
2022-10-25 18:06:58 <ikke> needed to add the hostnames to the webserver (traefik)
2022-10-25 18:11:13 <panekj> now we need monitoring checks for records returning proper IPs :P
2022-10-25 18:15:32 <ikke> ok, found where
2022-10-25 18:23:14 <ikke> fix one thing, break another :P
2022-10-25 18:23:22 <ikke> But at least we get alerts 
2022-10-26 19:45:17 <ikke> upgraded netbox to the latest minor version
2022-10-26 21:12:18 <ikke> hmm
2022-10-26 21:17:46 <psykose> kill one of the dotnets
2022-10-26 21:18:06 <ikke> I nuked .cargo go .cache/yarn .cache/go-build on all builders :P
2022-10-26 21:18:14 <psykose> i'd recommend not doing a dotnet too :)
2022-10-26 21:18:17 <psykose> and they are probably frozen
2022-10-27 10:24:51 <ikke> load >100
2022-10-27 14:12:59 <psykose> i see openjdk is still having fun getting stuck
2022-10-27 14:57:50 <ncopa> i killed java and rebooted build-edge-aarch64 and build-edge-ppc64le
2022-10-27 15:02:30 <ikke> ncopa: storage on dl-master has been increased
2022-10-27 15:03:10 <ncopa> awesome. thanks!
2022-10-27 16:32:16 <ikke> We have something weird in our internal dns. I sometimes get nonsense AAAA answers for hosts (in *.alpin.pw)
2022-10-27 16:32:23 <ikke> PING build-3-16-x86.nld8.alpin.pw(cc44:6f69:8255:0:609a:98b4:fe7f:0 
2022-10-27 16:32:36 <ikke> that cc44 address is nonsense
2022-10-27 16:32:51 <ikke> And there are no AAAA records
2022-10-28 07:09:11 <mps> clandmeter: good morning. I think we should enable static libxml2 (you are maintainer). it is needed to build crystal static version for bootstraping new releases
2022-10-28 07:11:24 <mps> it was disabled in dc015081c0d12a8cbcccac836c67d432e1f49ca3
2022-10-28 07:34:35 <ikke> mps: clandmeter is on vacation 
2022-10-28 07:39:30 <mps> ikke: ah, didn't know
2022-10-28 07:41:17 <mps> I think we should add this in libxml2, fix is simple https://tpaste.us/Z4v6
2022-10-28 07:55:18 <ncopa> mps: I'd also like to have a $pkgname-static subpackage
2022-10-28 07:55:24 <ncopa> otherwise all good
2022-10-28 08:03:20 <mps> I would left decision about -static subpkg to maintainer
2022-10-28 08:04:40 <mps> for now I think we should simply add --enable-static, for upgrading crystal to 1.6.1 to have it in 3.17 alpine
2022-10-28 08:05:14 <ncopa> argh! whats up with ppc64le-ci
2022-10-28 08:05:21 <ncopa> its the only ppc64le machine we have now
2022-10-28 08:05:30 <ncopa> i lost connectivity
2022-10-28 08:06:18 <ncopa> mps: i disagree. i'd still prefer have $pkgname-static subpackage. that is what we do for other packages
2022-10-28 08:06:46 <mps> ok, who will make it?
2022-10-28 08:06:50 <ncopa> i can
2022-10-28 08:07:11 <mps> nice, please do
2022-10-28 08:07:17 <ncopa> so in case maintainer (clandmeter) gets upset he don't blame you ;)
2022-10-28 08:07:30 <mps> hehe, right
2022-10-28 08:28:50 <ikke> Back apparently 
2022-10-28 08:29:06 <ncopa> yeah
2022-10-28 08:30:00 <mps> https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usbutils.git/commit/?id=79b796f945ea7d5c2b0e2a74f9b8819cb7948680
2022-10-28 08:30:22 <mps> usbutils release announce 6 days ago
2022-10-28 08:32:36 <mps> also would be nice to have it in 3.17
2022-10-28 08:34:28 <mps> when is planned release? at end of november?
2022-10-28 08:44:01 <ncopa> beginning or mid november
2022-10-28 08:44:14 <ncopa> but it depends on how many issue we bump into
2022-10-28 08:44:23 <ncopa> so realistically, mid nov
2022-10-28 08:44:51 <mps> aha, ok. so we have still some time to upgrade some pkgs and fix some other
2022-10-28 08:44:56 <ncopa> i think we can bump usbutils
2022-10-28 08:45:09 <ncopa> usbutils is low risk, isnt it?
2022-10-28 08:45:17 <mps> usually yes
2022-10-28 08:45:25 <ncopa> i mean, upgrade python is big risk, lots of packages to rebuild etc
2022-10-28 08:45:29 <ncopa> so its a no-go
2022-10-28 08:45:44 <ncopa> usbutils is a small tool, few things can break
2022-10-28 08:45:48 <ncopa> so i'd say push it
2022-10-28 08:46:07 <mps> also I think it is not good time upgrade big pkgs like python
2022-10-28 08:47:19 <mps> I'm working on crystal right now, will look later at usbutils if someone don't push it in meantime
2022-10-28 10:26:08 <ikke> ncopa: fyi, this evening we're going to upgrade the ppc64le ci host
2022-10-28 10:26:46 <ncopa> ok. thanks
2022-10-28 15:21:25 <Ramereth> ikke: good morning
2022-10-28 15:21:29 <ikke> Ramereth: o/
2022-10-28 15:22:42 <Ramereth> just getting settled in my office and getting prepped for the upgrade
2022-10-28 15:23:22 <ikke> nod
2022-10-28 15:27:20 <Ramereth> I need to take care of a minor fire first, please let me deal with that
2022-10-28 15:28:38 <ikke> sure, np
2022-10-28 15:28:45 <ikke> I'm stopping the builders in the mean time
2022-10-28 15:34:18 <ikke> Ramereth: the host is idle now
2022-10-28 15:37:30 <Ramereth> excellent. I'm almost ready on my end too
2022-10-28 15:41:33 <Ramereth> ikke: alright, I'm going to do the clean reboot first. Ready for me to do that?
2022-10-28 15:41:38 <ikke> yes
2022-10-28 15:44:28 <Ramereth> and it's back
2022-10-28 15:44:32 <ikke> good
2022-10-28 15:44:54 <Ramereth> alright, I'm going to follow your instructions to do the upgrade next
2022-10-28 15:45:05 <ikke> ack
2022-10-28 15:47:01 <Ramereth> upgrading now
2022-10-28 15:48:30 <Ramereth> alright, going to reboot now if that's oK?
2022-10-28 15:48:34 <ikke> Fine with me
2022-10-28 15:50:55 <Ramereth> and it's back online
2022-10-28 15:50:59 <ikke> nice
2022-10-28 15:51:13 <Ramereth> shall I go ahead and do the firmware upgrade now? I'll need to power down the system
2022-10-28 15:51:21 <ikke> Yes, please do
2022-10-28 15:51:27 <Ramereth> alright, starting that now
2022-10-28 15:56:03 <Ramereth> doing the BMC update right now
2022-10-28 15:59:27 <Ramereth> bmc update done. Now doing the system firmware
2022-10-28 16:04:40 <Ramereth> firmware update done. Powering up system
2022-10-28 16:05:55 <ikke> drum roll
2022-10-28 16:06:10 <Ramereth> it's writing the firmware into the cpus right now (so it has to reboot a few times)
2022-10-28 16:08:19 <Ramereth> and..... it's back!
2022-10-28 16:08:35 <ikke> Wonderful
2022-10-28 16:08:42 <Ramereth> confirm it all looks good?
2022-10-28 16:08:51 <ikke> Can ssh into it
2022-10-28 16:09:32 <ikke> Looks good sofar
2022-10-28 16:09:44 <Ramereth> excellent
2022-10-28 16:33:21 <ikke> for some reason, dns inside docker containers is not working yet, troubleshooting
2022-10-28 16:50:41 <ikke> Can't seem to figure out why :(
2022-10-28 16:53:44 <ncopa> does dns work outside containers?
2022-10-28 16:54:03 <ikke> yes
2022-10-28 16:54:53 <ikke> dial tcp: lookup gitlab.alpinelinux.org on 127.0.0.11:53: read udp 127.0.0.1:36793->127.0.0.11:53: i/o timeout
2022-10-28 16:55:14 <ikke> I already disabled all iptables rules and restarted the docker daemon
2022-10-28 16:55:26 <ikke> I already explicitly set dns servers in /etc/docker/daemon.json
2022-10-28 16:56:52 <ikke> evel=warning msg="[resolver] connect failed: dial udp 8.8.8.8:53: connect: network is unreachable"
2022-10-28 17:00:30 <ncopa> do you have default route in container?
2022-10-28 17:00:55 <ikke> default via 172.18.0.1 dev eth0
2022-10-28 17:01:19 <ncopa> can you ping 8.8.8.8?
2022-10-28 17:01:27 <ikke> no
2022-10-28 17:01:35 <ncopa> can you ping 172.18.0.1?
2022-10-28 17:01:54 <ikke> hmm, neither
2022-10-28 17:02:17 <ncopa> could it be a mixup of iptables legacy and iptables-nftables?
2022-10-28 17:02:28 <ncopa> have been battling that in k0s recently
2022-10-28 17:02:41 <ncopa> does iptables-save give any hints?
2022-10-28 17:02:51 <ncopa> if ip forwarding enabled?
2022-10-28 17:03:12 <ikke> ncopa: possibly
2022-10-28 17:03:16 <ikke> bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
2022-10-28 17:03:19 <ikke> I saw this in dmesg
2022-10-28 17:03:29 <ncopa> is the installed kernel the running kernel? (or was there an apk upgrade that pulled in newer kernel)
2022-10-28 17:03:33 <ikke> yes
2022-10-28 17:03:39 <ikke> that's the first thing I checked :)
2022-10-28 17:03:42 <ncopa> maybe modprobe br_netfilter?
2022-10-28 17:04:01 <ikke> doesn't help
2022-10-28 17:04:45 <ncopa> does networking work from lxc containers?
2022-10-28 17:04:50 <ikke> let me check
2022-10-28 17:04:55 <ikke> I didn't start them yet
2022-10-28 17:05:23 <ncopa> oh, there was an up conflict iirc in /etc/network/interfaces. I dont remember if I fixed it
2022-10-28 17:05:34 <ncopa> i think i did, but might be good to verify
2022-10-28 17:06:24 <ikke> I don't see it anymore
2022-10-28 17:06:47 <ikke> neither in /etc/network/interfaces as in ip -br a
2022-10-28 17:07:38 <ikke> cannot ping lxc containers
2022-10-28 17:09:04 <ikke> ncopa: it does look similar
2022-10-28 17:09:06 <ikke> as with the conflict
2022-10-28 17:12:57 <ikke> cannot ping over dmvpn either
2022-10-28 17:14:22 <ikke> ncopa: see these routes: https://tpaste.us/EBle 
2022-10-28 17:14:26 <ikke> never seen these throw routes before
2022-10-28 17:26:31 <ikke> ncopa: seems somethign strange going on
2022-10-28 17:26:37 <ikke> All these throw routes appear
2022-10-28 17:29:15 <ikke> I've managed to get the lxc routing working
2022-10-28 17:33:00 <ikke> ncopa: I've manually fixed it, but there is something really strange going on
2022-10-28 17:33:36 <ikke> I needed to manually add a route for each network, and the first time I do it, a 'throw' route appears. I delete it, add the same route again, and the correct route appears
2022-10-28 17:40:24 <Ramereth> just saw your email
2022-10-28 17:41:12 <ikke> I've never seen this before, so a bit puzzled
2022-10-28 17:42:47 <ikke> I did get everything working again for now
2022-10-28 17:43:07 <ikke> But the next boot or maybe after some network change, things might break again
2022-10-28 17:43:25 <Ramereth> where are the throw routes coming from?
2022-10-28 17:43:30 <ikke> I have no idea
2022-10-28 17:43:37 <ikke> It appears the kernel inserts them
2022-10-28 17:47:54 <Ramereth> it doesn't ring a bell to me
2022-10-28 17:48:01 <ikke> ok, thanks
2022-10-28 18:15:57 <ikke> This is quite anoying, not sure where this comes from and nothing to find about it
2022-10-30 03:34:20 <valerius> s/then/than
2022-10-30 06:46:08 <ikke> Heh
2022-10-30 15:51:48 <ikke> ..
2022-10-30 16:39:05 <psykose> did anyone order some icmp ping loss
2022-10-30 16:39:22 <ikke> Certainly not me
2022-10-30 16:39:23 <psykose> ikke: also there is still java-jna on 3.17-ppc stuck
2022-10-30 16:40:33 <ikke> stopped
2022-10-30 16:41:55 <ikke> I also freed up space on nld8. It turned out the 2.6 builder apparently did not clean up after each build, so ~50G of src/ and pkg/ dirs
2022-10-30 22:01:17 <ikke> ddevault: moved over listserv to the new host. You should be able to login via gbr-app-1.alpinelinux.org -p 22022
2022-10-31 05:52:03 <ikke> apparently the packetloss was an issue with linode
2022-10-31 07:48:02 <ddevault> ikke: ack
2022-10-31 19:43:26 <ikke> Ramereth: ping
2022-10-31 20:15:27 <Ramereth> pong
2022-10-31 20:15:30 <Ramereth> ikke: pong
2022-10-31 20:17:18 <ikke> going to try to install the 3.14 linux-lts package and reboot the machine
2022-10-31 20:17:36 <Ramereth> +1 I'm around if you need me. Do you need me to watch the serial console?
2022-10-31 20:17:58 <ikke> I'll just first try to reboot it. If it doesn't seem to come back, I can ping you
2022-10-31 20:18:23 <Ramereth> sounds good
2022-10-31 20:27:15 <ikke> Ramereth: ERROR: Failed to create boot/vmlinuz-lts: No space left on device
2022-10-31 20:29:31 <ikke> I moved the existing vmlinuz-lts out of there
2022-10-31 20:29:36 <Ramereth> ok
2022-10-31 20:37:37 <ikke> ok, it came back, same issue with older kernels
2022-10-31 20:38:53 <Ramereth> hrmph
2022-10-31 20:39:45 <ikke> So does not appear to be a kernel issue
2022-10-31 20:48:40 <ikke> Ramereth: back to the drawing board, thanks for being standby'
2022-10-31 20:55:47 <Ramereth> ikke: np