2022-03-01 05:20:42 <ikke> clandmeter: dmvpn2 is already hosted on linode
2022-03-01 05:20:51 <ikke> clandmeter: I guess the idea was to have them separate
2022-03-01 07:47:41 <clandmeter> ah ok, so that was the reason :)
2022-03-01 07:48:14 <ikke> I don't know, I just inferred it 
2022-03-01 07:49:55 <clandmeter> does it need to run on a dedicated host with dedicated ip?
2022-03-01 07:50:56 <ikke> For a hub that would make sense 
2022-03-01 07:51:36 <ikke> I suppose we could forward the ports to a container if necessary 
2022-03-01 08:43:10 <clandmeter> just feels like a waste of server resources using it like this dedicated
2022-03-01 11:20:34 <ikke> clandmeter: I suppose you wanted to move it to nld3 then?
2022-03-01 11:20:47 <clandmeter> could be, or gbr1
2022-03-01 11:21:07 <clandmeter> or i could even move it here at home
2022-03-01 11:23:45 <ikke> clandmeter: you have access to the rapidswitch backend, right?
2022-03-01 11:23:57 <clandmeter> yes?
2022-03-01 11:24:12 <ikke> gbr1 needs to be upgraded
2022-03-01 11:24:20 <clandmeter> oh ok
2022-03-01 11:24:25 <clandmeter> i domt have it now
2022-03-01 11:24:28 <clandmeter> you need to request it
2022-03-01 11:24:38 <clandmeter> they will connect some box to the serial port i think
2022-03-01 11:24:41 <ikke> ah ok
2022-03-01 11:25:04 <ikke> Can we request that then?
2022-03-01 11:25:13 <ikke> I can do the upgrade once it's arranged
2022-03-01 11:27:17 <clandmeter> ok give me some time, im in the middle of something 
2022-03-01 11:29:51 <ikke> np, there is no rush
2022-03-01 14:03:22 <clandmeter> ikke: i send the details via email
2022-03-01 14:03:50 <ikke> Thanks 
2022-03-02 14:16:04 <clandmeter> gbr1 is rebooting, i hope...
2022-03-02 14:18:48 <clandmeter> looks like it doesnt reboot...
2022-03-02 15:19:06 <ikke> \o/
2022-03-02 15:20:07 <psykose> victory
2022-03-02 15:21:24 <clandmeter> yup
2022-03-02 15:21:28 <clandmeter> old sshd config
2022-03-02 15:22:25 <clandmeter> i wonder how alpine was installed on this machine
2022-03-03 15:40:58 <clandmeter> ikke: ever had issues with docker build not able to fetch apkindex?
2022-03-03 15:44:24 <clandmeter> this is on linode
2022-03-03 15:47:54 <ikke> Not that I recall 
2022-03-03 15:50:49 <clandmeter> looks like its not working, need to switch to non cdn mirror
2022-03-03 15:53:18 <ikke> from which server?
2022-03-03 15:55:00 <clandmeter> gbr2-dev1
2022-03-03 15:55:04 <clandmeter> hmm
2022-03-03 15:55:20 <ikke> ipv6 maybe?
2022-03-03 15:55:21 <clandmeter> seems my lua script also does not want to download anything from cdn
2022-03-03 15:55:32 <clandmeter> uhm possible
2022-03-03 15:56:06 <clandmeter> looks like the container has ipv6
2022-03-03 16:03:51 <clandmeter>  yep its ipv6 issue
2022-03-03 16:29:41 <ikke> clandmeter: did you fix it?
2022-03-03 16:29:50 <clandmeter> nope
2022-03-03 16:29:54 <clandmeter> no idea how
2022-03-03 16:31:48 <ikke> I'll take a look at it in a bit
2022-03-04 08:52:48 <clandmeter> ikke: i guess you didnt fix it?
2022-03-04 08:56:36 <ikke> Can you check if ipv6 forwarding is enabled? 
2022-03-04 11:01:21 <clandmeter> ikke: looks enabled
2022-03-04 13:08:09 <ikke> clandmeter: hmm, apparently we're using the wrong ipv6 subnet?
2022-03-04 13:08:31 <ikke> yup
2022-03-04 13:10:30 <ikke> the wrong subnet has been added to /etc/docker/daemon.json
2022-03-04 13:11:13 <ikke> For traefik, I've added a separate network, which has a subnet in the correct ipv6 prefix, so that works
2022-03-04 13:12:20 <ikke> I've fixed it, but we need to restart the daemon to make it active
2022-03-04 13:34:54 <ikke> clandmeter: now it's just the firewall left
2022-03-04 13:46:29 <ikke> fixed
2022-03-04 16:28:24 <clandmeter> thanks!
2022-03-05 17:45:51 <mps> psykose: there were one request for driver in kernel few days ago beside your one. I can't find this because it is not assigned to me and forgot 'keyword'
2022-03-05 17:46:13 <mps> could you help me to find it
2022-03-05 17:46:14 <psykose> https://gitlab.alpinelinux.org/alpine/aports/-/issues/13571 this?
2022-03-05 17:46:33 <mps> yes, thanks
2022-03-05 17:46:36 <psykose> :) anytime
2022-03-05 17:46:53 <mps> :)
2022-03-05 23:03:45 <psykose> mps: mm, i thinks `# CONFIG_CGROUP_BPF is not set` was also needed still.. sadly the requirements are not fully mapped out
2022-03-05 23:04:25 <psykose> (i notice that is already on x86_64, where everything works fine, and it seems related)
2022-03-05 23:04:31 <psykose> i think it will test the changes myself then tell you
2022-03-05 23:21:41 <mps> psykose: yes, also I noticed this but after push
2022-03-05 23:21:54 <psykose> i started a built, will probably take all night on my rpi4 :)
2022-03-05 23:22:01 <psykose> tomorrow i will tell you if it is enough
2022-03-05 23:22:50 <mps> I hope it will, but I'm going to bed now, will add this tomorrow if I find time
2022-03-05 23:22:54 <psykose> mhm
2022-03-05 23:22:55 <psykose> me too
2022-03-05 23:22:56 <psykose> sleep well
2022-03-05 23:23:08 <mps> thanks, you too
2022-03-06 08:52:33 <rails> 
ACTION waves bye to old client

2022-03-06 12:23:36 <mps> psykose: ping
2022-03-06 16:39:55 <psykose> mps: yep, just that option now makes containers work
2022-03-06 16:39:56 <psykose> :)
2022-03-06 20:30:09 <mps> psykose: ok, thanks for confirmation. Do you need it right now or for next upgrade
2022-03-06 20:30:19 <psykose> next upgrade is fine, i have my rebuilt version already
2022-03-06 20:30:21 <psykose> thanks!
2022-03-06 20:30:42 <mps> ok
2022-03-06 20:31:13 <mps> just curios, how long it takes to build on RPi4
2022-03-06 20:33:31 <psykose> lets see
2022-03-06 20:33:42 <mps> approx is ok
2022-03-06 20:33:54 <psykose> started at.. 00:14
2022-03-06 20:34:04 <psykose> ended at.. 05:12
2022-03-06 20:34:05 <psykose> :)
2022-03-06 20:34:13 <mps> huh
2022-03-06 20:34:16 <psykose> 5 hours
2022-03-06 20:34:34 <mps> is the filesystem on mmc or usb ssd
2022-03-06 20:34:38 <psykose> https://img.ayaya.dev/RgmEQBqKfp7j.png
2022-03-06 20:34:42 <psykose> usb 3.0 
2022-03-06 20:34:46 <psykose> or well, 3.1 or something
2022-03-06 20:34:48 <psykose> pretty fast
2022-03-06 20:34:53 <psykose> ~250MB/s seq
2022-03-06 20:35:43 <mps> hm, I build kernels in less than hour on rk3399 chromebook with usb-c 3.1 ssd disk
2022-03-06 20:36:38 <mps> so RPi4 is slow for some reason
2022-03-06 20:37:19 <psykose> it is
2022-03-07 16:25:46 <clandmeter> interesting mirror request
2022-03-07 16:40:08 <ikke> :)
2022-03-07 16:41:40 <ikke> I think they are asking for instructions
2022-03-08 00:46:59 <psykose> the fcix one should probably be added too
2022-03-08 02:46:42 <rails> ooh?
2022-03-08 02:47:31 <psykose> http://mirror.fcix.net/alpine/
2022-03-08 07:21:30 <clandmeter> psykose: can we add you to the mirrors project? 
2022-03-08 07:21:50 <psykose> sure, ikke would have to show me how to add a mirror to the correct place and do the things
2022-03-08 07:22:29 <psykose> (or you) :)
2022-03-08 07:22:45 <clandmeter> no problem
2022-03-08 07:22:52 <clandmeter> i think its kind of obvious :)
2022-03-08 07:23:33 <psykose> sure, just want to make sure it's correct :p
2022-03-08 07:23:39 <clandmeter> ah you are alraedy 
2022-03-08 07:23:47 <psykose> i think infra is all of them
2022-03-08 07:23:51 <psykose> just not sure where to add what
2022-03-08 07:23:53 <clandmeter> due to your infra membership
2022-03-08 07:24:14 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/mirrors
2022-03-08 07:24:28 <clandmeter> there is a yml file which you need to update
2022-03-08 07:24:32 <psykose> ah
2022-03-08 07:24:34 <clandmeter> the rest should be automatic
2022-03-08 07:24:43 <clandmeter> should™️
2022-03-08 07:25:20 <clandmeter> its kind of hidden now, but i think we could also move this back to public
2022-03-08 07:25:20 <psykose> is there an ordering or is bottom of file fine
2022-03-08 07:25:38 <clandmeter> and let sysops add MR's 
2022-03-08 07:25:45 <clandmeter> just add to the bottom
2022-03-08 07:27:10 <clandmeter> the only thing that is kind of private is the mailing address, but most mirrors have a general mirror address anyways
2022-03-08 07:28:45 <psykose> well, i made an mr for that one
2022-03-08 07:28:50 <psykose> i can't actually merge anything into it :)
2022-03-08 07:30:22 <psykose> not sure how to respond to the mirror query afterward either- i assume just a gitlab reply doesn't send an email back (maybe it does)
2022-03-08 07:38:25 <clandmeter> i think it should
2022-03-08 07:38:34 <clandmeter> its like a ticket system
2022-03-08 07:39:08 <clandmeter> you cant merge the mr?
2022-03-08 07:39:38 <psykose> nope
2022-03-08 07:41:15 <psykose> the usual "Ask someone with write access.."
2022-03-08 07:41:36 <clandmeter> try now
2022-03-08 07:41:54 <psykose> hey it works :)
2022-03-08 07:42:25 <clandmeter> ikke: i hope you dont mind (i think you dont :)), i added alice to maintainer of mirrors
2022-03-08 07:43:38 <psykose> now i seem to remember something about it taking some amount of time to refresh
2022-03-08 07:43:53 <psykose> at least last time ikke amended something on it for the FIT CTU one
2022-03-08 07:44:38 <clandmeter> im not sure how i set it up
2022-03-08 07:44:42 <clandmeter> or if ikke changed something
2022-03-08 07:44:51 <psykose> maybe
2022-03-08 07:45:22 <clandmeter> some containers run from crond and others from mqtt-exec
2022-03-08 08:03:13 <psykose> added the whole backlog
2022-03-08 08:03:34 <psykose> no refresh yet, so probably cron
2022-03-08 08:07:17 <clandmeter> it is scanning all mirrors so it takes time
2022-03-08 08:07:34 <psykose> ah, that too
2022-03-08 08:07:43 <psykose> also are all these `it@alpinelinux.org` issues actual alerts or just spam
2022-03-08 08:08:15 <psykose> impressively spoofed email if so :p
2022-03-08 08:11:52 <clandmeter> which do you refer to?
2022-03-08 08:12:31 <psykose> the ones titled 'We have restricted..'
2022-03-08 08:12:39 <psykose> in the mirrors issues
2022-03-08 08:12:53 <clandmeter> looks like spam
2022-03-08 08:13:10 <psykose> mhm
2022-03-08 08:28:19 <clandmeter> it did not update automatically
2022-03-08 08:28:25 <clandmeter> i triggered it manually
2022-03-08 08:28:34 <clandmeter> so it will take some time to scan the mirrors
2022-03-08 08:33:04 <psykose> ah, okay
2022-03-08 08:33:05 <psykose> sounds good
2022-03-08 08:33:14 <psykose> i will respond to the issues when i see them updated on mirrors.a.o
2022-03-08 08:37:30 <clandmeter> thanks for helping out!
2022-03-08 08:37:50 <psykose> of course :)
2022-03-08 10:34:41 <rails> \o/
2022-03-08 10:35:45 <ikke> psykose: I also add the mirrors to zabbix a
2022-03-08 10:35:52 <psykose> ah
2022-03-08 10:35:59 <psykose> how does one press those buttons
2022-03-08 10:38:56 <ikke> Easiest to clone another 3rd party mirror
2022-03-08 10:39:15 <ikke> Change hostname and dns, and the HTTP_PATH macro
2022-03-08 10:39:54 <psykose> i would probably need some permissions for that
2022-03-08 10:40:01 <psykose> unless it's not on the inventory page
2022-03-08 10:40:42 <psykose> not sure what the dns port on them refers to as well
2022-03-08 10:43:46 <ikke> psykose: you should have the necessary permissions
2022-03-08 10:44:26 <ikke> under the configuration > hosts tab
2022-03-08 10:45:13 <psykose> ah, i see
2022-03-08 10:45:49 <psykose> what is that dns port
2022-03-08 10:45:56 <psykose> https://img.ayaya.dev/xLrbgNxbMzgE.png
2022-03-08 10:46:10 <ikke> You can ignore it
2022-03-08 10:46:14 <psykose> sure
2022-03-08 10:46:23 <ikke> It's normally the port used for SNMP
2022-03-08 10:46:30 <ikke> but there are not SNMP items, so it's not used
2022-03-08 10:47:52 <psykose> half the mirrors are labelled as snmp monitoring though
2022-03-08 10:48:00 <psykose> and like a few are zbx
2022-03-08 10:48:35 <ikke> Yeah, does not matter a lot. I switched to use the zabbix agent interface instead of snmp because it has less details / more generic
2022-03-08 10:48:36 <psykose> and HTTP_PATH `Value` should be the http:// url right
2022-03-08 10:49:05 <psykose> ah
2022-03-08 10:49:07 <psykose> slashpath
2022-03-08 10:49:08 <psykose> i see
2022-03-08 10:49:16 <psykose> there is also HTTP_SCHEME
2022-03-08 10:49:33 <ikke> if it's not http, you can fill in https
2022-03-08 10:50:43 <psykose> what if it has both
2022-03-08 10:51:51 <ikke> pick one
2022-03-08 10:55:43 <clandmeter> it screams automation :)
2022-03-08 10:55:48 <ikke> yes
2022-03-08 10:56:13 <psykose> the 2degrees one was already there
2022-03-08 10:56:18 <psykose> unless you just happened to do it at the same time
2022-03-08 10:56:19 <psykose> :p
2022-03-08 10:56:25 <psykose> but sure, all done
2022-03-08 10:56:31 <ikke> No, I did not add anything
2022-03-08 10:56:36 <psykose> strange
2022-03-08 10:56:50 <clandmeter> automagical automation 
2022-03-08 10:57:17 <psykose> is there an edit history
2022-03-08 10:57:19 <ikke> psykose: It might be that I've already added it before
2022-03-08 10:57:26 <psykose> possibly
2022-03-08 10:57:28 <ikke> to check the mirror stability
2022-03-08 10:57:32 <psykose> ah
2022-03-08 10:57:33 <psykose> makes sense
2022-03-08 10:57:52 <ikke> psykose: there is an auditlog for super admins
2022-03-08 10:57:56 <psykose> ah
2022-03-08 10:58:11 <ikke> but they changed it a bit in 6.0 and all history before that is gone
2022-03-08 10:58:18 <psykose> is there a double auditlog for triple extreme super duper admins too
2022-03-08 10:58:29 <psykose> :)
2022-03-08 10:58:32 <ikke> I cannot tell you
2022-03-08 10:59:25 <ikke> ;-)
2022-03-08 10:59:48 <psykose> :3
2022-03-08 10:59:52 <psykose> this rescanning does take a while
2022-03-08 11:10:34 <clandmeter> an audit log that disappears with an update, is there an audit entry for that task?
2022-03-08 11:50:00 <ikke> Who audits the audit log? 
2022-03-08 11:50:21 <psykose> how long does it normally take
2022-03-08 11:52:50 <ikke> psykose: to appear on mirrors.a.0?
2022-03-08 11:52:55 <psykose> ye
2022-03-08 11:53:02 <ikke> afaik the cron just runs once per day
2022-03-08 11:53:09 <psykose> sure, makes sense :)
2022-03-08 12:06:19 <clandmeter> no the cron is triggered by mqtt
2022-03-08 12:06:23 <clandmeter> err
2022-03-08 12:06:26 <clandmeter> the build
2022-03-08 12:06:37 <clandmeter> cron... /o\
2022-03-08 12:06:56 <clandmeter> ikke: not suer why it was not triggered
2022-03-08 12:07:16 <clandmeter> maybe because its a MR
2022-03-08 12:07:57 <clandmeter> 738 root      3h41 {generate-json.l} /usr/bin/lua5.3 ./generate-json.lua debug
2022-03-08 12:08:02 <psykose> i also did a bunch of master pushes, but afaik they are the same on the thing
2022-03-08 12:08:47 <clandmeter> the system is kind of stupid
2022-03-08 12:08:54 <clandmeter> as its made by me :)
2022-03-08 12:09:10 <clandmeter> but initially that was fine when there werent that much mirrors
2022-03-08 12:09:18 <clandmeter> now it just takes ages to scan all the mirrors
2022-03-08 12:10:10 <psykose> same
2022-03-08 15:11:38 <clandmeter> ikke: i have 10k on the azure account
2022-03-08 15:11:43 <clandmeter> i just dont know how to spend it
2022-03-08 15:12:20 <clandmeter> its not the most intuitive interface ive ever seen
2022-03-08 15:13:32 <Hello71> it's "modern"
2022-03-08 15:15:25 <clandmeter> i would not call it modern
2022-03-08 16:00:13 <ikke> I've a bit of experience with it
2022-03-08 16:00:35 <ikke> And believe me, it's not difficult to expend it :-p
2022-03-08 18:35:39 <danieli> I've used azure way way way too much
2022-03-08 18:35:54 <danieli> +1 it's super easy to waste money
2022-03-08 21:32:43 <clandmeter> looks like mirror page never updated
2022-03-08 23:37:22 <psykose> indeed it didn't
2022-03-09 06:16:10 <rails> azure almost ruined a confrence i ran
2022-03-09 06:16:17 <rails> that was... stressful
2022-03-09 06:16:17 <rails> xD
2022-03-09 11:22:49 <ikke> rails: oof
2022-03-09 11:22:59 <rails> yeah
2022-03-09 11:23:18 <rails> it rebooted all of our very big GPU instances at 8am the morning of
2022-03-09 11:23:28 <rails> y'know, the $250/hr ones
2022-03-09 11:30:48 <ikke> big ouch
2022-03-09 19:38:33 <clandmeter> ikke: did you ever run a different os as the ones that are offered on azure?
2022-03-09 19:38:54 <ikke> clandmeter: no
2022-03-09 19:42:11 <clandmeter> would be nice if we could look into that.
2022-03-09 19:42:22 <clandmeter> get alpine running on it
2022-03-09 19:47:43 <ikke> yes, would be nice
2022-03-09 19:56:37 <Hello71> maybe azure also has one of those $50000/year alpine images
2022-03-10 11:21:46 <ikke> sadly the "network issues" gitlab shows sometimes is not solved yet :(
2022-03-10 22:37:40 <psykose> indeed not
2022-03-11 20:19:38 <ikke> clandmeter: I guess we need to look at https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic
2022-03-11 21:53:26 <clandmeter> Yup saw that one
2022-03-11 21:54:19 <clandmeter>  and we need to upload it somewhere
2022-03-11 21:54:48 <clandmeter>  which we still need to config
2022-03-12 09:14:06 <psykose> the s390x ci is still disconnected entirely
2022-03-12 09:16:12 <psykose> and build-edge
2022-03-12 10:39:51 <ikke> Both hosts are unreachable
2022-03-12 10:41:01 <ikke> ppc64le builder as well
2022-03-12 10:41:07 <ikke> seems to be some upstream isseu
2022-03-12 10:45:00 <psykose> yep
2022-03-12 10:45:59 <psykose> finally the useless arches are gone 🙏 /s
2022-03-12 10:47:38 <mps> psykose: +1 :)
2022-03-13 07:22:52 <ikke> s390x seems to be back
2022-03-13 07:23:20 <psykose> yep
2022-03-13 07:27:39 <ikke> the host was rebooted
2022-03-13 07:27:44 <ikke> builder host
2022-03-13 07:28:00 <ikke> same for ci host
2022-03-13 07:29:06 <psykose> but didn't reconnect?
2022-03-13 07:29:43 <ikke> No, it was _just_ rebooted
2022-03-13 07:29:47 <psykose> ah
2022-03-13 07:29:48 <ikke> less than 30m uptime
2022-03-13 07:47:57 <psykose> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/659156 this seems to not be having fun
2022-03-13 07:48:04 <psykose> i think you need the seccomp profile stuff again
2022-03-13 07:48:12 <psykose> 3.15 too
2022-03-13 07:48:29 <psykose> since it was probably not saved between reboot
2022-03-13 07:48:35 <psykose> that whole faccessat2 thing
2022-03-13 08:04:22 <ikke> reenabled the custom seccomp profile
2022-03-13 08:04:28 <ikke> Was disabled in the config
2022-03-13 08:05:59 <psykose> thanks :3
2022-03-13 08:12:30 <ikke> Will ask if we can get help upgrading these hosts once more
2022-03-13 09:34:36 <clandmeter> Good luck with that
2022-03-14 09:05:42 <mps> anyone know good site which gives good info about CVEs (not those with a lot of marketing nonsense)
2022-03-14 10:22:41 <ikke> Just nivd / cve.org
2022-03-14 10:30:58 <mps> not so up to time
2022-03-14 10:31:24 <ikke> up-to-date?
2022-03-14 10:31:32 <mps> yes
2022-03-14 10:32:05 <mps> up-to-datetime :)
2022-03-14 10:33:34 <mps> it is not so much important to me, but need to add 'good info' for some security upgrades in aports
2022-03-14 10:33:51 <ikke> mps: this is an industry wide problem
2022-03-14 10:35:20 <mps> yes it is, but I have impression that was better when it was theater ;)
2022-03-14 10:35:31 <mps> now industry, good term
2022-03-14 17:28:02 <Ariadne> there is https://github.com/CVEProject/cvelist
2022-03-14 17:30:12 <Ariadne> integrating this into security.alpinelinux.org may be interesting, it uses the same format as NVD
2022-03-14 17:30:18 <Ariadne> the annoying side is it's a git repo
2022-03-14 18:23:35 <rails> 
ACTION eyes

2022-03-14 19:39:17 <mps> Ariadne: hmm, really annoying, you are right
2022-03-15 12:26:14 <psykose> ppc builders came back up but are stuck in git clone
2022-03-15 12:31:51 <ncopa> my guess is network issues. maybe MTU blackhole or similar?
2022-03-15 12:36:48 <ncopa> apk update also hangs on downloading the APKINDEX
2022-03-15 12:38:07 <ncopa> aw...
2022-03-15 12:41:26 <ncopa> phew... i thought i lost contact with the server
2022-03-15 12:41:40 <ncopa> i manually set mtu to 1480, but it does not seem to solve anything
2022-03-15 12:45:27 <ikke> hmm, it was not this host that had the MTU issue, at least not the one we had with the VMs
2022-03-15 12:45:56 <ikke> zabbix still reports the host as unreachabl
2022-03-15 12:48:32 <ikke> oh, it monitors it from the internal IP
2022-03-15 12:51:38 <ncopa> i can reset the mtu to 1500
2022-03-15 12:52:13 <ncopa> the network is broken in general. I cannot curl https://gitlab.alpinelinux.org
2022-03-15 12:53:02 <ikke> I cannot ping 8.8.8.8
2022-03-15 12:53:13 <ikke> firewall issue?
2022-03-15 12:53:33 <ncopa> possibly
2022-03-15 13:13:17 <ikke> I can ping the default route
2022-03-15 13:13:21 <ikke> maybe upstream issue?
2022-03-15 13:13:30 <ikke> Rafael mentioned they did work on the firewalls
2022-03-15 13:16:28 <ncopa> the ppc64le builders aare totally non-functional if they cannot git pull from gitlab.a.o
2022-03-15 13:17:34 <ikke> They cannot connect anywhere outside
2022-03-15 13:17:46 <ncopa> dns works
2022-03-15 13:18:00 <ncopa> unless its because its cached....
2022-03-15 13:18:28 <ikke> no, I can resolve arbitrary addresses
2022-03-15 13:18:58 <ikke> sounds like a strict firewall to me
2022-03-15 13:20:39 <ikke> hmm, and mqtt seems to work as well, since we get updates on build.a.o
2022-03-15 13:21:44 <mps> ppc64le 'keeps' openssh upgrade, anyone have access to it to check why it fail
2022-03-15 13:37:22 <ikke> ncopa: did you do something? I can ping now and dmvpn seems to be working as well
2022-03-15 13:40:07 <ikke> ok, edge builder seems to be building
2022-03-15 17:08:30 <ncopa> ikke: i went out for a walk. that seems to have solved it :)
2022-03-15 17:08:43 <ikke> ncopa: ha! always good :)
2022-03-15 17:08:56 <ncopa> i ran an apk upgrade on the host. it pulled in a new kernel. i wonder if I dare to reboot?
2022-03-15 17:10:04 <ncopa> seems like the edge and 3.15 builders are busy. maybe wait til they are done
2022-03-15 22:06:38 <mps> we need to merge ssl MRs I think
2022-03-15 23:34:06 <rails> apologies for the noise - docker is being a pain in the patootie
2022-03-16 15:15:37 <clandmeter> ikke: problems again with dmvpn?
2022-03-16 15:15:55 <ikke> hmm
2022-03-16 16:00:28 <ikke> clandmeter: btw, what do you think about https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/31987
2022-03-16 16:01:00 <clandmeter> oh python
2022-03-16 16:01:20 <ikke> yup
2022-03-16 16:01:56 <clandmeter> i think you have been busy...
2022-03-16 16:02:36 <clandmeter> but nice you did it :)
2022-03-16 16:04:19 <ikke> I'm wondering whether it's a good idea to actually commit it to aports
2022-03-16 16:04:28 <clandmeter> right
2022-03-16 16:04:36 <clandmeter> you could do it also in a container 
2022-03-16 16:04:48 <ikke> nod
2022-03-16 16:04:48 <clandmeter> i guess that is youir use case
2022-03-16 16:04:59 <ikke> Not my initial usecase
2022-03-16 16:05:04 <ikke> which is why I packaged it
2022-03-16 16:05:19 <clandmeter> you wanted to use it in ci right?
2022-03-16 16:05:48 <ikke> yes, though, I read recently somewhere that it might not even be required for tf, so..
2022-03-16 16:06:43 <clandmeter> i have no idea
2022-03-16 16:06:54 <clandmeter> python apps in aports are messy 
2022-03-16 16:07:03 <clandmeter> but sometimes its needed
2022-03-16 16:07:25 <ikke> yes, especially because it sometimes needs specific versions of packages
2022-03-16 16:07:29 <clandmeter> btw, i did boot alpine on qemu now
2022-03-16 16:07:44 <clandmeter> had to change some settings to get serial working
2022-03-16 16:08:11 <ncopa> i need to get new releases out today, but the arm builders are pretty busy. I wonder if I can lxc-freeze the build-edge-a* til the build-3-15-a* builders are done?
2022-03-16 16:08:41 <clandmeter> err i dont mean qemu :)
2022-03-16 16:09:04 <clandmeter> i mean azure of c
2022-03-16 16:10:12 <psykose> aarch64 edge seems to be stuck on openssl3
2022-03-16 16:10:25 <psykose> been like 2 hours
2022-03-16 16:10:47 <ncopa> i think it is becuase the machine is busy building kernels and other stuff
2022-03-16 16:11:05 <clandmeter> i have requested another machine
2022-03-16 16:11:10 <clandmeter> but that wont help now..
2022-03-16 16:11:18 <psykose> ah, could be
2022-03-16 16:11:26 <ncopa> htop shows 100% cpu usage on all cores
2022-03-16 16:11:27 <psykose> but hf/v7 was done much faster
2022-03-16 16:11:40 <ncopa>  Load average: 401.55 357.66 295.79 
2022-03-16 16:11:42 <clandmeter> actually, i do have another machine idle on my desk
2022-03-16 16:12:27 <ncopa> there was 5.15.28 kernels on 3.15-stable (armhf, armv7, aarch64) earlier today
2022-03-16 16:12:34 <ncopa> that is both -lts kernel and -rpi kernel
2022-03-16 16:12:56 <ncopa> they were not done building until the 5.15.29 kernel was released
2022-03-16 16:13:01 <ncopa> so I pushed those too
2022-03-16 16:13:44 <ncopa> so it was 3.15-stable, armhf, armv7, aarch64 building both -rpi kernel and -lts + kernels on edge branch
2022-03-16 16:13:59 <clandmeter> that strategy is kind of impossible now with only one machine
2022-03-16 16:14:13 <ncopa> and apparently someone else pushed qt5-qtwebengine to edge meanwhile
2022-03-16 16:14:41 <clandmeter> what does lxc-freeze do?
2022-03-16 16:14:53 <ncopa> i think it freezes the processes in the container
2022-03-16 16:14:54 <clandmeter> does it keep its memory?
2022-03-16 16:15:01 <ncopa> yes, i think it does
2022-03-16 16:15:10 <ncopa> i think its similar to ctrl-z
2022-03-16 16:15:24 <ncopa> then lxc-unfreeze to get them going again
2022-03-16 16:15:29 <ncopa> like 'fg'
2022-03-16 16:15:35 <clandmeter> i have no problem in freezing stuff to get stuff done
2022-03-16 16:15:53 <clandmeter> just a matter of setting prios i guess :)
2022-03-16 16:16:01 <clandmeter> have you tried it before?
2022-03-16 16:16:11 <ncopa> nope
2022-03-16 16:16:21 <ncopa> what can possibly go wrong? :)
2022-03-16 16:16:22 <clandmeter> maybe try on another container first :)
2022-03-16 16:16:39 <ncopa> i mean worst case we'll have to reboot the machine
2022-03-16 16:16:40 <ikke> at most, we would need to restart it
2022-03-16 16:16:42 <clandmeter> our infra is self healing 
2022-03-16 16:16:54 <clandmeter> with some help ofc ;-)
2022-03-16 16:17:39 <ncopa> i froze my ncopa-edge-armv7 while running a ping
2022-03-16 16:17:58 <psykose> ah, yep, just took a while
2022-03-16 16:18:02 <psykose> 400 load hah
2022-03-16 16:18:06 <psykose> over 2x the core count
2022-03-16 16:18:31 <ncopa> yup... i think we have been optimistic wrt how many cores we assign each builder
2022-03-16 16:18:51 <ncopa> lxc-freeze/lxc-unfreeze seems to work as intended
2022-03-16 16:18:54 <ikke> We could halve it?
2022-03-16 16:18:58 <ikke> 40 cores per container?
2022-03-16 16:19:21 <ncopa> it kinda sucks to have the idle under normal operation
2022-03-16 16:19:35 <clandmeter> i can look into bringing back the prev aarch64 machine
2022-03-16 16:19:46 <clandmeter> but its 64b only
2022-03-16 16:21:23 <ncopa> i think we can freeze the edge builders for a few mins for now
2022-03-16 16:21:55 <ncopa> i think this is exception
2022-03-16 16:22:18 <clandmeter> if it works, maybe its a good practice in the future when doing releases. 
2022-03-16 16:22:56 <psykose> fine by me
2022-03-16 16:22:59 <psykose> also a nice live test :)
2022-03-16 16:27:10 <clandmeter> we could add this feature to algitbot 
2022-03-16 16:27:13 <clandmeter> would be nice :)
2022-03-16 16:27:47 <ikke> algitbot: freeze clandmeter
2022-03-16 16:27:49 <ikke> :P
2022-03-16 16:28:12 <clandmeter> algitbot: send build-edge-aarch64 for lunch 
2022-03-16 16:28:38 <psykose> algitbot: thaw dinner
2022-03-16 16:31:13 <clandmeter> looks like git.a.o is not very responsive 
2022-03-16 16:32:03 <psykose> yep
2022-03-16 16:34:03 <ikke> restarted the containers
2022-03-16 16:36:03 <ncopa> i froze build-edge-aarch64 and build-edge-armv7
2022-03-16 16:36:36 <psykose> looks cold to me
2022-03-16 16:37:46 <psykose> <ncopa> and apparently someone else pushed qt5-qtwebengine to edge meanwhile <- that was me, sorry :p i thought you were only making the edge release at first
2022-03-16 16:38:54 <psykose> there are still the mariadb mrs as well that you might want to be in the release, but i'm not sure if it matters since minor releases are only for new .iso's
2022-03-16 16:39:30 <ikke> clandmeter: not sure if related, but mirrors is updating atm
2022-03-16 16:39:39 <ikke> 1586 root       20   0 21796 20968  3900 R 99.7  0.3     200h /usr/bin/lua5.3 ./generate-json.lua debug
2022-03-16 16:39:44 <ikke> on the host
2022-03-16 17:07:47 <ncopa> the htop is interesting. cores 80-159 are maxed out, and red
2022-03-16 17:07:56 <ncopa> the cores 0-79 are not
2022-03-16 17:08:43 <ikke> ncopa: we limit containers to certain cores 
2022-03-16 17:09:25 <ikke> Due to NUMA constraints 
2022-03-16 17:10:19 <ikke> So it's correct that you see that
2022-03-16 17:11:55 <ikke> ncopa: it showing a lot of red was not that good according to Ed Vilmetti, but no idea how to avoid that
2022-03-16 17:20:45 <psykose> not that good meaning it's effectively losing all that cpu, pretty much
2022-03-16 17:20:50 <psykose> i have no idea either
2022-03-16 18:04:07 <psykose> was aarch-3.15 frozen intentionally
2022-03-16 18:24:14 <psykose> ah, seems like it was just a visual bug
2022-03-16 19:53:48 <ikke> Testing multi-stage builds, but it's stuck: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/31987/pipelines
2022-03-16 19:57:37 <ikke> Ah: https://stackoverflow.com/a/70420880/20261
2022-03-16 19:58:15 <psykose> ooh
2022-03-16 19:58:16 <psykose> neat
2022-03-16 19:58:42 <psykose> could you also make the lint stage parallel to the others while you're at this
2022-03-16 19:58:45 <psykose> :3
2022-03-16 19:59:07 <ikke> Will look at it
2022-03-16 20:47:52 <ikke> Ooooh, lookie: https://gitlab.alpinelinux.org/alpine/aports/-/pipelines/115034
2022-03-16 20:49:27 <ikke> And it did not schedule the later stages yet!
2022-03-16 20:49:39 <ncopa> sweet!
2022-03-16 20:50:32 <ncopa> im gonna call it a day now. will try do 3.14, 3.13 and 3.12 releases tomorrow
2022-03-16 20:50:44 <ikke> o/
2022-03-16 20:50:58 <ikke> ncopa: If this works out, this should make it more feasible to do large rebuilds in CI
2022-03-16 21:48:13 <ikke> clandmeter: seems indeed a network issue again, I can reach dmvpn1 via OOB
2022-03-16 21:49:07 <ikke> clandmeter: I suppose that's something we could move to azure
2022-03-17 06:51:49 <ikke> psykose: https://gitlab.alpinelinux.org/alpine/aports/-/pipelines/115082
2022-03-17 07:32:11 <ikke> (it fails because the built packages still need to be put in place)
2022-03-17 09:44:14 <ncopa> I deleted lots of release candidates from dl-master
2022-03-17 09:44:23 <ncopa> all except 3.15.0_rc*
2022-03-17 09:44:30 <ncopa> maybe I should delete those too
2022-03-17 09:45:15 <ikke> wouldn't the builders just upload them there again? Or did you also remove them from the builders?
2022-03-17 09:47:23 <ncopa> i thikn I have removed them from the builders as well, but let me check that
2022-03-17 09:48:36 <ncopa> ok, 3.14 still had release candidates
2022-03-17 10:41:18 <nmeum> ikke: is my port forwarding (Port 22061) on ppc64le.alpinelinux.org still active? would like to debug the go 1.18 build failure on ppc64le but can't seem to access the machine over ssh
2022-03-17 10:42:20 <nmeum> > ssh: connect to host ppc64le.alpinelinux.org port 22061: Operation timed out
2022-03-17 10:43:43 <ikke> yes, the port-forward is still there
2022-03-17 10:43:53 <ikke> container ip is the same
2022-03-17 10:44:15 <ikke> sshd is also running, hmm
2022-03-17 10:44:24 <nmeum> weird, it did work fine a few weeks ago
2022-03-17 10:45:25 <ikke> forwarding is enabled
2022-03-17 10:46:14 <psykose> ikke: that looks awesome :)
2022-03-17 10:48:13 <nmeum> I can ping the server, I can also reach the sshd running on port 22 but I can't reach the one on port 22061
2022-03-17 10:50:03 <ikke> nmeum: I'll look into it in a bit
2022-03-17 10:50:35 <nmeum> sure, thanks. take your time. it's not urgent :)
2022-03-17 11:23:17 <ikke> nmeum: I don't even see the traffic reaching the host with tcpdump
2022-03-17 11:35:00 <ikke> nmeum: ibm mentioned they did work on firewalls, so maybe this traffic is blocked now
2022-03-17 11:35:40 <ikke> nmeum: Could we set you up to connect to our wg vpn? Then you should be able reach it over dmvpn
2022-03-17 12:26:22 <psykose> ikke: you should kick stunnel on the builders
2022-03-17 12:36:57 <ikke> a
2022-03-17 12:37:29 <clandmeter> psykose: lets move the discussion to here
2022-03-17 12:38:06 <clandmeter> so git.a.o is acting up sometimes, but we are not sure why.
2022-03-17 12:38:26 <clandmeter> the host shares a few services mostly in docker
2022-03-17 12:39:04 <clandmeter> ill add your key to the host
2022-03-17 12:41:14 <psykose> alright
2022-03-17 12:41:28 <ikke> We also need to do a system upgrade on that host, it's still running alpine 3.13
2022-03-17 12:41:33 <clandmeter> ikke: seems due1 is not on dmvpn?
2022-03-17 12:41:53 <ikke> clandmeter: We did not add it for docker hosts
2022-03-17 12:41:57 <ikke> (yet)
2022-03-17 12:42:04 <clandmeter> psykose: /srv/compose and /srv/docker are docker related dirs
2022-03-17 12:42:16 <psykose> right
2022-03-17 12:42:29 <clandmeter> you need some compose fu to get going :)
2022-03-17 12:42:48 <ikke> she already looked at aports-qa-bot before
2022-03-17 12:42:48 <psykose> this has a lot of services
2022-03-17 12:42:59 <psykose> yes, and i built it, but you never pointed me to gitlab-test :)
2022-03-17 12:43:00 <ikke> psykose: gbr2 has more :)
2022-03-17 12:43:05 <ikke> psykose: I'm aware
2022-03-17 12:43:16 <clandmeter> this has limited services :)
2022-03-17 12:43:19 <clandmeter> check the arm box
2022-03-17 12:43:23 <psykose> well, first things first
2022-03-17 12:43:28 <psykose> what's gonna break if we just bump to 3.15
2022-03-17 12:43:43 <clandmeter> the host?
2022-03-17 12:43:44 <ikke> psykose: I've been upgrading most infra to 3.15 without issue
2022-03-17 12:43:51 <psykose> ok, let's do it live :p
2022-03-17 12:43:59 <clandmeter> you can upgrade it
2022-03-17 12:43:59 <ikke> psykose: We always have to do it live :P
2022-03-17 12:44:02 <clandmeter> should not matter much
2022-03-17 12:44:06 <psykose> was the docker uhh
2022-03-17 12:44:07 <psykose> thing
2022-03-17 12:44:09 <psykose> fixed on 3.15
2022-03-17 12:44:10 <psykose> i forget
2022-03-17 12:44:12 <psykose> faccessat
2022-03-17 12:44:15 <clandmeter> just make sure you run update-conf
2022-03-17 12:44:37 <ikke> psykose: 3.14 I believe
2022-03-17 12:44:41 <clandmeter> as long as docker can run and compose, it should be fine.
2022-03-17 12:44:47 <psykose> ah, 'coffee' is not found
2022-03-17 12:45:08 <ikke> but we did not set anything special on these hosts to mitigate that
2022-03-17 12:45:14 <clandmeter> psykose: please keep host base pkg selection same as now
2022-03-17 12:45:20 <psykose> it's not found on 3.13 either
2022-03-17 12:45:25 <psykose> not sure what it's doing in the world file
2022-03-17 12:45:42 <clandmeter> only add what is really needed
2022-03-17 12:45:43 <psykose> and isn't even listed by apk policy
2022-03-17 12:45:48 <psykose> i'm not adding anything :)
2022-03-17 12:45:54 <psykose> it's just a thing in world that doesn't exist
2022-03-17 12:46:05 <clandmeter> its docker, so all magic should happen in the containers anwyays :)
2022-03-17 12:46:14 <psykose> https://img.ayaya.dev/n5JHlvZuAnWB.png
2022-03-17 12:46:15 <psykose> like so
2022-03-17 12:46:29 <clandmeter> wut
2022-03-17 12:46:39 <psykose> it's also not presently installed, so i think safe to remove
2022-03-17 12:46:53 <clandmeter> i have no idea how it came into world
2022-03-17 12:46:54 <psykose> wonder how that ended up there :p
2022-03-17 12:47:00 <clandmeter> normally i just blame ikke 
2022-03-17 12:47:12 <clandmeter> but now i have two options
2022-03-17 12:47:14 <ikke> 
ACTION whistles innocently

2022-03-17 12:47:40 <psykose> 3.15 upgrade wants to pull in pam
2022-03-17 12:47:40 <psykose> fun
2022-03-17 12:47:48 <ikke> uhh..
2022-03-17 12:48:12 <psykose> also moves lua5.3 stuff to 5.4
2022-03-17 12:48:22 <clandmeter> whats inside world?
2022-03-17 12:48:37 <psykose> https://img.ayaya.dev/GdvJCJxOfNOO.png
2022-03-17 12:49:01 <psykose> ah
2022-03-17 12:49:02 <psykose> util-linux
2022-03-17 12:49:14 <ikke> I had the suspicion
2022-03-17 12:49:15 <psykose> that's what uses it
2022-03-17 12:49:21 <clandmeter> you can kill it
2022-03-17 12:49:28 <psykose> rm util-linux?
2022-03-17 12:49:28 <psykose> sure
2022-03-17 12:49:43 <clandmeter> i sometimes install it cause i mess colors in dmesg
2022-03-17 12:49:49 <clandmeter> miss
2022-03-17 12:49:50 <psykose> https://img.ayaya.dev/fFsYKaDTbunP.png
2022-03-17 12:49:52 <psykose> full upgrade
2022-03-17 12:49:58 <psykose> -55MB :p
2022-03-17 12:50:21 <clandmeter> 👍
2022-03-17 12:51:05 <psykose> reboot should be done now, given kernel upgrade
2022-03-17 12:51:15 <psykose> bets on it coming back up? :)
2022-03-17 12:51:33 <ikke> they're vms, pretty high
2022-03-17 12:53:53 <ikke> it's back again
2022-03-17 12:53:55 <psykose> yay
2022-03-17 12:53:59 <psykose> happy bot
2022-03-17 12:54:02 <psykose> warm and shining
2022-03-17 12:54:19 <psykose> all containers up
2022-03-17 12:54:32 <psykose> git.a.o works for now
2022-03-17 12:54:35 <psykose> now we can go to that
2022-03-17 12:54:35 <clandmeter> psykose: did you run update-conf?
2022-03-17 12:54:38 <psykose> not yet
2022-03-17 12:54:40 <psykose> now i will
2022-03-17 12:54:54 <clandmeter> i think its better to run it before reboot 
2022-03-17 12:54:59 <psykose> yes
2022-03-17 12:55:01 <psykose> but i'm a dummy :)
2022-03-17 12:55:03 <psykose> do you use -i or not
2022-03-17 12:55:08 <clandmeter> its not the first time it will not return to sender :)
2022-03-17 12:55:28 <clandmeter> ok im going to take a look at my new kitchen :)
2022-03-17 12:57:31 <psykose> ok, update done
2022-03-17 12:57:36 <psykose> nothing new in confs
2022-03-17 12:59:10 <ikke> Yeah, usually I zap most new ocnfigs
2022-03-17 12:59:13 <ikke> configs*
2022-03-17 12:59:27 <psykose> it should probably be reworked a little
2022-03-17 12:59:33 <psykose> cause i'm sure someone has accidentally updated shadow
2022-03-17 12:59:44 <psykose> and broke their system
2022-03-17 12:59:56 <ikke> a 3-way merge would be a lot better
2022-03-17 13:00:01 <psykose> yeah
2022-03-17 13:00:05 <psykose> like git -p style
2022-03-17 13:00:09 <psykose> i think the gentoo tools support that
2022-03-17 13:00:21 <ikke> You need the original config for that though
2022-03-17 13:00:32 <ikke> git add -p is still 2-way
2022-03-17 13:00:43 <psykose> yes, but at least section-by-section is a start too
2022-03-17 13:00:52 <ikke> if you install vimdiff, you can use that
2022-03-17 13:00:57 <psykose> ooh
2022-03-17 13:01:43 <psykose> hm
2022-03-17 13:01:53 <psykose> why does traefik sit at 30% cpu while nothing is happening
2022-03-17 13:02:43 <psykose> we could bump it to 2.6 :) i'll have to read all the changelogs
2022-03-17 13:04:01 <ikke> We have it already running on some other hosts
2022-03-17 13:04:33 <psykose> the config file is nothing fancy so it seems fine
2022-03-17 13:04:41 <psykose> did you run into any issues on the others
2022-03-17 13:04:45 <ikke> The routing on the container side changes a bit
2022-03-17 13:05:06 <psykose> mm
2022-03-17 13:05:08 <psykose> what changes
2022-03-17 13:07:01 <psykose> https://doc.traefik.io/traefik/migration/v2/ commonname from 2.2->2.3 ?
2022-03-17 13:20:54 <ikke> hmm, maybe nothing changes there
2022-03-17 13:22:11 <ikke> psykose: btw, if you upgrade a host, can you also update the platform in netbox?
2022-03-17 13:24:13 <psykose> done
2022-03-17 13:24:23 <ikke> thanks :)
2022-03-17 13:24:33 <psykose> :3
2022-03-17 13:24:44 <psykose> i can try just upgrading traefik if you want
2022-03-17 13:24:52 <psykose> is it as simple as docker-compose down && up :)
2022-03-17 13:25:32 <ikke> just docker-compose up -d
2022-03-17 13:25:39 <clandmeter> you need to pull new container
2022-03-17 13:25:45 <ikke> right
2022-03-17 13:25:53 <ikke> although, if you change the image, it should do it already
2022-03-17 13:26:09 <psykose> recreating ~
2022-03-17 13:26:12 <psykose> done
2022-03-17 13:26:16 <psykose> lets see if everything crashed
2022-03-17 13:26:26 <clandmeter> traefik load is due to requests i think
2022-03-17 13:26:32 <psykose> could be
2022-03-17 13:26:34 <clandmeter> not sure we can fix that 
2022-03-17 13:26:39 <ikke> git.a.o is reachable
2022-03-17 13:26:43 <psykose> sweet
2022-03-17 13:26:57 <psykose> algitbot: ping
2022-03-17 13:27:03 <psykose> :)
2022-03-17 13:27:04 <ikke> secdb as well
2022-03-17 13:27:12 <ikke> psykose: algitbot does not use traefik :)
2022-03-17 13:27:16 <psykose> ah
2022-03-17 13:27:18 <clandmeter> i wonder if cgit has issues because of uwsgi 
2022-03-17 13:27:30 <psykose> it might
2022-03-17 13:27:40 <clandmeter> there are other ways to implement it
2022-03-17 13:27:41 <psykose> can it use any *sgi? there is asgi now
2022-03-17 13:27:55 <clandmeter> lightttp is another option
2022-03-17 13:28:07 <psykose> ah, it was a python thing
2022-03-17 13:28:08 <ikke> It's fcgi I think
2022-03-17 13:28:16 <clandmeter> fastcgi yes
2022-03-17 13:28:49 <clandmeter> i think jason did use uwsgi
2022-03-17 13:29:30 <ikke> jason?
2022-03-17 13:30:13 <clandmeter> Donenfeld
2022-03-17 13:30:31 <clandmeter> he took over development of cgit
2022-03-17 13:30:51 <clandmeter> before he started on wireguard iirc
2022-03-17 13:31:03 <ikke> ah
2022-03-17 13:31:42 <clandmeter> we used to use lighttpd
2022-03-17 13:32:00 <clandmeter> but i am not sure its because of the switch that we run into issue
2022-03-17 13:32:09 <clandmeter> or if its just the traffic that increased by 100x
2022-03-17 13:32:29 <clandmeter> i would think its #2
2022-03-17 13:32:36 <ikke> I think the traffic nowadays is not as much as it was
2022-03-17 13:33:37 <clandmeter> we dont have metrics about it i guess
2022-03-17 13:33:47 <ikke> ~20Mpbs before, and suddenly dropped to ~6Mbps
2022-03-17 13:33:48 <clandmeter> would be nice to have them from traefik 
2022-03-17 13:33:57 <ikke> https://zabbix.alpinelinux.org/history.php?action=batchgraph&itemids%5B31013%5D=31013&itemids%5B31014%5D=31014&graphtype=0
2022-03-17 13:34:05 <ikke> that's interface traffic
2022-03-17 13:34:15 <clandmeter> i am more interested in connections
2022-03-17 13:34:43 <clandmeter> initiating ssl connection cost cpu i guess
2022-03-17 13:34:50 <psykose> last year had a fun drop
2022-03-17 13:35:01 <psykose> https://img.ayaya.dev/QuChjIokWBvl.png
2022-03-17 13:35:14 <ikke> yes
2022-03-17 13:35:16 <psykose> wonder what that wawsw
2022-03-17 13:35:18 <psykose> was*
2022-03-17 13:35:23 <ikke> We did some blocking on user agent
2022-03-17 13:35:25 <clandmeter> we added some rules
2022-03-17 13:35:26 <psykose> heh
2022-03-17 13:35:28 <psykose> :)
2022-03-17 13:35:41 <clandmeter> claire was one of the issues
2022-03-17 13:35:43 <psykose> it's strange how consistent it is
2022-03-17 13:35:46 <clandmeter> or whatever its called
2022-03-17 13:35:50 <psykose> the secdb scanner
2022-03-17 13:35:55 <clandmeter> nod
2022-03-17 13:37:22 <psykose> we could enable the cgit cache
2022-03-17 13:37:35 <psykose> default is off, it's commented out in cgitrc
2022-03-17 13:37:43 <psykose> not sure how well it works
2022-03-17 13:37:48 <ikke> We did have it enabled
2022-03-17 13:37:55 <psykose> let me guess, everything broke
2022-03-17 13:45:07 <psykose> stunnel armhf stuck again
2022-03-17 13:45:08 <psykose> heh
2022-03-17 13:46:32 <ikke> fun, segfault when I strace the process
2022-03-17 13:46:46 <ikke> and _ofcourse_ it just continues
2022-03-17 13:47:17 <psykose> lol
2022-03-17 14:00:45 <psykose> for some reason there is only one aarch64 ci runner
2022-03-17 14:00:54 <psykose> as in job
2022-03-17 14:01:00 <psykose> weirdly intermittent stuff
2022-03-17 14:01:25 <psykose> sometimes it's other arches too
2022-03-17 14:02:16 <ikke> Someone is using our CI infra
2022-03-17 14:02:29 <psykose> whadya mean by using
2022-03-17 14:03:19 <ikke> Using our runners to host applications
2022-03-17 14:03:38 <psykose> like 'illegally'?
2022-03-17 14:03:39 <psykose> hm
2022-03-17 14:04:04 <psykose> what do you see on the admin overview
2022-03-17 14:04:15 <ikke> 'long' running jobs
2022-03-17 14:04:19 <psykose> ah
2022-03-17 14:04:31 <psykose> because i do have qt6-qtweb in one of them, but that usually doesn't matter, as there is still two
2022-03-17 14:04:41 <ikke> I killed the jobs
2022-03-17 14:04:46 <psykose> neat
2022-03-17 14:05:05 <psykose> i wish i could see that list myself sometimes, hah
2022-03-17 14:06:05 <ikke> It's a provite project, but first they hosted some remote vscode, and now laravel-server
2022-03-17 14:06:12 <psykose> lol
2022-03-17 14:06:27 <psykose> i think that might explain every instance i've seen of this then
2022-03-17 14:06:32 <psykose> where there is strangely one runner
2022-03-17 14:06:59 <ikke> clandmeter: I think this is a violation of our terms of usage (someone trying to host their personal projects on our gitlab instance
2022-03-17 14:07:08 <psykose> it is
2022-03-17 14:07:15 <ikke> https://gitlab.alpinelinux.org/agussusahnti
2022-03-17 14:07:17 <psykose> and if it isn't it should be made so :)
2022-03-17 14:07:20 <nmeum> ikke: yes, I can install wireguard, but I have no prior expierence with it yet and need to read the docs first :)
2022-03-17 14:07:29 <psykose> it took me like 30 minutes to set up
2022-03-17 14:07:31 <ikke> it should be relatively simple to setup
2022-03-17 14:07:32 <psykose> you'll be fine
2022-03-17 14:07:46 <clandmeter> ikke: right
2022-03-17 14:07:52 <clandmeter> maybe block the acc and send an email
2022-03-17 14:07:54 <psykose> nmeum: this is most of it https://gitlab.alpinelinux.org/alpine/infra/infra/-/wikis/Alpine-wireguard-VPN
2022-03-17 14:08:05 <psykose> i prefer the /etc/network/interfaces one
2022-03-17 14:08:10 <psykose> then it's just filling out some fields
2022-03-17 14:08:12 <nmeum> looks simple enough
2022-03-17 14:09:14 <ikke> clandmeter: I can first start with adding a notification limited to their user namespace
2022-03-17 14:09:44 <psykose> from what it looks like it usually takes a lot of effort to 'start' a ci job just to host something
2022-03-17 14:09:53 <psykose> i would normally assume it's intentional and they know what they're doing
2022-03-17 14:09:54 <clandmeter> if its really mis usage, i would be kind of strict about it.
2022-03-17 14:10:04 <ikke> ok
2022-03-17 14:10:13 <psykose> it's not as simple as pressing two buttons that say 'host thing' on the side and it happening
2022-03-17 14:10:52 <psykose> but if it's an honest mistake, i guess you will get an email
2022-03-17 14:11:00 <psykose> i doubt this will be the case :p
2022-03-17 14:14:05 <nmeum> ikke: should I just send you a public key for wireguard vpn access?
2022-03-17 14:15:12 <ikke> nmeum: Yes, and we need to reserve an IP for you
2022-03-17 14:15:28 <nmeum> *nod*
2022-03-17 14:16:20 <nmeum> can you do that? (:
2022-03-17 14:16:28 <psykose> 3-5 business days
2022-03-17 14:16:50 <psykose> let me go to see how to do it actually
2022-03-17 14:16:58 <nmeum> hehe
2022-03-17 14:20:23 <psykose> not sure where to add the public key stuff, but ip is reserved
2022-03-17 14:20:42 <psykose> 172.16.252.9
2022-03-17 14:20:47 <psykose> i would guess it's outside of netbox
2022-03-17 15:02:20 <psykose> hmm
2022-03-17 15:02:53 <psykose> it really doesn't like us huh :)
2022-03-17 15:08:50 <psykose> is bra1 actually alpine 3.8 or did someone forget to update it :p
2022-03-17 15:14:41 <nmeum> ok, I have wireguard setup now I think
2022-03-17 15:14:54 <ikke> The hardware dmvpn1 is running on is an older generation that will be discontinued by equinix, apparently they don't have redundant networking for it anymore
2022-03-17 15:14:58 <nmeum> psykose ikke: my public key is fE9qsFGQLCcLlU+TZFaFJrJnOVUhqqlSb54txRcD6Fo=
2022-03-17 15:15:06 <ikke> the host is running, I can access it via OOB console
2022-03-17 15:15:21 <ikke> nmeum: I'll add it to the wg hub later today
2022-03-17 15:15:31 <psykose> do you want to give me access to that too so i can add it :p
2022-03-17 15:15:32 <nmeum> thanks
2022-03-17 15:15:37 <ikke> or, clandmeter I guess we can give psykose access to that node as well
2022-03-17 15:15:40 <nmeum> afterwards I can just run `ifup wg0` and it should just work, right?
2022-03-17 15:15:40 <ikke> :)
2022-03-17 15:15:52 <psykose> assuming you wrote it out correctly in wg0.conf and networking, yeah
2022-03-17 15:16:29 <clandmeter> which node?
2022-03-17 15:17:02 <ikke> clandmeter: deu7 I believe
2022-03-17 15:17:11 <clandmeter> which is?
2022-03-17 15:17:13 <ikke> dedicated wg node
2022-03-17 15:17:17 <clandmeter> sorry i didint follow
2022-03-17 15:17:25 <ikke> the wg hub
2022-03-17 15:17:44 <ikke> The one mps setup
2022-03-17 15:17:50 <clandmeter> fine by me
2022-03-17 15:19:08 <ikke> psykose: you should now have access
2022-03-17 15:19:12 <ikke> deu7-dev1
2022-03-17 15:19:18 <psykose> works
2022-03-17 15:19:32 <ikke> The config is /etc/wireguard/wg0.conf
2022-03-17 15:19:37 <psykose> mm
2022-03-17 15:19:42 <psykose> there's an add-wg-user.sh
2022-03-17 15:19:51 <ikke> Then wg syncconf wg0 /etc/wireguard/wg0.conf
2022-03-17 15:20:00 <ikke> right
2022-03-17 15:20:10 <ikke> I haven't tested it myself yet
2022-03-17 15:20:26 <psykose> nmeum: added
2022-03-17 15:20:28 <psykose> ok, did both
2022-03-17 15:21:15 <psykose> make sure you set your ip thing to .9 from what i sent
2022-03-17 15:21:23 <nmeum> yep, I did
2022-03-17 15:21:53 <ncopa> github is down so i have to work on alpine...
2022-03-17 15:21:59 <psykose> oh no!
2022-03-17 15:22:00 <psykose> :)
2022-03-17 15:22:39 <ikke> I don't see a ocnnection yet for .9
2022-03-17 15:22:47 <psykose> me either
2022-03-17 15:24:19 <nmeum> is there any debugging facility for wireguard?
2022-03-17 15:24:28 <nmeum> i.e. log files
2022-03-17 15:24:53 <psykose> not sure where that is, but you could post your wg.conf and network interfaces (remove the private key)
2022-03-17 15:25:06 <psykose> `wg show` should show you wg state
2022-03-17 15:27:30 <nmeum> looks like one can debug wireguard using dynamic_debug
2022-03-17 15:27:34 <nmeum> https://tpaste.us/Z4Pl <- this is my wg show output
2022-03-17 15:28:07 <nmeum> doesn't contain any information about interface state though
2022-03-17 15:28:13 <nmeum> but it shows up in ip a so I guess it works?
2022-03-17 15:28:52 <nmeum> what's the address of the ppc64le thing in the 172.16.0.0/16 subnet?
2022-03-17 15:29:41 <ikke> 11.61
2022-03-17 15:29:50 <ikke> 172.16.11.61
2022-03-17 15:30:12 <ikke> but I don't see anything on the hub yet
2022-03-17 15:31:11 <psykose> can i see the config files? my network is https://img.ayaya.dev/Odinz8Q1S3ti and wg.conf is https://img.ayaya.dev/HtB1XU0wSg03
2022-03-17 15:34:09 <nmeum> sure https://tpaste.us/d4VL and https://tpaste.us/4oqX
2022-03-17 15:34:43 <psykose> i think you need a /32 after the address
2022-03-17 15:34:45 <psykose> not sure though
2022-03-17 15:35:01 <psykose> aside from that, it might be the same issue i had :)
2022-03-17 15:35:06 <psykose> where it didn't work for like 30 minutes
2022-03-17 15:35:09 <psykose> and then just magically did
2022-03-17 15:35:26 <nmeum> haha
2022-03-17 15:35:30 <nmeum> love those
2022-03-17 15:35:44 <psykose> not sure if wg needs more than syncconf or something
2022-03-17 15:35:57 <psykose> but since wg show shows your ip i would assume it should work
2022-03-17 15:37:16 <nmeum> https://tpaste.us/MkLP <- this is my ip address show dev wg0 output
2022-03-17 15:38:29 <ikke> psykose: we could try setconf instead
2022-03-17 15:38:40 <ikke> but that would also interrupt other connections
2022-03-17 15:38:54 <psykose> that disconnected everyone yes
2022-03-17 15:39:02 <psykose> :)
2022-03-17 15:39:07 <nmeum> according to the wg man page one could enable debug logs via the /sys/kernel/debug/dynamic_debug/control but I don't seem to have that file
2022-03-17 15:39:12 <psykose> but now maybe the reconnect works, i doubt it
2022-03-17 15:39:26 <psykose> nmeum: it probably requires a kconfig wg debug thing, and most debug stuff is disabled
2022-03-17 15:39:38 <nmeum> yea, that might be the case
2022-03-17 15:39:40 <psykose> i would guess
2022-03-17 15:39:51 <psykose> i reconnected after drop
2022-03-17 15:39:51 <psykose> hm
2022-03-17 15:40:12 <psykose> ikke: do you remember anything from last time that happened in the 30 minutes before it worked
2022-03-17 15:40:14 <ikke> still nothing for .9
2022-03-17 15:40:26 <ikke> psykose: nothing concrete
2022-03-17 15:40:43 <psykose> at first i tried forcing ipv4 but once it worked ipv6 worked too anyway
2022-03-17 15:42:02 <nmeum> hm
2022-03-17 15:42:32 <nmeum> without any sort of debugging output it's a bit hard to nail down the problem :D
2022-03-17 15:42:42 <psykose> yep
2022-03-17 15:42:45 <nmeum> maybe that dynamic_debug/control file just requires a specific kernel module to be loaded
2022-03-17 15:44:41 <psykose> does netbox needs something to be synced after adding an ip
2022-03-17 15:45:06 <nmeum> ah, we don't have CONFIG_DYNAMIC_DEBUG in our kernel config, that's why it doesn't work
2022-03-17 15:49:25 <nmeum> psykose: apart from the addresses our config files look identical to me
2022-03-17 15:49:32 <psykose> they are
2022-03-17 15:52:09 <nmeum> maybe this is a problem on the server side then? 🤔
2022-03-17 15:52:19 <nmeum> I could also try building a custom kernel with CONFIG_DYNAMIC_DEBUG
2022-03-17 15:53:16 <psykose> could you just try to ifdown/ifup it a few times :)
2022-03-17 15:53:36 <nmeum> hehe
2022-03-17 15:54:43 <nmeum> did that a few times, how would I know that it works apart from being able to ping 172.16.11.61?
2022-03-17 15:55:01 <psykose> wg show would show handshake stuff
2022-03-17 15:55:03 <psykose> and i would see it
2022-03-17 15:55:16 <psykose> i can't ping that ip either fwiw :p
2022-03-17 15:55:43 <nmeum> oh, that's good to know
2022-03-17 15:57:12 <psykose> wish i knew..
2022-03-17 15:57:21 <psykose> while you're on ppc there is another issue actually
2022-03-17 15:57:34 <psykose> binutils 2.38 broke a few ppc things
2022-03-17 15:59:10 <nmeum> hm, just running ifup/ifdown a few times doesn't seem to magically fix it unfourtunatly
2022-03-17 15:59:19 <ikke> hmm, me neither
2022-03-17 15:59:43 <ikke> ok, dmvpn does not work at on there :(
2022-03-17 16:00:34 <nmeum> psykose: yeah I saw the binutils issues on ppc. maybe even related as Go might use binutils for cgo, but idk without having a closer look
2022-03-17 16:00:46 <psykose> it might actually
2022-03-17 16:00:57 <psykose> but the binutils issue in the currently failing stuff is uh
2022-03-17 16:01:04 <psykose> a bunch of assertion failed on some ppc specific code check
2022-03-17 16:01:09 <psykose> no magic sigsegv
2022-03-17 16:01:11 <nmeum> I could try re-building go 1.17 on ppc via the CI and check if it also fails
2022-03-17 16:01:14 <psykose> unless it's actually two issues
2022-03-17 16:01:20 <psykose> i think we upgraded go since 2.38
2022-03-17 16:01:33 <psykose> it was mar 5 for the last upgrade
2022-03-17 16:01:45 <psykose> feb 10 for 2.38
2022-03-17 16:01:55 <psykose> so old version is fine
2022-03-17 16:02:15 <nmeum> *nod*
2022-03-17 16:02:25 <nmeum> in that case I really do need access to a ppc machine to debug it further :D
2022-03-17 16:03:35 <psykose> mhm
2022-03-17 16:03:40 <nmeum> ikke: doing port forwarding via a different port for ppc64le.alpinelinux.org is not possible probably, right?
2022-03-17 16:18:45 <ikke> nmeum: I don't suspect it's just specifically 22061 that's not working
2022-03-17 16:19:41 <nmeum> right, but I suppose they wouldn't block 80 for example
2022-03-17 16:23:56 <Hello71> wg should show peers
2022-03-17 16:27:18 <ikke> Hello71: we see it listed, but no active connection
2022-03-17 16:28:10 <Hello71> i mean in nmeum's previous paste, there are no peers listed
2022-03-17 16:29:30 <nmeum> right
2022-03-17 16:31:46 <nmeum> I just don't know why it doesn't list any peers and short of compiling a kernel with dynamic debug support to enable wireguard logging output I am not sure who I would debug this further
2022-03-17 16:32:07 <ikke> nmeum: you ran something like wg syncconf wg0 path/to/conf?
2022-03-17 16:33:24 <ikke> hmm
2022-03-17 16:33:28 <ikke> gitlab host
2022-03-17 16:33:36 <nmeum> ikke: yeah the /etc/network/interface config runs that as a pre-up rule
2022-03-17 16:33:43 <ikke> ok
2022-03-17 16:33:58 <Hello71> my guess is dns
2022-03-17 16:34:09 <Hello71> strace wg setconf
2022-03-17 16:37:00 <nmeum> here is the strace https://tpaste.us/0WLN
2022-03-17 16:37:10 <nmeum> > stat("/var/run/wireguard/wg0.sock", 0x7ffd3151c150) = -1 ENOENT (No such file or directory)
2022-03-17 16:37:12 <nmeum> is that expected?
2022-03-17 16:37:43 <nmeum> re dns: I can ping and resolve wg.alpinelinux.org, not sure how it would be related to dns apart from that
2022-03-17 16:41:25 <ikke> tcpdump?
2022-03-17 16:45:52 <Hello71> it means that it is using kernel wg, not userspace
2022-03-17 16:46:05 <Hello71> i don't see anything obviously wrong
2022-03-17 16:46:10 <Hello71> maybe try rebooting
2022-03-17 16:46:11 <nmeum> hm, I actually don't see any packages send to 172.104.203.112 (wg.alpinelinux.org) in wireshark
2022-03-17 16:46:22 <nmeum> s/packages/packets/
2022-03-17 16:46:22 <alpine-meetbot> nmeum meant to say: hm, I actually don't see any packets send to 172.104.203.112 (wg.alpinelinux.org) in wireshark
2022-03-17 16:46:38 <nmeum> so maybe it fails before even performing the handshake
2022-03-17 16:49:09 <Hello71> well if it is not registered then of course it will not send anything
2022-03-17 16:49:20 <Hello71> although wireguard does not proactively send packets just because a peer is registered
2022-03-17 16:51:37 <nmeum> I will look into building a kernel with dynamic debug support to access the wireguard debugging information, but won't have the time to do that today
2022-03-17 16:51:49 <nmeum> reboot didn't help either unfourtunatly
2022-03-17 16:56:40 <ikke> clandmeter: someone mentioned in the gitlab irc channel that due to cleanup issues they have temporarily disabled artifact cleanups
2022-03-17 17:01:01 <psykose> ikke: i noticed that on a lot of ci runs, that half of them say not cleaning volumes
2022-03-17 17:01:03 <psykose> i wonder if it's related
2022-03-17 17:01:35 <ikke> That message has been there before already
2022-03-17 17:01:40 <psykose> ah
2022-03-17 17:02:12 <ikke> https://gitlab.com/groups/gitlab-org/-/epics/7097
2022-03-17 17:08:11 <psykose> ah
2022-03-17 17:08:13 <psykose> that is quite the huge thread
2022-03-17 17:08:16 <ikke> yes
2022-03-17 17:09:15 <ikke> psykose: https://zabbix.alpinelinux.org/history.php?action=showgraph&itemids%5B%5D=29887 check the last 3 months
2022-03-17 17:12:18 <psykose> seems the past month is where it all goes up
2022-03-17 17:12:22 <psykose> before that is a void
2022-03-17 17:16:20 <ikke> 60.4 GB          alpine/aports
2022-03-17 17:19:41 <psykose> that looks like it needs a trim
2022-03-17 17:19:48 <psykose> i assume that is stray srcdir?
2022-03-17 17:22:54 <ikke> build artifacts / logs for projects
2022-03-17 17:23:02 <ikke> that's gitlab, not a builder
2022-03-17 17:23:16 <psykose> ah
2022-03-17 17:28:07 <ikke> psykose: using this recipe to clean up the artifacts: https://docs.gitlab.com/ee/administration/job_artifacts.html#delete-job-artifacts-and-logs-from-jobs-completed-before-a-specific-date
2022-03-17 17:28:41 <psykose> > Before running them, we highly recommend seeking guidance from a Support Engineer
2022-03-17 17:29:09 <psykose> looks good
2022-03-17 17:29:28 <ikke> yolo
2022-03-17 17:29:36 <psykose> yolo!
2022-03-17 17:29:41 <psykose> i would keep the logs though
2022-03-17 17:30:34 <psykose> which is the one above that one
2022-03-17 17:30:48 <psykose> bit late though i think :)
2022-03-17 17:36:55 <ikke> kunkku: ping
2022-03-17 17:37:19 <ikke> What external ports do we need to have open externally for dmvpn (a spoke)
2022-03-17 17:42:45 <ikke> I just realized I can verify what awall allows
2022-03-17 17:44:36 <psykose> firewalls are documentation :)
2022-03-17 17:44:52 <ikke> :)
2022-03-17 17:54:24 <ikke> I'm in contact with IBM regarding the firewall ports
2022-03-17 18:10:47 <psykose> hope they don't take weeks to respond
2022-03-17 18:36:45 <ikke> it's taking quite some time, even for a smaller project
2022-03-17 18:37:04 <psykose> mm
2022-03-17 18:37:05 <psykose> sad
2022-03-17 18:37:30 <ikke> maybe I should start with a larger period
2022-03-17 18:40:58 <ikke> At least something is happening :D
2022-03-17 18:42:01 <psykose> wao
2022-03-17 19:26:00 <ikke> Seems like we dmvpn is having issues (probably due to dmvpn1 being down for a longer period)
2022-03-17 19:30:39 <psykose> are they not fully independent
2022-03-17 19:30:53 <ikke> they?
2022-03-17 19:32:07 <psykose> the two dmvpn hosts
2022-03-17 19:32:40 <ikke> I expect so, but I'm not entirely sure
2022-03-17 19:33:59 <ikke> dmvpn.alpinelinux.org is a round-robin record
2022-03-17 19:36:29 <ikke> same with crl.alpinelinux.org
2022-03-17 19:37:33 <psykose> should be fine then, regardless of dmvpn1 not being friendly
2022-03-17 19:37:39 <psykose> lol
2022-03-17 19:37:44 <psykose> ok, maybe not very fine :)
2022-03-17 19:38:54 <ikke> :D
2022-03-17 19:40:45 <psykose> what happened to that one i wonder
2022-03-17 20:18:29 <ikke> kunkku: ping, we need to migrate dmvpn1 to a new host
2022-03-17 20:18:36 <ikke> kunkku: anything we need to take into account?
2022-03-17 20:30:04 <psykose> heh
2022-03-17 20:34:07 <ikke> psykose: somehow dmvpn1 is important
2022-03-17 20:34:48 <psykose> do things cache the first dns ip for a long time or something and never retry the other one
2022-03-17 20:35:37 <ikke> perhaps
2022-03-17 20:35:51 <ikke> I did remove dmvpn1 now from the records
2022-03-17 20:37:08 <psykose> if nothing is connecting to 1 and everything is fixed now that you removed it then it may indeed be dns
2022-03-17 20:37:50 <ikke> clandmeter: ping
2022-03-17 20:38:25 <clandmeter> sigh
2022-03-17 20:38:34 <clandmeter> just reading the email
2022-03-17 20:40:30 <ikke> clandmeter: where is the openvpn data?
2022-03-17 20:40:38 <clandmeter> on the host
2022-03-17 20:41:04 <ikke> ah, in your homedir
2022-03-17 20:41:16 <clandmeter> can you still access it?
2022-03-17 20:41:31 <ikke> yes, they restored access now
2022-03-17 20:41:34 <ikke> until tomorrow
2022-03-17 20:41:38 <clandmeter> ah ok
2022-03-17 20:41:45 <clandmeter> good :)
2022-03-17 20:41:59 <clandmeter> we could temp just run a simple one on linode
2022-03-17 20:42:06 <ikke> yea
2022-03-17 20:42:09 <clandmeter> think about it later
2022-03-17 20:42:13 <clandmeter> and*
2022-03-17 20:45:05 <ikke> can you think of anything that I should not forget to move off dmvpn1?
2022-03-17 20:45:50 <ikke> I have /etc /home and /var/lib/misc/dmvpn-ca.sqlite3
2022-03-17 20:47:57 <clandmeter> hmm
2022-03-17 20:48:01 <clandmeter> i dont know exactly
2022-03-17 20:48:08 <clandmeter> it was running dmvpn and openvpn iirc
2022-03-17 20:48:32 <clandmeter> i would tar whole var just to be sure
2022-03-17 20:48:42 <clandmeter> srv is empty i guess
2022-03-17 20:49:22 <ikke> yes
2022-03-17 21:01:07 <psykose> hm
2022-03-17 21:11:16 <psykose> it's been reachable all this time :p wonder what the retry interval is
2022-03-17 21:13:59 <ikke> 5 minutes
2022-03-17 21:14:32 <ikke> https://zabbix.alpinelinux.org/items.php?form=update&hostid=10255&itemid=28569&context=host
2022-03-17 21:19:40 <psykose> ah
2022-03-17 21:19:53 <psykose> wonder why it hasn't fixed itself
2022-03-17 21:21:40 <ikke> https://zabbix.alpinelinux.org/history.php?action=showgraph&itemids%5B%5D=28571
2022-03-17 21:21:48 <ikke> seems to timeout intermittently
2022-03-17 21:27:08 <psykose> when i curl it
2022-03-17 21:27:10 <psykose> and it takes 'forever'
2022-03-17 21:27:16 <psykose> i notice cgit.cgi is at 100% cpu
2022-03-17 21:27:56 <psykose> i am not sure if a) all requests are limited to one thread of that one cgit script or b) one request just hangs really long while burning cpu for that time
2022-03-17 21:27:59 <psykose> or both, and it blocks everything
2022-03-17 21:29:15 <psykose> but normally it's not even in htop output at all, when making random curls
2022-03-17 21:29:34 <psykose> and then occasionally is just 100
2022-03-17 21:29:49 <psykose> either any non-trivial request just takes 5 seconds and blocks everything or there's some specific dos
2022-03-17 21:29:50 <ikke> maybe someone making a request that is expensive?
2022-03-17 21:29:54 <psykose> it can be
2022-03-17 21:30:06 <psykose> i think a lot of the git requests that are specific are 'expensive'
2022-03-17 21:30:12 <psykose> but normally that should be fine if its parallel
2022-03-17 21:30:22 <psykose> i forget if it has any amount of multithreading for the action
2022-03-17 21:30:34 <psykose> i would assume so, because it's read-only, and no state is being modified in these requests
2022-03-17 21:30:40 <psykose> maybe we just have a bad config, or cgit is just that bad
2022-03-17 21:30:44 <psykose> i will have to read a bit tomorrow
2022-03-17 21:31:59 <psykose> ah
2022-03-17 21:32:17 <psykose> our uwsgi CMD does not have --async passed to it
2022-03-17 21:32:29 <psykose> https://uwsgi-docs.readthedocs.io/en/latest/Async.html anything from here
2022-03-17 21:32:36 <psykose> with multiple potential engines
2022-03-17 21:32:42 <psykose> we really are running it without concurrency
2022-03-17 21:32:57 <psykose> unless it's in the ini
2022-03-17 21:33:09 <psykose> it's not
2022-03-17 21:33:16 <psykose> i don't think it uses ugreen automatically ..?
2022-03-17 21:33:57 <ikke> I think clandmeter disabled something regarding that
2022-03-17 21:34:26 <clandmeter> i tried to play with async before
2022-03-17 21:34:31 <clandmeter> but we also had issues
2022-03-17 21:35:15 <clandmeter> psykose: but please try it again, whatever works is fine by me
2022-03-17 21:35:34 <ikke> clandmeter: I'ved deployed a new node on linode
2022-03-17 21:35:42 <clandmeter> ok good
2022-03-17 21:35:46 <clandmeter> thanks
2022-03-17 21:36:04 <clandmeter> i guess you will need to adjust dns 
2022-03-17 21:36:15 <ikke> Now figuring out how to properly set it up there
2022-03-17 21:38:47 <psykose> restarted with async 4
2022-03-17 21:39:55 <ikke> I'll have to continue tomorrow
2022-03-17 21:40:32 <psykose> same
2022-03-17 21:41:33 <psykose> not sure how much it helps, still seems to be a bit slow when one cgit.cgi is at 100%
2022-03-17 21:41:44 <psykose> i will look more into this mess tomorrow, someone must have figured it out a bit better than me
2022-03-17 21:51:12 <psykose> ah, i think i fixed it
2022-03-17 21:51:18 <psykose> :)
2022-03-17 21:51:25 <psykose> the async shit is <2.0 stuff
2022-03-17 21:51:33 <psykose> workers= is the new one
2022-03-17 21:51:46 <psykose> now it's more than one thread and everything resolves even while one cgit is at 100%
2022-03-17 21:51:59 <psykose> the issue is there can be multiple '100% cgits'
2022-03-17 21:52:02 <psykose> so it can dos the box
2022-03-17 21:52:03 <psykose> kinda
2022-03-17 21:52:14 <psykose> i set it to 3
2022-03-17 21:52:18 <psykose> 4 cores, should be ok
2022-03-17 21:52:42 <psykose> as for how to make cgit not awfully slow, that is a separate task for actually tomorrow
2022-03-17 21:52:53 <psykose> i'm a bit too much of a mess for today
2022-03-17 21:52:55 <psykose> goodnight
2022-03-17 22:36:07 <Hello71> shouldn't it be trivial to figure out the current requests?
2022-03-17 22:36:39 <Hello71> grep -z PATH_INFO /proc/*/environ | tr '\0' '\n'
2022-03-17 22:41:33 <psykose> they are all logged anyway
2022-03-17 23:12:41 <Hello71> that doesn't necessarily tell you what the current requests are though
2022-03-17 23:13:14 <Hello71> logs would be useful if they had processing duration, but i think typically http logs don't have that
2022-03-18 10:07:04 <nmeum> hmhm
2022-03-18 10:07:09 <nmeum> I think I fixed my wireguard issue from yesterdayd
2022-03-18 10:07:51 <nmeum> https://tpaste.us/5kwX
2022-03-18 10:07:55 <nmeum> this looks good, right?
2022-03-18 10:11:39 <nmeum> how do I reach the ppc64le machine now? (:
2022-03-18 10:17:25 <ikke> yes, looks good
2022-03-18 10:17:29 <ikke> what was the issue?
2022-03-18 10:18:49 <nmeum> I am afraid to say
2022-03-18 10:18:57 <nmeum> I entered the wrong public key under [Peers] in the config m)
2022-03-18 10:20:38 <ikke> nmeum: apparently the host is completely unreachable atm
2022-03-18 10:20:51 <nmeum> the ppc64le host?
2022-03-18 10:27:08 <ikke> yes
2022-03-18 10:27:36 <nmeum> unfourtunate, any idea when it will be back up?
2022-03-18 10:28:12 <ikke> not sure, I'll ping IBM
2022-03-18 10:31:25 <nmeum> thanks, I guess I will report the ppc64le issue to Go upstream without investigating it myself further for now in the meantime
2022-03-18 12:10:33 <ncopa> nmeum: :D I think we all have  been there with the wrong key in the hand...
2022-03-18 12:14:41 <psykose> git.a.o seems a lot faster now that it's not one thread for all the requests
2022-03-18 12:24:44 <clandmeter> psykose: 👍
2022-03-18 12:30:34 <psykose> i also noticed the git:// clone uri doesn't work
2022-03-18 12:30:42 <psykose> it's maybe a firewall issue given the random port
2022-03-18 12:30:52 <psykose> (9418)
2022-03-18 12:31:06 <psykose> but i think it's fine to also remove from the display given https clone works fine
2022-03-18 12:31:12 <psykose> and gitlab is preferred anyway
2022-03-18 12:36:05 <psykose> also updated all the containers for it since they were still on like 3.10
2022-03-18 12:39:46 <ikke> "our new gateway is misbehaving"
2022-03-18 12:39:50 <ikke> re ppc64le
2022-03-18 12:58:01 <nmeum> ok, so at least they are aware of the issue
2022-03-18 12:58:13 <ikke> Yes
2022-03-18 13:00:31 <psykose> strangely, if i make the uwsgi container 3.15 https cloning breaks and always returns a 429 too many requests
2022-03-18 13:00:35 <psykose> on 3.14 it works fine
2022-03-18 13:00:40 <psykose> the only difference is the git version
2022-03-18 13:01:03 <psykose> 2.32.0, 2.34.1
2022-03-18 13:01:17 <psykose> or whatever else transient dependency i guess
2022-03-18 16:18:44 <psykose> oh nice
2022-03-18 16:18:49 <psykose> it's back
2022-03-18 16:21:51 <ikke> Yes, but no ssh yet
2022-03-18 16:22:01 <ikke> I suspect due to MTU
2022-03-18 16:23:49 <psykose> i reached it with ssh, just not added to the host so got a password prompt
2022-03-18 16:25:22 <ikke> hmm, interesting
2022-03-18 16:25:57 <ikke> psykose: do you by chance have some lower MTU by default?
2022-03-18 16:26:18 <psykose> i dunno, i assume it goes through wg, which is '1420'
2022-03-18 16:26:26 <psykose> my regular eth is 1500
2022-03-18 16:26:44 <psykose> i don't think it affects anything, as the mtu is only to the router?
2022-03-18 16:27:14 <ikke> mtu is over the entire path
2022-03-18 16:27:17 <psykose> ah
2022-03-18 16:30:06 <ikke> yes, it works if I manually set a lower MTU for that route
2022-03-18 16:31:13 <psykose> strange
2022-03-18 16:32:41 <ikke> looks like packets that are too large >1464 are dropped
2022-03-18 16:33:36 <psykose> i don't think i did anything special when setting up wg
2022-03-18 16:34:09 <ikke> I think the mtu is automatically lowered to compensate
2022-03-18 16:36:17 <Hello71> only for wg-quick
2022-03-18 16:37:09 <Hello71> wg defaults to 1420
2022-03-18 16:39:16 <Hello71> i think that assumes 1500 - ipv6 - udp - wg, but if you use, say, 6in4 then it's too high
2022-03-18 16:39:57 <ikke> nmeum: you can try to see if you can access your container
2022-03-18 16:40:39 <ikke> but 1420 is already enough not run into the MTU issue with the ppc64le host
2022-03-18 17:44:59 <psykose> poof
2022-03-18 17:45:33 <ikke> sadtrombone
2022-03-18 17:47:12 <ikke> And I realize we have the same host twice in zabbix
2022-03-18 17:47:23 <psykose> yes, i almost noticed that too
2022-03-18 17:47:41 <psykose> and a duplicate dns entry too
2022-03-18 17:47:56 <ikke> That might be due to agent auto registration
2022-03-18 17:47:57 <rails> oop
2022-03-18 17:48:03 <rails> ello folks
2022-03-18 17:48:20 <ikke> no, we don't have any autoregistration setup
2022-03-18 17:48:22 <ikke> rails: hi
2022-03-18 18:55:48 <nmeum> ikke: which port should i use?
2022-03-18 18:56:06 <nmeum> 22061 gives me "ssh: connect to host 172.16.11.61 port 22061: Connection refused"
2022-03-18 18:58:15 <psykose> as user nmeum? (or whatever user)
2022-03-18 18:58:47 <ikke> nmeum: just 22
2022-03-18 18:58:57 <ikke> 22061 is forwarded to 22 
2022-03-18 18:59:03 <nmeum> that prompts me for a password
2022-03-18 18:59:23 <nmeum> nmeum@172.16.11.61 that is
2022-03-18 18:59:36 <nmeum> ahahh!!
2022-03-18 18:59:39 <nmeum> it's root@
2022-03-18 18:59:47 <nmeum> right right
2022-03-18 18:59:50 <nmeum> I can reach it :)
2022-03-18 18:59:52 <ikke> nice
2022-03-18 19:00:09 <nmeum> great, thanks a lot!
2022-03-18 19:00:41 <nmeum> will look into the go stuff tomorrow then
2022-03-18 19:07:05 <ikke> ok, apparently they temporarily moved back to the old FW 
2022-03-18 19:07:54 <psykose> their hosting is more messy than our infrastructure
2022-03-18 19:08:09 <psykose> i'm still confused why these issues are so persistent on this one singular host
2022-03-18 19:08:22 <psykose> it has been sooo long
2022-03-18 19:36:34 <ikke> we lost dmvpn1 again, let's hope I got everything off it
2022-03-18 19:36:46 <ikke> off of it*
2022-03-18 20:18:18 <ikke> clandmeter: I _think_ I finished installing the new dmvpn1
2022-03-18 20:36:00 <ikke> I've manually pointed gbr2 to the new host, it does establish a conneciton
2022-03-18 20:36:02 <ikke> connection
2022-03-18 20:37:22 <ikke> :)
2022-03-18 20:40:48 <psykose> sure do be establishing
2022-03-18 20:41:17 <psykose> wao it even has ipv6
2022-03-18 20:41:17 <psykose> fancy
2022-03-18 20:42:06 <ikke> what?
2022-03-18 21:45:04 <ikke> Ok, lets put the new host in action
2022-03-18 21:45:07 <ikke> hope I don't break anything
2022-03-18 21:45:12 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/21
2022-03-18 21:47:18 <psykose> didn't the crl record exist before
2022-03-18 21:47:25 <psykose> tf claims it is new and made
2022-03-18 21:47:26 <psykose> hah
2022-03-18 21:47:35 <ikke> I removed the old A record
2022-03-18 21:47:37 <psykose> ah
2022-03-18 21:50:24 <ikke> vtysh is quite cool (wasn't aware of it)
2022-03-18 21:51:32 <psykose> this looks far too advanced for me
2022-03-18 21:52:52 <ikke> I have experience with cisco routers, this interface is similar
2022-03-18 21:53:41 <ikke> first bgp session
2022-03-18 21:53:50 <ikke> 172.16.0.21     4 65000       7      10        0    0    0 00:00:45        1
2022-03-18 21:59:12 <ikke> The other hub joined now as well:
2022-03-18 21:59:14 <ikke> 172.16.0.254    4 65000      24      12        0    0    0 00:00:03       31
2022-03-19 08:25:23 <ikke> Everything did not explode (yet) :)
2022-03-19 08:25:53 <ikke> > Total number of neighbors 15
2022-03-19 08:26:06 <ikke> > Total num. of routes received     42
2022-03-19 13:33:13 <clandmeter> ikke: thx for following things up
2022-03-19 13:33:36 <clandmeter>  seen some emails in my mailbox
2022-03-19 13:33:58 <ikke> clandmeter: do we want to wipe the old dmvpn1?
2022-03-19 13:34:03 <ikke> I do still have access
2022-03-19 13:34:10 <psykose> it seems pretty gone to me
2022-03-19 13:34:21 <psykose> if you have some spare storage you could take a disk .img and keep it around
2022-03-19 13:34:27 <clandmeter>  maybe smart
2022-03-19 13:34:48 <clandmeter> I did it before
2022-03-19 13:35:51 <clandmeter>  i hope equinix will reply
2022-03-19 13:36:34 <ikke> clandmeter: btw, IBM invited me to a slack regarding ppc64le, should I ask them to invite you as well?
2022-03-19 16:27:10 <ikke> I have a disk image of /dev/sda3 of dmvpn1
2022-03-19 16:27:29 <ikke> dd if=/dev/sda3 >dmvpn1-sda3.img (over ssh)
2022-03-19 17:11:36 <psykose> that works
2022-03-19 20:48:29 <mps> how much space is allowed for me to use on dev.a.o
2022-03-19 20:49:51 <mps> with yesterday asahi alpha release announce people starting to ask how to install alpine on apple silicon (M1) machines
2022-03-19 20:50:27 <mps> and I'm thinking to create bootable image for download
2022-03-19 20:51:59 <mps> one user installed alpine today following my guides but it will be annoying to repeat for every new user
2022-03-19 20:53:46 <psykose> the regular aarch64 is 160, it's probably less than 200
2022-03-19 20:53:54 <psykose> to make a .iso
2022-03-19 20:54:07 <psykose> i can't tell you the limits but that sounds quite fair
2022-03-19 20:54:54 <psykose> feel free to go for it :)
2022-03-19 20:55:35 <mps> psykose: thanks but I want infra master confirmation
2022-03-19 20:55:51 <psykose> 
ACTION wears the infra master hat

2022-03-19 20:55:58 <psykose> 
ACTION emits in a deep grumbly voice "yeah go for it"

2022-03-19 20:55:59 <psykose> :)
2022-03-19 20:56:00 <mps> :D
2022-03-19 20:57:07 <mps> will create it tomorrow and as last option upload to my server with slow link
2022-03-19 22:43:23 <clandmeter> Go ahead:)
2022-03-19 22:46:44 <mps> clandmeter: thank you master ;)
2022-03-20 09:16:14 <kunkku> ikke: did you already complete the migration of dmvpn1?
2022-03-20 09:16:25 <ikke> kunkku: I think so
2022-03-20 09:17:05 <ikke> dmvpn1.a.o points to the new server
2022-03-20 09:17:30 <ikke> and has an A record in dmvpn.a.o
2022-03-20 09:19:20 <kunkku> sorry for not responding on time
2022-03-20 09:19:36 <kunkku> but probably everything went fine :)
2022-03-20 09:21:04 <ikke> I see the BGP sessions from the other hub and spokes
2022-03-20 09:21:10 <ikke> crl seems to be up-to-date
2022-03-20 09:22:17 <kunkku> did you set up the DB sync to dmvpn2?
2022-03-20 09:22:35 <ikke> I enabled the in-sync service
2022-03-20 09:22:39 <kunkku> ok good
2022-03-20 09:25:56 <kunkku> and seems the network kept working during the migration
2022-03-20 09:26:39 <ikke> yes
2022-03-20 10:17:48 <clandmeter> ikke: you can add me if you like to slack, with my gmail address preferable 
2022-03-20 10:20:57 <ikke> kunkku: one thing I did notice: after the old dmvpn1 was unreachable for about 1 day, things started to become unreachable
2022-03-20 10:22:52 <clandmeter> you are fast
2022-03-20 10:24:57 <ikke> are you able to ssh into deu5?
2022-03-20 10:28:00 <clandmeter> me?
2022-03-20 10:28:05 <ikke> yes
2022-03-20 10:28:10 <ikke> I mean, usa5
2022-03-20 10:28:28 <clandmeter> usa5-dev1?
2022-03-20 10:28:45 <ikke> yes
2022-03-20 10:29:00 <clandmeter> nope
2022-03-20 10:29:04 <clandmeter> cannot ping it 
2022-03-20 10:29:06 <ikke> I do have console access
2022-03-20 10:29:13 <clandmeter> which host is it?
2022-03-20 10:29:17 <ikke> oh, ping does work for me
2022-03-20 10:29:22 <ikke> risv64 builder
2022-03-20 10:29:43 <clandmeter> ah ok
2022-03-20 10:29:56 <clandmeter> the unstable one
2022-03-20 10:29:59 <clandmeter> you can ping it?
2022-03-20 10:30:01 <ikke> yes
2022-03-20 10:30:02 <clandmeter> i cannot 
2022-03-20 10:30:11 <ikke> 64 bytes from usa5-dev1.alpinelinux.org (147.75.77.241): icmp_seq=1 ttl=53 time=81.2 ms
2022-03-20 10:30:37 <clandmeter> --- usa5-dev1.alpinelinux.org ping statistics ---
2022-03-20 10:30:37 <clandmeter> 31 packets transmitted, 0 received, 100% packet loss, time 30727ms
2022-03-20 10:32:50 <ikke> ok, console is now hanging
2022-03-20 10:33:07 <ikke> suppose I reboot it again
2022-03-20 10:33:09 <clandmeter> i guess just kick the machine
2022-03-20 10:33:18 <clandmeter> did you reboot it already ones?
2022-03-20 10:33:33 <ikke> yes, couple of weeks ago
2022-03-20 10:33:45 <clandmeter> oh ok that long ago
2022-03-20 10:33:50 <clandmeter> yes its unstable
2022-03-20 10:34:50 <clandmeter> I was hoping ed would respond to my email so we could go over that box and find a long term solution
2022-03-20 10:36:11 <clandmeter> ikke: did they reply on changing the email address of our acc?
2022-03-20 10:36:38 <ikke> "I'll update the maintenance email straight away. "
2022-03-20 10:36:48 <clandmeter> ah ok
2022-03-20 10:36:53 <clandmeter> good
2022-03-20 10:37:06 <clandmeter> maybe we have more luck on following things up :)
2022-03-20 10:39:45 <ikke> ok, can ssh again
2022-03-20 13:54:32 <kunkku> ikke: the CRL is valid for 24 hours
2022-03-20 13:54:47 <ikke> is it only one node that generates it?
2022-03-20 13:54:55 <ikke> crl.a.o points to both hubs
2022-03-20 13:55:24 <kunkku> for how long was dmvpn1 unavailable?
2022-03-20 13:56:14 <ikke> Just a little over 24h
2022-03-20 13:56:32 <ikke> 29h
2022-03-20 13:58:35 <kunkku> yes, currently dmvpn2 just mirrors the CRL generated on dmvpn1
2022-03-20 13:58:43 <ikke> ok
2022-03-20 13:59:27 <ikke> That's done by a cron-job, correct?
2022-03-20 13:59:57 <kunkku> yes
2022-03-20 14:00:04 <ikke> dmvpn-crl-update
2022-03-20 14:00:26 <kunkku> correct
2022-03-20 14:01:39 <kunkku> should we increase the validity period?
2022-03-20 14:03:54 <ikke> is it easy to temporarily enable the crl generation on dmvpn2 if necessary?
2022-03-20 18:46:13 <kunkku> ikke: yes, you have to install the dmvpn-ca package and comment out the line in /etc/dmvpn-crl-dp.conf
2022-03-20 18:47:13 <ikke> ok, then I guess we can just do that the next time dmvpn1 for some reason is unreachable for a longer time
2022-03-20 19:01:33 <kunkku> sudo does not work for me on new dmvpn1
2022-03-20 19:02:10 <ikke> Ah, did not set that up yet
2022-03-20 19:02:49 <ikke> kunkku: can you try doas?
2022-03-20 19:04:31 <ikke> `doas -s` works for me
2022-03-20 22:55:37 <kunkku> ok, I'll use doas then
2022-03-20 23:21:54 <psykose> is it safe to reboot deu7-dev1 for a kernel upgrade
2022-03-21 05:37:45 <ikke> psykose: should be, yes
2022-03-21 12:03:20 <psykose> ok, done
2022-03-21 14:40:08 <mps> linux-edge is not linux-lts and I don't understand why we have it in aports if users wants to be same as linux-lts
2022-03-21 14:40:39 <mps> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/32294
2022-03-21 14:41:28 <ikke> mps: users sometimes need a newer version as lts provides. linux-edge appears to provide that. 
2022-03-21 14:41:45 <mps> clandmeter: 5.17 is released with fix for riscv64 reboot, will upgrade it in evening or tomorrow
2022-03-21 14:42:43 <ikke> But then they notice some features that lts has are missing 
2022-03-21 14:42:46 <mps> ikke: yes, we discussed this things and iirc you agreed with me then that they are not intended to be same
2022-03-21 14:43:41 <mps> and I'm not against enabling RBD in above MR, just have dilema
2022-03-21 14:44:09 <mps> contributor didn't explained/told why s/he need this
2022-03-21 14:45:20 <ikke> Seems to be related to CEPH 
2022-03-21 14:45:30 <mps> yes
2022-03-21 14:46:07 <mps> but wonder that people use unstable (linux-edge) for such tasks as CEPH
2022-03-21 14:46:42 <mps> ok, will enable it, who cares for stability nowadays ;)
2022-03-21 14:46:56 <mps> another something
2022-03-21 14:47:42 <mps> people on asahi channels ask me to add linux-asahi to testing. what do you (all) think about this
2022-03-21 14:48:45 <mps> maybe I should ask on #alpine-linux
2022-03-21 14:50:27 <psykose> seems fine to me
2022-03-21 14:52:24 <mps> psykose: somehow I knew your answer, thanks ;)
2022-03-21 14:52:30 <psykose> :p
2022-03-21 14:52:36 <psykose> tis but a single kernel for aarch64
2022-03-21 14:52:40 <psykose> have fun with it
2022-03-21 14:52:45 <psykose> i expect it will just be asahi master?
2022-03-21 14:53:02 <mps> psykose: we have linux-gru and linux-elm for more than a year
2022-03-21 14:53:06 <psykose> mhm
2022-03-21 14:53:50 <mps> I use this for work and for testing slimmed down and tailored for specific SoCs
2022-03-21 14:53:59 <mps> s/this/these/
2022-03-21 14:53:59 <alpine-meetbot> mps meant to say: I use these for work and for testing slimmed down and tailored for specific SoCs
2022-03-21 14:54:50 <mps> temped to add linux-sunxi
2022-03-21 16:12:47 <psykose> FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION seems to be a new 5.17 options that might be useful when lts rolls to next release
2022-03-21 16:12:56 <psykose> maybe then the things that don't boot with simpledrm will work
2022-03-21 16:15:58 <ikke> mps: What is confusing to users is that linux-edge appears to be just a more up-to-date version of linux-lts, not a completely different kernel with a different feature set. So if users are facing some kind of bug and want to check if it's fixed in the latest version of linux, they think they can switch to edge to test it, but then notice that some feature they rely on is
2022-03-21 16:16:00 <ikke> missing.
2022-03-21 16:18:21 <ikke> I don't think people expect edge to have the exact same configuration, but what they generally do not expact is that functionality is missing
2022-03-21 16:20:21 <psykose> i am not sure what to do about it either, but there should be a -current or something that is exactly that, just -lts but currentversion with the same new options picked for whatever new drivers get added
2022-03-21 16:21:15 <psykose> and -edge should be for.. testing or something, but given that simpledrm and kernel zstd and compressed modules were first done in -lts without going through -edge first (and we broke a lot of things by doing those without testing them first) i am not sure what niche edge fulfills
2022-03-21 16:21:23 <psykose> it is a better config for desktops, but it's not named as such
2022-03-21 16:23:14 <ikke> Right, but even using it for testing mostly implies trying out new features, not a completely different featureset
2022-03-21 16:23:23 <psykose> indeed
2022-03-21 16:25:21 <mps> psykose: re desktop, yes that was my idea when I added it and it proven to be better on servers also as a extra feature
2022-03-21 16:26:39 <mps> psykose: I already tested FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION, not sure it is worth to enable but probably will be ok for x86_64
2022-03-21 16:26:57 <psykose> it's unused entirely if it's not an old device that tries to load fbdev per the description
2022-03-21 16:27:08 <psykose> so i'm not sure how to test unless you have some 2004 x86 pc
2022-03-21 16:27:23 <psykose> ..or something
2022-03-21 16:27:25 <psykose> maybe i'm wrong
2022-03-21 16:27:40 <mps> also I don't have machine to test it
2022-03-21 16:27:46 <psykose> but maybe it's also used for some of the missing fbdev drivers with simpledrm
2022-03-21 16:28:00 <psykose> iirc there was something like some xen fbdev thing that didn't work with simpledrm either, but i forget now
2022-03-21 16:28:29 <mps> simpledrm works fine without it on apple M1
2022-03-21 16:28:41 <psykose> i would expect a brand new platform to work with simpledrm :p
2022-03-21 16:28:45 <psykose> as does everything i own
2022-03-21 16:28:49 <psykose> just hard to track down the things that don't
2022-03-21 16:29:01 <mps> yes
2022-03-21 17:08:30 <minimal> psykose: there was someone having problems about 1 month ago on a server with AST IPMI/BMC and DRM - I think the kernel at that time was missing a DRM driver that supported that particular BMC
2022-03-21 17:09:01 <psykose> yeah, that's the one i'm referring to i think
2022-03-21 17:09:05 <psykose> there was a gitlab issue for it
2022-03-21 17:11:41 <minimal> I also think there used to be (not sure if since fixed) and issue with either Nvidia's own binary driver or nouveau with SimpleDRM
2022-03-22 13:50:49 <mps> psykose: I'm against having linux-pam by default in alpine. you understand rest I'm sure
2022-03-22 13:50:55 <psykose> it is already there
2022-03-22 13:51:03 <psykose> and util-linux-login is not the default login
2022-03-22 13:51:05 <psykose> that one is from busybox
2022-03-22 13:51:08 <mps> I know, and this is bad imo
2022-03-22 13:51:13 <psykose> then make an mr to remove it
2022-03-22 13:51:18 <psykose> the curent configuration does not make sense
2022-03-22 13:51:21 <psykose> it is enabled and not configured
2022-03-22 13:51:49 <mps> linux-pam is pulled by util-linux-login
2022-03-22 13:52:03 <psykose> yes, because it is built against linux-pam
2022-03-22 13:52:12 <psykose> and then it cannot even use it, because the actual pam file is missing
2022-03-22 13:52:13 <mps> and util-linux-login is pulled by util-linux-misc
2022-03-22 13:52:45 <psykose> util-linux-misc does not pull login
2022-03-22 13:53:02 <psykose> https://img.ayaya.dev/B2PPgktqZHk3.png
2022-03-22 13:53:42 <psykose> *nothing* pulls util-linux-login or shadow-login
2022-03-22 13:53:52 <psykose> this is entirely opt in for people that want a pam login
2022-03-22 13:53:59 <mps> actually imo would be better to make util-linux-login not related to util-linux-misc
2022-03-22 13:54:05 <psykose> how is it related already?
2022-03-22 13:54:16 <psykose> they are completely distinct
2022-03-22 13:54:32 <mps> apk info -r util-linux-login
2022-03-22 13:54:43 <psykose> empty?
2022-03-22 13:54:53 <mps> huh
2022-03-22 13:55:17 <mps> util-linux-login-2.37.4-r0 is required by: util-linux-misc-2.37.4-r0
2022-03-22 13:55:30 <psykose> this is not the case on latest edge
2022-03-22 13:55:45 <mps> aha, lets update
2022-03-22 13:55:51 <psykose> or even 3.15
2022-03-22 13:56:02 <psykose> maybe you have just `util-linux` installed?
2022-03-22 13:56:19 <psykose> that pulls everything
2022-03-22 13:56:25 <psykose> or no, maybe just misc
2022-03-22 13:56:29 <psykose> either way, nothing depends on these pam logins
2022-03-22 13:56:45 <mps> psykose: nice ;)
2022-03-22 13:56:59 <mps> sorry for nonsense talk
2022-03-22 13:57:03 <psykose> is ok
2022-03-22 13:57:34 <mps> I have no problem with this MR anymore :)
2022-03-22 13:58:25 <mps> btw, I'm preparing linux 5.17 upgrade, any wishlist?
2022-03-22 13:58:44 <psykose> i looked at the new options and everything just looks like the usual new driver soup
2022-03-22 13:58:47 <psykose> nothing for me i don't think
2022-03-22 13:59:50 <mps> I intentionally didn't enabled sone new drivers, thinking it will be better if someone need them to tell us
2022-03-22 14:00:23 <mps> 'us' because I don't feel I'm owner of linux-edge
2022-03-22 14:02:00 <mps> received mail this morning from Jason Donefeld (wg upstream) (and I see Ariadne in receivers list) about rng but think we have to talk about this later
2022-03-22 14:02:20 <psykose> what kind of rng
2022-03-22 14:03:05 <mps> new implementation in 5.17 (use blake2 instead of sha1)
2022-03-22 14:03:38 <mps> psykose: https://www.zx2c4.com/projects/linux-rng-5.17-5.18/
2022-03-22 14:03:43 <psykose> ah
2022-03-22 14:04:05 <mps> tl; but read if you have time and interested
2022-03-22 14:04:35 <psykose> sounds like a good idea to me
2022-03-22 14:04:40 <mps> also list what distros do https://docs.google.com/spreadsheets/d/16shc-OEpzMWkYDrB9XmnRMmuW90CpvBXildme_TL3II/edit#gid=0
2022-03-22 14:04:51 <psykose> looks like 5.18 to me though
2022-03-22 14:05:14 <mps> blake2 is now default in 5.17
2022-03-22 14:05:35 <psykose> ah
2022-03-22 14:05:51 <mps> CONFIG_CRYPTO_ARCH_HAVE_LIB_BLAKE2S=y is default
2022-03-22 14:06:33 <mps> you are rught this is first step and in 5.18 it will be 'finish'
2022-03-22 14:06:49 <psykose> right, the rest of the changes are next version
2022-03-22 14:07:04 <psykose> also HAVE_LIB sounds like 'have' not 'use'
2022-03-22 14:07:32 <mps> I didn't yet found time to read all docs so not sure
2022-03-22 14:20:35 <minimal> psykose: one thing I noticed in the past about util-linux-login is that it defines replaces="$pkgname-misc" (i.e. util-linux-misc) in the APKBUILD
2022-03-22 14:20:48 <minimal> I don't really understand the purpose of that
2022-03-22 14:21:34 <ikke> minimal: the only thing that does is tell APK it's all right if packages have conflicting files, the packagine replacing the other would now own it
2022-03-22 14:24:49 <psykose> ye
2022-03-22 14:24:51 <minimal> ikke: ok, so I guess this covers the move of lslogins and nolign from util-linux-misc (prior to Edge) to util-linux-login
2022-03-22 14:25:03 <psykose> yeah
2022-03-22 14:25:09 <psykose> so on upgrade it doesn't error
2022-03-22 19:22:36 <mps> nmeum: clandmeter: linux-edge 5.17 is merged, you can test it on riscv64 and (if something is bad) report
2022-03-22 19:23:12 <mps> and anyone else who have riscv64 boards, ofc
2022-03-22 19:24:03 <mps> would be nice to know if it fails before cherry-pick to 3.15-stable
2022-03-23 09:11:55 <mps> I see that the nodejs is upgraded and that reminded me about this article https://www.technologyreview.com/2022/03/21/1047489/activists-are-targeting-russians-with-open-source-protestware
2022-03-23 09:13:05 <mps> I don't use nodejs (thanks god) but reading above article I'm scared with using FOSS (maybe this is intent on these 'actions') idk
2022-03-23 09:13:31 <mps> now I fear to upgrade anything
2022-03-23 11:18:44 <clandmeter> mps: did arm32 need a different initramfs? something with compression?
2022-03-23 11:21:04 <mps> clandmeter: I don't think so, for me it always worked as it is
2022-03-23 11:21:26 <clandmeter> mps: have you tried latest armv7 virt in qemu?
2022-03-23 11:21:36 <clandmeter> it panics on /init
2022-03-23 11:22:16 <mps> clandmeter: `file /boot/initramfs-edge` => /boot/initramfs-edge: gzip compressed data, max compression, from Unix, original size modulo 2^32 8499476
2022-03-23 11:22:52 <mps> clandmeter: you mean official armv7 iso?
2022-03-23 11:22:58 <clandmeter> yes
2022-03-23 11:23:08 <mps> it never worked for me
2022-03-23 11:23:50 <mps> because this I created separate one https://dev.alpinelinux.org/~mps/alpine-virt-210730-armv7.iso
2022-03-23 11:25:06 <mps> and guide is here https://arvanta.net/alpine/install-armv7-qemu/
2022-03-23 11:25:09 <clandmeter> ah they have different configs
2022-03-23 11:25:28 <mps> clandmeter: lts -virt kernel is problem
2022-03-23 11:26:11 <mps> BDFL don't like my style of work so I didn't fixed lts kernel
2022-03-23 11:27:09 <mps> clandmeter: did you tried latest (5.17) kernel on riscv64 board?
2022-03-23 14:16:54 <mps> Ariadne: `<ariadne@dereferenced.org>: host mx1.mailbun.net[170.39.20.100] said: 450 4.7.25 Client host rejected: cannot find your hostname, [109.72.52.77]`
2022-03-23 14:17:15 <mps> so my mail couldn't arrive to you
2022-03-23 14:17:35 <mps> I have to fix mail server
2022-03-23 14:18:19 <mps> actually not fix, but force to interface which could work
2022-03-23 18:16:20 <Ariadne> mps: yes, that seems to be due to missing reverse DNS
2022-03-23 18:30:53 <clandmeter> ikke?
2022-03-23 18:31:03 <ikke> clandmeter: ?
2022-03-23 18:32:23 <clandmeter> Linode?
2022-03-23 18:32:52 <ikke> Didn't touch it
2022-03-23 18:32:54 <psykose> me rebooting it
2022-03-23 18:32:57 <ikke> I had the feeling
2022-03-23 18:33:30 <clandmeter> Ok
2022-03-23 18:33:37 <clandmeter>  reason?
2022-03-23 18:35:50 <psykose> new kernel
2022-03-23 18:37:04 <clandmeter> Please next time leave a msg here
2022-03-23 18:37:30 <mps> it is custom to anounce
2022-03-23 18:37:47 <clandmeter>  it's on both systems i guess?
2022-03-23 18:38:29 <psykose> yes
2022-03-23 18:45:07 <clandmeter> Ok thx for getting it done
2022-03-23 18:45:14 <psykose> sorry for not notifying
2022-03-23 18:45:21 <psykose> forgot to do it '>.<
2022-03-23 18:45:39 <clandmeter> np
2022-03-23 18:45:42 <clandmeter>  next time
2022-03-24 11:34:57 <ikke> I suppose we should upgrade to gitlab 14.8
2022-03-24 11:35:00 <ikke> 14.9 is out
2022-03-24 11:39:39 <ncopa> i think you do an excellent work with our gitlab. thank you!
2022-03-24 11:41:46 <mps> well, ikke is excellent per se (and in everything I think)
2022-03-24 11:52:22 <ikke> It's running here: https://gitlab-test.alpinelinux.org/
2022-03-24 12:11:05 <mps> what is limit (size) of upload to tpaste.us? kernel config doesn't fit, I had to use ix.io on #asahi to share it
2022-03-24 12:11:46 <ikke> There is a limit set in the configuration, but I think the effective limit is lower
2022-03-24 17:13:32 <clandmeter> if it is lower it should be the proxy
2022-03-24 17:14:09 <clandmeter> mps: how big is your config?
2022-03-24 17:19:50 <mps> clandmeter: 25KB
2022-03-24 17:20:02 <clandmeter> strange
2022-03-24 20:34:38 <mps> where is list of packages which  setup-disk installs
2022-03-24 20:36:05 <mps> or better, how to tell setup-alpine to install linux-firmware-none only
2022-03-24 20:39:35 <ikke> setup-disk installs whatgs
2022-03-24 20:39:40 <ikke> Installed 
2022-03-24 20:39:52 <ikke> what is currently present
2022-03-24 20:43:58 <mps> I see
2022-03-24 20:44:25 <mps> hmm, so I should remove linux-firmare all
2022-03-24 20:44:58 <mps> hm, apk info doesn't show any linux-firmware
2022-03-24 20:46:10 <mps> and I can't find where on the iso are linux-firmware apks
2022-03-24 21:03:37 <ikke> It's automatically pulled in by linux-* through linux-firmware-any
2022-03-24 21:13:14 <mps> ikke: do you have idea how to disable it with setup-alpine, even if I have to edit some of setup-* scripts
2022-03-24 21:14:24 <Hello71> just apk add linux-firmware-none before setup-alpine
2022-03-24 21:14:28 <ikke> Like you suggested, make sure linux-firmware-none is installed 
2022-03-24 21:17:19 <mps> it doesn't work because before setup-alpine there is no network
2022-03-24 21:21:50 <Hello71> apk add -t linux-firmware-none maybe
2022-03-24 21:22:11 <Hello71> or echo linux-firmware-none >> /etc/apk/world might work
2022-03-24 21:23:23 <mps> aha, makes sense
2022-03-24 21:31:22 <ikke> Hello71: I don't think the virtual package would work, as it would not provide linux-firmware-any
2022-03-24 21:31:31 <Hello71> hm, good point
2022-03-24 21:31:53 <ikke> but perhaps apk add -t linux-firmware-any would
2022-03-24 21:32:14 <Hello71> idk if virtual packages are properly transferred when doing sys install either
2022-03-24 21:32:33 <ikke> no idea either
2022-03-24 21:38:16 <mps> Hello71: ikke: it worked
2022-03-24 21:38:36 <ikke> which option?
2022-03-24 21:38:54 <mps> 18:19 ........... mps| clandmeter: 25KB
2022-03-24 21:38:54 <mps> 18:20 ....@clandmeter| strange
2022-03-24 21:38:54 <mps> 21:34 ........... mps| where is list of packages which  setup-disk installs
2022-03-24 21:38:58 <mps> 21:36 ........... mps| or better, how to tell setup-alpine to install linux-firmware-none only
2022-03-24 21:39:01 <mps> 21:39 ..........@ikke| setup-disk installs whatgs
2022-03-24 21:39:03 <mps> 21:39 ..........@ikke| Installed
2022-03-24 21:39:06 <mps> 21:39 ..........@ikke| what is currently present
2022-03-24 21:39:08 <mps> 21:43 ........... mps| I see
2022-03-24 21:39:11 <mps> 21:44 ........... mps| hmm, so I should remove linux-firmare all
2022-03-24 21:39:13 <mps> uhm
2022-03-24 21:39:24 <mps> sorry, testing xf86-input-mtrack driver
2022-03-24 21:40:05 <mps> echo linux-firmware-none >> /etc/apk/world worked
2022-03-24 21:40:09 <ikke> ok
2022-03-24 21:41:10 <mps> next question is how to set BOOT_SIZE, boot partition size
2022-03-24 21:41:56 <mps> I did export BOOT_SIZE=64 but it created 192M size
2022-03-24 21:42:49 <mps> ok, could edit setup-disk, it is safer
2022-03-25 04:40:43 <psykose> edge-x86 seems to be poofed, along with riscv
2022-03-25 05:35:58 <ikke> x86 is still stuck on bb
2022-03-25 05:37:30 <psykose> oh was it
2022-03-25 05:37:35 <psykose> it was completely gone from the interface
2022-03-25 05:40:14 <ikke> mqtt somehow lost the retained will
2022-03-25 05:41:01 <psykose> that's okay mqtt
2022-03-25 05:41:02 <psykose> me too
2022-03-25 05:42:10 <ikke> :)
2022-03-25 05:51:48 <ikke> had to reboot usa5 (riscv64) again 
2022-03-25 06:16:54 <psykose> and x86 is having a busybox adventure now too
2022-03-25 06:16:59 <psykose> and i bet one restart will fix it again
2022-03-25 06:17:01 <psykose> as is customary
2022-03-25 20:15:59 <ikke> clandmeter: got a response from Ed
2022-03-25 22:23:57 <mps> I made this draft for installing alpine on apple m1 https://arvanta.net/alpine/install-alpine-m1/
2022-03-25 22:24:19 <mps> any help would be appreciated
2022-03-26 17:34:33 <ikke> Planning to upgrade gitlab in an hour
2022-03-26 20:08:18 <ikke> Nice, upgrade to 14.8 and we suddenly get our diskspace back :)
2022-03-26 21:45:56 <psykose> wowee
2022-03-26 21:45:58 <psykose> that easy huh
2022-03-26 22:18:28 <ikke> https://zabbix.alpinelinux.org/history.php?action=showgraph&itemids%5B%5D=29877
2022-03-26 22:18:46 <ikke> from ~35G to ~175G free space
2022-03-26 22:18:51 <ikke> I like those upgrades
2022-03-27 09:45:53 <ikke> have to reboot it again..
2022-03-28 10:19:03 <ncopa> are there cpu graphs of our builders available somewhere? I am interested in todays CPU load on the builders. I think we could use it as example of delayed release due to not enough build power
2022-03-28 10:19:35 <ikke> let me check
2022-03-28 10:21:06 <ncopa> i mean, it could be used as documentation of what happens when there are new kernel updates that are needed in all maintained branches
2022-03-28 10:24:26 <ikke> https://imgur.com/a/Zt57OXY
2022-03-28 10:24:28 <ikke> something like this?
2022-03-28 10:26:29 <ikke> ncopa: or are you specifically referring to usa9 (aarch builder)
2022-03-28 10:27:37 <ncopa> is that x86 builder?
2022-03-28 10:27:40 <ncopa> ah yes it is
2022-03-28 10:27:47 <ikke> x86_64
2022-03-28 10:28:32 <ncopa> im thinking of building cpu usage in general
2022-03-28 10:28:38 <ncopa> so those are nice
2022-03-28 10:28:50 <ncopa> but I am also interested in the usa9 builder
2022-03-28 10:29:33 <ncopa> and im interested in those at the end of the day, or when the build jobs are done
2022-03-28 10:31:13 <mps> ncopa: you think I upgrade kernels too often?
2022-03-28 10:31:27 <ikke> 
ACTION shakes fist at checkpath --directory failing when the provided path ends with /

2022-03-28 10:32:13 <psykose> lol really
2022-03-28 10:32:28 <ikke> * checkpath: unable to open directory: No such file or directory
2022-03-28 10:32:39 <psykose> there's lots of places with trailing / giving completely different meaning
2022-03-28 10:32:51 <psykose> install file /usr/bin/ # puts it in /usr/bin/file
2022-03-28 10:32:58 <psykose> install files /usr/bin # puts it as /usr/bin
2022-03-28 10:32:59 <psykose> haha
2022-03-28 10:33:21 <psykose> i really hate the behaviour
2022-03-28 10:34:25 <mps> psykose: hehe, why
2022-03-28 10:34:36 <mps> that is how it should work
2022-03-28 10:34:41 <psykose> absolutely not
2022-03-28 10:35:01 <mps> hah
2022-03-28 10:35:40 <ikke> if bin does not exist, how should install determine whether it should be a directory to put it in, or the filename?
2022-03-28 10:36:18 <mps> this is solved about 40-50 years ago
2022-03-28 10:37:17 <mps> so /usr/bin is file and /usr/bin/ is dir, always
2022-03-28 10:37:28 <ncopa> mps: no. i think we need more build power to do kernel updates even more frequent
2022-03-28 10:37:39 <mps> though in unix everything is file at the end ;)
2022-03-28 10:38:16 <mps> ncopa: ok, thanks for removing some 'fear' from me
2022-03-28 10:40:08 <mps> I will upgrade linux-edge 4 times today
2022-03-28 10:40:21 <ncopa> i mean, the timing may be a bit unfortunate, but its not your fault. and really the fundamental problem is how we share or schedule the build power
2022-03-28 10:40:21 <mps> one of them is already done
2022-03-28 10:40:33 <ncopa> i have 7 kernel builds in the queue
2022-03-28 10:40:39 <psykose> downgrade it 4 times too for good measure
2022-03-28 10:40:49 <ikke> ncopa: I sitll have to look at it, but nomad does scheduling as well, and might be simpler
2022-03-28 10:40:58 <mps> oh, actually I have 6 today
2022-03-28 10:41:16 <ncopa> mps do you need to have those builds out today, or can they wait for tomorrow? would probably help get the releases out faster if you could wait
2022-03-28 10:41:41 <ncopa> i guess the edge kernels are not really critical for the release (since they are not included in the release media)
2022-03-28 10:41:41 <mps> ncopa: ok, I can keep them
2022-03-28 10:41:47 <ncopa> appreciate. thank you
2022-03-28 10:42:27 <mps> ncopa: what's with some other pkgs, not big ones
2022-03-28 10:42:56 <ncopa> no problem with smaller packages
2022-03-28 10:43:32 <mps> I plan to upgrade monit this evening and maybe backport mutt to 3.15
2022-03-28 10:43:57 <ncopa> oh, actually, its more than 7 kernel builds in my queue, there are -virt kernels as well, so its more like 12 kernel builds in the queue from me
2022-03-28 10:44:11 <ncopa> those are no problem. just push
2022-03-28 10:44:34 <ncopa> things that takes less than 5-10 mins to build is no problem
2022-03-28 10:44:48 <mps> ok
2022-03-28 10:45:49 <mps> some people wants linux-asahi kernel in alpine but I think I will do it on wednesday
2022-03-28 10:47:12 <mps> btw, one of asahi men plan to make alpine installable straight from asahi-installer
2022-03-28 10:47:20 <ncopa> i think linux-asahi kernel sounds good, but not today
2022-03-28 10:47:30 <ncopa> that would be awesome
2022-03-28 10:48:02 <mps> ncopa: no, I didn't yet cleaned APKBUILD for linux-asahi, so no problem :)
2022-03-28 10:49:23 <ncopa> we're a bit offtopic but, i had a look at your APKBUILD. I have refactored the kernel config handling in linux-lts. we now use a minimal config from make savedefconfig
2022-03-28 10:50:13 <mps> ncopa: yes, few days ago I and one of our good friends criticized you in private about mess you made with -lts ;)
2022-03-28 10:53:33 <ncopa> kernel configs is messy regardless how you do it
2022-03-28 10:54:00 <mps> agree fully
2022-03-28 11:29:03 <clandmeter> ncopa: your new approach of kernel config, it means that its all based on defconfig right? so the new configs should be a lot different? 
2022-03-28 11:33:33 <ncopa> its based on defconfig yes
2022-03-28 11:33:39 <ncopa> its only a diff from the defconfig now
2022-03-28 12:47:00 <ikke> ncopa: https://imgur.com/a/vtTbFD3
2022-03-28 12:48:04 <ncopa> thank you!
2022-03-28 12:55:31 <mps> ncopa: forgot to tell that kernel 5.18 will not have '# CONFIG_something is not set' but will become 'CONFIG_something=n'
2022-03-28 12:57:27 <ncopa> oh, thats cool. about time
2022-03-28 14:19:46 <clandmeter> ncopa: ok, this means all missing defconfig options are now included in new kernels?
2022-03-28 14:32:05 <ncopa> yup
2022-03-28 14:32:19 <ncopa> it also means it is less visible what the new kernel options are
2022-03-28 14:51:20 <mps> and kernel is not anymore small, simple, I guess
2022-03-28 14:51:54 <mps> clandmeter: did you tested 5.17 kernel on riscv64 board?
2022-03-28 16:12:58 <clandmeter> I have not touched it for a long time
2022-03-28 16:13:39 <clandmeter> mps, i can give you access
2022-03-28 16:13:50 <clandmeter>  If you want too try
2022-03-28 16:14:17 <mps> clandmeter: I think you did already but I logged in just once
2022-03-28 16:15:24 <mps> I ask because 5.17 should have poweroff/reboot problem fixed, would be nice to know is it really
2022-03-28 19:16:37 <ikke> speaking about riscv boards
2022-03-28 19:22:46 <mps> ikke: ?
2022-03-28 19:23:36 <ikke> Just got an update about the shipments
2022-03-28 19:23:43 <mps> ah
2022-03-28 19:24:49 <ncopa> looks like build-edge-ppc64le had network issues
2022-03-28 19:24:55 <ncopa> on upload of lxc
2022-03-28 19:25:01 <ncopa> i just tagged edge snapshot
2022-03-28 19:25:45 <mps> ikke: also I got mail
2022-03-28 19:25:50 <mps> just read it
2022-03-28 19:38:24 <ikke> ncopa: compare: https://i.imgur.com/qCJSQUa.png
2022-03-28 19:40:26 <ncopa> nice! thank you!
2022-03-28 19:40:40 <ncopa> can you please save those two images?
2022-03-28 19:41:32 <mps> ikke: usa9 is arm?
2022-03-28 19:41:37 <ikke> correct
2022-03-28 19:41:54 <mps> stange
2022-03-28 19:42:04 <mps> s/stange/strange/
2022-03-28 19:42:04 <alpine-meetbot> mps meant to say: strange
2022-03-28 19:42:07 <ikke> It has 160 cores and 3 builders
2022-03-28 19:42:22 <ikke> well, builders for 3 arches I must say
2022-03-28 19:42:24 <mps> ah, yes. I forgot
2022-03-28 19:42:54 <mps> so not so strange
2022-03-28 19:43:30 <ikke> I'm thinking about normalizing the data
2022-03-28 19:44:10 <ikke> load / cores * 100
2022-03-28 19:46:48 <ncopa> ppc64le have issues with uploading the releases :-/
2022-03-28 19:47:04 <ncopa> or maybe its just slow...
2022-03-28 19:47:52 <ikke> https://i.imgur.com/HfFaDfL.png
2022-03-28 19:47:52 <ncopa> yeah, its just slow
2022-03-28 19:48:01 <ikke> That one is load per core
2022-03-28 19:48:03 <ikke> less distorted
2022-03-28 19:49:04 <ikke> Link for the interested (and that have access): https://zabbix.alpinelinux.org/zabbix.php?action=dashboard.view&dashboardid=9
2022-03-28 19:50:39 <ncopa> the zabbix favicon is suddenly a bit disturbing...
2022-03-28 19:50:46 <ikke> hmm
2022-03-28 19:50:55 <ikke> Didn't occur to me
2022-03-28 19:50:57 <ncopa> dont want to have a tab open with zabbix when sharing screen...
2022-03-28 19:52:10 <ncopa> i dont think we need to do anything about it, unless it disturbs the infra team.
2022-03-28 19:52:59 <ikke> Not me personally
2022-03-28 20:07:55 <ncopa> ppc64le is slowing us down now :-/
2022-03-28 20:09:40 <ikke> ncopa: I'll drop a message
2022-03-28 20:10:17 <ikke> 3.14.5 release?
2022-03-28 20:55:32 <ncopa> im on it
2022-03-28 20:56:54 <ncopa> ikke: can you send the email?
2022-03-28 20:57:00 <ncopa> im doing the docker images PR now
2022-03-28 20:58:11 <ikke> ncopa: I've sent a message via slack (Rafael invited me to one) 
2022-03-28 20:59:16 <ncopa> im not following? slack?
2022-03-28 20:59:27 <ncopa> ah for the ppc64le?
2022-03-28 21:00:52 <ikke> ye
2022-03-28 21:01:47 <ncopa> i think the upload finally went through?
2022-03-29 12:21:51 <ikke> ncopa: I suppose we do not have any number about how slow uploading the ppc64le releases was?
2022-03-29 12:27:44 <mps> ncopa: current load on usa9 is around 10, so i could push kernels to builders?
2022-03-29 12:28:15 <ikke> mps: releases are done, so I think it should be fine
2022-03-29 12:28:34 <mps> ok, thanks
2022-03-29 15:31:02 <ikke> apparently there is maintenance happening at the campus where ppc64le is hosted, will be back later today or tomorrow
2022-03-30 09:30:04 <ikke> hmm
2022-03-30 12:22:13 <mps> hmm, package name couldn't have upper case letters, right?
2022-03-30 12:24:50 <mps> aha, man APKBUILD is explicit, ok
2022-03-30 15:34:00 <ikke> ppc64le maintenance still in progress
2022-03-30 17:50:54 <ikke> dotnet builds
2022-03-31 18:53:06 <ikke> gitlab security anouncement