2023-02-01 14:58:34 <minimal> clandmeter: same issue with the Feb alpine-linux irc logs. Obviously there is very little in there so far. The olny thing out of the ordinary I can see there is the 3rd entriy, from alpine-meetbot, where the result of a rewritten message is highlighted (I see 2 unprintable characters either side of the word "musl" (the result of a person's "s/glibc/musl/")
2023-02-01 16:36:48 <ikke> minimal: and what is the issue again?
2023-02-01 16:37:04 <ikke> that chromium downloads the logs?
2023-02-01 16:40:21 <minimal> yupe
2023-02-01 16:47:29 <ikke> Hmm, even Content-Disposition: inline does not help
2023-02-01 16:48:51 <minimal> so much for uniform browser behaviour lol
2023-02-01 16:53:59 <ikke> may be this issue: https://bugs.chromium.org/p/chromium/issues/detail?id=106150
2023-02-01 16:56:08 <ikke> nope
2023-02-01 16:56:48 <minimal> the dev console shows "Resource intepreted as Document but transferred with MIME type application/octet-stream"
2023-02-01 16:57:17 <psykose> ikke: probably should have X-Content-Type-Options: nosniff anyway
2023-02-01 16:57:25 <psykose> just in the ole classic list of must have headers
2023-02-01 17:01:23 <ikke> psykose: yup, that works
2023-02-01 17:01:54 <ikke> It's head fix week I suppose
2023-02-01 17:01:59 <ikke> header*
2023-02-01 17:02:35 <minimal> yupe, great, thanks psykose
2023-02-01 20:33:59 <ikke> clandmeter: psykose upgraded gitlab to 15.5.7
2023-02-01 20:34:16 <ikke> sorry
2023-02-01 20:34:20 <ikke> 15.7.6 :p
2023-02-01 20:34:31 <psykose> whoa did i get super powers
2023-02-01 20:34:36 <psykose> i don't even remember doing that
2023-02-01 20:34:41 <psykose> (yes, yes, i know)
2023-02-01 20:35:08 <ikke> :)
2023-02-01 20:35:25 <ikke> It doesn't automatically add the colon for the 2nd autocomplete
2023-02-01 20:37:12 <psykose> aye
2023-02-01 21:21:04 <clandmeter> nice!
2023-02-01 21:21:24 <clandmeter> does she get this nice new overlay as well?
2023-02-01 21:22:27 <ikke> psykose: is not an admin
2023-02-01 21:22:30 <ikke> lol
2023-02-01 21:23:18 <clandmeter> yeah that would be really insecure to make her admin
2023-02-01 21:29:08 <psykose> in what sense
2023-02-01 21:34:49 <clandmeter> none at all, it was sarcasm. 
2023-02-01 21:35:06 <psykose> hah
2023-02-01 21:35:12 <clandmeter> just as well to make you an admin
2023-02-01 21:36:06 <clandmeter> if you want to take on the job that is.
2023-02-01 21:36:13 <clandmeter> its hard work and little pay
2023-02-01 21:36:19 <psykose> :-)
2023-02-01 21:37:06 <psykose> if you need me to do something i don't mind, mostly it gives me the global pipelines view which is an extra pair of eyes i guess for those pesky server hosters
2023-02-01 21:37:14 <psykose> i can't think of direct administration tasks however
2023-02-01 21:37:23 <psykose> but if you need me to help with something i can try
2023-02-01 21:37:30 <clandmeter> we can start with a beer or a juice of your choosing :)
2023-02-01 21:38:07 <psykose> too damn many good beers to pick from these days
2023-02-01 21:39:47 <clandmeter> i dont think admin brings a ton of things, but it comes in handy.
2023-02-01 21:39:55 <psykose> yeah
2023-02-01 21:39:56 <clandmeter> for sure the amazing overlay
2023-02-01 21:41:42 <clandmeter> but you will have to wait for the next critical security release, but that shouldnt be too long...
2023-02-01 21:42:04 <psykose> the most immediately obvious use is being able to move issues anywhere which i could've done a few times myself, hehe
2023-02-01 21:42:09 <psykose> ah, next critical
2023-02-01 21:42:13 <psykose> i give it to next tuesday
2023-02-01 21:42:14 <psykose> :-)
2023-02-01 22:12:25 <clandmeter> looks like our new arm setup should be reconfigured.
2023-02-01 22:12:36 <clandmeter> meaning proper mlan connections
2023-02-01 22:12:46 <clandmeter> but i have not tried it yet
2023-02-01 22:15:50 <psykose> Music Local Area Network? :)
2023-02-01 22:17:25 <clandmeter> management lan
2023-02-01 22:17:44 <clandmeter> gigabyte servers always have a dedicated management lan port
2023-02-01 22:17:50 <psykose> aha
2023-02-01 22:17:59 <clandmeter> and the bmc is license free
2023-02-01 22:18:07 <clandmeter> am i advertising ok?
2023-02-01 22:18:16 <clandmeter> :p
2023-02-01 22:18:58 <clandmeter> or should i say giga computing, as the branding is going to change.
2023-02-01 22:21:42 <psykose> :-)
2023-02-02 10:02:04 <clandmeter> psykose: you are admin now. have fun :)
2023-02-02 11:25:38 <ikke> not too much fun though :P
2023-02-02 11:40:25 <clandmeter> ikke: have you tried connecting to the arm boxes?
2023-02-02 11:51:11 <ikke> clandmeter: no, not yet
2023-02-02 15:02:38 <psykose> clandmeter: thanks :)
2023-02-02 18:34:24 <psykose> ikke: i wonder if we need some signup confirm or something
2023-02-02 18:34:37 <psykose> literally like 90% of these user creates are spam it seems
2023-02-02 19:15:59 <psykose> https://gitlab.alpinelinux.org/rosie4/economics_homework_help/-/issues/1 lol
2023-02-02 19:16:17 <psykose> can i just nuke spam like this or is there more process to it
2023-02-02 19:16:23 <ikke> Just nuke it
2023-02-02 19:16:28 <psykose> okie
2023-02-02 19:16:48 <ikke> I've bean clearing spam from time to time
2023-02-02 19:17:18 <psykose> noted
2023-02-02 19:18:16 <ikke> There is this captcha integration, but people are not a fan of that
2023-02-02 19:18:50 <psykose> it most likely does not have to be "google captcha" does it
2023-02-02 19:18:55 <psykose> because ime that is most of the issue
2023-02-02 19:19:16 <psykose> aside from that captchas are silly, but hmm
2023-02-02 19:19:20 <psykose> what other methods are there
2023-02-02 19:19:30 <psykose> aside from "confirm email or get an autobad in 7 days"
2023-02-02 19:21:54 <psykose> is it also possible to make ~alpine/aports/patches undeliverable? people don't exactly read the warning and keep posting to it anyway
2023-02-02 19:22:12 <psykose> i don't see a point to have something look mostly-working when it doesn't
2023-02-02 19:22:36 <psykose> and it's as much a waste of time for people using it as me monitoring it since it doesn't
2023-02-02 19:22:38 <ikke> I do watch it
2023-02-02 19:23:24 <psykose> i do too sometimes
2023-02-02 19:24:21 <ikke> because it's low traffic, it's doable to do manually
2023-02-02 19:26:53 <ikke> I cannot find the setting anymore where you enable something like captcha
2023-02-02 19:29:43 <psykose> https://docs.gitlab.com/ee/integration/recaptcha.html i guess
2023-02-02 19:30:04 <psykose> https://gitlab.alpinelinux.org/admin/application_settings/reporting
2023-02-02 19:30:16 <psykose> there's already a key in there
2023-02-02 19:30:51 <ikke> Yeah, we did have it
2023-02-02 19:30:58 <ikke> That's the page I was looking for
2023-02-02 19:31:05 <ikke> never thought to find it under reporting :D
2023-02-02 19:31:13 <psykose> indeed :D
2023-02-02 19:36:53 <ikke> There is this invisible captcha, but apparently not very effective
2023-02-02 19:37:26 <ikke> If you happen to be bored: https://gitlab.alpinelinux.org/explore/snippets :P
2023-02-02 19:37:56 <psykose> seen that :p
2023-02-02 19:38:06 <psykose> https://img.ayaya.dev/bk0ild7m65xO
2023-02-03 05:46:59 <psykose> ikke: could you kick the x86_64 build to save some time since i had to double rebuild on it :p
2023-02-03 05:47:40 <ikke> done
2023-02-03 05:50:21 <psykose> thanks
2023-02-03 06:28:07 <ikke> psykose: yesterday I created something that tries to classify snippets as spam :P
2023-02-03 06:29:07 <psykose> oh? whadya make
2023-02-03 06:29:43 <ikke> a simple script that checks for certain properties
2023-02-03 06:30:03 <ikke> age of account, amount of capitals in title, spam sensitive keywords etc
2023-02-03 06:49:06 <psykose> aha
2023-02-03 06:49:14 <psykose> well, hopefully no false positives
2023-02-03 06:51:28 <ikke> Yeah, I'm constantly verify, and when taking action, I will also look at if the user made contributions to aports for example
2023-02-03 06:52:17 <ikke> https://gitlab.alpinelinux.org/-/snippets/748
2023-02-03 06:54:21 <psykose> i assume "macbook":               4< is a typo on the end
2023-02-03 06:54:56 <ikke> yes
2023-02-03 06:55:53 <psykose> also you check if score < 5?
2023-02-03 06:56:21 <ikke> that's just me checking what's left
2023-02-03 06:56:37 <psykose> aha
2023-02-03 10:02:10 <clandmeter> blaboon: are you able to add an ipv6 /64 range to deu7-dev1.alpinelinux.org?
2023-02-03 13:57:52 <mps> anyone know how to see commit message for MR by downloading patch. for example this 'curl https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/43816.patch'
2023-02-03 14:09:34 <ptrc> mps: it shows the commit message though? unless you mean MR message
2023-02-03 14:10:21 <ptrc> ah, no, that's the MR title..
2023-02-03 14:10:30 <mps> ptrc: Subject? hm, then it is short ;-)
2023-02-03 14:11:07 <mps> 'Subject: [PATCH] community/iwd: upgrade to 2.3'
2023-02-03 14:30:11 <blaboon> clandmeter: done. the new range is 2a01:7e01:e001:46b::/64
2023-02-03 21:47:12 <psykose> ikke: did you ever deploy the setarch thing
2023-02-03 21:49:02 <ikke> Not yet 
2023-02-03 21:50:30 <psykose> i found it doesn't really work anyway
2023-02-03 21:50:37 <psykose> still gets armv8l from python or whateever
2023-02-03 21:50:42 <ikke> OK 
2023-02-03 23:24:53 <ikke> So what does python use as source for this
2023-02-03 23:29:27 <psykose> platform.machine() and whatever that does it think
2023-02-03 23:30:21 <psykose> ah
2023-02-03 23:30:25 <psykose> which is uname -m,
2023-02-03 23:30:49 <psykose> setarch just doesn't work for that for some reason
2023-02-03 23:31:04 <psykose> i swear that worked
2023-02-03 23:31:46 <psykose> gueess not
2023-02-03 23:32:59 <ikke> If I do setarch armhf sh, then uname -m still returns armv8l
2023-02-03 23:38:03 <psykose> yeah, it just does that name
2023-02-03 23:38:11 <psykose> i swear last time i tested it it did something else
2023-02-04 16:38:23 <ikke> strange, wg vpn to alpine suddenly stopped working for me
2023-02-04 16:38:50 <ikke> wg handshake appears to succeed, but no traffic
2023-02-04 16:43:56 <clandmeter> ikke: i think its me
2023-02-04 16:44:11 <clandmeter> i restarted it cause i edited the config
2023-02-04 16:44:23 <clandmeter> i dont even know if that is needed :)
2023-02-04 16:45:03 <ikke> Still does not work for me
2023-02-04 16:45:18 <ikke> you can do wg syncconf /etc/wg/wg0.conf
2023-02-04 16:45:33 <ikke> /etc/wireguard/wg0.conf
2023-02-04 16:45:35 <clandmeter> does that also manage the routing?
2023-02-04 16:45:53 <ikke> What routing?
2023-02-04 16:46:21 <clandmeter> looks like wg-quick enables some routing
2023-02-04 16:46:31 <ikke> hmm, wg0 no longer has an IP
2023-02-04 16:46:36 <ikke> on deu7
2023-02-04 16:46:57 <ikke> I don't use wg-quick
2023-02-04 16:47:34 <ikke> clandmeter: are you working on it?
2023-02-04 16:47:51 <clandmeter> let me check
2023-02-04 16:48:14 <mps> wg is down
2023-02-04 16:48:40 <ikke> yes, without an ip on the interface, it's not going to work
2023-02-04 16:48:57 <mps> ifup wg0
2023-02-04 16:49:17 <clandmeter> i tried it
2023-02-04 16:49:20 <clandmeter> but it does not come up
2023-02-04 16:49:48 <mps> ifup -f wg0
2023-02-04 16:49:56 <mps> now it is up
2023-02-04 16:50:11 <clandmeter> -f worked
2023-02-04 16:50:34 <mps> now everything works
2023-02-04 16:50:38 <clandmeter> nope
2023-02-04 16:50:43 <clandmeter> my config still not wkring :)
2023-02-04 16:50:47 <mps> for me works
2023-02-04 16:50:57 <clandmeter> ikke: i need to open the port for ipv6
2023-02-04 16:51:08 <clandmeter> any idea how that works with awall?
2023-02-04 16:51:30 <clandmeter> if we dont use wg-quick, maybe better to uninstall it?
2023-02-04 16:52:26 <mps> for now it is not needed on gateway
2023-02-04 16:52:28 <ikke> What port for ipv6?
2023-02-04 16:52:44 <ikke> you can already connect to wg via ipv6
2023-02-04 16:52:46 <clandmeter> wg port
2023-02-04 16:53:05 <clandmeter> then i dont know why it does not work for me
2023-02-04 16:53:21 <ikke> I'm connected to wg via ipv6
2023-02-04 16:53:26 <clandmeter> ah i works now
2023-02-04 16:53:41 <clandmeter> i guess the restart broke vpn
2023-02-04 16:53:56 <ikke> yes, since it removed the IP
2023-02-04 16:54:17 <clandmeter> ok with removing wireguard-tools-wg-quick?
2023-02-04 16:54:18 <ikke> awall automitically opens ports on both ipv4 and ipv6 if you don't restrict it
2023-02-04 16:54:23 <ikke> fine with me
2023-02-04 16:54:32 <clandmeter> mps: ?
2023-02-04 16:54:44 <mps> clandmeter: ok
2023-02-04 16:54:51 <clandmeter> hmm i cant
2023-02-04 16:55:06 <ikke> automatically pulled in?
2023-02-04 16:55:29 <mps> nor do I
2023-02-04 16:55:33 <clandmeter> yep
2023-02-04 16:55:38 <clandmeter> somebody installed wireguard-tools
2023-02-04 16:55:51 <clandmeter> it should be wireguard-tools-wg
2023-02-04 16:56:45 <clandmeter> but it removes a lot of stuff
2023-02-04 16:56:50 <clandmeter> hope it all keeps working 
2023-02-04 16:56:57 <mps> wireguard-tools-wg is installed
2023-02-04 16:57:36 <clandmeter> yes i just did
2023-02-04 16:58:32 <mps> ok
2023-02-04 17:22:41 <clandmeter> ikke: how does the wireguard route get added?
2023-02-04 17:24:32 <clandmeter> mps: i think you set it up right?
2023-02-04 17:31:49 <mps> clandmeter: routes are set by ikke 
2023-02-04 17:32:29 <clandmeter> i found the issue
2023-02-04 17:32:35 <mps> I even don't know where they are set
2023-02-04 17:32:38 <clandmeter> no ipv6 on the server interface
2023-02-04 17:32:51 <clandmeter> it works now
2023-02-04 17:33:45 <ikke> I don't recall adding IPv6 to the wg interface
2023-02-04 17:35:16 <ikke> clandmeter: btw, I reserved this prefix as a private but globally unique ipv6 address: https://netbox.alpin.pw/ipam/prefixes/54/
2023-02-04 17:35:33 <ikke> tried to get that working with dmvpn, but things stopped working, so I reverted it
2023-02-04 17:37:17 <clandmeter> ikke: how did you get dns working on your personal setup?
2023-02-04 17:37:28 <clandmeter> just adding DNS = ipv6 google  does not work
2023-02-04 17:37:45 <ikke> I set dnsmasq to resolve *.alpin.pw with 172.16.6.3
2023-02-04 17:38:37 <clandmeter> with an IPv6 address?
2023-02-04 17:44:22 <clandmeter> ipv6 is torture 
2023-02-04 17:44:34 <clandmeter> or lets say wg with ipv6
2023-02-04 17:50:18 <clandmeter> suddenly i dont get an headshake anymore, without any change... what fun.
2023-02-04 21:32:28 <ikke> psykose: clandmeter: I'm about to run this: https://gitlab.alpinelinux.org/-/snippets/754
2023-02-04 21:32:44 <psykose> ack
2023-02-04 21:32:56 <psykose> missed a few thousand (/s)
2023-02-04 21:48:29 <ikke> meh, that only blocked those users
2023-02-04 21:49:24 <ikke> oh, took a bit
2023-02-04 22:10:28 <psykose> -X DELETE ?hard_delete=true isn't a real delete?
2023-02-04 22:10:29 <psykose> :D
2023-02-04 22:10:35 <psykose> ah
2023-02-04 22:10:36 <psykose> right
2023-02-05 10:22:44 <ddevault> is dl-cdn down?
2023-02-05 10:24:18 <psykose> up for me
2023-02-05 10:24:32 <ddevault> over ipv6 maybe?
2023-02-05 10:24:34 <ddevault> I can reach it on v4
2023-02-05 10:24:44 <psykose> works on v6 too
2023-02-05 10:24:47 <psykose> it's fastly so who knows
2023-02-05 10:24:51 <ddevault> probably on my end
2023-02-05 10:24:52 <psykose> local geo cdn
2023-02-05 10:24:54 <ddevault> at FOSDEM network, who knows
2023-02-05 10:25:03 <psykose> i've definitely had unroutable v6 routes to it before
2023-02-06 11:16:59 <ikke> Spam script seems to work well, caught all new spam messages (still manually executed for now)
2023-02-06 13:41:13 <pj> \o/
2023-02-06 16:42:38 <ikke> psykose: clandmeter: I've set maximum_timeout to 24h on the x86 runner to prevent CI abuse
2023-02-06 16:43:20 <ikke> though, this time it was not a long running job
2023-02-06 17:49:44 <clandmeter> ikke: ok, did it get abused?
2023-02-06 17:50:18 <clandmeter> psykose, ikke can we plan a date to decommission the old arm servers? 
2023-02-06 17:50:52 <clandmeter> ikke: if you can get dmvpn working that would be great
2023-02-06 17:51:16 <clandmeter> i can start to copy the lxc part
2023-02-06 17:51:52 <clandmeter> just need to define what is docker and what is lxc
2023-02-06 17:52:40 <clandmeter> i guess we need to run dmvpn on the hosts, not the router.
2023-02-06 18:11:02 <ikke> clandmeter: how will we call them? Or do we keep the current names? 
2023-02-06 18:11:09 <ikke> Mainly for dmvpn site names 
2023-02-06 18:12:50 <ikke> clandmeter: and do we split on function (ci vs builders) and/or on arch?
2023-02-06 18:13:15 <ikke> If we separate on function, then we can probably just use plain docker on one, no more VMs for CI
2023-02-06 18:13:35 <ikke> clandmeter: re CI abuse: someone creates projects with long-running (and mostly a lot of space using) jobs
2023-02-06 18:13:50 <ikke> And setup some kind of way to connect to the running containers remotely
2023-02-06 18:14:08 <clandmeter> nod
2023-02-06 18:14:18 <clandmeter> let me take a walk and ill be back
2023-02-06 18:14:25 <ikke> ack
2023-02-06 19:04:20 <clandmeter> o/
2023-02-06 19:04:54 <clandmeter> for the hostname anything is fine for me
2023-02-06 19:05:15 <clandmeter> i would try to just split things between docker and lxc
2023-02-06 19:05:21 <clandmeter> one host docker one lxc
2023-02-06 19:05:21 <ikke> sounds good to me
2023-02-06 19:05:50 <clandmeter> its just a bit troublesome you need ipv6 currently
2023-02-06 19:06:04 <clandmeter> but i guess if we have dmvpn up and running we can just taht
2023-02-06 19:06:14 <ikke> I'll set it up soon
2023-02-06 19:06:36 <clandmeter> so lets use arm1 for lxc and arm2 for docker
2023-02-06 19:06:46 <clandmeter> and you can name them wahtever you want
2023-02-06 19:07:09 <ikke> alright
2023-02-06 19:09:08 <clandmeter> ill start with setting up lxc
2023-02-06 19:09:12 <clandmeter> and transfering things
2023-02-06 19:15:49 <clandmeter> ikke: why are we keeping all those old builders?
2023-02-06 19:16:18 <ikke> clandmeter: because no one bothers to clean up
2023-02-06 19:16:32 <clandmeter> :)
2023-02-06 19:16:55 <ikke> and the old builders are not very large, so cleanup recovers little
2023-02-06 19:18:02 <clandmeter> usa9-dev1 is our only builder right?
2023-02-06 19:18:07 <ikke> yes
2023-02-06 19:19:46 <clandmeter> already takes ages to do a du -hs * in /var/lib/lxc
2023-02-06 19:20:34 <ikke> maybe clean up before syncing?
2023-02-06 19:20:39 <ikke> There is a script in /var/lib/lxc
2023-02-06 19:21:40 <clandmeter> i guess we can also clean the user containers who are not active anmore
2023-02-06 19:21:59 <ikke> most likely yes
2023-02-06 19:22:05 <ikke> similar to nld-dev-1
2023-02-06 19:22:13 <ikke> I didn't even start some containers
2023-02-06 19:33:57 <clandmeter> ikke: any reason the script only cleans those specific builders?
2023-02-06 19:34:09 <ikke> clandmeter: mostly relevant there
2023-02-06 19:34:21 <ikke> I used it mostly during release builds
2023-02-06 19:34:25 <clandmeter> what about builddirs?
2023-02-06 19:34:45 <ikke> pkg/ and src/?
2023-02-06 19:34:52 <clandmeter> yes
2023-02-06 19:34:53 <ikke> I did that manually from time to time
2023-02-06 19:35:20 <clandmeter> ok let me adjust it and do it for all builders.
2023-02-06 19:35:33 <clandmeter> but it may break active build
2023-02-06 19:35:36 <ikke> yup
2023-02-06 19:35:41 <ikke> we'll just retry it 
2023-02-06 19:35:47 <clandmeter> nod
2023-02-06 19:35:52 <psykose> all is idle currently
2023-02-06 19:35:55 <psykose> just go for it
2023-02-06 19:47:46 <ikke> clandmeter: che-bld-1 and che-ci-1 I suppose
2023-02-06 19:58:37 <clandmeter> fine by me
2023-02-06 20:06:15 <clandmeter> looks like tmp is overcrowded as well 
2023-02-06 20:06:34 <ikke> oh, yeah, that gets filled up quite a lot
2023-02-06 20:06:41 <ikke> mostly amount of files, not size
2023-02-06 20:07:05 <clandmeter> which is terrible to rsync :)
2023-02-06 20:07:14 <ikke> why do you rsync /tmp?
2023-02-06 20:07:19 <clandmeter> arg is even too big :)
2023-02-06 20:07:26 <ikke> oh, for each builder 
2023-02-06 20:07:38 <clandmeter> i rsync var/lib/lxc
2023-02-06 20:07:40 <ikke> yeah
2023-02-06 20:07:53 <clandmeter> well i will
2023-02-06 20:08:01 <ikke> prune /tmp
2023-02-06 20:09:37 <clandmeter> need to adjust the script
2023-02-06 20:09:39 <clandmeter> but not now
2023-02-06 20:09:41 <clandmeter> too lazy
2023-02-06 20:11:44 <clandmeter> heh, that was a almost 100g of tmp i beliefe
2023-02-06 20:11:48 <clandmeter> v...
2023-02-06 20:15:51 <clandmeter> man those home dirs are messy. a gazillion .xxx dirs
2023-02-06 20:16:36 <ikke> clandmeter: what was the IP address of arm2?
2023-02-06 20:16:45 <clandmeter> maybe we should just delete everything in $HOME except what we want to keep
2023-02-06 20:17:29 <clandmeter> 2a0a:e5c1:517:cafe:da5e:d3ff:fee6:1f28
2023-02-06 20:17:59 <ikke> bedankt :)
2023-02-06 20:18:09 <clandmeter> it was from our pm :p
2023-02-06 20:18:17 <ikke> lol
2023-02-06 20:19:47 <ikke> hmm, for some reason arm2 cannot resolve dns
2023-02-06 20:20:13 <ikke> same setup as arm1
2023-02-06 20:20:20 <ikke> (che-bld-1 :p)
2023-02-06 20:23:27 <clandmeter> ikke: what do you think of rm all except packags, aports, any maybe something else?
2023-02-06 20:23:40 <clandmeter> we should really ahve rootbld do the build
2023-02-06 20:24:13 <ikke> afaik there were still some issues with rootbld that made it not good for the builders
2023-02-06 20:25:33 <clandmeter> that did not answer my question :)
2023-02-06 20:26:45 <ikke> packages, aports .abuild 
2023-02-06 20:27:12 <clandmeter> .ssh
2023-02-06 20:31:54 <clandmeter> about 12G of . files and directories for just one builder
2023-02-06 20:35:08 <ikke> yea
2023-02-06 21:01:04 <ikke> clandmeter: I'll do the dmvpn setup tomorrow
2023-02-06 21:01:12 <clandmeter> ok
2023-02-06 21:01:26 <clandmeter> im still thinking on how to properly cleanup those containers
2023-02-06 21:01:59 <clandmeter> if i make a mistake it will probably mean it will cleanup too much...
2023-02-06 21:02:47 <ikke> yup
2023-02-06 21:02:50 <ikke> been there, done that
2023-02-06 21:03:32 <ikke> I tried to build that script to be as safe as possible
2023-02-06 21:05:10 <clandmeter> when you start to play with find to exclude things, it gets more tricky..
2023-02-06 21:05:56 <clandmeter> i dont like to pipe finds output to xargs rm -rf :)
2023-02-06 21:06:20 <ikke> heh
2023-02-06 22:46:27 <clandmeter> ikke: i used an exclude list to provide to rsync, feels more safe :)
2023-02-07 15:49:06 <clandmeter> rsync got killed due to network timeout
2023-02-07 16:18:28 <ikke> oof
2023-02-07 20:49:35 <ikke> clandmeter: /usr/share/lua/5.2/asn1/rfc3779.lua:94: attempt to concatenate upvalue 'addr' (a table value) :/
2023-02-07 20:49:39 <ikke> trying to generate the certificate
2023-02-07 20:49:41 <ikke> trying to generate the certificates
2023-02-07 22:13:31 <clandmeter> ikke: for dmvpn?
2023-02-07 22:23:15 <clandmeter> it looks like the arm network is a bit slow
2023-02-07 22:39:02 <mps> finally linux 6.1 is promoted as LTS
2023-02-08 05:36:02 <ikke> clandmeter: yes, dmvpn
2023-02-08 12:43:35 <ikke> clandmeter: found the issue, wrong subnet, but the error reporting assumes something is a string while it's a table
2023-02-08 12:44:28 <ikke> (subnet length)
2023-02-08 14:50:27 <ncopa> do we have an email list for security team?
2023-02-08 14:51:19 <ncopa> seems we dont
2023-02-08 14:58:08 <ncopa> ugh. my email setup does not seem to work here
2023-02-08 15:18:03 <ncopa> i need to use mail.a.o as smtp for SPF to work, but i don't remember how to set that up. I don't remember the username/password, nor which port to connect to
2023-02-08 15:18:17 <ncopa> is it port 587, 465 or 25?
2023-02-08 15:18:32 <ncopa> is it username ncopa or ncopa@alpinelinux.org?
2023-02-08 15:20:25 <ncopa> bah, maybe i should just use gmail
2023-02-08 16:05:27 <ikke> ncopa: let me check
2023-02-08 16:07:17 <ikke> ncopa: port 587
2023-02-08 18:13:29 <ncopa> ikke: thanks! i got some progress now. port 587, No TLS, use STARTTLS
2023-02-08 18:14:12 <ncopa> but i get authentication error. Can I find out the username somehow? is it ncopa or ncopa@alpinelinux.org?
2023-02-08 18:31:46 <ncopa> hum, i think i have issues getting the wireguard connection up
2023-02-08 18:32:49 <ncopa> Feb  8 18:23:58 ncopa-mbp14 daemon.info dhcpcd[3183]: wg0: waiting for 3rd party to configure IP address
2023-02-08 18:42:02 <ncopa> ok. not figuring that out. anybody know the public hostname of ncopa-edge-x86_64 container?
2023-02-08 18:51:45 <ncopa> bah, my key is not there
2023-02-08 19:37:03 <ncopa> i think i may need some help with getting wireguard client working and getting access to the builders
2023-02-08 19:46:56 <mps> ncopa: what you use to set up wg client
2023-02-08 20:08:02 <ncopa> i created an /etc/wireguard/wg0.conf file
2023-02-08 20:08:27 <ncopa> then i did: wg-quick up wg0
2023-02-08 20:09:58 <clandmeter> you dont really need wg-quick
2023-02-08 20:10:09 <clandmeter> its a bash script
2023-02-08 20:10:16 <ncopa> so how do i start it?
2023-02-08 20:10:21 <clandmeter> from interfaces
2023-02-08 20:10:39 <clandmeter> check the example we have on wg.a.o
2023-02-08 20:10:42 <clandmeter> i think you have access
2023-02-08 20:11:00 <clandmeter> you can purge wg-quick by only adding wireguard-tools-wg
2023-02-08 20:11:05 <mps> I use wg-quick and keep wg0.conf in my home dir
2023-02-08 20:11:09 <clandmeter> if i have the naming right
2023-02-08 20:11:41 <mps> it is simpler than using /etc/network/interfaces
2023-02-08 20:12:00 <mps> and I could have few wg interfaces at the same time
2023-02-08 20:12:27 <ncopa> how should the interfaces look like? i need to set static ip addr?
2023-02-08 20:13:35 <mps> https://tpaste.us/lgWY
2023-02-08 20:13:42 <mps> without keys, ofc
2023-02-08 20:14:01 <ncopa> the wg0.conf i have correct i think
2023-02-08 20:14:10 <ncopa> i mean the /etc/network/interfaces config
2023-02-08 20:14:28 <mps> ah
2023-02-08 20:14:31 <clandmeter> https://tpaste.us/d1V5
2023-02-08 20:14:40 <clandmeter> thats what we have on wg.a.o
2023-02-08 20:14:48 <ncopa> I have ifupdown-ng-wireguard installed
2023-02-08 20:14:57 <ncopa> and i do have access to wg.a.o
2023-02-08 20:18:51 <clandmeter> ncopa: you can ofc also use wg-quick, its what you prefer. just wg with interfaces config feels more clean and obvious whats happening.
2023-02-08 20:19:04 <mps> probably need to set allowed IPs on your side
2023-02-08 20:19:26 <ncopa> i dont mind which way to get it up
2023-02-08 20:19:35 <ncopa> so it cannot get the ip from the server?
2023-02-08 20:19:44 <clandmeter> no
2023-02-08 20:19:55 <ncopa> aha
2023-02-08 20:19:56 <clandmeter> you need to define it
2023-02-08 20:20:00 <mps> clandmeter: I use /etc/network/interfaces on gateways but on clients I use wg-quick
2023-02-08 20:20:09 <ncopa> ok
2023-02-08 20:20:16 <ncopa> i will try later. thanks!
2023-02-08 20:20:25 <ncopa> need to leave this cafe now
2023-02-08 20:20:32 <clandmeter> and you can set the ip on the server side via allowedIPs 
2023-02-08 20:21:11 <clandmeter> when you are used to openvpn you expect to receive an ip automatically 
2023-02-08 20:21:38 <clandmeter> mps: I dont think it matters which side you are on"?
2023-02-08 20:21:51 <clandmeter> there is no client vs server concept with wireguard
2023-02-08 20:22:02 <mps> right
2023-02-08 20:22:10 <clandmeter> but wg-quick could be easier for some ppl.
2023-02-08 20:22:11 <mps> all are peers
2023-02-08 20:22:40 <mps> yes, wg-quick is simpler and more flexible
2023-02-08 20:22:43 <clandmeter> im still strugling to get ipv6 working on wg.a.o
2023-02-08 20:24:03 <mps> I have on my workstation 3 wg interfaces usually to different networks
2023-02-08 20:24:54 <mps> and can easily add other if I need, and can bring them up/down during day and when moving around
2023-02-08 20:25:38 <clandmeter> isnt that kind of similar with ifup ifdown?
2023-02-08 20:26:02 <mps> ofc, all this could be done by /etc/network/interfaces but it is simpler for me with wg-quick
2023-02-08 20:32:02 <ikke> ncopa: We have 172.16.252.14/24 reserved for you, if you need more, let me know
2023-02-08 21:33:57 <ncopa> i have also added 172.16.252.16/24 for my priv laptop. .14 is my work laptop
2023-02-08 21:34:07 <ncopa> and i think i got it working now. thanks
2023-02-08 21:35:29 <ikke> ncopa: ok, I'll record that
2023-02-08 21:35:53 <ikke> I track this in netbox as well fyi
2023-02-08 21:36:06 <ncopa> ok. thanks
2023-02-08 21:36:37 <ncopa> mps: how do you handle dns for wireguard?
2023-02-08 21:36:53 <ikke> intra dns?
2023-02-08 21:37:10 <ikke> I have dnsmasq forward requests for .alpin.pw to 172.16.6.3
2023-02-08 21:38:01 <mps> ncopa: as ikke told, dnsmasq is good for such things
2023-02-08 21:38:14 <ikke> clandmeter: If you have a moment, could you look at the certificates for che-bld-1 / che-ci-1? I have generated them, but on the servers it cannot decode them (bad mac)
2023-02-08 21:38:41 <ikke> the file contents and filenames are equal to what they are on dmvpn1
2023-02-08 21:45:10 <clandmeter> hmm
2023-02-08 21:45:20 <clandmeter> and both versions of dmvpn-tools are same?
2023-02-08 21:45:30 <clandmeter> and alpine version
2023-02-08 21:47:57 <clandmeter> ikke: what was the exact error?
2023-02-08 21:48:41 <ikke> /usr/bin/lua5.2: pkcs12.parse: p12_kiss.c:71:error:11800071:PKCS12 routines::mac verify failure
2023-02-08 21:48:47 <ikke> (after asking for a password)
2023-02-08 21:50:09 <clandmeter> dmvpn1 ssh key was updated?
2023-02-08 21:50:16 <clandmeter> i guess when we moved it
2023-02-08 21:50:34 <ikke> host-key? yes, I don't think I restored those
2023-02-08 21:52:39 <clandmeter> the ssl version are not the same
2023-02-08 21:52:48 <clandmeter> maybe that could be an issue?
2023-02-08 21:52:52 <ikke> perhaps
2023-02-08 21:53:12 <clandmeter> libss3 vs libssl1
2023-02-08 21:53:43 <clandmeter> i think kunkku needs to look at it.
2023-02-08 21:54:27 <ikke> ahuh
2023-02-08 21:55:29 <clandmeter> maybe we should create an issue for better visibility 
2023-02-08 21:56:26 <ikke> yup
2023-02-08 22:00:33 <clandmeter> ikke:  it was when running setup-dmvpn i guess?
2023-02-08 22:00:40 <ikke> yes
2023-02-08 22:00:51 <ikke> are you creating an issue?
2023-02-08 22:00:53 <clandmeter> ok adding infra issue
2023-02-08 22:00:53 <ikke> or should I?
2023-02-08 22:00:55 <ikke> ok
2023-02-08 22:01:18 <ikke> I as thinking of creating one here: https://gitlab.alpinelinux.org/alpine/dmvpn-tools/-/issues/
2023-02-08 22:02:14 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10788
2023-02-08 22:02:44 <clandmeter> if y ou want you can also move it
2023-02-08 22:27:40 <mps> ncopa: ikke: forgot to tell, DNS param in wg.conf also works, especially if combined with resolvconf to have more nameservers in /etc/resolv.conf
2023-02-09 12:21:38 <clandmeter> psykose: rv64 is still timing out?
2023-02-09 12:21:44 <psykose> nope
2023-02-09 12:21:57 <clandmeter> so it was not related to a timeout?
2023-02-09 12:35:02 <psykose> it seemed like it was at the time, i don't know what the actual cause was
2023-02-09 12:38:14 <ikke> It just said "killed"
2023-02-09 13:06:00 <kunkku> ikke, clandmeter: the setup-dmvpn error sounds like "wrong password"
2023-02-09 13:06:59 <kunkku> it tries to parse the password from the file name, but it the file was renamed, you need to know the password
2023-02-09 13:08:48 <ikke> kunkku: I used the exact same password
2023-02-09 13:08:53 <ikke> and also filename
2023-02-09 13:09:22 <ikke> the hashes of the files are the same as generated on the hub
2023-02-09 13:11:50 <kunkku> does it have the password in the file name?
2023-02-09 13:16:12 <ikke> it has some code in the filename yes
2023-02-09 13:16:47 <ikke> kunkku: I think you should still have access to distfiles, right?
2023-02-09 13:16:50 <ikke> sorry
2023-02-09 13:16:52 <ikke> dmvpn1
2023-02-09 13:17:17 <kunkku> I'm not in wheel
2023-02-09 13:17:45 <ikke> kunkku is in wheel
2023-02-09 13:17:54 <ikke> uid=1000(kunkku) gid=1000(kunkku) groups=1000(kunkku),10(wheel)
2023-02-09 13:18:15 <ikke> use doas
2023-02-09 13:27:20 <kunkku> I'm getting a different error
2023-02-09 13:28:45 <kunkku> 48DBB1F55C7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()
2023-02-09 13:29:15 <kunkku> ^^^ on edge
2023-02-09 13:29:39 <kunkku> on dmvpn1, it decodes correctly with openssl pkcs12 -in
2023-02-09 13:30:00 <clandmeter> the openssl version differ
2023-02-09 13:30:02 <ikke> the target server is on 3.17
2023-02-09 13:30:31 <kunkku> perhaps the pfx file has been encrypted with an algorithm not supported on 3.17?
2023-02-09 13:30:50 <kunkku> the error says RC2-40-CBC
2023-02-09 13:33:02 <ikke> openssl does return that as command
2023-02-09 13:33:32 <clandmeter> this all happens in userspace right? cannot be related to kernel
2023-02-09 13:35:23 <kunkku> it actually works if you use -legacy option on command line
2023-02-09 13:36:47 <kunkku> openssl pkcs12 -legacy -in <file>
2023-02-09 13:38:07 <ikke> so it does use some legacy cipher
2023-02-09 13:38:56 <kunkku> seems so
2023-02-09 13:39:00 <psykose> RC2-40-CBC is basically plaintext to be fair
2023-02-09 13:39:06 <psykose> and yes, those are disabled by default in openssl3
2023-02-09 13:39:14 <kunkku> and someone decided to break backward compatibility
2023-02-09 13:39:34 <psykose> insecure ciphers are not exactly what i would put in the list of "backward compatibility to keep"
2023-02-09 13:39:40 <psykose> but the -legacy provider has them
2023-02-09 13:40:08 <kunkku> well, breaking encrypting I can understand
2023-02-09 13:41:01 <kunkku> but IMO the argument does not hold for decryption
2023-02-09 13:41:05 <psykose> i would "agree" but rc2 has been broken for literally over 25 years
2023-02-09 13:41:59 <psykose> the fact it is even there with -legacy is the shocking aspect, not any broken compatibility with prehistory
2023-02-09 13:42:15 <kunkku> so does the pkcs12 thing use a better alg by default now?
2023-02-09 13:43:35 <ikke> kunkku: it's even in the man page
2023-02-09 13:43:40 <ikke> "When encountering problems loading legacy PKCS#12 files that involve, for example, RC2-40-CBC, try using the -legacy option and, if needed, the -provider-path option."
2023-02-09 13:43:44 <kunkku> if so, the solution is to 1) upgrade dmvpn1 and 2) convert the existing pfx files manually
2023-02-09 13:44:12 <ikke> kunkku: "The default encryption algorithm is AES-256-CBC with PBKDF2 for key derivation."
2023-02-09 13:44:18 <ikke> so yes
2023-02-09 13:47:53 <kunkku> it would have been fair to have a longer transition period
2023-02-09 13:48:32 <kunkku> instead of at the same time changing the default alg and blocking the old default
2023-02-09 13:48:43 <ikke> How difficult would it be to detect openssl v3 and use -legacy then?
2023-02-09 13:49:17 <ikke> Or fall-back to -legacy on mac error
2023-02-09 13:49:20 <kunkku> it does not use the cmd line util
2023-02-09 13:49:23 <ikke> ok
2023-02-09 13:50:09 <psykose> strictly speaking despite the fact that they only changed this default in openssl3, anyone using it should not have used the default in the first place for a very long time
2023-02-09 13:50:43 <ikke> psykose: or they should have changed the default a long time ago
2023-02-09 13:51:01 <psykose> «they» can always do a lot of things, yes
2023-02-09 13:51:38 <kunkku> many people (including myself) did not realize what the default was...
2023-02-09 13:51:39 <ikke> Using fancy french quotes are we :P
2023-02-09 13:52:00 <psykose> it is unfortunate, yes
2023-02-09 13:52:42 <ikke> Good API deprecation is to change the default at least some time before removing support for it
2023-02-09 13:53:17 <psykose> but again, from my perspective, said many people were always just encrypting-as-good-as-plaintext by using the default in the end
2023-02-09 13:53:35 <psykose> i can crack the key for that in under a day on my desktop pc, it's literally a waste of time for it to even be used
2023-02-09 13:54:25 <ikke> kunkku: what did you mean with manually converting the existing certs?
2023-02-09 13:56:45 <kunkku> I was thinking about 1) decrypting the pfx file using openssl cmd line, 2) re-encrypting it with the new default alg and 3) feeding the result to setup-dmvpn
2023-02-09 13:57:19 <kunkku> the alternative is to revoke the new cert and issue a new one
2023-02-09 13:57:21 <ikke> but just for new spokes, right?
2023-02-09 13:57:24 <kunkku> yes
2023-02-09 13:58:27 <kunkku> the pfx file can be deleted after running setup-dmpn
2023-02-09 13:59:38 <kunkku> or you can downgrade the spoke temporarily to openssl1 :)
2023-02-09 14:00:08 <ikke> I can manually reencrypt it
2023-02-09 14:00:21 <ikke> kunkku: do you see issues upgrading the hub?
2023-02-09 14:00:34 <kunkku> not yet :)
2023-02-09 14:00:40 <ikke> hehe :)
2023-02-09 14:00:59 <ikke> dmvpn itself is working with newer spokes
2023-02-09 14:01:46 <kunkku> do not upgrade both hubs at the same time...
2023-02-09 14:01:58 <ikke> yeah, makes sense
2023-02-09 14:02:12 <ikke> maybe it's safer to do 2 first?
2023-02-09 14:02:46 <kunkku> I think they are identical from routing point of view
2023-02-09 14:04:41 <kunkku> the pfx file contains 3 components: spoke private key, spoke cert and CA cert
2023-02-09 14:04:56 <kunkku> be sure to include them all when rebuilding the file
2023-02-09 14:05:01 <ikke> right
2023-02-09 14:06:11 <ikke> yeah, the exported pem file contains all 3 components
2023-02-09 14:07:26 <ikke> oh, it wants it seperately I guess
2023-02-09 14:10:15 <kunkku> the cmd line util is not very user friendly...
2023-02-09 14:10:38 <ikke> I have a bit of experience with it
2023-02-09 14:10:42 <ikke> but usually from pfx to pem
2023-02-09 14:11:16 <ikke> ok, seems to work, at least, pkcs12 without -legacy now accepts it
2023-02-09 14:11:34 <ikke> setup-dmvpn is happy now as well
2023-02-09 14:11:52 <ikke> kunkku: thanks
2023-02-09 14:12:29 <ikke> kunkku: when you have time (not necessarily now) it would be nice if we could also add ipv6 support to dmvpn. I tried it before (manually updating the config) but it broke routing, so I reverted it
2023-02-09 14:14:08 <ikke> clandmeter: che-bld-1 now has access to dmvpn
2023-02-09 14:15:36 <kunkku> actually there should have been four components
2023-02-09 14:17:21 <kunkku> the CA has different keys/certs for issuing/revoking
2023-02-09 14:26:25 <ikke> there are1
2023-02-09 14:26:32 <ikke> grep BEGIN crt.pem | wc -l -> 4
2023-02-09 14:28:36 <clandmeter> ikke: nice
2023-02-09 14:29:04 <clandmeter> so for all clients/spokes with updated openssl we need some workaround?
2023-02-09 14:30:07 <ikke> clandmeter: or we update the hub
2023-02-09 14:31:40 <clandmeter> nod
2023-02-09 14:31:49 <clandmeter> im stil trying to transfer usa9
2023-02-09 14:31:53 <clandmeter> but it keeps timing out
2023-02-09 14:32:18 <clandmeter> i think the network is doing 40mbits max
2023-02-09 14:32:41 <clandmeter> but it should be 1gbit
2023-02-09 14:33:29 <clandmeter> ikke: i would like to focus on moving the arm stuff first
2023-02-09 14:33:56 <clandmeter> i think we waited long enough to do this move
2023-02-09 14:34:15 <ikke> yes, agree
2023-02-09 14:34:24 <ikke> clandmeter: does it help if you tar the containers before syncing them?
2023-02-09 14:34:52 <clandmeter> not sure, i dont expect a huge difference
2023-02-09 14:34:57 <clandmeter> we just need to monitor it a bit
2023-02-09 14:35:10 <clandmeter> restart it if necessary 
2023-02-09 14:35:47 <clandmeter> 1.7T    824.3G    841.4G  49% /
2023-02-09 14:36:00 <clandmeter> its not like it did nothing :)
2023-02-09 14:36:24 <clandmeter> i think another 100G and it should be transfered 
2023-02-09 14:36:40 <kunkku> hm
2023-02-09 14:36:49 <clandmeter> i guess the CI should be easier to move over
2023-02-09 14:37:11 <kunkku> I just recalled there is "dmvpn-ca cert export"
2023-02-09 14:37:39 <kunkku> if you need to regenerate the pfx file
2023-02-09 14:38:19 <kunkku> ikke: so sorry about making you learn how to use "openssl pkcs12" :)
2023-02-09 14:38:20 <ikke> kunkku: yes, I have been using that
2023-02-09 14:38:35 <ikke> kunkku: no worry
2023-02-09 14:38:45 <clandmeter> nice switch around kunkku :p
2023-02-09 14:39:07 <ikke> clandmeter: for CI we do not need any data
2023-02-09 14:39:16 <ikke> just clone the repository and start the runner
2023-02-09 14:39:43 <ikke> clandmeter: one thing I'm thinking about is docker images
2023-02-09 14:39:54 <ikke> you cannot have the same image multiple times for different arches
2023-02-09 14:41:02 <clandmeter> i wonder if we should also keep a backup plan in the ci host
2023-02-09 14:41:12 <clandmeter> on*
2023-02-09 14:41:49 <clandmeter> i guess the lxc box will be abused much more
2023-02-09 14:50:08 <kunkku> ikke: dmvpn-tools README shows how to use IPv6 inside the VPN
2023-02-09 15:06:35 <ikke> kunkku: should it automatically work once you start adding ipv6 subnets / gre-addresses?
2023-02-09 15:06:52 <ikke> From what I understand, that assumes you set it up with the subnet already there
2023-02-09 15:08:04 <kunkku> you have to renew the cert
2023-02-09 15:08:26 <ikke> for spokes?
2023-02-09 15:08:33 <ikke> or the hub as well?
2023-02-09 15:09:05 <kunkku> for whatever you added the ipv6 addr
2023-02-09 15:09:40 <ikke> The readme mentions /etc/dmvpn-ca.conf, where you define the subnets
2023-02-09 15:10:00 <ikke> I've added it there, should I do anything afterwards?
2023-02-09 15:10:08 <kunkku> hm, good question
2023-02-09 15:12:41 <ikke> I checked the script and I noticed it sets up all kind of configuration in zebra
2023-02-09 15:14:13 <ikke> / quagga
2023-02-09 15:18:18 <ikke> it seems like it's dmvpn-ca that reads it, so maybe not
2023-02-09 15:18:49 <kunkku> quagga calls the nhrp-events script in run time
2023-02-09 15:19:23 <kunkku> it sets up ACL preventing the peer from advertising any other prefixes than those listed in the cert extension
2023-02-09 15:20:04 <kunkku> the subnet declarations in the conf file affect only hub certs
2023-02-09 15:20:17 <ikke> right
2023-02-09 15:20:23 <ikke> Maybe it's already working even
2023-02-09 15:20:28 <kunkku> hubs are allowed to advertise entire VPN address space
2023-02-09 15:20:42 <kunkku> spokes can advertise only their own subnets
2023-02-09 15:21:05 <ikke> I manually added the gre address to the hub
2023-02-09 15:21:10 <ikke> for ipv6
2023-02-09 15:21:20 <ikke> I have one spoke that has an ipv6 subnet
2023-02-09 15:21:26 <kunkku> I wonder why dvmpn1 seems to use the DEU2 cert
2023-02-09 15:21:51 <ikke> good question
2023-02-09 15:22:00 <ikke> where do you see that?
2023-02-09 15:22:25 <kunkku> /etc/swanctl/x509/dmvpn.pem 
2023-02-09 15:26:11 <ikke> oh
2023-02-09 15:26:28 <ikke> /usr/libexec/dmvpn-pfx-decode DEU2_1.MBhLBvYRXiYMcvSC.pfx might explain it
2023-02-09 15:26:54 <ikke> write_pem_file()
2023-02-09 15:27:49 <kunkku> right, but it is working because no one has restarted charon...
2023-02-09 15:28:04 <ikke> heh :)
2023-02-09 15:29:51 <kunkku> seems the dmvpn2 cert does not yet have the ipv6 addresses
2023-02-09 15:30:07 <kunkku> dmvpn1 I could not check :)
2023-02-09 15:31:02 <ikke> ok, so it does have to be regenerated with the new config?
2023-02-09 15:32:15 <kunkku> yes
2023-02-09 15:32:49 <kunkku> otherwise the IPv6 prefixes get blocked by the quagga ACL
2023-02-09 15:33:10 <ikke> ok
2023-02-09 15:33:17 <ikke> is an export enough for that?
2023-02-09 15:33:42 <kunkku> no, it just dumps out the old cert
2023-02-09 15:33:59 <ikke> oh, right
2023-02-09 15:34:20 <ikke> so cert generate
2023-02-09 15:34:39 <ikke> dmvpn-ca cert generate hub 1
2023-02-09 15:35:16 <kunkku> yes
2023-02-09 15:35:21 <ikke> and then run /usr/libexec/dmvpn-pfx-decode :P
2023-02-09 15:35:50 <kunkku> well what do you want to do?
2023-02-09 15:36:00 <ikke> let the hub use the new cert
2023-02-09 15:36:42 <kunkku> I think you need to run setup-dmvpn
2023-02-09 15:36:55 <kunkku> to set up other IPv6-related stuff
2023-02-09 15:36:59 <ikke> right
2023-02-09 15:37:08 <ikke> so you can run it multiple times
2023-02-09 15:37:58 <kunkku> yes
2023-02-09 15:39:59 <ikke> It promts 'NFLOG group [1]'
2023-02-09 15:40:19 <ikke> should I confirm?
2023-02-09 15:41:10 <kunkku> should be fine
2023-02-09 15:43:05 <kunkku> I wonder why there is no /etc/dmvpn.conf
2023-02-09 15:43:27 <kunkku> now there is, but I think IPv4 prefix length was 24
2023-02-09 15:43:30 <ikke> maybe I forgot to restore it after moving dmvpn?
2023-02-09 15:43:52 <ikke> oh, site prefix length, yes, that should be 24
2023-02-09 15:44:23 <kunkku> you may need to rm /etc/dmvpn.conf to try again
2023-02-09 15:44:26 <ikke> ok
2023-02-09 15:46:33 <ikke> No, there was no /etc/dmvpn.conf in the backup either
2023-02-09 15:50:34 <kunkku> oh, the conf file was introduced in dmvpn-tools-1.4.0
2023-02-09 15:52:38 <ikke> ok, appears the spoke cert does not have the ipv6 prefix yet either
2023-02-09 15:53:03 <ikke> should be under sbgp-ipAddrBlock, right?
2023-02-09 15:53:20 <kunkku> yes
2023-02-09 15:53:56 <kunkku> now dmvpn1's GRE address shows up in dmvpn2's IPv6 route table
2023-02-09 15:55:33 <ikke> ok, now the cert has the prefix
2023-02-09 15:56:27 <ikke> and neighbors added
2023-02-09 15:58:10 <ikke> hmm, cannot ping yet
2023-02-09 15:58:25 <ikke> ipv4 is working, ipv6 not
2023-02-09 15:59:25 <kunkku> did you update any other cert yet?
2023-02-09 15:59:48 <ikke> only dmvpn1 and NLD-DEV1
2023-02-09 16:00:09 <ikke> I'm trying to ping nld-dev1 form dmvpn1
2023-02-09 16:00:49 <ikke> I'm missing the ipv6 gre address in the routing table of dmvpn1
2023-02-09 16:01:36 <ikke> the gre-addr is defined (dmvpn-ca vpnc show does show it)
2023-02-09 16:02:14 <kunkku> true, only the subnet route is there
2023-02-09 16:03:09 <ikke> nld-dev1 has fde7:4eeb:852a::26/128 on gre1
2023-02-09 16:03:29 <ikke> and I could ping it before I renewed the certs
2023-02-09 16:03:54 <ikke> kunkku: do you explicitly need to define the gre range?
2023-02-09 16:04:00 <ikke> or is that for yourself to decide
2023-02-09 16:04:16 <kunkku> it should be inside the VPN range
2023-02-09 16:04:23 <ikke> yes, it is
2023-02-09 16:04:25 <kunkku> and not overlap site subnets
2023-02-09 16:04:43 <ikke> The range is fde7:4eeb:852a:/48
2023-02-09 16:05:12 <ikke> and the site is fde7:4eeb:852a:26::/64
2023-02-09 16:09:16 <ikke> This is on nld-dev-1: C>* fde7:4eeb:852a::26/128 is directly connected, gre1
2023-02-09 16:12:11 <kunkku> can you see N>* routes?
2023-02-09 16:12:59 <ikke> on nld-dev-1?
2023-02-09 16:13:12 <kunkku> yes
2023-02-09 16:13:23 <ikke> not in `show ipv6 route`
2023-02-09 16:13:50 <ikke> `show ip route does how some`
2023-02-09 16:13:55 <kunkku> what's on "show ipv6 nhrp cache" ?
2023-02-09 16:14:12 <ikke> no entries
2023-02-09 16:20:08 <kunkku> not even its own address?
2023-02-09 16:22:21 <ikke> I restart nhrpd and now that shows
2023-02-09 16:22:53 <kunkku> hm, setup-dmvpn should do that
2023-02-09 16:23:10 <ikke> yes, it did
2023-02-09 16:23:15 <ikke> I restarted it again
2023-02-09 16:28:22 <kunkku> perhaps restart quagga too
2023-02-09 16:29:03 <ikke> there is no quagga service
2023-02-09 16:29:11 <ikke> bgpd? zebra?
2023-02-09 16:29:36 <kunkku> well, everything :)
2023-02-09 16:29:45 <ikke> fair
2023-02-09 16:29:59 <kunkku> I mean on the spoke side
2023-02-09 16:30:09 <ikke> yes
2023-02-09 16:30:47 <ikke> restarting zebra restarts everything :)
2023-02-09 16:31:16 <ikke> no difference saidly
2023-02-09 16:36:39 <kunkku> should we restart also on hub side?
2023-02-09 16:37:51 <ikke> can try
2023-02-09 16:37:54 <ikke> feel free
2023-02-09 16:40:30 <kunkku> I'm running out of time today
2023-02-09 16:41:13 <kunkku> gotta investigate this later
2023-02-09 16:47:35 <ikke> sure, no problem, thanks so much so far :)
2023-02-09 17:25:54 <mps> hmm, ssh doesn't work between  172.16.26.16 and 172.16.26.17
2023-02-09 17:31:44 <ikke> can you ping between them?
2023-02-09 17:36:53 <mps> yes, ping works
2023-02-09 18:05:47 <ikke> I suppose something with the FW, need to figure out where it's blocked/dropped
2023-02-09 18:06:04 <ikke> There is a rule to allow everything on lxcbr0, but it is not hit
2023-02-09 18:08:18 <mps> these two lxc are on same host, afaik
2023-02-09 18:08:56 <ikke> yes, they are
2023-02-09 20:03:01 <kunkku> ikke: it would help if I had access to nld1-dev
2023-02-09 20:04:07 <ikke> kunkku: done
2023-02-09 20:04:31 <ikke> nld-dev-1.alpinelinux.org
2023-02-09 23:32:59 <kunkku> seems nhrpd gets confused if the GRE interface has multiple IPv6 addresses
2023-02-09 23:39:27 <kunkku> which leads to the question: why does "ifup gre1" add those addresses?
2023-02-09 23:39:56 <kunkku> the extra IPv6 addresses correspond to the IPv4 addresses of all interfaces in the system
2023-02-10 05:44:41 <psykose> ikke: for some reason i can't read the x86_64 dev container
2023-02-10 05:44:53 <psykose> on nld-dev-1
2023-02-10 05:44:58 <psykose> i guess that's related to the new networking :p
2023-02-10 05:45:11 <ikke> possibly
2023-02-10 05:47:14 <psykose> seemingly can't resolve the name
2023-02-10 05:47:24 <psykose> of psykose-edge-x86_64.nld-dev-1.alpin.pw
2023-02-10 05:47:47 <ikke> seems to work for me: psykose-edge-x86_64.nld-dev-1.alpin.pw has address 172.16.26.7
2023-02-10 05:48:39 <psykose> restarting wg fixed it
2023-02-10 05:48:41 <psykose> classique
2023-02-10 05:49:34 <ikke> right, I have that also from time to time (not that frequent though)
2023-02-10 05:49:37 <ikke> not sure why
2023-02-10 05:50:22 <psykose> tis a mystery
2023-02-10 05:50:47 <psykose> maybe it's time to restart deu7
2023-02-10 05:51:15 <ikke> it does get frequent restarts though
2023-02-10 05:51:25 <ikke> it's not like it has been up for months
2023-02-10 05:51:34 <psykose> yeah
2023-02-10 05:51:42 <psykose> just presently
2023-02-10 05:51:45 <psykose> moar updates
2023-02-10 05:52:21 <psykose> is it safe or did you have more networking to fix
2023-02-10 05:55:08 <ikke> should be safe
2023-02-10 05:55:49 <psykose> poof
2023-02-10 05:56:12 <psykose> may as well since the whole openssl thing et al
2023-02-10 06:12:33 <psykose> that should be deu1 and gbr2 too
2023-02-10 06:24:24 <psykose> not great
2023-02-10 06:26:57 <psykose> restarting traefik fixed it
2023-02-10 06:26:59 <psykose> bizarre
2023-02-10 08:29:56 <kunkku> ikke: the CA database sync from dmvpn1 to dmvpn2 is broken
2023-02-10 08:30:08 <kunkku> seems the SSH key is missing
2023-02-10 08:31:23 <ikke> kunkku: oh, I was not aware that it used ssh, so I never set that up probably 
2023-02-10 08:33:44 <kunkku> regarding IPv6, there seems to be two issues
2023-02-10 08:33:59 <kunkku> 1) ifup adding extra addresses to the GRE interface
2023-02-10 08:34:58 <kunkku> 2) NUD is not working with GRE properly and messes up the neigbor table entries
2023-02-10 08:40:30 <ikke> I'm wondering where those extra ips come from
2023-02-10 08:41:15 <ikke> Might also explain a dns issue I sometimes notice where it returns a ipv6 adres for a container with an Ipv4 address embedded. 
2023-02-10 08:42:10 <kunkku> it doesn't happen with AL 3.15
2023-02-10 08:44:06 <kunkku> at least the hubs don't have those
2023-02-10 11:05:57 <ikke> kunkku: I wonder if it's ifupdown that does it, or something that is done by the kernel
2023-02-10 13:35:48 <kunkku> yes, it requires more investigation
2023-02-10 13:36:47 <kunkku> I found a workaround for the NUD issue: echo 1 > /proc/sys/net/ipv6/neigh/gre1/app_solicit
2023-02-10 13:37:29 <kunkku> nhrpd is supposed to do the same on startup (via netlink) but it doesn't work for some reason
2023-02-10 16:28:55 <clandmeter> ikke: arm1 transfer is ready
2023-02-10 16:29:08 <clandmeter> i dont have time today but maybe this weekend or so we can move it
2023-02-10 16:29:33 <clandmeter> stil need to config lxc network and such
2023-02-10 16:58:07 <ikke> OK. For CI I think I'll still use separate vms for each arch 
2023-02-10 18:32:17 <ncopa> i need to send out release announcement soon, but my email set up does not work yet
2023-02-10 18:36:51 <ncopa> is anyone available to help me?
2023-02-10 18:37:09 <ncopa> i need to know my username for smtp.alpinelinux.org
2023-02-10 18:37:20 <ncopa> is it ncopa or ncopa@alpinelinux.org
2023-02-10 18:45:09 <ikke> ncopa: in sasldb it's the latter, but I think you authenticate just with the username
2023-02-10 18:47:01 <ikke> I'm not sure the password I provided is correct
2023-02-10 18:54:52 <ncopa> i have tried both
2023-02-10 18:55:46 <ikke> checking the logs
2023-02-10 18:56:23 <ikke> it just says authentication failure
2023-02-10 18:59:56 <ikke> I think it worked
2023-02-10 19:01:05 <ncopa> yeah, finally. Thank you very much
2023-02-11 13:30:08 <clandmeter> ikke: i need your advise
2023-02-11 13:30:34 <clandmeter> as you are our gitlab specialist :)
2023-02-11 13:30:35 <ikke> clandmeter: how can I help
2023-02-11 13:30:38 <ikke> lol
2023-02-11 13:31:00 <clandmeter> im having trouble triggering a pipeline when aports receives a tag
2023-02-11 13:31:21 <ikke> You mean in a downstream project?
2023-02-11 13:31:28 <clandmeter> yes kind of 
2023-02-11 13:31:55 <clandmeter> I dont think there exist a solution where a downstream pipeline is monitoring an upstream project
2023-02-11 13:32:08 <clandmeter> but i can be wrong 
2023-02-11 13:32:30 <ikke> I'm not aware of any mechanism either
2023-02-11 13:32:43 <clandmeter> you can define a child pipeline, but then you need to mess with the upstream proejct
2023-02-11 13:33:21 <clandmeter> so now i need to define a pipeline trigger
2023-02-11 13:33:30 <clandmeter> which will be triggered by webhooks
2023-02-11 13:33:41 <clandmeter> webhook.a.o
2023-02-11 13:33:46 <ikke> nod
2023-02-11 13:33:59 <clandmeter> but for the latest tag (.2) it did not run my pipeline
2023-02-11 13:34:05 <clandmeter> and i have no idea why
2023-02-11 13:34:14 <clandmeter> cause it does not log anything in my project
2023-02-11 13:34:34 <clandmeter> and webhook.a.o does not show any error in its logs
2023-02-11 13:34:50 <clandmeter> so to both sides, nothing went wrong :)
2023-02-11 13:35:17 <clandmeter> tbh, i dont really like this setup. its too complicated
2023-02-11 13:35:51 <ikke> how late was the tag pushed, do you know?
2023-02-11 13:36:02 <clandmeter> yesterday
2023-02-11 13:36:06 <clandmeter> 17:30 i think?
2023-02-11 13:36:14 <clandmeter> you can see it from aports tags
2023-02-11 13:36:37 <clandmeter> Feb 10, 2023 5:44pm GMT+0100
2023-02-11 13:37:06 <clandmeter> i can see it was hit on webhook.a.o
2023-02-11 13:37:34 <clandmeter> but as my script does not output anything, i dont see if it was actually run or not.
2023-02-11 13:37:48 <clandmeter> the mqtt script did run, as it has some debug output
2023-02-11 13:38:11 <clandmeter> i added some debug msgs now, so i know it for the next time.
2023-02-11 13:38:45 <ikke> https://gitlab.alpinelinux.org/admin/hooks/3/hook_logs/664263
2023-02-11 13:39:00 <clandmeter> yes
2023-02-11 13:39:10 <clandmeter> thats the one i can also see from webhook.a.o
2023-02-11 13:39:27 <clandmeter> but there are 2 scripts
2023-02-11 13:39:31 <ikke> We don't handle that
2023-02-11 13:39:41 <ikke> oh
2023-02-11 13:39:45 <ikke> push or tag_push
2023-02-11 13:40:12 <clandmeter> what do you mean?
2023-02-11 13:40:40 <ikke> in the mqtt-publish.lua script
2023-02-11 13:40:54 <ikke> We check the event type, but I see we do handle tag_push as well
2023-02-11 13:41:12 <clandmeter> yes
2023-02-11 13:41:21 <clandmeter> but i dont refer to this script
2023-02-11 13:41:23 <clandmeter> this script did run
2023-02-11 13:41:45 <clandmeter> alpine-disk-image should have also run
2023-02-11 13:41:45 <ikke> ok
2023-02-11 13:41:58 <clandmeter> and it should have run before the mqtt script
2023-02-11 13:42:02 <ikke> The one that is commented out?
2023-02-11 13:42:17 <clandmeter> commented out?
2023-02-11 13:42:28 <ikke> ttps://tpaste.us/NB7w
2023-02-11 13:42:32 <ikke> https://tpaste.us/NB7w
2023-02-11 13:42:48 <clandmeter> ah you are in webhook
2023-02-11 13:42:52 <ikke> yes
2023-02-11 13:42:52 <clandmeter> thats not running
2023-02-11 13:42:54 <ikke> ok
2023-02-11 13:42:58 <clandmeter> webhook-new :)
2023-02-11 13:43:03 <clandmeter> i think we need to rename that
2023-02-11 13:43:09 <ikke> oh, yeah, not first time I made that mistake
2023-02-11 13:43:16 <clandmeter> webhook and webhook-old :p
2023-02-11 13:43:19 <ikke> can we nuke webhook?
2023-02-11 13:43:25 <ikke> do we need to keep it?
2023-02-11 13:43:37 <clandmeter> i dont know, move it to attic or something
2023-02-11 13:43:58 <clandmeter> i dont remember why i kept it :)
2023-02-11 13:44:14 <ikke> moved it
2023-02-11 13:44:21 <ikke> didn't touch webhook-new yet
2023-02-11 13:44:24 <ikke> touched*
2023-02-11 13:44:32 <clandmeter> ok, its obvious now i think :)
2023-02-11 13:46:01 <clandmeter> it would have been nice if gitlab would support logging of pipeline triggers
2023-02-11 13:46:09 <clandmeter> but it looks like it. does not
2023-02-11 13:47:04 <ikke> check the api logs on deu2
2023-02-11 13:47:28 <clandmeter> on gitlab container?
2023-02-11 13:47:44 <ikke> storage/log
2023-02-11 13:48:07 <ikke> gitlab-api.[json|log]
2023-02-11 13:49:13 <ikke> did you manually trigger this?  [webhook] 2023/02/11 13:37:48 [ea51d1] command output: Running alpine-disk-image..Running mqtt-publish..push message has been send
2023-02-11 13:49:43 <clandmeter> yes
2023-02-11 13:50:09 <clandmeter> it should not trigger anything
2023-02-11 13:50:17 <clandmeter> just output those 2 new msgs 
2023-02-11 13:51:42 <clandmeter> which ip is for gitlab to ssh into?
2023-02-11 13:52:04 <clandmeter> ah its in my config :)
2023-02-11 13:53:02 <ikke> same IP, port 2222
2023-02-11 13:55:11 <clandmeter> you mean this file? /srv/docker/gitlab/log/gitlab/api_json.log
2023-02-11 13:55:23 <ikke> yes
2023-02-11 14:06:55 <clandmeter> dont see much useful info, kind of unclear what is what.
2023-02-11 14:07:38 <clandmeter> if it fails again, i might look into adding a child pipeline in aports
2023-02-11 14:09:15 <ikke> Have you tried manually invoking the webhook with the payload from gitlab, or is that what produced those logs
2023-02-11 14:09:45 <clandmeter> it will also trigger the mqtt part
2023-02-11 14:09:49 <clandmeter> which i dont want to touch 
2023-02-11 14:10:12 <clandmeter> i think the whole webhook setup is sub optimal
2023-02-11 14:12:21 <clandmeter> we could make ci send the msg directly to mqtt
2023-02-11 14:13:56 <ikke> why would that work better than a webhook?
2023-02-11 14:15:30 <clandmeter> you remove the webhook step?
2023-02-11 14:17:22 <clandmeter> now gitlab sends the payload to webhook.a.o which parses it and send its over to mqtt
2023-02-11 14:19:38 <clandmeter> we could probably do the whole release part via CI
2023-02-11 14:19:50 <clandmeter> no need for any messages
2023-02-11 14:22:01 <clandmeter> but thats not my problem right now
2023-02-11 14:37:11 <clandmeter> ikke: what i also can do is just copy the image generation part to aports. so each tag would generate also a disk image. And i could trigger a downnstream pipeline to do something with the image not directly associated to the project.
2023-02-11 15:23:59 <ikke> that's certainly possible
2023-02-11 15:24:28 <ikke> clandmeter: you had at one point an rv64 unmatched board, is that still there / connected?
2023-02-11 15:24:41 <ikke> (cleaning up netbox a bit)
2023-02-11 16:11:57 <clandmeter> yes should be online
2023-02-11 16:12:03 <clandmeter> not sure it actually is
2023-02-11 16:12:08 <clandmeter> it was some time ago
2023-02-11 16:12:15 <clandmeter> its on dmvpn
2023-02-11 16:34:07 <ikke> alright
2023-02-11 18:13:07 <clandmeter> ikke: i think ill do something like this https://tpaste.us/xyeQ
2023-02-11 18:13:58 <clandmeter> ill test it with another repo before adding it to aports
2023-02-11 18:29:32 <psykose> looks good in terms of intent :)
2023-02-11 21:19:51 <tomalok> ikke: https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/60 -- new set of cloud images have been build/published
2023-02-11 21:23:21 <ikke> tomalok: pushed
2023-02-11 21:35:16 <tomalok> ikke: thx!
2023-02-12 09:12:19 <ikke> kunkku: btw, did you fix the syncing between the hubs, or should I do that?
2023-02-12 11:06:36 <ikke> clandmeter: cleaned up sites in netbox: https://netbox.alpin.pw/dcim/sites/?status=active
2023-02-12 12:44:12 <ikke> Going to reboot dmvpn hub
2023-02-12 12:44:18 <ikke> upgraded it to 3.17
2023-02-12 12:44:26 <ikke> (dmvpn1)
2023-02-12 12:59:18 <ikke> clandmeter: openvpn no longer supports BF-CBC as cipher. Not sure if we still have clients using it, but if so, they might now fail to connect
2023-02-12 13:48:58 <ikke> clandmeter: ping, I see that all space in the vg for che-ci-1 has been assigned to the rootlv, which means that I cannot use the same VM setup as I have now
2023-02-12 15:05:25 <psykose> ikke: think the riscv ci doesn't work anymore https://gitlab.alpinelinux.org/hjaekel/aports/-/jobs/971683
2023-02-12 15:05:59 <ikke> psykose: yeah, had similar issues with building images
2023-02-12 15:06:13 <psykose> for riscv or sporadic in general?
2023-02-12 15:06:34 <psykose> the rest is ok (aside from that 1/1000 fail ig)
2023-02-12 15:07:19 <ikke> riscv
2023-02-12 15:13:21 <psykose> yeah weird
2023-02-12 15:30:55 <ikke> clandmeter: installed gitlab-runner-aarch64 now the new ci host
2023-02-12 15:31:00 <ikke> will do the other 2 later
2023-02-12 15:31:07 <ikke> (just os installation atm)
2023-02-12 19:02:46 <clandmeter> ikke: ok, we can resize the fs and lv
2023-02-12 19:03:35 <clandmeter> or you can use files instead of bdev
2023-02-12 19:13:57 <ikke> Yes, I used qcow2 files now
2023-02-12 19:15:55 <clandmeter> ikke: did you ever manage to cleanup the bot madness on gitlab?
2023-02-12 19:16:35 <ikke> Yes
2023-02-12 19:16:48 <ikke> The spam you mean? 
2023-02-12 19:17:06 <clandmeter> i have one MR which still has a gazillion labels operations
2023-02-12 19:17:46 <ikke> Ah
2023-02-12 19:18:06 <ikke> I cleaned up one
2023-02-12 19:18:16 <clandmeter> psykose: what happened to libguestfs?
2023-02-12 19:18:33 <clandmeter> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/42815
2023-02-12 19:19:17 <clandmeter> that mr consumes a lot of cpu on my macbook
2023-02-12 19:19:22 <ikke> yeaha
2023-02-12 19:22:34 <ikke> trying to find the query again
2023-02-12 19:27:46 <ikke> ok, found the table again
2023-02-12 19:27:53 <ikke> 36k events
2023-02-12 19:28:24 <clandmeter> [7177802.927919] EXT4-fs (dm-1): Inode 39849302 (00000000ab2db619): i_reserved_data_blocks (1) not cleared!
2023-02-12 19:28:32 <clandmeter> usa9-dev1
2023-02-12 19:47:50 <ikke> clandmeter: label events are gone
2023-02-12 19:50:27 <clandmeter> thx
2023-02-12 19:54:43 <ikke> is that message bad?
2023-02-12 22:04:52 <clandmeter> its red in dmesg and related to fs. i assume that's not good. 
2023-02-13 06:51:00 <psykose> clandmeter: it doesn't work at all
2023-02-13 06:51:03 <psykose> not much else to it
2023-02-13 06:51:35 <psykose> it relies on a bunch of per-distro hardcoded things like "what package from what repository to automatically fetch and install for stuff" so i concluded it's garbage and nobody should use it
2023-02-13 09:18:52 <kunkku> ikke: I didn't fix the CA sync
2023-02-13 09:19:55 <kunkku> what's needed is ssh-keygen on dmvpn1 and put that to auth keys on dmvpn2
2023-02-13 09:20:50 <kunkku> there already was one key which looked like the sync key. I wonder if it is for the old dmvpn1
2023-02-13 09:51:33 <ikke> Probably, but then I'll look at it
2023-02-13 09:51:45 <ikke> Upgrade to 3.17 worked fine 
2023-02-13 09:51:53 <ikke> And new certs are accepted 
2023-02-13 11:42:51 <ikke> psykose: let's continue here
2023-02-13 11:42:53 <ikke> POST /bearp2p/profit-crypto/-/issues HTTP/1.1" 429 65 "https://gitlab.alpinelinux.org/viktorp2p/p2p-dohod/-/issues/new"
2023-02-13 11:43:11 <ikke> spammer
2023-02-13 11:52:45 <psykose> ah
2023-02-13 11:52:46 <psykose> oops
2023-02-13 11:52:50 <psykose> didn't see this until after all the rest
2023-02-13 11:54:06 <ikke> np
2023-02-13 11:54:11 <psykose> most requests are 304'd, i wonder if that's some special cached content everyone is checking or if the webui mostly generates 304's on requests
2023-02-13 11:54:32 <ikke> possibly static assets
2023-02-13 11:54:37 <ikke> css / js
2023-02-13 11:54:40 <ikke> images
2023-02-13 11:54:59 <psykose> yea
2023-02-13 11:55:13 <ikke> no, also happens on page requests
2023-02-13 11:56:25 <psykose> yeah, all webui stuff then
2023-02-13 14:18:08 <mps> ncopa: what you think about adding versioned rust to alpine, i.e. 1.64 version? for now it is needed to build kernel with asahi gpu driver because kernel doesn't build with newest rust
2023-02-13 14:19:44 <mps> and asahi becoming more and more popular and probably alpine devs and users would like to try to build gpu driver
2023-02-13 14:21:54 <mps> and probably versioned rust will be needed in future for kernel for other things
2023-02-13 14:24:27 <psykose> no
2023-02-13 14:24:47 <psykose> what error do you get
2023-02-13 14:24:48 <mps> could you elaborate
2023-02-13 14:25:21 <mps> a lot of errors
2023-02-13 14:26:20 <psykose> post the errors
2023-02-13 14:28:41 <psykose> i.e. in a pastebin
2023-02-13 14:30:08 <mps> ok if you think so, I can build it for myself locally and don't care anymore
2023-02-13 14:30:19 <mps> as I did already
2023-02-13 14:30:33 <psykose> i asked for the errors specifically
2023-02-13 14:31:01 <psykose> rust doesn't support anything except the latest version (which is what it is i guess), but specifically there is very little actual breakage between them
2023-02-13 14:31:07 <psykose> i bet this is not as hard to fix as one thinks
2023-02-13 14:31:10 <mps> no, I will not because I know it will be can of worms to keep patches and fixes
2023-02-13 14:31:30 <psykose> there is also some likelihood the error is not related to the version
2023-02-13 14:31:54 <psykose> i'm not telling you to add random patches, i'm asking for the errors
2023-02-13 14:31:58 <psykose> can you please just post it
2023-02-13 14:32:00 <mps> rust 1.64 works fine while newest don't
2023-02-13 14:33:07 <mps> try to check kernel with 'make LLVM=1 rustavailable' and latest rust installed
2023-02-13 14:33:46 <mps> https://docs.kernel.org/next/rust/quick-start.html
2023-02-13 14:34:13 <psykose> it says it's available and nothing else
2023-02-13 14:34:19 <psykose> can you please just post the errors
2023-02-13 14:34:29 <mps> about versions?
2023-02-13 14:34:38 <psykose> it would have taken you less effort to just post a log by now
2023-02-13 14:35:21 <mps> no, will not be easy now for me, because I already 'fixed' my machine for old rust
2023-02-13 14:35:43 <psykose> ok, ping me when you reproduce the failure with the log if you ever do
2023-02-13 14:35:44 <mps> have to ruin it just to post a lot of errors of kernel build
2023-02-13 14:36:56 <mps> I can upload rust 1.64 pkgs to dev.a.o for those who need them
2023-02-13 14:38:12 <mps> (long time passed when I built rust pkgs for alpine, 3-4 years)
2023-02-13 15:02:48 <psykose> ok i figured it out
2023-02-13 15:03:09 <psykose> indeed it needs porting to new versions because the whole rust-linux kernel thing is a bunch of very unstable stuff
2023-02-13 15:03:28 <psykose> i.e. to .66 it's in https://github.com/Rust-for-Linux/linux/commit/3f893e11f62f880f6d4e2bf6cda199a4366203a7 for the next tree (and this gets merged to actual linux eventually)
2023-02-13 15:03:29 <psykose> very cute
2023-02-13 15:04:01 <psykose> you do indeed need older rust versions to build this, but afaict this will never be stable in this state
2023-02-13 15:04:25 <psykose> i.e. by the time "mainline rust drivers" exist they will have figured this out and not break all the time probably
2023-02-13 15:04:29 <psykose> currently it's a mess :D
2023-02-13 15:04:33 <mps> I don't expect stable of hipsters langs
2023-02-13 15:06:43 <mps> I know that the rust is mess and because this I stopped to work with it 3 years ago
2023-02-13 15:07:59 <mps> but it somehow (I guess how) managed to slip in kernel and that is thing which making me to think about switch to *BSD
2023-02-13 15:14:48 <psykose> i mean the kernel itself has the same issue even for the C code. you ever tried building a random arbitrary 3.x 4.x kernel from some android fork with gcc12? it's the same shit, you need "gcc6" or whatever, i.e. an older (or newer) compiler
2023-02-13 15:15:01 <psykose> the existing rust support is *more* unstable than that, yes, but in general nothing is perfect there
2023-02-13 15:16:21 <psykose> postmarketos keeps a gcc4 and gcc6 around exactly for that reason :D
2023-02-13 15:16:49 <psykose> if it comes to it in the end (for stable kernels), we will have to do that for exactly rust, but it's not there yet
2023-02-13 15:17:10 <mps> yes, because that I insisted to keep gcc6 in alpine ;-)
2023-02-13 15:17:32 <psykose> it's not kept for anyone to use, afaik you can't even install it and it fails
2023-02-13 15:17:40 <psykose> but it's used for the java bootstrap :p
2023-02-13 15:17:48 <psykose> otherwise you can't build gcc-gcj-compat
2023-02-13 15:17:52 <psykose> to build all of java
2023-02-13 15:18:03 <psykose> i heard someone was bringing gcj back so there might be a newer one soon
2023-02-13 15:18:07 <psykose> then we can drop gcc6 :-)
2023-02-13 15:18:47 <mps> last time I tried with gcc6 it worked but that was 2 years ago
2023-02-13 15:19:02 <psykose> yeah some stuff conflicts with it and libstdc++
2023-02-13 15:19:10 <psykose> i could fix it probably but it's not meant to be used, so..
2023-02-13 15:19:32 <psykose> ah, you can install gcc6
2023-02-13 15:19:35 <psykose> but not g++6
2023-02-13 15:20:06 <mps> ah, could be, I don't tried g++6
2023-02-13 15:20:16 <mps> didn't*
2023-02-13 15:21:04 <mps> reason for me was to build old 3.x kernels for some chromebooks
2023-02-13 15:23:01 <psykose> i think i fixed it, it was a very tiny issue
2023-02-13 15:23:04 <psykose> lemme test
2023-02-13 16:54:39 <ikke> hmm, curious, there is someone creating projects for each issue that they are facing and using CI
2023-02-13 16:54:59 <ikke> nothing abusive
2023-02-13 16:58:47 <psykose> which one
2023-02-13 16:59:25 <psykose> i think i saw something like that
2023-02-13 17:00:04 <ikke> https://gitlab.alpinelinux.org/dHannasch/curl-7.86-cannot-connect-to-hosts-that-curl-7.85-can
2023-02-13 17:02:03 <psykose> that's the one i was thinkin
2023-02-13 17:02:53 <psykose> yeah that's o
2023-02-13 17:02:55 <psykose> ok*
2023-02-13 17:02:56 <psykose> i guess
2023-02-13 17:04:22 <psykose> funnily the other issue wasn't one
2023-02-13 17:04:26 <psykose> https://gitlab.alpinelinux.org/dHannasch/latest-python3-refuses-to-trust-certificate-as-old-version-did/-/pipelines
2023-02-13 17:04:34 <psykose> the reproduction steps shows it worked correctly afaict
2023-02-13 17:05:01 <psykose> ah, comments in the yaml
2023-02-13 17:05:24 <psykose> if anything these are waaay better reports than normal
2023-02-13 17:05:26 <psykose> i quite like it
2023-02-13 17:07:38 <minimal> psykose: so it *is* actually working now as expected? It's not clear
2023-02-13 17:08:21 <psykose> idk
2023-02-13 17:09:07 <minimal> I have an interest as I had a MR merged 1 month ago for ca-certificates to fixup some minor stuff
2023-02-13 17:09:52 <psykose> it looks like it does and no aports issue was opened
2023-02-13 17:09:56 <psykose> i assume it is
2023-02-13 17:10:03 <minimal> yeah me too
2023-02-13 18:48:35 <ikke> mps: seems like the m.2 ssd is now working as well
2023-02-13 18:48:44 <ikke> I had to connect both USB cables of the enclosure
2023-02-13 18:49:58 <ikke> hmm, not working properly yet
2023-02-13 18:55:15 <mps> ikke: how it is seen with OS, as sda disk or something else
2023-02-13 18:55:25 <ikke> mps: yes
2023-02-13 18:55:33 <psykose> what enclosure is it
2023-02-13 18:55:55 <psykose> also sda sounds correct for an m.2 ssd if it's sata?
2023-02-13 18:56:14 <mps> I have no idea how nvme works over usb, as usual usb storage or something special to nvme
2023-02-13 18:56:25 <ikke> no brande
2023-02-13 18:56:33 <ikke> small metallic enclosure
2023-02-13 18:58:17 <ikke> lsblk: sdb            8:16   0    0B  0 disk
2023-02-13 18:58:19 <ikke> blkikd empty
2023-02-13 18:58:24 <ikke> sfdisk /dev/sdb
2023-02-13 18:58:29 <ikke> sfdisk: cannot open /dev/sdb: No such file or directory
2023-02-13 18:58:45 <psykose> aha
2023-02-13 18:58:48 <psykose> broken as hell
2023-02-13 18:59:10 <mps> does dmesg give more info
2023-02-13 19:00:07 <ikke> https://tpaste.us/KEd9 https://tpaste.us/KEd9 
2023-02-13 19:01:18 <ikke> Does not work on my laptop, but does work on my non-alpine desktop
2023-02-13 19:02:48 <psykose> that's weird
2023-02-13 19:04:22 <mps> maybe some other drivers and/or options must be enabled in kernel
2023-02-13 19:04:49 <ikke> case where it works: https://tpaste.us/ogQE 
2023-02-13 19:07:03 <mps> ah asmedia driver is not enabled in kernel, maybe this is problem
2023-02-13 19:08:21 <ikke> ah, yeah, maybe
2023-02-13 19:08:28 <minimal> both SATA and NVME devices connected via USB adaptor should appear as sda
2023-02-13 19:09:53 <ikke> minimal: I do see it as sd[x]
2023-02-13 19:09:59 <ikke> but I cannot use it 
2023-02-13 19:10:23 <minimal> ikke: was referring to mps' comment about nvme over usb
2023-02-13 19:12:19 <mps> maybe we could set CONFIG_USB_STORAGE_DEBUG=y in kernel to show more debug info
2023-02-13 19:12:47 <minimal> mps: from the output it is using UAS rather than USBSTORAGE though...
2023-02-13 19:13:03 <mps> minimal: yes
2023-02-13 19:13:43 <mps> but usbstorage is needed even for uas iirc
2023-02-13 19:13:57 <minimal> some USB adaptors are known to have problems with UAS, I think there's a blacklist in the kernel for such devices
2023-02-13 19:22:26 <mps> ikke: couldn't find anything as hint in kernel
2023-02-13 19:24:48 <mps> maybe we could ask on #riscv
2023-02-13 19:35:23 <minimal> ikke: maybe try disabling UAS for that device and see if it then works ok? https://leo.leung.xyz/wiki/How_to_disable_USB_Attached_Storage_(
2023-02-13 19:35:48 <minimal> oops, https://leo.leung.xyz/wiki/How_to_disable_USB_Attached_Storage_(UAS)
2023-02-13 19:36:44 <psykose> USB_UAS is already enabled for edge-riscv64 i think
2023-02-13 19:37:03 <ikke> I'm using the starfive kernel
2023-02-13 19:37:54 <ikke> https://gitlab.alpinelinux.org/nmeum/alpine-visionfive/-/tree/main/starfive/linux-starfive
2023-02-13 19:38:12 <psykose> ah
2023-02-13 19:38:32 <psykose> it has it on
2023-02-13 19:38:58 <psykose> also i misread as "enabling uas"
2023-02-13 19:39:17 <minimal> ikke: I meant try to modprobe the usb-storage module with a quirks param to (temporarily) disable UAS for that device
2023-02-13 19:40:25 <minimal> if usb-storage is compiled into the kernel then I assume there must be a similar method to pass a cmdline option
2023-02-13 19:40:26 <psykose> rmmod uas :p
2023-02-13 19:40:33 <psykose> but yes not a module
2023-02-13 19:41:10 <minimal> ah, that article does document cmdline also, "usb-storage.quirks=xxx:xxx:u"
2023-02-13 19:42:32 <minimal> Lots of Raspberry Pi users have found "random" USB-to-SATA (and I assume also NVME) adaptors/cables to be hit-and-miss for reliability. I think someone maintains a webpage with a list of tested adaptors
2023-02-13 19:43:32 <mps> ikke: could you post usb ID of device, maybe we can find something with this info
2023-02-13 19:43:39 <ikke> 174c:2362
2023-02-13 19:43:47 <minimal> ikke: indeed some of the dmesg errors you had are mentioned at the end of that article
2023-02-13 19:44:26 <ikke> interesting
2023-02-13 19:44:54 <ikke> still nothing
2023-02-13 19:45:38 <ikke> I've added usb-storage.quirks=174c:2362:u to my cmdlinie
2023-02-13 19:46:39 <minimal> ikke: could it be a power issue? You said you had to connect "both cables" - to 2 USB3 ports?
2023-02-13 19:46:46 <ikke> yes
2023-02-13 19:46:53 <minimal> I'd expect a single USB3 port should provide enough power
2023-02-13 19:47:13 <ikke> The 2nd time I tried it the 2nd cable didn't help, but after reboot I saw the disk again
2023-02-13 19:47:17 <minimal> USB2 would be a different matter (which is why I assume it has 2 connectors, for USB2 use)
2023-02-13 19:47:34 <mps> https://linux-hardware.org/?id=usb:174c-2362
2023-02-13 19:47:49 <ikke> The port has a blue slab, so I assume USB3
2023-02-13 19:48:15 <ikke> But it could be the usb adapter I use
2023-02-13 19:48:16 <minimal> ikke: the dmesg mentioned SuperSpeed, so is USB3.x
2023-02-13 19:48:22 <ikke> right
2023-02-13 19:49:05 <mps> I have to usb-to-ssd enclosures and on them always have to power it over another usb power adapter
2023-02-13 19:49:07 <minimal> but not RidiculousSpeed which is USB4 ;-)
2023-02-13 19:49:47 <minimal> mps: "it depends" on factors such as the storage device and adaptor's power draw
2023-02-13 19:50:22 <mps> minimal: yes
2023-02-13 19:50:34 <minimal> for some SSD/HDD vendors it is easy to find out the idle/busy power usage, for others they don't seem to provide that info
2023-02-13 19:50:44 <mps> 
ACTION is first hardware man

2023-02-13 19:53:48 <mps> according to linux-hardware.org all needed is enabled for this adapter
2023-02-13 19:57:19 <minimal> ikke: silly suggestion, if there's more than 1 USB3 port maybe try swapping the connectors around? It is possible that different ports may have different (USB3.x) capabilities...
2023-02-13 19:59:48 <ikke> Doesn't seem to help
2023-02-13 20:00:13 <ikke> Not sure if I should undo the querks
2023-02-13 20:00:21 <ikke> lsblk does not list it at all anymore
2023-02-13 20:00:24 <ikke> though /dev/sda appears
2023-02-13 20:00:59 <psykose> silly hack: download some debian riscv kernel :p
2023-02-13 20:01:10 <minimal> recent related article: https://www.martinrowan.co.uk/2023/01/argon-one-nvme-board-slower-than-sata/
2023-02-13 20:02:08 <ikke> I don't have such a board
2023-02-13 20:02:21 <minimal> its the same chipset
2023-02-13 20:02:23 <ikke> ok
2023-02-13 20:02:39 <minimal> "Some users on the Argon40 community forum also appear to have the board using the usb-storage driver instead of uas."
2023-02-13 20:02:51 <minimal> "Hooking the new board up and it has the same vid and pid, and uses usb-storage not uas driver"
2023-02-13 20:02:59 <ikke> Strange that it does work correctly on archlinux
2023-02-13 20:03:16 <minimal> ikke: with which kernel version?
2023-02-13 20:03:43 <ikke> 6.1.4
2023-02-13 20:04:15 <ikke> the rv64 board has 6.1.0
2023-02-13 20:05:15 <mps> ikke: it is 6.1-rc
2023-02-13 20:05:20 <ikke> rc6
2023-02-13 20:05:53 <minimal> ikke: you're testing 2 different SBCs for Arch and Alpine?
2023-02-13 20:05:59 <mps> yes, so nothing related changed with 6.1 release
2023-02-13 20:06:09 <ikke> minimal: no, arch is my desktop
2023-02-13 20:06:25 <ikke> trying my alpine laptop now, but needed to reboot it
2023-02-13 20:07:06 <minimal> ok, so you're not really testing like-with-like. One suspicion is power-related issues with the SBC
2023-02-13 20:07:19 <ikke> Yes, that might as well be
2023-02-13 20:07:22 <ikke> but I only have one sbc
2023-02-13 20:07:37 <ikke> so I don't have the luxury to test a lot of variations
2023-02-13 20:08:06 <ikke> yeah, it's working on my alpine laptop as well
2023-02-13 20:08:30 <minimal> again from that article:
2023-02-13 20:08:31 <minimal> "The NVMe solutions have the highest power consumption. Given there doesn’t appear to be a performance gain, due to the USB 3.0 limitations, there doesn’t appear to be a compelling case for considering an NVMe drive."
2023-02-13 20:08:39 <minimal> "Based on information from Sabrent, an SSD typically has a maximum power requirement of less than 4W, whereas NVMe will be between 3.5 and 8.5W. This increased power demand may be too much for what the Raspberry Pi 4 can deliver from its USB ports. Especially as the total available power from the Raspberry Pi 4 USB ports is shared between all ports and limited to 1.2A (~6W)."
2023-02-13 20:09:39 <minimal> whilst that is from a RPI perspective the same could apply for your SBC - using a NVME rather than a SATA SSD could perhaps be drawing too much from the board (especially depending on what else you have connected, USB or othewise)
2023-02-13 20:09:44 <ikke> I had this nvme lying around, so thought to use it for the vision-five board
2023-02-13 20:10:01 <mps> lsusb.py show milliampers for devices
2023-02-13 20:10:16 <minimal> ikke: got a spec sheet for the NVME to see its draw?
2023-02-13 20:11:04 <mps> minimal: right, '[USB 3.20, 10000 Mbps, 896mA] (Western Digital' one of my usb ssd
2023-02-13 20:11:30 <mps> very fast
2023-02-13 20:12:47 <ikke> samsung  pro
2023-02-13 20:12:53 <ikke> 650
2023-02-13 20:12:57 <ikke> 950*
2023-02-13 20:26:18 <minimal> whereas to use mps as an example, his Seagate SSD is 0.896A
2023-02-13 20:26:43 <mps> how much Ampers have adapter which come with v1 board
2023-02-13 20:26:49 <minimal> so your NVME is using 1/4 A more than his
2023-02-13 20:27:09 <ikke> minimal: the one that came with it was I believe 1.5A
2023-02-13 20:27:10 <mps> s/have/give/
2023-02-13 20:27:10 <alpine-meetbot> mps meant to say: how much Ampers give adapter which come with v1 board
2023-02-13 20:27:32 <ikke> mps: *
2023-02-13 20:28:02 <mps> I use another adapter for v1 which have 3A
2023-02-13 20:29:10 <minimal> ikke: sorry, I referring to the USB/NVME adaptor - it will have some (likely small) current use in addition to the NVME drive
2023-02-13 20:29:25 <mps> ikke: if you have another usb power supply with 2A at least try to power enclosure with it
2023-02-13 20:29:44 <ikke> mps: yes, I have a 2A adapter right now
2023-02-13 20:29:47 <minimal> so the draw on USB port will be 1.14 + (USB/NVME adaptor draw) A
2023-02-13 20:30:16 <minimal> ikke: have you a powered USB3 hub? you could try using that
2023-02-13 21:06:01 <ikke> minimal: no I have not
2023-02-14 10:08:43 <telmich> hello everyone
2023-02-14 10:49:36 <clandmeter> telmich: hi
2023-02-14 11:20:13 <ikke> telmich: o/
2023-02-14 11:22:32 <clandmeter> telmich: if you have an update regarding the network., please share here so infra team is up2date.
2023-02-14 11:22:34 <clandmeter> thanks!
2023-02-14 11:38:54 <telmich> clandmeter: I don't really have an update, I think it's maybe good that everyone is on the same page
2023-02-14 11:39:26 <telmich> From what I know is that the two arm servers are cnonected to vigir23 and from the dmesg that you sent it seems like they get a link up/down from time to time, is that the same status for everyone?
2023-02-14 11:40:38 <clandmeter> currently me and ikke are  the only ones working on the machines
2023-02-14 11:40:51 <clandmeter> ikke is up2date on this issue
2023-02-14 11:41:17 <ikke> telmich: only one of the ARM boxes appears to have the issue with the interface
2023-02-14 11:41:40 <ikke> But the main things we are noticing: max ~80mbit download speed, regular disconnects\
2023-02-14 11:41:49 <clandmeter> would be nice to conclude where the network performance issue is coming from
2023-02-14 11:42:05 <clandmeter> ikke: the disconnects are on both?
2023-02-14 11:42:24 <clandmeter> it would make sense it disconnects when the interface is jumping
2023-02-14 11:43:09 <ikke> I was still connected to che-bld-1 (where the interface is flapping) this afternoon
2023-02-14 11:43:34 <clandmeter> right, so that could be the issue
2023-02-14 11:44:00 <ikke> Last time it happened was 5 days aago
2023-02-14 11:44:03 <ikke> aago
2023-02-14 11:44:06 <clandmeter> telmich: would it be an option to double check the cable? maybe try another cable, and if possible switch it to another port on the virgir?
2023-02-14 11:44:11 <ikke> [Thu Feb  9 17:03:51 2023] igb 0003:03:00.0 eth0: igb: eth0 NIC Link is Down
2023-02-14 11:44:14 <ikke> [Thu Feb  9 17:03:54 2023] igb 0003:03:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
2023-02-14 11:46:24 <ikke> telmich: one thing I noticed is that the load of vigir23 is relatively high when we download things (avg load >5)
2023-02-14 11:48:31 <clandmeter> ikke: what is the model of that device?
2023-02-14 11:48:46 <clandmeter> i cant access its webif due to missing ipvt6
2023-02-14 11:48:54 <ikke> https://ungleich.ch/u/products/vigir/
2023-02-14 11:50:06 <clandmeter> https://openwrt.org/toh/zbtlink/zbt_wg3526   i guess this one
2023-02-14 12:36:26 <telmich> It is that one
2023-02-14 12:36:33 <telmich> I think both of you also have ssh access to the vigir?
2023-02-14 12:36:42 <telmich> If not, I can always add your key
2023-02-14 13:05:12 <telmich> clandmeter: In regards to cabling, I can see that they are exchanged. In regards to ports: the vigir has 4+1 (uplink) ports, so all of them are used at the moment
2023-02-14 14:06:45 <ikke> telmich: yes, we have 
2023-02-14 21:34:39 <ikke> rebooting x86_64 ci (upgraded to 3.17)
2023-02-14 21:35:49 <mps> ikke: still no ssh connection between lxc on 172.16.26.17 and 172.16.26.16
2023-02-14 21:36:10 <ikke> I have to look later this week
2023-02-14 21:36:40 <mps> hmm, have to upgrade linux-edge to 6.1.12
2023-02-14 21:38:06 <ikke> Do I need to copy something for you?
2023-02-14 21:38:15 <mps> maybe I can do rsync over few hops  more
2023-02-14 21:39:31 <mps> ok, I can use arm64 lxc as intermediate hop
2023-02-14 21:41:10 <ikke> that's a large hop :P
2023-02-14 21:42:23 <mps> yes it is, over ocean and back ;-)
2023-02-14 21:42:47 <ikke> an ocean and a continent
2023-02-14 21:43:00 <mps> yes, this too
2023-02-14 21:44:18 <mps> anyway this is tomorrow task
2023-02-15 05:38:43 <ikke> wheej
2023-02-15 05:39:55 <psykose> wheej
2023-02-15 05:40:28 <ikke> gitlab security upgrade
2023-02-15 05:40:44 <ikke> in case you hadn't noticed :P
2023-02-15 05:43:05 <psykose> oh yeah
2023-02-15 05:43:08 <psykose> definitely have :p
2023-02-15 05:43:14 <psykose> every single page, nondismissable
2023-02-15 05:43:35 <psykose> shall i just press "upgrade" ;P
2023-02-15 05:45:31 <ikke> It just leads you to a page with instructions :D
2023-02-15 05:47:17 <ikke> https://about.gitlab.com/releases/2023/02/14/critical-security-release-gitlab-15-8-2-released/
2023-02-15 05:47:39 <ikke> ok, related to the git CVEs
2023-02-15 05:49:10 <ikke> heh, this version also feature toggles the security warning
2023-02-15 05:49:15 <ikke> jhttps://gitlab.com/gitlab-org/gitlab/-/merge_requests/111585
2023-02-15 05:51:25 <ikke> fyi, I pushed the upgraded version, so it's being built now
2023-02-15 05:54:47 <psykose> yep
2023-02-15 05:54:48 <psykose> thanks
2023-02-15 06:43:55 <ikke> hmm, build timed out..
2023-02-15 06:43:59 <ikke> normally takes 20 minutes
2023-02-15 12:10:20 <mps> ikke: Esmil rebased VF v1 to 6.2-rc7 https://github.com/starfive-tech/linux/tree/visionfive
2023-02-15 12:10:53 <mps> I've built it and tested, works as previous version
2023-02-15 12:12:24 <mps> I'm thinking about changing pkgver to reflect 'rc' in pkgname but not sure how to do
2023-02-15 12:13:32 <mps> actually I have linux-rc aport in local branch
2023-02-15 12:14:11 <mps> maybe name it linux-starfivev1-rc, VF v2 is already around
2023-02-15 12:23:35 <ikke> mps: append _rc6 to the package for example
2023-02-15 12:26:04 <mps> to pkgname?
2023-02-15 12:27:08 <ikke> no, pkgver
2023-02-15 12:27:17 <ikke> pkgname should not change typically
2023-02-15 12:27:28 <ikke> I meant pkgver before
2023-02-15 12:29:22 <mps> aha
2023-02-15 12:32:30 <mps> lets see how
2023-02-15 12:33:45 <ikke> pkgver="1.2.3_rc1"
2023-02-15 12:34:06 <psykose> no quotes :p
2023-02-15 12:34:10 <ikke> heh
2023-02-15 13:01:19 <mps> building (with so slow qemu-user)
2023-02-15 13:21:00 <mps> ikke: I tested usb enclosure with old rotational disk and VF v1 can't even see disk, only see interface
2023-02-15 13:21:20 <mps> on my other computer it works fine
2023-02-15 13:22:25 <mps> so, it is probably not enough current on usb interface
2023-02-15 13:27:15 <ikke> I have one with an external power supply and that's working 
2023-02-15 13:27:22 <ikke> (spinning disk) 
2023-02-15 19:40:57 <psykose> ikke: did that rebuild ever go through for gitlab
2023-02-15 19:41:07 <ikke> yes, 2nd time it worked
2023-02-15 19:41:36 <ikke> I totally did not adblock the upgrade header in gitlab
2023-02-15 19:44:45 <psykose> hahaha
2023-02-15 19:44:49 <psykose> i still havent
2023-02-15 19:45:47 <ikke> I suppose I can do the upgrade soon
2023-02-15 20:35:34 <ikke> gitlab has been upgraded
2023-02-15 20:38:39 <psykose> hurra
2023-02-16 14:28:55 <ikke> another spam run in snippets appaarently
2023-02-16 21:02:49 <ikke> telmich: clandmeter fyi, I haven't seen anymore link flaps on that server
2023-02-16 21:02:55 <ikke> last one was from the 9th
2023-02-16 21:07:09 <ikke> bandwidth is still max ~80mbit
2023-02-16 21:07:32 <ikke> https://i.imgur.com/2VvsCzE.png
2023-02-16 21:07:46 <ikke> https://i.imgur.com/Azweh9K.png
2023-02-16 21:42:33 <ikke> psykose: added new arm* (aarch64 as well) gitlab ci runners. Let me know if you notice any issues
2023-02-16 21:42:40 <ikke> The names have che-ci-1 in them
2023-02-17 02:47:37 <psykose> ikke: i'll keep an eye out, thanks
2023-02-17 08:13:57 <ikke> clandmeter: guess the vms crashed? Can you take a look? 
2023-02-17 08:14:09 <clandmeter> vms?
2023-02-17 08:14:28 <ikke> Ci vms on usa9
2023-02-17 08:15:08 <clandmeter> oom
2023-02-17 08:16:00 <clandmeter> think the vm's are burning the cpus
2023-02-17 08:16:11 <clandmeter> armhf crashed
2023-02-17 08:16:25 <ikke> Oom is what usually happens 
2023-02-17 08:16:36 <ikke> And armv7? 
2023-02-17 08:16:41 <clandmeter> system is vveerryy slow
2023-02-17 08:16:54 <clandmeter> sorry v7 crasahed
2023-02-17 08:17:37 <clandmeter> memory usage on the host seems okisch
2023-02-17 08:18:28 <clandmeter> 08:18:16 up 92 days, 11:57,  load average: 252.86, 293.04, 277.65  :)
2023-02-17 08:21:49 <clandmeter> i restarted the vm
2023-02-17 08:36:14 <ikke> Wheej
2023-02-17 12:53:06 <clandmeter> https://www.phoronix.com/news/Linux-KVM-Lockless-Accessed-Bit  ikke should solve all our problems with vms ;)
2023-02-17 12:54:22 <psykose> time to run linux-ikke on all the prod servers
2023-02-17 12:54:24 <psykose> rebased daily
2023-02-17 12:54:35 <psykose> :D
2023-02-17 12:59:16 <mps> hm, looks interesting to add it
2023-02-17 13:00:22 <psykose> probably in 6.4
2023-02-17 13:01:32 <mps> heh, who knows but 6.3 merge window will be opened next monday, so
2023-02-17 13:02:05 <clandmeter> I like Linux-ikke let’s do it
2023-02-17 13:49:34 <ikke> :-D
2023-02-17 16:24:44 <ikke> dl-master had issues apperently (the host), but has been fixed by the provider
2023-02-18 01:26:06 <psykose> there's the che-ci
2023-02-18 14:26:22 <ikke> kunkku: The 2 new sites we're deploying are behind an IPv4 NAT, and I assume that is causing issues, as we noticed only one site can reach the dmvpn at the same time. For IPv6, they have a public routable address. Is it possible to force it to use IPv6?
2023-02-18 14:26:32 <ikke> The dmvpn hosts do have ipv6 addresses
2023-02-18 14:28:03 <ikke> s/hosts/hubs
2023-02-18 14:28:03 <alpine-meetbot> ikke meant to say: The dmvpn hubs do have ipv6 addresses
2023-02-19 15:13:12 <kunkku> ikke: IPsec NAT traversal should allow for multiple spokes behind a single IPv4 address
2023-02-19 15:13:58 <kunkku> is any of those new sites currently connected?
2023-02-19 15:15:33 <ikke> kunkku: strange, For some reason only one spoke has connection at the same time. When I run setup-dmvpn <crt> on the spoke that is not working, it starts working and the other looses connection
2023-02-19 15:16:08 <ikke> checked the cert at /etc/swanctl/x509/dmvpn.cert and they are what I expect
2023-02-19 15:16:55 <ikke> I do see this in the logs: "12[IKE] unable to resolve %any, initiate aborted"
2023-02-19 15:17:00 <ikke> (/var/log/messages)
2023-02-19 15:17:19 <ikke> 04[CFG] vici initiate CHILD_SA 'dmvpn', me 192.168.1.220, other %any, limits 0
2023-02-19 15:19:07 <kunkku> what's the public IP?
2023-02-19 15:19:22 <kunkku> or site name?
2023-02-19 15:20:49 <ikke> CHE-BLD-1 and CHE-CI-1
2023-02-19 15:20:53 <ikke> CHE-BLD-1 is not working atm
2023-02-19 15:21:27 <ikke> public ip: 194.230.148.145
2023-02-19 15:41:01 <kunkku> hm, might be that it is actually nhrpd that gets confused here
2023-02-19 15:41:32 <kunkku> even though ipsec is working
2023-02-19 15:55:45 <kunkku> I wonder if the NHRP protocol even supports having multiple mGRE tunnel endpoints behind a single address
2023-02-19 15:58:33 <ikke> Would it support IPv6?
2023-02-19 15:58:43 <ikke> They do have publically routable IPVv6 addresses
2023-02-19 16:00:30 <kunkku> I wonder if the kernel supports mGRE over IPv6
2023-02-19 16:01:01 <kunkku> this was not the case at least a few years ago
2023-02-19 16:02:15 <ikke> https://serverfault.com/questions/1053383/how-can-i-set-up-a-point-to-point-ipv6-gre-tunnel-encrypted-with-ipsec-between-m
2023-02-19 16:02:45 <ikke> Seems to imply it's possible
2023-02-19 16:06:18 <kunkku> that's point-to-point, not multipoint GRE
2023-02-19 16:06:22 <ikke> ah ok
2023-02-19 16:17:03 <kunkku> the issue was related to injecting the IPv6 address to the neighbor cache
2023-02-19 16:18:07 <kunkku> the netlink struct being too short for such an address
2023-02-19 16:22:57 <kunkku> I wonder if we could construct the network a bit differently
2023-02-19 16:23:23 <kunkku> those 2 "sites" are probably located in the same datacenter
2023-02-19 16:24:32 <kunkku> could they have a secure virtual network between them?
2023-02-19 16:29:54 <kunkku> according to fable, multiple spokes behind a single IPv4 address is not currently supported due to a kernel limitation
2023-02-19 16:30:25 <kunkku> it is also related to the neighbor cache
2023-02-19 16:52:25 <ikke> kunkku: they're even in the same lan
2023-02-19 16:52:40 <ikke> They have a private network where they are connected to
2023-02-19 16:54:11 <ikke> but they are not directly connected to eachother
2023-02-19 16:58:03 <kunkku> and the LAN cannot be trusted?
2023-02-19 16:58:13 <ikke> It can
2023-02-19 16:58:16 <ikke> It's ours
2023-02-19 17:00:00 <kunkku> I was thinking a setup where the container bridge is directly attached to the LAN
2023-02-19 17:00:30 <kunkku> and only one of the hosts would be a dmvpn spoke
2023-02-19 17:00:41 <ikke> yeah, but do we want to assign the public ipv6 address to the bridge?
2023-02-19 17:02:13 <ikke> Basically those machines are ipv6 first, and ipv4 for backwards compattibility
2023-02-19 17:02:26 <kunkku> ok
2023-02-19 17:02:39 <ikke> Maybe a gre tunnel between them to route the bridge traffic?
2023-02-19 17:02:51 <ikke> or something like that
2023-02-19 17:03:06 <kunkku> yes, we could set up static routing
2023-02-19 17:03:15 <ikke> The ci machine will not send a lot of traffic over the dmvpn anyway
2023-02-19 17:03:50 <kunkku> shortcut routes would not work anyway if both spokes are behind NAT
2023-02-19 17:04:48 <kunkku> so the traffic between ci<->bld would be routed via hubs
2023-02-19 17:05:12 <kunkku> ... if they both were dmvpn spokes
2023-02-19 17:06:21 <ikke> I don't expect much traffic between them
2023-02-19 17:07:25 <kunkku> GRE tunnel over LAN sounds a bit strange, though...
2023-02-19 17:07:43 <kunkku> I think VLAN tagging is the usual solution...
2023-02-19 17:08:02 <ikke> that could also be an option
2023-02-19 17:08:11 <ikke> Never done that just in a lan
2023-02-19 17:09:24 <ikke> So we add a subinterface to eth0 that is attached to the bridge or something like that?
2023-02-19 17:10:13 <kunkku> yes
2023-02-19 17:11:34 <kunkku> the other solution is start fixing the kernel :)
2023-02-19 17:11:37 <ikke> heh
2023-02-19 17:11:46 <ikke> Sadly that's a little out of my league
2023-02-19 17:12:46 <kunkku> that would take more time anyway
2023-02-19 17:13:27 <ikke> yup
2023-02-19 17:13:36 <kunkku> do we expect many similar sites with shared NAT?
2023-02-19 17:13:49 <ikke> I don't foresee anything
2023-02-19 17:14:08 <kunkku> so we can live with the above solution?
2023-02-19 17:14:12 <ikke> These are servers that are leased to us and we asked someone from the community with a hosting company to host them
2023-02-19 17:14:26 <ikke> Yeah, should be fine for now
2023-02-19 17:17:33 <kunkku> btw have you had time to investigate the weird IPv6 addresses on the GRE interface?
2023-02-19 17:17:59 <ikke> No, not yet. I saw it on these servers as well
2023-02-19 17:18:34 <ikke> ::7f00:1/96
2023-02-19 17:18:41 <ikke> ::c0a8:1dc/96
2023-02-19 17:19:12 <kunkku> and they match the IPv4 addrs on other interfaces
2023-02-19 17:20:07 <ikke> es
2023-02-19 17:20:13 <ikke> 192.168.1.220
2023-02-19 17:20:44 <kunkku> I tried commenting out stuff in /etc/network/interfaces
2023-02-19 17:20:54 <kunkku> but those addresses appeared anyway
2023-02-19 17:21:14 <kunkku> so it might be some kernel thing
2023-02-19 17:29:50 <kunkku> /proc/sys/net/ipv6/conf/gre1/addr_gen_mode ?
2023-02-19 17:29:52 <ikke> kunkku: would I still have a bridge interface on both with an ip in teh same lan?
2023-02-19 17:29:59 <ikke> same subnet I mean
2023-02-19 17:30:14 <ikke> returns 0
2023-02-19 17:30:56 <kunkku> mode 1 is "do not generate"
2023-02-19 17:31:53 <kunkku> are you using the VLAN method?
2023-02-19 17:31:57 <ikke> kunkku: yes
2023-02-19 17:32:22 <kunkku> then yes
2023-02-19 17:44:24 <ikke> https://tpaste.us/6M5W 
2023-02-19 17:44:35 <ikke> set it up like this, but cannot seem to ping between them yet
2023-02-19 17:45:28 <ikke> https://tpaste.us/Vqav 
2023-02-19 17:45:50 <ikke> So created a vlan interface eth0.2 with id 2
2023-02-19 17:46:31 <ikke> Then I set the master of those to the bridge interface
2023-02-19 17:48:11 <ikke> ip link add link eth0 name eth0.2 type vlan id 2
2023-02-19 17:49:59 <kunkku> the other bridge has only the broadcast address
2023-02-19 17:50:30 <ikke> Oh, my bad, I did add that myself
2023-02-19 17:50:44 <ikke> forgot that was the broadcast address
2023-02-19 17:51:45 <ikke> yes, now it works :)
2023-02-19 17:51:51 <kunkku> good
2023-02-19 17:52:23 <ikke> Was already looking at the switch if it was filtering vlan packets
2023-02-19 17:53:31 <kunkku> and the subnet should be within the site network range specified in the dmvpn cert
2023-02-19 17:53:36 <ikke> yes
2023-02-19 17:53:50 <ikke> 172.16.27.0/24, you mean
2023-02-19 17:54:31 <kunkku> yes
2023-02-19 19:48:21 <ikke> kunkku: everything is working btw, thanks for the suggestion
2023-02-19 22:50:32 <clandmeter> Another option is to switch to a apu2
2023-02-19 22:50:40 <clandmeter> with alpine on it
2023-02-19 22:51:08 <clandmeter>  and we run dmvpn on the device
2023-02-19 22:52:05 <clandmeter>  it would probably also improve wireguard performance 
2023-02-20 11:59:17 <ikke> clandmeter: don't know if you read it, but we've changed the dmvpn setup on che-*-1 slightly
2023-02-20 11:59:34 <clandmeter> i read a bit yes
2023-02-20 11:59:35 <ikke> ok
2023-02-20 11:59:49 <clandmeter> one server will run the dmvpn
2023-02-20 11:59:52 <ikke> correct
2023-02-20 12:00:47 <clandmeter> dmvpn will run over ipv4?
2023-02-20 12:01:44 <psykose> ikke: could you kick s390x
2023-02-20 12:02:24 <ikke> clandmeter: yes
2023-02-20 12:02:46 <clandmeter> so we will have the latency that it brings 
2023-02-20 12:05:59 <ikke> Yes
2023-02-20 12:06:12 <ikke> Not a lot to do about it 
2023-02-20 12:42:13 <ikke> psykose: done
2023-02-20 12:44:04 <psykose> thanks
2023-02-20 15:19:26 <mps> clandmeter: ikke: I think these new drivers in 6.2 kernel for ampere altra which monitors error would be useful
2023-02-20 16:04:51 <ikke> lesigh
2023-02-20 18:33:47 <ikke> mps: which drivers do you mean?
2023-02-20 18:36:44 <mps> CONFIG_SMPRO_ERRMON and CONFIG_SMPRO_MISC
2023-02-20 18:37:39 <mps> CONFIG_SENSORS_SMPRO
2023-02-21 05:44:37 <ikke> I hope the new CI setup will have less crashes due to OOM
2023-02-21 05:45:07 <psykose> bigger issue is probably that it doesn't restart
2023-02-21 05:45:09 <psykose> hehe
2023-02-21 05:45:24 <ikke> yes
2023-02-21 05:45:29 <psykose> as for the specific issue at hand however
2023-02-21 05:45:35 <psykose> let me throw clang at qt5-qtwebengine
2023-02-21 05:45:40 <psykose> then it will stop ooming everything all the time
2023-02-21 05:45:59 <ikke> I use qemu-openrc to start the VMs
2023-02-21 05:46:15 <ikke> Need to figure out how to get it to use supervisord
2023-02-21 08:24:51 <clandmeter> ikke: if they crash, cant you just use openrc supervision?
2023-02-21 08:38:52 <ikke> Yes, but the init script needs to be adjusted to use it 
2023-02-21 10:17:16 <clandmeter> ikke: the new ci's are already enabled?
2023-02-21 10:17:21 <clandmeter> for aarch64
2023-02-21 10:17:23 <ikke> Yes
2023-02-21 10:17:32 <ikke> Side by side
2023-02-21 10:17:34 <clandmeter> i want to see if i can add it
2023-02-21 10:17:41 <clandmeter> so ill shut it down and try
2023-02-21 10:17:52 <ikke> Add it where? 
2023-02-21 10:18:00 <clandmeter> to openrc
2023-02-21 10:18:23 <ikke> Ah
2023-02-21 10:18:25 <ikke> Sure
2023-02-21 10:18:40 <clandmeter> is there a magic cmd to shit it down
2023-02-21 10:18:43 <clandmeter> or just use rc?
2023-02-21 10:18:47 <clandmeter> shut :)
2023-02-21 10:18:54 <ikke> Openrc tries a clean shutdown 
2023-02-21 10:54:51 <clandmeter> ikke: did i understand correctly dmvpn is now fixed on che?
2023-02-21 10:58:06 <clandmeter> and how do i verify the host is online?
2023-02-21 10:58:11 <clandmeter> the ip's dont respond 
2023-02-21 11:02:37 <clandmeter> looks like dnsmasq is not running?
2023-02-21 11:12:46 <clandmeter> hmm
2023-02-21 11:13:03 <clandmeter> ikke: is network actually ok for qemu? 
2023-02-21 11:13:11 <clandmeter> looks like there are some changes
2023-02-21 11:16:57 <ikke> dnsmasq runs on bld
2023-02-21 11:17:46 <ikke> I've set the VMs to use static IPs though
2023-02-21 11:17:54 <ikke> Mostly because I wanted the default route to remain on the same host
2023-02-21 11:19:50 <ikke> clandmeter: 172.16.27.65-67
2023-02-21 11:26:28 <clandmeter> ikke: they dont run ssh?
2023-02-21 11:26:49 <ikke> clandmeter: I probably forgot to set that up
2023-02-21 11:26:56 <clandmeter> how do i connect?
2023-02-21 11:27:03 <clandmeter> just to verify its working
2023-02-21 11:27:12 <ikke> check the vms folder
2023-02-21 11:27:51 <ikke> vms/gitlab-runner-aarch64 connects via serial to that VM
2023-02-21 11:27:55 <clandmeter> i see
2023-02-21 11:28:44 <ikke> (via screen, use ctrl-a + k to exit)
2023-02-21 11:30:21 <clandmeter> ok i guess its working
2023-02-21 11:30:37 <clandmeter> but i think its easier to have them do ssh
2023-02-21 11:30:42 <ikke> yes, sure
2023-02-21 11:31:01 <ikke> This is just in case ssh is not working :)
2023-02-21 11:31:17 <clandmeter> weird ssh is running
2023-02-21 11:31:26 <ikke> And I have been mostly fixing network issues in the first place
2023-02-21 11:32:04 <ikke> hmm, connection refused
2023-02-21 11:34:53 <clandmeter> ping is ok
2023-02-21 11:35:12 <ikke> yeah
2023-02-21 11:35:29 <ikke> did you do something?
2023-02-21 11:35:32 <clandmeter> nope
2023-02-21 11:35:37 <ikke> lost access to the host
2023-02-21 11:35:39 <clandmeter> notwork
2023-02-21 11:35:41 <clandmeter> :)
2023-02-21 11:35:57 <clandmeter> killed im my vim session
2023-02-21 11:36:20 <ikke> ok, back
2023-02-21 11:37:09 <ikke> ok, fixed
2023-02-21 11:37:13 <ikke> awall enable adp-ssh-client
2023-02-21 11:37:13 <clandmeter> firewall?
2023-02-21 11:37:15 <ikke> yes
2023-02-21 11:37:32 <ikke> I setup adp for NAT
2023-02-21 11:37:38 <clandmeter> let me stop it ones more
2023-02-21 11:38:25 <clandmeter> not sure how stopping works
2023-02-21 11:38:34 <ikke> uses acpi
2023-02-21 11:38:36 <clandmeter> it runs ok
2023-02-21 11:38:40 <clandmeter> i mean
2023-02-21 11:38:51 <clandmeter> how does openrc know its stopped when its supervised
2023-02-21 11:39:05 <clandmeter> now it has an issue
2023-02-21 11:39:06 <ikke> it should run in the foreground
2023-02-21 11:39:17 <ikke> as a direct child of supervised
2023-02-21 11:39:22 <clandmeter> it kills it after 40 secs
2023-02-21 11:39:31 <clandmeter> i know how supervised works :)
2023-02-21 11:39:33 <ikke> ok
2023-02-21 11:39:45 <ikke> Then I misunderstood what you mean
2023-02-21 11:39:51 <clandmeter> np
2023-02-21 11:40:05 <clandmeter> the rc script does some magic i think to find out if its stopped
2023-02-21 11:40:19 <clandmeter> cause now it does not work anymore
2023-02-21 11:40:27 <ikke> there is an is_running function
2023-02-21 11:40:33 <ikke> depends on the pidfile
2023-02-21 11:40:58 <clandmeter> yes
2023-02-21 11:41:07 <clandmeter> but thats different with supervised
2023-02-21 11:41:14 <clandmeter> that is the pid of openrc supervised
2023-02-21 11:41:43 <clandmeter> so now it kills openrc supervised-daemon
2023-02-21 11:41:53 <ikke> right
2023-02-21 11:42:20 <clandmeter> but it does not shutdown qemu
2023-02-21 11:42:22 <clandmeter> not sure why not
2023-02-21 11:43:25 <clandmeter> so the qemush thingy is probably not working
2023-02-21 11:44:12 <ikke> That's the same thing that the attach-qemu script uses
2023-02-21 11:53:02 <clandmeter> interesting
2023-02-21 11:53:05 <clandmeter> the socket is gone
2023-02-21 11:53:07 <clandmeter> but the vm is up
2023-02-21 11:53:43 <ikke> hmm
2023-02-21 11:54:17 <clandmeter> and ssh is not reachable from outside host
2023-02-21 11:54:46 <clandmeter> and i cannot ssh now without my key
2023-02-21 11:55:13 <clandmeter> not sure my key was ever added
2023-02-21 11:59:06 <ikke> ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICHQbmOmCN7SOYpf+boeuShgnZy67XYqBa3tZChHHtIM
2023-02-21 11:59:08 <ikke> is in there
2023-02-21 11:59:10 <clandmeter> i think the rc file definitely need some extra work
2023-02-21 11:59:29 <ikke> You should be able to ssh into it
2023-02-21 11:59:42 <ikke> I have to go now
2023-02-21 12:01:56 <clandmeter> ah i see now
2023-02-21 12:02:09 <clandmeter> im jumping hosts due to ipv6
2023-02-21 12:02:16 <clandmeter> let me add a key to root
2023-02-21 12:02:26 <clandmeter> and ill add some entries to ssh config
2023-02-21 12:10:19 <clandmeter> ikke: out of simplicity i just added ssh config entries by arch. 
2023-02-21 12:10:26 <clandmeter> ssh aarch64 should do it
2023-02-21 12:20:43 <ikke> Ok
2023-02-21 12:22:11 <clandmeter> its weird the stop function does not stop the vm. its kind of obvious it does not kill the vm due to entries of start stop daemon in the stop function
2023-02-21 12:24:02 <clandmeter> ah i think its because of the pid
2023-02-21 12:31:53 <clandmeter> i think stop_post should actually check if the service is still running.
2023-02-21 14:29:22 <clandmeter> ikke: i created a qemu-supervised initd
2023-02-21 14:29:27 <clandmeter> its enabled for aarch64
2023-02-21 14:29:54 <clandmeter> it still needs some love
2023-02-21 14:30:18 <clandmeter> supervision is still set to defaults
2023-02-21 16:19:11 <ikke> clandmeter: alright, thanks
2023-02-21 17:26:47 <clandmeter> Ikke seen email?
2023-02-21 17:27:22 <clandmeter>  The old ci can already be released right?
2023-02-21 17:31:36 <ikke> If the new CI is stable enough
2023-02-21 17:32:10 <ikke> usa10 can be decommissioned from my part
2023-02-21 17:32:31 <clandmeter> Ok
2023-02-21 17:32:44 <clandmeter> Is it stable?
2023-02-21 17:37:10 <clandmeter> Can we plan an evening?
2023-02-21 17:37:31 <ikke> If network is stable
2023-02-21 17:37:52 <ikke> how about builders?
2023-02-21 17:38:43 <ikke> clandmeter: I'll return usa10, alright?
2023-02-21 17:38:45 <ikke> We never used it
2023-02-21 18:01:38 <clandmeter> Ok
2023-02-21 18:02:08 <clandmeter> Can we migrate builder this week
2023-02-21 18:02:12 <ikke> I think so
2023-02-21 18:02:30 <clandmeter> Not sure she needs it back so fast 
2023-02-21 18:02:57 <ikke> "We have limited inventory and have few requests pending for the systems."
2023-02-21 18:03:06 <ikke> So we can at least return 1
2023-02-21 18:03:59 <ikke> I'll replu
2023-02-21 18:04:02 <ikke> reply*
2023-02-21 18:19:35 <clandmeter> Thanks
2023-02-22 16:33:17 <psykose> ikke: the armhf/aarch64 CI seems to break
2023-02-22 16:34:04 <psykose> e.g. https://gitlab.alpinelinux.org/0564/aports/-/jobs/978068 just sat there for 6 minutes and https://gitlab.alpinelinux.org/0564/aports/-/jobs/978070 fails in fetches
2023-02-22 16:36:11 <ikke> Hmm, both on usa9
2023-02-22 16:51:18 <psykose> ah
2023-02-22 16:51:22 <psykose> could be related to overload
2023-02-22 16:51:26 <psykose> cause everything is building kernels
2023-02-22 16:51:27 <psykose> haha
2023-02-22 16:51:39 <psykose> it's like 600 load so
2023-02-22 16:52:02 <ikke> Totally not busy: Load average: 565.62 586.08 571.81
2023-02-22 20:40:32 <ikke> gitlab 15.9 released
2023-02-22 21:08:13 <psykose> upgrade time
2023-02-22 21:08:15 <psykose> send it ikke
2023-02-22 21:08:19 <psykose> letsgoletsgoletsgo
2023-02-22 21:08:32 <ikke> living on the edge
2023-02-22 21:09:54 <psykose> did they update ruby finally
2023-02-22 21:09:57 <psykose> or was that .10 target
2023-02-22 21:11:27 <ikke> I have not seen any mention
2023-02-22 21:11:35 <ikke> they do provide an alternative ssh server
2023-02-22 21:11:38 <ikke> gitlab-sshd
2023-02-22 21:12:57 <psykose> my experience of weird "internal" ssh's is quite poor
2023-02-22 21:13:25 <ikke> They say they have been using it since 15.2 internally for all of their traffic
2023-02-22 21:14:01 <psykose> you mean gitlab.com itself? probably
2023-02-22 21:14:15 <psykose> it's possible for it to be good, i'm not against it
2023-02-22 21:15:02 <psykose> or well
2023-02-22 21:15:25 <ikke> If possible we could run it on a different port side by side to try 
2023-02-22 21:15:26 <psykose> the sshd is probably one of the most security sensitive parts of the whole thing :-)
2023-02-22 21:16:22 <psykose> the blog post is funny
2023-02-22 21:16:23 <psykose> https://about.gitlab.com/blog/2022/08/17/why-we-have-implemented-our-own-sshd-solution-on-gitlab-sass/
2023-02-22 21:16:36 <psykose> >To mitigate the (security) risks, we performed multiple rounds of security reviews
2023-02-22 21:16:38 <psykose> damn i hope so
2023-02-22 21:17:07 <psykose> >Incompatibility with other in-progress features: Our first gitlab-sshd deployment consumed huge amounts of memory.
2023-02-22 21:17:11 <psykose> hehe
2023-02-22 21:18:09 <ikke> oof
2023-02-22 21:18:25 <psykose> ooh nice
2023-02-22 21:18:33 <psykose> they forked an entire go crypto library for it
2023-02-22 21:18:34 <psykose> https://gitlab.com/gitlab-org/golang-crypto/-/commits/master
2023-02-22 21:18:44 <psykose> then just didn't track any updates from https://github.com/golang/crypto/commits/master since doing that
2023-02-22 21:19:08 <ikke> ouch
2023-02-22 21:19:52 <psykose> nothing really major in there of course, but eh
2023-02-22 21:20:10 <minimal> psykose: you should be in Marketing, you're selling this so well ;-)
2023-02-22 21:20:15 <ikke> :D
2023-02-22 21:22:41 <ikke> They cannot say we didn't utilize usa9
2023-02-22 21:24:58 <psykose> aye
2023-02-22 23:51:06 <Hello71> no, see, it's fine, they performed multiple rounds of security reviews
2023-02-23 19:55:34 <psykose> ikke: any idea why https://gitlab.alpinelinux.org/alpine/aports/-/jobs/978322 fails instantly on start
2023-02-23 19:55:40 <psykose> seems to be x86_64 only and any mr
2023-02-23 20:29:18 <psykose> seems after some commits it gets fixed
2023-02-23 20:29:19 <psykose> weird
2023-02-23 20:41:23 <ikke> psykose: disk was full
2023-02-23 20:41:27 <psykose> ah
2023-02-23 20:41:29 <psykose> makes sense
2023-02-23 20:41:42 <ikke> There is a weekly cron that should clean up everything, but apparently it got full before
2023-02-23 20:41:54 <psykose> do we still use the .drone.yml
2023-02-23 20:41:59 <ikke> o
2023-02-23 20:42:01 <ikke> no
2023-02-23 20:42:05 <psykose> anything against rm
2023-02-23 20:42:15 <ikke> not from me
2023-02-23 20:42:24 <ikke> maybe confirm with clandmeter 
2023-02-23 20:42:30 <psykose> ack
2023-02-23 20:42:55 <clandmeter> del it
2023-02-23 20:45:11 <psykose> e57886d34bf416710ffede8672199ad5f8d2b01f
2023-02-24 13:48:25 <ikke> Hmm 
2023-02-24 13:48:47 <psykose> rip kdaudt-gitlab
2023-02-24 13:51:59 <ikke> :x
2023-02-24 14:26:22 <ikke> psykose: can you check the runner section to see if anything is going on?
2023-02-24 14:27:03 <psykose> shows offline
2023-02-24 14:27:31 <psykose> hm
2023-02-24 14:27:31 <psykose> no
2023-02-24 14:27:33 <psykose> only these https://img.ayaya.dev/EZi4OQ0UxhGq
2023-02-24 14:27:45 <psykose> https://img.ayaya.dev/EByWjaZVmiFJ
2023-02-24 14:27:50 <psykose> this is shared x86_64, nothing said
2023-02-24 18:22:47 <mps> latest aavmf (QEMU_EFI.fd) can't boot alpine aarch64 iso
2023-02-24 18:23:09 <mps> previous one works fine
2023-02-24 18:32:55 <ikke> From when was the latest?
2023-02-24 18:33:37 <ikke> I did install alpine aarch this week
2023-02-24 18:36:01 <mps> ikke: aavmf-0.0.202211-r0
2023-02-24 18:36:34 <mps> I mean, can't boot qemu VM in uefi mode
2023-02-24 18:36:54 <ikke> ah ok, I have aavmf-0.0.202208-r0
2023-02-24 18:37:06 <mps> ye, this one works fine
2023-02-24 18:37:29 <ikke> luckily
2023-02-24 18:37:45 <ikke> is there an issue?
2023-02-24 18:37:54 <ikke> / bug report
2023-02-24 18:38:25 <mps> didn't seen anything
2023-02-24 18:38:57 <mps> few minutes ago I noticed this behavior
2023-02-24 18:39:37 <ikke> ah, the server is on 3.17, so would not get that latest version
2023-02-24 19:53:17 <clandmeter> ikke: ping
2023-02-24 19:54:14 <ikke> pong
2023-02-24 19:54:39 <clandmeter> wanna do the transfer?
2023-02-24 19:56:07 <ikke> I already ran it once, it just finished
2023-02-24 19:56:56 <clandmeter> right, i mean the actual change
2023-02-24 19:57:02 <ikke> yes
2023-02-24 19:57:45 <ikke> Maybe start with the dev containers?
2023-02-24 19:59:09 <clandmeter> sure
2023-02-24 19:59:17 <clandmeter> did you do something with sshd config?
2023-02-24 19:59:23 <clandmeter> 5 days a go
2023-02-24 19:59:40 <clandmeter> there is a file in root called \ :)
2023-02-24 19:59:55 <ikke> oh, perhaps
2023-02-24 20:03:15 <psykose> 👉 \
2023-02-24 20:05:20 <clandmeter> i cannot start my dev container due to missing /usr/share/lxc/config/alpine.common.conf
2023-02-24 20:05:50 <clandmeter> legacy templates
2023-02-24 20:06:11 <clandmeter> yup
2023-02-24 20:06:58 <clandmeter> cgroup errors
2023-02-24 20:07:32 <clandmeter> ok cgroup rc started
2023-02-24 20:08:25 <ikke> lxc service should pull that in
2023-02-24 20:08:27 <clandmeter> ikke: .27 is the right network?
2023-02-24 20:08:43 <clandmeter> yes it should be lets not run that now
2023-02-24 20:09:12 <ikke> clandmeter: yes
2023-02-24 20:09:18 <clandmeter> ok
2023-02-24 20:09:26 <clandmeter> so the container received the right address
2023-02-24 20:10:26 <clandmeter> ok the container has no network
2023-02-24 20:10:39 <clandmeter> i mean it has network but not to the internet 
2023-02-24 20:11:08 <clandmeter> do we need to assign ipv6 addresses to the containers?
2023-02-24 20:11:26 <ikke> We need to still setup awall (adp) on that server
2023-02-24 20:11:36 <clandmeter> can you do that?
2023-02-24 20:11:47 <ikke> yes
2023-02-24 20:12:00 <ikke> Can I do that now
2023-02-24 20:12:05 <ikke> will interrup network possibly
2023-02-24 20:12:09 <clandmeter> go for it
2023-02-24 20:12:56 <clandmeter> but i think it will make sense to have ipv6 connectivity
2023-02-24 20:13:16 <clandmeter> as ipv6 seems to be the better uplink/route/network 
2023-02-24 20:22:53 <clandmeter> ikke: care to share progress? :)
2023-02-24 20:23:02 <ikke> sorry, got a phone call
2023-02-24 20:23:39 <clandmeter> ok, if its not convenient for you let me know. 
2023-02-24 20:36:32 <ikke> ok, available now
2023-02-24 20:40:41 <ikke> Ok, I think it's setup
2023-02-24 20:42:28 <clandmeter> network is broken now
2023-02-24 20:42:34 <clandmeter> no more dhcp lease
2023-02-24 20:44:40 <clandmeter> ikke: do we miss the local outbound traffic?
2023-02-24 20:45:07 <ikke> adp-local-outbound is enabled and adp_local_policy set to accept
2023-02-24 20:46:10 <clandmeter> still not getting a lease
2023-02-24 20:46:14 <ikke> hold on
2023-02-24 20:46:46 <clandmeter> https://tpaste.us/7Mv1
2023-02-24 20:46:55 <ikke> can you try agian?
2023-02-24 20:47:00 <ikke> dhcp zone was set incorrectly
2023-02-24 20:49:41 <clandmeter> still not working
2023-02-24 20:50:05 <ikke> hmm
2023-02-24 20:52:44 <clandmeter> are you sure that interface is correct?
2023-02-24 20:53:02 <clandmeter> usa9 uses the bond0
2023-02-24 20:53:08 <clandmeter> now its using the bridge
2023-02-24 20:53:42 <clandmeter> oh no im looking at the wrong line
2023-02-24 20:53:44 <ikke> ok
2023-02-24 20:53:57 <ikke> on other servers we have an explicit lxc zone
2023-02-24 20:54:00 <ikke> of dhcp
2023-02-24 20:54:03 <ikke> for*
2023-02-24 20:54:53 <clandmeter> yes lxc.json
2023-02-24 20:55:43 <ikke> can you try now?
2023-02-24 20:56:09 <clandmeter> thats better
2023-02-24 20:56:27 <clandmeter> yup network is also ok
2023-02-24 20:56:41 <clandmeter> what about ipv6?
2023-02-24 20:57:33 <ikke> what about it?
2023-02-24 20:57:42 <clandmeter> enable it on the containers?
2023-02-24 20:58:02 <ikke> what subnet?
2023-02-24 20:58:23 <clandmeter> i think we have a /48?
2023-02-24 20:58:28 <ikke> ok
2023-02-24 20:58:46 <clandmeter> so any /64 would do? i am not sure
2023-02-24 20:58:56 <ikke> yeah, should
2023-02-24 20:59:41 <clandmeter> can we use some auto configuration?
2023-02-24 21:00:02 <ikke> Need something to send router advertisements
2023-02-24 21:00:18 <ikke> Think linux can do that itself?
2023-02-24 21:00:42 <ikke> othwerwise there are some daemons
2023-02-24 21:00:45 <clandmeter> you are the ipv6 specialist :)
2023-02-24 21:04:16 <clandmeter> but if that takes more time lets do that another time
2023-02-24 21:04:33 <ikke> Yeah, can look into that later
2023-02-24 21:04:51 <clandmeter> do we want to turn of the builder?
2023-02-24 21:05:10 <ikke> sure
2023-02-24 21:05:46 <clandmeter> ok lets announce it 
2023-02-24 21:06:03 <ikke> everything is idle
2023-02-24 21:06:15 <ikke> mps: ^
2023-02-24 21:06:58 <clandmeter> its going down
2023-02-24 21:07:00 <clandmeter> lxc that is
2023-02-24 21:07:29 <clandmeter> i wonder what will break
2023-02-24 21:07:56 <ikke> plenty probably
2023-02-24 21:08:30 <clandmeter> there is not much that is ssh into the containers right?
2023-02-24 21:08:35 <mps> ikke: yes, in my case
2023-02-24 21:08:55 <clandmeter> still stopping...
2023-02-24 21:09:06 <clandmeter> gazillion containers
2023-02-24 21:09:12 <clandmeter> done
2023-02-24 21:09:25 <clandmeter> shall we run one last sync?
2023-02-24 21:10:15 <clandmeter> i take that as a yes
2023-02-24 21:19:54 <ikke> yes
2023-02-24 21:22:58 <clandmeter> synced
2023-02-24 21:24:42 <clandmeter> so we need to remove the following lines? https://tpaste.us/1Xne
2023-02-24 21:24:51 <clandmeter> i mean configs
2023-02-24 21:27:33 <ikke> yes
2023-02-24 21:45:04 <clandmeter> ikke: i disabled all dev containers for now
2023-02-24 21:45:13 <clandmeter> i think we could try to start lxc
2023-02-24 21:46:30 <ikke> clandmeter: go for it
2023-02-24 21:50:11 <clandmeter> ikke: looks good
2023-02-24 21:50:43 <clandmeter> they all have an ipv4 addr
2023-02-24 21:51:15 <ikke> good
2023-02-24 21:51:42 <clandmeter> but i still. notice ipv4 traffic is slowisch
2023-02-24 21:51:52 <clandmeter> so ipv6 connectivity should be nice to have
2023-02-24 21:54:56 <ikke> Do we use 2a0a:e5c1:517:1::/64 for that?
2023-02-24 21:56:04 <ikke> https://netbox.alpin.pw/ipam/prefixes/59/prefixes/?show_assigned=true&show_available=false
2023-02-24 22:00:24 <clandmeter> psykose: you got something to build?
2023-02-24 22:00:29 <clandmeter> or anybody else :)
2023-02-24 22:00:32 <psykose> in what sense
2023-02-24 22:00:35 <clandmeter> see if the builders act
2023-02-24 22:00:37 <psykose> ah
2023-02-24 22:00:52 <psykose> they don't wake
2023-02-24 22:00:57 <psykose> i just trigger via mqtt
2023-02-24 22:01:02 <psykose> unless it 'really' needs a commit
2023-02-24 22:01:06 <psykose> there is also a queued package
2023-02-24 22:01:42 <psykose> and it's not building even after trigger so
2023-02-24 22:02:13 <clandmeter> its building 
2023-02-24 22:02:58 <clandmeter> abuild is running
2023-02-24 22:02:59 <psykose> not visibly on build.a.o
2023-02-24 22:03:04 <psykose> but ok, the build works
2023-02-24 22:04:01 <clandmeter> its running rsync now
2023-02-24 22:04:09 <clandmeter> aarch64 edge
2023-02-24 22:04:46 <clandmeter> so its listening on mqtt but not publishing?
2023-02-24 22:07:46 <clandmeter> ikke: can you help debug this issue?
2023-02-24 22:09:30 <ikke> hmm
2023-02-24 22:11:33 <ikke> The containers should be equal
2023-02-24 22:12:21 <clandmeter> the network is different
2023-02-24 22:12:35 <clandmeter> maybe they want to connect over dmvpn?
2023-02-24 22:13:15 <clandmeter> but that should not matter if they can listen on the same address (if it uses the same)
2023-02-24 22:23:42 <clandmeter> ikke: l will leave it like this. maybe you can find whats wrong.
2023-02-25 15:31:36 <mps> what's happened with arm lxc? I can't ping them
2023-02-25 15:35:39 <ikke> Moved to a new host, with a new IP
2023-02-25 15:36:37 <mps> ah, what are my IPs
2023-02-25 15:37:27 <ikke> mps-edge-aarch64.che-bld-1.alpin.pw 172.16.27.110
2023-02-25 15:37:46 <ikke> mps-edge-armv7.che-bld-1.alpin.pw 172.16.27.111
2023-02-25 15:37:56 <mps> ikke: thanks
2023-02-25 15:38:08 <ikke> so instead of .23, it's .27
2023-02-25 15:39:32 <mps> yes, noticed
2023-02-25 15:42:22 <mps> still no ssh between rv64 and x86 lxcs
2023-02-25 15:45:23 <ikke> Yeah, I have it on a todo list to check
2023-02-25 15:45:30 <ikke> Need to dive into the iptables rules
2023-02-25 15:47:05 <mps> ok, I can use intermediate hop
2023-02-25 15:56:31 <ikke> can you ssh between your arm containers?
2023-02-25 16:06:33 <mps> ikke: yes
2023-02-25 16:06:57 <mps> and between rv64|x86 and arm
2023-02-25 16:07:48 <mps> I use rsync over ssh to keep things in sync on all of them
2023-02-25 16:14:28 <mps> fix for 6.2 linux-edge ext4 is just pushed, not serious bug but anyway
2023-02-25 16:19:52 <mps> do arm builders works
2023-02-25 16:20:46 <psykose> yeah
2023-02-25 16:21:07 <mps> hm, why they then doesn't build linux-edge
2023-02-25 16:21:10 <psykose> ikke: psykose-edge-aarch64.che-bld-1.alpin.pw doesn't seem to resolve
2023-02-25 16:21:15 <psykose> mps: they just don't publish to mqtt
2023-02-25 16:21:33 <mps> ah, ok
2023-02-25 16:22:27 <ikke> psykose: need to start the containers, will do in a bit 
2023-02-25 16:22:32 <psykose> thanks
2023-02-25 16:23:43 <psykose> (no rush)
2023-02-25 16:27:09 <ikke> psykose: should be available now
2023-02-25 16:30:04 <psykose> thanks :)
2023-02-25 19:08:58 <ikke> I cannot figure out why ssh on the same bridge between containers is not working
2023-02-25 19:09:03 <ikke> I can ping them
2023-02-25 19:10:46 <ikke> hmm
2023-02-25 19:10:56 <ikke> ok, found it, but cannot explain why
2023-02-25 19:11:16 <ikke> Even though the traffic is on the bridge, it requires a rule in the FORWARD table
2023-02-25 19:15:20 <ikke> And on another host, it works but we do not have a rule in forward 🤨
2023-02-25 19:21:12 <psykose> networking be like
2023-02-25 19:21:43 <ikke> ikr
2023-02-25 19:22:23 <mps> forwarding is needed if traffic is not destined for local process
2023-02-25 19:29:29 <ikke> It's the same subnet
2023-02-25 19:29:37 <ikke> on the same host
2023-02-25 19:29:40 <ikke> on the same interface
2023-02-25 19:39:48 <mps> host forwards these packets to another (virtual) interface and don't 'send' them to local process, so it is still forwarding
2023-02-25 19:41:00 <mps> nftables/iptables 'count' interfaces as main blocks/bricks
2023-02-25 19:42:49 <mps> if packet travel between any interfaces it pass through forward 'subsystem'
2023-02-25 19:43:45 <mps> only local packets (source or destination) doesn't pass through forward rules
2023-02-25 19:44:43 <mps> http://easillyy.com/wp-content/uploads/2020/07/hook.jpg
2023-02-25 19:44:51 <mps> simple image
2023-02-25 19:45:21 <mps> more detailed is here https://thermalcircle.de/doku.php?id=blog:linux:nftables_packet_flow_netfilter_hooks_detail
2023-02-25 21:20:14 <ikke> mps: psykose I'm going to reboot the host where the x86_64 dev containers are one
2023-02-25 21:25:25 <psykose> go for it
2023-02-25 21:27:39 <ikke> doen
2023-02-25 21:27:41 <ikke> done*
2023-02-25 21:29:32 <psykose> maybe it's time to restart all the builders
2023-02-25 21:29:38 <psykose> the uptime is severe by now :D
2023-02-25 21:29:50 <psykose> so many kernels
2023-02-25 22:21:05 <ikke> mps: I've put a temporary firewall rule into place that allows the traffic between containers
2023-02-25 22:21:14 <ikke> No idea why it's necessarya
2023-02-25 22:23:02 <mps> ikke: rv64 lxc is not started
2023-02-25 22:23:26 <mps> can't ping it
2023-02-25 22:23:36 <ikke> It is started, but does not have an ip
2023-02-25 22:25:19 <mps> intentionally?
2023-02-25 22:25:40 <ikke> no
2023-02-25 22:26:02 <ikke> It used to be fixed, but for some reason, rv64 containers can't get an ip from dchp
2023-02-25 22:28:45 <ikke> mps: ok, has an ip now
2023-02-25 22:30:29 <mps> ikke: now it works
2023-02-25 22:30:39 <mps> thank you
2023-02-25 23:09:41 <ikke> If we reboot the host, the rule is gone
2023-02-25 23:11:29 <ikke> IN=lxcbr0 OUT=lxcbr0 PHYSIN=vethXlPEp6 PHYSOUT=veth6LxHsJ MAC=fe:ff:12:68:e8:5f:fe:5b:cb:5a:a1:2c:08:00 SRC=172.16.26.8 DST=172.16.26.16 LEN=60 TOS=0x08 PREC=0x40 TTL=64 ID=64601 DF PROTO=TCP SPT=57936 DPT=22 WINDOW=42340 RES=0x00 SYN URGP=0
2023-02-25 23:11:45 <ikke> This is what requires the FORWARD rule and I don't understand it
2023-02-25 23:25:31 <psykose> maybe because physin/out is different
2023-02-26 04:05:49 <anonymous> wsp
2023-02-26 10:35:56 <telmich> Moin moin - anyone awake in here? We have to shift the arm servers from place6 to place5 due to power shifting (we always balance power between the two sites) - there will be a downtime, potentially today of about 1-2h
2023-02-26 11:51:33 <mps> telmich: it is already down
2023-02-26 11:53:18 <ikke> telmich: alright
2023-02-26 11:55:55 <clandmeter> ikke: what do you think of replacing dmvpn on the arm boxes with a wg tunnel to dmvpn.a.o?
2023-02-26 11:56:17 <ikke> clandmeter: means we cannot directly connect to the containers anymore over dmvpn?
2023-02-26 11:56:29 <ikke> Unless we add a static route
2023-02-26 11:57:54 <clandmeter> dmpvn hosts also has a subnet available?
2023-02-26 11:57:55 <ikke> clandmeter: and you want to have direct tunnels for each server?
2023-02-26 11:58:14 <ikke> clandmeter: you mean the hub?
2023-02-26 11:58:18 <clandmeter> yes
2023-02-26 11:58:21 <ikke> The hub itself has no dedicated subnet
2023-02-26 11:58:25 <ikke> only a gre endpoint
2023-02-26 11:58:29 <clandmeter> so waht do the clients get?
2023-02-26 11:58:36 <clandmeter> the wireguard clients
2023-02-26 11:58:52 <clandmeter> wg is on one of the hubs or not?
2023-02-26 11:58:56 <ikke> no, dedicated host
2023-02-26 11:58:58 <ikke> deu7
2023-02-26 11:59:00 <clandmeter> oh ok
2023-02-26 11:59:06 <clandmeter> confused with openvpn
2023-02-26 11:59:07 <ikke> which is also connected to dmvpn
2023-02-26 11:59:10 <ikke> yes
2023-02-26 11:59:12 <ikke> ovpn is on dmvpn1
2023-02-26 11:59:18 <clandmeter> ok so to wg.a.o
2023-02-26 11:59:29 <clandmeter> let all of them route via wg.a.o
2023-02-26 11:59:36 <clandmeter> wg.a.o is also in europe
2023-02-26 11:59:39 <ikke> right now each endpoint gets a single ip
2023-02-26 11:59:52 <clandmeter> and the peering via ipv4 is less optimal
2023-02-26 11:59:53 <ikke> on deu7
2023-02-26 12:00:12 <clandmeter> so we could connect over ipv6
2023-02-26 12:00:35 <clandmeter> now it forces all traffic over ipv4
2023-02-26 12:00:59 <ikke> dmvpn traffic you mean?
2023-02-26 12:01:15 <clandmeter> yes
2023-02-26 12:01:25 <clandmeter> but also all other if we dont assign ipv6 to the containers
2023-02-26 12:01:34 <ikke> right
2023-02-26 12:02:20 <clandmeter> i mean dmvpn will try to be smart to get the closest route to improve routing
2023-02-26 12:02:37 <clandmeter> but it seems ipv4 is way worse compared to ipv6
2023-02-26 12:03:07 <clandmeter> i think we did this before for some other builders
2023-02-26 12:03:32 <telmich> The arm boxes are back online, I can see the IPMI interfaces already got a lease again
2023-02-26 12:04:32 <ikke> can connect to one of the boxes, not yet the other
2023-02-26 12:04:39 <ikke> now to both
2023-02-26 12:05:39 <clandmeter> telmich: thanks!
2023-02-26 12:05:51 <clandmeter> telmich: is there anything we can do to improve the network?
2023-02-26 12:06:14 <telmich> clandmeter: you mean in regards to throughput?
2023-02-26 12:06:22 <clandmeter> and latency
2023-02-26 12:06:34 <clandmeter> looks like ipv4 is higher latency compared to ipv6
2023-02-26 12:06:43 <telmich> latency? In which direction is that one not good?
2023-02-26 12:07:10 <telmich> They should have about 10-15ms to Zurich
2023-02-26 12:07:14 <clandmeter> if you ping 1.1.1.1 there is a 20-30 ms difference between ipv4 and 6
2023-02-26 12:07:25 <clandmeter> when idle
2023-02-26 12:07:29 <telmich> Ohhh
2023-02-26 12:07:30 <telmich> That makes sense
2023-02-26 12:07:42 <telmich> Because the IPv4 translator is non-local at the moment
2023-02-26 12:08:22 <clandmeter> and i think we bump into the wg cpu bottleneck when doing bigger transfers
2023-02-26 12:08:28 <telmich> That can be addressed, we are currently improving/updating quite a few parts of the infra - we can certainly add a local nat64 translator
2023-02-26 12:08:29 <clandmeter> like ikke mentioned
2023-02-26 12:08:59 <clandmeter> telmich: could we run wg directly from the host?
2023-02-26 12:09:14 <clandmeter> i mean bypassing the router
2023-02-26 12:09:40 <telmich> Hmm, have to think about it
2023-02-26 12:10:17 <clandmeter> i mean its not a huge problem, but its nice to have more than 80mbits
2023-02-26 12:11:01 <telmich> Because the main purpose of the router is that the boxes are having their dedicated network. Let me keep that in my queue, if something smart/easy comes up during restructuring, I'll ping you
2023-02-26 12:11:13 <clandmeter> still weird that cpu can only handle 80mbs wg traffic
2023-02-26 12:11:29 <telmich> It indeed is
2023-02-26 12:12:00 <clandmeter> i think my apu2 can do full gigabit
2023-02-26 12:12:02 <telmich> I think I would have expected around 400 mbit/s - I think we actually measured some years ago
2023-02-26 12:12:17 <clandmeter> or atlease close
2023-02-26 12:12:45 <telmich> apu2 might be indeed an option
2023-02-26 12:13:13 <clandmeter> apu2 only has 3 nics
2023-02-26 12:13:17 <clandmeter> at least mine :)
2023-02-26 12:13:51 <clandmeter> if i should send over any stuff or devices to support, let me know.
2023-02-26 12:14:36 <ikke> clandmeter: fyi, I've created a simple oneshot service on che-bld-1 to add eth0.2 as member to lxcbr0. It appears it does not add it via ifupdown-ng in combination with dnsmasq.lxcbr0
2023-02-26 12:14:50 <clandmeter> i actually have a nice edgerouter with 8 ports. thats doing nothing...
2023-02-26 12:15:05 <telmich> If you have something that takes IPv6 on the "left" and does whatever is needed on the "right" (towards servers), feel free to - at the moment we are out of apu2 stock
2023-02-26 12:15:07 <clandmeter> which also runs openwrt
2023-02-26 12:15:09 <telmich> Uh
2023-02-26 12:15:13 <telmich> Edgerouter
2023-02-26 12:15:14 <telmich> Nice one
2023-02-26 12:15:26 <clandmeter> 19" 
2023-02-26 12:16:03 <clandmeter> i can do some tests to see the performance
2023-02-26 12:16:06 <telmich> Rack sizes don't matter so much here, as we don't use racks to spread the temperature better
2023-02-26 12:16:20 <clandmeter> oh :)
2023-02-26 12:16:26 <clandmeter> you just stash them?
2023-02-26 12:16:39 <clandmeter> giant pile of 19" servers
2023-02-26 12:16:43 <clandmeter> :D
2023-02-26 12:17:03 <clandmeter> i guess you are only doing air cooling atm?
2023-02-26 12:17:12 <ikke> clandmeter: btw, was looking at corerad for route advertisements for the containers
2023-02-26 12:17:14 <telmich> No stashing, spreading
2023-02-26 12:17:24 <telmich> We are on average using 3-4 m² per server
2023-02-26 12:17:42 <clandmeter> oh really? 
2023-02-26 12:17:44 <telmich> ikke: Not sure if you are aware, but bird also does radvd quite nicely
2023-02-26 12:17:50 <clandmeter> can you share pictures? 
2023-02-26 12:17:51 <ikke> telmich: I was not
2023-02-26 12:18:06 <telmich> We have replaced a few radvd's recently with bird, because we use bird everywhere
2023-02-26 12:18:08 <telmich> clandmeter: sure
2023-02-26 12:18:09 <ikke> telmich: but we are already running quagga on those
2023-02-26 12:18:29 <clandmeter> telmich: why scale horizontally? doesnt that cost more per m2?
2023-02-26 12:19:22 <clandmeter> ikke: i noticed the 0.2
2023-02-26 12:19:26 <clandmeter> didnt know what it was for
2023-02-26 12:19:36 <ikke> That's what we came up with with kunkku
2023-02-26 12:19:55 <ikke> a vlan interface that connects the 2 bridges
2023-02-26 12:20:16 <telmich> ikke: ahh, I never got warm with quagga and we actually have quite some performance issues with it in a few projects (we are bound to 7.5.1 of frr to be precise)
2023-02-26 12:20:28 <telmich> clandmeter: We are in a area where a lot of spinning and weaving companies used to be and we are re-using the old factory halls
2023-02-26 12:20:40 <ikke> telmich: our dmvpn solution uses quagga
2023-02-26 12:20:52 <telmich> The area here was famous for being most industrialised in the 19th century
2023-02-26 12:23:55 <ikke> telmich: I assume ipv6 traffic is also going over the wg tunnel? Because installing apk packages is also going very slow on these hosts
2023-02-26 12:25:43 <telmich> ikke: All traffic is going via wg on it
2023-02-26 12:25:49 <ikke> right
2023-02-26 12:26:01 <clandmeter> ikke: you mean pkgs on the host?
2023-02-26 12:26:05 <telmich> v4 is "emulated" only
2023-02-26 12:27:44 <ikke> clandmeter: yes
2023-02-26 12:27:55 <ikke> mtu is 1420
2023-02-26 12:28:00 <ikke> is that something we need to account for?
2023-02-26 12:28:09 <clandmeter> are you sure it was using ipv6?
2023-02-26 12:28:19 <ikke> clandmeter: at least curl is to dl-cdn
2023-02-26 12:29:08 <clandmeter> 80mbit should still be kind of okisch I would expect 
2023-02-26 12:29:22 <clandmeter> depending on pkgs size ofc
2023-02-26 12:29:47 <ikke> clandmeter: try an apk upgrade
2023-02-26 12:30:06 <ikke> Even with 80mbit, I feel it should be faster
2023-02-26 12:30:42 <ikke> if it's using ipv4 and packages get fragmented, it might explain why it it's slow
2023-02-26 12:30:48 <ikke> ipv6 does not do fragmentation though
2023-02-26 12:31:48 <telmich> Technically, v6 does, but at the sender side
2023-02-26 12:32:33 <ikke> right
2023-02-26 12:33:07 <clandmeter> when i wget an iso from cdn, the network also gets very slowish
2023-02-26 12:33:13 <ikke> but if the packets were to big, nothing would get through
2023-02-26 12:33:49 <clandmeter> and its not stable at all.
2023-02-26 12:33:53 <clandmeter> it fluctuates too much 
2023-02-26 12:34:56 <ikke> telmich: does dns need to be updated for vigir23?
2023-02-26 12:35:12 <clandmeter> telmich: if we could try a direct wg from one of the hosts, we could see if that improves things? maybe use the second nic in one of the servers to try something?
2023-02-26 12:37:59 <telmich> ikke: Should be fine, but I am checking it to be sure
2023-02-26 12:41:18 <telmich> opkg update on vigir23 works fine so far
2023-02-26 12:41:34 <ikke> telmich: I mean the dns records
2023-02-26 12:41:45 <ikke> But now it's working
2023-02-26 12:43:19 <telmich> Ahh, the dns record of the vigir itself, yes
2023-02-26 12:46:59 <telmich> I'll be doing some network reconfigurations in the next hours, there might be a few minutes of outages from time to time - we are switching over to new routers in place5 (this is where the arm boxes are now)
2023-02-26 12:49:46 <ikke> telmich: I keep track of it in our netbox instance :)
2023-02-26 12:50:43 <telmich> Nice, we are also using netbox intensively
2023-02-26 12:50:50 <ikke> I had imagined :)
2023-02-26 12:51:02 <telmich> ... thinking about federated netbox via activitypub now...
2023-02-26 12:51:21 <ikke> From our side it does not fit perfectly, because for us it's almost always on the 'customer' side
2023-02-26 12:51:31 <telmich> hmm
2023-02-26 12:51:42 <ikke> We are always the tennant
2023-02-26 12:51:43 <telmich> You don't have your own /48 or something-like-that space?
2023-02-26 12:51:51 <telmich> For within the vpn?
2023-02-26 12:52:20 <ikke> Not publically
2023-02-26 12:52:23 <ikke> no
2023-02-26 12:52:31 <ikke> I did reserve a ULA address
2023-02-26 12:53:16 <ikke> but that's not publically routable
2023-02-26 13:00:32 <ikke> But just for vpn it's fine
2023-02-26 14:11:30 <mps> I'm curious, what was reason to move arm machines from usa to europe
2023-02-26 14:11:50 <psykose> to lower your ping
2023-02-26 14:12:00 <psykose> (the old ones are EOL pretty sure)
2023-02-26 14:12:04 <mps> heh, is it
2023-02-26 14:12:34 <psykose> something something just 'move to new machine' by request
2023-02-26 14:12:36 <mps> now my ping is infinite 
2023-02-26 14:12:45 <psykose> ikke: is it the same specs? or a new specsheet :D
2023-02-26 14:13:23 <ikke> The new machines are single socket instead of dual socket
2023-02-26 14:13:32 <ikke> so half the cores
2023-02-26 14:13:45 <psykose> how many of them
2023-02-26 14:13:46 <ikke> 2
2023-02-26 14:13:58 <psykose> probably just an upgrade then given weird numa issues
2023-02-26 14:14:16 <ikke> ARM has limited hosting space in equinix
2023-02-26 14:14:29 <psykose> still wish we had some better job scheduling though
2023-02-26 14:14:43 <ikke> so now they are leasing us 2 machines that we are hosting with ungleich
2023-02-26 14:14:48 <psykose> aye
2023-02-26 14:16:06 <mps> maybe I could put somewhere my old M1 because I don't use it for anything
2023-02-26 14:32:42 <ikke> something something network
2023-02-26 14:32:45 <ikke> need to look at it later
2023-02-26 14:32:53 <ikke> or rather, notwork
2023-02-26 14:42:04 <mps> notwork, funny term :)
2023-02-26 20:54:10 <psykose> hurra
2023-02-26 20:54:16 <psykose> whadya hit
2023-02-26 20:54:20 <ikke> I think I messed up before
2023-02-26 20:54:38 <ikke> and apparently the build containers were not automatically started either
2023-02-26 20:54:55 <psykose> still can't reach mine :D
2023-02-26 20:55:11 <ikke> 172.16.27.20?
2023-02-26 20:55:29 <psykose> can't resolve psykose-edge-armv7.che-bld-1.alpin.pw at all
2023-02-26 20:55:31 <psykose> unless i typod it
2023-02-26 20:55:51 <psykose> or some weird dns cache
2023-02-26 20:56:10 <ikke> Probably saturated network
2023-02-26 20:56:22 <ikke> host psykose-edge-armv7.che-bld-1.alpin.pw 172.16.27.1
2023-02-26 20:56:26 <ikke> communications error to 172.16.27.1#53: timed out
2023-02-26 20:56:33 <psykose> gross indeed
2023-02-26 20:56:35 <ikke> heh
2023-02-26 20:57:01 <psykose> good work ikke
2023-02-26 20:57:17 <ikke> Was chekcing why build status is not updated
2023-02-26 20:57:29 <psykose> maybe some key not copied
2023-02-26 20:57:31 <ikke> I do see traffic to and from msg.a.o
2023-02-26 20:57:34 <psykose> since you authed it
2023-02-26 20:58:35 <ikke> The containers were rsynced, they should be identical
2023-02-26 20:59:02 <psykose> yeah, but it's something to check
2023-02-26 20:59:04 <ikke> Load average: 422.03 301.92 140.12
2023-02-26 20:59:14 <ikke> Not right now :p
2023-02-26 20:59:17 <psykose> :)
2023-02-26 21:01:37 <ikke> earlier today, I cancled apk upgrade (because it was going very slowly). I just rebooted it, and then it was hanging in "Loading initrd" :D
2023-02-26 21:01:55 <psykose> haha
2023-02-26 21:02:16 <ikke> There was also an lts kernel (it's running edge atm)
2023-02-26 21:02:22 <ikke> so booted that and could finish the upgrade
2023-02-26 21:02:37 <psykose> i would run edge too on that hardware
2023-02-26 21:04:55 <ikke> alpine-msg.gbr-app-1.alpin.pw has IPv6 address 8d62:a97::
2023-02-26 21:04:58 <ikke> lesigh
2023-02-26 21:05:33 <ikke> No idea where that is coming from
2023-02-26 21:06:42 <psykose> either dhcp or kernel magic autoconf or the ipv6 magic
2023-02-26 21:06:44 <psykose> yw :D
2023-02-26 21:07:08 <psykose> shouldn't it have one however
2023-02-26 21:07:59 <ikke> atm not via vpn
2023-02-26 21:08:29 <psykose> ah, right
2023-02-26 21:10:36 <ikke> It decodes to 141.98.10.151
2023-02-26 21:11:04 <ikke> some ISP in lithuania
2023-02-26 21:12:58 <psykose> ah that's why there's issues
2023-02-26 21:13:05 <psykose> :D
2023-02-26 21:15:14 <ikke> builders connect externally to msg.a.o
2023-02-26 21:19:03 <ikke> oh
2023-02-26 21:19:10 <ikke> I think I know what's going on
2023-02-26 21:20:14 <ikke> ~/.config/* was excluded from the sync
2023-02-26 21:20:18 <ikke> guess where the credentials are stored
2023-02-26 21:20:45 <ikke> clandmeter: ^
2023-02-26 21:20:54 <ikke> clandmeter: That means we cannot just nuke ~/.config either
2023-02-26 21:21:26 <clandmeter> ok :)
2023-02-26 21:22:08 <clandmeter> or put it in /etc somewhere?
2023-02-26 21:22:22 <ikke> It's something that mosquitto_sub reads
2023-02-26 21:22:32 <ikke> s/sub/pub
2023-02-26 21:22:32 <alpine-meetbot> ikke meant to say: It's something that mosquitto_pub reads
2023-02-26 21:27:30 <ikke> psykose: clandmeter: builders are reporting again
2023-02-26 21:27:34 <psykose> pog
2023-02-26 21:27:38 <psykose> double good work
2023-02-26 21:28:09 <ikke> Now I'm allowed to sleep :P
2023-02-26 21:28:14 <psykose> yeah
2023-02-26 21:28:15 <psykose> bedtime
2023-02-26 21:28:23 <psykose> also haha, they were building nothing this whole time it seems
2023-02-26 21:29:02 <ikke> I think they should have been building until at least this morning
2023-02-26 21:29:07 <ikke> subscribing should not require auth
2023-02-26 21:30:58 <psykose> ye
2023-02-26 21:31:11 <psykose> what's the load
2023-02-26 21:31:19 <psykose> they seem.. slow
2023-02-26 21:31:22 <psykose> but there's nothing really happening
2023-02-26 21:31:25 <psykose> except the 3 builders
2023-02-26 21:33:09 <ikke> 350
2023-02-26 21:33:19 <psykose> hmm
2023-02-26 21:33:20 <psykose> but ho
2023-02-26 21:33:21 <psykose> how*
2023-02-26 21:33:23 <psykose> what is it doing
2023-02-26 21:33:31 <ikke> 3.14 is building
2023-02-26 21:33:35 <psykose> ah
2023-02-26 21:33:42 <psykose> 3.14 didn't announce`
2023-02-26 21:33:44 <psykose> ok
2023-02-26 21:33:59 <ikke> maybe because it does not support the auth file
2023-02-26 21:34:07 <psykose> also should probably have armhf+armv7 on one machine and aarch64 on another
2023-02-26 21:34:13 <psykose> if you're doing the 2/1 split like before
2023-02-26 21:34:16 <psykose> priorities and all :D
2023-02-26 21:34:24 <ikke> we split builders versus ci
2023-02-26 21:34:42 <psykose> all on one, all on another?
2023-02-26 21:34:46 <ikke> yes
2023-02-26 21:35:08 <psykose> hmm
2023-02-26 21:35:55 <psykose> i mean it works, but in practice either one will be idle a lot of the time
2023-02-26 21:36:00 <psykose> the perils of no job scheduling
2023-02-26 21:36:03 <ikke> We'll see
2023-02-26 21:36:09 <ikke> we can shuffle things around
2023-02-26 21:36:11 <psykose> i.e. like right now
2023-02-26 21:36:34 <ikke> We first need to finish the migration and this was the easiest to do
2023-02-26 21:36:38 <psykose> yeah
2023-02-26 21:36:57 <psykose> good work :)
2023-02-26 21:37:13 <ikke> clandmeter did most of the work, at least for the builders :)
2023-02-26 21:37:36 <psykose> clandmeter: good work to you too :D
2023-02-26 21:37:58 <ikke> and ARM needs the old builder back
2023-02-26 21:38:00 <clandmeter> i had to finish it before my holiday
2023-02-26 21:38:07 <psykose> :)
2023-02-26 21:38:48 <psykose> deu1 rebooting time
2023-02-26 21:39:03 <ikke> rip algitbot
2023-02-26 21:41:14 <psykose> and it's always like those weird bunchofnumbers names
2023-02-26 21:41:16 <ikke> maybe I can extend my spam detecting script to check user profiles as well
2023-02-26 21:41:35 <psykose> or nah
2023-02-26 21:41:37 <psykose> just words
2023-02-26 21:41:46 <psykose> here's one https://gitlab.alpinelinux.org/admin/users/cntcmpsr
2023-02-26 21:50:33 <ikke> psykose: 3.14 is probably not reporting because it's building
2023-02-26 21:50:40 <psykose> makes sense
2023-02-26 21:50:40 <ikke> ie, it's still busy
2023-02-27 05:07:59 <ikke> Woosh
2023-02-27 07:02:52 <clandmeter> christmas is early this year
2023-02-27 11:17:42 <ikke> I've added vigir23 to zabbix and set a dependency, should reduce the amount of alerts
2023-02-27 13:53:33 <clandmeter> it looks slightly unstable
2023-02-27 13:58:20 <mps> does arm machines works at all. I can't ping them
2023-02-27 14:00:43 <mps> or they use only IPv6
2023-02-27 14:03:06 <ikke> The servers have only ipv6 publically 
2023-02-27 14:03:26 <ikke> But your container have an internal ipv4 address 
2023-02-27 14:04:52 <ikke> Annoying, need to tweak that 
2023-02-27 14:07:11 <mps> I see clandmeter have ipv6 address in wg config
2023-02-27 14:07:36 <mps> so we could use ipv6?
2023-02-27 14:14:02 <ikke> I think so 
2023-02-27 14:38:08 <mps> I added one ipv6 address for me but it isn't assigned on my client
2023-02-27 14:38:25 <mps> have to look later at these things
2023-02-27 15:23:26 <clandmeter> mps: we need to assign ipv6 addresses to the containers
2023-02-27 15:23:35 <clandmeter> not sure ikke has already done that
2023-02-27 15:24:13 <ikke> Not yet 
2023-02-27 15:28:10 <mps> clandmeter: I can't set my local wg to get ipv6 address. do you have config which do this?
2023-02-27 15:32:04 <clandmeter> since when can wg get remote ip config?
2023-02-27 15:32:59 <clandmeter> or im not catching your point
2023-02-27 15:33:32 <mps> I mean, how to set my local config to have ipv6 address
2023-02-27 15:33:47 <mps> on hub I already set it
2023-02-27 15:36:46 <mps> from hub 'ping 2a01:7e01:e001:46b::3' trying to ping my local endpoint but I didn't managed to set my local ipv6 to wg interface
2023-02-27 15:37:00 <clandmeter> there is a subnet set on the wg interface on wg.a.o
2023-02-27 15:37:29 <clandmeter> you need to assign one of the ip's from that subnet on your local wg interface
2023-02-27 15:37:42 <clandmeter> like Address = 2a01:7e01:e001:46b::2/128, 172.16.252.18/32
2023-02-27 15:38:23 <mps> I set it for me '2a01:7e01:e001:46b::3/128, 172.16.252.6/32'
2023-02-27 15:38:44 <clandmeter> so you have this ip on your lcoal interface?
2023-02-27 15:39:03 <mps> that is problem, I don't have it locally
2023-02-27 15:39:32 <clandmeter> i see it on my macos
2023-02-27 15:40:18 <clandmeter> although its tun instead of wg
2023-02-27 15:40:25 <clandmeter> golang implemetnation
2023-02-27 15:40:33 <mps> aha
2023-02-27 15:40:55 <clandmeter> can you share your config without private key?
2023-02-27 15:41:05 <mps> will look how to do it on linux, though it should work out-of-the-box
2023-02-27 15:41:25 <clandmeter> did you load ipv6 kernel module?
2023-02-27 15:41:41 <mps> it is compiled in-kernel
2023-02-27 15:41:46 <clandmeter> ok
2023-02-27 15:44:54 <mps> I thought it is enough to set 'Address = 2a01:7e01:e001:46b::3/128' in local config but it is not
2023-02-27 15:49:57 <ikke> mps: don't you need to manually add the address to your interface? 
2023-02-27 15:50:11 <mps> oh, yes. editing wrong config file will not solve problem :D
2023-02-27 15:52:58 <mps> now it works
2023-02-27 15:53:19 <clandmeter> yes, well it depends if you use wg-quick right?
2023-02-27 15:53:55 <clandmeter> wg-quick makes editing interfaces config obsolete
2023-02-27 15:54:12 <clandmeter> and also sets needed routes, iirc
2023-02-27 15:54:44 <mps> it set basic routes but not all
2023-02-27 15:56:30 <mps> what is DNS server address of the infra
2023-02-27 15:58:50 <ikke> 172.16.6.3, no ipv6 yet
2023-02-27 15:59:17 <mps> aha, ok
2023-02-27 16:02:13 <mps> any ipv6 address except hub in infra to test ping
2023-02-27 16:02:26 <clandmeter> me :)
2023-02-27 16:02:40 <clandmeter> but im not sure if that works
2023-02-27 16:03:30 <mps> your tunnel is down
2023-02-27 16:03:33 <mps> ?
2023-02-27 16:04:14 <clandmeter> no
2023-02-27 16:04:16 <mps> ah, it is up
2023-02-27 16:04:31 <clandmeter> if you login to wg.a.o you should be able to see it :)
2023-02-27 16:05:15 <mps> I'm already there and see
2023-02-27 16:05:28 <clandmeter> seeing is believing :)
2023-02-27 16:05:46 <mps> can ping your ip from wg.a.o but can't from my local machine
2023-02-27 16:05:56 <ikke> note that you would just go outside
2023-02-27 16:06:06 <ikke> not over dmvpn
2023-02-27 16:06:35 <clandmeter> yes, we do not have proper ipv6 in place
2023-02-27 16:06:45 <mps> ikke: I don't have public ipv6
2023-02-27 16:07:01 <clandmeter> but you should still be able to ping me i assume
2023-02-27 16:07:02 <ikke> mps: I gathered as much
2023-02-27 16:07:14 <clandmeter> as it wont touch dmvpn
2023-02-27 16:07:59 <mps> clandmeter: I can ping you over ipv4 but not over ipv6
2023-02-27 16:08:16 <ikke> mps: 2a01:7e00::f03c:93ff:fe0c:cd60
2023-02-27 16:08:55 <mps> ikke: for this I need to set ipv6 route
2023-02-27 16:09:28 <ikke> 2000::/3
2023-02-27 16:09:49 <mps> I can ping this address from one of my cloud servers
2023-02-27 16:12:11 <mps> I see my ping request on wg.a.o but not answer from clandmeter machine
2023-02-27 16:15:01 <ikke> Did you try the address I gave?
2023-02-27 16:16:03 <mps> yes
2023-02-27 16:16:07 <clandmeter> could be the fw is blocking it
2023-02-27 16:17:01 <mps> no. got this 'ping: sendmsg: Required key not available'
2023-02-27 16:17:57 <ikke> sounds like a wg issue
2023-02-27 16:18:36 <ikke> Did you allow the subnet in your wg config?
2023-02-27 16:21:17 <mps> yes
2023-02-27 16:25:32 <mps> ok, nearing there
2023-02-27 16:27:20 <mps> maybe ping echo response is blocked somewhere
2023-02-27 16:28:25 <mps> I must go now
2023-02-27 17:46:07 <psykose> ikke: armhf has been stuck for a while
2023-02-27 18:21:15 <ikke> clandmeter: anything left an usa9 we need to migrate, or can we return it to ARM?
2023-02-27 18:29:32 <ikke> psykose: building zola?
2023-02-27 18:29:37 <psykose> ye
2023-02-27 19:31:55 <ikke> seems like zola will be stuck again
2023-02-27 19:53:31 <psykose> yeah, guess it wasn't just a CI thing
2023-02-27 20:01:50 <ikke> Compiling futures-channel v0.3.26
2023-02-27 20:08:26 <psykose> sweet
2023-02-27 20:30:09 <psykose> i disabled it, just bounce it again
2023-02-27 20:30:39 <psykose> i also still can't resolve the armv7/aarch64 container dns for some reason
2023-02-27 20:31:26 <psykose> when you have time, you could make me an armhf one and i'd try and look at it a little closer
2023-02-27 20:39:06 <ikke> I'll do that tomorrow
2023-02-27 21:14:39 <Cogitri> ikke: Are the docker-alpine / ci-docker-image agents only available to the alpine/infra group?
2023-02-27 21:14:52 <Cogitri> And if so, can I get another repo for the CI images of apk-polkit-rs? :)
2023-02-27 21:23:34 <Cogitri> Here is the (untested) repo which could be migrated https://gitlab.alpinelinux.org/Cogitri/apk-polkit-rs-docker
2023-02-28 00:14:14 <psykose> ikke: urgh, somewhat sure it's now just a complete identical issue as the CI for armhf for everything
2023-02-28 00:14:34 <psykose> whatever changed between the older and the newer builder now exposes it there as well
2023-02-28 00:14:38 <psykose> maybe that's a clue
2023-02-28 05:48:52 <ikke> psykose: ci was running in an aarch64 vm on the same hardware as the builders were
2023-02-28 05:49:03 <psykose> i know that part, which is why it's weird
2023-02-28 05:49:12 <ikke> yes, thinking out loud
2023-02-28 05:49:24 <psykose> what's it stuck on now
2023-02-28 05:50:54 <ikke>  Running `rustc --crate-name futures_sink
2023-02-28 05:51:27 <psykose> from like ps
2023-02-28 05:51:29 <ikke> but, there are a lot of processes
2023-02-28 05:51:32 <psykose> the last thing in the log doesn't count
2023-02-28 05:51:36 <ikke> yea
2023-02-28 05:51:43 <psykose> if you find it uhh
2023-02-28 05:51:48 <psykose> gdb -p and bt full it i guess
2023-02-28 05:51:53 <psykose> maybe it's not useless
2023-02-28 05:52:03 <ikke> https://tpaste.us/wxne
2023-02-28 05:52:05 <psykose> inb4 futures-channel again
2023-02-28 05:52:18 <ikke> no features-channel in there
2023-02-28 05:52:19 <psykose> ugh
2023-02-28 05:52:23 <psykose> are they /all/ stuck
2023-02-28 05:52:28 <psykose> .. are they all 100% cpu?
2023-02-28 05:52:49 <psykose> just htop should show that
2023-02-28 05:53:49 <ikke> there are a bunch of processes that are owned by PID1
2023-02-28 05:54:04 <ikke> maybe left-over from zola?
2023-02-28 05:54:11 <psykose> ah, probably
2023-02-28 05:54:12 <ikke> none are 100%
2023-02-28 05:54:17 <psykose> nothing at all? that is weird
2023-02-28 05:54:36 <psykose> do you mind just killing all of them and starting by hand then, it will probably just hang again
2023-02-28 05:56:29 <psykose> hmm
2023-02-28 05:56:31 <psykose> ok i have more ideas
2023-02-28 05:56:35 <psykose> it looks very similar to https://github.com/rust-lang/rust/issues/75045
2023-02-28 05:57:01 <psykose> even though that's not the same target
2023-02-28 05:57:02 <ikke> says it would be fixed with llvm14
2023-02-28 05:57:08 <psykose> yeah
2023-02-28 05:57:11 <psykose> but it makes me curious
2023-02-28 05:57:18 <psykose> so first, reproduce the hang, we can check some stuff
2023-02-28 05:57:26 <psykose> and then after, you can try just.. change the opt level
2023-02-28 06:19:46 <ikke> I have to do that later, don't have time now
2023-02-28 06:20:40 <psykose> np
2023-02-28 10:49:44 <psykose> ikke: another networking thing- these above things break all the arm uploading so they take like 30 minutes or whatever to upload even for a small package diff
2023-02-28 11:08:03 <clandmeter> Maybe it’s related wireguard connection
2023-02-28 11:11:36 <clandmeter> Is there a reconnect option in wg?
2023-02-28 11:12:06 <mps> sending packet is enough
2023-02-28 11:13:31 <mps> wg doesn't have classical concept of connect/disconnect (though this discussion could end in nitpicking)
2023-02-28 11:15:09 <mps> aha, enjoy. everything is ok with smtp.a.o
2023-02-28 11:15:22 <mps> uh, sorry. wrong window
2023-02-28 11:16:19 <clandmeter> PersistentKeepalive
2023-02-28 11:17:10 <mps> yes
2023-02-28 11:17:42 <clandmeter> telmich:  looks like the connection is still unstable. As seen in this channel.
2023-02-28 11:31:17 <mps> "green friendly" ;P
2023-02-28 11:31:32 <mps> but also "red friendly"