2020-07-01 07:15:21 <clandmeter> ikke: see the billing email?
2020-07-01 07:15:29 <clandmeter> we are getting close to the limit
2020-07-01 07:16:30 <ikke> We should destroy the gitlab-test instance
2020-07-01 07:17:10 <clandmeter> yep
2020-07-01 07:17:20 <clandmeter> we should do it directly after we finish the test
2020-07-01 07:17:23 <ikke> Where do I see what the limit is?
2020-07-01 07:17:31 <clandmeter> i have no idea
2020-07-01 07:17:36 <ikke> ok
2020-07-01 07:17:36 <clandmeter> from my head it was 400
2020-07-01 07:19:14 <ikke> deleted it now
2020-07-01 07:31:01 <clandmeter> i see it no more
2020-07-01 15:33:57 <maxice8> x86_64 builder is stuck
2020-07-01 15:44:30 <ikke> mqtt-exec crashed
2020-07-01 15:44:38 <ikke> (and not using supervisor-daemon yet)
2020-07-01 15:45:12 <ikke> started it again
2020-07-01 15:46:24 <maxice8> rip
2020-07-01 16:00:45 <ikke> Now that it's started again, supervisor-daemon is also used
2020-07-01 16:35:01 <ikke> oooh: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html
2020-07-01 16:36:17 <ikke> still alpha though
2020-07-01 18:44:04 <MY-R> many (edge) mirrors got APKINDEX.tar.gz from 29-Jun and on mirrors.alpinelinux.org those show status OK, is it normal?
2020-07-01 18:47:01 <MY-R> hours ago I got for example "znc" package upgrade which was built 2 days ago but appeared just now on dl-cdn.alpinelinux.org
2020-07-01 18:47:45 <ikke> x86_64?
2020-07-01 18:48:07 <MY-R> yep
2020-07-01 18:48:23 <ikke> The builder was not functioning 
2020-07-01 18:49:12 <MY-R> oh ok, thought mirrors.a.o should catch that 
2020-07-01 18:50:31 <ikke> If the builder publishes nothing, there is nothing to get behind from
2020-07-01 18:51:10 <MY-R> :\
2020-07-01 19:01:55 <ikke> MY-R: is there any concrete issue you were facing, or just wondering?
2020-07-01 19:02:15 <MY-R> nah, just wondering, sorry for noise :)
2020-07-01 19:02:19 <ikke> no worry
2020-07-02 04:55:25 <ikke> clandmeter: ^
2020-07-02 04:57:33 <ikke> It's not growing atm
2020-07-02 04:57:40 <ikke> (significantly)
2020-07-02 09:33:56 <ncopa> can someone please help me trigger alpine-mksite build?
2020-07-02 09:34:08 <ncopa> https://alpinelinux.org/posts/ is still empty while fix was pushed to git
2020-07-02 09:34:18 <ncopa> not sure why it does not automatically update on push
2020-07-02 09:57:15 <ncopa> the 100 biggest packages in edge: https://tpaste.us/vJYQ
2020-07-02 09:57:36 <ncopa> we could save some space by removing xonotic, supertux and wesnoth
2020-07-02 09:58:29 <ncopa> i wonder if we should have a separate "games" repo or similar
2020-07-02 09:58:39 <ncopa> which are not distributed on the mirrors
2020-07-02 10:08:09 <clandmeter> i can check it later
2020-07-02 10:08:13 <clandmeter> busy with work atm
2020-07-02 10:08:40 <clandmeter> i made some modifications last time, not sure why its not working.
2020-07-02 20:46:24 <clandmeter> ncopa: i had to manually do a make clean && make all
2020-07-02 20:47:05 <maldridge> ncopa: void just restricts the data packages, so that they aren't built by the distro
2020-07-02 20:47:19 <maldridge> but that was mostly brought about by sauerbraten, which has almost a gig of high res assets
2020-07-02 20:54:55 <MY-R> there is no sauerbraten in Void :\
2020-07-02 20:56:03 <maldridge> it is a restricted package
2020-07-02 20:56:25 <maldridge> for pretty much the exact reason brought up above, its huge, and it takes valuable space on mirrors
2020-07-02 20:56:33 <MY-R> but I see urbanterror-data 1.35G
2020-07-03 05:58:54 <mps> is it safe to keep ssh keys on developers lxc containers which are needed for pushing aports to gitlab.a.o
2020-07-03 06:02:43 <maldridge> well, who has access to override the security policy protecting the containers, and could they obtain or forge those keys in another way?
2020-07-03 06:03:17 <maldridge> I personally find it helpful to reason about data security in terms of "if I didn't put this data there, could someone get it by some other way"
2020-07-03 06:04:19 <mps> at the bottom these boxes are owned by sponsors (packet) and I think they can read anything on them at the end
2020-07-03 06:05:17 <mps> but syncing aports and other things over my slow ADSL is quite irritating
2020-07-03 06:06:06 <maldridge> right, but in this case Packet would be an evil made, and that's fundamentally indefensible in a cloud environment
2020-07-03 06:06:37 <maldridge> s/made/maid/
2020-07-03 06:06:37 <alpine-meetbot> maldridge meant to say: right, but in this case Packet would be an evil maid, and that's fundamentally indefensible in a cloud environment
2020-07-03 06:06:39 <mps> yes, the question is how much we trust them
2020-07-03 06:24:19 <ikke> mps: I would say it's not safe honestly
2020-07-03 06:26:25 <mps> that is what I thought
2020-07-03 06:27:41 <maldridge> I would still posit that that ship has already sailed
2020-07-03 06:30:27 <mps> also, gitlab is on linode, so we are already 'in hand' of one of sponsors
2020-07-03 06:32:01 <mps> but we would do our best to not compromise our private keys
2020-07-03 06:47:54 <maldridge> ofc
2020-07-03 07:31:11 <Cogitri> ikke: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/9903
2020-07-03 07:32:02 <Cogitri> mps: I'd say it's always best practice to make a seperate set of keys per machine so you'd only have to invalidate that on set in case things go south
2020-07-03 07:32:23 <Cogitri> s/on//
2020-07-03 07:32:23 <alpine-meetbot> Cogitri meant to say: mps: I'd say it's always best practice to make a seperate set of keys per machine so you'd ly have to invalidate that on set in case things go south
2020-07-03 07:32:43 <Cogitri> Replacing ssh keys everywhere is no fun
2020-07-03 07:40:07 <mps> Cogitri: that was not a question at all, it is assumed
2020-07-03 07:40:49 <mps> but keep keys to push changes on our sponsors boxes is the question
2020-07-03 07:40:57 <Cogitri> Ah okie
2020-07-03 07:41:45 <mps> np :)
2020-07-03 20:11:17 <Ariadne> MY-R: urbanterror-data was actually dropped because of licensing, but still exists in the repo due to a bug
2020-07-03 20:12:24 <MY-R> ah, ok!
2020-07-04 20:46:27 <Cogitri> ikke: Can you maybe check if something suspicious is in appstream-generator's log? Seems like only for a few pkgs the icons are included
2020-07-04 20:46:34 <Cogitri> I guess for the applications which got updated in the meantime?
2020-07-04 20:51:05 <ikke> Cogitri: https://tpaste.us/b4OW
2020-07-04 20:51:12 <ikke> I don't see anything suspicious
2020-07-04 20:54:03 <Cogitri> Hm, same, weird...
2020-07-05 08:56:41 <Cogitri> ikke: Changed the cron file a bit for asgen, maybe it properly exports all icons now (can't really test on a 16MBit/s connection, I'd download all week long :D)
2020-07-05 08:57:03 <ikke> I'll update it later
2020-07-05 08:57:27 <Cogitri> Great, thanks. No rush :)
2020-07-05 10:31:25 <ikke> updated the container
2020-07-06 06:03:48 <mps> I think I will need x86_64 lxc on alpine infra, I don't have it locally anymore
2020-07-06 10:23:08 <Cogitri> ikke: Seems like my change to asgen didn't fix the icons either (so I'll have yo look into that later), but caused files to not be put into the dated dir. I've reverted my changes, so could you restart the container (and maybe remove the folders which aren't in the dated folders?)
2020-07-06 10:29:02 <ikke> would it also be possible to switch to nginx-alpine?
2020-07-06 10:38:42 <Cogitri> Sure thing 
2020-07-06 10:41:44 <ikke> These ubuntu / debian images have to ommit a lot of things that Alpine has through bb to keep small
2020-07-06 10:42:18 <Cogitri> Pushed
2020-07-06 10:42:44 <ikke> just pulled :P
2020-07-06 10:43:11 <ikke> ok, should be fixed now
2020-07-06 11:22:40 <ncopa> ikke: i wonder if we could have a meeting about how to set up the new x86/x86_64 builders
2020-07-06 11:24:14 <ncopa> i'd like to get it done today
2020-07-06 11:24:18 <ncopa> i mean this week
2020-07-06 11:43:16 <ikke> ncopa: fine with me
2020-07-07 08:30:40 <ncopa> clandmeter: FYI: me and ikke are setting up nld8 and nld9 machines to replace the old bld1 and bld2 machines. They are for x86 and x86_64 builders
2020-07-07 08:31:15 <ncopa> ikke: dmvpn is running on both of them now
2020-07-07 08:31:39 <ikke> ikke: ok, good
2020-07-07 08:31:57 <ikke> lxcbr0 is still down
2020-07-07 08:32:23 <ncopa> because we dont have any containers running
2020-07-07 08:32:24 <ikke> I guess it needs containers to bring it up
2020-07-07 08:32:26 <ikke> yeah
2020-07-07 08:32:32 <ncopa> it should come up with first container
2020-07-07 08:33:14 <ncopa> we should also update dns, delegate {nld8,nld9}.alpin.pw
2020-07-07 08:35:00 <ikke> yes, I can arrange that
2020-07-07 08:35:35 <ncopa> hm
2020-07-07 08:35:51 <ncopa> i wonder if we should start with the build-newedge-x86
2020-07-07 08:36:41 <ikke> I'm trying to get local dns working, but it does not resolve yet
2020-07-07 08:37:01 <ikke> ie, nld8.alpin.pw or nld8-dev1.alpin.pw
2020-07-07 08:37:09 <ikke> local dnsmasq should be able to resolve that
2020-07-07 08:37:24 <ncopa> lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> 
2020-07-07 08:37:32 <ncopa> maybe the NO-CARRIER thing?
2020-07-07 08:37:45 <ikke> well, I do get a response
2020-07-07 08:47:11 <ncopa> did you restrat dnsmasq after config changes?
2020-07-07 08:47:25 <ikke> hmm, good one :)
2020-07-07 08:47:45 <ikke> I think I changed it before I started the service though
2020-07-07 08:48:23 <ncopa> im copying over build-newedge-x86 to nld8
2020-07-07 08:48:43 <ikke> ok
2020-07-07 08:54:35 <ncopa> dns resolving is non-working
2020-07-07 08:56:19 <ncopa> ;; Got recursion not available from 172.16.21.2, trying next server
2020-07-07 08:58:00 <ikke> can you verify /etc/lxc/dnsmasq.conf? 
2020-07-07 09:00:52 <ncopa> looks good to me
2020-07-07 09:09:43 <ncopa> ok, i think i know whats wrong
2020-07-07 09:10:33 <ncopa> wrong ip in /etc/resolv.conf
2020-07-07 09:10:52 <ncopa> it was using the .2 address that is auth listener
2020-07-07 09:10:56 <ncopa> the resolver is on .1
2020-07-07 09:11:07 <ncopa> works now
2020-07-07 09:11:21 <ikke> hmm, ok, I copied that from the current bld servers
2020-07-07 09:49:57 <ncopa> something is wrong with lxc in 3.12
2020-07-07 09:50:02 <ncopa> lxc-start: a1: cgroups/cgfsng.c: cgfsng_monitor_create: 1264 Failed to create cgroup "(null)"
2020-07-07 10:31:02 <ikke> any idea what is causing that?
2020-07-07 10:31:19 <ncopa> no idea, but i dont think that is the blocker i have at hand
2020-07-07 10:32:08 <ikke> oh ok
2020-07-07 10:32:23 <ncopa> same error happens in a fresh 3.12 install wher creating a new alpine lxc
2020-07-07 10:32:35 <ncopa> but the container will start
2020-07-07 10:32:55 <ncopa> seems like it fails to set up lxc.monitor in cpuset
2020-07-07 10:33:19 <ncopa> it manages to create it, but fails later, and then does not clean it up
2020-07-07 10:37:42 <ikke> Anything I can help with?
2020-07-07 10:38:20 <ncopa> it failed to boot due to missing /var/cache/distfiles dir. I have solved that, but it does not get dhcp address now
2020-07-07 10:38:22 <ncopa> not sure why
2020-07-07 10:38:39 <ncopa> i also wonder what that cpuset monitor thing error is
2020-07-07 10:39:03 <ikke> lxcbr0 is still down
2020-07-07 10:43:11 <ncopa> it comes up as soon i start up the container
2020-07-07 10:43:14 <ncopa> but no dhcp
2020-07-07 10:43:35 <ikke> hmm
2020-07-07 10:44:11 <ncopa> ok, enabling awall policy adp-dhcp solved that
2020-07-07 10:44:40 <ncopa> aha, maybe the cpuset monitor thing needs iptables of some sort
2020-07-07 10:45:14 <ncopa> ikke: did you configure the delegation of dns to nld8,nld9?
2020-07-07 10:45:21 <ikke> no, will do that now
2020-07-07 10:49:38 <ncopa> i thikn we need awall policy that allows lxcbr0
2020-07-07 10:49:52 <ncopa> or 172.16/21
2020-07-07 10:50:33 <ikke> hmm
2020-07-07 10:50:34 <ikke> ok
2020-07-07 10:51:06 <ncopa> the build-newedge-x86 does not seem to be able to get anything from internet either
2020-07-07 10:51:09 <ncopa> maybe NAT?
2020-07-07 10:51:53 <ikke> Yeah, I think we need to add that
2020-07-07 10:51:56 <ikke> and forwarding probably as well
2020-07-07 10:52:08 <ikke> does awall handle that?
2020-07-07 10:52:10 <ncopa> adp-router policy?
2020-07-07 10:52:41 <ikke> yes, I believe so
2020-07-07 10:56:33 <ikke> hmm, it also responds when I push to another branch apparently
2020-07-07 10:58:15 <ikke> ncopa: the zones should have been added now
2020-07-07 10:58:22 <ncopa> thanks!
2020-07-07 10:59:13 <ncopa> i think maybe firewall is blocking inbound dns traffic
2020-07-07 10:59:27 <ikke> yeah, you need to explicitly allow everything with adp
2020-07-07 10:59:32 <ncopa> it also blocks traffic to and from the lxc
2020-07-07 10:59:43 <ikke> even outbound traffic I noticed
2020-07-07 11:00:05 <ncopa> there is an adp policy for outbound
2020-07-07 11:00:11 <ncopa> i enable ssh-client
2020-07-07 11:00:24 <ncopa> but i think we need allow the dmvpn traffic somehow
2020-07-07 11:00:30 <ikke> local-outbound
2020-07-07 11:00:43 <ikke> there is a config.json
2020-07-07 11:00:47 <ikke> we might need to add it there
2020-07-07 11:00:54 <ncopa> ok
2020-07-07 11:00:57 <ikke> adp-config.json
2020-07-07 11:01:00 <ncopa> my time is up for today
2020-07-07 11:01:03 <ikke> ok
2020-07-07 11:01:07 <ikke> adp_dhcp_zones
2020-07-07 11:01:20 <ikke> adp_lan_private_addrs
2020-07-07 11:01:28 <ikke> Will try to specify those
2020-07-07 11:01:35 <ncopa> feel free to config the firewall. build-newedge-x86.nld8.alpin.pw should be running
2020-07-07 11:01:46 <ikke> ok
2020-07-07 11:01:57 <ncopa> it needs to fetch things from internet
2020-07-07 11:02:05 <ncopa> and will need to scp the logs
2020-07-07 11:02:14 <ikke> right
2020-07-07 11:02:35 <ncopa> i think we should use a *alpin.pw address to push logs
2020-07-07 11:35:48 <ikke> I've added the dhcp profile and set the interface, but it dhcp does not work yet
2020-07-07 17:16:02 <mps> ikke: is there again disk issue on usa4, dmesg show something suspicious
2020-07-07 17:17:15 <ikke> mps: do you also notice issues?
2020-07-07 17:19:05 <ikke> sadly bb dmesg doesn't support -T (absolute time)
2020-07-07 17:19:18 <ikke> So hard to verify when these issues happened
2020-07-07 17:19:22 <mps> no, everything works fine for me, but I had segfault in tests for libcap and wanted to look in dmesg
2020-07-07 17:19:47 <mps> ikke: -T can be missleading
2020-07-07 17:19:53 <ikke> oh?
2020-07-07 17:20:05 <ikke> yeah, help says "may be inaccurate"
2020-07-07 17:20:24 <mps> it calculates relative time, not absolute
2020-07-07 17:21:04 <mps> or the dmesg from util-linux is buggy
2020-07-07 17:21:04 <ikke> You mean, it calculates absolute time based on relative time
2020-07-07 17:21:16 <mps> yes
2020-07-07 17:21:34 <ikke> ok, but it looks like those messages were yestday
2020-07-07 17:21:45 <ikke> (If I may believe the time it gives :))
2020-07-07 17:21:58 <mps> sometimes it works ok but sometimes doesn't
2020-07-07 17:23:00 <mps> if we have syslog messages there they can be more accurate
2020-07-07 17:24:03 <ikke> apparently we only have a couple of hours of syslog
2020-07-07 17:24:15 <mps> ah
2020-07-07 17:25:14 <mps> you can install util-linux temporary and try with its dmesg -T
2020-07-07 17:25:25 <ikke> yes, apparently it was already installed and that's what I did
2020-07-07 17:31:25 <Cogitri> Do we happen to have a script to cleanup CI runners? My runner is slowly approaching 100G just for docker containers which are months old now
2020-07-07 17:31:37 <ikke> Cogitri: docker system prune -v
2020-07-07 17:31:40 <ikke> in a cron job
2020-07-07 17:31:51 <ikke> (add -f as well)
2020-07-07 17:32:06 <Cogitri> Ah okay, wasn't sure if that would cause problems for the runner
2020-07-07 17:32:09 <ikke> nope
2020-07-07 17:32:16 <ikke> it will only remove things that are no longer in use
2020-07-07 17:32:34 <ikke> Just run it once a week a or so
2020-07-07 17:32:49 <Cogitri> Okay, thanks :) 
2020-07-07 17:32:57 <Cogitri> 
ACTION makes very sure I don't accidentally purge my mail server

2020-07-07 17:33:16 <ikke> hehe :P
2020-07-07 17:33:30 <ikke> make sure to add autostart
2020-07-07 17:34:31 <Cogitri> Yup, should be enabled since I don't have to manually start it after restarting my VPS
2020-07-07 17:38:52 <Cogitri> > Total reclaimed space: 55.65GB
2020-07-07 17:38:53 <Cogitri> Wew
2020-07-07 17:39:02 <ikke> heh
2020-07-07 17:39:48 <Cogitri> And mail server and sharelatex instance still seem to be alive, excellent :) 
2020-07-07 17:39:50 <Cogitri> Thanks!
2020-07-07 17:40:09 <ikke> no problem
2020-07-08 06:42:20 <ikke> ncopa: I have dhcp working and traffic out from the container
2020-07-08 07:01:58 <clandmeter> morning
2020-07-08 07:04:56 <clandmeter> i have a feeling the awall policies are too restrictive, would be nice if it would be configurable with a setting, or a question/option when running setup-firewall.
2020-07-08 07:59:15 <ikke> doesn't sound like a bad plan
2020-07-08 08:11:41 <ncopa> +1
2020-07-08 10:27:11 <ikke> clandmeter: there is a local-outbound profile that sets the policy through a variable
2020-07-08 12:25:04 <clandmeter> ikke: we need to update gitlab
2020-07-08 12:25:12 <clandmeter> i just pushed a new patch release
2020-07-08 12:25:17 <clandmeter> but we are 2 behind
2020-07-08 12:25:36 <ikke> ok
2020-07-08 12:25:37 <clandmeter> if you can help by pull/restart that would be nice.
2020-07-08 12:25:51 <clandmeter> there are also a few weird repos
2020-07-08 12:25:57 <clandmeter> well atleast 1
2020-07-08 12:26:24 <ikke> You mean that fsck is complaining about?
2020-07-08 12:55:16 <clandmeter> ikke: no something like this: https://gitlab.alpinelinux.org/admin/projects/dHannasch/try-to-use-kubernetes
2020-07-08 12:55:25 <ikke> ah
2020-07-08 20:08:03 <PureTryOut[m]> So the Gitlab team thingy requires me to setup 2FA. I want to do that, but I can't seem to configure it with my NitroKey (which does support TOTP and I use it for gitlab.com for example)
2020-07-08 20:08:30 <PureTryOut[m]> Gitlab in this case gives me a key I need to enter in "the OTP app", but the application that comes with NitroKey doesn't support something like that
2020-07-08 20:09:33 <PureTryOut[m]> Oh, never mind, got it
2020-07-08 20:09:42 <ikke> PureTryOut[m]: ok, cool
2020-07-08 20:09:58 <PureTryOut[m]> It's a bit of strange UX but it seems I was just too late the first time I tried it 🤷‍♂️
2020-07-08 20:10:09 <ikke> Yeah, it's kind of flaky for me as well
2020-07-08 20:30:06 <ikke> clandmeter: I've upgraded gitlab btw
2020-07-08 20:30:18 <clandmeter> i saw your msg
2020-07-08 20:38:56 <ikke> ok, good
2020-07-08 20:55:34 <ikke> One issue with the current team structure is that the parent group does need to have an owner, which is automatically member of all subgroups
2020-07-08 20:55:44 <ikke> (and owner as well)
2020-07-08 21:53:53 <maldridge> sorry if I've asked this before
2020-07-08 21:53:58 <maldridge> who actually owns alpine's infrastructure?
2020-07-08 21:54:45 <maldridge> I get that a lot of it is donations, but presumably there's some owner somewhere at the bottom
2020-07-09 06:37:58 <ncopa> maldridge: the linode servers are owned by linode, the packet.net servers are owned by packet.net, the s390x and ppc64le machines are owned by IBM, the alpinelinux.org DNS name is owned by me
2020-07-09 06:39:06 <ncopa> ikke: nld8.alpin.pw does not resolve for me here locally
2020-07-09 06:40:04 <maldridge> ncopa: is there an overarching alpine entity that's the consumer of services from those hosts?
2020-07-09 06:40:11 <maldridge> I thought y'all had DO stuff as well?
2020-07-09 06:54:04 <ncopa> no, there is no alpine entity (yet)
2020-07-09 06:54:18 <ncopa> nobody has cared enough to actually do the paperwork for that
2020-07-09 06:54:50 <maldridge> interesting
2020-07-09 07:00:21 <ncopa> ikke: I have enabled DNS from dmvpn -> _fw on nld8
2020-07-09 07:17:06 <ikke> ncopa: no, I have not
2020-07-09 07:19:57 <ikke> ncopa: on the other hosts we have these rules:
2020-07-09 07:19:59 <ikke> -A INPUT -i lxcbr+ -j ACCEPT
2020-07-09 07:20:01 <ikke> -A INPUT -i gre1 -j ACCEPT
2020-07-09 07:20:03 <ikke> Do we want something similar?
2020-07-09 07:21:39 <ncopa> i think that may be a good idea
2020-07-09 07:23:46 <ikke> aha, usa1 is using adp as well
2020-07-09 07:24:25 <ikke> ok, nice, it has an lxc policy that I can copy
2020-07-09 07:28:24 <ikke> ncopa: I've copied the basic setup of usa1 now
2020-07-09 08:16:33 <ncopa> thanks!
2020-07-09 08:52:50 <ikke> ncopa: is everything working now?
2020-07-09 10:31:51 <ncopa> ikke: i think so. just need to migrate the containers
2020-07-09 10:31:55 <ncopa> but im out of time again..
2020-07-09 10:31:59 <ikke> ncopa: ok
2020-07-09 10:32:15 <ikke> if you want me to migrate some containers, let me know
2020-07-09 10:33:49 <ncopa> would be nice. thanks
2020-07-09 10:34:00 <ncopa> maybe take the old, unsupported containers first
2020-07-09 10:34:14 <ncopa> may need modify lxc config
2020-07-09 10:34:19 <ikke> right
2020-07-09 10:34:24 <ncopa> i added an /etc/lxc/alpine.common.conf
2020-07-09 10:34:41 <ncopa> there seems to some issue with cpuset monitor cgroup something that I dont know what is
2020-07-11 09:05:55 <ikke> (I've raised the limit a bit for this one ^)
2020-07-11 10:33:46 <Cogitri> Hm, how come https://gitlab.alpinelinux.org/Cogitri/alpine-qa-bot/-/merge_requests/8 doesn't run CI? 🤔
2020-07-11 10:34:38 <Cogitri> The project itself has a private runner (just the one I have for aports too, wanted to see if I could do without shared runners) and I've went ahead and also enabled shared runners to see if that fixed it but apparently to no avail
2020-07-11 10:40:19 <ikke> Do you have one of these tags defined in the CI job? "ci-build docker-alpine x86_64 "
2020-07-11 10:41:13 <Cogitri> Oh no, I think I enabled tagless running on my private runner so I didn't keep in mind that I need that for shared runners, thanks :)
2020-07-11 10:41:24 <ikke> But it should work for your private runner
2020-07-11 10:41:35 <Cogitri> Ah, wasn't sure if those work for forks
2020-07-11 10:41:39 <ikke> aha, no
2020-07-11 10:42:07 <Cogitri> Also, do we have gitlab pages enabled for our instance? Would be neat if I could host the HTML pages of the coverage on gitlab (but it's not problem for me to host them elsewhere either)
2020-07-11 10:45:46 <Cogitri> Hm, I've added tags now and shared runners should be available, but seems like it still doesn't like running the pipeline for it: https://gitlab.alpinelinux.org/Cogitri/alpine-qa-bot/-/merge_requests/8
2020-07-11 10:56:55 <ikke> is that part of the MR?
2020-07-11 10:57:00 <ikke> or is the MR rebased?
2020-07-11 11:00:29 <Cogitri> Yes, I just rebased the MR on top of master
2020-07-11 11:00:40 <Cogitri> https://gitlab.alpinelinux.org/Leo/alpine-qa-bot/-/blob/move-to-should-be-move-from/.gitlab-ci.yml
2020-07-11 11:01:18 <ikke> I don't see that the pipeline was triggered again
2020-07-11 11:01:27 <ikke> https://gitlab.alpinelinux.org/Leo/alpine-qa-bot/pipelines
2020-07-11 11:05:01 <Cogitri> Huh, but I pushed new commits to the MR 🤔
2020-07-11 11:05:16 <Cogitri> Let's see if closing the MR and opening a new one fixes it
2020-07-11 11:05:25 <Cogitri> maxice8: Could you do that? ^
2020-07-11 11:05:52 <maxice8> ok
2020-07-11 11:08:09 <Cogitri> Huh, seems like it worked now
2020-07-11 11:08:58 <Cogitri> Seems like the job is stuck now though, I guess the shared runners aren't enabled in the fork for some reason? https://gitlab.alpinelinux.org/Leo/alpine-qa-bot/-/jobs/163436
2020-07-11 11:13:01 <ikke> at least now they are
2020-07-11 11:13:09 <ikke> might be that they inherit that from the parent repo?
2020-07-11 11:13:33 <Cogitri> Ah, I enabled shared runners in the parent repo, but only after the fork, so maybe that's it
2020-07-12 12:59:25 <Cogitri> Could we maybe make the filelist differences in CI an artifact of the pipeline? I want the qa-bot to comment file difference on the MR because otherwise it's really easy to miss them (especially when multiple packages are built)
2020-07-12 13:00:07 <Cogitri> And I want to add ABI scanning to pipeline too and use the same mechanism for that, but I'd probably do that at a later point
2020-07-12 14:51:29 <Cogitri> ikke: Also, could I maybe get another docker&-compose repo for alpine-qa-bot? :) 
2020-07-12 20:55:28 <Cogitri> ikke: ^ 😅
2020-07-12 20:55:46 <ikke> I'll do that tomowwor or so, it's kind of bed-time for me :)
2020-07-13 07:11:32 <clandmeter> Cogitri: good morning, im interested to know why you choose vala to program your bot?
2020-07-13 07:16:19 <Cogitri> Morning. I mainly chose Vala because I wanted to learn it
2020-07-13 07:16:49 <Cogitri> And I didn't have a strong reason to go with another language, Vala has pretty nice tooling for JSON and event loops/async 
2020-07-13 07:17:09 <Cogitri> And it's not as safe as Rust, but should be reasonable due to reference counting still
2020-07-13 07:17:35 <Cogitri> ikke: Ah yes, no rush. Just wasn't sure if you saw the messages :)
2020-07-13 07:19:01 <mps> Cogitri: what about lua for such things
2020-07-13 07:19:30 <mps> does it have all needed 'batteries'
2020-07-13 07:19:35 <Cogitri> I don't know Lua and have a strong preference against using weakly typed langs
2020-07-13 07:20:02 <ikke> mps: certainly not included
2020-07-13 07:20:29 <ikke> mps: the base language / run-time is very light, so to do most things you need a fair amount of libraries
2020-07-13 08:20:25 <ncopa> morning
2020-07-13 08:20:43 <ncopa> i thikn golang would have been a better tool for gitlab hooks server
2020-07-13 08:21:44 <ncopa> i think vala is ok(ish) for gnome gui stuff, but the future for vala is uncertain 
2020-07-13 08:22:27 <ikke> This is not a general hooks server I guess?
2020-07-13 08:22:31 <ikke> just a hook consumer
2020-07-13 08:22:35 <ncopa> i gave up vala for xfce projects. I ended up fighting the language and framework too much
2020-07-13 08:37:25 <Cogitri> The future of Vala is a bit uncertain indeed with GNOME moving to Rust (at least it seems like it) and its main usage being in elementary OS 
2020-07-13 08:37:51 <Cogitri> But recently new community efforts have been started, like a new LSP server which is pretty great
2020-07-13 08:38:52 <Cogitri> And Vala is pretty optimized towards dev productivity while still having excellent execution speed and being somewhat safe (by using reference counting), so it's really nice for me. Coding in Rust seems so slow in comparison (and it's async really is a quite the mindbender)
2020-07-13 08:39:26 <Cogitri> I like the GLib API quite a bit, so Vala seemed like a nice choice and I didn't have problems with it yet 
2020-07-13 08:40:34 <Cogitri> And yes, right now it mainly responds to Gitlab's webhooks (e.g. allowing collaboration on MRs), but I guess eventually I'll have to start to poll the Gitlab API because webhooks are rather limited
2020-07-13 08:41:03 <Cogitri> They don't work for forks of the original project unfortunately, so I can't just use webhooks for pipeline status and things like that 
2020-07-13 08:42:46 <Cogitri> I guess I'll have to throw in a SQLite database or something for that to remember the state of MRs
2020-07-13 08:48:26 <ncopa> there are various things like with vala
2020-07-13 08:52:22 <ncopa> ive done some golang for $dayjob and i think that golang is specially good for infra/devops kind of apps
2020-07-13 08:53:59 <Cogitri> I've only done some Golang for a JSON API parser thingie (I think that was actually my first somewhat serious project, as in I really committed to that), but somehow I'm not a huge fan of it 
2020-07-13 08:58:18 <Cogitri> How come Go is good for infra other than that you can compile it into one big statically linked exe that is easy to deploy? I feel like Docker somewhat does that for other langs too
2020-07-13 08:58:46 <ncopa> its good at concurrency
2020-07-13 08:59:28 <ncopa> i think static compile makes sense for docker deployments
2020-07-13 08:59:49 <ncopa> you lose various of the benefits with shared libs in containers
2020-07-13 09:00:09 <ncopa> but, thats my opinion anyway
2020-07-13 09:20:39 <mps> libcap upgrade fail on aarch64 builder, but pass on aarch64 CI and in 'my' aarch64 lxc
2020-07-13 09:21:21 <mps> and pass on my local native aarch64 machine
2020-07-13 09:22:19 <mps> error is '2020/07/13 09:19:02 go text rep not parsable by c'
2020-07-13 09:22:40 <mps> something about/with go lang
2020-07-13 19:30:07 <mps> can someone with builder access look at this above, libcap fails on aarch64
2020-07-13 19:31:42 <mps> or disable check() temporarily till the aarc64 builder is 'fixed'
2020-07-13 19:53:58 <ikke> mps: apparently it's something from the test suite itself
2020-07-13 19:54:26 <ikke> ie, a test failure
2020-07-13 19:55:20 <ikke> sadly, it does not give a lot of detail about what failed
2020-07-13 19:56:18 <ikke> mps: apparently there is some addition _cap_debug logging
2020-07-13 20:01:51 <ikke> mps: and of course it all works in a container :-/
2020-07-13 20:03:23 <ikke> mps: it seems it does not run that compare-cap
2020-07-13 20:04:55 <mps> ikke: yes, but I can't reproduce this issue on any of my aarch64 'boxes'
2020-07-13 20:05:02 <ikke> mps: I think I found it
2020-07-13 20:05:13 <ikke> yup
2020-07-13 20:05:26 <ikke> there were dependencies left for syncthing, which included go
2020-07-13 20:05:33 <ikke> without go, it does not run those
2020-07-13 20:05:34 <mps> only I can think is maybe go is left behind something on 
2020-07-13 20:05:38 <ikke> yes
2020-07-13 20:05:43 <mps> ah, nice
2020-07-13 20:05:52 <ikke> try installing go
2020-07-13 20:05:58 <ikke> I bet you get the error as well
2020-07-13 20:06:02 <mps> I wanted to tell, please remove go from builder
2020-07-13 20:06:32 <ikke> It was not installed explicitly
2020-07-13 20:06:40 <ikke> but I checked /etc/apk/world
2020-07-13 20:07:08 <ikke> and saw the left-behind .makedepends
2020-07-13 20:08:12 <mps> iiuc, go left only on aarch64 builder?
2020-07-13 20:08:25 <ikke> yes
2020-07-13 20:08:36 <mps> and it should be there?
2020-07-13 20:08:38 <ikke> no
2020-07-13 20:08:44 <mps> shouldn't *
2020-07-13 20:08:46 <ikke> yes
2020-07-13 20:08:53 <mps> aha
2020-07-13 20:08:55 <ikke> it was there as a dependency for syncthing
2020-07-13 20:09:06 <ikke> it was not properly cleaned up
2020-07-13 20:09:10 <ikke> maybe due to some crash
2020-07-13 20:09:34 <ikke> but, I just installed go in a container and ran it again, it and seems to work :S
2020-07-13 20:09:37 <mps> yes, that is what I thought when I was error msg,
2020-07-13 20:10:02 <mps> when I saw*
2020-07-13 20:10:28 <ikke> LD_LIBRARY_PATH=../libcap ./compare-cap
2020-07-13 20:10:30 <ikke> 2020/07/13 20:09:49 skipping file cap tests - insufficient privilege
2020-07-13 20:10:32 <ikke> 2020/07/13 20:09:49 skipping proc cap tests - insufficient privilege
2020-07-13 20:10:34 <ikke> 2020/07/13 20:09:49 compare-cap success!
2020-07-13 20:10:36 <ikke> strange
2020-07-13 20:10:56 <ikke> maybe to to the insufficient privileges
2020-07-13 20:11:03 <mps> that's ok I think, upstream proposed that
2020-07-13 20:11:43 <mps> I removed checkroot from options
2020-07-13 20:11:56 <ikke> That only happens in a container
2020-07-13 20:11:59 <ikke> I think
2020-07-13 20:12:02 <ikke> a docker container
2020-07-13 20:12:24 <mps> aha, but CI is also docker?
2020-07-13 20:12:26 <ikke> yes
2020-07-13 20:12:41 <ikke> but there is no go installed normally
2020-07-13 20:12:45 <ikke> it's not a dependency
2020-07-13 20:13:01 <ikke> and because go is in community and this is in main, it cannot be dependency either
2020-07-13 20:13:14 <mps> yes
2020-07-13 20:13:30 <ikke> s/be/be a/
2020-07-13 20:13:31 <alpine-meetbot> ikke meant to say: and be acause go is in community and this is in main, it cannot be dependency either
2020-07-13 20:13:37 <ikke> :/
2020-07-13 20:13:55 <mps> CIs is created clean on every run?
2020-07-13 20:14:59 <ikke> yes
2020-07-13 20:16:18 <ikke> Each job starts a new container based on a prestine image
2020-07-13 20:16:57 <mps> nice, so my understand and experience is better now
2020-07-13 20:17:13 <mps> understanding*
2020-07-14 17:47:49 <ikke> Cogitri: It might be possible to use the gitlab API to get a diff
2020-07-14 17:47:55 <ikke> at least, changed files
2020-07-14 17:53:14 <ikke> https://gitlab.alpinelinux.org/api/v4/projects/alpine%2faports/merge_requests/10332/versions/26490
2020-07-14 17:57:34 <ikke>  https://gitlab.alpinelinux.org/api/v4/projects/alpine%2faports/merge_requests/10325/versions/26435  | jq -r '.diffs[] | .new_path' shows all the changes files :)
2020-07-14 18:27:31 <Cogitri> Oh great, thanks :) 
2020-07-14 18:28:04 <Cogitri> I had expected that I'd need to parse the CI log otherwise, that's a lot nicer :) 
2020-07-14 18:28:09 <ikke> nope
2020-07-14 18:29:10 <ikke> hmm
2020-07-14 18:29:11 <ikke> https://docs.gitlab.com/ee/ci/metrics_reports.html
2020-07-14 18:29:13 <ikke> what about that?
2020-07-14 18:29:38 <ikke> ah, ee only
2020-07-15 20:21:18 <ikke> clandmeter: ping
2020-07-15 21:05:19 <clandmeter> Pong
2020-07-15 21:05:49 <ikke> trying to figure out why I cannot resolve infra-pkgs.alpin.pw on the builders
2020-07-15 21:06:51 <ikke> dnsmasq is set to forward to 172.16.8.3
2020-07-15 21:07:15 <ikke> if I ask .8.3 directly, it works
2020-07-16 07:36:23 <ikke> ncopa: usa4 is low on diskspace due to the newedge builders. I already freed up some space, but bound to fill up again
2020-07-16 08:06:13 <ncopa> morning
2020-07-16 08:10:56 <ikke> morning
2020-07-16 08:11:13 <ncopa> im trying to clean up some of my stuff
2020-07-16 08:11:28 <ncopa> but i think we will have a challenge with this
2020-07-16 08:19:10 <ncopa> ikke: we have another arm machine right?
2020-07-16 08:19:21 <ikke> We have usa1 and usa4
2020-07-16 08:19:26 <ikke> usa1 is aarch64
2020-07-16 08:19:38 <ikke> at least, hosts aarch64
2020-07-16 08:20:02 <ncopa> 90G free on that host
2020-07-16 08:22:56 <ncopa> do we have a machine with much diskspace we can use for archving?
2020-07-16 08:23:12 <ncopa> we could archive build-3-[0-8]-a*
2020-07-16 08:24:36 <ncopa> we could ask if those are still used: disaksen-edge-aarch64 hriomar-edge-aarch64 jirutka-edge-aarch64
2020-07-16 09:47:43 <mps> whom I have to ask for x86_64 lxc
2020-07-16 10:29:02 <ncopa> i guess clandmeter
2020-07-16 10:37:15 <mps> ok, thanks
2020-07-16 10:37:20 <mps> clandmeter: ping
2020-07-16 10:51:37 <ncopa> i have deleted build-3-[5-8]-armhf from usa4. I have a local copy if ever needed
2020-07-16 11:16:09 <ikke> 72% / 134G free
2020-07-16 11:16:25 <ikke> (72% used)
2020-07-16 20:08:13 <mps> huh, newedge fails become annoying on #alpine-commits
2020-07-17 05:01:35 <ikke> clandmeter: received an e-mail from linode that most of our hosts will be rebooted in about a week or 3
2020-07-17 05:05:03 <maldridge> very specific timeline
2020-07-17 05:06:33 <ikke> yes
2020-07-17 05:11:48 <ikke> ie, not all at the same time
2020-07-17 05:11:50 <ikke> over a couple of days
2020-07-17 06:25:09 <clandmeter> ikke: ok
2020-07-17 06:25:23 <clandmeter> regarding space, there is one host on linode with 1TB storage
2020-07-17 06:26:07 <ikke> deu5? with a 1tb volume attached?
2020-07-17 06:26:20 <clandmeter> i think so
2020-07-17 06:26:23 <clandmeter> there is only 1
2020-07-17 06:26:30 <ikke> that one then
2020-07-17 06:26:49 <ikke> I think ncopa wants to move distfiles off the builders
2020-07-17 06:26:52 <clandmeter> ncopa: any update on the linode email?
2020-07-17 06:27:12 <clandmeter> ncopa: regarding the interveiw
2020-07-17 06:27:59 <clandmeter> ikke: that was my plan, but i do not have much motivation recently, probably due lack of time.
2020-07-17 06:28:25 <ikke> ok
2020-07-17 06:28:46 <ikke> That was also the host you bootstrapped with ansible, right?
2020-07-17 06:29:04 <clandmeter> depends what you call boostrapping :)
2020-07-17 06:29:12 <ikke> hehe
2020-07-17 06:29:23 <clandmeter> i dont use linode api
2020-07-17 06:29:28 <ikke> yeah, sure
2020-07-17 06:29:43 <clandmeter> actually im not 100% sure
2020-07-17 06:29:46 <clandmeter> it could be
2020-07-17 06:34:23 <clandmeter> ikke: https://gitlab.alpinelinux.org/alpine/infra/ansible/-/blob/master/known_hosts
2020-07-17 07:16:18 <ikke> clandmeter: mps btw requested an x86_64 lxc container for building
2020-07-17 07:40:15 <clandmeter> sure go ahead
2020-07-17 07:43:39 <ikke> alright
2020-07-17 07:46:50 <ncopa> morning
2020-07-17 07:47:19 <ncopa> clandmeter: i missed another email from linode :-( i responded and gave them my phone number
2020-07-17 10:51:14 <ikke> mps: I've created a container for you on nld5
2020-07-17 10:51:40 <ikke> 172.16.4.44 (mps-edge-x86_64.nld5.alpin.pw)
2020-07-17 10:52:23 <ikke> I copied your key from your aarch64 container
2020-07-17 11:55:35 <mps> ikke: thanks
2020-07-17 11:56:01 <mps> do I need some Port in ssh .config
2020-07-17 11:56:35 <ikke> Do you have vpn access/
2020-07-17 11:56:38 <ikke> ?
2020-07-17 12:00:31 <mps> ikke: no
2020-07-17 12:00:40 <ikke> aha, ok
2020-07-17 12:00:56 <ikke> clandmeter: is that something we can arrange for mps, given he's helping us with infra?
2020-07-17 12:02:08 <mps> iirc, clandmeter told me some time ago that he will create VPN for me but I didn't wanted to bother him with this
2020-07-17 12:57:42 <clandmeter> ikke: yes we can
2020-07-17 12:58:08 <clandmeter> Just simply create lxc
2020-07-17 12:58:24 <clandmeter> mps: you don't have vpn yet?
2020-07-17 12:59:21 <clandmeter> Same box as our own container
2020-07-17 13:26:14 <ikke> clandmeter: I created the LXC container
2020-07-17 13:37:59 <mps> clandmeter: no, not yet VPN access
2020-07-17 14:05:44 <clandmeter> OK i will fix it later
2020-07-17 14:05:51 <clandmeter> Remind me plz
2020-07-17 14:07:25 <mps> at what time, approx?
2020-07-17 14:27:25 <ikke> .at 19:00:00 Europe/Amsterdam clandmeter: arrange a VPN profile for mps
2020-07-17 14:27:25 <alpine-meetbot> ikke: Okay, will remind at 2020-07-17 - 18:59:59UTC
2020-07-17 14:32:39 <mps> hah, algitbot knows 'at' command
2020-07-17 14:33:37 <ikke> this is not algitbot
2020-07-17 14:33:43 <ikke> this is alpine-meetbot ;)
2020-07-17 14:34:19 <mps> oh, I'm blind
2020-07-17 16:47:12 <mps> for VPN access OpenVPN is used?
2020-07-17 16:55:52 <ikke> yes
2020-07-17 16:57:04 <mps> no plan to switch to wireguard
2020-07-17 16:57:45 <ikke> Nothing concrete :)
2020-07-17 16:58:52 <mps> didn't used openvpn for more than 10 years
2020-07-17 16:59:43 <mps> have to reread docs to refresh knowledge
2020-07-17 17:00:26 <ikke> You get a plug-and-play ovpn profile
2020-07-17 17:01:35 <mps> you mean ready-made?
2020-07-17 17:01:40 <ikke> yes
2020-07-17 17:01:46 <ikke> it contains all the details you need
2020-07-17 17:01:52 <mps> ah, nice
2020-07-17 17:03:19 <mps> I don't need (use) my private key?
2020-07-17 17:03:36 <ikke> No
2020-07-17 17:03:44 <ikke> One will be generated for you
2020-07-17 17:03:52 <ikke> It's part of the profile
2020-07-17 17:04:02 <mps> ah, simple like these comercial OVPN offers
2020-07-17 17:04:10 <ikke> yeah
2020-07-17 17:04:16 <mps> yes, I know how it works
2020-07-17 17:05:56 <mps> looks like I triggered newedge again to spam us
2020-07-17 17:06:42 <mps> didn't know that 'retry master' will also retry newedge
2020-07-17 17:21:31 <ikke> It's based on git branch
2020-07-17 17:21:41 <ikke> so everything tracking master will be retried
2020-07-17 19:00:00 <alpine-meetbot> ikke: Europe/Amsterdam clandmeter: arrange a VPN profile for mps
2020-07-17 19:00:23 <ikke> :D
2020-07-17 19:01:01 <maldridge> that reads almost like something from the godfather
2020-07-17 19:01:18 <maldridge> "that's a nice VPN profile you've got there.  It would be a real shame if something were to happen to it"
2020-07-17 19:15:19 <ikke> maldridge: :D
2020-07-17 19:39:04 <mps> ikke: it is two hours late
2020-07-17 19:39:51 <ikke> No, it's right on time, just not the time I wanted to
2020-07-17 19:39:58 <ikke> It didn't take the timezone
2020-07-17 19:40:15 <ikke> "alpine-meetbot │ ikke: Okay, will remind at 2020-07-17 - 18:59:59UTC"
2020-07-17 19:40:24 <ikke> 19:00 utc is 21:00 CEST
2020-07-17 19:40:26 <mps> yes, I see, just kidding a little
2020-07-17 19:40:30 <ikke> :)
2020-07-17 19:41:56 <ikke> Ok, you need to concatenate the timezone
2020-07-17 19:48:36 <mps> ikke: it asks me for password
2020-07-17 19:48:56 <ikke> hmm, ok, let me check
2020-07-17 19:49:00 <ikke> you have a ovpn profile now?
2020-07-17 19:49:02 <mps> uh, wait
2020-07-17 19:49:09 <mps> I have to login as root?
2020-07-17 19:49:14 <Cogitri> Yes
2020-07-17 19:49:24 <ikke> yup
2020-07-17 19:49:28 <ikke> I did not create any other user
2020-07-17 19:49:29 <mps> eh, yes, that is
2020-07-17 19:49:29 <Cogitri> You can setup a different user afterwards
2020-07-17 19:49:54 <mps> I forgot that I have to add 'normal' user for builder
2020-07-17 19:50:05 <mps> works fine
2020-07-17 19:50:07 <ikke> Only thing I did is install and enable sshd and add your root key
2020-07-17 19:50:24 <mps> can I access 'my' other containers over this VPN
2020-07-17 19:50:45 <ikke> yes
2020-07-17 19:50:58 <ikke> Everything is connected :)
2020-07-17 19:51:24 <mps> ikke: thanks
2020-07-17 19:51:51 <mps> now I can use even mosh
2020-07-17 19:51:57 <ikke> yes, indeed
2020-07-17 19:52:12 <ikke> Only thing is that mosh has no agent forwarding
2020-07-17 19:52:22 <mps> yes
2020-07-17 19:52:45 <mps> but that is not important
2020-07-17 19:52:59 <ikke> I push from my build containers, so for me it kind of is
2020-07-17 19:53:18 <mps> ah, understand
2020-07-17 19:54:12 <ikke> And do you use something to get a scroll-back buffer?
2020-07-17 19:54:13 <mps> I push from my local machine, don't like to keep private keys on 'remote' containers
2020-07-17 19:54:23 <ikke> Me neither, hence agent forwarding
2020-07-17 19:54:56 <mps> that sounds as good idea, will look at that when I find some time
2020-07-17 19:55:47 <ikke> ofcourse, agent forwarding is not perfectly secure if they can hijack the socket
2020-07-17 19:55:49 <mps> and yes, I run tmux first when login after machines are rebooted
2020-07-17 19:55:55 <ikke> aha ok
2020-07-17 19:56:09 <ikke> I'm already in a tmux session on my local machine, so nesting is not that pleasant
2020-07-17 19:56:43 <mps> heh, I always open new terminal for any remote
2020-07-17 19:57:51 <ikke> I have 13 tmux sesions, most with multiple terminals
2020-07-17 19:58:10 <ikke> Would be a lot of terminal windows :)
2020-07-17 19:58:40 <ikke> And now I can take over this tmux session over ssh from my laptop 
2020-07-17 19:59:03 <mps> you beat me, I have 12 open
2020-07-17 19:59:34 <ikke> heh
2020-07-17 19:59:51 <ikke> Makes it a lot easier to manage alpine infra
2020-07-17 20:00:05 <mps> but awesome wm with key shortcuts is really helpful 
2020-07-17 20:02:30 <ikke> Yes, I use that as well
2020-07-17 20:02:40 <ikke> but it doesn't scale that much
2020-07-17 20:08:19 <mps> not perfect, but I can't find anything better for my habit
2020-07-17 20:13:56 <mps> btw, besides tmux I build st term with scrollback patches, using keyboard shortcuts also
2020-07-17 20:18:49 <ikke> yes, but that does not work once you're in a tmux session :)
2020-07-17 20:19:36 <mps> true, but nice to have
2020-07-17 20:20:35 <ikke> I don't have to patch my terminal to get scrolling :P
2020-07-17 20:21:45 <mps> well, only usable term for me on these chromebook keyboards are those which I patched
2020-07-18 12:22:05 <ikke> clandmeter: seems like dmvpn issue. I have connection with ovpn, but I cannot ping anything else
2020-07-18 12:33:18 <mps> also I can't ping my lxc
2020-07-18 12:33:26 <ikke> yeah, that's how I noticed
2020-07-18 12:34:52 <ikke> ok, fixed
2020-07-18 12:34:59 <ikke> nhrpd on the dmvpn crashed
2020-07-18 12:35:08 <ikke> dmvpn hub
2020-07-18 12:35:35 <mps> yes, it works again. thanks
2020-07-18 12:35:52 <ikke> [9560634.412058] nhrpd[2870]: segfault at 55c6ee781bd0 ip 000055c6ee781bd0 sp 00007ffebf20b958 error 15
2020-07-18 12:36:22 <mps> hm, you discovered bug
2020-07-18 13:08:26 <ikke> ugh, it crashed again
2020-07-18 13:08:40 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/issues/11755
2020-07-18 14:01:26 <mps> time to consider wireguard? ;)
2020-07-18 14:22:40 <ikke> this is dmvpn
2020-07-18 14:22:47 <ikke> wireguard does not offer what dmvpn does
2020-07-18 14:23:38 <ikke> openvpn is only used for users, and openvpn was stable
2020-07-18 16:20:01 <mps> ok, understand
2020-07-18 16:20:45 <ikke> fyi, it crashed again
2020-07-18 16:20:46 <mps> I don't know how complex is alpina infra network
2020-07-18 16:21:23 <mps> ah, can it be run under supervisor
2020-07-18 16:21:32 <ikke> yes
2020-07-18 16:21:34 <ikke> I think so
2020-07-18 16:21:59 <mps> runit or s6
2020-07-18 16:22:14 <ikke> supervisord would suffice
2020-07-18 16:22:38 <ikke> but crashing so often in so little time sure indicates some bigger issue
2020-07-18 16:22:44 <mps> can it send alert in case of crashed supervised process
2020-07-18 16:24:07 <mps> or even gdb to see where and why it crashes
2020-07-18 16:33:23 <ikke> no
2020-07-18 16:35:28 <ikke> can runnit or s6?
2020-07-18 16:36:04 <ikke> monit can send alerts on crash and is just a simple add-on
2020-07-18 16:37:31 <mps> runit run script is shell so it can do a lot of things
2020-07-18 16:40:01 <mps> but I think nhrpd can be run in tmux/screen session under gdb and when it crashes 'bt' can be seen
2020-07-18 16:41:35 <ikke> might not be a bad idea
2020-07-18 16:41:53 <ikke> I saw that nhrpd-dbg was already installed, so apparently it has been crashing before
2020-07-18 16:44:46 <ikke> ok, running it under gdb
2020-07-18 16:44:48 <mps> well, all software have bugs, some less some more
2020-07-18 16:44:57 <ikke> vpn is up
2020-07-18 16:45:05 <mps> ok
2020-07-18 16:45:19 <mps> lets try crash them :)
2020-07-18 16:58:43 <ikke> I don't think it will cost a lot of effort
2020-07-18 18:04:54 <ikke> mps: succes
2020-07-18 18:08:45 <ikke> switched nhrpd to use supervisor-daemon for now
2020-07-18 18:08:50 <ikke> supervise-daemon*
2020-07-18 18:12:30 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/issues/11755#note_103296
2020-07-18 18:24:29 <mps> ok (though I dislike when the servers are run under any supervisor by default, this can hide problems)
2020-07-18 18:25:03 <ikke> yes
2020-07-18 18:25:18 <ikke> In this case, we know there is an issue, and we need to work around it for now
2020-07-18 18:26:56 <mps> can't 'we' (you) in this case enable supervisor locally, by editing conf.d file file or init script
2020-07-18 18:28:31 <ikke> that's exactly what I did
2020-07-18 18:28:46 <mps> ah, nice then
2020-07-18 18:28:49 <ikke> I did not adjust the package, just the init file in place
2020-07-18 18:35:34 <mps> that is ok imo
2020-07-18 18:35:50 <ikke> imo too
2020-07-18 18:36:27 <ikke> if ncopa agrees, I want to add monitoring to that server, the we get actual alerts when the service crashes
2020-07-18 18:36:34 <mps> if nhrpd is run already under some supervisor you will probably didn't notice issue so fast
2020-07-18 18:38:13 <mps> some (but not much complicated) monitoring is always helpful
2020-07-18 18:38:49 <ikke> I've added native openrc service monitoring to the zabbix go agent
2020-07-18 18:39:44 <ikke> https://gitlab.alpinelinux.org/alpine/infra/zabbix-agent2-plugins/-/blob/master/plugins/alpine/openrc.go :)
2020-07-18 18:40:17 <mps> I don't know how it works, but by following this channel I'm sure it will be good because I've seen that you do a lot of things with this zabbix agent
2020-07-18 18:40:59 <ikke> Most monitoring historically has been monitoring network services
2020-07-18 18:41:15 <ikke> Only recently I've started adding internal monitoring
2020-07-18 18:46:57 <mps> I think your approach is good, i.e. monitor what is needed but not 'overmonitor' too much
2020-07-18 18:51:03 <Cogitri> ikke: Have you looked into the repos for alpine-qa-bot already? :)
2020-07-18 18:51:23 <ikke> No, will create them
2020-07-18 18:52:14 <ikke> should we call it aports-qa-bot?
2020-07-18 18:53:08 <Cogitri> Ah sure, sounds good too
2020-07-18 18:56:31 <ikke> Cogitri: done
2020-07-18 18:56:40 <ikke> I make them public as soon as you have pushed some content
2020-07-18 19:07:22 <Cogitri> Thanks!
2020-07-18 19:09:11 <mps> ikke: we didn't waited too long for crash of nhrpd
2020-07-18 19:09:35 <ikke> Nope, it crashes quite soon
2020-07-18 22:26:33 <kunkku> ikke: the backtrace in /var/crash/nhrpd differs slightly from what you posted in gitlab
2020-07-19 07:05:21 <ikke> kunkku: it crashed again afterwards
2020-07-19 07:26:53 <mps> how long supervisor takes time to restart it after crash
2020-07-19 07:27:38 <mps> seems it sleep too long in 'crashed' state
2020-07-19 07:30:24 <ikke> according to the documentation 'immediately' unless you change the restart delay
2020-07-19 07:30:45 <ikke> but I noticed that too
2020-07-19 07:30:59 <ikke> right now it's up for >9h though
2020-07-19 07:32:08 <mps> ah good news, maybe I can finish linux-edge upgrade
2020-07-19 08:07:54 <mps> is it possible to mount file system or dirs between alpine infra containers
2020-07-19 08:08:03 <mps> sshfs for example
2020-07-19 08:08:49 <mps> i.e. 'modprobe fuse' on these machines
2020-07-19 08:54:59 <ikke> clandmeter: Any objections to that?
2020-07-19 09:24:03 <clandmeter> Hi
2020-07-19 09:24:46 <clandmeter> Did kunkku get the bt?
2020-07-19 09:25:20 <clandmeter> mps: between you own containers?
2020-07-19 09:27:44 <mps> clandmeter: yes, but I don't insist on fuse, just asked
2020-07-19 09:28:03 <clandmeter> It's ok for me
2020-07-19 09:29:36 <mps> whatever you think I'm ok
2020-07-19 10:24:23 <kunkku> clandmeter: yes, I have a copy of the core dump
2020-07-19 10:24:39 <kunkku> fabled will also investigate it
2020-07-19 10:41:47 <kunkku> based on the logs, usa5 has stormed the hub with IKE requests at the time of the crash
2020-07-19 10:42:09 <kunkku> which version of the dmvpn package is installed there? I don't have access
2020-07-19 10:44:08 <ikke> dmvpn-1.2.1-r0                          = 1.2.1-r0
2020-07-19 10:58:17 <kunkku> I recommend updating all spokes with dmvpn < 1.2.3
2020-07-19 10:59:32 <kunkku> indeed, there is a race in nhrpd which should be fixed, but ^^^ should recude the likelihood of such a race occuring
2020-07-19 11:00:50 <ikke> 3.11 only has 1.2.1
2020-07-19 11:03:59 <kunkku> 3.11 has the fix backported
2020-07-19 11:04:51 <kunkku> 3.10 too
2020-07-19 11:05:11 <kunkku> just run "apk upgrade"?
2020-07-19 11:06:47 <ikke> ah, I see
2020-07-19 11:06:49 <ikke> -r3
2020-07-19 15:32:19 <ikke> kunkku: ah, I see fabled pushed a fix?
2020-07-19 16:01:32 <kunkku> ikke: yes, it's related to the crash
2020-07-19 16:01:42 <kunkku> we have been investigating it today
2020-07-19 16:02:44 <ikke> kunkku: thanks
2020-07-20 11:26:15 <ikke> Added dmvpn1 to Zabbix
2020-07-20 13:02:03 <ncopa> good! thanks!
2020-07-21 06:24:05 <fcolista> ine-team
2020-07-21 06:24:39 <ikke> :P
2020-07-21 08:18:09 <ncopa> I get "bad gateway" errors from cgit log: https://git.alpinelinux.org/aports/log/main/redis
2020-07-21 08:18:28 <ikke> timeout?
2020-07-21 08:18:35 <ikke> Seems to take quite some time
2020-07-21 08:19:39 <ikke> ncopa: https://gitlab.alpinelinux.org/alpine/aports/-/commits/master/main%2Fredis :P
2020-07-21 14:04:17 <ncopa> heh... i rsynced build-newedge-armhf to usa4.alpinelinux.org which apparently is not the same machine as usa4.alpin.pw
2020-07-21 14:04:30 <ikke> heh
2020-07-21 14:04:33 <ikke> fun
2020-07-21 14:04:50 <ncopa> seems like usa4.alpinelinux.org is usa5-dev1
2020-07-21 14:05:50 <ikke> strange, both take me to the same machine
2020-07-21 14:05:58 <ncopa> man... im so stupid
2020-07-21 14:06:35 <ncopa> correction: I rsynced it to usa5.alpinelinux.org which is not same machine as usa4.alpin.pw
2020-07-21 14:06:46 <ncopa> which I guess is a good thing :)
2020-07-21 14:06:46 <ikke> Yes, I can imagine that
2020-07-22 14:49:07 <clandmeter> we have an email from IBM
2020-07-22 14:50:36 <ikke> Not me
2020-07-22 14:50:57 <clandmeter> question is if we want to move to a cloud based solution instead of baremetal
2020-07-22 14:51:38 <ikke> is that for s390x?
2020-07-22 18:39:21 <clandmeter> ppc
2020-07-22 18:41:32 <clandmeter> ikke: which platform did we have to most issues with?
2020-07-22 18:41:41 <clandmeter> s390x?
2020-07-22 19:12:11 <ikke> I think ppc64le a little bit more
2020-07-22 19:12:20 <ikke> I haven't seen a lot of stuck jobs on s390x lately
2020-07-22 19:12:33 <ikke> but ppc64le still has network issues when having to upload things back
2020-07-22 19:24:32 <clandmeter> ikke: I added you to the thread
2020-07-22 19:35:19 <ikke> thanks
2020-07-23 10:38:44 <ncopa> maybe ask what hardware we get from cloud based solution. eg number of cpu's, RAM, if we can get power9 (insetad of current power8)
2020-07-23 20:28:54 <ikke> clandmeter: any particular reason the armv7 ci VM is running an old kernel? 4.19.75-0-vanilla
2020-07-24 05:27:04 <clandmeter> I don't think so
2020-07-24 05:28:26 <maldridge> somewhere I still have the source code for a horrifically bad daemon I wrote called "apr" (any particular reason).  It worked to document weird infrastructure problems at a past job
2020-07-24 05:29:50 <ikke> heh
2020-07-24 05:29:55 <ikke> apr it was a daemon? :P
2020-07-24 05:30:09 <maldridge> yes, it served an incredibly shitty web page
2020-07-24 05:30:19 <maldridge> though my team had always joked about rewritting it as a gopher service
2020-07-24 05:30:25 <ikke> ha
2020-07-24 10:07:02 <tmhoang> ikke: it seems there will be a 1 hour maintenaince downtime (upgrading the hypervisor) for the s390x machines today (friday) around 10-11am ET time . I mmissed that early this week ... 
2020-07-24 10:07:22 <ikke> aha, ok. Thanks for the heads-up
2020-07-24 10:07:48 <ikke> That should be around 4pm CEST I gather
2020-07-24 10:09:21 <tmhoang> I lost track of saving time thing ... it's been 4-5 hours difference between CET and ET I'd say 
2020-07-24 10:10:26 <ikke> 6 hours
2020-07-24 10:11:46 <ikke> ET is UTC-4 CEST is UTC+2
2020-07-24 10:15:06 <tmhoang> +1
2020-07-24 11:55:25 <Ariadne> re ppc64[le], we almost have colo up, and could probably sponsor some machines with power9 cpu
2020-07-24 11:56:41 <Ariadne> but there is not much difference between power8 and power9
2020-07-24 12:08:27 <ncopa> tmhoang: thank you
2020-07-24 12:35:51 <ncopa> dl-cdn has ~455 req/sec
2020-07-24 12:35:58 <ncopa> thats pretty heavy...
2020-07-26 15:42:16 <Cogitri> ikke: I think the aports-qa-bot docker repo can't upload things yet: https://gitlab.alpinelinux.org/alpine/infra/docker/aports-qa-bot/-/jobs/172308
2020-07-26 15:42:30 <ikke> ah, right
2020-07-26 15:42:47 <ikke> need to create a repo on docker hub
2020-07-26 15:43:05 <Cogitri> Also, I'd somehow have to pass secrets to aports-qa-bot (the secret token for the API so the bot can do things and the secret token Gitlab uses for webhook auth), how would I best do that?
2020-07-26 15:43:34 <Cogitri> Currently I just have a config file that is in the dockercompose repo (not pushed yet), but then the secrets are leaked in the repo (and I guess the repo would need to be private then?)
2020-07-26 15:44:09 <ikke> We typically use .env files in the docker-compose project
2020-07-26 15:46:48 <ikke> We are also looking into using hashicorp vault
2020-07-26 15:53:23 <Cogitri> Alright, I guess I'll just put it in the config then, since that'll effectively be the same as having iy in an .env file
2020-07-26 15:58:49 <ikke> right, but note that we do not commit these secrets
2020-07-26 15:59:23 <Cogitri> Oh, how does that work then?
2020-07-26 15:59:34 <ikke> We create these files when we deploy the project
2020-07-26 15:59:50 <ikke> Eventually, we use something like ansible to automate that
2020-07-26 15:59:54 <Cogitri> Ah okay, I thought that the docker compose stuff was auto deployed so it'd just use what ever is in the repo
2020-07-26 15:59:55 <Cogitri> Ah
2020-07-26 16:00:20 <ikke> no, we just checkout the repo and run docker-compose up -d :P
2020-07-26 16:00:48 <Cogitri> Oh :D 
2020-07-26 16:01:06 <ikke> We do not have something like kubernetes
2020-07-26 16:01:06 <Cogitri> I'll just commit the config file with the secret tokens not filled in then
2020-07-26 17:47:29 <ikke> Cogitri: ok, it should now be able to push the image
2020-07-26 17:49:49 <eth01> hi all
2020-07-26 17:51:17 <ikke> hi
2020-07-26 17:56:15 <eth01> my project would be happy to give the alpine project some hosting
2020-07-26 17:56:39 <eth01> compute power
2020-07-26 17:59:45 <ikke> What kind of computing power are we talking about?
2020-07-26 18:00:38 <eth01> virtual machines with very decent specs, globally. unmetered links.
2020-07-26 18:00:45 <eth01> ipv6/ipv4 
2020-07-26 18:09:08 <Cogitri> ikke: seems like the push worked, thanks
2020-07-26 18:47:22 <clandmeter> eth01: can you share your project?
2020-07-26 18:48:04 <eth01> fosshost.org
2020-07-26 18:50:57 <clandmeter> I didn't know about fosshost
2020-07-26 18:53:37 <ikke> clandmeter: in case you might have missed it, the armv7 edge builder is now upgraded to 3.12 with linux-lts
2020-07-26 18:53:47 <ikke> I mean
2020-07-26 18:53:49 <ikke> CI host
2020-07-26 18:54:07 <ikke> the VM
2020-07-26 18:54:14 <clandmeter> Right
2020-07-26 18:54:17 <clandmeter> Thx
2020-07-26 18:54:33 <clandmeter> I guess it was a simple upgrade
2020-07-26 18:54:50 <ikke> indeed
2020-07-26 18:55:15 <clandmeter> Adv of running it in qemu
2020-07-26 18:56:10 <ikke> There is still some weird issue when building armv7 gitlab runner images due to the musl upgrade
2020-07-26 19:15:40 <eth01> Not many projects know about us...
2020-07-26 21:14:53 <mps> eth01: do you offer it for alpine official or also for alpine developers
2020-07-26 21:16:38 <maldridge> eth01: yeah, coming from void I can say that I've only seen passing references to you
2020-07-26 21:20:11 <eth01> mps: open source projects / developers unofficial or official
2020-07-26 21:21:24 <eth01> At the moment we have some spare capacity and it’s sat there not being utilised, so I’m trying to push the word out, to projects I think will benefit.
2020-07-26 21:21:52 <eth01> maldridge: cool, do you know a project we help?
2020-07-26 21:23:37 <mps> eth01: I'm thinking to make alpine unofficial images which could be 'flashed' to mmc cards for some arm32 boards, but can't keep them on my local web servers because of slow link
2020-07-26 21:24:16 <mps> and I'm not sure it is good idea to put them on alpine infrastructure
2020-07-26 21:24:39 <mps> so, some third party hosting could be good idea
2020-07-26 21:24:59 <maldridge> mps: any particular board you're targeting?
2020-07-26 21:27:37 <eth01> mps: sounds reasonable, we could help with that
2020-07-26 21:28:23 <eth01> Fosshost.org/platform
2020-07-26 21:28:34 <eth01> Here are our locations and specs in each location
2020-07-26 21:30:24 <mps> maldridge: for now I have few, bananapi, lamobo r1, and exynos 5800 (peach pi chromebook)
2020-07-26 21:30:39 <maldridge> interesting
2020-07-26 21:30:42 <mps> maybe cubie board 
2020-07-26 21:32:05 <mps> and (if I found time) maybe RPi zero with mainline kernel
2020-07-26 21:32:51 <maldridge> now that would be something I'm interested in
2020-07-26 21:33:11 <maldridge> of course I say this with a number of rpi zero's sitting powered off not doing anything
2020-07-26 21:35:22 <mps> I started to work on RPi zero with mainline kernels (5.6 and 5.7) but don't have much time for this to finish these days
2020-07-26 21:35:35 <mps> I hope I will have more time in a month
2020-07-27 07:20:51 <Ariadne> those host servers are quite old, and vulnerable to meltdown
2020-07-27 07:23:57 <maldridge> the proxmox ones?
2020-07-27 07:26:26 <Ariadne> yes the ones eth01 is talking about
2020-07-27 07:26:59 <maldridge> I'd be more concerned about the quarter million lines of perl that make up proxmox rather than the kernel it runs on
2020-07-27 07:27:23 <Ariadne> I'm talking about the cpu
2020-07-27 07:27:25 <Ariadne> :)
2020-07-27 07:28:07 <Ariadne> Xeon 5620 is from 2010 for example
2020-07-27 07:28:50 <Ariadne> and meltdown is very relevant in a multi-user vps environment
2020-07-27 07:29:25 <Ariadne> performance impact of the mitigations is significant
2020-07-27 07:29:45 <mps> I'm concerned with any cpu I didn't made, and I didn't made any ;)
2020-07-27 07:30:21 <Ariadne> sure but some CPUs are more broken than others :p
2020-07-27 07:30:40 <maldridge> to be clear, its all just sand that's been taught to think
2020-07-27 07:31:20 <ikke> maldridge: reminds me of this: https://xkcd.com/505/
2020-07-27 07:31:20 <alpine-meetbot> https://xkcd.com/505 | A Bunch of Rocks | Alt-text: I call Rule 34 on Wolfram's Rule 34.
2020-07-27 07:31:50 <maldridge> yep
2020-07-27 07:33:56 <Ariadne> what kind of virt is it anyway?  kvm?
2020-07-27 07:34:56 <maldridge> yeah proxmox is all kvm based
2020-07-27 07:35:24 <maldridge> its a 1/4 million lines of perl on top of libvirt
2020-07-27 07:35:43 <Ariadne> yuck :)
2020-07-27 07:36:53 <maldridge> my biggest problem with it is that its not bootstrappable
2020-07-27 07:41:20 <maldridge> if you want small scale virtualization, kimchi is quite nice, if not particularly easy to run
2020-07-27 08:58:57 <eth01> We can run containers if you’d prefer that which uses LXC
2020-07-27 09:00:33 <Ariadne> well the point I was making is that alpine itself has much newer hardware
2020-07-27 09:00:57 <Ariadne> for example we have amd epyc 128 thread builder for x86
2020-07-27 09:01:39 <Ariadne> what Foss devs need more than x86 hosting is non-x86 hosting
2020-07-27 09:05:39 <eth01> Ariadne: we cant offer that at the minute but I am aware that it is something that projects want so I am looking at different options
2020-07-27 11:47:36 <tmhoang> it seems when we moved git.a.o to gitlab (instead of gitloite ?), full git clone from git.a.o seems slower. Reading git.a.o/log also seems slower and sometimes (most of the time) I get 500. Maybe the infra already aware of this. 
2020-07-27 11:48:16 <ikke> git.a.o/log is still cgit
2020-07-27 11:48:29 <tmhoang> agreed, but the backend is gitlab ? 
2020-07-27 11:48:31 <ikke> ncopa did mention hey got 500s there from time to time
2020-07-27 11:48:45 <ikke> cgit is it's own backend
2020-07-27 11:49:00 <ikke> it's not directly coupled to gitlab
2020-07-27 11:49:06 <tmhoang> interesting
2020-07-27 11:49:31 <ikke> We synchronize from gitlab.a.o to git.a.o
2020-07-27 11:49:48 <ikke> git.a.o is still gitolite
2020-07-27 11:50:04 <tmhoang> more interesting, I wonder what caused the slow 
2020-07-27 11:50:05 <ikke> We did move git.a.o however
2020-07-27 12:09:54 <ncopa> i wonder if cache is enabled in our cgit
2020-07-27 12:11:59 <ikke> where is that enabled?
2020-07-27 12:13:29 <ikke> cache-size
2020-07-27 12:13:31 <ikke> https://linux.die.net/man/5/cgitrc
2020-07-27 12:13:36 <ikke> which defaults to 0
2020-07-27 12:14:05 <ikke> ncopa: So I guess you are right
2020-07-27 12:15:06 <ikke> /var/cache/cgit is empty
2020-07-27 12:23:51 <ikke> ncopa: enabled caching
2020-07-27 12:23:59 <ikke> 1000 entries, no idea how much we should do
2020-07-27 13:16:34 <ncopa> lets try 1000 and see how that goes
2020-07-27 13:17:56 <ikke> I increased it even to 10K, but I don't see the load decreasing
2020-07-27 13:18:00 <tmhoang> nice !
2020-07-27 13:18:08 <ikke> the load of that host is ~18, with 4 cores
2020-07-27 13:19:11 <ikke> it's at 2800 cache items now
2020-07-27 13:19:18 <ikke> almost 2900
2020-07-27 13:19:40 <tmhoang> I'm pulling :) 
2020-07-27 13:21:16 <ikke> We still get a lot of traffec on secdb
2020-07-27 13:21:21 <ikke> I thought that clandmeter redirected that
2020-07-27 13:21:55 <clandmeter> cache will not improve cgit
2020-07-27 13:22:03 <ikke> ok
2020-07-27 13:22:10 <ikke> Which I noticed
2020-07-27 13:22:13 <clandmeter> the problem is that the host io is not that fast
2020-07-27 13:22:30 <clandmeter> i have a hack 
2020-07-27 13:22:34 <clandmeter> but its not live 
2020-07-27 13:22:46 <clandmeter> there is a tmpfs mount on the host
2020-07-27 13:22:56 <clandmeter> which we could use as git storage
2020-07-27 13:23:09 <ikke> 8G of memory
2020-07-27 13:23:41 <clandmeter> but it would need to clone/update all repos on restart
2020-07-27 13:24:23 <ikke> nod
2020-07-27 13:24:33 <clandmeter> i think i did a test and it was better but for sure not perfect.
2020-07-27 13:24:50 <clandmeter> git log is fundamentally slow
2020-07-27 13:25:06 <clandmeter> except if you offload it to some different storage
2020-07-27 13:25:11 <clandmeter> as in db 
2020-07-27 13:25:25 <clandmeter> thats what gitlab does i think
2020-07-27 13:25:35 <clandmeter> maybe redis or similar
2020-07-27 13:25:55 <tmhoang> maybe an init.d service would help on reboot 
2020-07-27 13:26:03 <tmhoang> if the cost of tmpfs on ram is ok 
2020-07-27 13:26:28 <clandmeter> its docker, you can add logic to entrypoint
2020-07-27 13:26:50 <clandmeter> but on a system reboot you will need to know which repos to clone.
2020-07-27 13:27:07 <clandmeter> so you need to define a list or grab the list from gitlab.
2020-07-27 13:27:15 <tmhoang> we don't have too much repos, dont we ? 
2020-07-27 13:27:18 <tmhoang> < 10 
2020-07-27 13:27:33 <tmhoang> aports seems to be the most cloned on a daily basis, ie. by CI 
2020-07-27 13:27:49 <clandmeter> CI should clone from gitlab
2020-07-27 13:27:55 <tmhoang> +1
2020-07-27 13:27:57 <ikke> it does
2020-07-27 13:28:13 <clandmeter> the list is small
2020-07-27 13:28:13 <ikke> including the builders
2020-07-27 13:28:20 <clandmeter> but its nice if its all automatic
2020-07-27 13:28:46 <clandmeter> the api can provide a list i think
2020-07-27 13:28:57 <clandmeter> i did some logic before, i just lost interest shortly after.
2020-07-27 13:29:32 <tmhoang> ah I think git@gitlab.alpinelinux.org:alpine/aports.git is just fine. I'll start doing that then. Just used to git://git.alpinelinux.org/aports
2020-07-27 13:29:49 <clandmeter> git.a.o is just bw compat
2020-07-27 13:29:50 <ikke> yeah, you should just be able to fetch from gitlab
2020-07-27 13:29:52 <tmhoang> remembered that url by heart 
2020-07-27 13:30:07 <ikke> gitlab has a beastly server backing it
2020-07-27 13:30:08 <clandmeter> but its slow sometimes to clone also
2020-07-27 13:30:11 <clandmeter> because its hammered 
2020-07-27 13:30:55 <ikke> clandmeter: still mostly secdb?
2020-07-27 13:31:02 <ikke> I see a lot of 301 replies
2020-07-27 13:31:16 <clandmeter> the person that implemented the logic in those security scanners didn't think about the consequences for our server.
2020-07-27 13:32:05 <clandmeter> ikke: yes i think 99% is sec scanner traffic
2020-07-27 13:34:10 <clandmeter> we redirect from nginx as i couldnt find the proper config for traefik
2020-07-27 13:34:27 <ikke> Perhaps we should take more effective measures..
2020-07-27 13:34:53 <ikke> clandmeter: afaik, upstream already changed the url they used, right?
2020-07-27 13:35:01 <clandmeter> iptables based on req/m
2020-07-27 13:35:25 <clandmeter> im not sure
2020-07-27 13:35:33 <clandmeter> i htink its the claire scanner that does it.
2020-07-27 13:35:45 <clandmeter> or similar name
2020-07-27 13:36:03 <ikke> http4s-blaze
2020-07-27 13:36:28 <clandmeter> thats the agent?
2020-07-27 13:36:31 <ikke> One of them
2020-07-27 13:36:39 <clandmeter> scala
2020-07-27 13:36:40 <ikke> but most it seems to be diretly cloning via git
2020-07-27 13:36:51 <ikke> git/2.17.0
2020-07-27 13:36:59 <ikke> git/2.11.2
2020-07-27 13:38:27 <ikke> some kind of tarpitting :P
2020-07-27 13:54:19 <clandmeter> if you have good ideas, please share :)
2020-07-27 13:54:28 <clandmeter> or better, implement ;-)
2020-07-27 15:30:41 <Cogitri> ```
2020-07-27 15:30:41 <Cogitri> kex_exchange_identification: read: Connection reset by peer
2020-07-27 15:30:41 <Cogitri> Connection reset by 172.16.4.39 port 22
2020-07-27 15:30:41 <Cogitri> ```
2020-07-27 15:30:41 <Cogitri> I think I just grilled my x86_64 container?
2020-07-27 15:31:05 <ikke> hmm\
2020-07-27 15:31:15 <ikke> Has it been a while since you upgraded it?
2020-07-27 15:31:33 <Cogitri> Yes, I just upgraded it now and logged out
2020-07-27 15:31:40 <Cogitri> Oh right...I should've restarted sshd after that
2020-07-27 15:31:50 <ikke> aha, will do it now
2020-07-27 15:31:58 <Cogitri> Thanks
2020-07-27 15:32:02 <ikke> done
2020-07-27 15:32:10 <Cogitri> Thanks :) 
2020-07-27 15:32:24 <Cogitri> Works
2020-07-27 15:32:46 <ikke> no worry
2020-07-27 17:48:42 <maldridge> clandmeter: what redirect were you trying to set, I can check my traefik notes in a bit
2020-07-27 17:50:47 <ikke> maldridge: traffic to one git repository over cgit redirecting to github
2020-07-27 17:51:06 <ikke> GET /cgit/alpine-secdb/info/refs?service=git-upload-pack HTTP/1.1" 301
2020-07-27 18:01:46 <maldridge> you want to proxy to github?
2020-07-27 18:02:38 <ikke> Just redirect traffic
2020-07-27 18:03:01 <ikke> there are hordes of users cloning alpine-secdb
2020-07-27 18:03:24 <ikke> which is basicaly causing the majority of the load on git.alpinelinux.org
2020-07-27 18:03:46 <ikke> But, we kind of already are doing that
2020-07-27 18:12:34 <maldridge> oh, I see now
2020-07-27 18:12:56 <maldridge> you'd need to load a redirect middleware, I'll have to dig through my notes to see how to do that in a bit
2020-07-28 07:08:35 <clandmeter> maldridge: if  you have an example that actually works, please share. 
2020-07-28 07:08:41 <clandmeter> i gave up on it long time ago.
2020-07-28 07:10:38 <maldridge> clandmeter: I'll take a look, I've been trying to get void to run in clouds and writing a crappy version of cloud-init for the last few days, so haven't had much time for searching my archives yet 
2020-07-28 07:11:57 <clandmeter> traefik.http.middlewares.cgit-secdb.redirectregex.regex
2020-07-28 07:12:02 <clandmeter> its somet hing like that 
2020-07-28 07:12:23 <maldridge> I think you can likely just do a PathPrefix and a PathStrip
2020-07-28 07:12:32 <maldridge> regex shouldn't be necessary here
2020-07-28 07:13:16 <clandmeter> how would you handle the replacement?
2020-07-28 07:13:43 <maldridge> well, I'm more of an ass than  you, I'd return a 410 for the old path
2020-07-28 07:13:57 <ikke> heh
2020-07-28 07:14:08 <clandmeter> i see that you dont know me very well
2020-07-28 07:14:11 <clandmeter> :p
2020-07-28 07:14:16 <maldridge> if I needed to keep both running, I'd probably stick varnish in front of it and proxy through to github
2020-07-28 07:14:26 <maldridge> try to serve as much as possible from cache
2020-07-28 07:14:36 <clandmeter> i dont want to serve anything
2020-07-28 07:14:39 <clandmeter> its too much
2020-07-28 07:14:50 <clandmeter> even redirecting is causing high cpu
2020-07-28 07:14:51 <ikke> shouldn't a redirect be much lighter?
2020-07-28 07:15:02 <maldridge> its still a socket and a couple of syscalls
2020-07-28 07:15:09 <maldridge> and if people are being dumb and not caching the db
2020-07-28 07:15:21 <clandmeter> what i like is to set a limit and then block the address for 1h
2020-07-28 07:15:49 <clandmeter> via fw
2020-07-28 07:16:19 <maldridge> you could potentially do that via fail2ban right?
2020-07-28 07:16:22 <clandmeter> too bad i cant see the header with iptables
2020-07-28 07:16:47 <clandmeter> i guess i can
2020-07-28 07:16:57 <clandmeter> i didnt spend too much time on it.
2020-07-28 07:17:30 <ikke> iptables doesn't do dpi :)
2020-07-28 07:20:54 <clandmeter> maldridge: this is what i found in the compose file: https://tpaste.us/xk0w
2020-07-28 07:21:09 <clandmeter> not sure what state that is in. long time ago..
2020-07-28 07:21:19 <maldridge> I think most of your cpu time is probably spent on regex
2020-07-28 07:21:23 <maldridge> its not particularly cheap
2020-07-28 07:21:45 <ikke> Noticed something similar with graylog using regexp to separate streams
2020-07-28 07:21:56 <ikke> they cost a lot of CPU time indeed
2020-07-28 07:22:56 <maldridge> that, among other reasons, is why I don't use graylog
2020-07-28 07:23:58 <clandmeter> pathprefix is gone in 2.2?
2020-07-28 07:24:34 <maldridge> hmm I didn't think it was
2020-07-28 07:29:35 <clandmeter> looking at the docs, i have no clue how to return any http code based on path.
2020-07-28 07:31:20 <Cogitri> Is b.a.o down right now?
2020-07-28 07:31:29 <clandmeter> b?
2020-07-28 07:31:39 <ikke> build
2020-07-28 07:32:02 <clandmeter> loads for me
2020-07-28 07:32:13 <Cogitri> Seems to either not load or say that msg.a.o is unavailable
2020-07-28 07:32:23 <clandmeter> says connected for me
2020-07-28 07:32:26 <Cogitri> Ah, now it loads :)
2020-07-28 07:33:09 <ikke> I saw the same
2020-07-28 08:41:43 <Cogitri> Btw, can the aports-qa-bot be setup so it's only reachable for the gitlab container and not the public? Currently the only authentication it has to check webhooks are really coming from Gitlab is the HTTP header with the secret token Gitlab sends (but that might be sufficient too as long as we choose a secure token)
2020-07-28 11:23:17 <Cogitri> And do we have a test instance for our gitlab/can we do a temporary copy of the aports repo to make sure nothing blows up when we enable the bot? I have pretty good test coverage on alpine-qa-bot but I didn't have a chance to test on such a huge repo yet
2020-07-28 11:25:29 <ikke> Cogitri: We usually spin up a test instance when we upgrade gitlab
2020-07-28 11:25:35 <ikke> which is a complete copy
2020-07-28 11:26:14 <Cogitri> Ah nice
2020-07-28 11:26:45 <ikke> if clandmeter agrees, we can start looking into the upgrade to 13.1
2020-07-28 11:27:45 <clandmeter> ok here
2020-07-28 11:40:49 <ikke> Cogitri: That's usually how webhooks work right? Over the internet with a secure token to authenticate?
2020-07-28 11:42:50 <Cogitri> Yup, I don't think Gitlab has another way of authentication
2020-07-28 11:43:18 <Cogitri> I just thought that if they're on the same internal net anyway we don't have to expose it to the internet :)
2020-07-28 11:46:21 <ikke> we have dedicated servers for gitlab
2020-07-28 11:46:56 <ikke> We would most likely host these containers on a different server
2020-07-28 11:47:38 <ikke> and gitlab is not (yet) part of the dmvpn
2020-07-28 11:48:52 <clandmeter> i dont see the issue here?
2020-07-28 11:48:52 <Cogitri> Ah okie. Then it shouldn't be a problem to host it publicly either
2020-07-28 11:49:03 <ikke> Me neither
2020-07-28 11:49:07 <clandmeter> api+token is fine
2020-07-28 11:49:13 <Cogitri> 👍
2020-07-28 11:49:13 <clandmeter> its what all bots and such do
2020-07-28 11:49:33 <clandmeter> over https ofc :)
2020-07-28 11:49:56 <Cogitri> If the reverse proxy that's in front of the bot does https, then sure :D
2020-07-28 11:50:02 <ikke> it does
2020-07-28 11:50:31 <clandmeter> gitlab talks to the bot?
2020-07-28 11:50:42 <ikke> webhooks
2020-07-28 11:51:00 <ikke> The bot needs to know when to respond
2020-07-28 11:51:05 <clandmeter> right
2020-07-28 11:51:15 <clandmeter> so it also has a token itself
2020-07-28 11:51:23 <ikke> a shared secret
2020-07-28 11:51:35 <ikke> which what tokens usually are :P
2020-07-28 11:54:14 <Cogitri> Yup, when setting up the webhook you can set up the token (I wonder why gitlab doesn't auto-generates it, but oh well)
2020-07-28 11:54:46 <Cogitri> Right now the bot only handles merge request events (since that's the only interesting hook for us AFAICS)
2020-07-28 11:55:00 <Cogitri> So the other event types don't have to be enabled for the webhook
2020-07-28 11:55:07 <ikke> Cogitri: maybe because the assumption is that the receiving side is determining the token
2020-07-28 11:55:43 <Cogitri> Ah, yes, that makes sense
2020-07-28 11:58:14 <clandmeter> maybe there is a token policy or the other side
2020-07-28 11:58:35 <ikke> Or it's just generated 
2020-07-28 12:20:07 <Cogitri> Anyway, just generated the docker container, I think the bot is ready for testing once we have a gitlab instance up
2020-07-28 12:22:25 <Cogitri> Actually, maybe we should have `G_MESSAGES_DEBUG=all` specified for the initial run to get all the debug info we can in case things go wrong
2020-07-28 12:22:52 <ikke> We can add that to the .env file
2020-07-28 12:40:20 <Cogitri> Yup, that'd be good, shouldn't be required once things work
2020-07-31 08:41:05 <ncopa> morning
2020-07-31 08:41:15 <ncopa> ikke: did we have a machine we could use for distfiles?
2020-07-31 08:50:08 <mps> gitlab.a.o web interface is sooooo slow
2020-07-31 08:53:06 <MY-R> mps, any example where? :D
2020-07-31 08:54:06 <MY-R> or just in general when switching/pushing buttons? :P
2020-07-31 08:55:55 <mps> in general
2020-07-31 08:56:08 <mps> heavy JS I think
2020-07-31 08:56:35 <mps> but maybe server is overloaded for about two days
2020-07-31 08:58:02 <MY-R> exactly, another reason why I wouldn't be abble to use ARM until those get at least strong single thread or working good enough GPU acceleration 
2020-07-31 08:59:11 <MY-R> working god for me, longer time ago it could notice that was slow but now is snappy
2020-07-31 08:59:45 <ikke> mps: gitlab webif seems quite fast for me
2020-07-31 09:02:00 <MY-R> mps, if you using firefox then you can test "webrender" stuff so most of it is done by GPU, is amazing fast (but even with radeon it spit out lot of weird errors in dmesg)
2020-07-31 09:03:16 <mps> MY-R: it worked 'bearable' till few days ago on same machine and I didn't upgrade anything in meantime, nor changed configuration
2020-07-31 09:03:56 <MY-R> gfx.webrender.all=true and gfx.webrender.enabled=true but better save all current projects ;)
2020-07-31 09:04:32 <MY-R> maybe some JS stuff from CDN's were updated
2020-07-31 09:05:41 <mps> this is most probable
2020-07-31 09:06:10 <mps> something which interfere with dark reader plugin
2020-07-31 09:07:25 <MY-R> oh ye, with that plugin I noticed a huge slow down (more like few seconds delays) that was the reason why uninstalled it
2020-07-31 09:09:29 <MY-R> 
ACTION using this now: https://github.com/m-khvoinitsky/dark-background-light-text-extension

2020-07-31 09:20:31 <mps> MY-R: thanks, but I used this earlier, before switch to dark reader
2020-07-31 09:38:31 <MY-R> :<
2020-07-31 09:39:03 <mps> MY-R: switched back to 'Aa' and disabled DR, and now it gitlab is more responsive
2020-07-31 09:39:25 <mps> it is*
2020-07-31 09:41:15 <MY-R> DR was working for me nice maybe 3-4 weeks ago but in that time many things could change
2020-07-31 09:43:16 <mps> it worked for me till few days ago. and it looks better than 'Aa'
2020-07-31 09:43:49 <ncopa> i am the one who took down the x86 builders. i am moving them to new machine
2020-07-31 09:47:48 <mps> waiting for someone to package https://github.com/MichaelMure/git-bug
2020-07-31 10:12:47 <ncopa> waiting for someone to package https://github.com/weaveworks/footloose
2020-07-31 10:13:27 <ikke> How does that compare to lxc?
2020-07-31 10:14:50 <ncopa> gives you lxc-like containers but with docker
2020-07-31 10:15:27 <ncopa> and let you docker manage the port forwards
2020-07-31 10:15:36 <ikke> aha ok, sounds a bit like dokken