2020-11-02 06:59:26 <ncopa> morning
2020-11-02 06:59:55 <ncopa> I have cleaned up the s390x dev machine (usa2-dev) a bit so there should be space for a container for Ariadne
2020-11-02 07:29:12 <ikke> z(y)
2020-11-02 08:09:45 <Ariadne> thanks ncopa.
2020-11-02 08:12:15 <ikke> For some reason lxc-create complains about the arch not being supported
2020-11-02 08:15:36 <ikke> That seems to come from our own template
2020-11-02 08:55:20 <ikke> ncopa: the lxc-templates-legacy-alpine script does not support s390x. How should we create containers there?
2020-11-02 09:47:38 <ikke> clandmeter: apparently travis is adjusting it's pricing model, which also affects foss projects
2020-11-02 09:49:01 <ikke> clandmeter: perhaps we could just move it to our gitlab instance, and make sure we have a way to build / publish images outside of gitlab if necessary
2020-11-02 09:54:53 <ncopa> ikke: i thikn we should fix the template and backport
2020-11-02 09:55:32 <ikke> ncopa: alright
2020-11-02 09:55:58 <ncopa> hum
2020-11-02 09:56:05 <ncopa> lxc 4 does not haev those template
2020-11-02 09:56:20 <ikke> I think that's the reason we have a legacy template package, right?
2020-11-02 09:56:40 <ncopa> oh, ok
2020-11-02 09:57:09 <ikke> https://pkgs.alpinelinux.org/packages?name=lxc-templates-legacy*&branch=edge
2020-11-02 10:03:01 <ncopa> ikke: i pushed an untested update
2020-11-02 10:03:08 <ikke> ncopa: ok, thanks
2020-11-02 10:03:16 <ikke> I'll add it pinned from edge first
2020-11-02 10:03:35 <ncopa> and if that works we can backport it to 3.12 and maybe 3.11
2020-11-02 10:04:26 <ikke> We need to upgrade our infra to 3.12
2020-11-02 10:05:05 <ikke> Hmm, strange, even with 10 characters in the hash, cgit says invalid object id
2020-11-02 10:07:05 <ikke> Most of our infra is still on 3.10
2020-11-02 10:22:29 <ikke> ncopa: seems it's working now
2020-11-02 10:25:27 <ikke> Ariadne: Which ssh keys should I use? one of these https://gitlab.alpinelinux.org/kaniini.keys
2020-11-02 11:10:34 <ncopa> 👍
2020-11-02 12:15:50 <ikke> clandmeter: checking gitlab logs, but can't really find any error message
2020-11-02 12:55:34 <clandmeter> ikke: ?
2020-11-02 12:56:41 <ikke> clandmeter: On our gitlab-test instance, was looking for errors regarding the API issues
2020-11-02 13:07:01 <clandmeter> K
2020-11-02 15:42:33 <Ariadne> ikke: yes
2020-11-02 15:42:50 <ikke> Do you already have VPN access?
2020-11-02 15:42:55 <Ariadne> ikke: no :)
2020-11-02 15:43:08 <ikke> oh, hmm
2020-11-02 15:43:22 <Ariadne> i think i did a long time ago but have lost the openvpn stuff
2020-11-02 15:48:40 <ikke> can try root@usa2.alpinelinux.org -p 22008
2020-11-02 15:59:04 <Ariadne> that container is a bit faster clock wise
2020-11-02 15:59:13 <Ariadne> cool
2020-11-02 15:59:25 <Ariadne> now, once i get the cross built rust, i can supply you with a rebuilt rust
2020-11-02 15:59:40 <Ariadne> and we can rebuild from that rust on the builder
2020-11-02 16:00:22 <Ariadne> of course rust developers have not given me an answer yet on how to proceed -- grr :)
2020-11-02 16:02:18 <mps> heh, like my experience with rust devs when I tried to make it for aarch64 and armv7
2020-11-02 16:07:58 <Ariadne> "just drop in our zulip, we'll help you!" they said
2020-11-02 16:20:55 <ikke> https://xkcd.com/1254/
2020-11-02 16:20:55 <alpine-meetbot> https://xkcd.com/1254 | Preferred Chat System | Alt-text: If you call my regular number, it just goes to my pager.
2020-11-02 18:22:30 <ikke> clandmeter: looks like nothing is wrong with the API
2020-11-02 18:22:49 <ikke> the test I ran still used an old docker image that did not contain API endpoint changes
2020-11-02 18:22:57 <ikke> so that would return an error
2020-11-02 19:37:58 <ikke> clandmeter: that gitally patch does not apply anymore
2020-11-03 07:37:16 <clandmeter> ikke: ah ok so all is ok?
2020-11-03 07:39:24 <ikke> clandmeter: I think so, yes
2020-11-03 07:39:32 <clandmeter> looks like the MR has been merged
2020-11-03 07:39:34 <ikke> I guess the sidekiq warning that was shown on boot is ok
2020-11-03 07:39:45 <clandmeter> which warning?
2020-11-03 07:39:58 <ikke> that sidekiq is not running
2020-11-03 07:40:09 <clandmeter> yes i think thats normal
2020-11-03 07:40:14 <clandmeter> iirc
2020-11-03 07:40:42 <ikke> Yeah, because I saw it running :)
2020-11-03 07:41:00 <ikke> So my plan was to upgrade tonight
2020-11-03 07:41:15 <ikke> regarding that MR, I could not even find the files anymore that it changed
2020-11-03 07:46:04 <clandmeter> file added Jan 6 2020
2020-11-03 07:55:51 <ikke> The whole folder was gone
2020-11-03 07:56:09 <ikke> probably restructured
2020-11-03 10:37:22 <clandmeter> ikke: which version are you upgrade to?
2020-11-03 10:40:55 <ikke> 13.2 for now
2020-11-03 10:44:07 <clandmeter> any reason you didnt jump a few releases?
2020-11-03 10:45:01 <ikke> smaller jumps are easier to troubleshoot
2020-11-03 10:45:52 <clandmeter> ok
2020-11-03 10:50:01 <ikke> But want to upgrade soon to newer versions
2020-11-03 14:50:07 <ikke> hmm
2020-11-03 14:57:06 <ikke> gbr2 is the zabbix proxy we use to montir services on nld, so these alerts are related
2020-11-03 16:30:55 <ikke> clandmeter: hmm, as this MR is quite new, it's possible that the files don't exist _yet_
2020-11-03 18:07:27 <ikke> clandmeter: gitlab 13.6 will start requiring postgres 12, so we need to think about upgrading that
2020-11-03 18:18:46 <clandmeter> right
2020-11-03 18:19:33 <clandmeter> maybe use something like https://github.com/tianon/docker-postgres-upgrade
2020-11-03 18:20:33 <ikke> gitlab 13.2 introduced servicedesk, which might be convenient for mirror requests
2020-11-03 18:21:10 <ikke> https://gitlab.alpinelinux.org/help/user/project/service_desk
2020-11-03 18:25:12 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10704
2020-11-03 18:32:50 <ikke> also https://gitlab.alpinelinux.org/alpine/infra/infra/-/alert_management
2020-11-03 18:57:53 <clandmeter> hmm
2020-11-03 18:58:35 <clandmeter> ikke: something changed with gitlab?
2020-11-03 18:58:52 <ikke> clandmeter: I've upgraded it?
2020-11-03 18:58:55 <ikke> to 13.2
2020-11-03 18:59:01 <clandmeter> i can no longer login
2020-11-03 18:59:04 <ikke> so, I would expect _something_ to be changed
2020-11-03 18:59:07 <ikke> hmm
2020-11-03 18:59:21 <ikke> I could login to the test instance
2020-11-03 18:59:49 <clandmeter> if i login it says invalid user/pass
2020-11-03 18:59:55 <clandmeter> or redirects me to 404
2020-11-03 19:00:19 <ikke> Hmm, no issues logging in myself
2020-11-03 19:01:09 <clandmeter> i was logged in before
2020-11-03 19:03:01 <ikke> 2020-11-03T10:45:22.369Z: Successful Login: username=clandmeter ip=x.x.x.x method=two-factor admin=true
2020-11-03 19:03:10 <ikke> That was this morning
2020-11-03 19:03:15 <clandmeter> yup
2020-11-03 19:03:20 <clandmeter> and now it doesnt work anymore
2020-11-03 19:03:24 <clandmeter> i can login with another acc
2020-11-03 19:03:34 <ikke> hmm, strange
2020-11-03 19:03:42 <ikke> I don't see a failed login attempt in the logs
2020-11-03 19:03:55 <ikke> I see you logging in as root
2020-11-03 19:04:12 <ikke> or rather, logging out
2020-11-03 19:04:23 <clandmeter> huh
2020-11-03 19:04:24 <clandmeter> wtf
2020-11-03 19:04:30 <clandmeter> gitlab send me an email
2020-11-03 19:04:39 <ikke> what bout?
2020-11-03 19:04:47 <clandmeter> As part of our commitment to keeping GitLab secure, we have identified and addressed a vulnerability in GitLab that allowed some users to bypass the email verification process in a recent security release.
2020-11-03 19:05:09 <ikke> aha, I saw that your e-mail was unconfirmed in your profile
2020-11-03 19:08:08 <ikke> your primary e-mail is still not confirmed
2020-11-03 19:08:29 <clandmeter> i have no idea how to do tht
2020-11-03 19:08:38 <ikke> I can force confirmation :P
2020-11-03 19:08:49 <ikke> done
2020-11-03 19:09:01 <ikke> Can you login now?
2020-11-03 19:09:58 <ikke> clandmeter: I haven't seen a lot of spam lately?
2020-11-03 19:10:45 <clandmeter> nope
2020-11-03 19:10:52 <clandmeter> im unable to do anything
2020-11-03 19:10:54 <ikke> No login?
2020-11-03 19:10:58 <clandmeter> i can reset my password
2020-11-03 19:11:03 <clandmeter> but that does nothing
2020-11-03 19:11:15 <clandmeter> i can request confirm my email
2020-11-03 19:11:20 <clandmeter> but it does not send an email
2020-11-03 19:12:11 <clandmeter> Almost there...
2020-11-03 19:12:11 <clandmeter> Please check your email to confirm your account
2020-11-03 19:12:24 <clandmeter> i dont get any email mr gitlab...
2020-11-03 19:13:33 <clandmeter> now it works
2020-11-03 19:14:27 <ikke> strange
2020-11-03 19:18:46 <clandmeter> have you tried upgrading pg on test?
2020-11-03 19:18:50 <ikke> no, not yet
2020-11-03 19:21:10 <clandmeter> how did you crfeate the support ticket?
2020-11-03 19:21:18 <clandmeter> via email?
2020-11-03 19:22:40 <ikke> yes
2020-11-03 19:22:44 <ikke> a dedicated e-mail address
2020-11-03 19:23:03 <clandmeter> dedicated as in?
2020-11-03 19:23:27 <ikke> https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/service_desk
2020-11-03 19:23:45 <clandmeter> heh
2020-11-03 19:24:44 <clandmeter> i hope i wont have to provide it via phone
2020-11-03 19:24:52 <ikke> lol
2020-11-03 19:25:31 <clandmeter> can we alias it via alpinelinux.org?
2020-11-03 19:26:05 <ikke> I guess so?
2020-11-03 19:26:55 <clandmeter> what about adding it to https://gitlab.alpinelinux.org/alpine/infra/mirrors
2020-11-03 19:27:55 <ikke> either that, or adding it to the recipients for alpine-mirrors@alpinelinux.org
2020-11-03 19:28:21 <clandmeter> yes thats what i mean
2020-11-03 19:36:29 <ikke> I don't see it yet in gitlab
2020-11-03 19:36:42 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues/service_desk
2020-11-03 19:37:38 <ikke> ah, now I understand what you meant
2020-11-03 19:37:44 <clandmeter> :)
2020-11-03 19:39:43 <clandmeter> they are by default confidential?
2020-11-03 19:39:50 <ikke> yes, apparently
2020-11-03 19:39:57 <ikke> could find a setting to control that
2020-11-03 19:40:05 <clandmeter> nt?
2020-11-03 19:40:09 <ikke> yes
2020-11-03 19:40:13 <clandmeter> :)
2020-11-03 19:40:48 <clandmeter> so we might need to add a template 
2020-11-03 19:41:23 <ikke> Yes, but I could not find where to do that either :P
2020-11-03 19:41:37 <clandmeter> ist in the help i think
2020-11-03 19:41:44 <clandmeter> you can put an .md file on fs
2020-11-03 19:42:06 <ikke> aha, ok
2020-11-03 19:42:33 <clandmeter> aha
2020-11-03 19:42:36 <clandmeter> i did get an reply
2020-11-03 19:42:40 <clandmeter> but it was in my filters
2020-11-03 19:43:46 <clandmeter> I find it a bit weird
2020-11-03 19:43:56 <ikke> A typical service desk e-mail
2020-11-03 19:44:22 <clandmeter> the reply does not really show its me who replied to it.
2020-11-03 19:44:34 <clandmeter> its kind of obvious, but still.
2020-11-03 19:47:44 <clandmeter> ah
2020-11-03 19:47:48 <clandmeter> its not on fs
2020-11-03 19:47:50 <clandmeter> its on git
2020-11-03 19:48:00 <clandmeter> The body of these email messages can customized by using templates. To create a new customized template,
2020-11-03 19:48:01 <clandmeter> create a new Markdown (.md) file inside the .gitlab/service_desk_templates/
2020-11-03 19:48:01 <clandmeter> directory in your repository. Commit and push to your default branch.
2020-11-03 19:48:15 <ikke> ah, that makes more sense
2020-11-03 19:48:50 <clandmeter> yeah, i thought it was needed for a responce email
2020-11-03 19:49:05 <clandmeter> but its ok like this
2020-11-04 01:09:45 <mckaygerhard> hey, alpine wiki reset password are working?
2020-11-04 01:20:20 <mckaygerhard> ping?
2020-11-05 10:10:17 <ikke> clandmeter: first real mirror request has been added now :)
2020-11-05 15:18:11 <clandmeter> ikke: yup seen it :)
2020-11-08 07:22:30 <ikke> mutes alerts for this mirror
2020-11-08 07:22:36 <ikke> muted*
2020-11-08 07:56:02 <mps> ikke: heh, this one is just to tell you 'something'
2020-11-08 07:56:50 <ikke> the solved would still come through after the problem alert
2020-11-09 07:47:08 <ncopa> i will try get the 3.13 builders up this week. we need to do somethign about disk space. i think we dont have enough free space for 3.13 release on our master mirror?
2020-11-09 07:47:25 <ncopa> should we move some older releases to ancient.a.o?
2020-11-09 07:48:23 <ncopa> for example v3.1 - v3.5?
2020-11-09 08:44:27 <ikke> Do we have enough space on ancient? 
2020-11-09 09:09:27 <ncopa> we have 100G free
2020-11-09 09:09:54 <ncopa> i wonder if we should set up a new ancient.a.o
2020-11-09 09:09:58 <ncopa> its just an archive
2020-11-09 16:18:36 <clandmeter> LBlaboon: ping, any update?
2020-11-09 16:39:29 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/issues/2
2020-11-09 16:41:02 <ncopa> oh, they want to reuse pkgs.a.o?
2020-11-09 16:41:05 <ncopa> thats cool
2020-11-09 16:45:45 <clandmeter> they also want to swirtch to apktools iirc
2020-11-09 16:51:51 <mps> clandmeter: yes, someone from openwrt was on our devel (or -linux) channel and told they thinking to use apk
2020-11-09 16:54:07 <clandmeter> i think there is an ml thread for it.
2020-11-09 16:54:21 <clandmeter> ikke: thx for pinging me
2020-11-09 16:54:39 <clandmeter> i think it should be no problem, as long as they remove our branding.
2020-11-09 19:14:56 <ikke> clandmeter: huh, we already have spam on that e-mail adres :-/
2020-11-09 19:15:01 <ikke> the service desk e-mail
2020-11-09 19:15:31 <ikke> probably to alpine-mirrors I guess
2020-11-09 19:15:45 <ikke> yeag
2020-11-09 20:45:27 <danieli> mps: that's pretty awesome
2020-11-09 20:48:53 <mps> danieli: hello, nice to see you :)
2020-11-09 20:49:15 <danieli> hiya :)
2020-11-09 20:49:48 <danieli> I did say I still lurk around IRC on a regular basis 
2020-11-09 20:49:58 <mps> yes, they use busybox, alpine, some kind of init which is 'similar' from admin perspective to openrc, so why not apk
2020-11-09 20:50:11 <mps> s/alpine/musl/
2020-11-09 20:50:11 <alpine-meetbot> mps meant to say: yes, they use busybox, musl, some kind of init which is 'similar' from admin perspective to openrc, so why not apk
2020-11-09 20:50:38 <mps> alpine become synonym in brain for musl :)
2020-11-09 20:51:51 <mps> even thought to rejoin their ML to help with apk, but I'm short with time
2020-11-09 21:00:23 <danieli> do they still use uclibc or do they use another libc now?
2020-11-09 21:00:28 <danieli> perhaps this is a better fit for -offtopic
2020-11-09 21:01:01 <mps> they switched to musl some time ago for most routers afaik
2020-11-09 21:01:55 <mps> they keep uclibc on devices which don't have resources for musl, old ones and ones with small flesh memory and/or ram
2020-11-09 21:02:12 <mps> flash*
2020-11-09 21:02:18 <danieli> "By default it is musl for most embedded targets, but glibc and others are also available."
2020-11-09 21:14:43 <LBlaboon> clandmeter: sorry for the delay. no update yet, but i will poke management again. i'll let you know as soon as i hear anything
2020-11-09 21:23:26 <mps> usa4 disk is full
2020-11-09 21:24:33 <mps> even 'make clean' in kernel tree doesn't work
2020-11-09 21:25:33 <ikke> ncopa bootstrapping 3.13 I guess
2020-11-09 21:25:52 <ikke> it went from ~70% to ~100%
2020-11-09 21:25:59 <mps> will fail, I guess :)
2020-11-09 21:30:46 <ikke> Not a lot I can do about atm
2020-11-09 21:30:50 <ikke> it seems to eat up all space 
2020-11-09 21:35:02 <mps> np, I don't have anything important to do there, just wanted to add snd-aloop module for issue request you assigned to me
2020-11-09 21:35:14 <mps> but this is not much important
2020-11-09 21:35:28 <clandmeter> ikke: we cannot delete issues?
2020-11-09 21:35:44 <mps> :)
2020-11-09 21:35:59 <mps> ah, you mean spam ones. sorry
2020-11-09 21:36:52 <ikke> clandmeter: perhaps through the API
2020-11-09 21:37:19 <ikke> clandmeter: was looking for it, and saw that they flapped a bit on supporting it or not
2020-11-09 21:37:30 <ikke> but that was around GL 8
2020-11-09 21:40:47 <ikke> (added that check)
2020-11-09 21:53:54 <clandmeter> ikke: click edit and delete :)
2020-11-09 21:54:14 <ikke> aha
2020-11-09 21:54:20 <ikke> thanks
2020-11-09 21:56:49 <clandmeter> i didnt notice pmos was also using pkgs
2020-11-09 21:57:12 <ikke> you mean aports-turbo?
2020-11-09 21:57:21 <clandmeter> nod
2020-11-10 06:49:21 <ncopa> morning
2020-11-10 06:49:27 <ncopa> usa4 has disk full yes
2020-11-10 06:49:30 <ncopa> due to 3.13
2020-11-10 06:51:52 <ncopa> not sure how to deal with the full disk
2020-11-10 06:56:31 <ncopa> there is an interesting bug in apk-tools
2020-11-10 06:56:53 <ncopa> it is a bug that eats the disk space for dinner
2020-11-10 06:57:00 <ncopa> and now also for breakfast
2020-11-10 07:57:05 <ikke> ncopa: it sure has an appetite
2020-11-10 08:48:09 <ncopa> yes
2020-11-10 08:48:17 <ncopa> and 'yes' was the bug
2020-11-10 08:48:38 <ikke> oh, fun
2020-11-10 08:48:58 <ncopa> it kinda was :)
2020-11-10 08:49:47 <ikke> ncopa: seems like caching in cgit causes issues indeed
2020-11-10 08:49:56 <ncopa> ok?
2020-11-10 08:50:26 <ikke> I notice that the links in #alpine-commits often give bad object back
2020-11-10 08:50:37 <ikke> probably because cgit does not have the commit yet
2020-11-10 08:50:50 <ikke> I thought initially that it might be because it was ambiguous
2020-11-10 08:51:01 <ikke> but now I notice that if I remove a character from the hash, it works again
2020-11-10 08:51:13 <ikke> so it seems like cgit cached the bad object id result for that hash
2020-11-10 08:51:28 <ikke> compare: https://git.alpinelinux.org/aports/commit/?id=bac0dff892e4
2020-11-10 08:51:31 <ikke> vs https://git.alpinelinux.org/aports/commit/?id=bac0dff892e
2020-11-10 08:53:33 <ikke> So I think I'll disable caching again?
2020-11-10 08:59:15 <ncopa> both urls works?
2020-11-10 08:59:47 <ikke> yeah, now :P
2020-11-10 09:00:09 <ikke> after I disabled the cache
2020-11-10 09:00:25 <ncopa> ok
2020-11-10 15:01:27 <HRio> any problem with the ppc64le gitlab runner? or just much work in the queue?
2020-11-10 15:04:01 <HRio> it "woke-up" :-)
2020-11-10 15:04:18 <mckaygerhard> hey, alpine wiki reset password are working?
2020-11-10 15:04:52 <ikke> HRio: I see 2 jobs running and 1 pending
2020-11-10 15:05:32 <HRio> ok took 18 minutes before picking my job
2020-11-10 15:05:43 <ikke> maybe something heave was building before
2020-11-10 15:05:46 <ikke> heavy
2020-11-10 15:05:52 <ikke> mckaygerhard: I think it's broken
2020-11-10 15:07:14 <mckaygerhard> err .. i try to be respectfull but i work hardly in documenting all the things made in alpine and well registration is broken and now password recovery too ? ikke ? you got it my point?
2020-11-10 15:09:40 <mckaygerhard> something can help me with that ? i mean.. 
2020-11-10 16:33:42 <mckaygerhard> emm no solution?
2020-11-10 16:34:38 <ikke> THis is something I have to delve into, which I don't have time for right nwo
2020-11-10 16:35:06 <mckaygerhard> err ok i got we are all bussy but that has months.. several months 
2020-11-10 16:35:19 <mckaygerhard> several months
2020-11-11 21:04:11 <clandmeter> ikke: how quick does gitlab.a.o log somebody out again?
2020-11-11 21:05:26 <ikke> clandmeter: I think it depends on when you last visited
2020-11-11 21:05:34 <ikke> The setting is 10 years
2020-11-11 21:05:48 <ikke> but obviously that's not the case
2020-11-11 21:07:49 <clandmeter> i guess its the session duration
2020-11-11 21:08:00 <clandmeter> 5256000
2020-11-11 21:08:56 <ikke> 60 days
2020-11-11 21:10:23 <clandmeter> 60 days?
2020-11-11 21:11:10 <ikke> 5256000 / 86400 = 60.3333
2020-11-11 21:11:27 <clandmeter> thats minutes
2020-11-11 21:11:30 <ikke> ah
2020-11-11 21:11:34 <clandmeter> so 10y
2020-11-11 21:11:38 <ikke> nod
2020-11-11 21:11:41 <clandmeter> but it makes no sense
2020-11-11 21:11:46 <clandmeter> i was logged in last week
2020-11-11 21:11:53 <ikke> Yes, it's not working
2020-11-11 21:11:56 <clandmeter> when i had that issue when you upgraded
2020-11-11 21:12:09 <ikke> dalias complained about it as well
2020-11-11 21:12:19 <clandmeter> i also have a sh*tload of sessions in my profile
2020-11-11 21:12:20 <ikke> btw, I setup 2fa for algitbot
2020-11-11 21:13:02 <ikke> because every time I impersonated algitbot, it required you to set it up, but I could not stop impersonation anymore
2020-11-11 21:13:29 <ikke> .calc 5256000 / (60*24)
2020-11-11 21:13:29 <alpine-meetbot> ikke: 3650
2020-11-11 21:13:33 <ikke> cool
2020-11-11 21:13:54 <clandmeter> you can also just use a calculator ;-)
2020-11-11 21:14:24 <ikke> well, that _is_ a calculator
2020-11-11 21:14:47 <ikke> just one in the channel
2020-11-11 21:17:46 <clandmeter> its still weird
2020-11-11 21:17:58 <clandmeter> i dont remember having this before
2020-11-11 21:18:19 <clandmeter> or maybe i used a private tab cause of the issues i was having.
2020-11-11 21:18:41 <ikke> Hmm, I'm still logged in
2020-11-11 21:18:51 <ikke> ie, I'm not having issues of getting logged out
2020-11-11 21:19:33 <clandmeter> im going to kill my gazillion sessions i still have open
2020-11-11 21:19:57 <ikke> clandmeter: did you get all the news about travis?
2020-11-11 21:20:13 <clandmeter> i remember you mention something
2020-11-11 21:20:25 <clandmeter> its going to stop the free tier completely?
2020-11-11 21:20:38 <ikke> I think they are giving free minutes or something like that
2020-11-11 21:21:16 <clandmeter> but not for us? :)
2020-11-11 21:21:18 <ikke> And they switched from travis.org to tracis.com
2020-11-11 21:21:21 <ikke> clandmeter: I have no idea
2020-11-11 21:21:24 <ikke> I didn't check
2020-11-11 21:21:33 <clandmeter> is that a typo?
2020-11-11 21:21:44 <ikke> yes
2020-11-11 21:21:46 <ikke> travis.com
2020-11-11 21:21:48 <clandmeter> :)
2020-11-11 21:21:57 <clandmeter> ci could have been real
2020-11-11 21:22:10 <ikke> Oh, maybe I'm stupid
2020-11-11 21:22:41 <clandmeter> travis.org is something else it seems
2020-11-11 21:22:46 <ikke> We're not using travis for gitlab
2020-11-11 21:22:57 <clandmeter> nope
2020-11-11 21:23:24 <clandmeter> i think jirutka is still using it
2020-11-11 21:23:33 <ikke> aha, ok
2020-11-11 21:23:43 <ikke> we still have a .travis.yml in aports
2020-11-11 21:24:11 <clandmeter> i guess the url is/was https://travis-ci.org/
2020-11-11 21:24:25 <ikke> oh, right
2020-11-11 21:24:44 <clandmeter> i dont think they build software in churches 
2020-11-11 21:24:47 <ikke> :D
2020-11-11 21:27:04 <clandmeter> i have logged into gitlab with another browser, ill check it next week if its still working.
2020-11-11 21:27:56 <clandmeter> thats weird
2020-11-11 21:28:06 <clandmeter> both sessions say it has the same time
2020-11-11 21:28:36 <clandmeter> Signed in on 11 Nov 21:03
2020-11-16 15:20:22 <LBlaboon> clandmeter: quick update - our marketing team is still working out the details of a contract, but in the meantime they are gonna go ahead and increase the monthly credit on your account to $1000
2020-11-16 15:20:28 <LBlaboon> that should be in place by the end of the day today
2020-11-16 15:22:44 <ncopa> :-O thats awesome! thank you \o/
2020-11-16 15:33:21 <clandmeter> LBlaboon: nice.
2020-11-16 15:34:34 <clandmeter> LBlaboon: would be nice if we could have a talk/chat soon and discus details.
2020-11-16 15:36:09 <LBlaboon> yea, once our marketing team has everything done on their end i'll setup a meeting so we can all discuss the details
2020-11-16 15:37:44 <LBlaboon> looks like the new credit just went through, so you should see that reflected in the billing section of your account
2020-11-16 16:09:03 <mps> LBlaboon: oh, very nice
2020-11-17 14:34:23 <Ariadne> let me guess, sourcehut has no ability to ban people from the mailing lists
2020-11-17 14:34:43 <Ariadne> sigh, i will try to get on the mailman stuff asap, i just keep having new shit popping up
2020-11-17 14:35:01 <ikke> user list?
2020-11-17 14:35:05 <ikke> I mean, the users mailing list?
2020-11-17 14:36:49 <Ariadne> yes
2020-11-17 14:36:52 <ikke> But I don't see an option to block by e-mail
2020-11-17 14:37:03 <Ariadne> you know who i'm talking about, i'm sure
2020-11-17 14:37:59 <ikke> Yes
2020-11-17 14:38:08 <ikke> Possibly it's possible
2020-11-17 14:38:31 <ikke> welcome to tautology club
2020-11-17 14:40:03 <ikke> Should we notify them?
2020-11-17 14:42:47 <ikke> Apparently you can disable just replies
2020-11-18 10:29:07 <ncopa> I wonder if we should offer an openvpn config with a certificate valid for a few weeks so gcc dev can access our mips64 machine:  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96734#c9
2020-11-18 10:29:16 <ncopa> what do you think?
2020-11-18 10:30:49 <ikke> You mean to a container on the mips host?
2020-11-18 10:31:06 <ncopa> to a mips container
2020-11-18 10:31:19 <ncopa> i could give him access to my ncopa-edge-mips64 for example
2020-11-18 10:31:58 <ncopa> but the mips64 host does not have public ip address so i think openvpn access with dmvpn routing woudl be needed
2020-11-18 10:32:05 <ncopa> unless you have better ideas
2020-11-18 10:32:19 <ikke> No, without public address I guess that's the only option
2020-11-18 12:40:39 <ikke> Should not be an issue, but I don't know how to generate ovpn certificates
2020-11-19 10:43:35 <ncopa> I think i figured out how to generate openvpn configs/certs
2020-11-19 10:44:18 <ikke> ok
2020-11-19 10:51:33 <ncopa> clandmeter: FYI: >>> Generated: ovpn/alpine-infra-mliska.ovpn
2020-11-19 10:51:46 <ncopa> expire=30
2020-11-19 10:51:50 <ncopa> so its only valid for a month
2020-11-19 11:00:49 <ncopa> Ariadne: FYI, im giving mliska access to my mips64 container https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96734#c9
2020-11-19 11:01:20 <Ariadne> go for it
2020-11-19 20:19:27 <Ariadne> fedora is moving further along with their evaluation of matrix
2020-11-19 20:19:42 <Ariadne> it may be worthwhile to consider reevaluating it
2020-11-19 20:51:56 <clandmeter> ncopa: ok
2020-11-19 20:52:02 <clandmeter> glad you found it
2020-11-19 20:52:47 <clandmeter> ikke: did you ever look into geodns?
2020-11-19 20:53:00 <ikke> clandmeter: not any further
2020-11-19 20:55:12 <clandmeter> i wonder what happens when a client is using cloudflare or google dns. i guess it kind of breaks the logic.
2020-11-19 20:55:15 <ikke> I think we had an issue with getting a geoip databse, right?
2020-11-19 20:56:23 <clandmeter> not sure
2020-11-19 20:59:20 <ikke> I do recall, but not in detail, that it did passthrough the /24 of the requestor
2020-11-19 20:59:21 <clandmeter> i dont think geoip is an issue
2020-11-19 20:59:31 <ikke> In what sense?
2020-11-19 20:59:45 <clandmeter> that we need to register for it
2020-11-19 20:59:58 <clandmeter> and with that info we can use the updater
2020-11-19 21:01:49 <ikke> https://www.maxmind.com/en/geolite2/signup
2020-11-19 21:02:03 <clandmeter> ikke: ok so what happens if somebody else makes the request via cloudflare?
2020-11-19 21:02:57 <ikke> trying to find the details
2020-11-19 21:03:30 <clandmeter> it looks like pdns has a geoip module 
2020-11-19 21:03:33 <clandmeter> in aports
2020-11-19 21:05:07 <clandmeter> i guess we will need to turn rsync.a.o into a zone
2020-11-19 21:08:08 <ikke> I guess we'll need to figure out what happens if you request through cloudflare
2020-11-19 21:09:04 <ikke> https://doc.powerdns.com/authoritative/backends/geoip.html#
2020-11-19 21:28:02 <clandmeter> ok, maybe it makes more sense to introduce geo.a.o like the example does.
2020-11-20 11:39:45 <ncopa> im struggling to send gnupg encrypted message
2020-11-20 11:40:10 <ncopa> claws-mail does not show any key for the email recipient
2020-11-20 11:41:03 <ncopa> $ gpg --list-keys mliska | tpaste
2020-11-20 11:41:03 <ncopa> https://tpaste.us/R4Eb
2020-11-20 11:41:24 <ikke> No experience with claws
2020-11-20 11:41:29 <ncopa> also evolution email gives error
2020-11-20 11:41:40 <ncopa> but i wonder what is wrong with the key?
2020-11-20 11:42:59 <ncopa> need to go out for a while. bbl
2020-11-20 11:43:52 <ncopa> maybe i need to add trust to the key?
2020-11-20 11:44:00 <ikke> might be
2020-11-20 11:44:13 <ikke> did you try to encrypt something directly with gpg?
2020-11-20 11:46:07 <ncopa> will try when i get back. thanks
2020-11-20 11:53:27 <ncopa> im back. decided to not go out.
2020-11-20 11:55:44 <ikke> you can set trust with gpg --edit-key <fingerprint>
2020-11-20 11:55:48 <ikke> then the trust command
2020-11-20 11:56:36 <ncopa> gpg: error retrieving 'mliska@suse.cz' via Local: Unusable public key
2020-11-20 11:56:36 <ncopa> gpg: error retrieving 'mliska@suse.cz' via WKD: General error
2020-11-20 11:56:36 <ncopa> gpg: mliska@suse.cz: skipped: General error
2020-11-20 11:56:36 <ncopa> gpg: alpine-infra-mliska.ovpn: encryption failed: General error
2020-11-20 11:57:01 <ncopa> that happens when i tried to encrypt from command line
2020-11-20 11:57:35 <ikke> did you use gpg or gpg2?
2020-11-20 12:00:57 <ikke> The keys you showed only have SC (signing, certification), not encryption key usage
2020-11-20 12:09:21 <ikke> Ariadne: We could try to deploy dendrite
2020-11-20 12:10:22 <ncopa> yeah, i figured they keys cannot be used for encryption. just signing
2020-11-20 12:11:02 <ncopa> I think I'll create a temp ssh account on dev.a.o instead
2020-11-20 13:03:02 <clandmeter> ikke: please be careful with deploying new services. Lets first outweigh the pro and cons.
2020-11-20 13:03:26 <Ariadne> ikke yeah that's what I was thinking.  but even synapse is a lot better now than it used to be
2020-11-20 13:03:43 <clandmeter> We already have enough drama about the ml and still no solution.
2020-11-20 13:04:04 <Ariadne> ml solution is coming.  mps is working on it
2020-11-20 13:04:25 <clandmeter> It's coming...
2020-11-20 13:04:39 <clandmeter> ™️
2020-11-20 13:06:51 <Ariadne> note that redeploying matrix can be done in an orderly way that minimizes drama
2020-11-20 13:10:26 <ikke> And just internal testing matrix is not the same as suddenly switching the mailing list for everyone 
2020-11-20 13:36:12 <Ariadne> 
ACTION fires up a linode to experiment with dendrite

2020-11-20 14:08:24 <ikke> clandmeter: fyi, I totally agree that we should be careful with making changes. My goal was just to be able to have something to test with
2020-11-21 11:21:53 <Ariadne> i deployed dendrite to a linode and its way more responsive than synapse ever was
2020-11-21 11:33:23 <ikke> And using less memory I gather
2020-11-21 12:18:19 <Ariadne> about 300mb RSS for everything
2020-11-21 12:33:22 <Ariadne> 45mb RSS with GODEBUG=madvdontneed=1
2020-11-21 12:33:32 <Ariadne> 
ACTION mutters things about Go

2020-11-21 12:36:54 <ikke> "This is less efficient, but causes RSS numbers to drop
2020-11-21 12:36:57 <ikke> more quickly.
2020-11-21 12:36:59 <ikke> "
2020-11-21 12:46:15 <ikke> So I guess it returns pages more quickly, but with a chance that it needs to request a new page immediately after?
2020-11-21 12:56:50 <Ariadne> yes, it makes Go's memory allocator behave like every other memory allocator
2020-11-21 12:56:55 <Ariadne> :P
2020-11-21 12:57:22 <Ariadne> and 'less efficient' isnt true for a long time
2020-11-21 12:57:32 <ikke> :)
2020-11-21 12:57:39 <Ariadne> but is commonly believed still
2020-11-22 00:22:52 <clandmeter> ive set ratelimiting on git.a.o
2020-11-22 00:22:55 <clandmeter> could be too strict
2020-11-22 10:02:25 <ikke> clandmeter: "Too Many Requests"
2020-11-22 10:57:21 <clandmeter> you click too much :)
2020-11-22 10:58:10 <clandmeter> ikke: you got that yourself?
2020-11-22 10:58:18 <ikke> Yes
2020-11-22 10:58:27 <clandmeter> with normal usage?
2020-11-22 10:58:33 <ikke> Was just going to a link
2020-11-22 10:58:42 <clandmeter> a single one?
2020-11-22 10:58:46 <ikke> Yes
2020-11-22 10:58:47 <clandmeter> single click?
2020-11-22 10:59:11 <ikke> Yes
2020-11-22 10:59:21 <clandmeter> do you share your outoging ip?
2020-11-22 10:59:34 <ikke> It's on my VPS, so no
2020-11-22 10:59:46 <clandmeter> it works just fine for me
2020-11-22 11:00:01 <ikke> Now it works fine for me as well
2020-11-22 11:00:49 <clandmeter> probably caching
2020-11-22 11:01:10 <clandmeter> i can increase the limit 
2020-11-22 11:01:16 <clandmeter> i think its 50req/min
2020-11-22 11:01:19 <clandmeter> which is kind of low
2020-11-22 11:01:24 <clandmeter> but i didnt hit the limit
2020-11-22 11:01:36 <clandmeter> cpu went down from 100% to 25 or so
2020-11-22 11:01:42 <ikke> not bad
2020-11-22 11:03:08 <ikke> Load average: 1.14 1.35 1.40
2020-11-22 11:05:09 <clandmeter> yes its still not perfect :)
2020-11-22 11:05:18 <clandmeter> but htop showed like 100 on all cores
2020-11-22 11:05:34 <clandmeter> now round 25-35
2020-11-22 11:06:02 <clandmeter>  /srv/compose/cgit/docker-compose.yml
2020-11-22 11:06:09 <clandmeter> you can see the limit
2020-11-22 11:06:39 <clandmeter> i finally found out how to actually make traefik work
2020-11-22 11:06:57 <clandmeter> i increased t he limit from 50 to 100 req/min
2020-11-22 11:07:08 <clandmeter> it should be ipbased
2020-11-22 11:08:48 <ikke> so we have single ips making more than 100 requests per minute?
2020-11-22 11:09:07 <clandmeter> yes 
2020-11-22 11:09:12 <ikke> oof
2020-11-22 11:09:14 <clandmeter> a lot
2020-11-22 11:09:28 <clandmeter> you can hit 50 a min also easily if you want to
2020-11-22 11:10:01 <clandmeter> the compose log is also a lot more silent
2020-11-22 11:10:13 <clandmeter> before it was like the matrix
2020-11-22 11:11:54 <clandmeter> so we now have rate limiting in place and if req passes it and goes t i cgit secdb it will redirect to github.
2020-11-22 11:12:14 <clandmeter> to*
2020-11-23 11:54:57 <clandmeter> ikke: did you turn of cgit caching?
2020-11-23 11:55:01 <ikke> yes
2020-11-23 11:55:20 <clandmeter> next time, can you put a note in the config please? :)
2020-11-23 11:55:39 <ikke> ok
2020-11-23 11:55:51 <clandmeter> why did y ou turn it of?
2020-11-23 11:56:06 <ikke> The reason was that it cached bad object pages
2020-11-23 11:56:22 <clandmeter> has it been reported upstream?
2020-11-23 11:56:23 <ikke> Probably because the repo didn't have the commit yet
2020-11-23 11:56:33 <clandmeter> ah ok
2020-11-23 11:56:45 <ikke> so any subsequent requests always returned that
2020-11-23 11:56:50 <clandmeter> so it responded 404
2020-11-23 11:57:30 <ikke> it gave a "bad object" error, the equivalent of a 404
2020-11-23 11:57:36 <clandmeter> :)
2020-11-23 11:57:56 <clandmeter> maybe cgit should not cache those items at all
2020-11-23 11:58:25 <clandmeter> do you know why it happens?
2020-11-23 11:58:53 <clandmeter> why would any client request it when its not yet available?
2020-11-23 12:00:56 <ikke> It's from the browser, not a git client
2020-11-23 12:01:35 <ikke> https://git.alpinelinux.org/aports/commit/?id=abcdef1234
2020-11-23 12:34:58 <clandmeter> ikke: yes i know its a brower, i mean how would a client/browser be able to access it before its available?
2020-11-23 12:38:14 <clandmeter> i know there is a delay, but cgit is listening over mqtt so that delay should be limited.
2020-11-23 12:39:41 <ikke> I don't know how it happens, but I noticed quite often the the links from #alpine-commits resulted in a message like that
2020-11-23 12:39:58 <ikke> and when I changed the url (added or removed one character from the hash), it worked
2020-11-23 12:41:24 <clandmeter> ok
2020-11-23 12:41:45 <clandmeter> but still, the cache should only work for 5 min iirc
2020-11-23 12:41:54 <clandmeter> ttl
2020-11-23 12:42:31 <clandmeter> ah there is an exception
2020-11-23 12:43:11 <ikke> sure, but waiting 5 minutes to be able to load a page is anoying :P
2020-11-23 12:44:42 <clandmeter> thats not what i meant to say
2020-11-23 12:45:12 <clandmeter> the man page is not explicit about it, but i think they mean that urls with fixed sha1 will stay cached.
2020-11-23 12:45:49 <clandmeter> disabling cache will increase cpu usage a lot
2020-11-23 12:46:14 <clandmeter> mostly on log pages which consume lots of cpu
2020-11-23 12:46:26 <ikke> we didn't have caching for most of the time
2020-11-23 12:46:36 <ikke> I enabled it to see if it reduced CPU usage
2020-11-23 12:46:46 <ikke> but I didn't see a noticable difference
2020-11-23 12:46:51 <ikke> but left it on
2020-11-23 12:47:03 <clandmeter> im trying to do the same
2020-11-23 12:47:11 <clandmeter> but this time i started from traefik
2020-11-23 12:47:28 <clandmeter> so now i blocked all secdb req
2020-11-23 12:47:44 <ikke> That should matter a lot I guess
2020-11-23 12:47:54 <clandmeter> but there is still some piece of software hammering log pages
2020-11-23 12:48:00 <ikke> hmm
2020-11-23 12:48:06 <clandmeter> and this consumes 100% cpu
2020-11-23 12:48:08 <ikke> And do we know what?
2020-11-23 12:48:14 <clandmeter> well actually 400%
2020-11-23 12:48:33 <clandmeter> the client is http4s-blaze/0.21.8
2020-11-23 12:48:51 <ikke> I think we've seen that before
2020-11-23 12:49:09 <ikke> https://github.com/http4s/blaze
2020-11-23 12:49:10 <clandmeter> some scala based client
2020-11-23 12:49:31 <ikke> But that doesn't necessarily tell much about the origin
2020-11-23 12:49:47 <clandmeter> it tells you its not a browser
2020-11-23 12:49:53 <ikke> yes
2020-11-23 12:50:04 <clandmeter> and looking at the qty and diff ip's
2020-11-23 12:50:52 <clandmeter> it also doesnt honor robots
2020-11-23 12:51:06 <ikke> should we return https://httpstatuses.com/429?
2020-11-23 12:51:18 <clandmeter> so i would not have any issue blocking those attempts 
2020-11-23 12:51:38 <ikke> Me neither
2020-11-23 12:51:51 <clandmeter> we could, but im not sure i can do it from traefik
2020-11-23 12:51:55 <ikke> ok
2020-11-23 12:52:24 <ikke> If we give a clear signal that they are sending too much traffic, they might reconsider their approach when users start complaining
2020-11-23 12:52:34 <ikke> or their systems stop working
2020-11-23 13:08:41 <clandmeter> [23/Nov/2020:13:07:58 +0000] "GET /cgit/aports/log/?qt=grep&q=cve&showmsg=1 HTTP/1.1" 429 169 "-" "http4s-blaze/0.21.8"
2020-11-23 13:09:57 <clandmeter> i added it to nginx, traefik makes my head spin.
2020-11-23 13:10:13 <ikke> nginx is in front of traefik, right?
2020-11-23 13:10:18 <ikke> or behind?
2020-11-23 13:10:19 <clandmeter> vice versa
2020-11-23 13:10:22 <ikke> ok
2020-11-23 13:10:38 <clandmeter> we need nginx for uwsgi
2020-11-23 13:11:10 <clandmeter> if you can find a solution for traefik, be my guest
2020-11-23 16:43:13 <clandmeter> ikke: https://imgur.com/a/h9km5fa
2020-11-23 16:43:50 <ikke> Yes, I saw that
2020-11-23 16:44:00 <ikke> A big difference
2020-11-23 16:44:08 <clandmeter> yup
2020-11-23 16:44:20 <clandmeter> we can add more UA's if needed
2020-11-23 16:44:24 <ikke> nod
2020-11-23 16:44:34 <clandmeter> but looks sane now
2020-11-23 16:44:44 <clandmeter> maybe can even upper the limit again
2020-11-23 16:45:08 <clandmeter> the problem with prev limit was that i didnt config burst
2020-11-23 16:45:32 <clandmeter> so a single click would do 2 reqs, page and css
2020-11-23 16:45:46 <ikke> yes
2020-11-23 16:45:59 <clandmeter> so burst is set to 2 now
2020-11-23 16:46:02 <clandmeter> that should allow them
2020-11-23 16:46:33 <clandmeter> i also tried to move the git db to tmpfs
2020-11-23 16:46:44 <clandmeter> seems to improve for around 15%
2020-11-23 16:46:49 <clandmeter> git log
2020-11-23 16:47:42 <clandmeter> but that would not survive a reboot, so i would need to add logic to refetch git on reboot.
2020-11-23 16:47:48 <ikke> right
2020-11-23 16:48:29 <clandmeter> i think ill just take the win as it is now. :)
2020-11-23 16:50:05 <clandmeter> geo.a.o is also active on that server
2020-11-23 16:50:13 <ikke> ok, good to know
2020-11-23 16:51:05 <clandmeter> i spun up one additinal server
2020-11-23 16:51:13 <ikke> deu6
2020-11-23 16:51:18 <clandmeter> it has a 1.2TB store
2020-11-23 16:51:25 <clandmeter> can be extended ofc
2020-11-23 16:51:37 <clandmeter> think the linodes have 5Gb connections max
2020-11-23 16:51:39 <ikke> Were you already syncing data there? 
2020-11-23 16:51:47 <ikke> got a traffic alert from linode
2020-11-23 16:51:47 <clandmeter> and 5TB max traffic
2020-11-23 16:52:01 <ikke> hmm, 5TB max traffic can be an issue
2020-11-23 16:52:08 <clandmeter> could be
2020-11-23 16:52:19 <clandmeter> but if we spread it over 3 servers it should be okish
2020-11-23 16:52:25 <ikke> ok
2020-11-23 16:52:37 <clandmeter> need to check current usage on one of the packet servers
2020-11-23 16:53:02 <clandmeter> i dont see a switch to turn on more traffic
2020-11-23 16:53:09 <clandmeter> except upgrade the vps
2020-11-23 16:53:40 <clandmeter> and yes, the new one is synced up
2020-11-23 16:53:59 <clandmeter> ill spin up the other 2 soon
2020-11-23 16:54:25 <clandmeter> geo.a.o was working nicely
2020-11-23 17:58:21 <clandmeter> LBlaboon: is there a way to increase traffic on a vps without upgrading it
2020-11-23 17:58:58 <clandmeter> s/traffic/traffic limit/
2020-11-23 17:58:58 <alpine-meetbot> clandmeter meant to say: LBlaboon: is there a way to increase traffic limit on a vps without upgrading it
2020-11-23 19:35:28 <LBlaboon> clandmeter: you have to upgrade to a higher plan if you want to increase your base traffic limit, however it might be cheaper to just pay the overage fee ($0.10/GB) depending on how far over you go
2020-11-23 19:39:35 <LBlaboon> one thing to keep in mind however is that network traffic is counted at the account level, rather than the individual linode level
2020-11-23 19:41:01 <LBlaboon> so for example if you have two linodes that each have a 1TB traffic limit, it's ok if one of them uses 1.5TB as long as the other doesn't use more than 500GB
2020-11-23 19:41:27 <ikke> Ok, that is good to know
2020-11-23 19:42:15 <LBlaboon> you can see the overall network usage for your account on the home page on the manager
2020-11-23 19:42:38 <ikke> Monthly Network Transfer Pool
2020-11-23 19:42:38 <LBlaboon> on the right-hand side, labelled "Monthly Network Transfer Pool"
2020-11-23 19:42:46 <LBlaboon> yep
2020-11-23 19:44:00 <LBlaboon> sometimes we have customers add small 1GB linode instances to their account just to increase their total network transfer
2020-11-23 19:52:35 <mps> LBlaboon: nice to know this, and not related to alpine but two private linodes I manage. thanks for explanation
2020-11-23 19:57:03 <clandmeter> LBlaboon: understand, thx
2020-11-23 20:11:07 <clandmeter> ikke: its not possible to fetch user.keys anymore from gitlab when you are not logged in?
2020-11-23 20:12:35 <ikke> apparently not, was not aware this was possible before
2020-11-23 20:43:02 <clandmeter> i always used github as source
2020-11-23 20:43:14 <clandmeter> but found out gitlab has it as well 
2020-11-23 20:43:18 <ikke> yes
2020-11-23 20:43:27 <clandmeter> kind of useless like this :)
2020-11-23 20:43:31 <ikke> I think it has to do with whether you allow public repos or not
2020-11-23 20:43:47 <ikke> I read somewhere that some setting like that makes user profiles private as well
2020-11-23 20:45:10 <clandmeter> ah could be
2020-11-23 20:46:22 <ikke> fsdfhttps://docs.gitlab.com/ee/public_access/public_access.html#visibility-of-users
2020-11-23 20:50:29 <clandmeter> thats a new protocol?
2020-11-23 20:50:38 <ikke> yup :P
2020-11-23 20:51:41 <clandmeter> but 
2020-11-23 20:51:44 <clandmeter> why did it change?
2020-11-23 20:51:56 <clandmeter> i mean we were private from the start right?
2020-11-23 20:52:00 <ikke> yes
2020-11-23 20:52:33 <ikke> Maybe someone decided that this should affect being able to obtain users keys as well? no clue
2020-11-23 20:53:10 <clandmeter> might be
2020-11-23 21:18:44 <clandmeter> other 2 sites are also syncing 
2020-11-23 21:19:16 <ikke> (y)
2020-11-23 21:20:01 <clandmeter> i think dallas, frankfurt and tokio should cover most of the world
2020-11-23 21:20:50 <ikke> Right
2020-11-23 21:21:07 <ikke> Missing Africa, but that's generally more difficult
2020-11-23 21:21:17 <clandmeter> yes
2020-11-23 21:21:29 <clandmeter> linode does not have grounds in afrika afaik
2020-11-23 21:21:32 <ikke> heh
2020-11-23 21:21:44 <clandmeter> Africa :) 
2020-11-23 21:22:50 <ikke> Toto :P
2020-11-23 21:23:38 <danieli> did linode decide what they want in return?
2020-11-23 21:24:24 <ikke> They were still working on that, but in the mean time, they upped our balance a bit
2020-11-23 21:24:26 <clandmeter> not conclusive, we already did some parts.
2020-11-23 21:24:54 <danieli> I see, I 'overheard' something about it the other day, wasn't sure if it was decided yet
2020-11-23 21:30:50 <ikke> clandmeter: Do we update our password on alpine-mirrors regularly? I received an e-mail saying that a new security policy requires that :P
2020-11-23 21:35:09 <clandmeter> uhm, not following?
2020-11-23 21:35:49 <clandmeter> i updated geodns to use real records
2020-11-23 21:35:56 <ikke> sorry, that was a spam mail
2020-11-23 21:36:04 <clandmeter> ah ok
2020-11-23 21:36:16 <clandmeter> thats probably in my gmail spam :)
2020-11-23 21:36:25 <ikke> yes, same for me
2020-11-23 21:36:36 <ikke> was just reviewing if it didn't caught to much
2020-11-23 21:38:30 <clandmeter> i just went over it
2020-11-23 21:38:34 <clandmeter> i c an stop working
2020-11-23 21:41:06 <clandmeter> ah bummer
2020-11-23 21:41:43 <clandmeter> docker doesnt take file changes when its directly mounted
2020-11-24 17:11:58 <clandmeter> ikke: how do you handle acme requests on a geo based zone entry? :)
2020-11-24 17:18:17 <ikke> good question
2020-11-24 17:28:06 <ikke> It means we need to be able to handle / forward requests from any of the targets
2020-11-24 17:28:56 <clandmeter> its actually pretty simple
2020-11-24 17:29:05 <clandmeter> just use dns like we do now
2020-11-24 17:34:09 <clandmeter> it would be a challenge to do it http based.
2020-11-24 17:35:57 <ikke> indeed
2020-11-24 18:13:46 <clandmeter> ikke: https://rsync.geo.alpinelinux.org
2020-11-24 18:14:02 <ikke> cool
2020-11-24 18:14:11 <ikke> and rsync.a.o redirects to there?
2020-11-24 18:14:16 <ikke> or will?
2020-11-24 18:14:19 <clandmeter> will
2020-11-24 18:14:28 <clandmeter> need to check if it auto updates correctly 
2020-11-24 18:18:08 <clandmeter> i dont think it does
2020-11-24 18:20:16 <clandmeter> and we need t o think about that issue we had before
2020-11-24 18:20:26 <clandmeter> when the mirror rebooted and dint mount 
2020-11-24 18:20:39 <clandmeter> and all clients rm -rf 
2020-11-24 18:21:27 <ikke> ugh, yes
2020-11-24 19:02:55 <ikke> We are running into docks rate limiting
2020-11-24 19:02:58 <ikke> docker*
2020-11-24 19:03:00 <ikke> https://gitlab.alpinelinux.org/andypost/aports/-/jobs/254412
2020-11-24 19:29:16 <ikke> clandmeter: I've set the pull_policy to not-if-present for the time being
2020-11-24 19:30:22 <ikke> maybe we can even keep that
2020-11-24 19:30:37 <ikke> We clean the images weekly, so should be updated after that
2020-11-24 19:31:04 <ikke> maybe we can prune images nightly
2020-11-24 20:03:53 <clandmeter> ikke: ok
2020-11-24 20:04:02 <clandmeter> algitbot: retry master
2020-11-25 06:35:44 <ikke> clandmeter: https://git.alpinelinux.org/aports/commit/?id=09ec5bfc6f84 -> bad object id
2020-11-25 06:44:22 <ikke> now it works, ofcourse
2020-11-25 07:36:24 <clandmeter> ikke: im still getting bad object
2020-11-25 08:20:59 <ikke> Interesting, for me it's working
2020-11-25 08:24:25 <clandmeter> yes cause i disabled caching :)
2020-11-25 08:27:12 <ikke> Well, it was working before you responded :P
2020-11-25 08:27:27 <ikke> a few minutes after I first looaded it
2020-11-25 08:28:10 <clandmeter> i refreshed it a couple of time
2020-11-25 08:28:15 <clandmeter> only got bad object
2020-11-25 08:28:25 <clandmeter> the object was also in the cache
2020-11-25 08:28:48 <clandmeter> anyway its disabled now
2020-11-25 08:28:52 <ikke> k
2020-11-25 08:29:54 <clandmeter> i guess the mapping is done in memory as the object file name is serial.
2020-11-26 11:48:06 <ncopa> i have restarted the s390x lxc containers
2020-11-26 11:48:19 <ncopa> Ariadne: seems like mips64 machine is gone again
2020-11-26 12:43:00 <Ariadne> hmm
2020-11-26 12:44:55 <Ariadne> indeed, something is odd
2020-11-26 12:45:50 <Ariadne> connection is up.  hope the SSD has not failed :s
2020-11-27 07:07:46 <ncopa> seems like we are close to finish build of v3.13/main we should probably check that we have disk space on the mirrors
2020-11-27 08:41:23 <clandmeter> you could switch mirrors
2020-11-27 09:24:14 <ncopa> do the new dl-master?
2020-11-27 09:29:38 <fcolista> where is  mirrors.yaml located?
2020-11-27 09:30:04 <fcolista> which repo, I mean
2020-11-27 09:30:28 <ncopa> i think its stored online, not in repo
2020-11-27 09:30:43 <ncopa> https://mirrors.alpinelinux.org/
2020-11-27 09:30:45 <fcolista> ah ok. So if we want to remove a mirror?
2020-11-27 09:31:00 <ikke> Private repo: https://gitlab.alpinelinux.org/alpine/infra/mirrors
2020-11-27 09:31:06 <ikke> that's the source
2020-11-27 09:31:13 <fcolista> so we can make a commit there
2020-11-27 09:31:16 <ikke> yes
2020-11-27 09:31:21 <fcolista> good
2020-11-27 09:31:23 <fcolista> thx ikke 
2020-11-27 09:31:49 <fcolista> what should I do to have access to that repo?
2020-11-27 09:32:50 <ikke> Thought you already had access
2020-11-27 09:32:59 <fcolista> Page Not Found
2020-11-27 09:33:01 <ikke> Yea
2020-11-27 09:33:03 <ikke> I see why
2020-11-27 09:33:10 <fcolista> With GitHub I had
2020-11-27 09:33:16 <fcolista> with gitlab nope
2020-11-27 09:33:36 <ikke> ncopa: no issue in adding fcolista to the alpine group on gitlab, right?
2020-11-27 09:37:57 <ncopa> i have no objections to that
2020-11-27 09:38:29 <ikke> fcolista: You should have access now
2020-11-27 09:38:34 <fcolista> thx ikke 
2020-11-27 09:38:46 <fcolista> yes, I see that now
2020-11-27 09:53:45 <fcolista> ikke, does it looks good?
2020-11-27 09:53:46 <fcolista> https://dpaste.org/j4pG
2020-11-27 09:54:25 <ikke> seems you removed one extra line at the end?
2020-11-27 09:54:47 <fcolista> correct
2020-11-27 09:55:17 <ikke> Oh, sorry, the next entry just starts with a '-'
2020-11-27 09:55:25 <fcolista> otherwise there are two lines 
2020-11-27 09:55:27 <fcolista> yes
2020-11-27 09:56:43 <ikke> dpaste doesn't show it as fixed width, which is not helping to read patches :)
2020-11-27 09:57:08 <ikke> looks good
2020-11-27 09:57:16 <ikke> Had to click 'edit' to get a readable version
2020-11-27 09:57:30 <fcolista> https://tpaste.us/ranV
2020-11-27 09:57:55 <fcolista> maybe this looks better
2020-11-27 09:58:27 <ikke> yes, certainly
2020-11-27 09:58:30 <fcolista> if looks good, it's enough git push?
2020-11-27 09:58:36 <ikke> should be, yes
2020-11-27 09:58:41 <fcolista> Or there's some validation I should do?
2020-11-27 09:58:43 <fcolista> Ok
2020-11-27 09:58:44 <fcolista> thanks!
2020-11-27 09:59:00 <ikke> As long as the yaml is valid :)
2020-11-27 09:59:21 <ikke> There is a cron job that picks up changes
2020-11-27 09:59:24 <fcolista> :)
2020-11-27 09:59:28 <ikke> and renders the new mirror list
2020-11-27 10:03:16 <clandmeter> ncopa: we have 3 new mirrors
2020-11-27 10:03:20 <clandmeter> geo based
2020-11-27 10:03:35 <clandmeter> frankfurt, dallas, tokio
2020-11-27 10:03:48 <clandmeter> atm its rsync.geo.alpinelinux.org
2020-11-27 10:03:59 <ncopa> sweet!
2020-11-27 10:04:02 <clandmeter> we could make the german one master
2020-11-27 10:04:15 <clandmeter> if the load gets too high we could add another
2020-11-27 10:04:31 <clandmeter> storage is 1.2TB
2020-11-27 10:04:32 <ncopa> master is the one we push to
2020-11-27 10:04:50 <clandmeter> yes but the current one is limtied
2020-11-27 10:04:53 <ncopa> i think we can only have one master
2020-11-27 10:05:06 <clandmeter> yes only one master
2020-11-27 10:06:15 <clandmeter> alternative we can transform the uk dedicated server to master
2020-11-27 10:07:23 <clandmeter> if the german on in rsync.geo.a.o would also be the master the load could get too high, thats why i mention we could introduce another one.
2020-11-27 10:07:36 <clandmeter> 1.2T cost 120 p/m
2020-11-27 10:07:52 <clandmeter> the vps is 40 iirc
2020-11-27 10:08:04 <clandmeter> so 160 for a single mirror
2020-11-27 10:08:55 <clandmeter> adding another one would fit the credit
2020-11-27 10:11:40 <ncopa> those rsync.go.a.o are the tier-1 mirrors?
2020-11-27 10:11:53 <clandmeter> That's the Idea
2020-11-27 10:12:36 <ncopa> i kinda like the idea of letting the uk mirror to be a dedicated master
2020-11-27 10:12:49 <ncopa> the uk mirror has pretty big storage?
2020-11-27 10:13:07 <clandmeter> I think so
2020-11-27 10:13:41 <ncopa> whats the hostname for uk mirror? ssh uk.a.o logs me in to nld3-dev
2020-11-27 10:13:51 <clandmeter> Gbr1
2020-11-27 10:13:57 <clandmeter> Dev1
2020-11-27 10:14:54 <ncopa> i like this better and better
2020-11-27 10:15:03 <ncopa> it as 1.8T
2020-11-27 10:15:16 <ncopa> maybe we could move ancient.alpinelinux.org to here too
2020-11-27 10:15:39 <ncopa> as archive for the old releases
2020-11-27 10:19:45 <clandmeter> you could
2020-11-27 10:20:00 <clandmeter> but i think this was already second hand when we got it.
2020-11-27 10:20:17 <clandmeter> something to keep in mind.
2020-11-27 10:21:09 <clandmeter> not sure you keep a copy local of old releases.
2020-11-27 11:28:16 <clandmeter> ikke: https://gitlab.com/gitlab-org/gitlab/-/issues/235756
2020-11-27 11:38:18 <ikke> so we need to upgrade
2020-11-27 12:09:33 <clandmeter> Would be nice
2020-11-27 14:43:49 <clandmeter> ikke: did you write down your steps on how to upgrade?
2020-11-27 14:44:34 <clandmeter> to run the tests
2020-11-27 14:44:46 <ikke> no
2020-11-27 14:45:03 <ikke> cd /srv/compose/gitlab-acceptance-tests
2020-11-27 14:45:07 <ikke> docker-compose run tests
2020-11-27 14:45:26 <clandmeter> but you first restore from a recent backup correct
2020-11-27 14:45:30 <ikke> yes
2020-11-27 14:45:36 <ikke> create a new instance from the last backup
2020-11-27 14:58:11 <ikke> There are some settings you need to change
2020-11-27 14:58:29 <ikke> Otherwise you keep being redirected back to gitlab.a.o :)
2020-11-28 12:41:22 <clandmeter> ikke: ci stages run in its own container?
2020-11-28 12:46:54 <ikke> yes, each job is a separate container
2020-11-28 12:47:11 <ikke> jobs can get files from earlier stages through artifacts
2020-11-30 08:27:41 <clandmeter> ikke: anything i need to turn of when starting the gitlab test instance?
2020-11-30 08:33:58 <ikke> clandmeter: The only thing I turn of is sending mails
2020-11-30 08:34:16 <clandmeter> i was also looking at imap
2020-11-30 08:34:23 <clandmeter> but it uses an internal hostname
2020-11-30 08:35:08 <clandmeter> how do you disable outgoing emails? change hostname?
2020-11-30 08:36:29 <ikke> No, there is a way to just disable it, hold on
2020-11-30 08:37:18 <ikke> In /srv/docker/gitlab/config/gitlab/initializers/smtp_settings.rb
2020-11-30 08:38:47 <ikke> perform_deliveries: false
2020-11-30 08:40:32 <ikke> You need to set that on be Base class
2020-11-30 08:41:52 <clandmeter> right
2020-11-30 08:42:02 <clandmeter> and you said i needed to set soome setting to make itwork?
2020-11-30 08:43:02 <ikke> In the gitlab.yml you need to set the correct hostname
2020-11-30 08:43:13 <ikke> and in the admin settings, you need to fix the redirect 
2020-11-30 08:43:45 <clandmeter> correct hostname is some specific value?
2020-11-30 08:43:52 <ikke> the hostname you are testing with
2020-11-30 08:44:08 <ikke> otherwise some links would redirect you back to production
2020-11-30 08:44:38 <clandmeter> ah i see y ou have gitlab-test in dns
2020-11-30 08:45:07 <ikke> yes
2020-11-30 08:46:09 <clandmeter> nginx needs no change?
2020-11-30 08:46:30 <ikke> no
2020-11-30 08:46:49 <ikke> Oh, traefik needs to know the hostname
2020-11-30 08:46:59 <ikke> otherwise you get a 404
2020-11-30 08:48:29 <clandmeter> yeah im looking at that
2020-11-30 08:49:09 <ikke> So you ned to adjust the labels in docker-compse.yml
2020-11-30 08:49:15 <clandmeter> done
2020-11-30 08:49:20 <ikke> I wanted to make a script to automate all thise :)
2020-11-30 08:50:20 <clandmeter> you are reading my mind
2020-11-30 08:50:28 <ikke> heh :P
2020-11-30 08:50:41 <clandmeter> actually i think we could have 2 compose files
2020-11-30 08:50:51 <ikke> yes
2020-11-30 08:50:54 <ikke> I already tested this
2020-11-30 08:51:16 <clandmeter> and have an additional container to prep things
2020-11-30 08:51:29 <ikke> oh, you mean something different :)
2020-11-30 08:51:35 <ikke> But, hold on
2020-11-30 08:52:34 <ikke> check /srv/compose/aports-qa-bot on gbr2-dev1
2020-11-30 08:54:48 <clandmeter> and what should i see?
2020-11-30 08:55:02 <ikke> docker-compose.override.yml
2020-11-30 08:55:36 <clandmeter> yes?
2020-11-30 09:02:59 <ikke> Just a nice way to split the general application config from the deployment config
2020-11-30 09:03:17 <clandmeter> yes, i mention that the other day.
2020-11-30 09:03:35 <clandmeter> but we need some way to edit the original configs.
2020-11-30 09:03:39 <clandmeter> or just use sed
2020-11-30 09:04:10 <ikke> https://pkgs.alpinelinux.org/package/edge/community/x86_64/yq maybe?
2020-11-30 09:04:17 <ikke> never used it
2020-11-30 09:04:32 <clandmeter> or use patches?
2020-11-30 09:04:40 <ikke> works as well
2020-11-30 09:04:43 <mps> docker-compose allows overrides (overlays) config files?
2020-11-30 09:04:47 <ikke> yes
2020-11-30 09:05:27 <ikke> https://docs.docker.com/compose/extends/
2020-11-30 09:07:04 <mps> not easy to collaborate with you two, always something new to read and learn ;)
2020-11-30 09:07:11 <ikke> heh
2020-11-30 12:26:51 <clandmeter> ikke: i think we have a few challenges with gitlab
2020-11-30 12:26:59 <ikke> Just a few? :P
2020-11-30 12:27:01 <clandmeter> looks like 13.6 needs ruby 2.7
2020-11-30 12:27:14 <ikke> right
2020-11-30 12:27:19 <clandmeter> and need to upgrade pg
2020-11-30 12:27:22 <ikke> yes
2020-11-30 12:27:44 <clandmeter> i want at least .3 soon
2020-11-30 12:27:59 <ikke> Sounds like a good idea
2020-11-30 12:28:05 <ikke> does 13.3 give challenges?
2020-11-30 12:28:16 <clandmeter> always :)
2020-11-30 12:28:21 <ikke> heh
2020-11-30 12:28:33 <clandmeter> but not obvious ones yet
2020-11-30 12:28:41 <ikke> I have 2 days off, today and tomorrow, so I have some time to work on things :)
2020-11-30 12:28:56 <clandmeter> i dont mind to skip a version
2020-11-30 12:29:28 <ikke> We're on 13.2 right now, correct?
2020-11-30 12:29:32 <clandmeter> nod
2020-11-30 12:29:40 <clandmeter> .3 fixes the keys thingy
2020-11-30 12:29:49 <clandmeter> recent .3 at least
2020-11-30 12:29:52 <ikke> ok
2020-11-30 12:30:06 <ikke> Maybe we should focus on 13.3 first?
2020-11-30 12:30:09 <clandmeter> let me check the changelog for 4 and 5
2020-11-30 12:30:34 <clandmeter> i would like to upgrade pg before we move to ruby 2.7
2020-11-30 12:30:48 <ikke> ok
2020-11-30 12:30:51 <clandmeter> i guess we have too due to versions
2020-11-30 12:30:58 <clandmeter> to*
2020-11-30 12:31:29 <clandmeter> ah it was also 13.6
2020-11-30 12:31:55 <ikke> yes, so 13.6 will be ruby 2.7 and pg 12
2020-11-30 12:31:56 <clandmeter> so 13.6 is new pg and new ruby
2020-11-30 12:32:16 <ikke> :P
2020-11-30 12:32:19 <ikke> gmta
2020-11-30 12:32:57 <clandmeter> pg should be okish, just a bit more work.
2020-11-30 12:33:15 <clandmeter> but i suspect 2.7 will give us headaches 
2020-11-30 12:33:28 <ikke> Why?
2020-11-30 12:33:39 <clandmeter> because ruby is a monster 
2020-11-30 12:33:45 <ikke> afaik 2.7 just introduces more deprecations
2020-11-30 12:34:09 <clandmeter> have not followed aports much
2020-11-30 12:34:19 <clandmeter> any known issues due to ruby upgrade?
2020-11-30 12:34:38 <ikke> Haven't heard anything
2020-11-30 12:34:46 <ikke> no issues reported on gitlab afaik
2020-11-30 12:34:48 <clandmeter> i hope my feeling is wrong
2020-11-30 12:34:57 <clandmeter> but ruby has a lot of deps
2020-11-30 12:35:04 <clandmeter> i mean gems
2020-11-30 12:35:20 <clandmeter> gitlab has a lot of gems and thats what makes me nervous 
2020-11-30 12:35:22 <ikke> My (limited) experience with 2.7 is mostly more noice due to deprecations
2020-11-30 12:35:35 <ikke> so up-to-date dependencies help there
2020-11-30 12:35:42 <ikke> (They are preparing for 3.0)
2020-11-30 12:36:16 <ikke> github switched to 2.7 already, and they did a lot of legwork getting rid of all the deprecation in warnings
2020-11-30 12:36:24 <ikke> in gems*
2020-11-30 12:49:20 <clandmeter> 13.6 also wants a newer git version
2020-11-30 12:51:28 <ikke> Newer than 2.26?
2020-11-30 12:52:07 <clandmeter> yes
2020-11-30 12:52:08 <clandmeter> https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/install/requirements.md
2020-11-30 12:52:18 <clandmeter> rom GitLab 13.6:
2020-11-30 12:52:18 <clandmeter> Git 2.29.x and later is required.
2020-11-30 12:52:26 <ikke> We'd have to wait for alpine 3.13 or switch to edge
2020-11-30 12:53:04 <clandmeter> by the time we are ready, 3.13 will be deprecated ;-) 
2020-11-30 12:54:52 <clandmeter> i was playing with the ruby console
2020-11-30 12:55:01 <clandmeter> but starting it takes forever
2020-11-30 12:55:42 <clandmeter> i guess any cmd exec by bundler has that delay
2020-11-30 13:07:54 <clandmeter> ikke: what if we upgrade like:  13.2 => 13.4  with pg update. and 13.4 => 13.6 with ruby 2.7 and recent git on alpine 13.3
2020-11-30 13:08:35 <ikke> sounds like a good idea
2020-11-30 13:08:40 <ikke> do you have experience with pg upgrades?
2020-11-30 13:08:56 <clandmeter> i read a bit on the subject
2020-11-30 13:11:37 <clandmeter> there are 2 ways iirc 
2020-11-30 13:14:08 <clandmeter> im reading it again,  its kind of a headache...
2020-11-30 13:14:26 <ikke> Is it feasible to export + import?
2020-11-30 13:15:22 <clandmeter> you need to dump with new bins
2020-11-30 13:21:08 <ikke> and dump as sql?
2020-11-30 22:51:19 <mps> what are kernel versions on builders?
2020-11-30 22:51:30 <mps> and what on CIs
2020-11-30 22:51:42 <mps> do we have these data somewhere
2020-11-30 22:54:36 <ikke> No, I don't think so
2020-11-30 22:54:42 <ikke> most builders are still on alpine 3.10
2020-11-30 22:54:54 <mps> huh
2020-11-30 22:55:09 <mps> that means 4.19?
2020-11-30 22:56:48 <ikke> x86(_64) are new, they are on 5.4.43
2020-11-30 22:57:53 <ikke> hmm, seems moth builders are already on 5.4.x