2020-08-03 14:47:49 <Cogitri> https://gitlab.gnome.org/GNOME/epiphany/-/issues/service_desk GNOME's gitlab has a feature for users to create issues without an email (a bot posts the issue as confidental), do we want something similar?
2020-08-03 14:48:06 <Cogitri> Might be nice for having a security@a.o email address
2020-08-03 14:49:04 <Cogitri> s/without an email/without a gitlab account/
2020-08-03 14:49:04 <alpine-meetbot> Cogitri meant to say: https://gitlab.gnome.org/GNOME/epiphany/-/issues/service_desk GNOME's gitlab has a feature for users to create issues without a gitlab account (a bot posts the issue as confidental), do we want something similar?
2020-08-03 14:49:28 <Cogitri> Oh wow, that worked, good bot
2020-08-03 15:04:12 <ikke> Uses service desk, which we need to enable somehow
2020-08-03 18:11:38 <clandmeter> maxice8: +¹ on the move of mkmr
2020-08-03 18:12:34 <ikke> heh
2020-08-03 18:12:44 <ikke> did my comment wake you up? :P
2020-08-03 18:17:34 <maxice8> @ikke can you make gitlab.alpinelinux.org/Leo/mkmr public ?
2020-08-03 18:17:49 <ikke> sure
2020-08-03 18:18:46 <ikke> done
2020-08-03 18:22:41 <mps> yes, I need to learn mkmr. But time :\
2020-08-03 18:23:26 <ikke> I wanted to look into mkmr
2020-08-03 18:23:31 <ikke> hence me commenting on that issue
2020-08-03 18:23:52 <ikke> But it needs one change before it's usable to me, because I don't use origin as a remote
2020-08-03 18:25:20 <maxice8> What do you use as a remote ?
2020-08-03 18:25:27 <ikke> gitlab
2020-08-03 18:25:39 <maxice8> that points to ikke/aports ?
2020-08-03 18:25:43 <ikke> yes
2020-08-03 18:25:55 <maxice8> mkmr --origin=gitlab
2020-08-03 18:26:03 <ikke> yes, but I don't want to specify that all the time :)
2020-08-03 18:26:31 <ikke> So was thinking adding configuration options for that
2020-08-03 18:26:35 <maxice8> heh
2020-08-03 18:26:42 <maxice8> I was thinking about that right now
2020-08-03 18:27:01 <ikke> You could just use git config to store that
2020-08-03 18:28:58 <maxice8> If required now is the time to split the configuration into 2 files
2020-08-03 19:40:13 <maxice8> problem is that everything depends on a remote being set, even getting a configuration file path and which section of the configuration file depends on a remote being given to us
2020-08-03 19:42:51 <ikke> how so?
2020-08-03 19:43:42 <maxice8> we need a remote to guess the URL we will interact it
2020-08-03 19:45:00 <maxice8> instead of giving --upstream=https://gitlab.alpinelinux.org/alpine/aports every time we just look at the remote 'upstream' or whatever you pass with --upstream=<REMOTE_NAME> and get the URL from there
2020-08-03 19:45:15 <ikke> yes. understood
2020-08-03 19:45:34 <ikke> But that doesn't prevent us from configuring what remote to use, right?
2020-08-03 19:45:55 <ikke> And maybe even explicitly specifying the API endpoint if required for some reason
2020-08-03 19:50:04 <maxice8> you can pass it every time via --upstream, but currently it needs the remote before starting because it uses the URI for the name of the section in the config file
2020-08-03 19:50:11 <maxice8> if you look in $XDG_CONFIG_HOME/etc/mkmr/config you will see
2020-08-03 19:50:14 <maxice8> [gitlab.alpinelinux.org]
2020-08-03 19:50:20 <maxice8> url = https://gitlab.alpinelinux.org
2020-08-03 19:50:27 <maxice8> personal_token = XXXXXXXXXXXXX
2020-08-03 19:50:56 <ikke> ok, but couldn't you check the git config even before that?
2020-08-03 19:51:25 <maxice8> it is possible
2020-08-03 19:51:41 <maxice8> or we can split into a XDG_CONFIG_HOME/mkmr/remotes that has only configuration for the remotes
2020-08-03 19:51:56 <maxice8> and a XDG_CONFIG_HOME/mkmr/config which has configuration for mkmr remotes
2020-08-03 19:53:29 <ikke> watever works
2020-08-03 20:11:25 <maxice8> I'll go with the .git/config route
2020-08-03 20:11:45 <maxice8> create section called [mkmr] write an option called remote with the value wanted
2020-08-03 20:16:04 <ikke> sounds reasonable :)
2020-08-03 20:26:16 <maxice8> It is kinda messy but eh
2020-08-03 21:35:27 <eth01> 
ACTION waves

2020-08-04 09:17:54 <clandmeter> ikke: did you add the recent mirror requests?
2020-08-04 09:18:55 <ikke> not yet, keep forgetting it
2020-08-04 09:19:30 <clandmeter> ill try to update status
2020-08-04 09:20:30 <clandmeter> ikke: maybe we should remove non existing mirrors
2020-08-04 09:20:37 <clandmeter> which are just cnames
2020-08-04 09:20:43 <clandmeter> like uk
2020-08-04 09:21:00 <ikke> Yeah, makes sense
2020-08-04 09:21:06 <ikke> keep the cnames, but don't advertise them
2020-08-04 09:21:15 <clandmeter> nod
2020-08-04 09:23:32 <clandmeter> i bumped into https://github.com/shellhub-io/shellhub recently.
2020-08-04 09:23:39 <clandmeter> the concept is interesting.
2020-08-04 09:45:53 <clandmeter> bummer, releases.json does not list the repos.
2020-08-04 09:50:18 <clandmeter> looks like master mirror has storage limitations
2020-08-04 09:51:59 <ikke> is that the reason that we have package mismatches?
2020-08-04 10:18:33 <clandmeter> i think jirutka removed some older stuff
2020-08-04 10:18:41 <clandmeter> so it has some space
2020-08-04 10:20:06 <clandmeter> ncopa: any specific reason you didnt add repos to releases.json?
2020-08-04 11:13:37 <ikke> I freed up a bit of space from my container
2020-08-04 11:13:41 <ikke> maybe others could do as well ^^^
2020-08-04 11:14:03 <mps> which host?
2020-08-04 11:14:12 <ikke> nld5, x86_64 lxc host
2020-08-04 11:15:11 <mps> I don't use much space on it, except temporarily when building big pkgs
2020-08-04 11:15:54 <clandmeter> how much is mine?
2020-08-04 11:16:54 <ikke> which one? :P
2020-08-04 11:16:54 <clandmeter> 58g
2020-08-04 11:17:14 <clandmeter> i think i normally only use 1
2020-08-04 11:17:28 <ikke> mine are 5.5G combined
2020-08-04 11:17:46 <ikke> sorry, 15G, forgot one
2020-08-04 11:18:15 <clandmeter> im checking htem all
2020-08-04 11:19:46 <mps> my is 1.2GB :)
2020-08-04 11:24:01 <clandmeter> mps: my stats say your container use 21.2G
2020-08-04 11:24:39 <mps> aarch64?
2020-08-04 11:24:48 <clandmeter> nld5
2020-08-04 11:24:49 <ikke> x86_64
2020-08-04 11:24:54 <mps> huh
2020-08-04 11:25:01 <mps> impossible
2020-08-04 11:25:12 <ikke> mps-edge-x86_64
2020-08-04 11:25:52 <mps> as root I did 'cd / ; du -sh' and got 1.2G
2020-08-04 11:26:07 <clandmeter> maybe check tmp dir?
2020-08-04 11:26:11 <ikke> 21.2G   mps-edge-x86_64/
2020-08-04 11:26:24 <mps> ok, will check
2020-08-04 11:28:04 <mps> hah, I didn't seen 2 in 21.2G ;)
2020-08-04 11:28:56 <clandmeter> ill cleanup mine as well 
2020-08-04 11:28:58 <mps> clandmeter: :D
2020-08-04 11:29:05 <clandmeter> :D
2020-08-04 11:29:15 <clandmeter> mps: i couldnt resist, sorry.
2020-08-04 11:29:23 <mps> np
2020-08-04 11:29:48 <clandmeter> ikke: there is still some space left on the vg
2020-08-04 11:29:54 <clandmeter> in case you need it
2020-08-04 11:30:12 <ikke> clandmeter: ok
2020-08-04 11:30:20 <ikke> but cleanup is better :)
2020-08-04 11:30:25 <mps> you have to hurry, cellar becoming empty because I need more 'sedatives' caused by induced fear on pandemic :)
2020-08-04 11:30:39 <ikke> clandmeter: see what you did :P
2020-08-04 11:31:10 <clandmeter> yes, everytime we see you missed the % sign :p
2020-08-04 11:32:27 <clandmeter> mps: the pandemic is killing my shipment from USA, send some wine to them!
2020-08-04 11:34:42 <mps> I would send some bottles to you but DHL boy will arrive with airplane at the end of august, and bottles are forbidden by air companies
2020-08-04 11:34:58 <clandmeter> dhl boy :D
2020-08-04 11:36:19 <clandmeter> maybe they should allow alcohol again on flights to cover the losses 
2020-08-04 11:36:24 <mps> yes, he doesn't like to be in quarantine for about 14 days on every border he have to cross if he go by car :)
2020-08-04 11:36:39 <clandmeter> ah ok
2020-08-04 11:39:47 <mps> can I enter plane if I drink a bottle of brandy before entering?
2020-08-04 11:40:23 <mps> ofc, if I find entry point :)
2020-08-04 12:21:14 <ncopa> clandmeter: i dont remember why i didnt add repos to releases.json
2020-08-05 04:46:26 <ikke> clandmeter: apparently mirrors.a.o hasnt updated since yesterday
2020-08-05 08:08:52 <clandmeter> ikke: should be updating again
2020-08-05 08:17:06 <clandmeter> i think we should probably skip non supported releases on mirror status
2020-08-05 08:18:57 <clandmeter> ncopa: are you able to introduce repos in rel.json?
2020-08-05 08:24:03 <ncopa> i can look into it
2020-08-05 08:24:14 <ncopa> what is it supposed to be used for?
2020-08-05 08:24:38 <clandmeter> so i can construct all repo urls
2020-08-05 08:24:50 <clandmeter> its used for mirror status
2020-08-05 08:25:48 <ncopa> so, for v3.x: repos: main community
2020-08-05 08:26:03 <ncopa> and for edge: main community testing
2020-08-05 08:26:13 <clandmeter> yes
2020-08-05 08:26:17 <clandmeter> but we also have 2.x
2020-08-05 08:26:29 <clandmeter> but i think i can skip those
2020-08-05 08:26:34 <ncopa> yeah
2020-08-05 08:26:55 <clandmeter> i think it makes sense to only support supported branches for these types of services.
2020-08-05 08:26:58 <ncopa> i wonder if we need to set eol_date for repos
2020-08-05 08:27:05 <ncopa> yes
2020-08-05 08:27:14 <clandmeter> didnt we do that already?
2020-08-05 08:27:24 <ncopa> we do for releases, but not for repos
2020-08-05 08:27:28 <clandmeter> ah ok
2020-08-05 08:27:29 <ncopa> i mean, we dont have repos
2020-08-05 08:27:42 <ncopa> because eol_date is different for main and community
2020-08-05 08:27:47 <clandmeter> nod
2020-08-05 08:28:04 <clandmeter> if the diff changes, does it change for all hostory?
2020-08-05 08:28:23 <clandmeter> now its 6m and 2y?
2020-08-05 08:28:42 <ncopa> well, it shouldnt change for all history
2020-08-05 08:28:51 <clandmeter> ok
2020-08-05 08:28:54 <ncopa> which is why i think we need a db of historical data
2020-08-05 08:29:08 <clandmeter> so ones set we keep a reference about it.
2020-08-05 08:31:37 <ncopa> something like this: https://tpaste.us/6VNn
2020-08-05 08:32:09 <clandmeter> nod
2020-08-05 08:32:36 <ncopa> if the eol_date is not set, fall back to the eol_date for the release branch
2020-08-05 08:32:54 <clandmeter> why would it not be set?
2020-08-05 08:33:17 <ncopa> to avoid duplicate data
2020-08-05 08:33:27 <ncopa> to simplify
2020-08-05 08:33:36 <clandmeter> ok, so your example would change.
2020-08-05 08:33:42 <ncopa> yes
2020-08-05 08:33:49 <clandmeter> :)
2020-08-05 08:34:05 <ncopa> https://tpaste.us/V4zd
2020-08-05 08:34:28 <clandmeter> lgtm
2020-08-05 08:36:01 <ncopa> or maybe even: https://tpaste.us/mEW7
2020-08-05 08:47:52 <clandmeter> i like the first one more
2020-08-05 08:48:37 <ncopa> or we could let the reponame be the key in the key value pari
2020-08-05 08:48:47 <ncopa> s/pari/pair/
2020-08-05 08:48:47 <alpine-meetbot> ncopa meant to say: or we could let the reponame be the key in the key value pair
2020-08-05 08:49:34 <ncopa> https://tpaste.us/Rzm9
2020-08-05 08:50:01 <ncopa> i guess the first option is the most verbose way to do it
2020-08-05 08:50:22 <clandmeter> first or last is ok for me.
2020-08-05 08:51:29 <clandmeter> in such tiny data, verbosity does not matter and probably improves readability.
2020-08-05 08:54:20 <ikke> yaml is already structured, so no need to embed structured data in values
2020-08-05 08:54:32 <ikke> (imho :))
2020-08-05 08:54:49 <ncopa> i already do that for arches
2020-08-05 08:55:30 <ikke> is there a reason for that?
2020-08-05 08:55:45 <ncopa> easier to write
2020-08-05 08:56:59 <ncopa> btw
2020-08-05 08:57:07 <ncopa> what is the eol date for community repo?
2020-08-05 08:57:13 <ncopa> branch date of next version right?
2020-08-05 08:57:40 <clandmeter> that how i understand it.
2020-08-05 08:58:33 <ncopa> that means that eol_date for 3.12 is currently unknown
2020-08-05 08:59:04 <ncopa> i mean eol_date for v3.12/community  is currently unknown
2020-08-05 08:59:16 <clandmeter> right
2020-08-05 08:59:42 <ncopa> i think we can set it to 2020-11-05 anyway
2020-08-05 08:59:50 <ncopa> and when we do v3.13 we change it
2020-08-05 09:00:05 <ncopa> better to extend the support date than shorten it
2020-08-05 09:00:50 <ncopa> like this: https://tpaste.us/9NYn
2020-08-05 09:03:57 <ncopa> some of the historical data is wrong
2020-08-05 10:25:38 <ncopa> ikke: I'm powering off bld1 now
2020-08-05 10:27:20 <ikke> OK, thanks 
2020-08-05 10:27:43 <ikke> Hmm, probably will get alerts here though
2020-08-05 10:27:53 <ikke> Could you wait about 10 minutes? 
2020-08-05 10:27:58 <ncopa> yup
2020-08-05 10:28:13 <ncopa> sleep 15m && poweroff ?
2020-08-05 10:29:46 <MY-R> speaking of mirror page :D that order is still confusing v3.11 -> v3.12 -> v3.2 -> v3.3
2020-08-05 10:35:55 <ikke> ncopa: should be fine now
2020-08-05 11:04:25 <ncopa> ok. im powering it iff
2020-08-05 11:04:28 <ncopa> off*
2020-08-05 11:04:31 <ncopa> its off
2020-08-05 11:21:11 <clandmeter> MY-R: we accept PR for reordering :)
2020-08-05 11:40:53 <MY-R> for sure it is something very tricky and that is why nobody wanna touch it! :D
2020-08-05 16:17:29 <mps> could some add more time for !10288 on CIs
2020-08-05 16:18:45 <Cogitri> mps: The submitter of the MR has to increase it in their fork
2020-08-05 16:19:32 <mps> ah, he mentioned you there, you can give advice :)
2020-08-05 16:25:20 <mps> Cogitri: I would, but I don't know how to increase time in MR, till now I didn't needed it
2020-08-05 16:27:56 <Cogitri> Go to https://gitlab.alpinelinux.org/$username/aports/-/settings/ci_cd and set Timeout to a higher value
2020-08-05 16:28:42 <mps> aha, thanks
2020-08-05 19:28:32 <ncopa> ikke:
2020-08-05 19:28:33 <ncopa> Filesystem      Size  Used Avail Use% Mounted on
2020-08-05 19:28:34 <ncopa>  /dev/simfs      1.0T  889G  136G  87% /
2020-08-05 19:28:45 <ncopa> i thikn we are good til tomorrow at least
2020-08-05 19:29:37 <ikke> nod
2020-08-05 19:33:04 <maldridge> 
ACTION steals your space to download more ram

2020-08-05 19:33:36 <ikke> :D
2020-08-05 19:35:41 <clandmeter> ncopa: did you see the gitlog on that system?
2020-08-05 19:35:52 <clandmeter> seems jirutka clean up some stuff 
2020-08-05 19:35:57 <clandmeter> not sure what he did 
2020-08-05 19:38:05 <clandmeter> i noticed when i searched for apkindex files and got errors about .tmp dirs from rsync which means it was probably out of space.
2020-08-06 06:20:29 <ncopa> morning. i didnt see the gitlog
2020-08-06 06:43:58 <clandmeter> ncopa: morning, its displayed as motd
2020-08-06 06:44:18 <ncopa> ok
2020-08-06 06:44:22 <ncopa> yes i saw that
2020-08-07 20:24:45 <ikke> So now it's my fault, huh? 😋
2020-08-07 20:24:57 <clandmeter> yes you restarted that box :)
2020-08-07 20:27:14 <clandmeter> algitbot: ping
2020-08-07 20:28:44 <ikke> That error on the mqtt-exec service is quite vague btw
2020-08-07 20:28:45 <clandmeter> should be ok n ow
2020-08-07 20:28:54 <clandmeter> which one?
2020-08-07 20:29:06 <clandmeter> on old git.a.o?
2020-08-07 20:29:18 <ikke> A stack trace without a clear error 
2020-08-07 20:29:27 <ikke> No, deu1-dev1 
2020-08-07 20:29:40 <clandmeter> i have no idea what you are talking about :)
2020-08-07 20:30:30 <clandmeter> mps: let me know if algitbot still does not play nice
2020-08-07 20:31:56 <ikke> docker-compose logs --tail=20 mqtt-exec
2020-08-07 20:37:27 <clandmeter> i think those are normal :)
2020-08-07 20:37:28 <mps> clandmeter: just saw first one merge announced there
2020-08-07 20:38:03 <clandmeter> ikke: the bot is not yet online but exec wants to send some messages which are retained from mqtt
2020-08-08 08:00:10 <Cogitri> Seems like beta.docs.a.o doesn't have a valid https cert
2020-08-08 08:03:36 <ikke> Yeah, we don't have a dedicated cert for beta.docs.a.o, and the wildcard doesn't cover it
2020-08-08 08:03:42 <ikke> https://docs.alpinelinux.org contains everything though
2020-08-08 09:15:36 <Cogitri> Okie
2020-08-10 08:10:15 <HRio> whats up with the armv7 builder for v3.12?
2020-08-10 08:11:34 <ikke> stuck on fail2ban?
2020-08-10 08:11:36 <HRio> seems mips64 for v3.12 is stuck as well
2020-08-10 08:11:49 <HRio> but on pulling git instead
2020-08-10 08:12:23 <HRio> ikke: fail2ban, could be the tests that are stuck
2020-08-10 08:12:34 <HRio> can we kill them and retry?
2020-08-10 08:12:34 <ikke> yes
2020-08-10 08:12:39 <ikke> done
2020-08-10 08:13:31 <HRio> the fail2ban test suite sometimes hangs in gitlab ci as well
2020-08-10 09:06:45 <ikke> HRio: fixed mips64 on 3.12 as well
2020-08-10 09:46:57 <Cogitri> I think edge/aarch64 is stuck on Libreoffice?
2020-08-10 09:47:22 <Cogitri> And I'm not sure if openjdk* just takes really long to compile on the stable branches or if that's stuck again too
2020-08-10 09:48:11 <ikke> openjdk took at least a long time in CI
2020-08-10 10:08:06 <HRio> ikke: thanks
2020-08-10 10:08:41 <ikke> Cogitri: seems like libreoffice is indeed stuck
2020-08-10 10:08:52 <ikke> no log updates for 10 minutes
2020-08-10 11:45:27 <Cogitri> Okie, let's hope it goes through next time
2020-08-10 11:45:33 <Cogitri> Thanks for checking
2020-08-10 11:48:53 <ikke> openjdk seems to still continue, at least on one builder, so I assume it just takes very long
2020-08-10 11:56:09 <Cogitri> Okie. I think previous versions got stuck sometimes though
2020-08-10 11:56:31 <ikke> yeah, might be
2020-08-10 14:31:45 <kunkku> something is wrong with the 3.9 builders
2020-08-10 14:35:48 <Cogitri> ikke: Can you make Cogitri/apk-polkit-rs public?
2020-08-10 16:30:17 <ncopa> kunkku: i think its distfiles that got bad archive for some reason
2020-08-10 16:32:49 <ncopa> https://tpaste.us/DD6X
2020-08-11 07:26:45 <clandmeter> ncopa: did you add repos to releases.json?
2020-08-11 07:26:54 <clandmeter> and good morning to all 
2020-08-11 07:42:27 <maldridge> morning clandmeter 
2020-08-11 07:42:35 <ikke> morning
2020-08-11 07:43:15 <clandmeter> maldridge: i guess its night for you?
2020-08-11 07:43:59 <maldridge> eh, UGT
2020-08-11 07:44:14 <maldridge> and past midnight, so I claim morning
2020-08-11 07:44:30 <clandmeter> :D
2020-08-11 07:44:34 <ikke> :D
2020-08-11 07:45:08 <clandmeter> its only morning with a big cup of coffee
2020-08-11 07:45:18 <maldridge> ah, then its never morning for me
2020-08-11 07:45:22 <ikke> did anyone say coffee?
2020-08-11 07:45:24 <ikke> \
2020-08-11 07:46:57 <clandmeter> you can replace it with tea, but please dont put any milk inside :)
2020-08-11 07:52:27 <ikke> clandmeter: don't say that to a brittish person :P
2020-08-11 08:13:31 <clandmeter> ikke: i first scanned all hostmasks here and i think we are safe.
2020-08-11 08:13:50 <ikke> hehe
2020-08-12 17:27:31 <ikke> ... 
2020-08-13 00:53:07 <Hello71> seems that https://mirrors.alpinelinux.org/ is still broken on ipv6
2020-08-13 06:15:03 <clandmeter> Hello71: i am not able to check here but maybe the system reboot killed something. 
2020-08-13 06:15:09 <clandmeter> ikke:  can you check it?
2020-08-13 06:16:40 <mps> goede morgen
2020-08-13 06:17:09 <mps> is the aarch64 edge builder stuck on libreoffice
2020-08-13 06:20:20 <ikke> clandmeter: yes, I already took a brief look (someone reported it on gitlab)
2020-08-13 06:34:03 <Ariadne> it appears our aarch64 builder may be MIA
2020-08-13 06:34:11 <Ariadne> oh, mps already reported it (:
2020-08-13 06:34:14 <ikke> heh
2020-08-13 06:34:26 <Ariadne> haha maybe i should read :P
2020-08-13 06:36:41 <ikke> killed the processes
2020-08-13 09:29:56 <ikke> I sent an e-mail to iweb 
2020-08-13 09:30:14 <ikke> it's constantly flapping
2020-08-13 10:49:59 <Cogitri> What's the status for a test instance of gitlab for testing the bot? 
2020-08-13 11:21:34 <ikke> I will try to set it up tonight
2020-08-13 11:22:09 <Cogitri> Great, thanks :) 
2020-08-13 12:07:40 <ncopa> im moved build-3-9-x86_64 no nld9-dev1
2020-08-13 12:07:51 <ncopa> s/no/to/
2020-08-13 12:07:52 <alpine-meetbot> ncopa meant to say: im moved build-3-9-x86_64 to nld9-dev1
2020-08-13 12:08:09 <ikke> ack
2020-08-13 12:08:49 <ncopa> i rsyncing distfiles data now, and the bulid logs
2020-08-13 12:10:19 <ncopa> build-3-10-x86_64 successfully moved too
2020-08-13 12:10:52 <ncopa> i'm moving 3.11 and 3.12 now
2020-08-13 13:07:34 <ncopa> build-3-1[12]-x86_64 is up and running on the nld9-dev1 now
2020-08-13 13:07:48 <ncopa> im cleaning up build-edge a bit and will soon move that over too
2020-08-13 13:07:57 <ncopa> then only distfiles is left
2020-08-13 13:09:17 <ncopa> i thikn we need a dedicated distfiles service, that catches git pushes and only fetches all the needed source files
2020-08-13 13:09:39 <ncopa> ideally the builders should wait til distfiles has downloaded them
2020-08-13 13:09:47 <ikke> sounds complicated :)
2020-08-13 13:09:52 <ncopa> yes :)
2020-08-13 13:10:08 <ncopa> and the services needs to delete old sources etc
2020-08-13 13:10:28 <ncopa> keep the sources for stable branches forever
2020-08-13 13:11:10 <ncopa> we also need somethign similar for build logs
2020-08-13 13:11:26 <ikke> Would be nice if buildlogs could be streamed
2020-08-13 13:11:34 <ncopa> indeed
2020-08-13 13:11:44 <ikke> now we only have that for x86*
2020-08-13 13:12:14 <ncopa> i dont know what the simplest solution for that would be
2020-08-13 13:12:25 <ncopa> maybe stream to redis?
2020-08-13 13:12:44 <ncopa> once the stream is complete, it should be downloaded and archives some place
2020-08-13 13:13:34 <ikke> Does redis have primitives for streaming? Don't see anything relevant
2020-08-13 13:14:21 <ncopa> i think redis has
2020-08-13 13:14:30 <ncopa> but im open to other solutions
2020-08-13 13:14:42 <ncopa> i dont know what things like travis and gitlab does
2020-08-13 13:14:54 <ncopa> would be interesting to know how they solve it
2020-08-13 13:14:57 <ikke> https://redis.io/topics/streams-intro
2020-08-13 13:16:07 <ncopa> i alos found out that logfiles are a bit complicated. there have been logfile lines that are bigger than 2MB iirc
2020-08-13 13:16:41 <ikke> ouch
2020-08-13 13:17:17 <ikke> I cannot phantom what would generate a single line that is 2MB large
2020-08-13 13:18:09 <ikke> ugh, s/phantom/???
2020-08-13 13:19:09 <ikke> s/cannot phantom/cannot fathom/
2020-08-13 13:19:10 <alpine-meetbot> ikke meant to say: I cannot fathom what would generate a single line that is 2MB large
2020-08-13 13:32:46 <Hello71> https://mirrors.alpinelinux.org/ ipv6 is still broken?
2020-08-13 13:34:00 <Hello71> hm, someone raised it in #-linux
2020-08-13 13:34:10 <ikke> yes, will look at it tonight
2020-08-13 13:39:07 <ikke> clandmeter: can we arrange ipv6 for nld3? Then we can properly monitor it via zabbix
2020-08-13 13:43:00 <ikke> It already has one assigned I see, so we just need to make sure it is used
2020-08-13 15:50:28 <ikke> clandmeter: docker just anounced new ToS, which includes a maximum retention for docker images of 6 months for free plans
2020-08-13 16:36:32 <clandmeter> Hmm
2020-08-13 16:36:47 <clandmeter> That sound like trouble
2020-08-13 16:37:13 <ikke> The FAQ says they look at activity per image
2020-08-13 16:37:29 <ikke> So I think as long as your keep updating images, it should be fine
2020-08-13 16:37:56 <ikke> https://www.docker.com/pricing/retentionfaq
2020-08-13 16:38:10 <ikke> "An inactive image is a container image that has not been either pushed or pulled from the image repository in 6 or months."
2020-08-13 16:39:09 <ikke> alpinelinux/mariadb hasn't been updated recently
2020-08-13 16:39:35 <ikke> turbo-paste
2020-08-13 16:39:41 <ikke> lua-turbo
2020-08-13 16:39:47 <ikke> docker-alpine
2020-08-13 16:43:58 <ikke> clandmeter: We can make sure they are in gitlab and set to update weekly, then we should be fine in any case
2020-08-13 20:31:30 <mps> hmm s390x CI have problem https://gitlab.alpinelinux.org/mps/aports/-/jobs/185814
2020-08-14 06:14:17 <mps> looks like the aarch64 edge builder stuck on libreoffice again
2020-08-14 07:19:09 <ncopa> let me have a look at it
2020-08-14 07:22:06 <ncopa> there are java processes using 32.4G memory
2020-08-14 07:23:06 <ncopa> >>> libreoffice: Building community/libreoffice 6.4.5.2-r0 (using abuild 3.6.0-r1) started Thu, 13 Aug 2020 18:34:21 +0000
2020-08-14 07:23:21 <ncopa> Fri Aug 14 07:23:10 UTC 2020
2020-08-14 07:23:42 <maldridge> "there are java processes using 32.4G memory" sounds like my minecraft host
2020-08-14 07:24:03 <ncopa> have been working for ́almost 13 hours
2020-08-14 07:25:25 <ncopa> it has more than enough memory
2020-08-14 07:26:10 <ncopa> hum
2020-08-14 07:26:21 <ncopa> i wonder if this is deadlock in java
2020-08-14 08:32:27 <clandmeter> ncopa: i think i already have a copy of distfiles on a linode box
2020-08-14 08:32:44 <clandmeter> if you are interested in that.
2020-08-14 08:34:28 <clandmeter> ikke: right, i read that new tos incorrectly. 
2020-08-14 08:34:46 <clandmeter> so it kind of means trouble for older versions i guess?
2020-08-14 08:34:53 <ncopa> clandmeter: where can i find it?
2020-08-14 08:35:02 <ikke> What I understood, it's about the entire 'image'
2020-08-14 08:35:05 <ncopa> i suppose it needs to be resynced
2020-08-14 08:35:13 <ikke> so older tags should remain, as long as the image is updated
2020-08-14 08:35:15 <ikke> but not sure
2020-08-14 08:35:27 <ikke> maybe it *is* about older versions
2020-08-14 08:35:37 <clandmeter> hmm
2020-08-14 08:35:39 <clandmeter> thats weird
2020-08-14 08:35:48 <clandmeter> i can login to linode without 2fa
2020-08-14 08:36:21 <clandmeter> LBlaboon: ^?
2020-08-14 08:36:26 <clandmeter> did i hold it wrong?
2020-08-14 08:38:38 <clandmeter> ncopa: ssh root@deu5-dev1.alpinelinux.org
2020-08-14 08:39:19 <clandmeter> but it synced all files to same directory (no branch separation) 
2020-08-14 08:46:18 <ncopa> so its not a drop-in replacent that is ready
2020-08-14 08:46:25 <ncopa> it asks me for password
2020-08-14 08:46:43 <ncopa> i think i'll have to move it to other place for now. im a bit short of time
2020-08-14 08:47:03 <clandmeter> it was an idea to move disfiles to a centralized server
2020-08-14 08:47:12 <clandmeter> do periodic syncs and cleanup builders
2020-08-14 08:47:19 <ncopa> i think thats a good diea
2020-08-14 08:47:32 <clandmeter> but its not dropin as you would like :)
2020-08-14 08:47:41 <clandmeter> else it would have been active ;-)
2020-08-14 08:48:01 <ncopa> i promised to poweroff the current machine this week so...
2020-08-14 08:48:51 <clandmeter> in current setup a remote distfiles is not really optimal?
2020-08-14 08:49:06 <clandmeter> i think current one was mounted via nfs iirc
2020-08-14 08:49:12 <ncopa> its not optimal
2020-08-14 08:49:23 <ncopa> it was only shared between x86 and x86_64 builders
2020-08-14 08:49:27 <ncopa> the two first ones
2020-08-14 08:49:41 <clandmeter> yes, thast what you want to replicate now right?
2020-08-14 08:49:52 <ncopa> in the new setup (that needs to be finished today) it will no longer be any nfs
2020-08-14 08:49:54 <ikke> more like replace I guess?
2020-08-14 08:50:01 <ncopa> so x86 will be like the other arches
2020-08-14 08:50:12 <clandmeter> aha
2020-08-14 08:50:20 <ncopa> only x86_64 will be accessing it directly
2020-08-14 08:50:40 <clandmeter> its a time limiting feature? or just change of mind?
2020-08-14 08:50:57 <ncopa> i think we will drop nfs
2020-08-14 08:51:00 <ncopa> forever
2020-08-14 08:51:13 <ncopa> i mean
2020-08-14 08:51:33 <ncopa> i think what i really want is a dedicated distfiles server that is not tied to any arch
2020-08-14 08:51:50 <clandmeter> ncopa: your keys are on the box now.
2020-08-14 08:51:52 <maldridge> we do that for void
2020-08-14 08:51:54 <ncopa> it may be shared by the different builders with nfs or not
2020-08-14 08:51:58 <maldridge> its not as useful as you might think
2020-08-14 08:52:14 <ncopa> problem is that we need to save the sources for our releases
2020-08-14 08:52:21 <ncopa> in case sources disapear from upstram
2020-08-14 08:52:36 <clandmeter> nod
2020-08-14 08:53:02 <clandmeter> i guess for stable branches we want to keep it. and have some special logic for edge
2020-08-14 08:53:17 <ncopa> exactly
2020-08-14 08:53:24 <ncopa> for edge we want delete relatively soon
2020-08-14 08:53:51 <ncopa> i was thining: keep all sources that are in current git and delete everythign else that is more than N days old
2020-08-14 08:55:16 <clandmeter> i guess for stable we could just keep it? i dont think it will grow that much?
2020-08-14 08:55:34 <clandmeter> i have a feeling we had this conversation before :)
2020-08-14 09:03:02 <mps> clandmeter: yes
2020-08-14 09:06:12 <ncopa> im super afraid of making a mistake. im currently uncomfortable stressed...
2020-08-14 09:07:38 <ncopa> can anyone help me to add a CNAME buildlogs.alpin.pw -> distfiles.nld9.alpin.pw?
2020-08-14 09:08:16 <ncopa> i think we should separate out the buildlogs in the future from the distfiles
2020-08-14 09:09:28 <ncopa> and a CNAME distfiles.alpin.pw -> distfiles.nld9.alpin.pw
2020-08-14 09:12:10 <ncopa> hum.
2020-08-14 09:12:36 <ncopa> dns is broken on dev.a.o. it does not resolve distfiles.nld9.alpin.pw
2020-08-14 09:12:48 <ncopa> same name resolves on my desktop
2020-08-14 09:25:11 <clandmeter> ncopa: im adding cname
2020-08-14 09:25:18 <ncopa> thanks
2020-08-14 09:25:34 <clandmeter> i need to run to fysio in a few min
2020-08-14 09:28:02 <ncopa> ikke: im having issues with git (git-old aka dev.a.o)
2020-08-14 09:28:13 <ncopa> same as before, it does not ping the default gw
2020-08-14 09:28:55 <ikke> hmm
2020-08-14 09:29:43 <ncopa> not sure if its same issue
2020-08-14 09:30:01 <ikke> so the host has no issues, just the containers?
2020-08-14 09:30:10 <ikke> or one of them
2020-08-14 09:30:17 <ncopa> correct. only one of them
2020-08-14 09:30:29 <ncopa> the one with a bridged public ip
2020-08-14 09:30:38 <ncopa> last time a reboot solved it
2020-08-14 09:45:08 <ncopa> ikke: problem looks very similar to the same issue last time
2020-08-14 09:45:21 <ikke> aha ok
2020-08-14 09:45:32 <ikke> Very strange
2020-08-14 09:49:50 <ncopa> here is what i observe from the host: guest does arp lookup of the default gw. the host sees the arp response. the guest never sees the arp response
2020-08-14 09:50:32 <ikke> hmm, so something is not forwarding it properly
2020-08-14 10:08:08 <ncopa> i discovered one thing
2020-08-14 10:08:25 <ncopa> the mac address was not configured in the lxc config
2020-08-14 10:08:52 <ncopa> which means it gets random (and changes) each time the container restarts
2020-08-14 10:09:24 <ikke> But that doesn't happen so often
2020-08-14 10:09:34 <ncopa> i restarted it today
2020-08-14 10:09:54 <ncopa> i have configured a mac address for it now
2020-08-14 10:10:00 <ncopa> so from now on it should be the same
2020-08-14 10:10:08 <ikke> ok
2020-08-14 10:10:31 <ikke> should git-old be reachable?
2020-08-14 10:10:40 <ncopa> the good news is that dmvpn is working
2020-08-14 10:10:50 <ncopa> git-old is the machine with this arp problem
2020-08-14 10:10:55 <ikke> yeah
2020-08-14 10:11:02 <ncopa> which is why its not reachable at the moment
2020-08-14 10:11:08 <ikke> ok
2020-08-14 11:12:23 <ncopa> ok. this is super weird. i have a bridge interface to a lxc container with public ip
2020-08-14 11:12:44 <ncopa> the container does not get the arp response for the default gw over the bridge
2020-08-14 11:13:03 <ncopa> i can see the arp response with tcpdump on the host
2020-08-14 11:13:14 <ncopa> but i cannot see the arp response in the container
2020-08-14 11:13:23 <ncopa> any idea whats going wrong?
2020-08-14 11:13:50 <ikke> The setup is the same like we had, right?
2020-08-14 11:14:02 <ikke> Well, perhaps not exactly the same
2020-08-14 11:14:29 <ncopa> not sure what you mean with "we had"
2020-08-14 11:15:28 <ikke> bld1/2
2020-08-14 11:15:33 <ncopa> yes
2020-08-14 11:20:06 <ncopa> if i manually set the arp table with -s
2020-08-14 11:20:09 <ncopa> the it works
2020-08-14 11:20:16 <ncopa> but who knows when it will break again
2020-08-14 11:25:29 <ikke> There should be some kind of way to debug this
2020-08-14 11:28:56 <ncopa> i cannot ping nld9-dev1 from the same container
2020-08-14 11:28:58 <ncopa> for same reasons
2020-08-14 11:34:22 <ikke> So seems like a general arp issue
2020-08-14 11:41:44 <ncopa> i foudn this: https://bridge.linux-foundation.narkive.com/5pSQfHW3/linux-bridge-does-not-forward-arp-reply-back-packets-in-a-vmware-vm
2020-08-14 11:43:28 <ncopa> "everything starts working only when I set ageing time to 0, because in that case all packets are flooded on all ports and the bridge behaves like a hub."
2020-08-14 11:43:39 <ncopa> i tried to set ageing time to 0
2020-08-14 11:43:45 <ncopa> turning it to a hub
2020-08-14 11:43:49 <ncopa> and it works
2020-08-14 11:57:01 <ikke> so some issue with the bridging code / filters
2020-08-14 12:05:55 <ncopa> maybe its trelated that lxcbr0 has same mac addr on both nld8 and nl9?
2020-08-14 12:06:02 <ncopa> shouldnt matter
2020-08-14 12:06:06 <ncopa> but who knows
2020-08-14 12:06:18 <ncopa> ok my time is up
2020-08-14 12:12:13 <clandmeter> i hope for just this project :)
2020-08-14 12:12:39 <ikke> lol
2020-08-14 13:13:14 <LBlaboon> clandmeter: which user account were you logging in as?
2020-08-14 13:13:35 <clandmeter> LBlaboon: my own?
2020-08-14 13:14:10 <clandmeter> username is clandmeter
2020-08-14 13:14:33 <LBlaboon> ok, just making sure since there are several logins under your account (but i guess i should have figured :D)
2020-08-14 13:15:54 <LBlaboon> 2FA is enabled for that account. did you perhaps have your device set as a "trusted device"?
2020-08-14 13:18:20 <LBlaboon> i do see one active trusted device for your user, so if you were logging in from the trusted device then you will skip 2FA
2020-08-14 13:26:34 <clandmeter> ah ok
2020-08-14 13:26:47 <clandmeter> maybe i dont understand it
2020-08-14 13:27:29 <LBlaboon> when logging in, there is a checkbox under the login form to remember your device for 30 days. that makes your device a "trusted device"
2020-08-14 13:28:06 <clandmeter> ok but i still need to login?
2020-08-14 13:28:20 <LBlaboon> yea, if you log out then you still need to log back in again, but 2FA is skipped
2020-08-14 13:28:22 <clandmeter> trusted device disables 2FA only?
2020-08-14 13:28:25 <LBlaboon> yep
2020-08-14 13:28:30 <clandmeter> ok
2020-08-14 13:28:33 <clandmeter> thx
2020-08-14 13:29:16 <clandmeter> LBlaboon: i just tried it
2020-08-14 13:29:18 <clandmeter> if i logout
2020-08-14 13:29:28 <clandmeter> i still need to provide 2FA
2020-08-14 13:32:55 <LBlaboon> hmm....ahh i think manually logging out automatically expires the trusted device (ahead of when it would normally expire). so probably what happened in your case is you didn't manually log out, but your session expired and you had a re-login, but 2FA was skipped because your device was trusted
2020-08-14 13:33:32 <clandmeter> if you think its ok, lets leave it at taht :)
2020-08-14 13:34:04 <LBlaboon> i can see that the previously trusted device for your user just expired when you wrote me that message (presumably, when you hit the "logout" button)
2020-08-14 13:37:23 <LBlaboon> yea, i think it's working fine (although our implementation is perhaps a bit strange). i do think it's weird that you got prompted for username/password instead of being automatically logged in on your trusted device, but i'm not an expert on the details of our login system
2020-08-14 15:41:20 <ncopa> ikke: this makes it work: nld8-dev1 [~]# echo 0 > /sys/class/net/br0/bridge/ageing_time 
2020-08-14 15:41:40 <ncopa> resetting it to 29999 and flushing the arp cache in the container makes the problem come back
2020-08-14 15:41:46 <ikke> That's what was suggested in that post, right? 
2020-08-14 15:42:12 <ncopa> i didnt have time to read the entire post, i just saw that setting aging time to zero make things work
2020-08-14 15:42:31 <ikke> I think that forces it to hub mode, right? 
2020-08-14 15:42:33 <ncopa> i need to make the upload of build logs work. i think they are uploaded to old host currently
2020-08-14 15:42:35 <ncopa> yes
2020-08-14 15:43:00 <ncopa> so problem is that the bridge does not learn some hw address for some reason
2020-08-14 15:44:20 <ncopa> ugh... seems like i need to step out again
2020-08-14 15:54:44 <ikke> Good weekend
2020-08-14 17:28:17 <clandmeter> https://www.theregister.com/2020/08/14/docker_container_retention_policy/
2020-08-14 19:56:28 <ncopa> ikke: i filed an issue for the arp/bridge problem: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10701
2020-08-14 19:56:51 <ikke> ok
2020-08-14 19:56:56 <ikke> thanks
2020-08-15 09:11:24 <ikke> Looks like the latency to our s390x-ci host has increased from ~80ms to ~300ms
2020-08-16 13:54:30 <ikke> clandmeter: ping
2020-08-16 16:21:18 <clandmeter> Pong
2020-08-16 16:21:28 <ikke> Working on the gitlab 13.1 image
2020-08-16 16:22:52 <ikke> I just switched it to ruby:2.6-alpine3.12, because it complained about needing go 1.13
2020-08-16 16:22:58 <ikke> It was alpine 3.10
2020-08-16 16:23:17 <ikke> I needed to fix one patch because the context changed
2020-08-16 16:23:26 <ikke> but it's taking a long time to build each time
2020-08-16 16:25:11 <clandmeter> Long as in?
2020-08-16 16:27:37 <ikke> Well, 10-20 minutes I guess each time
2020-08-16 16:30:31 <clandmeter> That's normal
2020-08-16 16:30:49 <clandmeter> Around 16 min
2020-08-16 16:30:57 <ikke> yeah
2020-08-16 16:31:07 <ikke> but when you're trying to fix issues, that's a long time :P
2020-08-16 16:37:06 <ikke> clandmeter: Any way I can see what protobuf version is required?
2020-08-16 16:38:22 <clandmeter> Yes in some file in the root
2020-08-16 16:39:11 <clandmeter> Gemfile.lock
2020-08-16 16:44:02 <ikke> google-protobuf (3.8.0)
2020-08-16 16:44:04 <ikke> ?
2020-08-16 16:44:42 <ikke> That's older then we have now (3.11.4)
2020-08-16 16:45:47 <ikke> Should I check it for gitlab-floss?
2020-08-16 17:37:05 <clandmeter> Does it fail to build?
2020-08-16 18:17:52 <ikke> no
2020-08-17 09:14:44 <ikke> clandmeter: I pushed a 13.1-stable branch for alpine-docker-gitlab
2020-08-17 10:18:58 <clandmeter> ikke: nice
2020-08-17 10:19:06 <clandmeter> did you had to fix a lot of things?
2020-08-17 10:19:17 <ikke> No, just a couple
2020-08-17 10:19:23 <ikke> testing it now on gitlab-test
2020-08-17 10:19:53 <clandmeter> it seems they switch protobuf from time to time.
2020-08-17 10:20:06 <clandmeter> newer version can be build without patching i think
2020-08-17 10:26:35 <ikke> clandmeter: https://gitlab-test.alpinelinux.org
2020-08-17 10:26:40 <ikke> seems to be running
2020-08-17 10:27:49 <clandmeter> is that hostname public?
2020-08-17 10:30:52 <ikke> yes
2020-08-17 11:57:59 <clandmeter> hostfile :)
2020-08-17 11:58:09 <ikke> haha :P
2020-08-17 12:00:50 <clandmeter> you had to explicitly set it ti 3.12?
2020-08-17 12:01:43 <ikke> yes
2020-08-17 12:02:06 <ikke> ruby:2.6 is alpine 3.100
2020-08-17 12:02:14 <ikke> s/100/10
2020-08-17 12:02:14 <alpine-meetbot> ikke meant to say: ruby:2.6 is alpine 3.10
2020-08-17 12:03:28 <clandmeter> ah ok
2020-08-17 12:03:35 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/commit/4438a1330a85d94dac321bf8f0acc6829589b71a
2020-08-17 12:04:15 <ikke> ruby:2.6-alpine I must say
2020-08-17 12:04:40 <ikke> Not sure if gitlab is compatibly with ruby 2.7 yet
2020-08-17 12:05:15 <ikke> ncopa: It seems we are still getting corrupt tarbals on distfiles
2020-08-17 12:06:44 <ncopa> new tarballs?
2020-08-17 12:06:52 <ncopa> or old ones
2020-08-17 12:06:53 <ikke> yes, it seems so
2020-08-17 12:06:57 <ikke> hiredis was just pushed
2020-08-17 12:07:20 <ikke> http://dup.pw/alpine/aports/137de502
2020-08-17 12:10:52 <clandmeter> ikke: no i dont think so, not last time i checked.
2020-08-17 12:11:19 <ikke> clandmeter: right, so better to stay at 2.6
2020-08-17 12:12:09 <clandmeter> https://gitlab.com/groups/gitlab-org/-/epics/2380
2020-08-17 12:12:17 <clandmeter> fun with protobuf it seems
2020-08-17 12:13:14 <clandmeter> or ruby-grpc
2020-08-17 12:31:15 <ikke> clandmeter: if you agree, I will upgrade gitlab tonight
2020-08-17 12:34:22 <clandmeter> I sure do, thanks for following this up.
2020-08-17 12:34:51 <ikke> Cogitri: fyi, this also means that gitlab-test will disappear soon after
2020-08-17 12:37:16 <ikke> clandmeter: any idea how we can completely disabled smtp? I've tried to rename smtp_settings.rb in the initializers folder and restart, but it still sent e-mails
2020-08-17 12:40:59 <ikke> clandmeter: oh, API gives 500 internal server error :(
2020-08-17 12:42:16 <Cogitri> ikke: ah, I'd still like to test with email disabled
2020-08-17 12:42:35 <ikke> I think we can leave it for one or two days
2020-08-17 12:43:20 <ikke> Cogitri: fyi, this instance is now upgraded to 13.1, and apparently the API is returning 500, I'm investigating
2020-08-17 12:44:53 <ikke> clandmeter: "exception.class": "GRPC::Unavailable",
2020-08-17 12:45:04 <ikke> so grpc is giving issues
2020-08-17 12:45:16 <clandmeter> sounds like fun
2020-08-17 12:45:28 <clandmeter> i think i had that before
2020-08-17 12:46:11 <Cogitri> Ah okie, I can wait until it's resolved, no problem. Thanks for the ping :)
2020-08-17 12:46:29 <Cogitri> Thought you meant that the main instance is being upgraded and gitlab-test will be gone for good
2020-08-17 12:47:06 <ikke> Yes, that will happen
2020-08-17 12:47:27 <ikke> but we can decide when we delete the test instance
2020-08-17 12:49:30 <ikke> clandmeter: any hints on how to debug this?
2020-08-17 12:56:32 <clandmeter> ikke: how can i ssh to the instance?
2020-08-17 12:57:07 <clandmeter> got it
2020-08-17 13:31:03 <clandmeter> ikke: build log is on drone?
2020-08-17 13:45:50 <ikke> clandmeter: yes
2020-08-17 13:46:19 <ikke> https://cloud.drone.io/alpinelinux/alpine-docker-gitlab/135/1/2
2020-08-17 13:50:36 <clandmeter> gitaly does not want to start
2020-08-17 13:50:44 <clandmeter> time="2020-08-17T13:49:55Z" level=fatal msg="could not create GitLab API client:  is not a valid url"
2020-08-17 13:51:15 <clandmeter> ikke: did you check the installation steps?
2020-08-17 13:54:26 <ikke> I fllowed it, but did not notice that
2020-08-17 13:55:11 <clandmeter> i looks like some url is not set
2020-08-17 13:57:05 <ikke> Where did you see that line?
2020-08-17 13:57:40 <ikke> clandmeter: https://gitlab.com/gitlab-org/gitaly/-/issues/2981
2020-08-17 13:58:00 <ikke> https://gitlab.com/gitlab-org/gitaly/blob/4290807efcf2de64d5c1e8abce15399361da42f9/config.toml.example#L95
2020-08-17 13:59:11 <clandmeter> yup
2020-08-17 13:59:20 <clandmeter> looks like a few new config options
2020-08-17 13:59:59 <clandmeter> i think i added a few infos about which parts need to check when upgrading.
2020-08-17 14:00:06 <clandmeter> like check a diff on install from source part
2020-08-17 14:02:14 <ikke> It should be the same socket as for gitlab-shell I guess?
2020-08-17 14:02:32 <ikke> http+unix://%2Fhome%2Fgit%2Fgitlab%2Ftmp%2Fsockets%2Fgitlab.socket
2020-08-17 14:03:15 <ikke> Do I manually need to add it to /src/docker/gitlab/config/gitaly?
2020-08-17 14:09:19 <ikke> Ok, that did the trick
2020-08-17 14:09:54 <clandmeter> ok nice
2020-08-17 14:10:23 <clandmeter> we should add a url to the gitlog of those config files
2020-08-17 14:11:17 <ikke> https://gitlab.com/gitlab-org/gitaly/-/commits/master/config.toml.example ?
2020-08-17 14:11:44 <ikke> oh, hmm: https://gitlab.com/gitlab-org/gitaly/-/commit/a5584c2dac3a4863c4c7701c81ac77e97bbe5599
2020-08-17 14:12:01 <clandmeter> yup
2020-08-17 14:12:15 <ikke> They explicitly point it to the workhorse socket?
2020-08-17 14:17:25 <clandmeter> yeah
2020-08-17 14:17:39 <clandmeter> its a kind of a proxy
2020-08-17 14:20:34 <ikke> and https://gitlab.com/gitlab-org/gitaly/-/commit/c531efb0de798a8c83ac135be780524161184034
2020-08-17 14:20:52 <ikke> So apparently gitlab-shell config is no longer used?
2020-08-17 14:54:31 <ikke> Cogitri: ping
2020-08-17 14:58:25 <Cogitri> Pong 
2020-08-17 15:00:06 <ikke> Cogitri: I tried another thing to disable e-mails, can we test it?
2020-08-17 15:00:25 <ikke> Maybe you can create an issue and I will respond to it
2020-08-17 15:02:13 <Cogitri> Sure
2020-08-17 15:04:27 <Cogitri> ikke: https://gitlab-test.alpinelinux.org/alpine/aports/-/issues/11844
2020-08-17 15:05:35 <ikke> replied
2020-08-17 15:06:22 <ikke> Haven't received something yet
2020-08-17 15:11:36 <ikke> So I guess it's working?
2020-08-17 15:13:30 <Cogitri> Didn't get anything yet either, so seems to work :)
2020-08-17 15:16:57 <Cogitri> Time to test the gitlab bit then, thanks for looking into it
2020-08-17 15:18:05 <ikke> NP 
2020-08-17 16:14:20 <Cogitri> ikke: Would it be possible to restore gitlab-test to the state it was  after initially restoring the backup?
2020-08-17 16:24:10 <clandmeter> Restore again?
2020-08-17 16:25:14 <clandmeter> Maybe I don't follow :)
2020-08-17 16:27:17 <ncopa> im gonna do the interview tomorrow for linode, and I was asked to fill in a for that has: "focus on the interview" and "seed questions (up to 5)". I have no clue!
2020-08-17 16:28:18 <clandmeter> I don't even know what that means
2020-08-17 16:28:32 <clandmeter> Seed questions?
2020-08-17 16:28:51 <ncopa> i think what they want me to them to ask
2020-08-17 16:29:19 <ncopa> i think the idea is that they want me to talk about something, not just respond to yes/no questions
2020-08-17 16:30:33 <ncopa> instead of asking: "do you like linux?" they ask something that makes me talk, like "what is it with linux that makes you spend so much time on it?"
2020-08-17 16:30:48 <ncopa> thats how interviews are normally done (i think)
2020-08-17 16:33:16 <ncopa> they have seed questions that make me talk about something, and they can add follow up questions to that
2020-08-17 16:33:24 <ncopa> my problem is i dont know what to talk about
2020-08-17 16:33:32 <Cogitri> clandmeter: gitlab-test was initially restored from a backup of the actual gitlab instance, it'd be nice if that could be done again
2020-08-17 16:34:01 <Cogitri> So I can do another test of the stale mechanism because right now it won't mark anything as stale anymore because everything that is stale has been marked already :)
2020-08-17 17:11:44 <ikke> Cogitri: I don't think we still have a copy of the DB at point of restore
2020-08-17 17:12:39 <Cogitri> Hm, okay
2020-08-17 17:13:16 <Cogitri> Well, not the worst thing ever, the marking itself seems to work, so I guess the changes I did today should be OK 
2020-08-17 17:13:23 <clandmeter> Linode has some dated backups
2020-08-17 17:14:15 <ikke> I can do an export of the production database
2020-08-17 17:14:21 <ikke> and import that to the test instance
2020-08-17 17:16:12 <ncopa> hum... 3.12 builders does not upload buildlogs
2020-08-17 17:22:52 <Cogitri> ikke: Oh, that'd work too :) 
2020-08-17 18:05:51 <ikke> clandmeter: upgrade to 13.1 succeeded
2020-08-18 07:04:24 <clandmeter> ikke: 👊
2020-08-18 07:10:02 <ikke> Nice, you can now subscribe to new releases in gitlab (under custom notification settings)
2020-08-18 07:10:05 <ikke> (to just them)
2020-08-18 07:28:58 <ncopa> awesome! thank you ikke
2020-08-18 07:48:35 <ikke> hmmm, cgit is giving bad object ids back for recently pushed commits, but it seems the repo sync is still working
2020-08-18 07:49:32 <ikke> ncopa: I think we need to increase the commit length
2020-08-18 07:49:48 <ikke> https://git.alpinelinux.org/aports/commit/?id=f10a27ab doesno't work
2020-08-18 07:49:50 <ikke> https://git.alpinelinux.org/aports/commit/?id=f10a27abc does
2020-08-18 08:05:44 <ncopa> hum. ok
2020-08-18 08:12:58 <ncopa> should be done now
2020-08-18 08:21:51 <ikke> Thanks 
2020-08-18 13:03:46 <Cogitri> ikke: Can you make https://gitlab.alpinelinux.org/Cogitri/apk-polkit-rs public?
2020-08-18 16:33:17 <ikke> Cogitri: done
2020-08-18 16:34:50 <Cogitri> Thanks 👍
2020-08-21 06:14:04 <Ariadne> can someone make https://gitlab.alpinelinux.org/kaniini/alpine-gcc-patches public, and/or set it up so i don't have to bother people
2020-08-21 06:14:28 <Ariadne> i'm working on gcc 10.2 and i think we're going to just go git-first for gcc patchset tracking like we did with uclibc
2020-08-21 10:07:13 <Cogitri> appstream.a.o gives 404s for everything :/
2020-08-21 10:07:25 <ikke> that's not good :)
2020-08-21 10:08:10 <ikke> nginx container crashed
2020-08-21 10:08:19 <ikke> no, exit 0
2020-08-21 10:08:39 <ikke> It's missing the restart setting
2020-08-21 10:08:43 <ikke> so it was never restarted again
2020-08-21 10:08:46 <ikke> at boot
2020-08-21 10:12:08 <Cogitri> Oh
2020-08-21 10:15:22 <ikke> Do we need to clean up older reports at some point?
2020-08-21 10:32:21 <Cogitri> I guess we can do that depending on how much space they take up
2020-08-23 08:38:37 <clandmeter> ikke: few interesting new features in gitlab
2020-08-23 08:38:52 <ikke> Any in particular? 
2020-08-23 08:39:18 <clandmeter> Matrix
2020-08-23 08:39:30 <clandmeter> Not sure how it could be useful
2020-08-23 08:39:48 <ikke> Ugh, whats up with these mirrors
2020-08-23 08:40:27 <clandmeter> Project access tokens
2020-08-23 08:40:41 <clandmeter> No longer based on user
2020-08-23 08:40:52 <ikke> Yeah that's nice
2020-08-23 08:41:28 <clandmeter> Ci matrix sound interesting
2020-08-23 08:46:35 <ikke> Yes, indeed, it does 
2020-08-23 08:47:55 <maldridge> clandmeter: yeah, the project access tokens is amazing, it solves a lot of wonky access control problems
2020-08-23 08:48:04 <maldridge> and should help with not needing random root level bot users
2020-08-23 12:08:33 <ikke> clandmeter: Seems like gitaly has issues or at least some jobs are taking a long time
2020-08-23 12:20:23 <ikke> https://tpaste.us/zap1 
2020-08-23 12:20:31 <ikke> These seem to be stuck
2020-08-23 12:36:52 <ikke> apparently cloudflare is providing an alpinelinux mirror as well: https://cloudflaremirrors.com/alpine/
2020-08-23 13:21:43 <ikke> clandmeter: seems like they're deadlocked
2020-08-24 04:47:29 <ikke> hmpfs
2020-08-24 04:57:28 <ikke> strange, I can reach dl-2, but not from zabbix
2020-08-24 04:57:35 <ikke> I can ping it, but no http
2020-08-24 07:06:57 <clandmeter> morning
2020-08-24 07:07:27 <clandmeter> ikke: reg deadlocks, how did you notice this?
2020-08-24 07:10:10 <ikke> Rebasing in the web interface not completing 
2020-08-24 07:10:30 <clandmeter> so its a new version issue?
2020-08-24 07:10:54 <ikke> I guess so? It's not happening always 
2020-08-24 07:11:09 <clandmeter> did it happen before?
2020-08-24 07:11:39 <ikke> Not that I'm aware of 
2020-08-24 07:11:56 <clandmeter> did you check upstream?
2020-08-24 07:12:20 <clandmeter> from what i could tell, there were a lot of changes in gitalhy
2020-08-24 07:12:59 <clandmeter> and i know gitaly needs a specific git version, pretty recent.
2020-08-24 07:13:13 <clandmeter> err not specific but pretty recent..
2020-08-24 07:17:24 <clandmeter> ikke: from which version did we upgrade?
2020-08-24 07:17:30 <clandmeter> 3.0.x?
2020-08-24 07:17:59 <ikke> I don't recall exactly 
2020-08-24 07:18:42 <clandmeter> but it was 3.0 right?
2020-08-24 07:18:52 <clandmeter> 13.0
2020-08-24 07:18:53 <clandmeter> sorry
2020-08-24 07:19:47 <ikke> Yes
2020-08-24 07:19:57 <clandmeter> ah you also upgraded alpine
2020-08-24 07:20:03 <ikke> Indeed
2020-08-24 07:20:13 <clandmeter> from 11 to 12 right?
2020-08-24 07:24:13 <clandmeter> # Make sure Git is version 2.24.0 or higher (recommended version is 2.28.0)
2020-08-24 07:53:58 <ikke> Alpine 3.12 has 2.26
2020-08-24 09:13:11 <ikke> sigh
2020-08-24 10:42:04 <clandmeter> danieli: there is a new lounge version out
2020-08-24 10:42:51 <ikke> clandmeter: seems there are a number of stuck processes again
2020-08-24 10:44:37 <clandmeter> im checking
2020-08-24 10:44:48 <ikke> 24770
2020-08-24 10:44:50 <clandmeter> the cat files are from tree browsing
2020-08-24 10:45:00 <ikke> Yes, those are not stuck
2020-08-24 10:45:07 <clandmeter> ikke: please use pids from container :)
2020-08-24 10:45:17 <ikke> git pull / git fetch / gitaly-ssh
2020-08-24 10:45:21 <clandmeter> its confuing to do it from host
2020-08-24 10:45:53 <clandmeter> 3270 git       0:00 /usr/bin/git fetch --no-tags ssh://gitaly/internal.git 0323b71cb35062acfa199af88c926835c351142a
2020-08-24 10:48:35 <ikke> I count 7 processes
2020-08-24 10:50:51 <clandmeter> right
2020-08-24 10:51:00 <clandmeter> and we cant strace them due to perms
2020-08-24 10:51:05 <ikke> You can from the host :P
2020-08-24 10:51:11 <ikke> which I did previously
2020-08-24 10:51:32 <ikke> One process was hanging on wait4
2020-08-24 10:51:37 <ikke> the other on futexes
2020-08-24 10:52:06 <clandmeter> ah ok
2020-08-24 10:53:02 <clandmeter> so this is related to git itself?
2020-08-24 10:53:58 <ikke> gitaly-ssh is involved as well
2020-08-24 10:54:38 <ikke> gitaly-ssh is waiting on a futex
2020-08-24 10:55:05 <ikke> another one on epol_wait4
2020-08-24 10:55:10 <ikke> sorry, epol_pwait
2020-08-24 10:55:26 <ikke> all git processes are hanging on wait4
2020-08-24 10:55:38 <ikke> so I guess waiting for child processes to finish?
2020-08-24 10:55:55 <ikke> which is gitaly-ssh
2020-08-24 10:56:01 <clandmeter> right
2020-08-24 10:57:24 <ikke> this is an older issue: https://gitlab.com/gitlab-org/gitaly/-/issues/1248
2020-08-24 11:05:59 <clandmeter> https://docs.gitlab.com/ee/administration/gitaly/
2020-08-24 11:10:09 <clandmeter> ikke: i think we need to disable the askimet integration
2020-08-24 11:10:14 <clandmeter> it looks totally useless
2020-08-24 11:14:04 <clandmeter> hmm, the logs section has been removed from webui?
2020-08-24 11:17:13 <ikke> clandmeter: https://gitlab.alpinelinux.org/admin/spam_logs
2020-08-24 11:17:17 <ikke> this>
2020-08-24 11:17:37 <clandmeter> For spam yes
2020-08-24 11:17:42 <clandmeter> That's useless
2020-08-24 11:18:08 <clandmeter> It only blocks good things
2020-08-24 11:18:24 <clandmeter> For server logs the option is gone
2020-08-24 11:18:36 <ikke> aha, ok
2020-08-24 11:18:59 <clandmeter> Maybe it turned into a config option
2020-08-24 11:19:06 <ikke> https://gitlab.alpinelinux.org/admin/application_settings/reporting here we can disable akismet
2020-08-24 11:22:51 <clandmeter> done
2020-08-24 11:23:54 <clandmeter> ikke: can you replicate the issue?
2020-08-24 11:24:13 <ikke> I think it
2020-08-24 11:24:16 <ikke> it's intermittent
2020-08-24 11:25:07 <clandmeter> lets first upgrade it
2020-08-24 11:25:09 <clandmeter> we are behind
2020-08-24 11:25:24 <ikke> ok
2020-08-24 11:25:32 <ikke> Can you upgrade the image?
2020-08-24 11:27:27 <clandmeter> im going to build it locally
2020-08-24 11:27:49 <ikke> k
2020-08-24 11:30:59 <clandmeter> i think the local protobuf we build is not used
2020-08-24 11:31:09 <clandmeter> not anymore
2020-08-24 11:33:21 <clandmeter> tmhoang: you are also using lounge right?
2020-08-24 11:33:35 <tmhoang> clandmeter: yes, thanks to you 
2020-08-24 11:33:44 <clandmeter> there is a new version
2020-08-24 11:33:48 <clandmeter> so i plan to upgrade
2020-08-24 11:34:00 <tmhoang> yes please
2020-08-24 11:34:12 <clandmeter> danieli prefers a heads up
2020-08-24 11:34:31 <clandmeter> not sure when it will happen, but should be asap
2020-08-24 11:37:29 <tmhoang> :3
2020-08-24 11:38:01 <clandmeter> there is a delay in typing when using chrome
2020-08-24 11:38:05 <clandmeter> i like to kill that issue
2020-08-24 11:38:23 <tmhoang> I was wondering how did you get a .us domain like  tpaste.us given you are NL  citizen ? 
2020-08-24 11:38:31 <tmhoang> I have no issue with firefox on VPN 
2020-08-24 11:44:29 <clandmeter> tmhoang: i can j ust register it with my provider
2020-08-24 11:44:57 <tmhoang> interesting
2020-08-24 11:46:24 <clandmeter> and still can
2020-08-24 11:46:30 <clandmeter> around 12 euro p/y
2020-08-24 12:04:41 <mps> hmm, to try to fix dovecot on 3.12 I have to create 3.12-stable armv7 (or armhf/x86) lxc
2020-08-24 12:05:15 <mps> I think it doesn't make sense to test builds on edge
2020-08-24 12:12:02 <mps> I can't run lxc under lxc, I think?
2020-08-24 12:14:53 <clandmeter> i dont think so
2020-08-24 12:17:20 <mps> if I interpret correctly this !tautology I can run lxc under lxc
2020-08-24 12:19:21 <Ariadne> afaik you can nest lxc under privileged lxc container
2020-08-24 12:26:34 <mps> yes
2020-08-24 12:26:53 <mps> I forgot to mention that I have unprivileged ones
2020-08-24 12:29:22 <clandmeter> everything is possible, but your current container wont allow it.
2020-08-24 12:31:20 <mps> np, I will crate local one for 3.12-stable
2020-08-24 12:36:52 <clandmeter> mps: you ever tried rootbld?
2020-08-24 12:37:10 <ikke> That doesn't work in lxc either :P
2020-08-24 12:37:23 <clandmeter> it does on mine :p
2020-08-24 12:37:31 <ikke> cheater :P
2020-08-24 12:38:16 <clandmeter> its not that intrusive iirc
2020-08-24 12:38:44 <clandmeter> or less difficult to setup on both sides
2020-08-24 12:39:43 <clandmeter> ikke: https://cloud.drone.io/alpinelinux/alpine-docker-gitlab/136
2020-08-24 12:40:16 <clandmeter> i hope disabling local protobuf building does not break gitlab.
2020-08-24 12:40:19 <ikke> did you fix the protobuf issue (if it even was an issue)?
2020-08-24 12:40:23 <ikke> ok
2020-08-24 12:40:33 <clandmeter> i think it now depends on a more recent version
2020-08-24 12:40:42 <ikke> Cogitri: reminds me, do you still need the test server?
2020-08-24 12:40:43 <clandmeter> which does not have build issues
2020-08-24 12:41:35 <ikke> I'd like to take it down soon
2020-08-24 12:42:42 <Cogitri> It'd be nice if you could import a backup of the current gitlab so I can do another test run
2020-08-24 12:42:51 <ikke> ah, right
2020-08-24 12:43:03 <Cogitri> After that it can be taken down and the bot can be deployed I think
2020-08-24 12:48:11 <mps> clandmeter: no, for debugging it is easier to use VM or native machine, at least for me
2020-08-24 12:48:45 <mps> than rootbld and similar, I mean
2020-08-24 12:55:21 <tmhoang> personally I still prefer non-root (daemon) container platform like podman - been using it nearly a year now - smooth 
2020-08-24 12:55:33 <tmhoang> cant wait to get it to community 
2020-08-24 12:56:02 <tmhoang> currently testing extensively and would prob. pr to community before 3.13
2020-08-24 13:02:38 <clandmeter> tmhoang: the one we have in testing?
2020-08-24 13:02:53 <tmhoang> yes
2020-08-24 13:03:07 <clandmeter> i remember i tried it but it  was not in working shape
2020-08-24 13:04:21 <clandmeter> mps: i dont think you can use something more simple than rootbld.
2020-08-24 13:04:45 <clandmeter> if your container allows it ofc :)
2020-08-24 13:05:02 <ikke> You need CAP_ADMIN apparently :)
2020-08-24 13:05:11 <ikke> sys_admin
2020-08-24 13:05:33 <ikke> at least, that's how you solved it
2020-08-24 13:06:01 <tmhoang> clandmeter: how so ? 
2020-08-24 13:06:21 <clandmeter> tmhoang: its long time ago, so dont ask details :)
2020-08-24 13:06:43 <clandmeter> and reading your comment, it sounded you fixed something?
2020-08-24 13:07:14 <tmhoang> not fix, but more like adding missing steps in post-setup script
2020-08-24 13:07:36 <tmhoang> (I also havent used it recently on alpine though ...)
2020-08-24 13:19:40 <clandmeter> ikke: do we want to upgrade gitlab without testing it?
2020-08-24 13:21:08 <ikke> Do you consider it a 'risky' upgrade?
2020-08-24 13:22:16 <clandmeter> its a patch release, but it could be an issue with the local protobuf. i dont think it will but...
2020-08-24 13:22:25 <clandmeter> the risk is 20m downtime :)
2020-08-24 13:22:32 <ikke> heh
2020-08-24 13:22:57 <clandmeter> can you upgrade the test instance?
2020-08-24 13:23:06 <ikke> sure
2020-08-24 13:23:22 <clandmeter> if that runs and api works, it should be fine.
2020-08-24 13:31:01 <tmhoang> clandmeter: do you know what is the right approach to add per-user config to, for example, /etc/subuid and /etc/subgid something like : tmhoang:100000:65536. Since this is per-user, I'm not sure how to do it in post-script 
2020-08-24 13:31:29 <tmhoang> adding for root or specific user (i.e. qemu) is fine I guess 
2020-08-24 13:31:58 <tmhoang> wondering if there was any post script in some packages alrady done this 
2020-08-24 13:33:30 <tmhoang> maybe rely on /etc/shadow ? 
2020-08-24 14:03:02 <clandmeter> is that a task for a post install script?
2020-08-24 14:08:55 <ikke> clandmeter: gitlab-test is upgraded, and API is still working
2020-08-24 14:09:13 <clandmeter> sounds good
2020-08-24 14:09:17 <clandmeter> lets upgrade it
2020-08-24 14:09:32 <clandmeter> but i need to go now to other office
2020-08-24 14:09:48 <clandmeter> we can also do it later.
2020-08-24 14:10:04 <ikke> yeah, will do it later
2020-08-24 20:17:59 <ikke> clandmeter: ping
2020-08-24 20:18:06 <clandmeter> hola
2020-08-24 20:18:20 <ikke> some strange http(s) related issues lately
2020-08-24 20:18:49 <ikke> for instance, all those mirrors reporting as unreachable
2020-08-24 20:19:00 <clandmeter> its a host or zabbix issue?
2020-08-24 20:19:25 <ikke> well, the x86 ci host is timing out on gitlab as well from time to time
2020-08-24 20:19:44 <ikke> curl: (28) Failed to connect to gitlab.alpinelinux.org port 443: Operation timed out
2020-08-24 20:20:24 <ikke> I can reach mirrors.gigenet.com without issues
2020-08-24 20:20:40 <clandmeter> where does zabbix run?
2020-08-24 20:20:43 <ikke> nld3
2020-08-24 20:20:50 <ikke> packet
2020-08-24 20:20:50 <clandmeter> thats linode?
2020-08-24 20:20:53 <clandmeter> ah ok
2020-08-24 20:21:05 <ikke> there is a proxy on gbr2
2020-08-24 20:21:14 <ikke> but that's used to monitor services on nld3
2020-08-24 20:23:35 <ikke> zabbix reports 7 mirrors unreachable atm
2020-08-24 20:26:14 <clandmeter> could it be fw?
2020-08-24 20:26:29 <ikke> which fw?
2020-08-24 20:26:37 <clandmeter> zabbix host?
2020-08-24 20:26:58 <ikke> doesn't explain the issue from nld7 to deu2
2020-08-24 20:35:41 <clandmeter> so we have https network issues across our whole network?
2020-08-24 20:36:23 <ikke> I guess mostly on packet hosts
2020-08-24 20:36:47 <clandmeter> but its https only right?
2020-08-24 20:37:20 <ikke> http as well
2020-08-24 20:37:24 <ikke> not sure anything else
2020-08-24 20:37:41 <ikke> dl-2 is http only for example
2020-08-24 20:37:47 <clandmeter> its not network related?
2020-08-24 20:39:32 <clandmeter> maybe something ipv6 related
2020-08-24 20:39:50 <ikke> no, curl shows it's connecting to ipv4
2020-08-24 20:40:09 <ikke> *   Trying 172.105.69.85:443...
2020-08-24 20:40:25 <ikke> some requests return immediately, others time out
2020-08-24 20:48:24 <ikke> .isup dl-2.alpinelinux.org
2020-08-24 20:48:43 <ikke> .uptime
2020-08-24 20:48:43 <alpine-meetbot> I've been sitting here for 105 days, 12:32:52 and I keep going!
2020-08-24 20:48:50 <ikke> .isup http://dl-2.alpinelinux.org
2020-08-24 20:49:07 <ikke> 22:48:11 alpine-meetbot │ http://dl-2.alpinelinux.org looks fine to me.
2020-08-24 20:50:34 <alpine-meetbot> http://dl-2.alpinelinux.org looks down from here.
2020-08-24 20:50:59 <alpine-meetbot> http://dl-2.alpinelinux.org looks down from here.
2020-08-24 20:51:00 <ikke> lol
2020-08-25 11:53:09 <fabled> hmm.. i tagged new apk-tools, but the git hook that created the tarball from it, is apparently gone or creates the tarballs in new place?
2020-08-25 11:54:53 <ikke> fabled: does this work? https://gitlab.alpinelinux.org/alpine/apk-tools/-/archive/v2.12.0_rc1/apk-tools-v2.12.0_rc1.tar.gz
2020-08-25 11:56:30 <ikke> gitlab also has explicit releases, where you can attach your own tarbal if we need it
2020-08-25 11:57:00 <ikke> gitlab 13.3 introduces project level authentication tokens, which makes it easier to do these things in a CI/CD pipeline
2020-08-25 11:58:13 <fabled> ok, i kinda like it when it happens automatic either implicitly or explicitly via CI/CD pipeline
2020-08-25 11:58:38 <ikke> The url I linked to exists automatically for every tag
2020-08-25 11:58:45 <fabled> i'll use gitlab url. i think the only concern is that if the content changes it's invalidating the apkbuild checksum
2020-08-25 11:59:13 <ikke> except very rare ocassions, it should be stable
2020-08-25 12:03:01 <fabled> yes, that's my understanding as well. i remember once it broke due to gzip parameters changing somehow. but that should be good. thanks!
2020-08-25 12:03:44 <ikke> We're already using github (and less frequently gitlab) generated archives in aports already
2020-08-25 12:04:09 <ikke> for github I've seen confirmed that they're using git archive as a back-end
2020-08-25 12:04:15 <ikke> I assume gitlab uses the same
2020-08-26 08:42:26 <ikke> In addition to image retention, docker also added rate limits (100 image pull requests per 6 hours) 
2020-08-26 08:42:42 <ikke> Per ip if anonymous 
2020-08-26 09:01:18 <clandmeter> ikke: seems the http errors are gone?
2020-08-26 09:01:25 <clandmeter> or did you silence them?
2020-08-26 09:02:45 <ikke> No
2020-08-26 09:03:44 <clandmeter> did you upgrade gitlab?
2020-08-26 09:29:54 <Cogitri> ikke: Tested gitlab bot again, you can off gitlab-test now
2020-08-26 09:31:00 <Cogitri> Thanks for making the test instance 👍
2020-08-26 10:19:54 <ikke> clandmeter: not yet
2020-08-26 10:19:58 <ikke> Cogitri: cheers
2020-08-26 20:40:54 <ikke> clandmeter: want to upgrade gitlab tomorrow
2020-08-27 07:38:37 <Cogitri> ikke: Can you maybe send me the stage1/CMakeFiles/CMakeOutput.log from the x86_64 builder for ldc? No clue why it fails on the builders, worked on CI and still does locally
2020-08-27 07:40:44 <clandmeter> ikke: upgrade as patch or new version?
2020-08-27 07:45:26 <ikke> Patch
2020-08-27 07:45:53 <clandmeter> ill do it in a minute
2020-08-27 07:45:59 <ikke> Ok
2020-08-27 09:07:45 <clandmeter> ikke: i think we should review gitlab.yml config when we have some time
2020-08-27 09:08:04 <clandmeter> seems it has changed a lot over time. 
2020-08-27 09:08:28 <clandmeter> im adding: https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example#L205 and set it to true
2020-08-27 09:15:26 <ikke> Yes, it would make sense to update the configuration 
2020-08-27 10:08:21 <ikke> Cogitri: https://tpaste.us/Nmbl
2020-08-27 10:08:59 <Cogitri> Thanks 👍
2020-08-27 10:09:20 <ikke> clandmeter: Maybe we should keep a base version for the config files. Then we can do a proper 3-way merge
2020-08-27 10:09:22 <Cogitri> Also, not sure what went wrong here: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/11706#note_109957
2020-08-27 15:04:28 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/11706#note_109957
2020-08-27 16:59:22 <clandmeter> It's reproducible?
2020-08-27 17:01:50 <ikke> I haven't tried to reproduce it
2020-08-27 20:48:13 <clandmeter> looks like the git directory exists but its empty
2020-08-27 20:48:27 <clandmeter> the wiki git dir does seem right
2020-08-27 20:55:19 <clandmeter> ikke: i removed the project from his acc
2020-08-27 20:57:42 <clandmeter> ok i forked it for him and it seemed to work
2020-08-28 04:42:15 <ikke> clandmeter: alright
2020-08-28 14:10:53 <clandmeter> Dear Alpine Linux,
2020-08-28 14:10:53 <clandmeter> We are currently suffering an interruption to our network which will may impact the following of your services:
2020-08-28 14:10:53 <clandmeter> IS-57415
2020-08-28 14:34:08 <ikke> Does not ring a bell..
2020-08-28 18:26:01 <clandmeter> ikke: that was for UK
2020-08-28 18:26:36 <ikke> ah, understood
2020-08-30 12:07:24 <Cogitri> I think we talked about using MargeBot when we started using gitlab but I'm not sure why we decided to not use it. Does one of you happen to still know that? Would be nice if merging things was quicker
2020-08-30 12:07:46 <Cogitri> Right now it takes like 1~2 minutes for me to merge things because I have to go to the MR, rebase it, wait until rebasing is done and then finally merge it :/
2020-08-30 12:13:12 <ikke> Cogitri: nothing prevents you from rebasing locally and force pushing that back btw (if you can rebase in the first place)
2020-08-30 12:14:18 <Cogitri> That isn't that much quicker because then I have to setup a temporary remote for the user's repo, checkout their branch, pull, push and then switch to the webUI to merge
2020-08-30 12:14:44 <Cogitri> I sometimes do that when there are merge conflicts which are trivial to fix (e.g. pkgrel conflicts) so I don't have to annoy contributors with that
2020-08-30 12:16:47 <ikke> You can do that even without adding any remote 
2020-08-30 12:17:33 <Cogitri> Oh? How do I pull and push then? :o 
2020-08-30 12:17:56 <ikke> directly to the url
2020-08-30 12:18:10 <ikke> git fetch <remote_url>
2020-08-30 12:18:16 <ikke> this would leave the branch in FETCH_HEAD
2020-08-30 12:18:36 <ikke> gitlab has instructions on the page
2020-08-30 12:19:30 <Cogitri> Oh fancy, didn't know that I could use the URL directly
2020-08-30 12:35:40 <mps> interesting, what are '<remote_url>'
2020-08-30 12:36:19 <ikke> git@gitlab.alpinelinux.org:username/aports.git
2020-08-30 12:37:39 <mps> ah, that what we talked about two weeks ago. thanks
2020-08-30 12:37:52 <ikke> yes
2020-08-31 07:09:28 <clandmeter> 2020/08/31 07:08:45 invalid number of arguments, expected at least 1, got 1
2020-08-31 07:09:46 <clandmeter> :) must be quality software
2020-08-31 07:11:37 <maldridge> I take your error and raise you this one: xbps-rindex: invalid repository, existing!
2020-08-31 07:12:10 <maldridge> not sure how to console a repo manager that's in an existential crisis
2020-08-31 07:46:17 <clandmeter> ikke: im talking to ncopa about our gitlab deadlock issue
2020-08-31 07:47:39 <clandmeter> i think ncopa is busy atm, maybe we can check together later and see how to move further. 
2020-08-31 07:49:31 <ncopa> is there some way to attach to a deadlocking process?
2020-08-31 07:49:40 <ncopa> with gdb -p
2020-08-31 07:49:44 <ncopa> or strace
2020-08-31 07:51:35 <clandmeter> yes 
2020-08-31 07:51:40 <clandmeter> just ssh into gitlab server
2020-08-31 07:51:47 <clandmeter> but its on port 2222
2020-08-31 07:52:04 <clandmeter> else you will start the clone things :)
2020-08-31 07:52:28 <clandmeter> i have a list of gitaly errors here
2020-08-31 07:52:35 <clandmeter> ill pm them
2020-08-31 07:52:50 <clandmeter> im not sure its very useful but you never know.
2020-08-31 07:53:22 <clandmeter> ncopa: you need to strace from host not container
2020-08-31 08:05:12 <ncopa> what host is having problems? can i have a look at a strace?
2020-08-31 08:06:03 <ncopa> ok i found it
2020-08-31 08:12:43 <ncopa> so its gitaly-ssh upload-pack that hangs
2020-08-31 08:12:49 <ncopa> it looks like a grpc thing
2020-08-31 08:13:11 <ncopa> which talks with gitaly?
2020-08-31 08:14:03 <ncopa> i'd like to know what gitaly-ssh talks to
2020-08-31 08:21:08 <clandmeter> yes its probably related to grpc
2020-08-31 08:21:25 <clandmeter> but this gitaly stuff hooks into multiple things
2020-08-31 08:24:37 <ncopa> the usual suspect is fork/malloc/exec
2020-08-31 08:25:04 <ncopa> golang does not have any fork so any go code should not be affected by it
2020-08-31 08:25:19 <ncopa> but i see that there are gitaly ruby grpc too
2020-08-31 08:26:04 <clandmeter> there is gitaly-ruby which does rpc calls
2020-08-31 08:26:26 <ncopa> does the gitaly-ruby execute any external commands? using fork/exec?
2020-08-31 08:26:54 <clandmeter> https://docs.gitlab.com/ee/administration/gitaly/#gitaly-ruby
2020-08-31 08:26:56 <ncopa> i think we may want ask help from gitaly devs to debug this
2020-08-31 08:27:58 <clandmeter> i wanted to ask ikke where he noticed the issues
2020-08-31 08:28:13 <clandmeter> if its related to: RPCs that create commits on behalf of a user, such as merge commits.
2020-08-31 08:44:19 <clandmeter> ikke: the issue was reproducible from webif? 
2020-08-31 08:45:13 <ikke> clandmeter: I only had it once so far
2020-08-31 08:45:37 <clandmeter> and it was a common task you did?
2020-08-31 08:45:43 <ikke> rebasing an MR
2020-08-31 08:45:52 <ikke> through the webif
2020-08-31 08:45:55 <clandmeter> from webif
2020-08-31 08:46:01 <clandmeter> butr you do that more often?
2020-08-31 08:46:15 <clandmeter> its not like all those ops are broken now?
2020-08-31 08:47:27 <clandmeter> i wonder if we are bumping into: https://docs.gitlab.com/ee/administration/gitaly/#limit-rpc-concurrency
2020-08-31 08:49:08 <ncopa> i have found one potential problem in ruby
2020-08-31 08:49:42 <ncopa> i am not sure this is what we hit, but it looks like there is one place where it mallocs after fork
2020-08-31 08:50:35 <clandmeter> PROTOBUF?
2020-08-31 08:55:20 <ikke> clandmeter: no, it's not consistently broken
2020-08-31 09:37:58 <ikke> ncopa: We do use alpine:3.12 for gitlab, so shouldn't we be affected as much by the malloc after fork issue?
2020-08-31 10:34:43 <ncopa> i think 3.12 introduces some things that triggers the fork/exec issue
2020-08-31 10:35:00 <ncopa> at least libvirt started to deadlock with alpine 3.12
2020-08-31 10:35:00 <ikke> ah ok
2020-08-31 10:35:20 <ikke> I thought it was musl-1.2.1 that mostly introduced it
2020-08-31 10:35:45 <ncopa> musl 1.2 did it worse yes
2020-08-31 10:36:48 <ikke> ok
2020-08-31 10:41:12 <mps> sorry for nitpicking but musl didn't introduced this bug, musl just revealed it
2020-08-31 10:41:49 <ikke> It introduced this new behavior ;-)
2020-08-31 10:42:09 <ikke> We didn't call it a bug in musl
2020-08-31 10:42:38 <mps> well, we can twist whatever we want :)
2020-08-31 10:44:13 <ikke> In a discussion about finding possible causes for issues, it's quite relevant to talk about what version of a library introduced a certain behavior
2020-08-31 10:54:29 <mps> I posted url few days ago with suspect commit in dbus and nmeum added it in bug report
2020-08-31 10:55:22 <mps> sorry, no commit but bug report on dbus site
2020-08-31 10:56:03 <mps> (look like my head is full of commits :D )
2020-08-31 10:56:51 <ikke> This one? https://gitlab.freedesktop.org/dbus/dbus/-/issues/173
2020-08-31 10:57:02 <nmeum> yes
2020-08-31 12:02:06 <clandmeter> ikke: why did you upgrade to 3.12?
2020-08-31 12:03:42 <ikke> clandmeter: It required a newer version of Go
2020-08-31 12:04:37 <clandmeter> right
2020-08-31 12:06:18 <clandmeter> ncopa: which part were you talking about?
2020-08-31 12:17:38 <ncopa> https://github.com/ruby/ruby/blob/master/process.c#L2746
2020-08-31 12:17:53 <ncopa> but i dont know if that is relevant at all
2020-08-31 12:18:21 <ncopa> i thikn we should check what gitaly-ssh connects to, using netstat or similar
2020-08-31 12:28:57 <ncopa> hum, netstat does not show the process
2020-08-31 12:29:15 <ncopa> ls: /proc/16055/fd/3: cannot read link: Permission denied
2020-08-31 12:29:15 <ncopa> lrwx------    1 git      git             64 Aug 31 12:22 /proc/16055/fd/3
2020-08-31 12:31:37 <ncopa> i dont know how to figure out what it connects to
2020-08-31 12:32:24 <ikke> To a socket?
2020-08-31 12:32:26 <ikke> lsof
2020-08-31 12:33:25 <ncopa> nsenter -t 4541 -n netstat -p 
2020-08-31 12:33:31 <ncopa> from the host
2020-08-31 12:33:52 <ikke> 4541 is the PID?
2020-08-31 12:34:29 <ikke> Seems like it:
2020-08-31 12:34:31 <ikke> 4541 root       20   0  1600   572   488 S  0.0  0.0  0:02.25 sh /usr/local/bin/entrypoint.sh start
2020-08-31 12:36:14 <ikke> ncopa: lrwx------    1 clandmet clandmet        64 Aug 31 12:24 3 -> socket:[76223028] 
2020-08-31 12:36:17 <ikke> it has one socket open
2020-08-31 12:36:25 <ikke> pid 4341
2020-08-31 12:52:31 <ncopa> that is the gitaly-ssh process
2020-08-31 12:52:38 <ncopa> im interested in the other end
2020-08-31 12:53:01 <ikke> yeah, Not sure how you can find what's on the other side of the socket
2020-08-31 12:54:14 <ncopa> neither can i. but i suspect its the gitaly process
2020-08-31 13:00:47 <ikke> https://serverfault.com/a/417946/1615
2020-08-31 13:05:35 <ikke> u_str ESTAB  0      0                                                       * 76358185            * 76363823
2020-08-31 13:05:38 <ikke> u_str ESTAB  0      0      /home/git/gitlab/tmp/sockets/private/gitaly.socket 76363823            * 76358185
2020-08-31 13:06:04 <ikke> pid 4868
2020-08-31 13:06:15 <ncopa> which is gitaly
2020-08-31 13:06:17 <ikke> yes
2020-08-31 13:06:36 <ncopa> my thinking here is that: gitaly is implemented in go, so musl version should not make any difference
2020-08-31 13:06:46 <ncopa> however, it looks like it forwards something to ruby
2020-08-31 13:06:57 <ncopa> there are a couple of ruby processes
2020-08-31 13:07:12 <ikke> Doesn't the musl version matter when compiled?
2020-08-31 13:07:23 <ikke> or does it not use musl significantly?
2020-08-31 13:08:13 <ncopa> it is dynamically linked, so even if its built with musl 1.1, it may use musl 1.2 during runtime, if musl was updated
2020-08-31 13:08:23 <ikke> aha, ok
2020-08-31 13:08:58 <ncopa> but, inspecting those two ruby process
2020-08-31 13:09:23 <ncopa> shows that they are in pthread_cancel
2020-08-31 13:10:26 <ncopa> but this is a bit over my head
2020-08-31 13:15:10 <ncopa> this is the backtrace for the two ruby processes: https://tpaste.us/Ypmw
2020-08-31 13:16:08 <ikke> One being gitaly-ssh and the other gitaly?
2020-08-31 13:16:27 <ncopa> no
2020-08-31 13:16:59 <clandmeter> there is no unique gitaly process right?
2020-08-31 13:17:09 <clandmeter> its made up out of different processes 
2020-08-31 13:17:41 <ikke> gitaly-ruby and gitaly-ssh?
2020-08-31 13:17:48 <ncopa>   163 git       20   0  195.6m  78.1m   0.0   0.5  61:26.39 S              `- gitaly /etc/gitlab/gitaly/config.toml                                                                                                                                                                   
2020-08-31 13:17:48 <ncopa> 31921 git       20   0  227.1m  80.0m   0.0   0.5   0:02.89 S                  `- ruby /home/git/gitaly-ruby/bin/gitaly-ruby 163 /tmp/gitaly-internal201546454/ruby.0                                                                                                                 
2020-08-31 13:17:48 <ncopa>   436 git       20   0  227.1m  80.3m   0.0   0.5   0:02.08 S                  `- ruby /home/git/gitaly-ruby/bin/gitaly-ruby 163 /tmp/gitaly-internal201546454/ruby.1    
2020-08-31 13:18:02 <ncopa> gitaly spawns gitaly-ruby
2020-08-31 13:18:17 <ncopa> the backtrace was from both gitaly-ruby processes
2020-08-31 13:18:20 <clandmeter> ah right there is one
2020-08-31 13:19:13 <ncopa> but really, its a long shot
2020-08-31 13:19:31 <ncopa> i dont know if it is any of those processes that causes the deadlock
2020-08-31 13:21:53 <clandmeter> what i find interesting is that it does not happen all the time.
2020-08-31 13:22:40 <ikke> maybe some kind of race condition?
2020-08-31 13:22:47 <clandmeter> or load
2020-08-31 13:23:00 <ikke> well, load might cause a bad race condition ;-)
2020-08-31 13:25:35 <clandmeter> there is a rcp concurrency setting
2020-08-31 13:25:42 <clandmeter> maybe when it gets hit, it fails.
2020-08-31 13:26:06 <clandmeter> Each repository served by the Gitaly server can have at most 20 simultaneous PostUploadPack RPC calls in flight, and the same for SSHUploadPack.
2020-08-31 13:26:06 <clandmeter> If another request comes in for a repository that has used up its 20 slots, that request gets queued.
2020-08-31 13:27:18 <ikke> right, but that should not cause deadlocks I recon?
2020-08-31 13:27:34 <clandmeter> if  the problem is in the queuing 
2020-08-31 13:28:11 <clandmeter> we can open an issue on gitlab.org
2020-08-31 13:28:19 <ncopa> sounds like a good idea
2020-08-31 13:28:28 <clandmeter> not sure we have enough info to get them kickstarted
2020-08-31 14:04:36 <Cogitri> Can I set `/proc/sys/kernel/core_pattern` in my aarch64 container somehow? I get a SEGFAULT during the build() of an APKBUILD and I can't really debug it without having the coredump
2020-08-31 14:04:44 <Cogitri> Seems like /proc is ro in the container
2020-08-31 14:05:55 <ikke> hmm, more specifically, /proc/sys is ro
2020-08-31 14:07:19 <ikke> Cogitri: it's currently set to 'core'
2020-08-31 14:08:12 <ikke> Cogitri: shouldn't that mean the coredump should be written in PWD?
2020-08-31 14:08:33 <ikke> Cogitri: what does ulimit -c return?
2020-08-31 14:14:40 <Cogitri> Uhhh, actually, seems like it magically fixed itself since I last looked at it, sorry for the noise :)
2020-08-31 14:14:55 <ikke> heh
2020-08-31 14:15:21 <ikke> But I think it should be enough to set ulimit -c to a higher value in the future to get coredumps
2020-08-31 14:30:15 <Cogitri> Okie, thanks 👍️
2020-08-31 14:31:02 <ikke> `ulimit -c unlimited`
2020-08-31 17:45:37 <Cogitri> ikke: I've adjusted asgen's config, could you restart the container? I also think we cab delete the generated archived from June and July