2021-07-01 08:23:12 <clandmeter> ikke: looks like bundler is broken
2021-07-01 09:42:03 <ikke> clandmeter: ^
2021-07-01 09:42:19 <clandmeter> nod
2021-07-01 10:35:42 <clandmeter> ikke: i think i have a fix
2021-07-01 10:36:18 <ikke> ah, nice
2021-07-01 10:39:22 <clandmeter> the prlblem is gitaly
2021-07-01 10:40:16 <clandmeter> the offending commit is: https://gitlab.com/gitlab-org/gitaly/-/commit/365674695fcbc1ac6332cb8bbc646441e1e0f15a
2021-07-01 10:42:24 <clandmeter> deployment is the issue
2021-07-01 10:43:05 <clandmeter> need to look how to properly manage it
2021-07-01 10:43:13 <clandmeter> but for now i just it to false
2021-07-01 10:43:17 <clandmeter> ill push 13.12
2021-07-01 10:52:07 <ikke> ok
2021-07-01 10:54:57 <clandmeter> ikke: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/pipelines/86080
2021-07-01 10:55:06 <clandmeter> 🤞
2021-07-01 10:55:37 <ikke> btw: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/issues/7
2021-07-01 10:55:51 <ikke> That's something we need to change
2021-07-01 10:56:15 <clandmeter> yes
2021-07-01 10:56:25 <clandmeter> thats what i mean
2021-07-01 10:56:29 <clandmeter> we need to look at bundler
2021-07-01 10:56:32 <clandmeter> how to manage it
2021-07-01 10:56:37 <clandmeter> there are some changes also in docs
2021-07-01 10:58:10 <clandmeter> its a bit tricky 
2021-07-01 10:58:27 <clandmeter> in principle we need to use bundler config whatever
2021-07-01 10:58:40 <clandmeter> but it seems that ie gitaly does not honor those settings 
2021-07-01 10:58:57 <clandmeter> it will only look at those env vars
2021-07-01 10:59:14 <ikke> right, so they are doing some custom things?
2021-07-01 10:59:59 <clandmeter> yes
2021-07-01 11:00:06 <clandmeter> we are actually doing a hack
2021-07-01 11:00:13 <clandmeter> and they unhack us :)
2021-07-01 11:00:37 <clandmeter> in short
2021-07-01 11:00:40 <ikke> :D
2021-07-01 11:00:41 <clandmeter> from wh at i understand
2021-07-01 11:01:02 <clandmeter> in production you need to set deployment = true
2021-07-01 11:01:18 <clandmeter> and it will bundle the deps per bundle into its own bundle
2021-07-01 11:02:03 <clandmeter> its the normal world nowdays
2021-07-01 11:02:21 <clandmeter> copy a gazillion similar deps everywhere
2021-07-01 15:57:44 <ikke> clandmeter: build passed
2021-07-01 17:49:05 <ikke> oh
2021-07-01 18:56:15 <ikke> clandmeter: as you might have noticed, I started restoring an up-to-date version of gitlab to gitlab-test
2021-07-01 18:56:28 <clandmeter> yup
2021-07-01 18:57:20 <ikke> was the update already pushed to github?
2021-07-01 18:57:30 <clandmeter> github?
2021-07-01 18:57:53 <ikke> we still build gitlab with drone, right?
2021-07-01 18:58:06 <ikke> build to push to hub
2021-07-01 18:58:12 <clandmeter> we do?
2021-07-01 18:58:15 <ikke> yes
2021-07-01 18:58:31 <ikke> You felt that we needed to be able to build gitlab outside of gitlab
2021-07-01 18:58:40 <clandmeter> ah ok
2021-07-01 18:58:43 <clandmeter> we also build on gitlab
2021-07-01 18:58:47 <ikke> yes, just as CI
2021-07-01 18:58:48 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/jobs/427221
2021-07-01 18:58:52 <clandmeter> ah ok
2021-07-01 18:58:53 <ikke> I've added that
2021-07-01 18:58:59 <clandmeter> yeah
2021-07-01 18:59:06 <ikke> If you want, I can also let it push to hub
2021-07-01 18:59:16 <clandmeter> it was pushed
2021-07-01 18:59:20 <clandmeter> as it shows on hub
2021-07-01 18:59:23 <ikke> ok
2021-07-01 18:59:29 <clandmeter> but i had no idea it was via github
2021-07-01 18:59:33 <clandmeter> too long ago :)
2021-07-01 18:59:37 <ikke> yeah, I see the 13.12 branch there
2021-07-01 18:59:58 <ikke> ok, then I'll upgrade gitlab-test as soon as it's ready
2021-07-01 19:00:29 <ikke> although, wasn't 13.12 the version that Thalheim mentioned?
2021-07-01 19:01:30 <clandmeter> 3.12.5 definitely works, and 14.x works too if y'all want to upgrade to latest
2021-07-01 19:01:42 <ikke> ah
2021-07-01 19:01:44 <ikke> good
2021-07-01 19:28:35 <ikke> clandmeter: acme-client is failing on git.alpinelinux.org (which updates distfiles.a.o)
2021-07-01 19:28:54 <clandmeter> thats the old client?
2021-07-01 19:29:14 <clandmeter> i think that one is no l onger supported
2021-07-01 19:29:22 <ikke> ah ok
2021-07-01 19:29:25 <clandmeter> you will need something like uacme 
2021-07-01 19:29:36 <clandmeter> or use our wildcard 
2021-07-01 19:29:42 <ikke> yeah, was wondering about that
2021-07-01 19:29:50 <ikke> there is already a script which fetches the certs from cert.a.o
2021-07-01 19:30:09 <clandmeter> right, so maybe the certs are already on the server?
2021-07-01 19:31:32 <ikke> yes
2021-07-01 19:32:00 <ikke> and up-to-date
2021-07-01 19:33:01 <clandmeter> simple life :)
2021-07-01 19:33:11 <ikke> heh
2021-07-01 19:33:13 <clandmeter> easy i mean...
2021-07-01 19:33:36 <ikke> that's better
2021-07-01 19:33:48 <ikke> {"days_left": 56, "hours_left": 1350, "end_date": "Aug 27 02:00:13 2021 GMT"}
2021-07-01 19:34:32 <clandmeter> :)
2021-07-01 19:34:41 <clandmeter> weird that it happens this late
2021-07-01 19:34:51 <clandmeter> i think i had htis error on another one few months ago
2021-07-01 19:35:36 <ikke> probaby just updated right before support was dropped
2021-07-01 19:38:02 <ikke> distfiles was the only one left not using the wildcard yet
2021-07-01 19:38:33 <ikke> good that we monitor it :P
2021-07-01 19:38:44 <ikke> clandmeter: I have bad news for you :P
2021-07-01 19:38:59 <clandmeter> 
ACTION hides

2021-07-01 19:39:05 <ikke> GitLab Security Release 14.0.2, 13.12.6, and 13.11.6
2021-07-01 19:40:35 <clandmeter> cooking
2021-07-01 19:43:53 <ikke> rsync is hanging on ppc64le
2021-07-01 19:44:21 <clandmeter> should we send that email?
2021-07-01 19:46:02 <ikke> clandmeter: I suppose
2021-07-01 19:48:46 <ikke> \o/
2021-07-01 19:52:17 <minimal> I noticed on the main alpinelinux.org homepage the "Latest Development" that is supposed to show the last few merges to aports hasn't changed since 15th June
2021-07-01 19:52:26 <ikke> minimal: oh
2021-07-01 19:53:26 <ikke> I did change something to fix it responding to updates from gitlab, maybe I broke this part
2021-07-01 19:55:42 <minimal> ikke: the Microsoft approach of a new release bringing new features and also breaking existing stuff? ;-) What he gives with one hand he takes away with the other lol
2021-07-01 19:55:58 <ikke> right, learn from the best, they say, right?
2021-07-01 19:56:25 <minimal> so Alpine's going to require a TPM in the next release? lol
2021-07-01 19:57:17 <ikke> And a CPU that is 14 weeks old
2021-07-01 19:57:20 <ikke> 2 weeks*
2021-07-01 19:57:59 <minimal> 14 weeks old? so like 5 Intel microcode updates behind for security issues? :-)
2021-07-01 19:58:03 <ikke> :D
2021-07-01 20:01:54 <ikke> I think it should be fixed now
2021-07-01 20:02:01 <ikke> waiting for the next commit
2021-07-01 20:05:32 <clandmeter> ikke: did you break the internet?
2021-07-01 20:05:44 <ikke> as usual
2021-07-01 20:05:48 <clandmeter> https://build.alpinelinux.org/
2021-07-01 20:05:57 <ikke> ssssht
2021-07-01 20:06:16 <ikke> it means you are not active enough
2021-07-01 20:06:51 <Thalheim> thanks for the notice on the 14.0.2 and 13.12.6 releases; we've upgraded to 14.0.2 now.
2021-07-01 20:07:23 <ikke> Thalheim: You can subscribe to security e-mails from gitlab
2021-07-01 20:07:28 <ikke> That's how I knew
2021-07-01 20:07:47 <clandmeter> i told you, give me the credit :p
2021-07-01 20:07:50 <ikke> :D
2021-07-01 20:07:56 <Thalheim> 
ACTION applauds both of you

2021-07-01 20:08:32 <clandmeter> Thalheim: who is we if i may ask?
2021-07-01 20:09:25 <Thalheim> Adélie
2021-07-01 20:09:28 <ikke> ah
2021-07-01 20:09:34 <clandmeter> aha :)
2021-07-01 20:09:58 <ikke> do you use omnibus?
2021-07-01 20:10:22 <clandmeter> they did before 
2021-07-01 20:10:38 <Thalheim> I underwent the excruciating process of migrating from a source-built omnibus situation to the dockerized version, needing to hit all minor releases along the way :)
2021-07-01 20:10:56 <Thalheim> no more gem incompatibility and LD_PRELOAD hacks
2021-07-01 20:11:02 <ikke> heh
2021-07-01 20:11:11 <clandmeter> you build your own docker images?
2021-07-01 20:11:23 <Thalheim> no, using gitlab/gitlab-ce:tag ones
2021-07-01 20:11:36 <ikke> yeah, I expected, since you updated so quickly
2021-07-01 20:11:48 <clandmeter> those are still ubuntu based i guess?
2021-07-01 20:12:01 <ikke> minimal: fixed, thanks for reporting
2021-07-01 20:12:03 <Thalheim> I think so. tbh don't remember. I can share my notes on the transition and all the steps later this evening if that would be helpful.
2021-07-01 20:12:12 <ikke> We have our own docker images
2021-07-01 20:12:15 <ikke> alpine based
2021-07-01 20:12:21 <clandmeter> we eat our own dogfood :)
2021-07-01 20:12:36 <clandmeter> i kill our headaches with paracetamol :)
2021-07-01 20:12:41 <Thalheim> lol
2021-07-01 20:12:54 <clandmeter> its not that bad
2021-07-01 20:13:11 <clandmeter> just some minor breakage from time to time
2021-07-01 20:13:27 <ikke> the upgrade itself is really simple, but we test it before we deploy it to production
2021-07-01 20:13:44 <Thalheim> breaking from our previous policy, dogfood is the goal but not the requirement if it's impeding progress.
2021-07-01 20:14:11 <clandmeter> true
2021-07-01 20:14:20 <clandmeter> but the nice thing is, you know gitlab a bit better.
2021-07-01 20:14:43 <clandmeter> so if shit hits the fan, you know how to take it apart :)
2021-07-01 20:14:46 <Thalheim> I like to work on antique cars but I don't want to drive them to work
2021-07-01 20:14:49 <ikke> and, except for security issues, having the latest and greatest gitlab version is not that important
2021-07-01 20:15:31 <clandmeter> i remember adelie was using an old version for a longer time
2021-07-01 20:16:06 <clandmeter> but they dont support sec updates for older than few versions
2021-07-01 20:16:18 <ikke> 3 versions
2021-07-01 20:21:55 <clandmeter> ikke: gitlab is uplaoded
2021-07-01 20:22:01 <ikke> clandmeter: thanks
2021-07-01 20:22:18 <ikke> sadly, the backup restore takes quite a long time for some reason
2021-07-01 20:22:26 <clandmeter> its still running?
2021-07-01 20:22:29 <ikke> yes
2021-07-01 20:22:33 <clandmeter> hmm
2021-07-01 20:22:48 <ikke> I understood they restored things on file level
2021-07-01 20:23:02 <clandmeter> maybe too many files?
2021-07-01 20:23:05 <ikke> yeah
2021-07-01 20:23:13 <ikke> I already pruned a lot of empty dirs before
2021-07-01 20:24:51 <ikke> find . -type d -depth -empty | wc -l
2021-07-01 20:24:53 <ikke> 104224
2021-07-01 20:24:55 <ikke> oof
2021-07-01 20:26:57 <ikke> I guess we should prune that again
2021-07-01 20:55:51 <clandmeter> how far is that backup restore?
2021-07-01 20:56:50 <clandmeter> 8%? say what?
2021-07-01 20:57:29 <clandmeter> blaboon: is it common for the restore to be slow like this?
2021-07-01 22:17:09 <blaboon> that backup job does appear to be going unusually slow, although it is making progress
2021-07-01 22:18:19 <blaboon> sometimes we see this happen with backups that have a very large number of files in them, like on the order of millions
2021-07-02 04:32:45 <ikke> df -i -> 3.855.578
2021-07-02 07:03:50 <clandmeter> ikke: i pushed some changes to gitlab
2021-07-02 07:03:56 <clandmeter> specifically the ones you mentioned
2021-07-02 10:57:18 <ikke> clandmeter: "and display steps": \o/
2021-07-02 11:20:40 <ikke> clandmeter: https://gitlab-test.alpinelinux.org is now 13.12.6
2021-07-02 13:04:26 <Thalheim> 500 https://gitlab-test.alpinelinux.org/alpine/infra/alpine-mksite
2021-07-02 13:04:57 <Thalheim> most or all repos are 500
2021-07-02 13:06:57 <Thalheim> as well as issues and mr's https://gitlab-test.alpinelinux.org/groups/alpine/-/merge_requests
2021-07-02 13:10:14 <MY-R> maybe because it is just some test bed and not exact copy of orig repo, dunno
2021-07-02 13:58:47 <ikke> It is
2021-07-02 13:59:23 <MY-R> ^ Thalheim
2021-07-02 13:59:44 <ikke> I mean, it is a copy
2021-07-02 13:59:50 <MY-R> ah
2021-07-02 14:00:15 <MY-R> then I see 500 too :P
2021-07-02 14:00:20 <ikke> We restore a snapshot of the gitlab server
2021-07-02 14:00:53 <ikke> I already saw test failures in our acceptance tests 
2021-07-02 15:16:54 <ikke> seems like grpc issues
2021-07-02 17:21:18 <ikke> time="2021-07-02T17:21:03Z" level=fatal msg="unsupported Git version: \"2.30.2\""
2021-07-02 17:21:21 <ikke> sigh
2021-07-02 17:21:33 <ikke> gitaly is failing to start
2021-07-02 17:27:03 <ikke> using git from v3.14 fixes the issue
2021-07-02 17:27:07 <ikke> clandmeter: ^
2021-07-02 17:30:26 <clandmeter> Huh
2021-07-02 17:30:36 <ikke> gitaly requires git >2.31.0
2021-07-02 17:30:43 <clandmeter> Yes
2021-07-02 17:30:51 <clandmeter> We build it ourselves?
2021-07-02 17:30:58 <ikke> https://gitlab.com/gitlab-org/gitaly/-/tree/13-12-stable
2021-07-02 17:31:06 <ikke> build what ourselves?
2021-07-02 17:31:11 <clandmeter> Git
2021-07-02 17:31:12 <ikke> no
2021-07-02 17:31:18 <ikke> the image is using alpine 3.13
2021-07-02 17:31:26 <ikke> which has git 2.30.2
2021-07-02 17:31:47 <clandmeter> I thought w already build it local
2021-07-02 17:32:04 <ikke> apk add .. git
2021-07-02 17:32:13 <clandmeter> Nod
2021-07-02 17:32:26 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/blob/13.7-stable/overlay/usr/local/bin/setup.sh#L39
2021-07-02 17:32:42 <clandmeter> Right
2021-07-02 17:33:31 <ikke> So do we want to build git?
2021-07-02 17:33:37 <clandmeter> But the docs say to build it manually
2021-07-02 17:33:46 <clandmeter> I think I mention it before
2021-07-02 17:34:03 <ikke> Probably, we talked about it with the previous time it came up
2021-07-02 17:34:17 <ikke> I can see if I can add it in
2021-07-02 17:34:52 <ikke> clandmeter: do you know which docs?
2021-07-02 17:35:17 <clandmeter> Install from source doscs
2021-07-02 17:36:17 <ikke> https://docs.gitlab.com/ee/install/installation.html#installation-from-source
2021-07-02 17:36:44 <ikke> https://docs.gitlab.com/ee/install/installation.html#git
2021-07-02 17:36:57 <ikke> they have their own fork apparently
2021-07-02 17:37:52 <ikke> the difficulty is that we need git to fetch the source of a lot of things as well, so we need to make sure we build git before that
2021-07-02 17:40:11 <clandmeter> I don't think it matters
2021-07-02 17:40:30 <ikke> what?
2021-07-02 17:40:31 <clandmeter> Just uninstall git after
2021-07-02 17:41:48 <clandmeter> System git I mean
2021-07-02 17:43:00 <ikke> We need to change gitlab to point to the built git version
2021-07-02 17:43:17 <clandmeter> Yup
2021-07-02 17:43:33 <clandmeter> This has changed recently
2021-07-02 17:43:59 <ikke> ?
2021-07-02 17:44:21 <clandmeter> Their own version
2021-07-02 17:44:39 <clandmeter> It used to just need very recent
2021-07-02 17:45:21 <ikke> 13.11 changed it to 2.31
2021-07-02 17:46:26 <ikke> so I move git from runtime to buildtime
2021-07-02 17:47:43 <clandmeter> Nod
2021-07-02 17:47:55 <clandmeter> Maybe you need some deps to build it
2021-07-02 17:48:23 <ikke> yes
2021-07-02 17:48:51 <ikke> I just wonder how actually git gets compiled, I cannot really find it in the gitaly Makefile
2021-07-02 17:50:35 <clandmeter> I thought I saw something like that in the logs. Is that's possible?
2021-07-02 17:50:57 <ikke> `sudo make git GIT_PREFIX=/usr/local`
2021-07-02 17:50:59 <ikke> really
2021-07-02 17:51:23 <ikke> clandmeter: like what?
2021-07-02 17:51:41 <clandmeter> Libgit
2021-07-02 17:51:48 <clandmeter> Could be done gem
2021-07-02 17:51:54 <clandmeter> Some
2021-07-02 17:52:01 <ikke> libgit2 is not git
2021-07-02 17:52:47 <clandmeter> I'm on mobile so can't check
2021-07-02 17:53:01 <ikke> nod
2021-07-02 17:59:47 <ikke> We need to change the bin_path in gitlab.yml
2021-07-02 19:00:39 <ikke> Hmm, gitaly is not installing git
2021-07-02 19:01:09 <ikke> oh, my bad
2021-07-02 19:40:04 <clandmeter> got it wokring?
2021-07-02 19:42:44 <ikke> not yet
2021-07-02 19:42:50 <ikke> but it's building git now
2021-07-02 20:35:25 <ikke> clandmeter: btw, the /var/lib/docker directory contains 3m+ files/directories
2021-07-02 20:35:31 <ikke> wondering why so many
2021-07-02 20:36:00 <clandmeter> uhm
2021-07-02 20:36:05 <clandmeter> docker volume?
2021-07-02 20:36:39 <ikke> only 10k
2021-07-02 20:39:48 <ikke> overlay2
2021-07-02 20:39:58 <ikke> 3652924
2021-07-02 21:06:48 <clandmeter> ikke: so the files are in docker but not in the container?
2021-07-02 21:07:04 <ikke> I don't know
2021-07-02 21:08:24 <ikke> Currently fixing git in the container
2021-07-02 21:08:29 <ikke> was missing runtime dependencies
2021-07-02 21:09:37 <clandmeter> the script didnt find them?
2021-07-02 21:11:21 <ikke> I added the build-time dependencies, but they got removed after build
2021-07-02 21:11:30 <ikke> I'm used to abuild tracing the runtime dependencies :P
2021-07-02 21:13:15 <clandmeter> but there is a script included
2021-07-02 21:13:17 <clandmeter> it should do that
2021-07-02 21:13:25 <clandmeter> ah
2021-07-02 21:13:28 <clandmeter> i know
2021-07-02 21:13:39 <clandmeter> it only scans ruby path
2021-07-02 21:13:44 <ikke> :)
2021-07-02 21:13:57 <ikke> gemdeps
2021-07-02 21:14:11 <clandmeter> yes
2021-07-02 21:14:21 <clandmeter> well we could rename it and add paths as options
2021-07-02 21:15:15 <ikke> they are just 3 dependencies
2021-07-02 21:38:44 <ikke> clandmeter: now it's happy
2021-07-02 21:38:57 <ikke> Do you think it's worth it to change the script?
2021-07-02 21:44:33 <ikke> pushed it
2021-07-03 09:57:08 <ikke> clandmeter: gitlab-test is now running the latest image (with the compiled git version)
2021-07-03 10:19:31 <clandmeter> \o/
2021-07-03 10:22:32 <ikke> It's so nice to have a test environment to verify these things
2021-07-03 10:39:14 <ikke> clandmeter: so I think about upgrading gitlab this evening
2021-07-03 10:39:38 <clandmeter> fine by me, but im probably not around :)
2021-07-03 10:41:00 <ikke> ok
2021-07-03 14:01:40 <ikke> clandmeter: ^
2021-07-03 14:01:44 <ikke> we lost usa5 it seems
2021-07-03 15:56:16 <ikke> hmm, the host is responding
2021-07-03 15:56:55 <ikke> seems memory issues, do not get a prompt after login
2021-07-03 16:13:35 <ikke> clandmeter: I had to force reboot usa5. 
2021-07-03 16:53:41 <ikke>  sigh
2021-07-03 22:26:58 <ikke> blaboon: a message from linode about a trusted device was marked as 'could not be verified it came from the sender' in gmail for me (and ended up in the spam folder)
2021-07-03 22:28:20 <ikke> "Gmail could not verify that it actually came from linode.com. Avoid clicking links, downloading attachments, or replying with personal information."
2021-07-03 22:28:31 <ikke> With a large orange banner
2021-07-03 23:22:15 <blaboon> hmm, i just got one of those emails myself a few days ago that passed SPF, although i'm not using gmail. and i don't think we've made any recent changes to our policy
2021-07-03 23:22:38 <blaboon> would you mind forwarding that along to support@linode.com? they'll be able to check it and escalate if needed
2021-07-04 03:21:37 <rhatr> hey y'all! quick question: how exactly is https://hub.docker.com/_/alpine?tab=tags&page=1&ordering=last_updated maintained? Specifically the edge tag. What's the schedule of it being pushed?
2021-07-04 06:40:17 <ikke> rhatr: there is no regular schedule
2021-07-04 06:41:18 <ikke> rhatr: ncopa needs to crate a snapshot tag, and then make a pull request to get the image updated
2021-07-04 17:13:32 <nmeum> the aarch64 CI seems to have run out of disk space https://gitlab.alpinelinux.org/alpine/aports/-/jobs/429603
2021-07-04 17:13:34 <nmeum> > mkdir: can't create directory 'keys/': No space left on device
2021-07-04 17:13:44 <nmeum> ah, algibot already knows!
2021-07-04 17:14:37 <ikke> ceph...
2021-07-04 17:16:02 <ikke> right now 37G is available
2021-07-04 17:16:51 <ikke> CI has little to no persisted data
2021-07-04 17:17:51 <ikke> for some reason, ceph likes to use a lot of space to build
2021-07-04 17:22:55 <nmeum> ah!
2021-07-04 18:58:14 <rhatr> ikke: got it! so... what would it take to start publishing riscv64? convincing ncopa: ?
2021-07-04 19:00:02 <clandmeter> rhatr: you cannot
2021-07-04 19:00:13 <clandmeter> docker does not support rv64
2021-07-04 19:00:22 <clandmeter> in their infra
2021-07-04 19:00:31 <clandmeter> afaik
2021-07-04 19:01:31 <rhatr> well it very much does everywhere else
2021-07-04 19:02:14 <clandmeter> docker needs hardware to support the architecture.
2021-07-04 19:02:17 <rhatr> Mac OS, Windows, Linux -- any docker older than 19.x supports it pretty well (on Linux -- obviously -- it always requires qemu-static -- but that's for everything)
2021-07-04 19:02:29 <clandmeter> there is an issue on github about it
2021-07-04 19:02:32 <ikke> rhatr: docker needs to have hardware before they can publish official images in the library
2021-07-04 19:02:36 <ikke> you can push images yourself
2021-07-04 19:02:43 <ikke> but for official images, they need to build it
2021-07-04 19:03:05 <rhatr> wait... that was my entire question to begin with -- who builds it
2021-07-04 19:03:14 <ikke> rhatr: https://github.com/alpinelinux/docker-alpine/pull/148
2021-07-04 19:03:25 <ikke> rhatr: docker builds the official image
2021-07-04 19:04:05 <rhatr> ah! so it ain't ncopa: 
2021-07-04 19:04:21 <ikke> He provides the pull requests, but the actual image is built on docker infra
2021-07-04 19:04:24 <rhatr> (given the answer about I thought he physically is in charge of building and pushing the image)
2021-07-04 19:04:47 <rhatr> got it -- it makes it MUCH clear now -- let me talk to a friend of mine who happens to a CTO at Docker Inc. :-)
2021-07-04 19:05:23 <ikke> :D
2021-07-04 19:07:00 <ikke> rhatr: https://github.com/docker-library/official-images
2021-07-04 19:07:15 <ikke> https://github.com/docker-library/official-images/blob/master/library/alpine
2021-07-04 19:11:25 <ikke> clandmeter: I wonder if we should publish an alpinelinux/alpine image for riscv64
2021-07-04 19:13:58 <clandmeter> makes sense
2021-07-04 19:15:05 <clandmeter> rhatr: https://github.com/docker-library/official-images/issues/8794
2021-07-04 19:15:05 <ikke> clandmeter: fyi, forking projects is broken atm
2021-07-04 19:15:12 <clandmeter> wut?
2021-07-04 19:15:23 <clandmeter> what happend?
2021-07-04 19:15:33 <clandmeter> bug?
2021-07-04 19:15:37 <ikke> appears so
2021-07-04 19:15:41 <ikke> NoMethodError
2021-07-04 19:15:59 <clandmeter> hmm
2021-07-04 19:16:00 <ikke> https://github.com/docker-library/official-images/blob/master/library/alpine
2021-07-04 19:16:03 <ikke> https://tpaste.us/Bgeb
2021-07-04 19:16:06 <clandmeter> ruby is missing something?
2021-07-04 19:16:11 <ikke> oh, nilclass
2021-07-04 19:16:42 <ikke> there is a ServiceCreator that should return a Project
2021-07-04 19:16:49 <ikke> but apparently it returns nil
2021-07-04 19:25:46 <clandmeter> sounds scary
2021-07-04 19:34:03 <clandmeter> rhatr: https://twitter.com/clandmeter/status/1411770239637983236
2021-07-04 19:35:23 <ikke> clandmeter: Could not find anythign in the bugtracker
2021-07-04 20:12:54 <ikke> Not sure what to do with it
2021-07-04 20:13:08 <ikke> Not sure how to debug that code
2021-07-04 20:41:42 <clandmeter> ikke: i can take a look tomorrow
2021-07-04 20:42:33 <clandmeter> if i cant find it maybe ping somebody from gitlab
2021-07-04 20:43:13 <ikke> nod
2021-07-05 07:18:44 <clandmeter> logs are a bit difficult to read when we get hit that much
2021-07-05 07:19:45 <ikke> I use the json logs with jq
2021-07-05 07:20:25 <ikke> And tail 
2021-07-05 07:20:40 <clandmeter> https://tpaste.us/Qr44
2021-07-05 07:20:44 <clandmeter> thats it i guess
2021-07-05 07:22:14 <ikke> Yes, what I already posted yesterday 
2021-07-05 07:22:44 <ikke> @project somehow is nil
2021-07-05 07:27:47 <clandmeter> how did you come to that conclusion?
2021-07-05 07:38:28 <ikke> The error message, combined with the source code 
2021-07-05 07:39:02 <ikke> It tries to call persisted? on a nil object
2021-07-05 07:39:38 <ikke> NoMethodError (undefined method `persisted?' for nil:NilClass):
2021-07-05 07:39:40 <ikke> app/services/projects/fork_service.rb:35:in `fork_new_project'
2021-07-05 07:42:30 <clandmeter> ah ok you already did some research 
2021-07-05 07:45:25 <danieli> new_project = CreateService.new(current_user, new_fork_params).execute
2021-07-05 07:45:25 <danieli> return new_project unless new_project.persisted?
2021-07-05 07:45:44 <danieli> why would CreateService.new(...).execute return nil?
2021-07-05 07:47:59 <ikke> That's what I was wondering 
2021-07-05 07:49:55 <danieli> ikke: shot in the dark - try changing visibility of the project
2021-07-05 07:50:02 <danieli> i've seen a vaguely familiar bug before
2021-07-05 07:51:13 <clandmeter> related: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42607
2021-07-05 07:51:15 <ikke> It happ
2021-07-05 07:51:49 <danieli> clandmeter: looks like a different issue considering the error message in that one, but it could be related
2021-07-05 07:52:00 <clandmeter> yes just related
2021-07-05 07:52:07 <clandmeter> could be visibilty related
2021-07-05 07:52:18 <clandmeter> i tried to switch, but didnt fix it
2021-07-05 07:52:28 <clandmeter> also read something about caching 
2021-07-05 07:52:32 <danieli> i'm firing up emacs to grep around a bit
2021-07-05 07:52:44 <clandmeter> maybe we need to clear cache where possible 
2021-07-05 07:56:35 <clandmeter> danieli: you are still using thelounge?
2021-07-05 07:56:41 <danieli> clandmeter: yup
2021-07-05 07:57:02 <clandmeter> i think the new version will support searching (if i read the commits correctly)
2021-07-05 07:57:12 <clandmeter> history search
2021-07-05 07:57:26 <danieli> that's neat, i've checked third party logs or scrolled up and hit ctrl+f
2021-07-05 07:58:29 <clandmeter> ikke: we have some specific restriction settings right?
2021-07-05 08:01:07 <danieli> it's a shame the traceback says nearly nothing about what's interesting, it's all about the error boiling up through the web stack
2021-07-05 08:02:12 <clandmeter> ikke: did you report this upstream?
2021-07-05 08:11:44 <ikke> Not yet 
2021-07-05 09:17:30 <clandmeter> ikke: is inbound email still working?
2021-07-05 09:19:21 <ikke> No idea? 
2021-07-05 09:28:55 <clandmeter> ikke: https://tpaste.us/ovQn
2021-07-05 09:32:54 <clandmeter> does api allow forking?
2021-07-05 09:46:59 <clandmeter> ikke, danieli: https://gitlab.com/gitlab-org/gitlab/-/issues/335187
2021-07-05 09:49:14 <danieli> 👍
2021-07-05 09:58:36 <ikke> clandmeter: good
2021-07-05 10:04:08 <ikke> I hope we get a timely response
2021-07-05 12:03:29 <clandmeter> ikke: something is wrong with gitlab pull
2021-07-05 12:04:13 <ikke> You mean the delay? 
2021-07-05 12:04:17 <clandmeter> yes
2021-07-05 12:04:31 <clandmeter> there was no delay over https
2021-07-05 12:04:45 <clandmeter> but now its taking a lot of time to start fetching
2021-07-05 12:07:03 <clandmeter> when i noticed the delay over ssh, i switched my pull logic to https
2021-07-05 15:21:26 <clandmeter> ikke, ncopa: did you guys see the ppc emails?
2021-07-05 15:21:32 <ikke> yes
2021-07-05 15:21:35 <clandmeter> wonder if they actually fixed anything?
2021-07-05 15:22:18 <clandmeter> ikke: i think you already played with MTU 
2021-07-05 15:22:27 <ikke> yes
2021-07-05 15:31:40 <clandmeter> ikke: please test it and let me know. ill try to keep pushing them.
2021-07-05 15:31:58 <ikke> Need to wait until the builder is idle
2021-07-05 15:32:14 <ikke> Can check 3-14-ppc64le
2021-07-05 15:32:32 <clandmeter> nod
2021-07-05 15:33:27 <clandmeter> ikke: maybe we should test the fork issue on our test instance
2021-07-05 15:33:47 <ikke> yes, that's what I was doing
2021-07-05 15:33:56 <ikke> It fails there as well
2021-07-05 15:34:04 <clandmeter> if we dont get feedback, we could try upgrade to 14
2021-07-05 15:34:08 <ikke> Just wondering how to get debug output
2021-07-05 15:34:23 <ikke> hmm, all stable builders are stopped
2021-07-05 15:35:25 <ikke> same for edge
2021-07-05 15:36:03 <clandmeter> same what?
2021-07-05 15:36:18 <ikke> lxc containers stopped
2021-07-05 15:36:42 <clandmeter> oh you mean REALLY stopped
2021-07-05 15:37:03 <ikke> build-3-14-ppc64le STOPPED
2021-07-05 15:37:09 <clandmeter> powered off, if thats a thing for comtainers :)
2021-07-05 15:37:17 <ikke> heh
2021-07-05 15:37:35 <ikke> in my defense, lxc calls it stopped :P
2021-07-05 15:38:13 <clandmeter> i guess they dont auto start on boot?
2021-07-05 15:38:18 <ikke> yeah
2021-07-05 15:38:46 <ikke> I guess we need to add autostart
2021-07-05 15:39:16 <ikke> lxc.start.auto = 1
2021-07-05 15:39:20 <ikke> that's missing
2021-07-05 15:39:31 <clandmeter> i would assume so
2021-07-05 15:39:56 <ikke> It's the case :P
2021-07-05 15:40:17 <clandmeter> bad management of lxc containers ;-)
2021-07-05 15:40:36 <ikke> agreed
2021-07-05 15:41:44 <ikke> added it to all containers now
2021-07-05 15:42:16 <clandmeter> prefer bad config instead of bad networking...
2021-07-05 15:42:41 <ikke> bad config we can fix
2021-07-05 15:44:20 <ikke> 'pulling git' takes a long time
2021-07-05 15:46:03 <clandmeter> that was same for me
2021-07-05 15:46:28 <ikke> for me it's finished within 10 secs
2021-07-05 15:47:28 <clandmeter> now its also fast for me
2021-07-05 15:47:29 <clandmeter> weird
2021-07-05 15:57:26 <ikke> still showing that
2021-07-05 15:57:43 <ikke> hmm, even though the process is gone
2021-07-05 18:19:31 <ikke> clandmeter: ok, doing what rafael said actually helps
2021-07-05 18:19:37 <ikke> mtu 1450, and git continues
2021-07-05 19:01:29 <ikke> clandmeter: I've synced 3.14 up now (to get rid of the files in .~tmp~), so we'll see if that comes back
2021-07-05 19:47:12 <clandmeter> Nice
2021-07-05 19:47:42 <ikke> waiting for edge to finish as wel
2021-07-05 19:47:44 <ikke> well
2021-07-05 20:18:37 <clandmeter> not much activity from gitlab
2021-07-06 06:07:49 <ikke> I freed up space on usa2
2021-07-06 10:19:39 <ikke> clandmeter: syncing ppc64le community now, and it seems it still hangs randomly
2021-07-06 10:27:05 <ikke> clandmeter: but tcpdump also does not show a lot of packets going over
2021-07-06 10:42:07 <clandmeter> ok
2021-07-06 10:42:40 <ikke> and now broken pipe
2021-07-06 10:42:59 <clandmeter> can you add this info to the email? 
2021-07-06 10:43:01 <ikke> as if it randomly justs stops
2021-07-06 10:43:08 <ikke> is it a network issue?
2021-07-06 10:43:32 <clandmeter> what else could it be?
2021-07-06 10:43:36 <ikke> let me try tcpdump from both sides
2021-07-06 10:44:01 <clandmeter> what happens if you sync to another site?
2021-07-06 10:46:46 <clandmeter> ikke: i asked ncopa regarding ap and different repo's. it seems its a bug and should be fixed.
2021-07-06 10:47:22 <ikke> ah, ok
2021-07-06 10:47:30 <clandmeter> which means currently its much harder to find recursive revdeps for testing
2021-07-06 10:47:47 <clandmeter> I did see Ariadne commit some bundled changes
2021-07-06 10:47:52 <ikke> clandmeter: ok, I see that at some point, dl-master just does not receive the packets anymore
2021-07-06 10:47:52 <clandmeter> i guess that was a grep fix
2021-07-06 10:48:22 <clandmeter> ikke: should we try to setup gbr?
2021-07-06 10:48:28 <clandmeter> see what happens if we use that
2021-07-06 10:48:49 <clandmeter> the old uk.a.i
2021-07-06 10:48:52 <clandmeter> the old uk.a.o
2021-07-06 10:51:19 <clandmeter> its weird as the old boxes were in brazil iirc, and the new ones are in europe.
2021-07-06 10:51:28 <ikke> correct
2021-07-06 11:07:39 <clandmeter> ikke: ?
2021-07-06 11:08:33 <clandmeter> ikke: do we want to test sycning to another host?
2021-07-06 11:08:40 <clandmeter> just to rule out issues with cz.a.o
2021-07-06 11:08:58 <ikke> yeah, we can test that
2021-07-06 12:42:20 <ncopa> where is the proper place to report apk install returning BAD signature (in this case its riscv46)?
2021-07-06 12:42:42 <ncopa> i think its the rsync problem we have thats causing it, but i dont know
2021-07-06 12:43:32 <ncopa> is https://gitlab.alpinelinux.org/alpine/infra/infra the proper place?
2021-07-06 12:48:49 <clandmeter> ncopa: which pkg
2021-07-06 12:57:24 <clandmeter> some pkg i had to rebuild locally
2021-07-06 12:57:24 <clandmeter> i think i know what the issue is
2021-07-06 12:57:24 <clandmeter> let me do a check on all pkgs
2021-07-06 13:15:35 <clandmeter> hmm this is harder than i expected
2021-07-06 13:15:35 <clandmeter> how do you verify a pkg against the index but dont want to install it
2021-07-06 13:37:54 <ikke> Fetch? 
2021-07-06 13:46:34 <clandmeter> does fetch verify the index?
2021-07-06 13:52:18 <ncopa> question was from here: https://github.com/tonistiigi/xx/pull/21#issuecomment-870145130
2021-07-06 13:58:09 <ikke> infra/infra is a catch-all for infra issues 
2021-07-06 14:58:09 <clandmeter> ncopa: i already fixed that
2021-07-06 14:58:15 <clandmeter> 2 days ago iirc
2021-07-06 15:11:02 <ikke> ncopa: btw, we found out a little bit more about the rsync issue. Apparently it's a known problem when the .~tmp~ files are left behind, others have reported similar rsync issues
2021-07-06 15:18:57 <ncopa> ok. good
2021-07-06 15:22:25 <ikke> But network issues cause these files to be left behind
2021-07-07 11:13:57 <ikke> ugh
2021-07-07 20:19:42 <Slence> If one wants to be a mirror for alpine, what does one need to do?
2021-07-07 20:19:48 <Slence> We currently run a debian mirror at work.
2021-07-07 20:20:02 <Slence> (AS42708)
2021-07-07 20:22:46 <ikke> send an e-mail to alpine-mirrors@alpinelinux.org with details about the mirror
2021-07-07 20:22:59 <Slence> ok, will do.
2021-07-07 20:23:19 <Slence> thanks!
2021-07-08 18:04:34 <ikke> clandmeter: We still have the 2 block storage volumes in equinix for the old arm gitlab runners, I think I can remove them, right?
2021-07-08 18:12:57 <ikke> clandmeter: ooh, seems like 13.12.8 fixed the forking issue
2021-07-08 18:13:07 <ikke> running on gitlab-test now
2021-07-08 18:22:55 <clandmeter> Huh?
2021-07-08 18:23:29 <ikke> clandmeter: can you try to fork a project on gitlab-test?
2021-07-08 18:23:37 <clandmeter> But no reply on the issue?
2021-07-08 18:23:42 <ikke> No
2021-07-08 18:23:53 <clandmeter> I'm not behind pc
2021-07-08 18:23:56 <ikke> ok
2021-07-08 18:24:05 <clandmeter> I can later
2021-07-08 18:24:36 <ikke> ooh
2021-07-08 18:24:38 <ikke> n/m :(
2021-07-08 18:25:27 <ikke> I already had that project forked
2021-07-08 18:29:40 <ikke> Removed the restrictions on public repos on -test, and that does not fix it either
2021-07-08 19:58:59 <ikke> clandmeter: 2021-07-08T19:58:45.611Z: Unable to save project. Error: could not find any valid magic files!
2021-07-08 20:00:11 <clandmeter> ?
2021-07-08 20:00:52 <ikke> That appears in application.log when I try to fork a project
2021-07-08 20:01:38 <ikke> Seems to have to do with libmagic
2021-07-08 20:08:17 <ikke> clandmeter: can reproduce it with the ruby-magic gem
2021-07-08 20:10:45 <ikke> but the `file` command is working without issue
2021-07-08 20:11:35 <clandmeter> Aha
2021-07-08 20:12:15 <clandmeter> But I checked that log, or is it another one?
2021-07-08 20:12:49 <ikke> I don't know what you checked :)
2021-07-08 20:12:58 <ikke> It just stood out now
2021-07-08 20:13:14 <ikke> and i retried and the message appeared again
2021-07-08 20:13:42 <ikke> https://tpaste.us/XK6e 
2021-07-08 20:15:35 <clandmeter> Got some visitors, can check after
2021-07-08 20:22:00 <ikke> https://tpaste.us/Lm6W 
2021-07-08 20:22:46 <ikke> It tries to look for those files in a non-existing location
2021-07-08 20:37:58 <ikke> Magic.do_not_auto_load = true
2021-07-08 20:38:14 <ikke> m = Magic.new
2021-07-08 20:38:24 <ikke> m.paths
2021-07-08 20:38:32 <ikke> ["/usr/local/bundle/gems/ruby-magic-0.4.0/ports/x86_64-pc-linux-musl/libmagic/5.39/share/misc/magic"]
2021-07-09 07:58:36 <clandmeter> ikke: forking works 
2021-07-09 07:58:46 <clandmeter> :)
2021-07-09 07:59:01 <ikke> ‼️
2021-07-09 07:59:20 <ikke> What was it? 
2021-07-09 07:59:45 <clandmeter> i dont tell
2021-07-09 07:59:56 <ikke> 😥
2021-07-09 08:00:07 <clandmeter> too much cleanup :)
2021-07-09 08:00:29 <ikke> Heh
2021-07-09 08:01:08 <clandmeter> im pulling on stable
2021-07-09 08:01:18 <clandmeter> on production i mean
2021-07-09 08:02:43 <clandmeter> ikke: i guess you dont announce patch releases via gitlab?
2021-07-09 08:03:18 <ikke> I did yesterday for the security upgrade 
2021-07-09 08:03:33 <clandmeter> ah its already on the latest?
2021-07-09 08:03:37 <clandmeter> so this only has the fix?
2021-07-09 08:04:03 <ikke> Yes 
2021-07-09 08:04:12 <clandmeter> ok restarting...
2021-07-09 08:27:02 <ikke> But it was related to ruby-magic? 
2021-07-09 08:28:40 <clandmeter> it  was related that it rm -rf gems/*/ports directories
2021-07-09 08:28:52 <clandmeter> so ruby-magic was affected
2021-07-09 08:29:04 <ikke> Right 
2021-07-09 08:29:15 <clandmeter> im not sure why ports was added
2021-07-09 08:29:25 <clandmeter> its a bit undocumented what we delete
2021-07-09 08:29:34 <clandmeter> some are obvious, some are not.
2021-07-09 08:30:14 <clandmeter> not sure if there is  some spec about gem filesystem structure
2021-07-09 08:30:51 <clandmeter> danieli: maybe you have more experience with this?
2021-07-09 08:35:35 <clandmeter> ikke: i closed the issue on gitlab.com
2021-07-09 08:35:53 <ikke> Ack
2021-07-09 08:36:26 <clandmeter> note to ourselves, utilize the logs more carefully next time :D
2021-07-09 08:36:33 <clandmeter> ikke: and  thx for finding it out
2021-07-09 08:37:17 <clandmeter> excuse from my side, damn this thing has a lot of log files...
2021-07-09 08:37:27 <ikke> Nod 
2021-07-09 08:40:39 <ikke> clandmeter: I also started on 14.0
2021-07-09 08:40:50 <clandmeter> :)
2021-07-09 08:40:58 <clandmeter> escape route
2021-07-09 09:46:34 <ncopa> congrats with fixing the gitlab fork issue
2021-07-09 09:58:00 <clandmeter> algitbot: kick master
2021-07-09 09:58:26 <clandmeter> nmeum: ^ :)
2021-07-09 10:27:30 <ikke> glad it's solved
2021-07-09 12:23:11 <danieli> clandmeter: about what specifically, ruby gems?
2021-07-09 13:35:55 <ikke> Euhm.. 
2021-07-09 13:41:32 <nmeum> clandmeter: ty, good to know :)
2021-07-09 19:38:04 <clandmeter> ikke: how is the riscv64 runner doing?
2021-07-09 19:38:27 <ikke> https://build.alpinelinux.org/buildlogs/build-edge-riscv64/community/gitlab-runner/gitlab-runner-14.0.1-r0.log :P
2021-07-09 19:38:48 <clandmeter> heh
2021-07-09 19:38:51 <clandmeter> im pushing a fix
2021-07-09 19:38:57 <ikke> ah, I was looking at it as wel
2021-07-09 19:39:10 <clandmeter> im building it now
2021-07-09 19:39:27 <ikke> I'm just wondering why go mod edit -replace + go mod download does not work
2021-07-09 19:39:46 <clandmeter> wfm
2021-07-09 19:39:47 <ikke> https://tpaste.us/aZMm vs https://tpaste.us/qPoO
2021-07-09 19:40:58 <clandmeter> https://tpaste.us/NM7l
2021-07-09 19:41:32 <ikke> https://tpaste.us/ypke 
2021-07-09 19:42:08 <clandmeter> https://tpaste.us/YEvQ
2021-07-09 19:42:56 <ikke> huh
2021-07-09 19:45:27 <ikke> your patch fails to build for me :/
2021-07-09 19:45:35 <clandmeter> rm -rf src
2021-07-09 19:45:45 <ikke> I run clean each time
2021-07-09 19:45:58 <clandmeter> hmm
2021-07-09 19:46:02 <clandmeter> now it also fails for me
2021-07-09 19:46:25 <ikke> If you look the 2 diffs, after go mod download, there is one line less added
2021-07-09 19:46:45 <ikke> github.com/creack/pty v1.1.13 h1:rTPnd/xocYRjutMfqide2zle1u96upp1gm6eUHKi7us=
2021-07-09 19:46:48 <ikke> this line is missing
2021-07-09 19:47:31 <ikke> If I change the patch from v1.1.1 to v1.1.5, the patch works
2021-07-09 19:48:42 <clandmeter> it builds if you keep the patch :)
2021-07-09 19:48:48 <clandmeter> the old one
2021-07-09 19:49:19 <ikke> right, I guess because that line is still present, then
2021-07-09 19:57:17 <ikke> clandmeter: go mod tidy afterwards seems to fix it
2021-07-09 19:58:05 <ikke> though, it does mean the packages are already downloaded during prepare()
2021-07-09 20:01:43 <clandmeter> sounds ok
2021-07-09 20:01:51 <ikke> https://tpaste.us/5nwj
2021-07-09 20:01:58 <ikke> this works
2021-07-09 20:02:51 <clandmeter> i think it also works without the download?
2021-07-09 20:02:57 <ikke> probably
2021-07-09 20:03:02 <clandmeter> it does for me :)
2021-07-09 20:03:08 <ikke> ko
2021-07-09 20:03:10 <ikke> ok
2021-07-09 20:05:07 <clandmeter> lets just push it and move on :)
2021-07-09 20:05:24 <ikke> ahuh
2021-07-09 20:05:27 <ikke> Who will push it?
2021-07-09 20:05:36 <clandmeter> you ofc
2021-07-09 20:05:41 <clandmeter> you fix it
2021-07-09 20:05:43 <clandmeter> :)
2021-07-09 20:10:06 <ikke> here it goes
2021-07-09 20:14:31 <ikke> regarding the runner, what do we do with testing?
2021-07-09 20:14:51 <ikke> I suppose currently all packages with dependencies in testing would fail
2021-07-09 20:35:44 <ikke> clandmeter: This is a good reference: https://golang.org/ref/mod
2021-07-09 20:43:50 <ikke> clandmeter: any tips on dealing with the gitlab upgrade? Fixing things and rebuilding the image from scratch each time is going to take a long tim
2021-07-09 20:43:52 <ikke> time
2021-07-09 20:58:40 <clandmeter> It's different then other upgrades?
2021-07-09 21:00:07 <ikke> No
2021-07-09 21:00:12 <ikke> though, more patches to rebase
2021-07-09 21:03:54 <clandmeter> You mean v14?
2021-07-09 21:26:18 <ikke> yeah
2021-07-09 21:26:28 <ikke> I don't know how many
2021-07-09 21:27:03 <ikke> But I gather you don't have any specific approach
2021-07-09 22:21:19 <clandmeter> i did not look at it yet, but if you want i can take care of it.
2021-07-09 22:21:39 <clandmeter> its not that bad to build for me here and ill do it in steps.
2021-07-10 11:27:38 <clandmeter> ikke: please comment on what to do with v14. if you commit to it, ill not reserve the time for it, but ofc i can help.
2021-07-10 11:28:19 <ikke> I can continue, was just wondering if you had some tips
2021-07-10 11:29:14 <clandmeter> i would hack the dockerfile
2021-07-10 11:31:06 <clandmeter> have you checked how many patches currently fail? 
2021-07-10 11:31:22 <clandmeter> and did you look at the changes in the install from source documentation
2021-07-10 11:31:50 <ikke> The only way to find out it is to try to apply each patch
2021-07-10 11:44:42 <clandmeter> ikke: if you modify dockerfile and RUN each setup step individual, it should cache layers and it should be much faster to debug.
2021-07-10 11:45:23 <ikke> yes, we would need to splitup setup.sh
2021-07-10 11:46:01 <clandmeter> right
2021-07-10 11:47:06 <clandmeter> as its already functions you could add some logic to it like what we do with abuild
2021-07-10 11:47:22 <ikke> right
2021-07-10 11:48:16 <clandmeter> i also keep getting ideas about splitting all of it into seperate container. but in the end thinking of it, it will probably provide more headaches.
2021-07-10 11:49:50 <clandmeter> we would have to superglue the spaghetti monster together again. 
2021-07-10 11:51:10 <clandmeter> ikke: oh the setup is not functions
2021-07-10 11:51:16 <ikke> no, it's not
2021-07-10 11:51:25 <clandmeter> entrypoint is :)
2021-07-10 11:51:44 <clandmeter> but it should be simple to wrap them up
2021-07-10 11:52:09 <ikke> I do like the idea of splitting gitlab up :)
2021-07-10 11:54:30 <ikke> We could do it step by step
2021-07-10 11:56:33 <ikke> gitaly seems doable
2021-07-10 11:57:04 <ikke> I guess
2021-07-10 11:58:38 <ikke> Many things that you have patched seem to have been upstreamed
2021-07-10 12:01:53 <ikke> oh, I guess they already have been applied
2021-07-10 12:29:41 <ikke> One challenge splitting with splitting it up is how to get the correct versions of all components
2021-07-10 15:13:46 <clandmeter> Is was thinking of using some makefile or similar
2021-07-10 16:29:36 <ikke> I don't think we want to manually keep track of all the versions
2021-07-10 16:29:58 <ikke> I guess we could just fetch the files from gitlab to get the correct version
2021-07-10 17:28:52 <clandmeter> Ofc
2021-07-10 18:59:52 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/jobs/433762
2021-07-10 19:04:13 <ikke> clandmeter: [ACTION REQUIRED]: Block Storage Depreciation [FINAL NOTICE]: Block Storage Depreciation
2021-07-10 19:14:52 <clandmeter> ikke: where did you get that notice from?
2021-07-10 19:15:09 <ikke> e-mail
2021-07-10 19:15:29 <ikke> You received it as well
2021-07-10 19:15:39 <clandmeter> strange i dont see it
2021-07-10 19:15:42 <clandmeter> when?
2021-07-10 19:15:52 <ikke> 3 days ago
2021-07-10 19:15:55 <ikke> Juli 7th
2021-07-10 19:16:09 <ikke> It's the old arm ci volumes, they still exist
2021-07-10 19:16:24 <ikke> I think we can just remove them, right?
2021-07-10 19:17:55 <clandmeter> ah its tagged as spam
2021-07-10 19:23:31 <clandmeter> yes lets kill them
2021-07-10 19:23:35 <clandmeter> if you dont use them anymore
2021-07-10 19:24:08 <clandmeter> weird that im missing equinix emails. seems like  DMARC fails.
2021-07-10 19:25:32 <clandmeter> so you are now building the gitlab contain from gitlab
2021-07-10 19:25:49 <ikke> ?
2021-07-10 19:26:06 <ikke> curl -s https://gitlab.com/api/v4/projects/278964/repository/files/GITALY_SERVER_VERSION\?ref\=v14.0.2-ee | jq -r .content | base64 -d
2021-07-10 19:26:09 <clandmeter> ah wait you alraedy did taht
2021-07-10 19:26:16 <clandmeter> just not pushing it
2021-07-10 19:26:32 <ikke> I'm not following :)
2021-07-10 19:26:49 <clandmeter> you pasted a job here
2021-07-10 19:26:54 <ikke> right
2021-07-10 19:27:13 <clandmeter> so what should i look at?
2021-07-10 19:27:18 <ikke> It's 14.0.2
2021-07-10 19:27:31 <clandmeter> nice
2021-07-10 19:27:47 <clandmeter> did you need to fix a lot of patches?
2021-07-10 19:27:51 <ikke> no
2021-07-10 19:27:55 <ikke> just one in the end
2021-07-10 19:28:06 <clandmeter> i guess it would not be taht much
2021-07-10 19:28:21 <clandmeter> i think the __va or whatever it was got upstreamed
2021-07-10 19:28:36 <clandmeter> but im not sure we already pull that in
2021-07-10 19:28:55 <ikke> bundle config set --global build.google-protobuf --with-cflags=-D__va_copy=va_copy
2021-07-10 19:28:57 <ikke> that I guess
2021-07-10 19:29:09 <clandmeter> yes, but there is also a patch
2021-07-10 19:29:51 <clandmeter> and the link in it, mentions something about it being fixed upstream
2021-07-10 19:30:12 <clandmeter> not sure what the relationship with that patch and this config set is.
2021-07-10 19:30:22 <ikke> ruby-fix-cflags.patch
2021-07-10 19:31:35 <clandmeter> https://github.com/protocolbuffers/protobuf/commit/9abf6e2ab0d8847059edfc1720e6697c7282f527
2021-07-10 19:31:54 <clandmeter> https://github.com/protocolbuffers/protobuf/commit/9abf6e2ab0d8847059edfc1720e6697c7282f527#diff-814c9eb0903e070d523bb5ca2fc2caba7cade74087d6b13d9a883b6f46813924
2021-07-10 19:34:17 <clandmeter> btw, why 14.0.2 when 0.5 is out?
2021-07-10 19:34:42 <ikke> hmmm, good question :)
2021-07-10 19:34:56 <ikke> I guess I did not verify that latest version
2021-07-10 19:35:21 <clandmeter> https://about.gitlab.com/releases/2021/07/08/gitlab-14-0-5-released/
2021-07-11 09:55:10 <clandmeter> ikke: did you test your build?
2021-07-11 09:58:42 <ikke> not yet
2021-07-11 17:28:49 <ikke> I have a multi-stage docker file for gitaly
2021-07-11 17:28:55 <ikke> Still need to finish it
2021-07-11 17:43:27 <ikke> Hmm, need gitlab-shell as well
2021-07-12 10:44:16 <ikke> clandmeter: afaik, we do not do that yet, but I noticed we can save a lot of space if we strip ruby gems
2021-07-12 10:44:23 <ikke> grpc is a big one
2021-07-12 10:44:39 <ikke> for gitaly, it went from 500M to 200M
2021-07-12 11:00:08 <ikke> I'm quite anoyed that many ruby projects (and also python projects) require you to compile each dependency all the time
2021-07-12 11:00:17 <ikke> well, go as well, but at least that's fast
2021-07-12 11:01:03 <danieli> they usually don't, given that prebuilt binaries are available
2021-07-12 11:01:34 <ikke> If you are on the blessed platform
2021-07-12 11:01:39 <danieli> yes, bingo
2021-07-12 11:01:46 <danieli> if you aren't, you need to compile stuff allllll the damn time
2021-07-12 11:01:58 <ikke> We do have most libraries pre-compiled
2021-07-12 11:02:11 <ikke> but then the require version >1.2.3 <1.2.4
2021-07-12 11:02:22 <ikke> (exegerated)
2021-07-12 11:02:31 <danieli> yeah.. that's why they're often stored upstream in e.g. pypi
2021-07-12 11:03:01 <ikke> For blessed platforms
2021-07-12 11:44:26 <clandmeter> did I miss messages?
2021-07-12 11:44:26 <clandmeter> ikke: do not do what yet?
2021-07-12 11:44:48 <ikke> Sorry, strip binaries 
2021-07-12 11:45:25 <clandmeter> i think you can apply configure options to use external libraries
2021-07-12 11:45:54 <clandmeter> for some
2021-07-12 11:46:43 <clandmeter> but i wonder how much we need to tweak to not shoot ourselves in the foot all the time.
2021-07-12 11:46:56 <ikke> Nod
2021-07-12 11:47:28 <clandmeter> stripping should be ok
2021-07-12 11:47:45 <ikke> I would assume so 
2021-07-12 11:58:14 <clandmeter> danieli: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/blob/13.12-stable/overlay/usr/local/bin/setup.sh#L233
2021-07-12 11:58:27 <clandmeter> this is what i was referring to 
2021-07-12 11:58:52 <danieli> clandmeter: uhm, what is the context/
2021-07-12 11:58:53 <danieli> ?
2021-07-12 11:59:05 <clandmeter> my previous question
2021-07-12 11:59:46 <clandmeter> if there is some logic in rubys directory structure
2021-07-12 11:59:48 <clandmeter> some spec
2021-07-12 16:49:26 <ikke> clandmeter: Am I missing something, or can gemdeps.sh be mostly replaced with: `scanelf -BRF '%n#p' -E ET_DYN,ET_EXEC . | tr , '\n' | sort -u | awk '!/libruby/ { print "so:" $1 }'`
2021-07-12 16:55:39 <ikke> 7s vs 0.01s
2021-07-12 18:07:45 <ikke> https://tpaste.us/7Vvo 
2021-07-12 19:11:29 <clandmeter> ikke: dunno
2021-07-12 19:11:38 <clandmeter> if you say so :)
2021-07-12 19:11:47 <ikke> Output is the same
2021-07-12 19:12:24 <ikke> fun fact: for stripping binaries, I use scanelf -BARF :)
2021-07-12 19:19:28 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/compare/14.0-stable...split-gitaly?from_project_id=356
2021-07-12 19:21:47 <ikke> It's not complete yet, but this is what I have so far
2021-07-12 19:29:22 <clandmeter> why do you use the api to fetch versions?
2021-07-12 19:30:04 <ikke> Because that's the only thing I need from that repo
2021-07-12 19:30:19 <ikke> outside of the docker containers
2021-07-12 19:30:47 <clandmeter> you cant just curl https://gitlab.com/gitlab-org/gitlab/-/raw/v14.0.5-ee/GITALY_SERVER_VERSION ?
2021-07-12 19:31:00 <ikke> Yeah, somehow did not think of that :D
2021-07-12 19:31:21 <clandmeter> like not using -R with scanelf? :p
2021-07-12 19:31:26 <ikke> :D
2021-07-12 19:31:57 <clandmeter> tbf, i think i copied that from some example which was very slow and i added some logic to make it faster.
2021-07-12 19:32:08 <clandmeter> and i didnt see the -R switch i guess :)
2021-07-12 19:32:15 <clandmeter> and -E
2021-07-12 19:32:32 <clandmeter> but its been a long time 
2021-07-12 19:32:40 <ikke> it's forgiven :)
2021-07-12 19:32:58 <clandmeter> thx
2021-07-12 19:37:21 <ikke> I now use that also to trace the git deps
2021-07-12 19:37:53 <clandmeter> yes i saw
2021-07-12 19:38:28 <ikke> But I think I need to add git-shell to there as well (maybe copy it from another image)
2021-07-12 19:40:40 <clandmeter> you might want to move all dockerfiles into its own subdirs
2021-07-12 19:43:31 <ikke> I did for gitaly
2021-07-12 19:45:04 <clandmeter> its kind of best practice as docker build will add all deeper located files to its recipe or however you call it.
2021-07-12 19:46:14 <ikke> build context
2021-07-12 20:18:35 <ikke> clandmeter: better so? https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/compare/14.0-stable...split-gitaly?from_project_id=356#014d9d2e9ed07256aa29a172be69002b25fc644d_0_12
2021-07-12 20:23:54 <clandmeter> much better! :)
2021-07-12 20:24:10 <clandmeter> why not add sodeps to setup?
2021-07-12 20:29:10 <ikke> no particular reason
2021-07-12 21:07:32 <ikke> disadvantage of using a single setup file is that all stages have to be rebuilt if you change it
2021-07-12 21:28:11 <clandmeter> ikke: this has to do with sodeps?
2021-07-12 21:29:04 <ikke> no
2021-07-12 21:29:22 <ikke> this has to do with having to rebuild everything just because you just a small step at the end
2021-07-12 21:29:56 <ikke> even though things are separated in stages that are independent (except for the last and the first)
2021-07-12 21:30:47 <clandmeter> ok but in general you dont modify setup
2021-07-12 21:31:16 <clandmeter> i added a sep file so it could exec it self
2021-07-12 21:31:31 <clandmeter> which is not needed anymore iiuc
2021-07-12 21:31:39 <ikke> correct
2021-07-12 21:31:55 <ikke> that's what I gathered
2021-07-12 21:32:38 <clandmeter> but you can also split the whole file into sep files if you move everything into its own container
2021-07-12 21:32:51 <clandmeter> and let it source some shared fucntions
2021-07-12 21:33:16 <ikke> yes, but I'm now talking about the setup of a single container (gitaly), with multiple stages
2021-07-12 21:36:06 <ikke> I'm wondering how we can share functionality with separate build contexts
2021-07-12 21:36:18 <ikke> symlinks do not work
2021-07-12 21:36:23 <clandmeter> symlinks?
2021-07-12 21:36:31 <ikke> :P
2021-07-12 21:36:47 <clandmeter> hardlinks dont work?
2021-07-12 21:37:01 <ikke> hardlinks get broken
2021-07-12 21:37:54 <clandmeter> you will need to copy it from toplevel makefile
2021-07-12 21:38:05 <clandmeter> or whatever script you will use
2021-07-12 21:46:13 <clandmeter> ikke: are you going to use sockets or tcp?
2021-07-12 21:46:52 <ikke> first attempt will be sockets
2021-07-13 15:12:36 <ikke> https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/configuration/README.md#gitlab-shell2
2021-07-13 15:12:38 <ikke> https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/configuration/README.md#gitlab-shell
2021-07-13 15:12:53 <ikke> So it seems it only needs the code
2021-07-14 07:18:33 <ncopa> i deleted some files in distfiles and cleaned up my container(s) on usa9-dev1
2021-07-14 07:20:58 <ikke> thanks, I've also cleaned up nld9 yesterday
2021-07-14 07:21:12 <ikke> fyi: ~/.cache/yarn tends to grow big
2021-07-14 07:21:22 <ikke> and ~/.cache/go-build as well
2021-07-14 12:48:27 <eletrotupi> heya, is lists.alpinelinux.org down?
2021-07-14 12:48:35 <eletrotupi> it keeps 404 to me :(
2021-07-14 12:48:39 <ikke> not for me
2021-07-14 12:48:42 <ncopa> eletrotupi: works for me
2021-07-14 12:49:07 <eletrotupi> wtf, https://lists.alpinelinux.org?
2021-07-14 12:49:15 <eletrotupi> that's weird
2021-07-14 12:49:29 <ncopa> we have spam problems https://lists.alpinelinux.org/~alpine/aports/%3Cd08475fea13b918021b1bca14e2beb97%4010.57.86.120%3E
2021-07-14 12:49:52 <eletrotupi> can't open that link either
2021-07-14 12:50:21 <eletrotupi> were my IP blackholed?
2021-07-14 12:50:22 <ikke> What IP does it resolve to for you?
2021-07-14 12:50:46 <eletrotupi> 147.75.101.119
2021-07-14 12:51:32 <ikke> that's correct
2021-07-14 12:52:57 <ikke> eletrotupi: would you mind sharing me your IP address in private?
2021-07-14 12:55:54 <eletrotupi> yeah
2021-07-14 16:10:05 <ncopa> i get emails about storage on equinix metal
2021-07-14 16:10:13 <ncopa> Per our previous notifications, Equinix Metal began deprecating our Elastic Block Storage service on June 1st, 2021. I have noticed you still have unattached volumes in the following project(s):
2021-07-15 04:53:59 <ikke> oh boy
2021-07-15 05:15:57 <ikke> host seems down
2021-07-15 06:54:03 <ikke> 😕
2021-07-15 06:54:15 <ikke> so we effectively lost s390x
2021-07-15 06:56:15 <mps> ah, finally good news ;)
2021-07-15 10:01:51 <ikke> mps: too bad ^ :P
2021-07-15 10:02:55 <mps> ikke: you understand that I'm just kidding
2021-07-15 10:03:16 <ikke> yes, so I continue your joke :)
2021-07-15 10:03:26 <mps> I see
2021-07-17 18:08:09 <ikke> clandmeter: I split the build files for each component of gitaly now. Now changing one thing does not cause everything to be rebuilt :)
2021-07-17 18:14:20 <clandmeter> but it will introduce multiple layers?
2021-07-17 18:14:38 <ikke> Not in the final image
2021-07-17 18:14:55 <clandmeter> ok you are using copy from
2021-07-17 18:14:58 <ikke> yes
2021-07-17 18:14:59 <clandmeter> i guess
2021-07-17 18:15:14 <clandmeter> be sure to copy everything :)
2021-07-17 18:15:18 <ikke> ahuh :)
2021-07-17 18:15:33 <ikke> I'm looking at setup.sh and the existing gitlab instance for reference
2021-07-17 18:15:42 <ikke> testing running gitaly now
2021-07-17 18:15:47 <ikke> ran into __va_copy now
2021-07-17 18:18:35 <ikke> ok, gitaly is running now :)
2021-07-17 18:18:48 <ikke> I'm just wondering what we need to do with ssh
2021-07-17 18:19:05 <ikke> ssh <-> gitlab-shell <-> gitaly
2021-07-17 18:30:13 <ikke> clandmeter: https://github.com/protocolbuffers/protobuf/pull/7773
2021-07-17 19:01:39 <ikke> clandmeter: with DOCKER_BUILDKIT, it means it can build several stages in paralel
2021-07-17 21:31:33 <ikke> clandmeter: do you happen to know what triggers .ssh/authorized_keys to be written / generated?
2021-07-17 21:43:56 <clandmeter> Yes
2021-07-17 21:44:27 <clandmeter> It's in entrypoint
2021-07-17 21:47:28 <ikke> rake gitlab:shell:setup
2021-07-18 08:51:27 <ikke> Keep having rsync issues with ppc64le. Sent another e-mail about it
2021-07-19 07:13:34 <clandmeter> ikke: did you see if syncing builders to gbr1 makes any difference?
2021-07-19 10:15:36 <ikke> clandmeter: no, not yet
2021-07-19 10:27:34 <ikke> clandmeter: No issues atm, but it seems that syncing to dl-master.a.o is not having issues either
2021-07-19 10:27:35 <ikke> atm
2021-07-19 12:26:22 <clandmeter> but you are removing --delayed-updated?
2021-07-19 12:27:54 <ikke> yes
2021-07-19 12:28:10 <ikke> That's a separate issue
2021-07-19 12:28:35 <ikke> without --delayed-updates, it does try to update files, but for larger files, it hangs
2021-07-19 12:57:14 <clandmeter> why is that a sep issue?
2021-07-19 12:57:38 <clandmeter> "for large files it hangs" sounds like a network issue
2021-07-19 12:57:52 <ikke> Because we still have this issue without --delayed-updates
2021-07-19 12:58:09 <ikke> But that network issue does play into account with --delayed-updates
2021-07-19 14:29:04 <clandmeter> can we reproduce this issue and see if it differs on these two mirrors?
2021-07-19 14:29:51 <ikke> The rsync --delayed-updates issue we can reproduce
2021-07-19 14:30:12 <ikke> https://github.com/WayneD/rsync/issues/192
2021-07-19 14:30:15 <clandmeter> right, but thats not something we can really fix right now.
2021-07-19 14:30:19 <ikke> No
2021-07-19 14:30:28 <ikke> The network issue is intermittent
2021-07-19 14:30:36 <ikke> and so far, I only encountered it on ppc64le
2021-07-19 14:30:48 <clandmeter> i was wondering if we could try rsyncing large files
2021-07-19 14:30:55 <clandmeter> lets say 10GB or something
2021-07-19 14:31:17 <clandmeter> maybe that can reproduce it
2021-07-19 14:32:25 <ikke> One thing that I did notice is that it was usually the -dbg packages that hung
2021-07-19 14:32:37 <ikke> mesa-dbg for example
2021-07-19 14:33:30 <ikke> And it would hang after 1 or 2 seconds
2021-07-19 14:33:44 <ikke> different amounts of data transfered
2021-07-19 14:48:54 <clandmeter> do we have zabbix installed?
2021-07-19 14:49:18 <ikke> on the builders?
2021-07-19 14:50:25 <ikke> Not on the ppc64le hosts
2021-07-19 14:50:41 <clandmeter> and master
2021-07-19 14:50:56 <clandmeter> i wonder if we could graph network quality between them?
2021-07-19 14:51:09 <ikke> No, I asked before and at least at the time, you and ncopa didn't think it was a good idea
2021-07-19 14:51:42 <clandmeter> i guess we can run it in a container?
2021-07-19 14:52:51 <ikke> an agent in a container does not make a lot of sense
2021-07-19 14:53:09 <ikke> at least, if you want to monitor the host
2021-07-19 14:53:16 <ikke> if you want to monitor the container, then, yes
2021-07-19 14:56:19 <ikke> btw, I've noticed that gitlab-shell has 4 identical binaries about ~20M 
2021-07-19 14:56:32 <ikke> so we can save ~60M
2021-07-19 14:57:19 <clandmeter> ok, kill them
2021-07-19 14:57:30 <ikke> I've added rdfind in the build process
2021-07-19 14:57:54 <ikke> which automatically turns them into symlinks (which are preserved across build stages / copies)
2021-07-19 15:01:17 <ikke> So I have a gitaly container now, and working on a gitlab-shell container which will probably run sshd
2021-07-20 10:47:28 <ncopa> docs was updated in git this weekend (OFTC, link to git repository) but the page is not yet generated. what needs to be done to regenerate the docs? https://docs.alpinelinux.org/user-handbook/0.1a/index.html
2021-07-20 10:48:54 <ikke> It should respond to mqtt messages
2021-07-20 10:49:52 <ikke> I guess we need to update the topic it listens to
2021-07-20 10:55:03 <ikke> updated
2021-07-20 10:55:12 <ikke> still need to update the topic
2021-07-20 16:15:26 <nmeum> regarding foss video conferencing software for the tsc: I have used both jitsi and bigbluebutton at work, they both work fine bigbluebutton maybe better suited for our purpose though
2021-07-20 16:15:43 <nmeum> would probably be nice to have a video conferencing tool running on alpine infra, would also allow other teams to use it for coordination
2021-07-20 16:16:21 <ikke> Would be nice if at least the host OS could be alpine, but not sure if feasible
2021-07-20 16:17:33 <nmeum> we probably need to experiment with jitsi/bigbluebutton then and see which one is easier to host I guess?
2021-07-20 16:18:27 <nmeum> might be a bit ambitious to do that in two weeks, would it be possible to use Ariadne's bbb instance again as a fallback for the tsc meeting in two weeks or will it be gone by then?
2021-07-20 16:18:36 <Ariadne> yes
2021-07-20 16:18:51 <Ariadne> BBB server is at equinix, so i have more time for it
2021-07-20 16:19:02 <Ariadne> the larger issue is my s390x machine :(
2021-07-20 16:19:06 <Ariadne> need to find a new home for it
2021-07-20 16:20:26 <ikke> The basic install of jitsi is not too bad, got it running in one morning
2021-07-20 16:20:29 <ikke> in docker
2021-07-20 16:20:35 <Ariadne> same with BBB
2021-07-20 16:20:38 <Ariadne> you just install ubuntu
2021-07-20 16:20:42 <Ariadne> and run an install script
2021-07-20 16:20:48 <Ariadne> and then it installs docker and does a bunch of stuff
2021-07-20 16:20:48 <Ariadne> lol
2021-07-20 16:20:55 <nmeum> hehe
2021-07-20 16:21:01 <ikke> how much does it tinker on the host OS?
2021-07-20 16:21:02 <nmeum> I thought the goal was hosting it on alpine ;)
2021-07-20 16:21:13 <nmeum> but we could also just use ubuntu for now and migrate to alpine later
2021-07-20 16:21:22 <Ariadne> i dont think it does much tinkering
2021-07-20 16:21:47 <nmeum> I personally wouldn't mind hosting the video conferencing system on ubuntu tbh
2021-07-20 16:27:40 <Ariadne> i think there is still some salt about the numerous times canonical has tried to come after us
2021-07-20 16:27:58 <Ariadne> with FUD
2021-07-20 16:28:21 <Ariadne> i try to avoid ubuntu for that reason, anyway
2021-07-20 16:30:28 <nmeum> fair
2021-07-20 16:30:57 <nmeum> ikke: if you have prior experience with jitsi, would you be able/intersted in trying to get it running on alpine infra?
2021-07-20 16:31:11 <Ariadne> i think either solution is fine
2021-07-20 19:59:49 <clandmeter> ping
2021-07-20 19:59:54 <clandmeter> weird
2021-07-20 20:00:13 <clandmeter> i was under the impression jitsi didnt need a server side?
2021-07-20 20:00:55 <ikke> if you use the cloudservice, but you can run your own instance
2021-07-21 04:47:14 <clandmeter> ikke: ping
2021-07-21 04:48:25 <ikke> pong
2021-07-21 04:49:42 <clandmeter> we have gitlab issues
2021-07-21 04:51:07 <ikke> it appears so. Anything more specific?
2021-07-21 04:51:16 <clandmeter> disk
2021-07-21 04:51:22 <clandmeter> looks like pg is not happy
2021-07-21 04:51:46 <clandmeter>  /dev/sda        315G  298G  715M 100% /
2021-07-21 04:52:21 <clandmeter> shared is 154G
2021-07-21 04:52:40 <clandmeter> artifacts 
2021-07-21 05:07:17 <ikke> there is 60GB  worth of docker images
2021-07-21 05:07:25 <ikke> `system docker df`
2021-07-21 05:18:47 <ikke> are you doing something atm?
2021-07-21 05:25:51 <ikke> 38G available now
2021-07-21 05:54:10 <ikke> clandmeter: I've fixed the agent config on deu2 as well, so now we should get proper disk usage alerts
2021-07-21 05:55:00 <clandmeter> no
2021-07-21 05:55:05 <clandmeter> im not doing anything
2021-07-21 05:55:09 <clandmeter> showering :)
2021-07-21 05:55:13 <ikke> :)
2021-07-21 05:56:49 <clandmeter> can we do something to keep artifacts down a bit?
2021-07-21 05:58:41 <ikke> I though artifacts should only be kept for 24h
2021-07-21 05:59:44 <ikke> /dev/sda        315G  243G   56G  82% /
2021-07-21 05:59:53 <ikke> current istuation
2021-07-21 06:00:14 <clandmeter> its 153Gb
2021-07-21 06:00:29 <clandmeter> that does not sound like 24h :)
2021-07-21 06:01:18 <clandmeter> there are 2 dirs very large
2021-07-21 06:03:22 <ikke> There are circumstance where GL will keep artifacts
2021-07-21 06:03:35 <ikke> The latest artifacts for a branch
2021-07-21 06:13:41 <clandmeter> ikke: in any case its too much
2021-07-21 06:13:49 <clandmeter> we will bump into issues if that grows
2021-07-21 06:13:53 <clandmeter> https://tpaste.us/Qrnz
2021-07-21 06:14:03 <ikke> yes, I agree, need to look into it
2021-07-21 06:14:19 <ikke> are thise also buildlogs?
2021-07-21 06:14:23 <ikke> these*
2021-07-21 06:15:26 <clandmeter> not sure, just some simple du on some larger dirs
2021-07-21 06:15:47 <clandmeter> https://tpaste.us/KMyn
2021-07-21 06:15:52 <clandmeter> thats the root of artifacts
2021-07-21 06:16:05 <clandmeter> those 2 dirs look suspicious 
2021-07-21 06:16:16 <ikke> nod
2021-07-21 06:16:24 <clandmeter> not sure what the logic is behind the naming
2021-07-21 06:17:16 <ikke> does it follow the hashed repo structure?
2021-07-21 06:17:59 <ikke> yes
2021-07-21 06:18:13 <ikke> shared/artifacts/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/
2021-07-21 06:18:18 <ikke> @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git
2021-07-21 06:18:21 <ikke> that's aports
2021-07-21 07:01:13 <clandmeter> ikke: did you check docker compose logs?
2021-07-21 07:02:37 <ikke> for postgres?
2021-07-21 07:30:21 <ncopa> chromium is building for two architectures, aarch64 and x86_64. not sure if that is related
2021-07-21 08:51:00 <clandmeter> ikke: yes
2021-07-21 08:51:11 <clandmeter> or are those errors still related to that openstanding issues?
2021-07-21 08:53:09 <ikke> I'm not sure
2021-07-21 09:23:30 <clandmeter> does spf softfail result in dmarc fail?
2021-07-21 09:25:00 <clandmeter> looks like it :{
2021-07-21 10:42:28 <ikke> clandmeter: It's strange, shared/artifacts/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/2021_06_16/416494/854755/MR23042_x86_64.zip points to an MR that was created one week ago
2021-07-21 10:43:19 <ikke> But the directory mentions 2021_06_16
2021-07-21 10:49:48 <clandmeter> ikke: which mr do you think this is?
2021-07-21 10:51:17 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/23042
2021-07-21 10:51:34 <clandmeter> but the artifacts are mariadb
2021-07-21 10:51:54 <ikke> right
2021-07-21 10:54:18 <ikke> so that's strange
2021-07-21 10:55:00 <clandmeter> so the mr id is not from aports
2021-07-21 10:55:08 <clandmeter> or not from official aports
2021-07-21 10:56:53 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/.gitlab-ci.yml#L46
2021-07-21 11:01:08 <ikke> I, [2021-07-21T11:00:59.174851 #28558]  INFO -- : [DRY RUN] Processed 0 job artifact(s) to find and cleaned 0 orphan(s).
2021-07-21 11:01:22 <clandmeter> yes i know that part, but something does not compute
2021-07-21 11:02:32 <ikke> I think we both agree on that
2021-07-21 11:02:48 <clandmeter> thats nice for a change ;-)
2021-07-21 11:03:03 <ikke> 😆
2021-07-21 11:05:21 <clandmeter> i found 2 related MRs'
2021-07-21 11:05:41 <clandmeter> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/21244
2021-07-21 11:05:49 <clandmeter> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/21243
2021-07-21 11:05:58 <clandmeter> these match the artifacts
2021-07-21 11:14:52 <clandmeter> yup
2021-07-21 11:15:00 <clandmeter> the 21244 resolves to that file
2021-07-21 11:15:31 <ikke> "These artifacts are the latest. They will not be deleted (even if expired) until newer artifacts are available."
2021-07-21 11:16:19 <ikke> This used to be different, but at some point, gitlab changed that
2021-07-21 11:16:39 <clandmeter> where do you read that  text?
2021-07-21 11:17:21 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/416494
2021-07-21 11:19:51 <clandmeter> ok
2021-07-21 11:23:37 <clandmeter> ikke: none of the ID's match
2021-07-21 11:23:40 <clandmeter> its seems
2021-07-21 11:25:31 <clandmeter> ah
2021-07-21 11:25:49 <clandmeter> you need CI_MERGE_REQUEST_IID
2021-07-21 11:26:10 <clandmeter> CI_MERGE_REQUEST_ID is gitlab global ID
2021-07-21 11:26:21 <clandmeter> IID is per project
2021-07-21 11:26:37 <ikke> Ah, that explains
2021-07-21 11:27:08 <clandmeter> olk one issue solved :)
2021-07-21 11:28:16 <clandmeter> https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs
2021-07-21 11:28:19 <ikke> And those ids in the path are also global instead of project specific?
2021-07-21 11:28:20 <clandmeter> i guess we need to change this
2021-07-21 11:28:46 <clandmeter> im not sure what they are
2021-07-21 11:28:51 <ikke> ok
2021-07-21 11:29:02 <ikke> I guess we cannot toggle that per project
2021-07-21 11:29:27 <ikke> ah https://gitlab.com/gitlab-org/gitlab/-/issues/241026
2021-07-21 11:30:01 <ikke> https://i.imgur.com/iKt7QfH.png
2021-07-21 11:30:04 <ikke> So we need to disable that
2021-07-21 11:30:15 <ikke> But not sure if that needs to be done for all forks as well
2021-07-21 11:30:31 <clandmeter> i think global id's are no problem per se 
2021-07-21 11:30:37 <ikke> No
2021-07-21 11:30:50 <clandmeter> maybe even nicer
2021-07-21 11:30:53 <ikke> But it makes it easier to find the corresponding MR
2021-07-21 11:30:54 <clandmeter> in some cases
2021-07-21 11:31:04 <ikke> And can cause confusion
2021-07-21 11:31:13 <clandmeter> but im not sure how to search with global ids
2021-07-21 11:31:15 <ikke> It's also the name of the file that you get as a download
2021-07-21 11:31:26 <ikke> probably though gitlab shell
2021-07-21 11:35:27 <clandmeter> did you read https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines
2021-07-21 11:35:34 <clandmeter> its a global setting
2021-07-21 11:35:55 <ikke> It is, but they later also added a per-project setting
2021-07-21 11:36:04 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/settings/ci_cd
2021-07-21 12:52:08 <ncopa> i bisected rsync and found the commit that introduces the issue
2021-07-21 12:55:32 <ikke> nice!
2021-07-21 13:13:03 <ncopa> https://github.com/WayneD/rsync/issues/192#issuecomment-884163578
2021-07-21 13:13:08 <ncopa> im writing a test for the testsuite now
2021-07-21 13:13:36 <ikke> What version was that introduced?
2021-07-21 13:13:50 <ikke> ah, 3.2.3
2021-07-21 13:14:11 <ikke> no, 3.2.0 was the first
2021-07-21 14:39:44 <clandmeter> ncopa: nice!
2021-07-21 14:46:59 <clandmeter> so everything done except the fix :)
2021-07-21 14:50:47 <danieli> awesome
2021-07-21 15:42:35 <ikke> clandmeter: I disabled 'keep latest artifacts' for aports
2021-07-21 16:07:23 <ikke> ncopa: CI failed, no route to host 😑
2021-07-21 17:25:10 <clandmeter> ikke: 👍
2021-07-21 17:27:42 <ikke> I don't see a decrease in disk usage though (the task should run every 7 minutes)
2021-07-21 18:48:25 <ikke> clandmeter: I have gitlab running now with gitaly / sshd separated
2021-07-21 18:48:27 <ikke> locally
2021-07-21 18:48:36 <ikke> Still need to verify if everything is working
2021-07-21 18:51:30 <clandmeter> Nice
2021-07-21 18:51:39 <clandmeter> If it works
2021-07-21 18:51:57 <ikke> I did have to copy gitlab-shell into both gitlab and gitally
2021-07-21 18:52:08 <ikke> as they both still have dependencies on it
2021-07-21 18:55:29 <ikke> I do see this warning: s6-svwait: fatal: unable to subscribe to events for /run/s6/web: No such file or directory 
2021-07-21 18:55:40 <ikke> but I do not see /etc/s6/web being defined
2021-07-21 18:56:20 <ikke> I see that nginx waits for it
2021-07-21 19:20:41 <ikke> "Unexpected error" "Internal API unreachable" ..
2021-07-21 19:24:26 <ikke> apparently gitaly also needs the workhorse socket
2021-07-21 20:17:22 <ikke> Just the gitlab socket, forget to add the volume
2021-07-21 20:31:07 <ikke> Now I get "not authorized"
2021-07-21 21:25:41 <clandmeter> ikke: there is another socket for auth iirc
2021-07-21 21:26:08 <ikke> I have 2 sockets, one for gitaly, one for gitlab
2021-07-21 21:26:15 <ikke> is there another one?
2021-07-21 21:26:26 <clandmeter> i dont know anymore
2021-07-21 21:26:30 <clandmeter> but i remember something like that
2021-07-21 21:27:44 <ikke> It's too late now, will look tomorrow
2021-07-22 07:43:33 <nmeum> regarding video conferencing: should I create an issue in the infra gitlab group to further discuss this or how do we want to proceed with this? :)
2021-07-22 07:46:13 <clandmeter> sounds good
2021-07-22 08:07:44 <nmeum> https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10726
2021-07-22 09:32:54 <clandmeter> nmeum: TPC?
2021-07-22 09:33:15 <clandmeter> i guess you mean TSC?
2021-07-22 09:33:19 <nmeum> oops yes
2021-07-22 09:33:20 <nmeum> typo
2021-07-22 09:33:28 <clandmeter> double typo :D
2021-07-22 09:33:40 <nmeum> fixed :p
2021-07-22 09:35:21 <clandmeter> i looked briefly at both solutions
2021-07-22 09:35:35 <clandmeter> but it seems not that easy to transform it to alpine
2021-07-22 09:35:39 <nmeum> yeah, i thought so
2021-07-22 09:35:59 <clandmeter> i think we (infra) should spend our time in different places.
2021-07-22 09:36:30 <clandmeter> more closely to the core of our product, but thats my opinion of course
2021-07-22 09:36:53 <nmeum> i personally think it would be fine to just use ubuntu where both bbb and jitsi are probably a lot easier to install
2021-07-22 09:36:58 <clandmeter> if somebody wants to take on this task to build it on top of alpine thats wonderful :)
2021-07-22 09:37:42 <clandmeter> jitsi is mostly java if i understand correctly
2021-07-22 09:38:20 <nmeum> yep
2021-07-22 10:26:55 <danieli> clandmeter: i agree about the maintaining focus on the project itself part
2021-07-22 10:39:57 <ikke> clandmeter: [DRY RUN] Processed 156807 job artifact(s) to find and cleaned 56823 orphan(s).
2021-07-22 10:40:22 <ikke> bb ionice does not accept -c best-effort, so it would never find orphaned files
2021-07-22 10:40:52 <ikke> on gitlab-test, I installed util-linux, and now it finds a lot of files
2021-07-22 10:59:31 <clandmeter> ok interesting
2021-07-22 10:59:49 <clandmeter> ionice makes a difference?
2021-07-22 10:59:53 <clandmeter> in what context?
2021-07-22 11:01:23 <ikke> They run the find command with ionice
2021-07-22 11:01:32 <ikke> [DRY RUN] find command: '/bin/ionice -c best-effort find -L /home/git/gitlab/shared/artifacts -mindepth 6 -maxdepth 6 -type d
2021-07-22 11:01:34 <ikke> ionice: invalid number 'best-effort'
2021-07-22 11:02:05 <clandmeter> ah it just fails to start
2021-07-22 11:02:51 <ikke> Fixing that should result in quite a bit of diskspace recovery
2021-07-22 11:03:04 <clandmeter> looks like bb does support it
2021-07-22 11:03:12 <clandmeter> but you need to provide the number
2021-07-22 11:03:16 <ikke> And if I look the disk usage graph, I suppose that keep artifact toggle only affects new jobs
2021-07-22 11:03:49 <ikke> yes, it's specifically 'best-effort' that is not supported
2021-07-22 11:04:09 <clandmeter> 2:best-effort
2021-07-22 11:04:12 <clandmeter> so -c 2
2021-07-22 11:05:09 <clandmeter> so we could patch it instead of adding another dep.
2021-07-22 11:05:13 <clandmeter> maybe even upstream it
2021-07-22 11:05:51 <ikke> nod
2021-07-22 11:06:18 <clandmeter> nice catch :)
2021-07-22 11:06:30 <ikke> I decided to actually read the error message for once :P
2021-07-22 11:06:54 <clandmeter> makes sense 
2021-07-22 11:33:44 <ncopa> i think i have a fix for rsync --delay-updates
2021-07-22 11:34:04 <ikke> That would be nice
2021-07-22 11:34:24 <ikke> I regularly manually sync the repos for ppc64le
2021-07-22 11:34:29 <ncopa> https://github.com/WayneD/rsync/pull/204
2021-07-22 11:34:36 <ncopa> lets see what upstream says
2021-07-22 16:02:31 <ikke> clandmeter: for now, should I just install util-linux in the gitlab container and run that cleanup job?
2021-07-22 16:02:40 <clandmeter> nod
2021-07-22 16:08:54 <ikke> hmm, not a lot of space gained
2021-07-22 16:09:10 <ikke> 4GB :/
2021-07-23 15:24:57 <ikke> clandmeter: ping
2021-07-23 15:25:05 <clandmeter> pong
2021-07-23 15:25:18 <ikke> trying to fix that unauthorized issue with gitlab
2021-07-23 15:25:27 <clandmeter> ok
2021-07-23 15:25:35 <ikke> I've found this page: https://docs.gitlab.com/ee/administration/gitaly/troubleshooting.html
2021-07-23 15:25:46 <ikke> It mentions this specific issue
2021-07-23 15:25:58 <ikke> "You need to sync your gitlab-secrets.json file with your GitLab application nodes"
2021-07-23 15:26:18 <ikke> But as far as I understand, that specific file does not exist anymore
2021-07-23 15:27:30 <ikke> there is a secrets.yml
2021-07-23 15:28:07 <clandmeter> tbh, i dont know that file
2021-07-23 15:28:17 <clandmeter> only secrets.yml
2021-07-23 15:28:22 <ikke> right
2021-07-23 15:28:37 <ikke> But as far as I can tell, gitaly does not refer to that spefic file
2021-07-23 15:28:55 <ikke> at least, I did not find any configuration or documentation
2021-07-23 15:29:13 <clandmeter> ah
2021-07-23 15:29:17 <clandmeter> thats omnibus
2021-07-23 15:29:21 <ikke> nod
2021-07-23 15:29:23 <clandmeter> we do from source
2021-07-23 15:29:28 <clandmeter> For Omnibus:
2021-07-23 15:29:28 <clandmeter> For installation from source:
2021-07-23 15:29:32 <clandmeter> oops
2021-07-23 15:29:46 <clandmeter> https://docs.gitlab.com/ee/raketasks/backup_restore.html#storing-configuration-files
2021-07-23 15:30:09 <clandmeter> that json is siumilar like the yml
2021-07-23 15:30:22 <clandmeter> omnibus used json
2021-07-23 15:30:29 <ikke> But I understand that it's only the web front-end that uses it?
2021-07-23 15:30:29 <clandmeter> and will probably generate the yml files
2021-07-23 15:31:10 <clandmeter> i dont know that many details
2021-07-23 15:31:12 <clandmeter> too long ago
2021-07-23 15:31:39 <ikke> So gitaly _does_ refer to a secrets file
2021-07-23 15:32:37 <clandmeter> from what i know
2021-07-23 15:32:54 <clandmeter> most of that data is encrypted with keys in secrets.yml
2021-07-23 15:33:08 <ikke> "secret_file = "/home/git/gitlab-shell/.gitlab_shell_secret"
2021-07-23 15:33:15 <ikke> that's in the gitaly ocnfig
2021-07-23 15:33:37 <clandmeter> isnt that auto genereated?
2021-07-23 15:34:21 <ikke> I don't know? What should generate it?
2021-07-23 15:34:27 <ikke> For now I generated the seret myself
2021-07-23 15:36:49 <clandmeter> ah
2021-07-23 15:36:52 <clandmeter> i remember now
2021-07-23 15:37:04 <ikke> https://tpaste.us/LmYp 
2021-07-23 15:37:05 <clandmeter> i think that file is just a file that need to be available on both sides
2021-07-23 15:37:10 <ikke> right
2021-07-23 15:37:17 <ikke> so I shared it with 3 containers
2021-07-23 15:37:22 <ikke> gitlab, gitlab-shell and gitaly
2021-07-23 15:37:26 <clandmeter> whatever the secret doesnt matter
2021-07-23 15:37:42 <ikke> that's what I assumed
2021-07-23 18:47:23 <ikke> clandmeter: hmm, disk usage on deu2 seems to be slowly decreasing
2021-07-23 18:49:57 <ikke> https://imgur.com/a/QM28Slr
2021-07-23 18:51:09 <ikke> https://pasteboard.co/Kcxz9s8.png
2021-07-24 18:02:38 <ikke> clandmeter: oh, apparently the .gitlab directory has a .gitlab_shell_secret as well..
2021-07-24 18:02:44 <ikke> /home/git/gitlab
2021-07-24 18:21:08 <ikke> So apparently the secret is generated during build..
2021-07-24 19:21:33 <ikke> YES!
2021-07-24 19:40:34 <ikke> So committing via the webif succeeds now
2021-07-25 13:09:13 <ikke> pushing over ssh does not work yet
2021-07-25 14:08:19 <danieli> what broke it?
2021-07-25 14:08:44 <ikke> Oh, I'm working on separating parts from the monolithic docker image
2021-07-25 14:09:02 <ikke> This is not about production :)
2021-07-25 14:09:13 <danieli> ah I see, how's that going in general?
2021-07-25 14:09:45 <ikke> Quite good, I have gitaly and gitlab-shell separated
2021-07-25 14:09:52 <ikke> gitlab is running
2021-07-25 14:10:14 <danieli> i suppose you had to mess about with mounting a lot of directories in various places?
2021-07-25 14:10:38 <ikke> Not that much
2021-07-25 14:10:54 <ikke> except gitlab-shell is used all over the place
2021-07-25 14:11:13 <ikke> I embedded the binaries in each image
2021-07-25 14:11:38 <ikke> But the rest is just 2 sockets
2021-07-25 14:11:53 <danieli> I see, it makes sense that it would be everywhere given that it manages SSH keys and git over ssh sessions
2021-07-25 14:12:09 <ikke> They are working on limiting it to just gitaly
2021-07-26 10:51:12 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/7
2021-07-26 10:52:00 <ikke> I've added a redirect on nld3 for dl-2 to dl-cdn
2021-07-26 11:01:14 <Ariadne> it would be nice if pkgs.a.o indexed the riscv64 repos
2021-07-26 11:01:28 <Ariadne> i looked around to see if i could figure out how to implement that
2021-07-26 11:04:45 <ikke> Need to add it to the configuration, and manually run the import once
2021-07-26 11:51:14 <clandmeter> is testing complete now?
2021-07-26 11:51:23 <ikke> not yet
2021-07-26 11:52:19 <ikke> clandmeter: do you agreen with that MR ^?
2021-07-26 11:54:16 <clandmeter> yup
2021-07-26 11:54:25 <clandmeter> but
2021-07-26 11:54:33 <clandmeter> what about rsync?
2021-07-26 11:55:35 <ikke> is dl-2 an rsync source?
2021-07-26 11:55:59 <clandmeter> from what i remember most dl-x are rsync backed
2021-07-26 11:56:09 <clandmeter> except cdn ofc
2021-07-26 12:00:14 <ikke> they get the data from rsync, but do they also provide data to other mirrors?
2021-07-26 13:40:02 <danieli> they shouldn't but they probably do in reality
2021-07-26 13:41:05 <ikke> At least I don't think any of the other mirrors on mirrors.a.o
2021-07-26 13:41:40 <ikke> Hmmm, maybe linorg.usp.br?
2021-07-26 13:42:16 <ikke> Oof: https://mirrors.alpinelinux.org/#mirror39
2021-07-26 13:42:24 <ikke> we should remove that mirror
2021-07-26 13:50:30 <danieli> ouch, that looks pretty dead
2021-07-26 13:51:53 <danieli> foobar.turbo.net.id looks a bit stale too
2021-07-26 13:54:30 <ikke> According to Zabbix that was just fixed
2021-07-26 13:54:45 <clandmeter> the last updated seems fine
2021-07-26 13:54:54 <ikke> Time since last mirror update
2021-07-26 13:54:56 <ikke> 2021-07-26 15:42:403h 42m 40s
2021-07-26 13:55:15 <ikke> It used to be 3 months
2021-07-26 13:55:29 <ikke> 5 months even
2021-07-26 13:56:52 <danieli> ah i missed the message here
2021-07-26 13:56:57 <danieli> the solved one from algitbot 
2021-07-27 04:22:14 <ikke> -- │ ChanServ has unset topic for #alpine-infra
2021-07-27 04:22:16 <ikke> huh?
2021-07-27 06:14:38 <danieli> wut
2021-07-27 06:14:48 <danieli> did some oper accidentally run a bad command?
2021-07-27 07:15:29 <danieli> hmm, chanserv changed topics in a few other channels i'm in around midnight
2021-07-27 16:10:44 <Ariadne> i think services does that when they reboot it
2021-07-27 16:11:29 <ikke> But not on all channels?
2021-07-27 16:11:51 <Ariadne> idk
2021-07-28 15:27:43 <ikke> clandmeter: What do you think we should do with dl-2? The mirror owner says they will no longer provide a public mirror
2021-07-28 15:28:28 <ikke> and btw, sync issues might be related to dl-master, ppc64le seems to sync fine with gbr1
2021-07-28 15:29:05 <ikke> It gets stuck on a file to dl-master, but it syncs entire repos to gbr1
2021-07-28 15:44:43 <ncopa> can we remove dl-2? i guess we no longer need it?
2021-07-28 15:45:56 <ikke> Yes, the question is if we are going to redirect it somewhere else
2021-07-28 15:46:03 <ikke> A cname I guess?
2021-07-28 15:46:46 <ncopa> do we have some other dl-*?
2021-07-28 15:46:48 <ncopa> dl-4?
2021-07-28 15:52:14 <ncopa> dl-1, dl-6, dl-7, dl-8 all resolves to something but does not work
2021-07-28 15:52:23 <ncopa> i think we should remove those dns names
2021-07-28 15:52:57 <ncopa> dl-4 and dl-5 points both to nl3.alpinelinux.org
2021-07-28 15:53:37 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/1
2021-07-28 15:53:58 <ncopa> dl-3 points to nld3-dev1.alpinelinux.org which does a 301 redirect to dl-cdn
2021-07-28 15:55:42 <danieli> nld3-dev1 probably runs nginx with a vhost set up for it, so adding another one there makes sense
2021-07-28 15:56:28 <ikke> curl -v --resolve dl-2.alpinelinux.org:443:147.75.101.119 https://dl-2.alpinelinux.org/alpine/ :-)
2021-07-28 16:05:30 <ncopa> looks like there is movement in upstream rsync finally!
2021-07-28 16:07:43 <ikke> https://github.com/WayneD/rsync/pull/204/commits/15ec7de5503c57860fb73ea6e4a349f1e70b72db
2021-07-28 16:25:41 <ikke> nice, merged
2021-07-28 18:30:57 <ikke> https://www.linode.com/docs/guides/how-to-use-nftables
2021-07-28 18:56:07 <danieli> nftables is awesome
2021-07-28 18:56:23 <danieli> i've been experimenting a ton with it lately
2021-07-28 18:57:45 <ikke> Need to still look into it
2021-07-28 18:58:40 <mps> yes, nftables is a lot better than iptables
2021-07-28 19:00:02 <mps> but I need to 'invent' some of higher level framework for it to use in production
2021-07-28 19:01:09 <danieli> not necessarily, but a higher level abstraction is nice
2021-07-28 19:01:14 <danieli> like iptables-nft or firewall-cmd
2021-07-28 19:02:36 <danieli> s/firewall-cmd/firewalld/
2021-07-28 19:02:36 <alpine-meetbot> danieli meant to say: like iptables-nft or firewalld
2021-07-28 19:02:45 <mps> yes, I'm thinking to make simple shell script to revert changes in case I made mess on remote systems
2021-07-29 06:35:30 <clandmeter> good morning
2021-07-29 06:35:52 <danieli> morning
2021-07-29 06:36:05 <clandmeter> ikke: lets clean those up
2021-07-29 06:36:29 <clandmeter> and yes, i think making uk the new master sounds good to me
2021-07-29 06:36:55 <clandmeter> it has enough space iirc
2021-07-29 06:37:45 <clandmeter> we can also ask them to solve it, but that sounds like another can of worms.
2021-07-29 06:37:58 <clandmeter> danieli: hi
2021-07-29 06:38:00 <clandmeter> hows life?
2021-07-29 06:38:20 <danieli> could be better, had a rough couple of days, other than that it's pretty decent
2021-07-29 06:39:06 <clandmeter> me too, but just tired.
2021-07-29 06:39:11 <clandmeter> that will solve itself
2021-07-29 06:39:42 <danieli> for me it's related to some pretty severe issues in my family
2021-07-29 06:40:05 <clandmeter> oh, sorry to hear that.
2021-07-29 06:40:42 <danieli> it is what it is, it'll be alright eventually
2021-07-29 06:40:44 <danieli> at least this time :)
2021-07-29 07:13:28 <ncopa> danieli: sorry hear
2021-07-29 07:13:51 <danieli> issues with my mother, and my grandfather had a heart attack
2021-07-29 07:14:00 <ncopa> :-(
2021-07-29 07:14:00 <danieli> but they're both alive and doing 'okay' so it's not a major concern
2021-07-29 07:14:32 <mps> ouch :(
2021-07-29 07:15:00 <ikke> clandmeter: what is your suggestion regarding dl-2? Do we keep a cname for the time being?
2021-07-29 07:15:05 <ikke> There are people using it apparently
2021-07-29 07:28:50 <clandmeter> Sounds good
2021-07-29 08:55:51 <ncopa> i pushed rsync fix to 3.14-stable and 3.13-stable. we can re-enable --delay-updates
2021-07-29 08:56:05 <ikke> did we disable it?
2021-07-29 09:00:07 <ncopa> dunno
2021-07-29 09:01:28 <ikke> I did not at least, and given that I regularly had to rsync and noticed that .~tmp~ files were removed, I assume we did not
2021-07-29 09:01:41 <ncopa> ok
2021-07-29 09:01:43 <ncopa> good
2021-07-29 09:03:08 <ncopa> hopefully you will no longer need to regularily rsync
2021-07-30 12:03:22 <ikke> ncopa: strange, still having issues on ppc64le
2021-07-30 12:03:47 <ikke> I suppose we need to update rsync on dl-master?
2021-07-30 12:26:01 <ikke> Updated rsync on dl-master to -r4
2021-07-30 13:28:21 <ncopa> yeah, problem was at receiver side
2021-07-30 13:29:40 <ikke> the network issues are stil there, though
2021-07-30 19:37:26 <ikke> ^^^^^^^^
2021-07-30 19:37:40 <ikke> ncopa: clandmeter Ariadne 
2021-07-30 20:21:27 <clandmeter> :)
2021-07-31 01:33:18 <Ariadne> lol very good uptime
2021-07-31 01:35:16 <Ariadne> ohhh
2021-07-31 01:35:18 <Ariadne> that’s funny
2021-07-31 01:35:45 <Ariadne> they gave us our box back and then scheduled deletion of the VMs
2021-07-31 01:35:52 <Ariadne> but i guess it deleted immediately
2021-07-31 01:43:07 <Ariadne> i pinged rafael
2021-07-31 16:18:12 <mps> can I post mail from alpinelinux.org somehow. I posted bug report to kernel but one of the recipients server refuses mail from my smtp server
2021-07-31 16:18:52 <danieli> isn't it easier to get a throwaway gmail or something for that?
2021-07-31 16:19:11 <danieli> i don't see why it has to be a specific domain or self-hosted mail server :)
2021-07-31 16:19:38 <ikke> For bug reports it does not matter, but gmail mangles patches
2021-07-31 16:19:50 <mps> well, this is solution I know, but solution which I dislike
2021-07-31 16:20:29 <mps> gmail, yahoo .... no thanks
2021-07-31 16:24:33 <danieli> you don't *have* to go for any of the conglomerates, it was just an example
2021-07-31 16:25:01 <danieli> i use protonmail myself, but the issue there is that i have to run an application locally to get SMTP and IMAP
2021-07-31 16:26:58 <mps> I thought about protonmail but decided to not use it
2021-07-31 16:28:22 <mps> and similar problems do not happen often, very rarely actually and I ignore such recipients/server
2021-07-31 16:28:58 <mps> in this case bug is related to alpine kernels and because that I asked
2021-07-31 16:34:03 <danieli> just curious, how come it refuses mail from your server?
2021-07-31 16:35:12 <mps> 554-kundenserver.de (mxeue010) Nemesis ESMTP Service not available 554-No SMTP service 554-Bad DNS PTR resource record.
2021-07-31 16:36:00 <ikke> Ah, no PTR record?
2021-07-31 16:36:09 <ikke> or at least not matching
2021-07-31 16:36:19 <danieli> you should be able to adjust PTR / rDNS through your server provider
2021-07-31 16:37:06 <mps> it is in my local macine
2021-07-31 16:37:15 <mps> machine*
2021-07-31 16:37:25 <danieli> aha, then it's most likely going to refuse it if you're hosting it from home and can't control PTR
2021-07-31 16:37:58 <mps> right that