2024-12-02 10:01:17 <clandmeter> ncopa: ive been playing with incus. it looks nice, but it has some serious issues for me. I think the aport also can use some love, including the ecosystem.
2024-12-02 10:12:00 <ncopa> ok?
2024-12-02 10:12:20 <ncopa> what serius issues? and what improvement would you like in the aport?
2024-12-02 10:15:20 <clandmeter> its having issues to boot from lvm
2024-12-02 10:15:38 <clandmeter> not sure what exactly is the issue, seems it cannot find the boot device
2024-12-02 10:16:24 <clandmeter> the aport has a agent that pulls in all the server deps, i guess that is less optimal
2024-12-02 10:16:50 <clandmeter> and the ecosystem provides vm images that are also not "perfect" 
2024-12-02 10:17:44 <clandmeter> it would be nice if we had a generic vm image that could pull in the needed stuff for incus, somethign with tinycloud i guess
2024-12-02 10:18:21 <clandmeter> tinycloud could setup incus-agent so we can access the console
2024-12-02 10:18:35 <clandmeter> incus exec vm1 sh or similar
2024-12-02 10:20:41 <ncopa> ah, you are using the vm
2024-12-02 10:21:11 <ncopa> yeah, the VM bit may need some polishing. i only use incus system containers for now
2024-12-02 10:21:17 <ncopa> have not had time to test and clean up the VM bits
2024-12-02 10:22:25 <clandmeter> well, if we use docker, we need to put them in a vm i guess
2024-12-02 10:22:56 <clandmeter> its nice to have vm's managed by incus
2024-12-02 10:23:22 <ncopa> you think incus is better than libvirt for managing vms?
2024-12-02 10:23:51 <ncopa> i have also kept an eye on lima vm
2024-12-02 10:23:59 <ncopa> which is mostly for macos
2024-12-02 10:24:25 <ncopa> but it also works with linux
2024-12-02 10:25:07 <clandmeter> as a replacement for qemu?
2024-12-02 10:26:49 <ncopa> no. it uses qemu as backend
2024-12-02 10:27:00 <ncopa> its an alternative to incus (vm) and libvirt
2024-12-02 10:27:07 <ncopa> which also uses qemu backend
2024-12-02 10:28:35 <clandmeter> ah ok
2024-12-02 10:28:41 <clandmeter> never seen it before
2024-12-02 10:39:29 <ncopa> I think it was built by ranches as a docker desktop alternative, for mac
2024-12-02 10:39:40 <ncopa> rancher*
2024-12-02 10:40:00 <ncopa> so its a vm and docker/containerd/kubernetes
2024-12-02 10:40:21 <ncopa> its a relatively conveninet way to spin up VMs on mac
2024-12-02 10:40:34 <ncopa> and i use it for my alpine vms on mac
2024-12-02 12:09:56 <durrendal> clandmeter: you can run docker images inside of an incus system container. But you need to enable nesting at least. I have several examples I can share if you'd like. 
2024-12-02 12:10:10 <durrendal> I use incus as my primary hypervisor currently. 
2024-12-02 12:23:37 <clandmeter> durrendal: you run the docker daemon in the container? 
2024-12-02 12:24:15 <clandmeter> are you using lvm?
2024-12-02 12:24:51 <clandmeter> ive setup a second storage based on an existing vg
2024-12-02 12:24:56 <clandmeter> but this seems to cause issues
2024-12-02 13:32:50 <clandmeter> ikke: do we prefer docker in vm or cnt?
2024-12-02 13:33:28 <durrendal> Yes precisely, here's an example I setup recently that runs a Mealie docker container. https://krei.lambdacreate.com/durrendal/Verkos/src/branch/trunk/Generated/setup-mealie-server.sh
2024-12-02 13:34:08 <durrendal> You probably need to set the back end to use fuse-overlayfs, and need to mount the sysfs into the container
2024-12-02 13:34:28 <durrendal> And Incus set $container security.nesting=true before trying to run docker
2024-12-02 13:34:40 <durrendal> But afterwards it should run as normal. 
2024-12-02 13:35:22 <durrendal> I personally use the ZFS backend, so maybe LVM is adding some additional friction, that part I'm uncertain on.
2024-12-02 13:35:52 <durrendal> When I said set the backend, I meant the docker storage configuration specifically, inside the container
2024-12-02 13:37:30 <ikke> clandmeter: I suppose a vm has more overhead in storage and memory?
2024-12-02 13:54:24 <clandmeter> performance wise, i guess cnt is better
2024-12-02 13:54:27 <clandmeter> but security wise..
2024-12-02 14:13:19 <durrendal> you're obviously trading isolation for less overhead with containers, so maybe that makes less sense in the context of something like the aports CI where someone could feasibly inject malicious code and then automatically execute it
2024-12-02 14:13:56 <durrendal> it would be a very public and stupid idea, but possible.
2024-12-02 14:15:20 <durrendal> automatically executed with little review I mean. versus the builders, which require a trusted agent to make a change to trigger them.
2024-12-02 19:27:59 <fossdd[m]> uhh ppc64le CI can't resolve libvirt.org https://gitlab.alpinelinux.org/fossdd/aports/-/jobs/1631308
2024-12-02 19:28:08 <fossdd[m]> wget: bad address 'libvirt.org'
2024-12-02 19:31:16 <ikke> fossdd[m]: just a single incident or continuously?
2024-12-02 19:31:32 <fossdd[m]> also on the third retry
2024-12-02 19:32:48 <ikke> hmm, srvfail in docker
2024-12-02 19:33:34 <ikke> issues with ipv6
2024-12-02 19:33:54 <ikke> It's something I've already reported, but no answer just yet
2024-12-02 19:34:05 <fossdd[m]> ah..
2024-12-02 19:38:04 <ikke> Hmm, not sure what's happening, this host does not have a publically routable ipv6 address
2024-12-02 19:38:35 <ikke> but for some reason dns in docker tries to use ipv6 dns servers
2024-12-02 19:53:16 <ikke> strange, I already did harcode dns servers in the docker daemon configuration
2024-12-03 09:09:23 <omni> what's up with the arm package builders?
2024-12-03 10:01:51 <ncopa> i think I messed up
2024-12-03 10:08:55 <ncopa> seems like https://build.alpinelinux.org/ does not respond
2024-12-03 10:19:53 <omni> relax, get up, stretch, take a walk around the room
2024-12-03 10:22:50 <ikke> ncopa: Can I help?
2024-12-03 10:23:54 <ikke> ncopa: I suspect a storage issue, the nginx processes are all in disk sleep
2024-12-03 10:26:31 <ikke> I'm rebooting the server
2024-12-03 10:30:56 <ikke> build.a.o is back
2024-12-03 10:34:53 <ncopa> ikke: thanks!
2024-12-03 17:54:27 <durrendal> ikke: prodding openbao and I do in fact have questions. I assume there's no real desire to create any sort of vendor lock in, correct? ie: AWS KMS as an HSM is an unappealing solution for unlocking the vault.
2024-12-03 18:00:00 <ikke> Indeed, and we do not have any credits on AWS
2024-12-03 18:02:59 <durrendal> that's what I thought, but shamir's secrets make sense given the distributed nature of the team. 
2024-12-03 18:03:37 <durrendal> if everyone is willing to setup GPG keys it looks like openbao can sign each person's shamir key with their key, so only the intended user can read the secret.
2024-12-03 18:04:41 <durrendal> though that begs the question of how many people would have keys to unlock the vault, and how much of a quorum is needed to handle that unlocking. It should be greater than one at least.
2024-12-03 18:08:47 <durrendal> does the netbox instance document all of the various bits of infrastructure that exist which may need to communicate with the openbao instance, and if so, would it be possible to get read only access so I can better understand the environment?
2024-12-03 18:09:01 <durrendal> If that's a bridge too far at the moment, no worries, completely understand.
2024-12-03 19:30:37 <ikke> durrendal: Should not be an issue. clandmeter do you see any issue with ^?
2024-12-03 19:32:17 <clandmeter> i read it, now i need to understand it.
2024-12-03 19:32:43 <ikke> was mostly about netbox
2024-12-03 19:34:17 <ikke> durrendal: I think 2-out-of-3 quorum could be feasible
2024-12-03 19:37:39 <clandmeter> where is this conversation coming from?
2024-12-03 19:37:48 <clandmeter> i miss a bit of context
2024-12-03 19:37:49 <clandmeter> i guess
2024-12-03 19:40:04 <ikke> clandmeter: I've asked durrendal if het could look at creating a deployment for openbao (vault)
2024-12-03 19:42:26 <durrendal> yes that precisely, I asked Ikke a couple months ago if there was anything that I could help with on the infrastructure side.
2024-12-03 19:42:26 <clandmeter> tbh, i don't have experience with vault, and not sure which issue we are trying to solve here. 
2024-12-03 19:42:54 <clandmeter> yes, i am aware and happy you are willing to help
2024-12-03 19:43:01 <clandmeter> just trying to understand 
2024-12-03 19:43:13 <ikke> clandmeter: My question was if you have any issue giving durrendal readonly access to netbox
2024-12-03 19:43:34 <ikke> Which also implies vpn access
2024-12-03 19:44:00 <clandmeter> nod
2024-12-03 19:44:09 <clandmeter> im Ok with it i guess
2024-12-03 19:44:53 <clandmeter> i thought you were asking to let durrendal setup a secret store and manage it, which seems interesting as a first thing to do in a projects infrastructure :)
2024-12-03 19:46:00 <durrendal> hahaha I would object to that. I shouldn't be considered for one of the unseal keys even, that level of trust is earned, and I will earn it in due time.
2024-12-03 19:46:36 <clandmeter> i think having access to netbox gives a better understanding of the infrastructure
2024-12-03 19:46:38 <clandmeter> so im Ok 
2024-12-03 19:46:56 <ikke> We would do the actual deployment, but durrendal can certainly help with creating docker( compose) files
2024-12-03 19:48:13 <clandmeter> or make first steps in deploying with ansible or similar
2024-12-03 19:49:03 <durrendal> I had considered doing that, if for no other reason than to ease my own test setup.
2024-12-03 19:50:56 <clandmeter> is it correct that ansible doesnt have proper incus support?
2024-12-03 19:52:20 <durrendal> the last time I checked that was the case, I don't think there's a way to spin containers/vms up and down with it. Though there was support for LXD before the project was forked
2024-12-03 19:53:33 <durrendal> https://library.tf/providers/lxc/incus/latest/docs/resources/project <- there is however a terraform/opentofu module for incus
2024-12-03 19:54:11 <durrendal> so you could use terraform to manage the container/vm lifetimes, and use the local provisioner to apply scripts/ansible playbooks to the containers/vms are you deploy them
2024-12-03 19:54:48 <clandmeter> nod
2024-12-05 00:18:08 <omni> build-edge-ppc64le is stuck?
2024-12-05 03:45:50 <mio> seems s390x ran low on disk space
2024-12-05 03:45:56 <mio> # k8s.io/kubectl/pkg/cmd/patch
2024-12-05 03:45:58 <mio> compile: writing output: write $WORK/b3560/_pkg_.a: no space left on device
2024-12-05 03:46:06 <mio> in https://build.alpinelinux.org/buildlogs/build-edge-s390x/community/k3s/k3s-1.31.3.1-r0.log
2024-12-05 03:46:26 <omni> =(
2024-12-05 03:46:53 <omni> thanks for noticing that's what's up with it
2024-12-05 03:47:25 <mio> yw
2024-12-05 04:07:53 <mio> seems to be fine now
2024-12-05 04:08:30 <mio> maybe just needed 1-2 retries
2024-12-05 08:47:42 <ncopa> do we have an alpinelinux org on quay.io?
2024-12-05 09:01:14 <ikke> Not that I'm aware of 
2024-12-05 11:20:42 <ncopa> can someone help review the sponsors listing? https://gitlab.alpinelinux.org/ncopa/alpine-mksite/-/blob/release-notes-3.21/posts/Alpine-3.21.0-released.md
2024-12-05 12:47:21 <ncopa> i have updated the latest-stable symlink on dl-master. we need purge the cache for latest-stable
2024-12-05 13:02:01 <ikke> on it
2024-12-05 13:03:02 <ikke> It's running
2024-12-05 13:31:06 <ikke> ncopa: latest-stable should be purged now
2024-12-05 13:31:13 <ncopa> thank you
2024-12-05 13:52:30 <ncopa> ikke: can you please help write a proper toot?
2024-12-05 13:52:37 <ncopa> post a toot for 3.21
2024-12-05 13:54:12 <ikke> yes
2024-12-05 13:54:42 <ncopa> thank you sir!
2024-12-05 13:55:34 <ncopa> do we need anything for getting 3.21 into pkgs.a.o? https://pkgs.alpinelinux.org/packages?name=&branch=v3.21&repo=&arch=x86_64&origin=&flagged=&maintainer=
2024-12-05 13:55:37 <ncopa> or we jsut wait
2024-12-05 13:56:42 <ikke> I expect we just need to wait
2024-12-05 13:56:57 <ncopa> awesome
2024-12-05 13:57:02 <ikke> clandmeter let it look at releases.json
2024-12-05 13:57:14 <fossdd[m]> oh cool
2024-12-05 13:57:15 <ncopa> seems like it also shows up in rpi-imager
2024-12-05 13:58:19 <ikke> ncopa: https://tpaste.us/av0B 
2024-12-05 13:59:43 <ncopa> 6.12 kernel is not yet announced as LTS
2024-12-05 14:00:09 <ncopa> so maybe just: This release also comes with Linux 6.12 kernel.
2024-12-05 14:01:42 <ikke> Right
2024-12-05 14:02:11 <ikke> https://fosstodon.org/deck/@alpinelinux/113600582275026707
2024-12-06 20:16:28 <omni> https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/issues/24
2024-12-06 20:17:02 <omni> to do that, summary would need to be replaced, but I haven't figured out with what yet
2024-12-06 22:20:01 <omni> s390x out of disk
2024-12-06 22:21:35 <ikke> yes, was already cleaning up
2024-12-06 22:27:48 <omni> 👍
2024-12-07 05:03:41 <zv> the cgit syntax highlighter may be choking; e.g. https://git.alpinelinux.org/apk-tools/tree/src/database.c it is also adding code as HTML tags, which *may* be a security issue if anyone with write access decides to be a meanie?
2024-12-09 01:53:12 <Hello71> zv: it shouldn't be a security issue at least, because only gitlab has anything sensitive and its cookies are contained to gitlab
2024-12-09 04:51:35 <zv> Hello71: xss is still a thing..
2024-12-09 09:16:30 <ikke> zv: has this been reported upstream?
2024-12-09 09:16:42 <ikke> (or known upstream)
2024-12-09 10:24:01 <zv> ikke: it is likely to do with your highlighter, not cgit
2024-12-09 10:24:07 <zv> e.g. https://cgit.adelielinux.org/apk-tools/tree/src/database.c
2024-12-09 10:26:14 <zv> https://gitlab.alpinelinux.org/alpine/infra/compose/cgit/-/blob/master/config/cgit/cgitrc?ref_type=heads#L8
2024-12-09 10:29:43 <ikke> Thanks, will look into it
2024-12-09 10:32:17 <ikke> https://gitlab.alpinelinux.org/alpine/infra/compose/cgit/-/blob/master/uwsgi/scripts/highlight.sh
2024-12-10 12:41:40 <omni> something with CI generate-build-jobs?
2024-12-10 12:44:50 <ikke> Not aware of any issues?
2024-12-10 12:45:26 <ikke> Seeing https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1643504
2024-12-10 12:45:40 <ikke> That can happen if the CI host is busy
2024-12-10 12:48:10 <omni> ok, thanks, was about to links to that one
2024-12-13 22:11:35 <vkrishn> ikke: https://gitlab.alpinelinux.org/api/v4/projects/1/issues/16728 , description does not seem ok
2024-12-13 22:11:52 <vkrishn> compare with, https://gitlab.alpinelinux.org/alpine/aports/-/issues/16728
2024-12-13 22:26:16 <vkrishn> ... This is the issue template ... , should be there ??
2024-12-13 22:26:34 <vkrishn> html source has it though
2024-12-13 23:08:19 <vkrishn> nm, i used sed to clean it
2024-12-16 07:50:03 <PureTryOut> It seems the builders fail to upload packages. According to build.a.o every upload on edge failed 🤔
2024-12-16 09:14:04 <omni> perhaps you shouldn't push directly without going through CI
2024-12-16 10:01:15 <omni> PureTryOut: I fixed your cd171e1eb8eb85023153bb1bda0549f09c0e8401 with !77134
2024-12-16 10:05:30 <PureTryOut> oh derp, thanks
2024-12-16 12:10:36 <fossdd[m]> Not sure why the CI doesn't build any packages in https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/77137
2024-12-16 12:19:10 <ikke> fossdd[m]: is there a syntax error in an aport somewhere?
2024-12-16 12:21:04 <fossdd[m]> does not seem so
2024-12-16 12:23:04 <ikke> There was 6 commits ago 
2024-12-16 12:23:09 <ikke> omni fixed it
2024-12-16 12:23:30 <ikke> The source branch is 14 commits behind the target branch.
2024-12-16 12:23:42 <ikke> So if you rebase, it should be fixed 
2024-12-16 12:37:58 <fossdd[m]> ah alr thanks!
2024-12-16 20:43:23 <vkrishn> has edge-x86_64 builder stopped ?
2024-12-16 20:46:23 <mio> not exactly stopped as blocked
2024-12-16 20:46:51 <mio> there's a package currently ftbfs with failed test
2024-12-16 20:47:39 <mio> not sure who to report it to as it was not maintainer who moved it from testing
2024-12-16 20:48:16 <mio> probably simplest is to skip the test
2024-12-16 20:48:51 <mio> it keeps retrying the same package so didn't retry for now
2024-12-16 20:50:37 <vkrishn> of move back to testing with note
2024-12-16 20:51:31 <mio> it's probably needed by another aport, and moved as part of the grommunio MR
2024-12-16 20:53:32 <vkrishn> noticed, can't whole MR be moved
2024-12-16 20:56:32 <vkrishn> should'nt deps be moved firs
2024-12-16 20:56:35 <vkrishn> first
2024-12-16 20:57:02 <mio> suppose it could, adding a workaround for one aport (if it's the only one) would be less moving parts
2024-12-16 20:57:18 <mio> s/adding/just that adding/
2024-12-16 20:57:53 <mio> they can be moved together with dependents
2024-12-16 20:58:08 <vkrishn> ok
2024-12-16 20:58:30 <vkrishn> but becomes a mess if deps fail
2024-12-16 20:59:34 <ncopa> im on it
2024-12-16 21:01:45 <mio> thanks ncopa!
2024-12-16 21:36:12 <midasi> where can I see the builders?
2024-12-16 21:36:34 <mio> https://build.alpinelinux.org/
2024-12-16 21:36:47 <midasi> thanks
2024-12-16 22:52:01 <sertonix[m]> It looks like when building dcc ./configure is finding binaries inside of /usr/local/bin which causes wrong paths to end up in the package. Would it be possible to wipe /usr/local on all builders? And maybe prevent files from being created there in the future?
2024-12-16 23:11:50 <vkrishn> sertonix[m]: https://pkgs.alpinelinux.org/contents?file=&path=%2Fusr%2Flocal&name=&branch=edge&repo=&arch=x86_64 no-files
2024-12-16 23:13:18 <vkrishn> it should be there
2024-12-16 23:13:22 <vkrishn> should not
2024-12-16 23:15:00 <vkrishn> but I have noticed empty folder being created in /usr/local 
2024-12-16 23:16:40 <sertonix[m]> I know that this isn't in a package. I suspect that something else has left files in /usr/local on the builders.
2024-12-16 23:17:36 <vkrishn> hmm, yes, i replied quick, did not read full
2024-12-17 11:20:14 <ncopa> i wonder what we can do to improve performance for gitlab.a.o
2024-12-17 11:20:31 <ikke> ncopa: anything specifically?
2024-12-17 11:20:33 <ncopa> it looks like it is currently using 600% cpu and has done so for last months
2024-12-17 11:21:08 <ncopa> git pull from cli, git rebase/merge from web interface
2024-12-17 11:21:43 <ncopa> seems like we run everything on a single host?
2024-12-17 11:21:52 <ikke> reduce the size (in the sense of amount of commits) from history would help
2024-12-17 11:21:54 <ikke> yes, a single host
2024-12-17 11:22:32 <ncopa> i wonder what it is that uses the cpu. I dont have ssh access to the host
2024-12-17 11:22:55 <ncopa> i also wonder if it would make sense to split out the pg database and gitaly server to separate nodes
2024-12-17 11:23:43 <ncopa> sorry CPU load is 200-300%
2024-12-17 11:23:47 <ncopa> not 600
2024-12-17 11:24:03 <clandmeter> as mentioned before, it would be nice if we could get some eu colocation space
2024-12-17 11:24:16 <clandmeter> some close by
2024-12-17 11:24:20 <clandmeter> at least for me
2024-12-17 11:24:36 <clandmeter> i can introduce some computing power
2024-12-17 11:24:49 <ikke> https://imgur.com/a/o83vsqe
2024-12-17 11:25:06 <ncopa> hum
2024-12-17 11:25:41 <ncopa> https://cloud.linode.com/linodes/13397180/analytics say 297% avg
2024-12-17 11:25:47 <ikke> load average: 3.49, 5.38, 5.10
2024-12-17 11:25:51 <ncopa> which sounds crazy
2024-12-17 11:26:16 <ncopa> load avg 3.49 corresponds to 297% cpu usage
2024-12-17 11:26:26 <ikke> but with 6 cores
2024-12-17 11:26:55 <ikke> so ~0.5 per core
2024-12-17 11:27:07 <ncopa> load avg does not include all cores? its only for a single core?
2024-12-17 11:27:29 <ncopa> might be me who dont understand how it works
2024-12-17 11:27:58 <ikke> For a single core, a load of 1 means the cpu is constantly busy
2024-12-17 11:28:15 <ikke> The load is not exactly the same as cpu usage
2024-12-17 11:28:29 <ncopa> ok
2024-12-17 11:28:41 <ikke> "It refers to the number of processes which are either currently being executed by the CPU or are waiting for execution."
2024-12-17 11:29:39 <ikke> So for a system with 6 cores, a load <6 means that there is room left
2024-12-17 11:29:49 <ncopa> ok
2024-12-17 11:30:02 <ncopa> i wonder where the bottlenecks are though
2024-12-17 11:30:13 <ikke> gitaly is a bottleneck
2024-12-17 11:30:42 <ncopa> so we could maybe move it to separate node, in same datacenter?
2024-12-17 11:31:17 <ncopa> or even have multiple gitaly servers
2024-12-17 11:31:28 <ikke> That would increase complexity quite some bit
2024-12-17 11:31:36 <ncopa> yup
2024-12-17 11:32:34 <ikke> deployment, backups
2024-12-17 11:32:43 <ncopa> nod
2024-12-17 11:33:14 <ikke> Not sure how much it matters, but gitlab creates a _ton_ of internal references
2024-12-17 11:33:51 <ncopa> i dont know if we could tweak the gitaly settings, increase caches etc
2024-12-17 11:34:28 <ncopa> i also don't know how much read-only traffic we get (people/bots doing git clones/pulls in read-only mode, not logged in)
2024-12-17 11:34:31 <ikke> We already have packed object caching
2024-12-17 11:39:24 <ikke> There are 1.1m refs in the repository
2024-12-17 11:58:28 <clandmeter> which performance issues are we having? i guess only git repo related?
2024-12-17 12:01:46 <clandmeter> ncopa: you could try to enable the performance bar
2024-12-17 12:01:52 <clandmeter> it will help a bit analyzing 
2024-12-17 12:05:19 <ikke> There are moments where `git pull` on aports takes about half a minute before you see anything happening
2024-12-17 12:09:27 <clandmeter> yes i noticed that
2024-12-17 12:10:32 <clandmeter> did you look into: https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html ?
2024-12-17 12:10:55 <clandmeter> i mean we dont use it, but i guess it has some reference to the actual product
2024-12-17 12:14:35 <vkrishn> the webui does lots of async calls, means if some text don't show up, user does refresh, again recalls
2024-12-17 12:14:55 <vkrishn> this can be reduced if the same async calls are done in backend, clubbed the json into one big json , then send to browser, i don't know if its doable currently
2024-12-17 12:15:30 <vkrishn> lots of refresh increases loads significantly
2024-12-17 12:17:13 <vkrishn> coz it may need to call all web-assets
2024-12-17 12:18:21 <vkrishn> checking the page if static assets are already split into separate cdn
2024-12-17 12:18:57 <ikke> It's not feasible to change the webinterface
2024-12-17 12:19:05 <ikke> and I'm not convinced it will help a lot anyway
2024-12-17 12:20:18 <vkrishn> ok
2024-12-17 12:21:00 <vkrishn> nope, assets from same server, src:url(./gitlab-sans/GitLabSans-1e0a5107ea3bbd4be93e8ad2c503467e43166cd37e4293570b490e0812ede98b.woff2)
2024-12-17 12:21:48 <invoked> wonder if the reftables backend would help
2024-12-17 12:22:20 <ikke> It may
2024-12-17 12:22:21 <vkrishn> if just static assets can be changed, by changing templates it might be some help
2024-12-17 12:29:38 <invoked> i dunno how risky it would be to migrate to reftables, or how that should be done within gitlab
2024-12-17 12:32:49 <ikke> I would not do it in production unless gitlab supports it
2024-12-17 12:32:57 <ikke> it may completely break gitaly
2024-12-17 12:33:24 <invoked> a quick search shows that gitlab does support reftables but i don't see anything about migrating
2024-12-17 12:33:59 <invoked> (yet)
2024-12-17 12:37:17 <invoked> https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/8008/diffs?commit_id=4f09ab2f1d13dd16ab3d3161a26490bc1cf3b198#e96a8f8677435e8e6803b8dcce6cf064be884fb1_0_132
2024-12-17 12:38:21 <invoked> further down, "
2024-12-17 12:38:23 <invoked> For existing repositories, there could potentially be a situation where the migration corrupts the repository.
2024-12-17 12:49:08 <durrendal> Gitlab has an open telemetry endpoint built into it, I'm not certain if we enable it in our setup, it sounds like we don't if we don't have the performance bar enabled.
2024-12-17 12:49:11 <durrendal> https://docs.gitlab.com/ee/administration/monitoring/performance/
2024-12-17 12:50:49 <durrendal> I'd first look here personally, on the surface the system being under 50-60% load seems like it should be OK, but there's clearly something else going on here.
2024-12-17 12:56:28 <ikke> prometheus is enabled, but apparently we need to export prometheus_multiproc_dir for it to work
2024-12-17 12:56:43 <ikke> " WARNING: Environment variable prometheus_multiproc_dir does not exist or is not pointing to a valid directory. "
2024-12-17 13:11:53 <vkrishn> https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html#metrics-shared-directory
2024-12-17 13:17:32 <vkrishn> with prometheus 200M increase, https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html#disable-monitoring
2024-12-18 15:08:51 <prabu> Since the irc logs from all the alpine channels are available, can this be made searchable so that previously answered questions/doubts can help users. Since we currently do not have a forum, irc logs should be having a lot of valuable information.
2024-12-19 20:48:30 <durrendal> ikke: clandmeter: I may just just a few minutes delayed to our meeting. Left the office later than I expected, but I should be no later than 5 minutes at most. 
2024-12-19 20:49:01 <clandmeter> Np 
2024-12-19 22:06:25 <durrendal> Thanks again for the chat clandmeter ikke! Looking forward to helping out :) 
2024-12-19 22:06:34 <ikke> Yeah, it was nice meeting you
2024-12-19 22:06:36 <clandmeter> thank you!
2024-12-20 14:53:22 <ncopa> $ git pull --rebase
2024-12-20 14:53:22 <ncopa> Connection closed by 2a01:7e01:e001:15a:1::2 port 22
2024-12-20 14:53:22 <ncopa> fatal: Could not read from remote repository.
2024-12-20 14:53:22 <ncopa> Please make sure you have the correct access rights
2024-12-20 14:53:22 <ncopa> and the repository exists.
2024-12-20 14:54:49 <ncopa> seems like somethign in gitlab is broken. I cant pull or push over ssh
2024-12-20 15:03:07 <ikke> ncopa: i can check in a bit
2024-12-20 15:28:31 <ncopa> seems like it works now
2024-12-20 18:59:31 <ikke> ncopa: checking the logs, but cannot find anything that would explain it