2022-01-02 11:09:37 I seem to be no longer able to reach my riscv64 alpine container at nld5-dev1.alpinelinux.org:22204 has the port forwarding maybe stopped working for some reason? 2022-01-02 11:09:40 > ssh: connect to host nld5-dev1.alpinelinux.org port 22204: Host is unreachable 2022-01-02 11:13:22 perhaps, let me check 2022-01-02 11:15:13 Seems like the ip was changed, so we didn't create a static record 2022-01-02 11:16:30 oh, the container is just not started 2022-01-02 11:16:47 even though autostart is enabled 2022-01-02 11:17:12 ok, started, it should work again 2022-01-02 11:23:11 yes, does work again. thanks! :) 2022-01-02 11:26:21 for some reason, all the rv64 containers did not start 2022-01-04 09:32:01 clandmeter: what are we going to do about those mirrors? 2022-01-04 13:32:27 hi ikke 2022-01-04 13:32:43 ive been swamped recent days/weeks 2022-01-04 13:33:27 so im not sure which issue you are referring too? do we have one on gitlab? 2022-01-04 13:34:46 Mirror space that zabbix alerts about 2022-01-04 20:40:57 ikke: im not sure 2022-01-04 20:41:07 we have a few options 2022-01-04 20:41:20 remove some releases 2022-01-04 20:41:25 or ask for more storage 2022-01-06 18:59:05 clandmeter: I've migrated gitlab traefik to v2.5, and it now forwards ssh 2022-01-06 20:57:55 is the riscv64 builder stuck on zabbix? hasn't uploaded in a while… 2022-01-06 20:58:34 btw: would it be possible to make the riscv64 builder buildlog available before the entire software is build? like on the x86_64 builder 2022-01-06 20:59:04 i.e. the zabbix buildlog currently gives 404 here https://build.alpinelinux.org/buildlogs/build-edge-riscv64/community/zabbix/zabbix-5.4.9-r1.log 2022-01-06 20:59:16 It was only possible on x86_64 because that was the host that served distfiles 2022-01-06 20:59:32 We don't have any facility yet to upload it live 2022-01-06 20:59:55 distfiles has moved, so it's not live anymore for x86_64 2022-01-06 21:00:02 oh, unfourtunate 2022-01-06 21:00:52 always found that useful to figure out if there is a problem with the build, e.g. if it is stuck and such 2022-01-06 21:01:40 It's hanging on go 2022-01-06 21:01:57 the riscv64 builder? 2022-01-06 21:02:19 yes 2022-01-06 21:02:48 is it hanging on the go 1.7.4 upgrade? because that was a while ago :D 2022-01-06 21:03:50 no 2022-01-06 21:04:06 I pushed a fix 2022-01-06 21:04:19 https://tpaste.us/9P5P 2022-01-06 21:04:58 ah, hangang on the go parts in zabbix 2022-01-06 21:04:58 i see 2022-01-06 21:05:48 algitbot: kick master 2022-01-06 21:06:06 algitbot: retry master 2022-01-06 21:08:34 nice, ty 2022-01-06 21:08:59 regarding live log files: would it not be possible to setup a http server on the new distfiles server to get live logfiles again? 2022-01-06 21:09:10 nmeum: the issue is uploading the logs 2022-01-06 21:09:20 this happens now as a separate step after building the package 2022-01-06 21:10:29 ah 2022-01-06 21:10:30 hm 2022-01-06 21:14:19 I'm a bit puzzled 2022-01-06 21:15:52 So on most arches we have a plugin for buildrepo that uploads the logs and anounces it on mqtt 2022-01-06 21:16:11 but not on x86 and x86_64, yet it it's still working 2022-01-06 21:17:53 hehe 2022-01-06 21:50:59 alternative idea: should it not be possible to write log files from the builders to some sort of nfs and have that nfs mounted on the server powering buildlogs.alpinelinux.org for live buildlog updates? thereby avoiding the upload step altogether? 2022-01-07 07:24:18 is the riscv64 builder actually doing stuff on numpy or is it stuck again? 2022-01-07 10:19:11 this could be used to deter regular spaced text spams - https://tpaste.us/lyLn 2022-01-07 14:10:55 ikke: if you have a free minute: could you check what the riscv64 builder is doing on numpy? seems weird that it spends so many hours building it or am I just impatient? :D 2022-01-07 14:26:33 You can kick it yourself and get the buildlog 2022-01-07 16:33:10 algitbot: kick risv64 2022-01-07 16:33:12 algitbot: kick naster 2022-01-07 16:33:15 algitbot: kick master 2022-01-07 16:33:49 kick ikke's keyboard? 2022-01-07 21:04:45 heh, that's fast, just sent them an e-mail :) 2022-01-12 05:41:29 security anouncement for gitlab 2022-01-12 07:26:16 ikke: some of the fixes seem especially important, i was notified through a thing at work 2022-01-12 07:30:39 building new versions 2022-01-12 11:25:48 Upgraded gitlab-test to 14.4.5 2022-01-12 19:10:37 gitlab upgraded to 14.4.5 2022-01-14 17:27:41 are we using gitlab 11.3? is there any chance we could enable interactive web terminals for the ci? should ease debugging build/check failures, see https://docs.gitlab.com/ee/ci/interactive_web_terminal/ 2022-01-14 17:28:05 we are at 14.4 atm :) 2022-01-14 17:28:24 It requires things to be setup correctly 2022-01-14 17:28:33 and 2022-01-14 17:28:40 it does not work with docker executors afaik 2022-01-14 17:29:42 "The docker executor does not keep running after the build script is finished. At that point, the terminal automatically disconnects and does not wait for the user to finish." 2022-01-14 17:59:46 unfourtunate 2022-01-15 13:42:12 mps: you still have access to netbox, right? 2022-01-15 13:57:37 I think so, only not sure if I have password 2022-01-15 14:10:54 You do have an account there, so if I need to reset your pw, let me know 2022-01-15 14:17:22 I don't often connect there 2022-01-15 14:19:30 but probably would be good to have password if I need to look something or change/add 2022-01-15 14:19:44 Sent via pm 2022-01-15 14:20:38 thanks 2022-01-20 05:48:54 clandmeter: mips64 is back at the moment 2022-01-20 09:28:24 ok, but it cant upload iirc 2022-01-20 09:30:16 Correct, its constantly trying 2022-01-20 22:40:05 ikke: I cleaned my lxc containers, they are ready for removal 2022-01-21 13:53:40 did the lounge die for some hours? 2022-01-21 15:05:11 danieli: it did 2022-01-21 15:05:18 i see 2022-01-21 16:45:27 clandmeter: when you have time, can you check usa5? 2022-01-21 16:45:48 /var/lib/docker has 50G data which I cannot account for 2022-01-23 18:28:49 :.' 2022-01-23 18:28:51 :/ 2022-01-24 13:55:11 ikke: are we getting hammered? 2022-01-24 20:55:12 clandmeter: doesn't look like it 2022-01-24 20:55:34 load is reasonable on the host 2022-01-24 20:55:37 not a lot of requests 2022-01-24 20:56:11 restarted the workers 2022-01-25 12:28:17 I tried to get my password on https://meta.alpinelinux.org/forgot but it says 'No account found with this email address.' for mps@arvanta.net though I receive mails 2022-01-25 12:29:15 how can I unsubscribe 2022-01-25 12:32:14 Send an email to ~alpine/devel+unsubscribe 2022-01-25 12:32:29 @lists.alpinelinux.org 2022-01-25 12:32:36 ikke: thanks 2022-01-25 12:32:39 Same for other lists 2022-01-25 20:00:03 clandmeter: ping 2022-01-25 20:00:10 ping 2022-01-25 20:00:14 pong 2022-01-25 20:00:22 i mean pong 2022-01-25 20:00:25 :) 2022-01-25 20:00:25 was a typo :) 2022-01-25 20:00:32 I assumed so 2022-01-25 20:00:45 so on usa5, we have the contents of /srv duplicated 2022-01-25 20:00:59 one time on vg0 and one time on vg1 2022-01-25 20:01:30 I think docker already uses the directory before vg1/storage was mounted on /srv 2022-01-25 20:01:52 So there was 60G of space I could not locate 2022-01-25 20:02:16 The issue is, I don't really know how to properly merge it 2022-01-25 20:02:36 It's not only data, it's the entire structure that is nearly duplicated 2022-01-25 20:03:00 I have mounted vg1/storage on /mnt/storage 2022-01-25 20:03:04 (and stopped docker) 2022-01-25 20:03:24 im looking but dont understand yet :) 2022-01-25 20:03:35 compare /srv with /mnt/storage 2022-01-25 20:05:20 ah now i start to understand 2022-01-25 20:05:41 currently vg1 was not used? 2022-01-25 20:05:47 it was mounted on /srv 2022-01-25 20:05:55 so hiding /srv on vg0 2022-01-25 20:06:02 nod 2022-01-25 20:06:35 so why not just kill it? 2022-01-25 20:06:43 kill what? 2022-01-25 20:06:55 srv data on vg0? 2022-01-25 20:07:05 because I think the builder has been writing to there 2022-01-25 20:07:18 it has 60G of data 2022-01-25 20:07:33 the rv64 builder 2022-01-25 20:08:41 but if you mount it on another disk the other data is not accessible right? 2022-01-25 20:08:54 correct, unless a process already has it open 2022-01-25 20:09:05 so if docker happens to start before the volume is mounted 2022-01-25 20:11:12 hmm 2022-01-25 20:16:13 if you ask me the data in /srv is just old 2022-01-25 20:16:19 the volume isnt in fstab? 2022-01-25 20:16:28 and the disk probably got full 2022-01-25 20:16:39 and created a new volume and copied the data 2022-01-25 20:16:43 then mounted it 2022-01-25 20:16:44 MY-R: it is 2022-01-25 20:17:04 so how docker can start before localmount service finish? :\ 2022-01-25 20:17:07 clandmeter: it was vg0 that was getting full lately 2022-01-25 20:17:13 nod 2022-01-25 20:17:22 so something was writing to vg0 2022-01-25 20:17:38 yes its root 2022-01-25 20:17:44 and the only things that take space is /var/lib/docker and /srv 2022-01-25 20:18:07 / is mounted on vg0 right? 2022-01-25 20:18:10 yes 2022-01-25 20:18:19 /dev/mapper/vg0-lv_root on / type ext4 (rw,relatime) 2022-01-25 20:18:37 so that volume was getting full 2022-01-25 20:18:56 and builds failed because of disk usage 2022-01-25 20:19:11 so the builder was definitely writing to vg0-lv_root 2022-01-25 20:19:18 in the past yes 2022-01-25 20:19:22 no 2022-01-25 20:19:24 today 2022-01-25 20:19:36 yes 2022-01-25 20:19:40 but not in srv i think 2022-01-25 20:19:48 probably in docker 2022-01-25 20:19:50 why would it fail with disk usage? 2022-01-25 20:20:02 the containers and such are on vg0 2022-01-25 20:20:13 probably somethign is filling up the container itself? 2022-01-25 20:20:54 -rw-r--r-- 1 dnsmasq dnsmasq 523K Jun 5 2021 /srv/compose/aports-build/storage/packages/main/riscv64/APKINDEX.tar.gz 2022-01-25 20:20:57 right 2022-01-25 20:21:04 thats the date i did the change (i think) 2022-01-25 20:21:06 /var/lib/docker/overlay2/d6fdd8b5997463d6c4e324f771965705ffb42c1beeaef5fe2730ab4d2cb83ee0/diff/home/buildoze 2022-01-25 20:21:11 /var/lib/docker/overlay2/d6fdd8b5997463d6c4e324f771965705ffb42c1beeaef5fe2730ab4d2cb83ee0/diff/home/buildozer 2022-01-25 20:21:34 so if we destroy the container, the packages are gone? 2022-01-25 20:21:58 no, packages is empty 2022-01-25 20:22:03 no i dont think so 2022-01-25 20:22:07 look at the compose 2022-01-25 20:22:23 some parts are bind mouinted to ./storage 2022-01-25 20:22:32 .cache 14G 2022-01-25 20:22:40 yeah 2022-01-25 20:22:44 go 5G 2022-01-25 20:22:46 i didnt mount all of it 2022-01-25 20:22:53 yeah, ok, that makes sense 2022-01-25 20:23:21 we could if you want 2022-01-25 20:23:30 if you want to reuse cache parts 2022-01-25 20:23:47 Not sure 2022-01-25 20:24:13 well in this way its also useless 2022-01-25 20:24:33 in a perfect world the container would be spin for each package cleaning it self up after 2022-01-25 20:25:22 i think we also discussed about moving it out of docker 2022-01-25 20:25:37 we have an lxc container on nld5 already 2022-01-25 20:25:56 oh, that's 3.15 I think 2022-01-25 20:26:12 yeah, was a test to bootstrap 3.15 2022-01-25 20:27:23 we can move it 2022-01-25 20:27:34 but does the destination have the space needed? 2022-01-25 20:28:49 vg1 has enough space 2022-01-25 20:29:02 on nld5? 2022-01-25 20:29:05 oh 2022-01-25 20:29:32 /var/lib/lxc has 129G 2022-01-25 20:29:37 if i understand correctly, for usa5 we can just rm -rf /srv (if not mounted) :) 2022-01-25 20:29:59 nld5 is equinix? 2022-01-25 20:30:00 ok, feel free :P 2022-01-25 20:30:02 clandmeter: yes 2022-01-25 20:30:35 ah that the dev box 2022-01-25 20:30:38 yes 2022-01-25 20:31:09 ok, what do we gain if we move it to nld5? 2022-01-25 20:31:40 usa5 is a system only used by CI iirc 2022-01-25 20:31:56 only thing we would gain is that it uses the same structure as other builders 2022-01-25 20:31:57 its a troubled system which could fail on high load. 2022-01-25 20:32:17 at least it did that long time ago 2022-01-25 20:33:27 if we start doing releases for riscv the space is going to be a bottleneck again 2022-01-25 20:33:31 yes 2022-01-25 20:33:45 it would be nice if we could abuse usa5 for that 2022-01-25 20:33:59 we could start using it with lxc again 2022-01-25 20:34:10 but then again, we said we only do stables if we have the hw 2022-01-25 20:34:17 nod 2022-01-25 20:34:29 which I think is still quite a while out 2022-01-25 20:35:08 adding lxc to usa5 sounds like the best plan 2022-01-25 20:35:17 not sure how much trouble it will make 2022-01-25 20:36:16 how would you manage the CI instances? 2022-01-25 20:36:27 via qemu or run docker and lxc together? 2022-01-25 20:37:07 afaik docker and lxc can run together without issue 2022-01-25 20:38:00 do you have it on your bucketlist? :) 2022-01-25 20:38:20 Runing lxc and docker on one system? 2022-01-25 20:38:23 Yes, ofcourse! 2022-01-25 20:38:36 :) 2022-01-25 20:39:00 I think I've even already ticked that one 2022-01-25 20:39:18 if you can get lxc and docker running on usa5, i dont mind moving the builder to it 2022-01-25 20:39:28 try ls /var/lib/docker on nld3 :) 2022-01-25 20:40:33 but no daemon running :) 2022-01-25 20:40:38 no, we stopped it again 2022-01-25 20:40:42 but it did run 2022-01-25 20:40:52 sure it can run 2022-01-25 20:40:57 but? 2022-01-25 20:40:59 but without issues preferabel :) 2022-01-25 20:41:07 what kind of issues? 2022-01-25 20:41:14 without network madness 2022-01-25 20:41:30 but i think awall has some logic to handle it now 2022-01-25 20:41:37 As long as docker has it's own subnet, it should not be an issue 2022-01-25 20:41:41 yes 2022-01-25 20:41:54 but i have not looked in it for a long time 2022-01-25 20:41:58 what we cannot do easily afaik is let docker use part of the dmvpn subnet space 2022-01-25 20:42:06 but I don't think that matters 2022-01-25 20:42:09 i think ncopa also had docker running in lxc iirc 2022-01-25 20:42:23 right, but with a lot of extra privileges I think 2022-01-25 20:42:32 I tried it, but could not get it quite running 2022-01-25 20:42:37 yes 2022-01-25 20:42:47 but running docker on the host is that same issue i guess 2022-01-25 20:42:57 what same issue? 2022-01-25 20:43:07 afaik it runs ootb 2022-01-25 20:43:15 you need to give it perms to run it in lxc 2022-01-25 20:43:30 right, and provide the cgroup interface 2022-01-25 20:43:32 but thats the same as running it on the host itself 2022-01-25 20:43:39 yeah, basically 2022-01-25 20:43:48 it would not be for security reasons to do it 2022-01-25 20:44:02 indeed, so I don't see the advantage 2022-01-25 20:44:04 could be practical, but im not sure. 2022-01-25 20:44:18 a bit like docker in docker 2022-01-25 20:44:38 not even sure how iptables would work 2022-01-25 20:45:02 so if you are up to it, please go ahead and add lxc on usa5 2022-01-25 20:45:21 I've enabled awall on deu2 (gitlab host) 2022-01-25 20:45:30 as long as you have the dedicated chains enabled 2022-01-25 20:45:31 then i will try to reserve some time to move it (when my house is in a state of less attention) 2022-01-25 20:46:19 or you can also try it, if you can find the time. all fine by me. 2022-01-25 20:47:19 ill go and watch some tv now to relax a bit. i feel i need it :) 2022-01-25 20:47:24 o/ 2022-01-25 20:47:32 so regarding the builder 2022-01-25 20:47:39 I probably can restore it for now? 2022-01-25 20:47:40 let me know what you decide 2022-01-25 20:47:47 I mean, in the interrim 2022-01-25 20:47:51 sure 2022-01-25 20:47:53 9G free space 2022-01-25 20:47:59 i think you can kill /srv 2022-01-25 20:48:03 ok, will do that 2022-01-25 20:48:11 if that helps 2022-01-25 20:48:12 or I can move it asside even 2022-01-25 20:48:18 oh no 2022-01-25 20:48:23 that's on a different partition 2022-01-25 20:48:37 Have a good evening 2022-01-26 07:08:40 clandmeter: we are not using tls-alpn-01 for letsencrypt, right? 2022-01-26 07:09:16 Afaik we only use dns-01 2022-01-26 07:38:27 uhm, we are mostly using the wildcard dns cert 2022-01-26 07:39:20 not sure if we have some service that fetches one itself 2022-01-26 07:44:27 But that would most likely use an http challenge 2022-01-26 08:42:22 yes it will 2022-01-26 08:42:32 what is it about alpn? 2022-01-26 08:50:10 They found issues with the implementation and are revoking certs 2022-01-26 08:50:44 https://community.letsencrypt.org/t/2022-01-25-issue-with-tls-alpn-01-validation-method/170450 2022-01-26 20:04:41 dns-01 is best 2022-01-26 20:06:14 wildcard on LE requires dns-01 so this was unaffecting if everything was using a wildcard 2022-01-26 20:06:26 (unless they changed it in the past 3 months..) 2022-01-26 20:14:13 Yes, and tls-alpn-01 is fairly new 2022-01-26 20:14:26 We use wildcard for most, but there might be a dns-01 cert here and there 2022-01-26 20:15:01 uhh 2022-01-26 20:15:16 do you get your wildcard from somewhere not-LE 2022-01-26 20:16:11 (because LE wildcards require dns-01 so those words confused me) 2022-01-26 20:22:14 wildcards are dns-01 only :D 2022-01-26 20:23:15 psykose: we use wildcars, therefore dns-01 in most situations 2022-01-26 20:44:09 not a great option, log folder can be squash`ed fs and remounted(overlayfs) there itself 2022-01-26 20:44:17 would save tons of space 2022-01-26 20:44:22 vkrishn: wrong channel? 2022-01-26 20:44:41 nope, just toooo late response 2022-01-26 20:44:43 ah 2022-01-26 20:45:03 What was that a response to? 2022-01-26 20:45:19 logs hogging the spaces in servers 2022-01-26 20:51:51 ikke: https://tpaste.us/eV1X 2022-01-26 20:56:04 oh, heh 2022-01-26 21:00:02 How would squashfs help for a log directory? It's read-only? 2022-01-26 21:01:26 overlayfs 2022-01-26 21:01:48 yes, but what would squashfs add? 2022-01-26 21:02:45 We want these logs continuously rotating, not keeping them indefinitely 2022-01-26 21:03:05 this would be ideal in situation where there only logs are still needed, else it can be purged or backed'up 2022-01-26 21:03:26 I where old logs are still needed 2022-01-26 21:03:35 That's not the case here 2022-01-26 21:03:36 I mean 2022-01-26 21:03:41 ok 2022-01-27 16:55:34 clandmeter: I have built gitlab 14.5 and deployed it on gitlab-test 2022-01-27 17:37:32 I've cleaned up the .cache dirs on the builders 2022-01-29 15:35:03 clandmeter: could you clean up your container? clandmeter-edge-x86_64 73G :-) 2022-01-29 17:16:31 If you have time, could you also review https://gitlab.alpinelinux.org/alpine/infra/mirror-status/-/merge_requests/6 2022-01-31 18:42:07 clandmeter: another MR, this one is just an esthetic change 2022-01-31 18:42:09 https://gitlab.alpinelinux.org/alpine/infra/mirror-status/-/merge_requests/5