2020-12-01 08:45:48 So what do we still need to do for the gitlab 13.4 upgrade? 2020-12-01 08:45:56 Find out how to upgrade pg? 2020-12-01 08:58:44 clandmeter: what about https://www.postgresql.org/docs/12/pgupgrade.html 2020-12-01 09:01:25 You need to be able to run both at the same time, but that might not be an issue with docker 2020-12-01 09:03:35 I think pgupgrade works on same system only, not two 'boxes' 2020-12-01 09:05:14 with docker you can pretty easy link things together in a single container 2020-12-01 09:35:46 ikke: i think there is a docker project that supports upgrading like that. 2020-12-01 09:36:05 aha, was already working on something :P 2020-12-01 09:36:10 https://github.com/tianon/docker-postgres-upgrade 2020-12-01 09:36:24 nice 2020-12-01 09:38:32 My idea was to use docker-compose to automate this a bit more 2020-12-01 09:39:48 Not sure if it matters, but this project uses postgres: rather than postgres:-alpine 2020-12-01 10:13:14 ikke: i guess we need to push a new branch to gitlab docker repo and let it try to build? 2020-12-01 10:13:21 or build it local 2020-12-01 10:15:07 referring to gitlab or postgres? 2020-12-01 10:16:29 gitlab 2020-12-01 10:16:43 i used to build it locally 2020-12-01 10:17:03 ah, right 2020-12-01 10:20:56 hmm, aarch64 ci vm is awol 2020-12-01 10:21:12 oom 2020-12-01 10:22:12 started the VM again 2020-12-01 10:23:45 im building 13.4.6 local 2020-12-01 10:23:49 ok 2020-12-01 10:23:50 on windows :) 2020-12-01 10:23:53 :D 2020-12-01 10:24:19 it will take probably 2 hours 2020-12-01 10:33:56 ikke: i tried to run docker on my lxc 2020-12-01 10:34:07 me too 2020-12-01 10:34:11 but it looks like somebody upgraded system without a reboot 2020-12-01 10:34:19 ah 2020-12-01 10:34:38 so you are missing modules 2020-12-01 10:34:51 yes matching ones 2020-12-01 12:01:37 "Your Linode credit expires soon 2020-12-01 12:01:40 " 2020-12-01 12:01:45 I assume this is renewed for 2021? 2020-12-01 12:54:19 no idea 2020-12-01 12:54:28 its not obvious on linodes system 2020-12-01 12:55:19 LBlaboon: how is the credit arrange atm? and what happens if it really expires, lets say due to misconfig. Will our vps all stop working? 2020-12-01 13:31:14 ikke: we are having the __va_copy linkage issue again when building gitlab 2020-12-01 13:31:52 google protobuf 2020-12-01 13:32:02 im adding a workaround 2020-12-01 13:32:16 ugh, ok 2020-12-01 14:59:17 ikke: is that from an email? i just checked the alpine account and the current credit is valid through december 2021 2020-12-01 14:59:31 LBlaboon: yes, received an e-mail about it 2020-12-01 14:59:58 yea, i think that's an automatic email cause the previous credit would have expired at the end of this month 2020-12-01 15:00:11 right 2020-12-01 15:00:20 but the new credit is already in place, so there shouldn't be any problem 2020-12-01 15:00:28 Was expecting that, just wanted to verify 2020-12-01 15:00:47 thanks 2020-12-01 15:03:28 clandmeter: if your account credit were to expire (due to misconfig, or any other reason), then there would be a positive balance on the account which would need to be paid 2020-12-01 15:03:54 that sounds scary 2020-12-01 15:03:58 so basically your account would work just like any other account, where you have (X) number of days to pay the balance before we start suspending services (i forget what the exact policy is) 2020-12-01 15:04:16 i read about something on "hacker" "news" where some ruby on rails dev had that happen to him on digitalocean 2020-12-01 15:04:28 (fuck digitalocean in general, for many reasons but i digress) 2020-12-01 15:05:21 i remember reading that. i think that was a result of their fraud system mis-classifying them as fraud, and then it happened a second time 2020-12-01 15:06:05 anyway i would assume linode would immediately fix such an issue before anything got turned off 2020-12-01 15:07:23 yea, we have your account tagged appropriately, so anyone on our support team can see you are a sponsored account and they would reach out before suspending or cancelling anything 2020-12-01 15:20:33 LBlaboon: thanks for the clarification 2020-12-01 15:21:39 ikke: 13.4 is uploaded 2020-12-01 15:21:55 nice 2020-12-01 15:22:32 ikke: will you focus on db migration? 2020-12-01 15:22:39 yes 2020-12-01 15:22:47 nice 2020-12-01 15:23:01 teamwork :) 2020-12-02 10:44:50 ikke: did you make any progress on the pg migration? 2020-12-02 11:29:08 clandmeter: I've hit some issues, trying to figure out how to get it to work. Will try to get something tonight 2020-12-02 18:22:05 hmmm 2020-12-04 15:31:40 ikke: Seems like appstream.a.o is still 404'ing :/ 2020-12-04 15:34:37 oh 2020-12-04 15:35:35 the nginx container crashed 2020-12-04 15:35:39 or is stopped 2020-12-04 15:36:59 ah, restart was not defined 2020-12-04 15:37:03 so it was not started at boot 2020-12-04 15:38:27 It works again now 2020-12-04 15:43:18 Oh nice 2020-12-04 15:43:35 Thanks 2020-12-04 17:04:44 . 2020-12-07 15:32:24 seems like the 3.13 builders working on community repo now 2020-12-07 15:32:43 i created the v3.13/main directory so next run the packages should be uploaded to master mirror 2020-12-07 15:32:53 do we have enough disk space or will we get problems? 2020-12-07 15:33:38 185G available 2020-12-07 15:33:40 I don't know the diskspace situation on master 2020-12-07 15:34:50 looks like v3.12 takes 108G 2020-12-07 15:34:55 we will be full 2020-12-07 15:59:46 ikke: any luck with pg? 2020-12-07 16:00:04 not yey 2020-12-07 16:00:06 yet 2020-12-07 16:00:16 pg_upgrade keeps complaining about not having root access 2020-12-07 16:00:19 on the db 2020-12-07 16:00:41 root access? 2020-12-07 16:02:11 connection to database failed: FATAL: must be superuser to connect in binary upgrade mode 2020-12-07 16:05:50 superuser is postgres, not root 2020-12-07 16:07:05 though I think 'pg_dump.... | pg_restore ...' is more safe than pg_upgrade 2020-12-07 16:19:35 Hmm 2020-12-07 19:57:40 mps: would you just use the plain text format for pg_dump? 2020-12-07 19:58:09 ikke: usually yes 2020-12-07 19:58:15 ok 2020-12-07 19:59:06 how do you make sure things like permissions are backed up as well 2020-12-07 19:59:08 if need to transfer it over net or save space I usually add 'gzip' 2020-12-07 19:59:49 you mean filesystem permissions of DB server? 2020-12-07 20:00:39 no pg users 2020-12-07 20:01:35 aha, roles/users. you have to export them with pg_dump and they are imported 'automatically' 2020-12-07 20:01:37 So I want to backup one instance (pg11) and import it in a new instance 2020-12-07 20:02:04 pg_dumpall then is good choice 2020-12-07 20:02:15 aha 2020-12-07 20:03:45 or you can do this in more steps. pg_dumpall have option '--roles-only dump only roles, no databases or tablespaces' 2020-12-07 20:04:17 I'll first try it in one go 2020-12-07 20:05:38 will be back in 10-15 minutes 2020-12-07 20:41:05 ikke: I think you know but just in case, pg_dumpall dumps all databases in one postgresql installation (server) and pg_dump dumps particular databases 2020-12-07 20:41:31 right, we just have a single schema 2020-12-07 20:42:56 this is how I move db from one server to another machine 'busybox time pg_dumpall -h 127.0.0.1 -U postgres | gzip > pg.dumpall-20201125.gz' 2020-12-07 20:43:16 one example 2020-12-07 20:45:19 then on another server 'gunzip -c /pg.dumpall-20200914.gz | psql -U postgres -h 127.0.0.1 -d postgres' 2020-12-07 20:46:47 never had problems with this method 2020-12-07 21:08:36 imorting is taking a while 2020-12-07 21:09:32 how big is DB 2020-12-07 21:10:01 2G3G\ 2020-12-07 21:10:03 3G 2020-12-07 21:10:14 and yes, import takes more time then dump 2020-12-07 21:10:34 on linode? 2020-12-07 21:11:39 yes 2020-12-07 21:16:18 3GB is dump size, non gziped? 2020-12-07 21:17:00 no, that's the db size on disk 2020-12-07 21:17:07 aha 2020-12-07 21:20:14 last time i worked with 5GB db size on linode server it took me less than 5 minutes 2020-12-07 21:20:22 forgot exact time 2020-12-07 21:22:39 still running 2020-12-07 21:23:17 not sure if it's doing a lot 2020-12-07 21:23:22 hmm, more than 20 minutes? 2020-12-07 21:23:36 too much 2020-12-07 21:24:32 pg_restore -h /run/postgresql12/ -U postgres -f gitlab.sql 2020-12-07 21:28:10 ah, I thought you use psql 2020-12-07 21:28:53 oh, you mention pg_restore :P 2020-12-07 21:29:01 it seems to read from stdin 2020-12-07 21:29:07 so that's why it's waiting 2020-12-07 21:29:33 using psql now 2020-12-07 21:29:33 heh 2020-12-07 21:29:48 More is happening now :) 2020-12-07 21:30:23 pg_restore --help => '-f, --file=FILENAME output file name' 2020-12-07 21:30:48 yeah, noticed that 2020-12-07 21:31:25 hope it didn't destroyed your gitlab.sql 2020-12-07 21:31:51 apparently it didn't 2020-12-07 21:32:07 nice from 'him' 2020-12-07 21:32:33 seems to have worked 2020-12-07 21:33:24 good \o/ 2020-12-07 21:34:20 starting gitlab now with pg12 2020-12-07 21:34:24 (test instance) 2020-12-07 21:35:42 ok, gitlab is not happy :) 2020-12-07 21:39:42 doesn't like pg 12? 2020-12-07 21:40:32 It says the DB does not exist 2020-12-07 21:40:59 hmm, and you can connect with psql? 2020-12-07 21:42:41 yes 2020-12-07 21:42:50 Seems like the db name was renamed 2020-12-07 21:43:08 huh, oh 2020-12-07 21:45:55 trying to rename it, but the name remains the same 2020-12-07 21:46:17 interesting 2020-12-07 21:46:25 very one 2020-12-07 21:46:29 in the container where I imported it, the name is correct 2020-12-07 21:46:56 ah, my bad 2020-12-07 21:48:05 I adjusted the target mount path instead of the source 2020-12-07 21:48:47 heh, as usual 2020-12-07 21:49:13 most errors on administration are ours 2020-12-07 21:49:29 absolutely 2020-12-07 21:50:34 seems it's working ;) 2020-12-07 21:52:03 gitlab is no longer complaining about pg11 :) 2020-12-07 21:53:07 \o/ 2020-12-08 10:06:47 ikke: nice 2020-12-08 10:07:04 And the procedure is rather simple 2020-12-08 10:13:29 ikke: did you try to upgrade gitlab? 2020-12-08 10:13:47 no, not yet 2020-12-08 10:13:53 first focues on the db migration :) 2020-12-08 10:14:00 :) 2020-12-08 10:14:10 you said it was simple ;-) 2020-12-08 10:14:27 sure, after you know how :P 2020-12-08 10:17:07 it could be more simple :) 2020-12-08 10:19:27 'pg_dumpall | psql -h hostname -p port -U postgres', something like this, but you have to run two postgresql servers on different ports if you do this on one machine 2020-12-08 10:19:50 yes, so that's why I have a small docker-compose project that allows you to do that 2020-12-08 10:29:40 ikke: so its just running two instances and dump it to the other instance? 2020-12-08 10:29:49 yes 2020-12-08 10:30:00 it mounts the socket dir from both running containers 2020-12-08 10:30:13 and then it's basically what mps suggested 2020-12-08 10:33:53 socket or ip doesnt make a diff i guess? 2020-12-08 10:34:05 clandmeter: right 2020-12-08 10:34:19 maybe a few miliseconds :) 2020-12-08 10:34:52 clandmeter: should not matter, no 2020-12-08 10:35:09 But authentication via socket is easier :) 2020-12-08 17:43:43 clandmeter: btw, I created a simple python script that reads a yaml file and allows you quickly connect / switch between hosts using fzf. I use it for easily switching between different releases for each build arch 2020-12-08 18:33:30 clandmeter: still seeing stuck gitaly-ssh processes 2020-12-08 18:34:33 let's see if the upgrade is fixing it 2020-12-08 19:16:26 hi 2020-12-08 19:16:35 ah you checked before the upgrade 2020-12-08 19:17:01 yes, and production :) 2020-12-11 05:32:54 deu1 was rebooted 2020-12-11 05:39:55 clandmeter: for some reason, we need to set `net.ipv6.conf.eth0.accept_ra = 2` on deu1 after reboot, even though it's in /etc/sysctl.d and set in /etc/network/interfaces 2020-12-11 05:40:17 otherwise, ipv6 is not reachable 2020-12-11 05:52:34 ikke: be sure that "ipv6" module is added to /etc/modules or /modules-load.d/ like in last versions of Alpine 2020-12-11 05:53:20 for me that was the reason when ipv6 stuff in sysctl didnt work 2020-12-11 05:53:52 it's in /etc/modules 2020-12-11 05:54:11 twice even :P 2020-12-11 05:54:17 hehe 2020-12-11 05:54:41 maybe interface eth0 isnt ready yet before sysctl apply? 2020-12-11 05:56:05 https://tpaste.us/N84y 2020-12-11 05:58:33 I dont have luck with put stuff in that place... maybe better use "net.ipv6.conf.default.accept_ra = 2" and "net.ipv6.conf.all.accept_ra = 2" 2020-12-11 05:59:11 and change it for rest interfaces which dont need to be set to "2" 2020-12-11 05:59:57 I don't think it matters for the other interfaces 2020-12-11 06:00:38 so try with those two above in sysctl.d 2020-12-11 06:01:06 yea 2020-12-11 06:01:51 What usually disables this is when the interface gets set to forwarding 2020-12-11 06:01:58 so it might be docker that is interfering with this 2020-12-11 06:02:07 oh ye... 2020-12-11 06:03:10 I hate docker when touching too many things like iptables too... 2020-12-11 06:06:36 Hmm, everywhere it says just to set eth0.accept_ra = 2 2020-12-11 06:07:05 oh, but I think the trick is to set forwarding yourself so that docker doesn't touch it 2020-12-11 06:09:12 is still good to set default and all 2020-12-11 06:10:30 smarty docker :D 2020-12-11 06:11:02 docker wants to be a run-and-go sollution 2020-12-11 06:13:55 ye and after you are the one who have to predict what docker doing or will do to not interfere with system :\ 2020-12-11 06:16:27 Ok, let's see what happens on the next reboot 2020-12-11 06:17:23 ACTION fingers crossed 2020-12-11 06:22:54 MY-R: no dice 2020-12-11 06:23:51 damn :\ 2020-12-11 06:24:33 but the interesting thing is that there is in upv6 adres, but the default route is missing 2020-12-11 06:27:12 buuut accept_ra 2 is now set or no? 2020-12-11 06:30:19 maybe try with "net.ipv6.conf.all.forwarding = 0" and "net.ipv6.conf.default.forwarding = 0" ... 2020-12-11 06:32:20 setting it to 2 with forwarding 0 makes little sense 2020-12-11 06:32:33 because 2 is explicitly enable it when forwarding = 1 2020-12-11 06:40:59 wait, ye, so to be sure that accept_ra 2 is set need to be sure that forwarding is 1 (not zero ye) 2020-12-11 06:44:07 annoying because to check if all that working you have to reboot :D 2020-12-11 07:14:28 i pushed to https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite but https://alpinelinux.org does not seem to be updated 2020-12-11 07:15:04 maybe it will next time someone pused to aports? 2020-12-11 07:47:55 I feel this is an issue every time 2020-12-11 07:48:09 MY-R: yes, indeed 2020-12-11 07:48:48 MY-R: I guess I can spin up a test instance with a similar setup and see if I can fix it there 2020-12-11 07:53:11 ikke: dunno if that is relevant in your case but looks familiar: https://test-dockerrr.readthedocs.io/en/latest/userguide/networking/default_network/ipv6/ 2020-12-11 07:53:31 yes, I found that page 2020-12-11 08:02:25 morning 2020-12-11 08:02:58 ikke: something unsets it? 2020-12-11 08:03:38 clandmeter: I'm not sure if it's unset or something else is going on 2020-12-11 08:04:15 what is the value pre manual set? 2020-12-11 08:05:21 I did not check 😑 2020-12-11 08:38:37 ikke: i just tested with a test instance but docker does not touch accept_ra 2020-12-11 08:39:02 clandmeter: no, it does affect forwarding 2020-12-11 08:39:19 which in turn disables accept_ra when it's set to 1 2020-12-11 08:40:00 I mean, accept_ra does nothing when set to 1 and forwarding is set to 1 2020-12-11 08:40:24 But, I see that eth0 already has an ipv6 address, so it must have gotten an RA 2020-12-11 08:40:37 But the default route is missing 2020-12-11 08:43:19 ikke: maybe i dont follow correctly 2020-12-11 08:43:24 i see the route 2020-12-11 08:44:32 where? 2020-12-11 08:44:52 https://tpaste.us/Yod6 2020-12-11 08:45:07 right 2020-12-11 08:45:12 on deu1, that one is missing right after boot 2020-12-11 08:45:15 (default) 2020-12-11 08:47:18 clandmeter: https://tpaste.us/5VPx 2020-12-11 08:53:38 right 2020-12-11 08:53:59 ikke: try 172.105.95.177 2020-12-11 08:54:06 i added a docker container 2020-12-11 08:54:11 to auto start 2020-12-11 08:54:23 rebooted and it seems to be ok 2020-12-11 08:54:29 but i dont have forwarding enabled 2020-12-11 08:55:54 Doesn't docker do that? 2020-12-11 08:57:07 cat /proc/sys/net/ipv6/conf/eth0/forwarding 2020-12-11 08:57:07 0 2020-12-11 09:00:39 ok 2020-12-11 09:01:30 What happens when you expose some ports on a container? 2020-12-11 09:04:49 what do you expect? 2020-12-11 09:05:12 i dont even know anymore how docker run works :) 2020-12-11 09:15:44 not sure about that but if doecker uses iptables nat why it need to touch forwarding 2020-12-12 12:41:33 clandmeter: cpu usage on deu1 is back btw 2020-12-12 13:00:30 clandmeter: so I've added a dmvpn network, and now forwarding is enabled 2020-12-12 13:35:43 clandmeter: So now when I reboot, the default route for ipv6 is gone 2020-12-12 14:10:10 MY-R: So I can reproduce it now on a test server 2020-12-12 14:13:26 ikke: heh, so let the fun begin - reboot number 50 - accept_ra is still 1 and no default route :P 2020-12-12 14:15:36 so indeed, accept_ra = 1 2020-12-12 14:15:39 after reboo 2020-12-12 14:25:05 ikke: /lib/sysctl.d/00-alpine.conf 2020-12-12 14:26:05 yes? 2020-12-12 14:27:08 This is what I have now: https://tpaste.us/g6Od 2020-12-12 14:29:07 ahh thought something can be in conflict with ipv6 with stuff in 00-alpine.conf but dont see anything 2020-12-12 14:29:14 me niether 2020-12-12 14:29:17 neither 2020-12-12 14:30:21 Hmm, strange, if I do sysctl -p, accept_ra remains 1 2020-12-12 14:33:28 do from hand net.ipv4.conf.default.forwarding = 1 net.ipv4.conf.all.forwarding = 1 and same with ipv6 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.all.forwarding = 1 2020-12-12 14:33:43 well, by hand it works 2020-12-12 14:35:22 and then try add above and all rest to sysctl.d/99-something.conf 2020-12-12 14:35:36 let /etc/sysctl.conf be empty 2020-12-12 14:37:42 it works if I restart the sysctl service after boot 2020-12-12 14:37:58 so some service overwriting it 2020-12-12 14:39:00 iptables together with docker? or those services restart interface but then "all/default" in sysctl should deal with it 2020-12-12 14:39:03 weird 2020-12-12 14:43:06 passing --ip-forward=false to docker does not work :( 2020-12-12 14:43:41 ikke: did you try set accept_ra = 2, check it if is still 2 and then restart docker to see if docker doing something? 2020-12-12 14:45:09 still = 2 after docker restart 2020-12-12 14:45:51 is iptables service running too? 2020-12-12 14:46:09 there is no iptables service 2020-12-12 14:46:15 well 2020-12-12 14:46:28 not running at least 2020-12-12 14:47:04 well, docker doing own magic stuff so probably when creating own network bridges then accept ra is gone... 2020-12-12 14:47:13 nod 2020-12-12 14:53:27 https://forums.docker.com/t/docker-removes-host-ipv6-default-route/83238 2020-12-12 14:54:09 No solution though 2020-12-12 14:55:56 ye I was curious too about it and there is too many issues with it and few still open ones and many MR... 2020-12-12 14:57:16 so the only dirty one would be add accept_ra to /local.d/ ... 2020-12-12 15:04:18 testing that now 2020-12-12 15:08:37 Interesting 2020-12-12 15:08:47 local service was not enabled yet 2020-12-12 15:08:51 I removed the sysctl config 2020-12-12 15:08:56 now it works 2020-12-12 15:09:05 but forwarding=0 and accept_ra=1 2020-12-12 15:09:19 ye local isnt enabled by default 2020-12-12 15:09:22 right 2020-12-12 15:09:34 but it does mean that --ip-forward=false is effective 2020-12-12 15:09:38 ye 2020-12-12 15:10:26 but you need forward to open ports in containers? :) 2020-12-12 15:11:42 if docker is so good with adjusting forward to 1 why cant make setting to do accept_ra=2 :D 2020-12-12 15:16:42 funny, less than 1ms latency to google.com from that vm 2020-12-12 15:18:02 google got servers everywhere so :) 2020-12-12 15:18:33 ikke: sysctl -a |grep accept_ra_defrtr 2020-12-12 15:18:44 is 1 one interfaces? 2020-12-12 15:18:52 s/one/on 2020-12-12 15:18:52 MY-R meant to say: is 1 on interfaces? 2020-12-12 15:19:10 yes 2020-12-12 15:20:11 for sure is docker issue, enough if type "accept_ra docker" in google 2020-12-12 15:20:22 right 2020-12-12 15:20:27 but if I disable that function 2020-12-12 15:20:31 which it seems it does now 2020-12-12 15:20:41 and I set forwarding to 1, and accept_ra to 2 2020-12-12 15:20:45 I expect it to work 2020-12-12 15:21:54 agree 2020-12-12 15:23:15 except it doesn't 2020-12-12 15:23:43 forwarding = 1, but accept_ra remains 1 2020-12-12 15:23:44 some hardcoded magic when docker bring up own interfaces 2020-12-12 15:25:00 what doesn't help is that the lifetime of the default route is 0 sec 2020-12-12 15:25:07 so it's immediately gone 2020-12-12 15:25:17 omg 2020-12-12 15:26:40 docker should respect at least default.accept_ra/all.accept_ra but instead doing own voodoo 2020-12-12 15:28:17 oof 2020-12-12 15:28:32 setting it in local doesn't appear to work 2020-12-12 15:29:11 tried with sleep 10? :D 2020-12-12 15:29:47 I was thinking along the same lines 2020-12-12 15:29:58 But not sure if it would help 2020-12-12 15:30:33 ye cus rly hard to do anything else than start dig in docker code to patch it.... 2020-12-12 15:30:58 I mean, if things are executed serially, it just takes longer to boot 2020-12-12 15:31:06 probably after bring up container it would again reset to 1 .... 2020-12-12 15:32:03 local.d is executed as a last service in "default" 2020-12-12 15:34:03 I see, after '*' 2020-12-12 15:39:29 I can disable docker by default and see what happens then 2020-12-12 15:39:32 before and after starting it 2020-12-12 15:40:38 not sure if the local script is executed 2020-12-12 15:40:49 I think libnetwork which is part of docker doing that stuff with reseting sysctls 2020-12-12 15:40:52 I've added logging to it, but that's not appearing 2020-12-12 15:42:36 ikke: /etc/rc.conf and change uncomment and change to rc_logger="YES" 2020-12-12 15:42:50 then will have /var/log/rc.log 2020-12-12 15:43:26 ah you done it 2020-12-12 15:43:46 MY-R: well, custom logging, not that 2020-12-12 15:43:54 MY-R: even without docker, accept_ra is still 1 2020-12-12 15:44:37 ah 2020-12-12 15:44:47 so it's not docker 2020-12-12 15:47:05 * Configuring kernel parameters ... [ ok ] 2020-12-12 15:47:37 I would try remove that hook from /etc/network/interfaces, be sure that sysctl.conf is empty and create in sysctl.conf/blabla.conf with all those default/all/eth0 stuff accept_ra and forward 2020-12-12 15:48:16 that's already the case 2020-12-12 15:48:28 oh, 2020-12-12 15:48:34 the hook is still there 2020-12-12 15:48:56 🤦 2020-12-12 15:49:16 that would explain a lot 2020-12-12 15:49:27 I thought I removed it 2020-12-12 15:49:30 :] 2020-12-12 15:51:45 ugh 2020-12-12 15:51:49 linode is resetting it 2020-12-12 15:52:03 Need to disable auto configure networking 2020-12-12 15:53:15 heh 2020-12-12 15:55:59 ok, that's more like it 2020-12-12 15:57:10 after starting docker, accept_ra is still 2 2020-12-12 15:57:20 so it's linode's setup 2020-12-12 15:59:23 what exactly they do? 2020-12-12 16:00:08 echo 1 > accept_ra 2020-12-12 16:00:11 is it VM and they doing what want and overwriting things in guest? 2020-12-12 16:00:24 yes, they have auto configuration helpers 2020-12-12 16:00:29 ahh 2020-12-12 16:00:35 nice helpers 2020-12-12 16:00:58 one of the first things I did was to remove that line 2020-12-12 16:01:04 but didn't realize that it was overwritten 2020-12-12 16:01:10 yeee 2020-12-12 16:01:13 ikke: pre-up echo 1 > /proc/sys/net/ipv6/conf/eth0/accept_ra 2020-12-12 16:01:17 yes 2020-12-12 16:01:22 But that's not what we need 2020-12-12 16:01:24 we need it set to 2 2020-12-12 16:01:31 my /etc/network/.interfaces.linode-orig 2020-12-12 16:01:55 I changed it right after install 2020-12-12 16:02:11 Did you disable the network configuration helper? 2020-12-12 16:02:27 "Auto-configure networking" 2020-12-12 16:02:59 Im glad you now figured this out, and sorry Docker for "bad" words! :) 2020-12-12 16:03:00 yes 2020-12-12 16:03:10 ACTION knock-knock 2020-12-12 16:03:22 I set it to 'fixed' IP 2020-12-12 16:03:44 The configuration they do is also fixed 2020-12-12 16:03:54 and LBlaboon told earlier (here or in private, I forgot) that this is safe to do 2020-12-12 16:03:59 ok 2020-12-12 16:04:11 The only difference we need is to set accept_ra to 2 2020-12-12 16:04:25 by imo it's nicer to do that via /etc/sysctl.d 2020-12-12 16:04:56 much elegant and easier to find :) 2020-12-12 16:06:19 now I'm wondering how many people fought with this and blamed docker :D 2020-12-12 16:06:28 heh 2020-12-12 16:06:38 I saw at least one reference to linode on google 2020-12-12 16:07:04 MY-R: thanks for pairing on this with me :) 2020-12-12 16:08:16 no problem, from experience I know that writing about problems on chat or talk about it at loud can help :) 2020-12-12 16:08:22 now we don't lose ipv6 support on mirrors.a.o every time we reboot 2020-12-12 16:08:28 MY-R: yes, exactly 2020-12-12 16:10:59 yeah, more predictable setup :) gj! mps could help earlier! 2020-12-12 16:11:22 We need to work on our ipv6 support 2020-12-12 16:11:25 this helps 2020-12-12 16:14:14 MY-R: sorry, I'm busy and didn't read carefully 2020-12-12 16:17:12 I would love for iproute2 to have some option to automaticaly ignore all veth* interfaces 2020-12-12 16:18:56 ikke: afaik if something don't instruct iproute it will not touch veth 2020-12-12 16:19:22 I mean, when listing addresses, etc 2020-12-12 16:19:31 ah 2020-12-12 16:19:34 now, ip a will flood your terminal 2020-12-12 16:19:50 mps: no worry, is saturday! 2020-12-12 16:22:36 ip -c a type veth 2020-12-12 16:23:30 heh no !veth 2020-12-12 16:25:48 With a bit of luck, alpinelinux.org should be available on ipv6 soon 2020-12-12 16:26:02 nice 2020-12-13 07:09:52 MY-R: sadly, on the production host it did not work yet :( 2020-12-13 07:10:44 ikke: huh? accept_ra going back to 1? 2020-12-13 07:10:52 yes 2020-12-13 07:11:12 oh, I think because I have default in sysctl.d, not eth0 2020-12-13 07:11:43 :P 2020-12-13 07:12:20 see, enough to write about problem and solutions coming right after :) 2020-12-13 07:12:57 for some reason, that host does get rebooted a lot 2020-12-13 07:13:22 they already moved it to a complete different physical host 2020-12-13 07:17:53 maybe machine was already overloaded by some user or random ddos sponsored by crap bots which trying use every machine in their amplify attacks 2020-12-13 08:32:05 [09:23:28] heh no !veth 2020-12-13 08:32:12 i could add a filter for that 2020-12-13 08:32:34 we have iproute2 upstream people in our team (: 2020-12-13 08:34:34 interesting idea, didn't seriously thought about it 2020-12-13 08:34:53 but maybe it makes sense 2020-12-13 08:46:57 hmm 2020-12-13 08:47:11 Would be really nice to have 2020-12-13 08:47:16 IFLA_LINKKIND is null for real interfaces 2020-12-14 02:56:04 Somebody with GL admin rights, please move https://gitlab.alpinelinux.org/alpine/infra/infra-packages/-/issues/3 to aports. 2020-12-14 05:30:32 TBK[m] done 2020-12-14 05:30:46 ty 2020-12-14 13:07:07 ikke: ping 2020-12-14 13:07:14 pong 2020-12-14 13:07:26 whats up with gitlab? 2020-12-14 13:07:33 Was testing the migration again 2020-12-14 13:07:43 I guess you mean, gitlab-test or prod? 2020-12-14 13:07:49 test :) 2020-12-14 13:07:52 heh 2020-12-14 13:07:54 prod is fine right? 2020-12-14 13:07:57 yes 2020-12-14 13:08:01 as far as I know :P 2020-12-14 13:08:03 its not made by goole 2020-12-14 13:08:13 google* 2020-12-14 13:08:15 lol 2020-12-14 13:08:21 I migrated the data again 2020-12-14 13:08:27 with a script 2020-12-14 13:08:42 you mean db migrations? 2020-12-14 13:08:46 yes 2020-12-14 13:08:47 pg 2020-12-14 13:09:04 it was not ok? 2020-12-14 13:09:12 i remember you did it before 2020-12-14 13:09:15 yes 2020-12-14 13:09:26 so I tried to do the same thing again, but now automated 2020-12-14 13:09:46 right 2020-12-14 13:09:54 nice to have for next version 2020-12-14 13:09:55 ah, now it's working 2020-12-14 13:10:16 (I just recreated the containers) 2020-12-14 13:10:50 So the idea is that you stop the gitlab pg container 2020-12-14 13:10:57 then start the upgrader pg containers 2020-12-14 13:11:05 and then run the pgupgrade container 2020-12-14 13:11:20 After that is finished, you can stop them again 2020-12-14 13:11:26 and then point gitlab pg to the new directory 2020-12-14 13:11:39 (with a new version) 2020-12-14 13:12:26 hmm 2020-12-14 13:12:30 bit confusing 2020-12-14 13:12:43 check the gitlab-pg-upgrade dir in /src/copose 2020-12-14 13:12:50 you would have 2 containers, both with its own volume? 2020-12-14 13:13:10 one is old one is new version i guess 2020-12-14 13:13:14 yes 2020-12-14 13:13:21 you cross mount socket dir 2020-12-14 13:13:21 the containers expose the socket 2020-12-14 13:13:23 yes 2020-12-14 13:13:38 with new tools you export and import in the right socket 2020-12-14 13:13:38 to the upgrade container 2020-12-14 13:14:14 https://tpaste.us/zxMj 2020-12-14 13:14:15 so the new volume is your new db store, and you n eed to attach it to the new versioned container 2020-12-14 13:14:37 note that we use (rightfully so) host mounted volumes 2020-12-14 13:15:45 so currently it's mounted at /srv/docker/gitlab/postgres 2020-12-14 13:15:50 the last one i dont understand 2020-12-14 13:16:04 they are not docker volumes 2020-12-14 13:16:04 you mean bind mounts? 2020-12-14 13:16:18 right 2020-12-14 13:16:25 and why rightfully so? 2020-12-14 13:16:55 my experience is that you should not keep data in docker volumes that you do not want to loose 2020-12-14 13:17:58 named volumes are ok imho 2020-12-14 13:19:09 you have to explicitly delete them to get rid of them. 2020-12-14 13:19:24 docker-compose down -v 2020-12-14 13:20:03 thats pretty explicit 2020-12-14 13:20:20 or you would think it means verbose :D 2020-12-14 13:26:19 ikke: did you try to run the upgrade script already? 2020-12-14 13:26:27 Yes 2020-12-14 13:26:39 Pg upgrade right? 2020-12-14 13:26:46 no gitlab one 2020-12-14 13:27:08 Oh, no, that was my next step 2020-12-14 13:28:06 If you want, you can start that 2020-12-14 13:28:52 If all goes well, I wanted to do the upgrade tomorrow 2020-12-14 17:33:41 clandmeter: before starting gitlab, we need to run `CREATE EXTENSION IF NOT EXISTS btree_gist;` 2020-12-14 17:36:17 gitlab-test is upgraded :) 2020-12-14 17:36:22 only, we need to upgrade it again :D 2020-12-14 17:37:45 13.4.7 2020-12-14 20:33:09 ikke: nice 2020-12-14 20:33:15 yes its a security update 2020-12-14 20:35:03 I've alreadu pusjed 2020-12-14 20:35:05 pushed it 2020-12-14 22:38:52 ikke: that extension is a one timer right? 2020-12-14 22:39:13 clandmeter: Once it's enabled it remains enabled 2020-12-14 22:40:47 Yes I think there is another one 2020-12-14 22:40:59 We need to add it 2020-12-14 22:41:09 pg_trgm 2020-12-14 22:41:18 But I did not need to add that one 2020-12-14 22:41:42 Check setup script 2020-12-14 22:42:43 Does not mention any extensio 2020-12-14 22:43:29 Hmm 2020-12-14 22:43:57 ah, entrypoint 2020-12-14 22:44:08 create_db 2020-12-14 22:44:14 Ok 2020-12-14 22:44:30 but that's only the first time, I guess? 2020-12-14 22:44:54 yes 2020-12-15 09:54:36 LBlaboon: just a bit of feedback we run into with the linode images: By default the interface script sets accept_ra to 1, which breaks ipv6 slaac as soon as something (ie. docker) enables forwarding on the interfaces. Would it make sense to set the value to 2 so that RA's are still working? 2020-12-15 09:55:00 Another thing I noticed is that the slaac default route lifetime is 0 secs 2020-12-15 14:19:17 ikke: im getting an upgrade msg, is that for this evening? 2020-12-15 14:20:36 clandmeter: yes 2020-12-15 14:20:42 That was the plan 2020-12-15 14:20:47 nice 2020-12-15 14:21:11 first db? 2020-12-15 14:21:25 Yes, I would think so 2020-12-15 14:21:33 should not take too long 2020-12-15 14:22:17 will you include the logic in the repo? 2020-12-15 14:22:59 the compose file / dockerfile? 2020-12-15 14:23:05 nod 2020-12-15 14:23:32 yeah, either that, or a separate project 2020-12-15 14:23:56 maybe with another name so you need to specify it 2020-12-15 14:24:15 not sure it makes sense 2020-12-15 14:58:34 ikke: sure, we can update the template on our end to make that the default. we're gonna be pushing out a few related updates tomorrow so i can include that as part of it 2020-12-15 14:59:01 LBlaboon: nice, thanks 2020-12-15 18:13:17 clandmeter: https://tpaste.us/x1jl 2020-12-15 18:13:23 appeared during the upgrade 2020-12-15 18:18:16 clandmeter: but I think everything is working 2020-12-15 18:18:47 curl https://gitlab.alpinelinux.org/clandmeter.keys :) 2020-12-15 18:19:15 Looks good 2020-12-15 18:19:21 Nice work 2020-12-15 18:19:27 You as well :) 2020-12-15 18:19:54 I still have a copy of the pg11 db 2020-12-15 18:20:23 And we have a backup of the vps 2020-12-15 18:20:34 yes 2020-12-15 19:49:12 ikke: \o/ 2020-12-15 19:49:41 is your ansible code working again? 2020-12-15 19:50:01 https://gitlab.alpinelinux.org/alpine/infra/ansible/-/pipelines/60435 2020-12-15 19:50:31 ncie 2020-12-15 19:54:26 ikke: are you monitoring the fix for those processes? 2020-12-15 19:54:43 not really 2020-12-15 19:54:53 :) 2020-12-15 19:54:54 this one? https://bugs.ruby-lang.org/issues/17189 2020-12-15 19:55:05 no 2020-12-15 19:55:10 ok 2020-12-15 19:55:13 gitaly i think 2020-12-15 19:55:29 they had some fix for those hanging processes 2020-12-15 19:55:38 i think its in .4 2020-12-15 19:55:53 reminds me 2020-12-15 19:56:19 his one ?https://bugs.ruby-lang.org/issues/17189 2020-12-15 19:56:25 arg 2020-12-15 19:56:28 https://gitlab.com/gitlab-org/gitaly/-/issues/3194 2020-12-15 19:57:00 that one is in 13.6 2020-12-15 19:57:13 we now have vault support iirc 2020-12-15 19:57:39 I was looking into setting up vault 2020-12-15 19:57:55 (properly) 2020-12-15 20:01:05 Ariadne talked about a credential broker for cloud images, for which vault would be suitable solution 2020-12-15 20:08:22 weird 2020-12-15 20:08:28 what? 2020-12-15 20:08:31 i thought it was backported 2020-12-15 20:08:36 ah, that fix 2020-12-15 20:10:06 https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#create-a-matrix-of-jobs-using-a-simple-syntax 2020-12-15 20:12:43 https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#run-pipelines-in-the-parent-project-for-mrs-from-forks sad, not in core :( 2020-12-15 20:13:04 but, project access tokens are :)_ 2020-12-15 20:15:18 i dont know exactly what you mean with cred broker. 2020-12-15 20:17:11 "The resources relating to the the community AMI image effort would be transferred to the Infrastructure team. This primarily consists of a small application that acts as a credential broker." 2020-12-15 20:23:02 https://about.gitlab.com/releases/2020/09/22/gitlab-13-4-released/#lock-the-latest-job-artifact-to-prevent-deletion 2020-12-15 20:23:05 should be aware 2020-12-15 21:48:38 hmm, why is there a double // in buildlog urls from edge, but not from tagged releases? e.g. https://build.alpinelinux.org/buildlogs//build-edge-mips64/testing/cfssl/cfssl-1.5.0-r0.log 2020-12-15 21:53:35 TBK[m]: probably an extra slash in the url defined on the builder 2020-12-15 21:55:19 ikke: you are right, it is a mips64 builder specific thing. 2020-12-15 21:55:56 did not spot that at first glance 2020-12-15 21:57:02 TBK[m]: should be fixed now 2020-12-15 21:57:34 ikke: 👍️ 2020-12-16 12:39:12 LBlaboon: ping 2020-12-16 13:00:53 on 3.11 x86_64 builder: tar: linux-5.4/sound/soc/codecs/cs4271-i2c.c: Cannot write: No space left on device 2020-12-16 13:01:32 /var/cache/distfiles is full 2020-12-16 13:01:50 basically / is full 2020-12-16 13:01:56 yeah 2020-12-16 13:02:00 im looking at it now 2020-12-16 13:03:20 we should have distfiles on a separate machine 2020-12-16 13:04:05 What would upload the distfiles there? 2020-12-16 13:15:33 build-*-x86_64 2020-12-16 13:15:55 yes 2020-12-16 13:15:57 ok 2020-12-16 13:15:58 i mean 2020-12-16 13:16:07 But is there a mechanism already to do so? 2020-12-16 13:16:13 no 2020-12-16 13:16:18 right 2020-12-16 13:16:28 i wonder if we could use nfs or sometehing 2020-12-16 13:17:05 Not sure if nfs over the internet is a good idea 2020-12-16 13:17:20 (mostly referring to blocking syscalls) 2020-12-16 13:18:30 i deleted edge build logs 2020-12-16 15:15:07 Ariadne: pong? 2020-12-16 15:16:15 LBlaboon: a question about linode and distribution images. alpine is planning with 3.14 to produce official images, and ncopa mentioned linode might be a possible cloud provider we want to support with these images. but we're not sure if linode actually uses distribution-provided images or if you all just master your own in all cases. 2020-12-16 15:18:37 we typically master our own images, since our platform is a bit non-standard in that all of our disks are basically just plain filesystems without a partition table 2020-12-16 15:20:48 thats what i figured 2020-12-16 15:20:51 just wanted to be sure 2020-12-16 15:21:21 LBlaboon: we are talking about making official cloud images with our releases. is there anything we can do on our side to make it easier for you to build linode alpine images? 2020-12-16 15:21:27 but we're currently in the process of re-working the way we build distros, so if one of the available formats was a plain tarball of the root fs then we would be able to use that 2020-12-16 15:21:57 on our end images are essentially just gzip-compressed ext4 filesystems 2020-12-16 15:22:06 that we can provide 2020-12-16 15:22:07 :) 2020-12-16 15:22:14 we have minirootfs already 2020-12-16 15:22:25 yeah, we can just expand that a little bit 2020-12-16 15:22:30 to include virt kernel and so on 2020-12-16 15:22:30 but it lacks kernel and things 2020-12-16 15:22:50 cause it would be great to have alpine images in linode on day 1 2020-12-16 15:22:58 since linode is cloud of choice for most of us :P 2020-12-16 15:23:06 yeah 2020-12-16 15:23:27 AWS? i can't afford that 2020-12-16 15:23:40 especially since the "alpine" image on there is $2500 yearly ;) 2020-12-16 15:23:46 even as things currently are, we can certainly ensure day 1 support. it only takes us ~30-45 minutes to build and deploy an alpine image with our current process 2020-12-16 15:24:43 well, what would be cool is if we could ping you with a URI to a filesystem as part of the release process 2020-12-16 15:25:06 Ariadne: :D 2020-12-16 15:25:34 ok. good. adding official support for linode would not add any overhead at all 2020-12-16 15:26:35 like, if we could do that 2020-12-16 15:27:02 cut release -> build server makes the tar.gz -> ping linode -> linode automatically does the thing and publishes the image 2020-12-16 15:27:06 that would be sick 2020-12-16 15:27:28 because if that was in place, alpine could cut a release and people could be live in minutes 2020-12-16 15:27:51 which is what we're trying to enable in AWS and all those guys 2020-12-16 15:28:07 but we're broke FOSS developers and thus cannot afford those clouds 2020-12-16 15:28:09 ;0 2020-12-16 15:28:11 )* 2020-12-16 15:28:17 wow fuck this keyboard :) 2020-12-16 15:28:46 yea totally. we're not quite at the point yet where we'd be able to automatically trigger a build (but that's coming), but for now a ping on IRC should suffice 2020-12-16 15:30:11 LBlaboon: i tagged 3.12.3 release today ;) 2020-12-16 15:30:15 (that's tongue in cheek of course, but AWS does not give me personally any more value for money over linode, where i have like 10 VMs running) 2020-12-16 15:31:15 1GB $5/month throwaway VMs is what i want and linode excels at giving me that 2020-12-16 15:31:50 sweet. i'll add that to my to-do list today 2020-12-16 15:32:13 (and unlike vultr, the network is stable) 2020-12-16 15:37:52 Ariadne: +1 about AWS :) 2020-12-16 21:31:11 mcrute: welcome, and ping us any time 2020-12-16 21:42:54 mcrute: I've added you to a new 'cloud' group in gitlab. This is just a group being used as a team 2020-12-16 21:47:46 @ikke: thanks. do I have the ability to add other members? 2020-12-16 21:50:09 Not at the moment 2020-12-16 21:51:45 okay, I have a co-maintainer that'll be joining me with the cloud stuff 2020-12-16 21:52:19 tomalok is his GitLab username 2020-12-16 21:53:41 ikke: could you add me to alpine, devs, and infra as well? 2020-12-16 22:09:31 mcrute: why hurry, I usually wait till some of infra team decide to call me to join 2020-12-16 22:10:13 mps: oh we just had a chat about it and I've got some contributions over there as well related to the images :-) 2020-12-16 22:21:01 ikke: ncopa: just wanted to let you guys know that we pushed 3.12.3 a few hours ago and also updated our network helper templates with the accept_ra change, so that's all ready to go 2020-12-17 07:01:57 LBlaboon: awesome! thanks! 2020-12-17 22:11:14 ikke: ping 2020-12-17 22:11:38 pong 2020-12-17 22:12:54 did you add the uacme compose on test instnace? 2020-12-17 22:13:19 Yes 2020-12-17 22:13:37 ah its custom made 2020-12-17 22:13:42 not the one from gitlab 2020-12-17 22:13:47 nope 2020-12-17 22:13:52 was just trying something 2020-12-17 22:13:57 :) 2020-12-17 22:15:32 you know we have a general cron file to fetch the certs 2020-12-17 22:16:47 yes 2020-12-17 22:17:21 I just didn't want to use the wildcard certs on a test instance 2020-12-17 22:17:45 right, you mean the key 2020-12-17 22:18:09 nod 2020-12-17 22:18:33 traefik should be able to do it, but i have not tried it yet. 2020-12-17 22:19:55 ikke: do you mind if i turn off traefik on that host? 2020-12-17 22:20:06 I don't mind 2020-12-17 23:14:59 :/ 2020-12-17 23:17:10 seems like packet has issues? 2020-12-17 23:27:28 yup 2020-12-17 23:27:34 all in ewr 2020-12-18 05:40:47 https://status.equinixmetal.com/incidents/pfgmgy1fnjcp 2020-12-18 05:42:01 partial power outage 2020-12-18 07:16:53 what is the reason that we dont have mips64 CI? due to we didnt have gitlab-runner? 2020-12-18 07:17:53 and docker... 2020-12-18 07:18:04 i dont think we have official docker images for mips64 2020-12-18 07:40:36 missing go 2020-12-18 07:40:43 so yes, missing docker / gitlab-runner 2020-12-18 07:41:15 base images for mips64 would be usefull too 2020-12-18 07:41:17 yes 2020-12-18 07:42:30 ncopa: nld3 is full btw 2020-12-18 07:43:01 hm, that's more a clandmeter / me thing though 2020-12-18 07:43:23 It's the mirror that's hosted there 2020-12-18 07:50:47 and there is nld5 2020-12-18 07:56:21 ncopa: I saw that edge has all kinds of old iso's. Do we still want to keep them there? 2020-12-18 08:39:05 Some have more space 2020-12-18 08:39:26 Just need lvm fix 2020-12-18 10:04:41 nld3|5 have 100G extra now for mirror 2020-12-18 10:04:54 thanks 2020-12-18 10:05:21 ikke: there is a simple txt that has a 2 step action if it happens again. 2020-12-18 10:05:28 i think we still have some additional space 2020-12-18 10:06:03 clandmeter: can even be one step if you add -r to lvresize ;-) 2020-12-18 10:06:14 lvextend* 2020-12-18 10:06:32 it is a bit tricky 2020-12-18 10:06:45 i prefer to do it myself 2020-12-18 10:06:49 hehe :) 2020-12-18 10:06:54 in case tools are mising 2020-12-18 10:07:08 lvm will call resize2fs iirc 2020-12-18 10:08:01 yes 2020-12-18 10:10:22 clandmeter: maybe after 3.13 is released, we should make some effort on upgrading our infra 2020-12-18 10:11:13 as in alpine version? 2020-12-18 10:11:17 yes 2020-12-18 10:11:20 sure 2020-12-18 10:11:25 and record it in netbox 2020-12-18 10:11:27 Most is still running 3.10 2020-12-18 10:11:35 maybe we can also upgrade netbox as the same time 2020-12-18 10:11:39 its a bit behind 2020-12-18 10:11:42 yes, I had that in mind 2020-12-18 10:12:33 would be nice if we can move things into iac 2020-12-18 10:12:39 we can do it one step at a time. 2020-12-18 10:13:03 Yes, that's best 2020-12-18 10:13:13 trying to do everything at once is daunting :) 2020-12-18 10:13:32 yes, and its not needed. 2020-12-18 10:13:42 the thing is how to test things in iac 2020-12-18 10:13:57 its kind of challenging 2020-12-18 10:14:49 chef has kitchen, which is nice 2020-12-18 10:15:16 I think you can adopt kitchen also for other things, though 2020-12-18 10:15:35 but it remains a challenge to verify actual effects on production 2020-12-18 10:17:36 i am trying to convert some compose stuff into ansible, but it consumes time and i dont have an easy way to test it. 2020-12-18 10:17:52 and thats a problem when doing it on running infra 2020-12-18 10:18:53 does ansible have a dry-run? 2020-12-18 10:21:16 something like --check i guess 2020-12-18 11:08:43 i think we can delete old edge iso images 2020-12-18 12:32:05 clandmeter: playing a bit with terraform? 2020-12-18 13:27:22 ikke: big brother? :p 2020-12-18 13:27:47 linode is spamming me :P 2020-12-18 13:27:55 i was checking what gitlab supports 2020-12-18 13:28:51 looks like hashicorp is their favorite 2020-12-18 13:29:01 nod 2020-12-19 11:10:49 ssh can't access my armv7 lxc on usa4, while aarch64 there works fine 2020-12-19 11:11:29 let me check 2020-12-19 11:12:00 mps: it has no ip :) 2020-12-19 11:12:47 hmm, why? 2020-12-19 11:12:51 ip thieves 2020-12-19 11:12:59 /etc/init.d/networking was gone 2020-12-19 11:13:09 aha 2020-12-19 11:13:09 fixed now 2020-12-19 11:13:34 ifupdown-ng again in action? :P 2020-12-19 11:13:50 the removal of ifupdown-ng-openrc 2020-12-19 11:14:15 for some reason, that does not restore /etc/init.d/networking from openrc 2020-12-19 11:14:40 yes, known bug 2020-12-19 11:16:00 it overwrites it with replaces 2020-12-19 11:18:54 ikke: thanks 2020-12-20 07:20:30 clandmeter: can we destroy the gitlab-test instance? 2020-12-20 12:33:30 should we try upgrading it to 13.6? 2020-12-20 12:34:15 do we want a new snapshot? 2020-12-20 12:34:49 we could first test on this, and if it works do ones more? 2020-12-20 12:34:58 whatever you prefer 2020-12-20 12:35:12 I guess we first need to build 13.6, right? 2020-12-20 12:35:28 yup 2020-12-20 12:35:39 I was about to run a test 2020-12-20 12:35:42 ok 2020-12-20 12:43:07 clandmeter: I think I've now really disabled mails on gitlab-test 2020-12-20 12:43:11 I was still receiving them 2020-12-20 12:43:27 ok 2020-12-20 12:43:45 there is some project that can act as an smtp and queue them 2020-12-20 12:44:44 upstream changed nginx config it seems 2020-12-20 15:00:44 clandmeter: https://gitlab.com/gitlab-org/gitlab/-/issues/273558#note_441441155 2020-12-20 15:08:35 A workaround for the related issues form not rendering properly 2020-12-21 08:36:19 ikke: i think we need to start using the bundled git in gitlab. 2020-12-21 08:37:49 `due to version constraints? 2020-12-21 08:38:14 they kind of depend on the latest and greatest git features 2020-12-21 08:38:26 and they would like to support in tree patches.... 2020-12-21 08:39:11 You mean they want to be able to patch git? 2020-12-21 08:39:23 nod 2020-12-21 08:39:58 gitaly has a make git t arget 2020-12-21 08:40:49 13.6 wants git .29 2020-12-21 08:41:12 which we only have in 3.13 2020-12-21 08:41:15 maybe even more recent version 2020-12-21 08:41:27 not sure what 13.4 wants 2020-12-21 08:41:35 somewhere they made the switch 2020-12-21 08:42:46 # Make sure Git is version 2.24.0 or higher (recommended version is 2.28.0) 2020-12-21 08:42:53 so .24 is still ok 2020-12-21 08:54:46 ikke: did you have to manually create the btree_gist extension? 2020-12-21 08:55:46 yes, I think so 2020-12-21 08:55:48 I did not automate it 2020-12-21 08:56:01 how did you know y ou had to do so? 2020-12-21 08:56:29 migrations failed? 2020-12-21 08:57:10 yes 2020-12-21 08:59:12 so the current user does not have that permission 2020-12-21 08:59:25 correct 2020-12-21 08:59:34 the 'gitlab' user 2020-12-21 08:59:47 https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/install/postgresql_extensions.md#L69 2020-12-21 10:28:50 clandmeter: The other day I was trying to make ipv6 work for git.a.o, but because it's now a cname for deu1-dev1.a.o, I cannot point it to a different ipv6 address 2020-12-21 10:29:36 clandmeter: I guess we could change it to an A record instead of a CNAME 2020-12-21 10:29:36 hmm 2020-12-21 10:29:42 interesting :) 2020-12-21 10:29:54 ipv6 makes trouble ;-) 2020-12-21 10:30:16 more like ipv4 making it hard for ipv6 to function properly :) 2020-12-21 10:30:33 i know, but its still trouble :) 2020-12-21 10:30:50 im ok in chainging it to a record 2020-12-21 10:30:59 ok 2020-12-21 10:31:23 is there an issue attached to that request? 2020-12-21 12:55:58 clandmeter: maybe we want to manage our DNS records as code as well 2020-12-21 13:15:26 nod 2020-12-21 13:23:42 have played with Ansible to generate BIND zone files 2020-12-21 13:25:50 minimal: we use linode, so probably we'd integrate with their API 2020-12-21 13:27:40 ok, main thing to think about is the SOA serial - do you keep it typical date-type format with simple increments 2020-12-21 13:29:06 you dont have to manage it with linode 2020-12-21 13:29:39 khost -t soa alpinelinux.org ns1.linode.com. 2020-12-21 13:29:48 as using "infrastructure-as-code" if you've lots of potential changes any day and you're using date-format (plus 1/2 digit extra) serial you could run into problems 2020-12-21 13:30:08 We make like at most a couple of changes per month :P 2020-12-21 13:30:31 2021000024 2020-12-21 13:30:42 2021 2020-12-21 13:31:00 welcome to the future :-) 2020-12-21 13:40:23 https://www.linode.com/docs/api/domains 2020-12-21 13:46:24 ikke: seems they take care of serial management, don't see if mentioned in that doc 2020-12-21 13:47:22 Yes. You can just add records to the webif, they handle the zone files 2020-12-21 13:47:52 makes life too easy ;-) 2020-12-21 16:21:28 I recently realized the aconf repo was lost on the gitlab migration 2020-12-21 16:21:54 https://git.alpinelinux.org/aports/tree/main/aconf/APKBUILD#n7 2020-12-21 16:22:20 We still have everything 2020-12-21 16:22:49 should it be hosted on AL gitlab or shall I take it elsewhere? 2020-12-21 16:22:59 I think we can host it on AL 2020-12-21 16:23:02 gitlab 2020-12-21 16:26:55 Do you still have the repo? 2020-12-21 16:43:11 yes, I have a local copy 2020-12-21 16:48:13 kunkku: You could create a repo on gitlab and push it to there, then I can make it public 2020-12-21 16:48:56 as it was a user repo before, that's the equivalent 2020-12-21 16:49:56 what should the path be? 2020-12-21 16:50:13 /kunkku/aconf? 2020-12-21 16:50:53 would that be the only user repo? 2020-12-21 16:51:13 What do you mean? 2020-12-21 16:51:50 of form // ? 2020-12-21 16:52:48 The core alpine projects live under /alpine 2020-12-21 16:52:56 there are others who have projects hosted under their username 2020-12-21 16:54:02 are all of them private because I can't see them? 2020-12-21 16:56:07 You should be able to see them here: https://gitlab.alpinelinux.org 2020-12-21 16:56:53 I can see 72 repos, all starting with alpine/ 2020-12-21 16:57:12 I guess because you are redirected to /alpine/ ? 2020-12-21 16:57:46 https://gitlab.alpinelinux.org/explore 2020-12-21 16:58:57 no matter the filters, I can see 72 repos at most 2020-12-21 16:59:15 And that last link? 2020-12-21 17:01:25 ok, now I can see 2020-12-21 17:05:10 now it's there 2020-12-21 17:05:48 made it public 2020-12-21 17:08:17 thanks 2020-12-21 17:08:40 Note that gitlab exposes archives that you can use as source for apkbuilds 2020-12-22 10:40:12 ikke: ping 2020-12-22 10:40:15 pong 2020-12-22 10:40:23 damn you are fast 2020-12-22 10:40:27 haha :D 2020-12-22 10:40:28 is that a script? 2020-12-22 10:40:37 I was just looking at the alert here 2020-12-22 10:40:52 we had an email about a mirror not long ago 2020-12-22 10:41:04 or was i the only one? 2020-12-22 10:41:06 I handled all mirrors requests yesterday 2020-12-22 10:41:17 the one about the rebranding? 2020-12-22 10:41:22 yes 2020-12-22 10:42:50 Was handled 2020-12-22 10:43:01 https://mirrors.alpinelinux.org/#mirror29 2020-12-22 10:43:14 nice 2020-12-22 10:43:17 i was thinking 2020-12-22 10:43:23 should we make that public? 2020-12-22 10:43:35 is there really private info in that list? 2020-12-22 10:44:16 im asking as we could allow operators to create MR's 2020-12-22 10:45:20 I have no idea what is considered private 2020-12-22 10:46:01 the only private thing would be the contact window 2020-12-22 10:46:25 but i think 99 out of 100 those are public email addresses 2020-12-22 11:00:32 I sent an e-mail about the speglar mirror yesterday, and now it seems they pulled it (but the mirror is still available under a different address) 2020-12-22 11:27:04 aarch64 gitlab runner down, or just busy? 2020-12-22 11:28:10 just busy 2020-12-22 11:29:10 thx 2020-12-22 12:58:38 ikke: wwwtest does not work anymore? 2020-12-22 12:59:09 You tell me :) 2020-12-22 12:59:26 not managed by me :) 2020-12-22 12:59:42 I have the feeling it's not managed by anyone 2020-12-22 12:59:45 i dont think i have access, its on old-git 2020-12-22 12:59:49 aha 2020-12-22 13:00:54 I don't think there is a reason to keep it there 2020-12-22 13:01:37 why is it not working? 2020-12-22 13:01:43 container not started? 2020-12-22 13:02:31 nginx is giving a tiemout 2020-12-22 13:02:36 (the ipv6-test server is still running as well) 2020-12-22 13:02:42 i guess its maybe an ip that has changed? 2020-12-22 13:03:33 can you check the nginx container to see where it points to? 2020-12-22 13:03:43 right, nginx still points to an old address 2020-12-22 13:04:57 ok, working now 2020-12-22 13:15:25 :( 2020-12-22 13:15:32 www does not update automatically 2020-12-22 13:17:00 yeah, ncopa already mentioned that 2020-12-22 13:18:03 config looks correct 2020-12-22 13:26:16 no idea why it does not work 2020-12-22 13:46:52 clandmeter: restarted the build-3-13-x86_64 container, and now ns1 does no longer resolve it, but if I query nld9 directly, it resolves 2020-12-22 15:12:40 hmm, now it works again :( 2020-12-22 18:31:40 https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released 2020-12-22 18:32:03 Review requests 2020-12-22 18:32:17 Yes, I see, nice 2020-12-22 18:37:33 The docker proxy is also interesting 2020-12-25 10:24:12 clandmeter: fyi, I switched the records for git.a.o now to A/AAAA records 2020-12-25 12:20:53 👍 2020-12-25 13:26:17 clandmeter: with this version of gitlab it seems there are still hanging processes 2020-12-25 15:11:31 seems like the compose file documentation got removed from docs.docker.com and they just link to a github spec.md file 2020-12-25 18:22:45 ikke: v2 still online 2020-12-25 18:25:07 v2? 2020-12-25 18:25:11 doc? 2020-12-25 18:26:37 https://docs.docker.com/compose/compose-file/compose-file-v2/ 2020-12-27 13:32:09 what happened to #alpine? 2020-12-27 14:10:35 did something happen? 2020-12-27 14:11:36 it's cleared out. now bounces to ##namespace 2020-12-27 14:11:44 looks like someone at freenode booted it 2020-12-27 14:12:23 oh, maybe i'm just an idiot and it was changed to invite only 2020-12-27 14:12:57 but ... then it wouldn't be bouncing me 2020-12-27 19:03:03 \o/ 2020-12-28 13:16:27 ncopa: nld9-dev1 disk is full (x86_64 builder) 2020-12-28 14:09:19 ok ill clean up 2020-12-28 15:28:02 :( 2020-12-28 15:34:15 i have free'd up a few gigs 2020-12-28 15:34:26 deleted some old build logs, and old distfiles 2020-12-29 01:59:09 cornfeedhobo: the alpine e-mail client is at ##alpine, alpine linux is at #alpine-linux 2020-12-29 01:59:18 this is intended to reduce confusion 2020-12-29 06:37:02 I'm banned from alpine project https://gitlab.alpinelinux.org/alpine/aports/-/project_members ? 2020-12-29 06:37:27 :) 2020-12-29 06:40:00 https://gitlab.alpinelinux.org/groups/team/developers/-/group_members 2020-12-29 06:42:38 ah, not yet :P 2020-12-29 06:43:57 but little confusing because I'm not listed in previous url 2020-12-29 08:32:06 Ariadne: thanks. strange it didn't show up in /list 2020-12-29 08:33:41 cornfeedhobo: because this channels are set to private, to reduce spam 2020-12-29 08:33:54 ah 2020-12-29 08:34:27 There was a period last year where there was a lot of spam on freenode 2020-12-31 08:05:25 the 3.13 package are being uploaded on the master mirror. do we have enough space? 2020-12-31 08:05:55 103G free