2020-12-01 08:45:48 <ikke> So what do we still need to do for the gitlab 13.4 upgrade?
2020-12-01 08:45:56 <ikke> Find out how to upgrade pg?
2020-12-01 08:58:44 <ikke> clandmeter: what about https://www.postgresql.org/docs/12/pgupgrade.html
2020-12-01 09:01:25 <ikke> You need to be able to run both at the same time, but that might not be an issue with docker
2020-12-01 09:03:35 <mps> I think pgupgrade works on same system only, not two 'boxes'
2020-12-01 09:05:14 <ikke> with docker you can pretty easy link things together in a single container
2020-12-01 09:35:46 <clandmeter> ikke: i think there is a docker project that supports upgrading like that.
2020-12-01 09:36:05 <ikke> aha, was already working on something :P
2020-12-01 09:36:10 <clandmeter> https://github.com/tianon/docker-postgres-upgrade
2020-12-01 09:36:24 <ikke> nice
2020-12-01 09:38:32 <ikke> My idea was to use docker-compose to automate this a bit more
2020-12-01 09:39:48 <ikke> Not sure if it matters, but this project uses postgres:<version> rather than postgres:<version>-alpine
2020-12-01 10:13:14 <clandmeter> ikke: i guess we need to push a new branch to gitlab docker repo and let it try to build?
2020-12-01 10:13:21 <clandmeter> or build it local
2020-12-01 10:15:07 <ikke> referring to gitlab or postgres?
2020-12-01 10:16:29 <clandmeter> gitlab
2020-12-01 10:16:43 <clandmeter> i used to build it locally
2020-12-01 10:17:03 <ikke> ah, right
2020-12-01 10:20:56 <ikke> hmm, aarch64 ci vm is awol
2020-12-01 10:21:12 <ikke> oom
2020-12-01 10:22:12 <ikke> started the VM again
2020-12-01 10:23:45 <clandmeter> im building 13.4.6 local
2020-12-01 10:23:49 <ikke> ok
2020-12-01 10:23:50 <clandmeter> on windows :)
2020-12-01 10:23:53 <ikke> :D
2020-12-01 10:24:19 <clandmeter> it will take probably 2 hours
2020-12-01 10:33:56 <clandmeter> ikke: i tried to run docker on my lxc 
2020-12-01 10:34:07 <ikke> me too
2020-12-01 10:34:11 <clandmeter> but it looks like somebody upgraded system without a reboot
2020-12-01 10:34:19 <ikke> ah
2020-12-01 10:34:38 <ikke> so you are missing modules
2020-12-01 10:34:51 <clandmeter> yes matching ones 
2020-12-01 12:01:37 <ikke> "Your Linode credit expires soon
2020-12-01 12:01:40 <ikke> "
2020-12-01 12:01:45 <ikke> I assume this is renewed for 2021?
2020-12-01 12:54:19 <clandmeter> no idea
2020-12-01 12:54:28 <clandmeter> its not obvious on linodes system
2020-12-01 12:55:19 <clandmeter> LBlaboon: how is the credit arrange atm? and what happens if it really expires, lets say due to misconfig. Will our vps all stop working?
2020-12-01 13:31:14 <clandmeter> ikke: we are having the __va_copy linkage issue again when building gitlab
2020-12-01 13:31:52 <clandmeter> google protobuf
2020-12-01 13:32:02 <clandmeter> im adding a workaround
2020-12-01 13:32:16 <ikke> ugh, ok
2020-12-01 14:59:17 <LBlaboon> ikke: is that from an email? i just checked the alpine account and the current credit is valid through december 2021
2020-12-01 14:59:31 <ikke> LBlaboon: yes, received an e-mail about it
2020-12-01 14:59:58 <LBlaboon> yea, i think that's an automatic email cause the previous credit would have expired at the end of this month
2020-12-01 15:00:11 <ikke> right
2020-12-01 15:00:20 <LBlaboon> but the new credit is already in place, so there shouldn't be any problem
2020-12-01 15:00:28 <ikke> Was expecting that, just wanted to verify
2020-12-01 15:00:47 <ikke> thanks
2020-12-01 15:03:28 <LBlaboon> clandmeter: if your account credit were to expire (due to misconfig, or any other reason), then there would be a positive balance on the account which would need to be paid
2020-12-01 15:03:54 <Ariadne> that sounds scary
2020-12-01 15:03:58 <LBlaboon> so basically your account would work just like any other account, where you have (X) number of days to pay the balance before we start suspending services (i forget what the exact policy is)
2020-12-01 15:04:16 <Ariadne> i read about something on "hacker" "news" where some ruby on rails dev had that happen to him on digitalocean
2020-12-01 15:04:28 <Ariadne> (fuck digitalocean in general, for many reasons but i digress)
2020-12-01 15:05:21 <LBlaboon> i remember reading that. i think that was a result of their fraud system mis-classifying them as fraud, and then it happened a second time
2020-12-01 15:06:05 <Ariadne> anyway i would assume linode would immediately fix such an issue before anything got turned off
2020-12-01 15:07:23 <LBlaboon> yea, we have your account tagged appropriately, so anyone on our support team can see you are a sponsored account and they would reach out before suspending or cancelling anything
2020-12-01 15:20:33 <clandmeter> LBlaboon: thanks for the clarification
2020-12-01 15:21:39 <clandmeter> ikke: 13.4 is uploaded
2020-12-01 15:21:55 <ikke> nice
2020-12-01 15:22:32 <clandmeter> ikke: will you focus on db migration?
2020-12-01 15:22:39 <ikke> yes
2020-12-01 15:22:47 <clandmeter> nice
2020-12-01 15:23:01 <ikke> teamwork :)
2020-12-02 10:44:50 <clandmeter> ikke: did you make any progress on the pg migration?
2020-12-02 11:29:08 <ikke> clandmeter: I've hit some issues, trying to figure out how to get it to work. Will try to get something tonight
2020-12-02 18:22:05 <ikke> hmmm
2020-12-04 15:31:40 <Cogitri> ikke: Seems like appstream.a.o is still 404'ing :/
2020-12-04 15:34:37 <ikke> oh
2020-12-04 15:35:35 <ikke> the nginx container crashed
2020-12-04 15:35:39 <ikke> or is stopped
2020-12-04 15:36:59 <ikke> ah, restart was not defined
2020-12-04 15:37:03 <ikke> so it was not started at boot
2020-12-04 15:38:27 <ikke> It works again now
2020-12-04 15:43:18 <Cogitri> Oh nice
2020-12-04 15:43:35 <Cogitri> Thanks
2020-12-04 17:04:44 <maxice8> .
2020-12-07 15:32:24 <ncopa> seems like the 3.13 builders working on community repo now
2020-12-07 15:32:43 <ncopa> i created the v3.13/main directory so next run the packages should be uploaded to master mirror
2020-12-07 15:32:53 <ncopa> do we have enough disk space or will we get problems?
2020-12-07 15:33:38 <ncopa> 185G available
2020-12-07 15:33:40 <ikke> I don't know the diskspace situation on master
2020-12-07 15:34:50 <ncopa> looks like v3.12 takes 108G
2020-12-07 15:34:55 <ncopa> we will be full
2020-12-07 15:59:46 <clandmeter> ikke: any luck with pg?
2020-12-07 16:00:04 <ikke> not yey
2020-12-07 16:00:06 <ikke> yet
2020-12-07 16:00:16 <ikke> pg_upgrade keeps complaining about not having root access
2020-12-07 16:00:19 <ikke> on the db
2020-12-07 16:00:41 <clandmeter> root access?
2020-12-07 16:02:11 <ikke> connection to database failed: FATAL:  must be superuser to connect in binary upgrade mode
2020-12-07 16:05:50 <mps> superuser is postgres, not root
2020-12-07 16:07:05 <mps> though I think 'pg_dump.... | pg_restore ...' is more safe than pg_upgrade
2020-12-07 16:19:35 <ikke> Hmm
2020-12-07 19:57:40 <ikke> mps: would you just use the plain text format for pg_dump?
2020-12-07 19:58:09 <mps> ikke: usually yes
2020-12-07 19:58:15 <ikke> ok
2020-12-07 19:59:06 <ikke> how do you make sure things like permissions are backed up as well
2020-12-07 19:59:08 <mps> if need to transfer it over net or save space I usually add 'gzip'
2020-12-07 19:59:49 <mps> you mean filesystem permissions of DB server?
2020-12-07 20:00:39 <ikke> no pg users
2020-12-07 20:01:35 <mps> aha, roles/users. you have to export them with pg_dump and they are imported 'automatically'
2020-12-07 20:01:37 <ikke> So I want to backup one instance (pg11) and import it in a new instance
2020-12-07 20:02:04 <mps> pg_dumpall then is good choice
2020-12-07 20:02:15 <ikke> aha
2020-12-07 20:03:45 <mps> or you can do this in more steps. pg_dumpall have option '--roles-only             dump only roles, no databases or tablespaces'
2020-12-07 20:04:17 <ikke> I'll first try it in one go
2020-12-07 20:05:38 <mps> will be back in 10-15 minutes
2020-12-07 20:41:05 <mps> ikke: I think you know but just in case, pg_dumpall dumps all databases in one postgresql installation (server) and pg_dump dumps particular databases
2020-12-07 20:41:31 <ikke> right, we just have a single schema 
2020-12-07 20:42:56 <mps> this is how I move db from one server to another machine 'busybox time pg_dumpall -h 127.0.0.1 -U postgres | gzip > pg.dumpall-20201125.gz'
2020-12-07 20:43:16 <mps> one example
2020-12-07 20:45:19 <mps> then on another server 'gunzip -c /pg.dumpall-20200914.gz  | psql -U postgres -h 127.0.0.1 -d postgres'
2020-12-07 20:46:47 <mps> never had problems with this method
2020-12-07 21:08:36 <ikke> imorting is taking a while
2020-12-07 21:09:32 <mps> how big is DB
2020-12-07 21:10:01 <ikke> 2G3G\
2020-12-07 21:10:03 <ikke> 3G
2020-12-07 21:10:14 <mps> and yes, import takes more time then dump
2020-12-07 21:10:34 <mps> on linode?
2020-12-07 21:11:39 <ikke> yes
2020-12-07 21:16:18 <mps> 3GB is dump size, non gziped?
2020-12-07 21:17:00 <ikke> no, that's the db size on disk
2020-12-07 21:17:07 <mps> aha
2020-12-07 21:20:14 <mps> last time i worked with 5GB db size on linode server it took me less than 5 minutes
2020-12-07 21:20:22 <mps> forgot exact time
2020-12-07 21:22:39 <ikke> still running
2020-12-07 21:23:17 <ikke> not sure if it's doing a lot
2020-12-07 21:23:22 <mps> hmm, more than 20 minutes?
2020-12-07 21:23:36 <mps> too much
2020-12-07 21:24:32 <ikke> pg_restore -h /run/postgresql12/ -U postgres -f gitlab.sql
2020-12-07 21:28:10 <mps> ah, I thought you use psql
2020-12-07 21:28:53 <ikke> oh, you mention pg_restore :P
2020-12-07 21:29:01 <ikke> it seems to read from stdin 
2020-12-07 21:29:07 <ikke> so that's why it's waiting 
2020-12-07 21:29:33 <ikke> using psql now
2020-12-07 21:29:33 <mps> heh
2020-12-07 21:29:48 <ikke> More is happening now :)
2020-12-07 21:30:23 <mps> pg_restore --help => '-f, --file=FILENAME      output file name'
2020-12-07 21:30:48 <ikke> yeah, noticed that
2020-12-07 21:31:25 <mps> hope it didn't destroyed your gitlab.sql
2020-12-07 21:31:51 <ikke> apparently it didn't
2020-12-07 21:32:07 <mps> nice from 'him'
2020-12-07 21:32:33 <ikke> seems to have worked
2020-12-07 21:33:24 <mps> good \o/
2020-12-07 21:34:20 <ikke> starting gitlab now with pg12
2020-12-07 21:34:24 <ikke> (test instance)
2020-12-07 21:35:42 <ikke> ok, gitlab is not happy :)
2020-12-07 21:39:42 <mps> doesn't like pg 12?
2020-12-07 21:40:32 <ikke> It says the DB does not exist
2020-12-07 21:40:59 <mps> hmm, and you can connect with psql?
2020-12-07 21:42:41 <ikke> yes
2020-12-07 21:42:50 <ikke> Seems like the db name was renamed 
2020-12-07 21:43:08 <mps> huh, oh
2020-12-07 21:45:55 <ikke> trying to rename it, but the name remains the same
2020-12-07 21:46:17 <ikke> interesting
2020-12-07 21:46:25 <mps> very one
2020-12-07 21:46:29 <ikke> in the container where I imported it, the name is correct
2020-12-07 21:46:56 <ikke> ah, my bad
2020-12-07 21:48:05 <ikke> I adjusted the target mount path instead of the source
2020-12-07 21:48:47 <mps> heh, as usual
2020-12-07 21:49:13 <mps> most errors on administration are ours
2020-12-07 21:49:29 <ikke> absolutely
2020-12-07 21:50:34 <ikke> seems it's working ;)
2020-12-07 21:52:03 <ikke> gitlab is no longer complaining about pg11 :)
2020-12-07 21:53:07 <mps> \o/
2020-12-08 10:06:47 <clandmeter> ikke: nice
2020-12-08 10:07:04 <ikke> And the procedure is rather simple
2020-12-08 10:13:29 <clandmeter> ikke: did you try to upgrade gitlab?
2020-12-08 10:13:47 <ikke> no, not yet
2020-12-08 10:13:53 <ikke> first focues on the db migration :)
2020-12-08 10:14:00 <clandmeter> :)
2020-12-08 10:14:10 <clandmeter> you said it was simple ;-)
2020-12-08 10:14:27 <ikke> sure, after you know how :P
2020-12-08 10:17:07 <mps> it could be more simple :)
2020-12-08 10:19:27 <mps> 'pg_dumpall | psql -h hostname -p port -U postgres', something like this, but you have to run two postgresql servers on different ports if you do this on one machine
2020-12-08 10:19:50 <ikke> yes, so that's why I have a small docker-compose project that allows you to do that
2020-12-08 10:29:40 <clandmeter> ikke: so its just running two instances  and dump it to the other instance?
2020-12-08 10:29:49 <ikke> yes
2020-12-08 10:30:00 <ikke> it mounts the socket dir from both running containers
2020-12-08 10:30:13 <ikke> and then it's basically what mps suggested
2020-12-08 10:33:53 <clandmeter> socket or ip doesnt make a diff i guess?
2020-12-08 10:34:05 <mps> clandmeter: right
2020-12-08 10:34:19 <clandmeter> maybe a few miliseconds :)
2020-12-08 10:34:52 <ikke> clandmeter: should not matter, no
2020-12-08 10:35:09 <ikke> But authentication via socket is easier :)
2020-12-08 17:43:43 <ikke> clandmeter: btw, I created a simple python script that reads a yaml file and allows you quickly connect / switch between hosts using fzf. I use it for easily switching between different releases for each build arch
2020-12-08 18:33:30 <ikke> clandmeter: still seeing stuck gitaly-ssh processes
2020-12-08 18:34:33 <ikke> let's see if the upgrade is fixing it
2020-12-08 19:16:26 <clandmeter> hi
2020-12-08 19:16:35 <clandmeter> ah you checked before the upgrade
2020-12-08 19:17:01 <ikke> yes, and production :)
2020-12-11 05:32:54 <ikke> deu1 was rebooted
2020-12-11 05:39:55 <ikke> clandmeter: for some reason, we need to set `net.ipv6.conf.eth0.accept_ra = 2` on deu1 after reboot, even though it's in /etc/sysctl.d and set in /etc/network/interfaces
2020-12-11 05:40:17 <ikke> otherwise, ipv6 is not reachable
2020-12-11 05:52:34 <MY-R> ikke: be sure that "ipv6" module is added to /etc/modules or /modules-load.d/ like in last versions of Alpine
2020-12-11 05:53:20 <MY-R> for me that was the reason when ipv6 stuff in sysctl didnt work
2020-12-11 05:53:52 <ikke> it's in /etc/modules
2020-12-11 05:54:11 <ikke> twice even :P
2020-12-11 05:54:17 <MY-R> hehe
2020-12-11 05:54:41 <MY-R> maybe interface eth0 isnt ready yet before sysctl apply?
2020-12-11 05:56:05 <ikke> https://tpaste.us/N84y
2020-12-11 05:58:33 <MY-R> I dont have luck with put stuff in that place... maybe better use "net.ipv6.conf.default.accept_ra = 2" and "net.ipv6.conf.all.accept_ra = 2"
2020-12-11 05:59:11 <MY-R> and change it for rest interfaces which dont need to be set to "2"
2020-12-11 05:59:57 <ikke> I don't think it matters for the other interfaces
2020-12-11 06:00:38 <MY-R> so try with those two above in sysctl.d
2020-12-11 06:01:06 <ikke> yea
2020-12-11 06:01:51 <ikke> What usually disables this is when the interface gets set to forwarding
2020-12-11 06:01:58 <ikke> so it might be docker that is interfering with this
2020-12-11 06:02:07 <MY-R> oh ye...
2020-12-11 06:03:10 <MY-R> I hate docker when touching too many things like iptables too...
2020-12-11 06:06:36 <ikke> Hmm, everywhere it says just to set eth0.accept_ra = 2
2020-12-11 06:07:05 <ikke> oh, but I think the trick is to set forwarding yourself so that docker doesn't touch it
2020-12-11 06:09:12 <MY-R> is still good to set default and all 
2020-12-11 06:10:30 <MY-R> smarty docker :D
2020-12-11 06:11:02 <ikke> docker wants to be a run-and-go sollution
2020-12-11 06:13:55 <MY-R> ye and after you are the one who have to predict what docker doing or will do to not interfere with system :\
2020-12-11 06:16:27 <ikke> Ok, let's see what happens on the next reboot
2020-12-11 06:17:23 <MY-R> 
ACTION fingers crossed

2020-12-11 06:22:54 <ikke> MY-R: no dice
2020-12-11 06:23:51 <MY-R> damn :\
2020-12-11 06:24:33 <ikke> but the interesting thing is that there is in upv6 adres, but the default route is missing
2020-12-11 06:27:12 <MY-R> buuut accept_ra 2 is now set or no?
2020-12-11 06:30:19 <MY-R> maybe try with "net.ipv6.conf.all.forwarding = 0" and "net.ipv6.conf.default.forwarding = 0" ...
2020-12-11 06:32:20 <ikke> setting it to 2 with forwarding 0 makes little sense
2020-12-11 06:32:33 <ikke> because 2 is explicitly enable it when forwarding = 1
2020-12-11 06:40:59 <MY-R> wait, ye, so to be sure that accept_ra 2 is set need to be sure that forwarding is 1 (not zero ye)
2020-12-11 06:44:07 <MY-R> annoying because to check if all that working you have to reboot :D
2020-12-11 07:14:28 <ncopa> i pushed to https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite but https://alpinelinux.org does not seem to be updated
2020-12-11 07:15:04 <ncopa> maybe it will next time someone pused to aports?
2020-12-11 07:47:55 <ikke> I feel this is an issue every time
2020-12-11 07:48:09 <ikke> MY-R: yes, indeed
2020-12-11 07:48:48 <ikke> MY-R: I guess I can spin up a test instance with a similar setup and see if I can fix it there
2020-12-11 07:53:11 <MY-R> ikke: dunno if that is relevant in your case but looks familiar: https://test-dockerrr.readthedocs.io/en/latest/userguide/networking/default_network/ipv6/
2020-12-11 07:53:31 <ikke> yes, I found that page
2020-12-11 08:02:25 <clandmeter> morning
2020-12-11 08:02:58 <clandmeter> ikke: something unsets it?
2020-12-11 08:03:38 <ikke> clandmeter: I'm not sure if it's unset or something else is going on
2020-12-11 08:04:15 <clandmeter> what is the value pre manual set?
2020-12-11 08:05:21 <ikke> I did not check 😑
2020-12-11 08:38:37 <clandmeter> ikke: i just tested with a test instance but docker does not touch accept_ra
2020-12-11 08:39:02 <ikke> clandmeter: no, it does affect forwarding
2020-12-11 08:39:19 <ikke> which in turn disables accept_ra when it's set to 1
2020-12-11 08:40:00 <ikke> I mean, accept_ra does nothing when set to 1 and forwarding is set to 1
2020-12-11 08:40:24 <ikke> But, I see that eth0 already has an ipv6 address, so it must have gotten an RA
2020-12-11 08:40:37 <ikke> But the default route is missing
2020-12-11 08:43:19 <clandmeter> ikke: maybe i dont follow correctly
2020-12-11 08:43:24 <clandmeter> i see the route
2020-12-11 08:44:32 <ikke> where?
2020-12-11 08:44:52 <clandmeter> https://tpaste.us/Yod6
2020-12-11 08:45:07 <ikke> right
2020-12-11 08:45:12 <ikke> on deu1, that one is missing right after boot
2020-12-11 08:45:15 <ikke> (default)
2020-12-11 08:47:18 <ikke> clandmeter: https://tpaste.us/5VPx 
2020-12-11 08:53:38 <clandmeter> right
2020-12-11 08:53:59 <clandmeter> ikke: try 172.105.95.177
2020-12-11 08:54:06 <clandmeter> i added a docker container
2020-12-11 08:54:11 <clandmeter> to auto start
2020-12-11 08:54:23 <clandmeter> rebooted and it seems to be ok
2020-12-11 08:54:29 <clandmeter> but i dont have forwarding enabled
2020-12-11 08:55:54 <ikke> Doesn't docker do that?
2020-12-11 08:57:07 <clandmeter> cat /proc/sys/net/ipv6/conf/eth0/forwarding
2020-12-11 08:57:07 <clandmeter> 0
2020-12-11 09:00:39 <ikke> ok
2020-12-11 09:01:30 <ikke> What happens when you expose some ports on a container?
2020-12-11 09:04:49 <clandmeter> what do you expect?
2020-12-11 09:05:12 <clandmeter> i dont even know anymore how docker run works :)
2020-12-11 09:15:44 <mps> not sure about that but if doecker uses iptables nat why it need to touch forwarding
2020-12-12 12:41:33 <ikke> clandmeter: cpu usage on deu1 is back btw
2020-12-12 13:00:30 <ikke> clandmeter: so I've added a dmvpn network, and now forwarding is enabled
2020-12-12 13:35:43 <ikke> clandmeter: So now when I reboot, the default route for ipv6 is gone
2020-12-12 14:10:10 <ikke> MY-R: So I can reproduce it now on a test server
2020-12-12 14:13:26 <MY-R> ikke: heh, so let the fun begin - reboot number 50 - accept_ra is still 1 and no default route :P
2020-12-12 14:15:36 <ikke> so indeed, accept_ra = 1
2020-12-12 14:15:39 <ikke> after reboo
2020-12-12 14:25:05 <MY-R> ikke: /lib/sysctl.d/00-alpine.conf 
2020-12-12 14:26:05 <ikke> yes?
2020-12-12 14:27:08 <ikke> This is what I have now: https://tpaste.us/g6Od
2020-12-12 14:29:07 <MY-R> ahh thought something can be in conflict with ipv6 with stuff in 00-alpine.conf but dont see anything
2020-12-12 14:29:14 <ikke> me niether
2020-12-12 14:29:17 <ikke> neither
2020-12-12 14:30:21 <ikke> Hmm, strange, if I do sysctl -p, accept_ra remains 1
2020-12-12 14:33:28 <MY-R> do from hand net.ipv4.conf.default.forwarding = 1 net.ipv4.conf.all.forwarding = 1 and same with ipv6 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.all.forwarding = 1
2020-12-12 14:33:43 <ikke> well, by hand it works
2020-12-12 14:35:22 <MY-R> and then try add above and all rest to sysctl.d/99-something.conf
2020-12-12 14:35:36 <MY-R> let /etc/sysctl.conf be empty
2020-12-12 14:37:42 <ikke> it works if I restart the sysctl service after boot
2020-12-12 14:37:58 <MY-R> so some service overwriting it
2020-12-12 14:39:00 <MY-R> iptables together with docker? or those services restart interface but then "all/default" in sysctl should deal with it
2020-12-12 14:39:03 <MY-R> weird
2020-12-12 14:43:06 <ikke> passing --ip-forward=false to docker does not work :(
2020-12-12 14:43:41 <MY-R> ikke: did you try set accept_ra = 2, check it if is still 2 and then restart docker to see if docker doing something?
2020-12-12 14:45:09 <ikke> still = 2 after docker restart
2020-12-12 14:45:51 <MY-R> is iptables service running too?
2020-12-12 14:46:09 <ikke> there is no iptables service
2020-12-12 14:46:15 <ikke> well
2020-12-12 14:46:28 <ikke> not running at least
2020-12-12 14:47:04 <MY-R> well, docker doing own magic stuff so probably when creating own network bridges then accept ra is gone...
2020-12-12 14:47:13 <ikke> nod
2020-12-12 14:53:27 <ikke> https://forums.docker.com/t/docker-removes-host-ipv6-default-route/83238
2020-12-12 14:54:09 <ikke> No solution though
2020-12-12 14:55:56 <MY-R> ye I was curious too about it and there is too many issues with it and few still open ones and many MR...
2020-12-12 14:57:16 <MY-R> so the only dirty one would be add accept_ra to /local.d/ ...
2020-12-12 15:04:18 <ikke> testing that now
2020-12-12 15:08:37 <ikke> Interesting
2020-12-12 15:08:47 <ikke> local service was not enabled yet
2020-12-12 15:08:51 <ikke> I removed the sysctl config
2020-12-12 15:08:56 <ikke> now it works
2020-12-12 15:09:05 <ikke> but forwarding=0 and accept_ra=1
2020-12-12 15:09:19 <MY-R> ye local isnt enabled by default
2020-12-12 15:09:22 <ikke> right
2020-12-12 15:09:34 <ikke> but it does mean that --ip-forward=false is effective
2020-12-12 15:09:38 <MY-R> ye
2020-12-12 15:10:26 <MY-R> but you need forward to open ports in containers? :)
2020-12-12 15:11:42 <MY-R> if docker is so good with adjusting forward to 1 why cant make setting to do accept_ra=2 :D
2020-12-12 15:16:42 <ikke> funny, less than 1ms latency to google.com from that vm
2020-12-12 15:18:02 <MY-R> google got servers everywhere so :)
2020-12-12 15:18:33 <MY-R> ikke: sysctl -a |grep accept_ra_defrtr
2020-12-12 15:18:44 <MY-R> is 1 one interfaces?
2020-12-12 15:18:52 <MY-R> s/one/on
2020-12-12 15:18:52 <alpine-meetbot> MY-R meant to say: is 1 on interfaces?
2020-12-12 15:19:10 <ikke> yes
2020-12-12 15:20:11 <MY-R> for sure is docker issue, enough if type "accept_ra docker" in google
2020-12-12 15:20:22 <ikke> right
2020-12-12 15:20:27 <ikke> but if I disable that function
2020-12-12 15:20:31 <ikke> which it seems it does now
2020-12-12 15:20:41 <ikke> and I set forwarding to 1, and accept_ra to 2
2020-12-12 15:20:45 <ikke> I expect it to work
2020-12-12 15:21:54 <MY-R> agree
2020-12-12 15:23:15 <ikke> except it doesn't
2020-12-12 15:23:43 <ikke> forwarding = 1, but accept_ra remains 1
2020-12-12 15:23:44 <MY-R> some hardcoded magic when docker bring up own interfaces
2020-12-12 15:25:00 <ikke> what doesn't help is that the lifetime of the default route is 0 sec
2020-12-12 15:25:07 <ikke> so it's immediately gone
2020-12-12 15:25:17 <MY-R> omg
2020-12-12 15:26:40 <MY-R> docker should respect at least default.accept_ra/all.accept_ra but instead doing own voodoo
2020-12-12 15:28:17 <ikke> oof
2020-12-12 15:28:32 <ikke> setting it in local doesn't appear to work
2020-12-12 15:29:11 <MY-R> tried with sleep 10? :D
2020-12-12 15:29:47 <ikke> I was thinking along the same lines
2020-12-12 15:29:58 <ikke> But not sure if it would help
2020-12-12 15:30:33 <MY-R> ye cus rly hard to do anything else than start dig in docker code to patch it....
2020-12-12 15:30:58 <ikke> I mean, if things are executed serially, it just takes longer to boot
2020-12-12 15:31:06 <MY-R> probably after bring up container it would again reset to 1 ....
2020-12-12 15:32:03 <MY-R> local.d is executed as a last service in "default"
2020-12-12 15:34:03 <ikke> I see, after '*'
2020-12-12 15:39:29 <ikke> I can disable docker by default and see what happens then
2020-12-12 15:39:32 <ikke> before and after starting it
2020-12-12 15:40:38 <ikke> not sure if the local script is executed
2020-12-12 15:40:49 <MY-R> I think libnetwork which is part of docker doing that stuff with reseting sysctls
2020-12-12 15:40:52 <ikke> I've added logging to it, but that's not appearing
2020-12-12 15:42:36 <MY-R> ikke: /etc/rc.conf and change uncomment and change to rc_logger="YES"
2020-12-12 15:42:50 <MY-R> then will have /var/log/rc.log
2020-12-12 15:43:26 <MY-R> ah you done it
2020-12-12 15:43:46 <ikke> MY-R: well, custom logging, not that
2020-12-12 15:43:54 <ikke> MY-R: even without docker, accept_ra is still 1
2020-12-12 15:44:37 <MY-R> ah
2020-12-12 15:44:47 <ikke> so it's not docker
2020-12-12 15:47:05 <ikke> * Configuring kernel parameters ... [ ok ]
2020-12-12 15:47:37 <MY-R> I would try remove that hook from /etc/network/interfaces, be sure that sysctl.conf is empty and create in sysctl.conf/blabla.conf with all those default/all/eth0 stuff accept_ra and forward
2020-12-12 15:48:16 <ikke> that's already the case
2020-12-12 15:48:28 <ikke> oh,
2020-12-12 15:48:34 <ikke> the hook is still there
2020-12-12 15:48:56 <ikke> 🤦
2020-12-12 15:49:16 <ikke> that would explain a lot
2020-12-12 15:49:27 <ikke> I thought I removed it
2020-12-12 15:49:30 <MY-R> :]
2020-12-12 15:51:45 <ikke> ugh
2020-12-12 15:51:49 <ikke> linode is resetting it
2020-12-12 15:52:03 <ikke> Need to disable auto configure networking
2020-12-12 15:53:15 <MY-R> heh
2020-12-12 15:55:59 <ikke> ok, that's more like it
2020-12-12 15:57:10 <ikke> after starting docker, accept_ra is still 2
2020-12-12 15:57:20 <ikke> so it's linode's setup
2020-12-12 15:59:23 <MY-R> what exactly they do?
2020-12-12 16:00:08 <ikke> echo 1 > accept_ra
2020-12-12 16:00:11 <MY-R> is it VM and they doing what want and overwriting things in guest?
2020-12-12 16:00:24 <ikke> yes, they have auto configuration helpers
2020-12-12 16:00:29 <MY-R> ahh
2020-12-12 16:00:35 <MY-R> nice helpers
2020-12-12 16:00:58 <ikke> one of the first things I did was to remove that line
2020-12-12 16:01:04 <ikke> but didn't realize that it was overwritten
2020-12-12 16:01:10 <MY-R> yeee
2020-12-12 16:01:13 <mps> ikke: pre-up echo 1 > /proc/sys/net/ipv6/conf/eth0/accept_ra
2020-12-12 16:01:17 <ikke> yes
2020-12-12 16:01:22 <ikke> But that's not what we need
2020-12-12 16:01:24 <ikke> we need it set to 2
2020-12-12 16:01:31 <mps> my /etc/network/.interfaces.linode-orig
2020-12-12 16:01:55 <mps> I changed it right after install
2020-12-12 16:02:11 <ikke> Did you disable the network configuration helper?
2020-12-12 16:02:27 <ikke> "Auto-configure networking"
2020-12-12 16:02:59 <MY-R> Im glad you now figured this out, and sorry Docker for "bad" words! :)
2020-12-12 16:03:00 <mps> yes
2020-12-12 16:03:10 <MY-R> 
ACTION knock-knock

2020-12-12 16:03:22 <mps> I set it to 'fixed' IP
2020-12-12 16:03:44 <ikke> The configuration they do is also fixed
2020-12-12 16:03:54 <mps> and LBlaboon told earlier (here or in private, I forgot) that this is safe to do
2020-12-12 16:03:59 <ikke> ok
2020-12-12 16:04:11 <ikke> The only difference we need is to set accept_ra to 2
2020-12-12 16:04:25 <ikke> by imo it's nicer to do that via /etc/sysctl.d
2020-12-12 16:04:56 <MY-R> much elegant and easier to find :)
2020-12-12 16:06:19 <MY-R> now I'm wondering how many people fought with this and blamed docker :D
2020-12-12 16:06:28 <ikke> heh
2020-12-12 16:06:38 <ikke> I saw at least one reference to linode on google
2020-12-12 16:07:04 <ikke> MY-R: thanks for pairing on this with me :)
2020-12-12 16:08:16 <MY-R> no problem, from experience I know that writing about problems on chat or talk about it at loud can help :)
2020-12-12 16:08:22 <ikke> now we don't lose ipv6 support on mirrors.a.o every time we reboot
2020-12-12 16:08:28 <ikke> MY-R: yes, exactly
2020-12-12 16:10:59 <MY-R> yeah, more predictable setup :) gj! mps could help earlier!
2020-12-12 16:11:22 <ikke> We need to work on our ipv6 support
2020-12-12 16:11:25 <ikke> this helps
2020-12-12 16:14:14 <mps> MY-R: sorry, I'm busy and didn't read carefully
2020-12-12 16:17:12 <ikke> I would love for iproute2 to have some option to automaticaly ignore all veth* interfaces
2020-12-12 16:18:56 <mps> ikke: afaik if something don't instruct iproute it will not touch veth
2020-12-12 16:19:22 <ikke> I mean, when listing addresses, etc
2020-12-12 16:19:31 <mps> ah
2020-12-12 16:19:34 <ikke> now, ip a will flood your terminal
2020-12-12 16:19:50 <MY-R> mps: no worry, is saturday!
2020-12-12 16:22:36 <mps> ip -c a type veth
2020-12-12 16:23:30 <mps> heh no !veth
2020-12-12 16:25:48 <ikke> With a bit of luck, alpinelinux.org should be available on ipv6 soon
2020-12-12 16:26:02 <maxice8> nice
2020-12-13 07:09:52 <ikke> MY-R: sadly, on the production host it did not work yet :(
2020-12-13 07:10:44 <MY-R> ikke: huh? accept_ra going back to 1?
2020-12-13 07:10:52 <ikke> yes
2020-12-13 07:11:12 <ikke> oh, I think because I have default in sysctl.d, not eth0
2020-12-13 07:11:43 <MY-R> :P
2020-12-13 07:12:20 <MY-R> see, enough to write about problem and solutions coming right after :)
2020-12-13 07:12:57 <ikke> for some reason, that host does get rebooted a lot
2020-12-13 07:13:22 <ikke> they already moved it to a complete different physical host
2020-12-13 07:17:53 <MY-R> maybe machine was already overloaded by some user or random ddos sponsored by crap bots which trying use every machine in their amplify attacks
2020-12-13 08:32:05 <Ariadne> [09:23:28] <mps> heh no !veth
2020-12-13 08:32:12 <Ariadne> i could add a filter for that
2020-12-13 08:32:34 <Ariadne> we have iproute2 upstream people in our team (:
2020-12-13 08:34:34 <mps> interesting idea, didn't seriously thought about it
2020-12-13 08:34:53 <mps> but maybe it makes sense
2020-12-13 08:46:57 <Ariadne> hmm
2020-12-13 08:47:11 <ikke> Would be really nice to have 
2020-12-13 08:47:16 <Ariadne> IFLA_LINKKIND is null for real interfaces
2020-12-14 02:56:04 <TBK[m]> Somebody with GL admin rights, please move https://gitlab.alpinelinux.org/alpine/infra/infra-packages/-/issues/3 to aports.
2020-12-14 05:30:32 <ikke>  TBK[m] done
2020-12-14 05:30:46 <TBK[m]> ty
2020-12-14 13:07:07 <clandmeter> ikke: ping
2020-12-14 13:07:14 <ikke> pong
2020-12-14 13:07:26 <clandmeter> whats up with gitlab?
2020-12-14 13:07:33 <ikke> Was testing the migration again
2020-12-14 13:07:43 <ikke> I guess you mean, gitlab-test or prod?
2020-12-14 13:07:49 <clandmeter> test :)
2020-12-14 13:07:52 <ikke> heh
2020-12-14 13:07:54 <clandmeter> prod is fine right?
2020-12-14 13:07:57 <ikke> yes
2020-12-14 13:08:01 <ikke> as far as I know :P
2020-12-14 13:08:03 <clandmeter> its not made by goole
2020-12-14 13:08:13 <clandmeter> google*
2020-12-14 13:08:15 <ikke> lol
2020-12-14 13:08:21 <ikke> I migrated the data again
2020-12-14 13:08:27 <ikke> with a script
2020-12-14 13:08:42 <clandmeter> you mean db migrations?
2020-12-14 13:08:46 <ikke> yes
2020-12-14 13:08:47 <clandmeter> pg
2020-12-14 13:09:04 <clandmeter> it was not ok?
2020-12-14 13:09:12 <clandmeter> i remember you did it before
2020-12-14 13:09:15 <ikke> yes
2020-12-14 13:09:26 <ikke> so I tried to do the same thing again, but now automated
2020-12-14 13:09:46 <clandmeter> right
2020-12-14 13:09:54 <clandmeter> nice to have for next version
2020-12-14 13:09:55 <ikke> ah, now it's working
2020-12-14 13:10:16 <ikke> (I just recreated the containers)
2020-12-14 13:10:50 <ikke> So the idea is that you stop the gitlab pg container
2020-12-14 13:10:57 <ikke> then start the upgrader pg containers
2020-12-14 13:11:05 <ikke> and then run the pgupgrade container
2020-12-14 13:11:20 <ikke> After that is finished, you can stop them again
2020-12-14 13:11:26 <ikke> and then point gitlab pg to the new directory
2020-12-14 13:11:39 <ikke> (with a new version)
2020-12-14 13:12:26 <clandmeter> hmm
2020-12-14 13:12:30 <clandmeter> bit confusing
2020-12-14 13:12:43 <ikke> check the gitlab-pg-upgrade dir in /src/copose
2020-12-14 13:12:50 <clandmeter> you would have 2 containers, both with its own volume?
2020-12-14 13:13:10 <clandmeter> one is old one is new version i guess
2020-12-14 13:13:14 <ikke> yes
2020-12-14 13:13:21 <clandmeter> you cross mount socket dir
2020-12-14 13:13:21 <ikke> the containers expose the socket
2020-12-14 13:13:23 <ikke> yes
2020-12-14 13:13:38 <clandmeter> with new tools you export and import in the right socket
2020-12-14 13:13:38 <ikke> to the upgrade container
2020-12-14 13:14:14 <ikke> https://tpaste.us/zxMj
2020-12-14 13:14:15 <clandmeter> so the new volume is your new db store, and you n eed to attach it to the new versioned container
2020-12-14 13:14:37 <ikke> note that we use (rightfully so) host mounted volumes
2020-12-14 13:15:45 <ikke> so currently it's mounted at /srv/docker/gitlab/postgres
2020-12-14 13:15:50 <clandmeter> the last one i dont understand
2020-12-14 13:16:04 <ikke> they are not docker volumes
2020-12-14 13:16:04 <clandmeter> you mean bind mounts?
2020-12-14 13:16:18 <clandmeter> right
2020-12-14 13:16:25 <clandmeter> and why rightfully so?
2020-12-14 13:16:55 <ikke> my experience is that you should not keep data in docker volumes that you do not want to loose
2020-12-14 13:17:58 <clandmeter> named volumes are ok imho
2020-12-14 13:19:09 <clandmeter> you have to explicitly delete them to get rid of them.
2020-12-14 13:19:24 <ikke> docker-compose down -v
2020-12-14 13:20:03 <clandmeter> thats pretty explicit 
2020-12-14 13:20:20 <clandmeter> or you would think it means verbose :D
2020-12-14 13:26:19 <clandmeter> ikke: did you try to run the upgrade script already?
2020-12-14 13:26:27 <ikke> Yes
2020-12-14 13:26:39 <ikke> Pg upgrade right? 
2020-12-14 13:26:46 <clandmeter> no gitlab one
2020-12-14 13:27:08 <ikke> Oh, no, that was my next step 
2020-12-14 13:28:06 <ikke> If you want, you can start that
2020-12-14 13:28:52 <ikke> If all goes well, I wanted to do the upgrade tomorrow
2020-12-14 17:33:41 <ikke> clandmeter: before starting gitlab, we need to run `CREATE EXTENSION IF NOT EXISTS btree_gist;`
2020-12-14 17:36:17 <ikke> gitlab-test is upgraded :)
2020-12-14 17:36:22 <ikke> only, we need to upgrade it again :D
2020-12-14 17:37:45 <ikke> 13.4.7
2020-12-14 20:33:09 <clandmeter> ikke: nice
2020-12-14 20:33:15 <clandmeter> yes its a security update
2020-12-14 20:35:03 <ikke> I've alreadu pusjed
2020-12-14 20:35:05 <ikke> pushed it
2020-12-14 22:38:52 <clandmeter> ikke: that extension is a one timer right?
2020-12-14 22:39:13 <ikke> clandmeter: Once it's enabled it remains enabled
2020-12-14 22:40:47 <clandmeter> Yes I think there is another one
2020-12-14 22:40:59 <clandmeter> We need to add it 
2020-12-14 22:41:09 <ikke> pg_trgm
2020-12-14 22:41:18 <ikke> But I did not need to add that one
2020-12-14 22:41:42 <clandmeter> Check setup script
2020-12-14 22:42:43 <ikke> Does not mention any extensio
2020-12-14 22:43:29 <clandmeter> Hmm
2020-12-14 22:43:57 <ikke> ah, entrypoint
2020-12-14 22:44:08 <ikke> create_db
2020-12-14 22:44:14 <clandmeter> Ok
2020-12-14 22:44:30 <ikke> but that's only the first time, I guess?
2020-12-14 22:44:54 <ikke> yes
2020-12-15 09:54:36 <ikke> LBlaboon: just a bit of feedback we run into with the linode images: By default the interface script sets accept_ra to 1, which breaks ipv6 slaac as soon as something (ie. docker) enables forwarding on the interfaces. Would it make sense to set the value to 2 so that RA's are still working?
2020-12-15 09:55:00 <ikke> Another thing I noticed is that the slaac default route lifetime is 0 secs
2020-12-15 14:19:17 <clandmeter> ikke: im getting an upgrade msg, is that for this evening?
2020-12-15 14:20:36 <ikke> clandmeter: yes
2020-12-15 14:20:42 <ikke> That was the plan
2020-12-15 14:20:47 <clandmeter> nice
2020-12-15 14:21:11 <clandmeter> first db?
2020-12-15 14:21:25 <ikke> Yes, I would think so
2020-12-15 14:21:33 <ikke> should not take too long
2020-12-15 14:22:17 <clandmeter> will you include the logic in the repo?
2020-12-15 14:22:59 <ikke> the compose file / dockerfile?
2020-12-15 14:23:05 <clandmeter> nod
2020-12-15 14:23:32 <ikke> yeah, either that, or a separate project
2020-12-15 14:23:56 <clandmeter> maybe with another name so you need to specify it 
2020-12-15 14:24:15 <clandmeter> not sure it makes sense
2020-12-15 14:58:34 <LBlaboon> ikke: sure, we can update the template on our end to make that the default. we're gonna be pushing out a few related updates tomorrow so i can include that as part of it
2020-12-15 14:59:01 <ikke> LBlaboon: nice, thanks
2020-12-15 18:13:17 <ikke> clandmeter: https://tpaste.us/x1jl 
2020-12-15 18:13:23 <ikke> appeared during the upgrade
2020-12-15 18:18:16 <ikke> clandmeter: but I think everything is working
2020-12-15 18:18:47 <ikke> curl https://gitlab.alpinelinux.org/clandmeter.keys  :)
2020-12-15 18:19:15 <clandmeter> Looks good
2020-12-15 18:19:21 <clandmeter> Nice work
2020-12-15 18:19:27 <ikke> You as well :)
2020-12-15 18:19:54 <ikke> I still have a copy of the pg11 db
2020-12-15 18:20:23 <clandmeter> And we have a backup of the vps
2020-12-15 18:20:34 <ikke> yes
2020-12-15 19:49:12 <clandmeter> ikke: \o/
2020-12-15 19:49:41 <ikke> is your ansible code working again?
2020-12-15 19:50:01 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/ansible/-/pipelines/60435
2020-12-15 19:50:31 <ikke> ncie
2020-12-15 19:54:26 <clandmeter> ikke: are you monitoring the fix for those processes?
2020-12-15 19:54:43 <ikke> not really
2020-12-15 19:54:53 <clandmeter> :)
2020-12-15 19:54:54 <ikke> this one? https://bugs.ruby-lang.org/issues/17189
2020-12-15 19:55:05 <clandmeter> no
2020-12-15 19:55:10 <ikke> ok
2020-12-15 19:55:13 <clandmeter> gitaly i think
2020-12-15 19:55:29 <clandmeter> they had some fix for those hanging processes
2020-12-15 19:55:38 <clandmeter> i think its in .4
2020-12-15 19:55:53 <clandmeter> reminds me
2020-12-15 19:56:19 <ikke> his one ?https://bugs.ruby-lang.org/issues/17189
2020-12-15 19:56:25 <ikke> arg
2020-12-15 19:56:28 <ikke> https://gitlab.com/gitlab-org/gitaly/-/issues/3194
2020-12-15 19:57:00 <ikke> that one is in 13.6
2020-12-15 19:57:13 <clandmeter> we now have vault support iirc
2020-12-15 19:57:39 <ikke> I was looking into setting up vault
2020-12-15 19:57:55 <ikke> (properly)
2020-12-15 20:01:05 <ikke> Ariadne talked about a credential broker for cloud images, for which vault would be suitable solution
2020-12-15 20:08:22 <clandmeter> weird
2020-12-15 20:08:28 <ikke> what?
2020-12-15 20:08:31 <clandmeter> i thought it was backported
2020-12-15 20:08:36 <ikke> ah, that fix
2020-12-15 20:10:06 <ikke> https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#create-a-matrix-of-jobs-using-a-simple-syntax
2020-12-15 20:12:43 <ikke> https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#run-pipelines-in-the-parent-project-for-mrs-from-forks sad, not in core :(
2020-12-15 20:13:04 <ikke> but, project access tokens are :)_
2020-12-15 20:15:18 <clandmeter> i dont know exactly what you mean with cred broker.
2020-12-15 20:17:11 <ikke> "The resources relating to the the community AMI image effort would be transferred to the Infrastructure team.  This primarily consists of a small application that acts as a credential broker."
2020-12-15 20:23:02 <ikke> https://about.gitlab.com/releases/2020/09/22/gitlab-13-4-released/#lock-the-latest-job-artifact-to-prevent-deletion
2020-12-15 20:23:05 <ikke> should be aware
2020-12-15 21:48:38 <TBK[m]> hmm, why is there a double // in buildlog urls from edge, but not from tagged releases? e.g. https://build.alpinelinux.org/buildlogs//build-edge-mips64/testing/cfssl/cfssl-1.5.0-r0.log
2020-12-15 21:53:35 <ikke> TBK[m]: probably an extra slash in the url defined on the builder
2020-12-15 21:55:19 <TBK[m]> ikke: you are right, it is a mips64 builder specific thing.
2020-12-15 21:55:56 <TBK[m]> did not spot that at first glance
2020-12-15 21:57:02 <ikke> TBK[m]: should be fixed now
2020-12-15 21:57:34 <TBK[m]> ikke: 👍️
2020-12-16 12:39:12 <Ariadne> LBlaboon: ping
2020-12-16 13:00:53 <ncopa> on 3.11 x86_64 builder: tar: linux-5.4/sound/soc/codecs/cs4271-i2c.c: Cannot write: No space left on device
2020-12-16 13:01:32 <ikke> /var/cache/distfiles is full
2020-12-16 13:01:50 <ikke> basically / is full
2020-12-16 13:01:56 <ncopa> yeah
2020-12-16 13:02:00 <ncopa> im looking at it now
2020-12-16 13:03:20 <ncopa> we should have distfiles on a separate machine
2020-12-16 13:04:05 <ikke> What would upload the distfiles there?
2020-12-16 13:15:33 <ncopa> build-*-x86_64
2020-12-16 13:15:55 <ikke> yes
2020-12-16 13:15:57 <ikke> ok
2020-12-16 13:15:58 <ncopa> i mean
2020-12-16 13:16:07 <ikke> But is there a mechanism already to do so?
2020-12-16 13:16:13 <ncopa> no
2020-12-16 13:16:18 <ikke> right
2020-12-16 13:16:28 <ncopa> i wonder if we could use nfs or sometehing
2020-12-16 13:17:05 <ikke> Not sure if nfs over the internet is a good idea
2020-12-16 13:17:20 <ikke> (mostly referring to blocking syscalls)
2020-12-16 13:18:30 <ncopa> i deleted edge build logs
2020-12-16 15:15:07 <LBlaboon> Ariadne: pong?
2020-12-16 15:16:15 <Ariadne> LBlaboon: a question about linode and distribution images.  alpine is planning with 3.14 to produce official images, and ncopa mentioned linode might be a possible cloud provider we want to support with these images.  but we're not sure if linode actually uses distribution-provided images or if you all just master your own in all cases.
2020-12-16 15:18:37 <LBlaboon> we typically master our own images, since our platform is a bit non-standard in that all of our disks are basically just plain filesystems without a partition table
2020-12-16 15:20:48 <Ariadne> thats what i figured
2020-12-16 15:20:51 <Ariadne> just wanted to be sure
2020-12-16 15:21:21 <ncopa> LBlaboon: we are talking about making official cloud images with our releases. is there anything we can do on our side to make it easier for you to build linode alpine images?
2020-12-16 15:21:27 <LBlaboon> but we're currently in the process of re-working the way we build distros, so if one of the available formats was a plain tarball of the root fs then we would be able to use that
2020-12-16 15:21:57 <LBlaboon> on our end images are essentially just gzip-compressed ext4 filesystems
2020-12-16 15:22:06 <Ariadne> that we can provide
2020-12-16 15:22:07 <Ariadne> :)
2020-12-16 15:22:14 <ncopa> we have minirootfs already
2020-12-16 15:22:25 <Ariadne> yeah, we can just expand that a little bit
2020-12-16 15:22:30 <Ariadne> to include virt kernel and so on
2020-12-16 15:22:30 <ncopa> but it lacks kernel and things
2020-12-16 15:22:50 <Ariadne> cause it would be great to have alpine images in linode on day 1
2020-12-16 15:22:58 <Ariadne> since linode is cloud of choice for most of us :P
2020-12-16 15:23:06 <ncopa> yeah
2020-12-16 15:23:27 <Ariadne> AWS?  i can't afford that
2020-12-16 15:23:40 <Ariadne> especially since the "alpine" image on there is $2500 yearly ;)
2020-12-16 15:23:46 <LBlaboon> even as things currently are, we can certainly ensure day 1 support. it only takes us ~30-45 minutes to build and deploy an alpine image with our current process
2020-12-16 15:24:43 <Ariadne> well, what would be cool is if we could ping you with a URI to a filesystem as part of the release process
2020-12-16 15:25:06 <ikke> Ariadne: :D
2020-12-16 15:25:34 <ncopa> ok. good. adding official support for linode would not add any overhead at all
2020-12-16 15:26:35 <Ariadne> like, if we could do that
2020-12-16 15:27:02 <Ariadne> cut release -> build server makes the tar.gz -> ping linode -> linode automatically does the thing and publishes the image
2020-12-16 15:27:06 <Ariadne> that would be sick
2020-12-16 15:27:28 <Ariadne> because if that was in place, alpine could cut a release and people could be live in minutes
2020-12-16 15:27:51 <Ariadne> which is what we're trying to enable in AWS and all those guys
2020-12-16 15:28:07 <Ariadne> but we're broke FOSS developers and thus cannot afford those clouds
2020-12-16 15:28:09 <Ariadne> ;0
2020-12-16 15:28:11 <Ariadne> )*
2020-12-16 15:28:17 <Ariadne> wow fuck this keyboard :)
2020-12-16 15:28:46 <LBlaboon> yea totally. we're not quite at the point yet where we'd be able to automatically trigger a build (but that's coming), but for now a ping on IRC should suffice
2020-12-16 15:30:11 <ncopa> LBlaboon: i tagged 3.12.3 release today ;)
2020-12-16 15:30:15 <Ariadne> (that's tongue in cheek of course, but AWS does not give me personally any more value for money over linode, where i have like 10 VMs running)
2020-12-16 15:31:15 <Ariadne> 1GB $5/month throwaway VMs is what i want and linode excels at giving me that
2020-12-16 15:31:50 <LBlaboon> sweet. i'll add that to my to-do list today
2020-12-16 15:32:13 <Ariadne> (and unlike vultr, the network is stable)
2020-12-16 15:37:52 <mps> Ariadne: +1 about AWS :)
2020-12-16 21:31:11 <clandmeter> mcrute: welcome, and ping us any time
2020-12-16 21:42:54 <ikke> mcrute: I've added you to a new 'cloud' group in gitlab. This is just a group being used as a team
2020-12-16 21:47:46 <mcrute> @ikke: thanks. do I have the ability to add other members?
2020-12-16 21:50:09 <ikke> Not at the moment
2020-12-16 21:51:45 <mcrute> okay, I have a co-maintainer that'll be joining me with the cloud stuff
2020-12-16 21:52:19 <mcrute> tomalok is his GitLab username
2020-12-16 21:53:41 <mcrute> ikke: could you add me to alpine, devs, and infra as well?
2020-12-16 22:09:31 <mps> mcrute: why hurry, I usually wait till some of infra team decide to call me to join
2020-12-16 22:10:13 <mcrute> mps: oh we just had a chat about it and I've got some contributions over there as well related to the images :-)
2020-12-16 22:21:01 <LBlaboon> ikke: ncopa: just wanted to let you guys know that we pushed 3.12.3 a few hours ago and also updated our network helper templates with the accept_ra change, so that's all ready to go
2020-12-17 07:01:57 <ncopa> LBlaboon: awesome! thanks!
2020-12-17 22:11:14 <clandmeter> ikke: ping
2020-12-17 22:11:38 <ikke> pong
2020-12-17 22:12:54 <clandmeter> did you add the uacme compose on test instnace?
2020-12-17 22:13:19 <ikke> Yes
2020-12-17 22:13:37 <clandmeter> ah its custom made
2020-12-17 22:13:42 <clandmeter> not the one from gitlab
2020-12-17 22:13:47 <ikke> nope
2020-12-17 22:13:52 <ikke> was just trying something
2020-12-17 22:13:57 <clandmeter> :)
2020-12-17 22:15:32 <clandmeter> you know we have a general cron file to fetch the certs
2020-12-17 22:16:47 <ikke> yes
2020-12-17 22:17:21 <ikke> I just didn't want to use the wildcard certs on a test instance
2020-12-17 22:17:45 <clandmeter> right, you mean the key
2020-12-17 22:18:09 <ikke> nod
2020-12-17 22:18:33 <clandmeter> traefik should be able to do it, but i have not tried it yet.
2020-12-17 22:19:55 <clandmeter> ikke: do you mind if i turn off traefik on that host?
2020-12-17 22:20:06 <ikke> I don't mind
2020-12-17 23:14:59 <ikke> :/
2020-12-17 23:17:10 <ikke> seems like packet has issues?
2020-12-17 23:27:28 <ikke> yup
2020-12-17 23:27:34 <ikke> all in ewr
2020-12-18 05:40:47 <ikke> https://status.equinixmetal.com/incidents/pfgmgy1fnjcp
2020-12-18 05:42:01 <ikke> partial power outage
2020-12-18 07:16:53 <ncopa> what is the reason that we dont have mips64 CI? due to we didnt have gitlab-runner?
2020-12-18 07:17:53 <ncopa> and docker...
2020-12-18 07:18:04 <ncopa> i dont think we have official docker images for mips64
2020-12-18 07:40:36 <ikke> missing go
2020-12-18 07:40:43 <ikke> so yes, missing docker / gitlab-runner
2020-12-18 07:41:15 <ikke> base images for mips64 would be usefull too
2020-12-18 07:41:17 <ikke> yes
2020-12-18 07:42:30 <ikke> ncopa: nld3 is full btw
2020-12-18 07:43:01 <ikke> hm, that's more a clandmeter / me thing though
2020-12-18 07:43:23 <ikke> It's the mirror that's hosted there
2020-12-18 07:50:47 <ikke> and there is nld5
2020-12-18 07:56:21 <ikke> ncopa: I saw that edge has all kinds of old iso's. Do we still want to keep them there?
2020-12-18 08:39:05 <clandmeter> Some have more space
2020-12-18 08:39:26 <clandmeter> Just need lvm fix
2020-12-18 10:04:41 <clandmeter> nld3|5 have 100G extra now for mirror
2020-12-18 10:04:54 <ikke> thanks
2020-12-18 10:05:21 <clandmeter> ikke: there is a simple txt that has a 2 step action if it happens again.
2020-12-18 10:05:28 <clandmeter> i think we still have some additional space
2020-12-18 10:06:03 <ikke> clandmeter: can even be one step if you add -r to lvresize ;-)
2020-12-18 10:06:14 <ikke> lvextend*
2020-12-18 10:06:32 <clandmeter> it is a bit tricky
2020-12-18 10:06:45 <clandmeter> i prefer to do it myself 
2020-12-18 10:06:49 <ikke> hehe :)
2020-12-18 10:06:54 <clandmeter> in case tools are mising
2020-12-18 10:07:08 <clandmeter> lvm will call resize2fs iirc
2020-12-18 10:08:01 <ikke> yes
2020-12-18 10:10:22 <ikke> clandmeter: maybe after 3.13 is released, we should make some effort on upgrading our infra
2020-12-18 10:11:13 <clandmeter> as in alpine version?
2020-12-18 10:11:17 <ikke> yes
2020-12-18 10:11:20 <clandmeter> sure
2020-12-18 10:11:25 <clandmeter> and record it in netbox
2020-12-18 10:11:27 <ikke> Most is still running 3.10
2020-12-18 10:11:35 <clandmeter> maybe we can also upgrade netbox as the same time
2020-12-18 10:11:39 <clandmeter> its a bit behind
2020-12-18 10:11:42 <ikke> yes, I had that in mind
2020-12-18 10:12:33 <clandmeter> would be nice if we can move things into iac
2020-12-18 10:12:39 <clandmeter> we can do it one step at a time.
2020-12-18 10:13:03 <ikke> Yes, that's best 
2020-12-18 10:13:13 <ikke> trying to do everything at once is daunting :)
2020-12-18 10:13:32 <clandmeter> yes, and its not needed.
2020-12-18 10:13:42 <clandmeter> the thing is how to test things in iac 
2020-12-18 10:13:57 <clandmeter> its kind of challenging 
2020-12-18 10:14:49 <ikke> chef has kitchen, which is nice
2020-12-18 10:15:16 <ikke> I think you can adopt kitchen also for other things, though
2020-12-18 10:15:35 <ikke> but it remains a challenge to verify actual effects on production
2020-12-18 10:17:36 <clandmeter> i am trying to convert some compose stuff into ansible, but it consumes time and i dont have an easy way to test it.
2020-12-18 10:17:52 <clandmeter> and thats a problem when doing it on running infra
2020-12-18 10:18:53 <ikke> does ansible have a dry-run?
2020-12-18 10:21:16 <clandmeter> something like --check i guess
2020-12-18 11:08:43 <ncopa> i think we can delete old edge iso images
2020-12-18 12:32:05 <ikke> clandmeter: playing a bit with terraform?
2020-12-18 13:27:22 <clandmeter> ikke: big brother? :p
2020-12-18 13:27:47 <ikke> linode is spamming me :P
2020-12-18 13:27:55 <clandmeter> i was checking what gitlab supports
2020-12-18 13:28:51 <clandmeter> looks like hashicorp is their favorite
2020-12-18 13:29:01 <ikke> nod
2020-12-19 11:10:49 <mps> ssh can't access my armv7 lxc on usa4, while aarch64 there works fine
2020-12-19 11:11:29 <ikke> let me check
2020-12-19 11:12:00 <ikke> mps: it has no ip :)
2020-12-19 11:12:47 <mps> hmm, why?
2020-12-19 11:12:51 <clandmeter> ip thieves
2020-12-19 11:12:59 <ikke> /etc/init.d/networking was gone
2020-12-19 11:13:09 <mps> aha
2020-12-19 11:13:09 <ikke> fixed now
2020-12-19 11:13:34 <mps> ifupdown-ng again in action? :P
2020-12-19 11:13:50 <ikke> the removal of ifupdown-ng-openrc
2020-12-19 11:14:15 <ikke> for some reason, that does not restore /etc/init.d/networking from openrc
2020-12-19 11:14:40 <mps> yes, known bug
2020-12-19 11:16:00 <clandmeter> it overwrites it with replaces
2020-12-19 11:18:54 <mps> ikke: thanks
2020-12-20 07:20:30 <ikke> clandmeter: can we destroy the gitlab-test instance?
2020-12-20 12:33:30 <clandmeter> should we try upgrading it to 13.6?
2020-12-20 12:34:15 <ikke> do we want a new snapshot?
2020-12-20 12:34:49 <clandmeter> we could first test on this, and if it works do ones more?
2020-12-20 12:34:58 <clandmeter> whatever you prefer
2020-12-20 12:35:12 <ikke> I guess we first need to build 13.6, right?
2020-12-20 12:35:28 <clandmeter> yup
2020-12-20 12:35:39 <clandmeter> I was about to run a test
2020-12-20 12:35:42 <ikke> ok
2020-12-20 12:43:07 <ikke> clandmeter: I think I've now really disabled mails on gitlab-test
2020-12-20 12:43:11 <ikke> I was still receiving them
2020-12-20 12:43:27 <clandmeter> ok
2020-12-20 12:43:45 <clandmeter> there is some project that can act as an smtp and queue them
2020-12-20 12:44:44 <clandmeter> upstream changed nginx config it seems
2020-12-20 15:00:44 <ikke> clandmeter: https://gitlab.com/gitlab-org/gitlab/-/issues/273558#note_441441155
2020-12-20 15:08:35 <ikke> A workaround for the related issues form not rendering properly
2020-12-21 08:36:19 <clandmeter> ikke: i think we need to start using  the bundled git in gitlab.
2020-12-21 08:37:49 <ikke> `due to version constraints?
2020-12-21 08:38:14 <clandmeter> they kind of depend on the latest and greatest git features
2020-12-21 08:38:26 <clandmeter> and they would like to support in tree patches....
2020-12-21 08:39:11 <ikke> You mean they want to be able to patch git?
2020-12-21 08:39:23 <clandmeter> nod
2020-12-21 08:39:58 <clandmeter> gitaly has a make git t arget
2020-12-21 08:40:49 <clandmeter> 13.6 wants git .29
2020-12-21 08:41:12 <ikke> which we only have in 3.13
2020-12-21 08:41:15 <clandmeter> maybe even more recent version
2020-12-21 08:41:27 <clandmeter> not sure what 13.4 wants
2020-12-21 08:41:35 <clandmeter> somewhere they made the switch
2020-12-21 08:42:46 <clandmeter> # Make sure Git is version 2.24.0 or higher (recommended version is 2.28.0)
2020-12-21 08:42:53 <clandmeter> so .24 is still ok
2020-12-21 08:54:46 <clandmeter> ikke: did you have to manually create the btree_gist extension?
2020-12-21 08:55:46 <ikke> yes, I think so
2020-12-21 08:55:48 <ikke> I did not automate it
2020-12-21 08:56:01 <clandmeter> how did you know y ou had to do so?
2020-12-21 08:56:29 <clandmeter> migrations failed?
2020-12-21 08:57:10 <ikke> yes
2020-12-21 08:59:12 <clandmeter> so the current user does not have that permission
2020-12-21 08:59:25 <ikke> correct
2020-12-21 08:59:34 <ikke> the 'gitlab' user
2020-12-21 08:59:47 <clandmeter> https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/install/postgresql_extensions.md#L69
2020-12-21 10:28:50 <ikke> clandmeter: The other day I was trying to make ipv6 work for git.a.o, but because it's now a cname for deu1-dev1.a.o, I cannot point it to a different ipv6 address
2020-12-21 10:29:36 <ikke> clandmeter: I guess we could change it to an A record instead of a CNAME
2020-12-21 10:29:36 <clandmeter> hmm
2020-12-21 10:29:42 <clandmeter> interesting :)
2020-12-21 10:29:54 <clandmeter> ipv6 makes trouble ;-)
2020-12-21 10:30:16 <ikke> more like ipv4 making it hard for ipv6 to function properly :)
2020-12-21 10:30:33 <clandmeter> i know, but its still trouble :)
2020-12-21 10:30:50 <clandmeter> im ok in chainging it to a record
2020-12-21 10:30:59 <ikke> ok
2020-12-21 10:31:23 <clandmeter> is there an issue attached to that request?
2020-12-21 12:55:58 <ikke> clandmeter: maybe we want to manage our DNS records as code as well
2020-12-21 13:15:26 <clandmeter> nod
2020-12-21 13:23:42 <minimal> have played with Ansible to generate BIND zone files
2020-12-21 13:25:50 <ikke> minimal: we use linode, so probably we'd integrate with their API
2020-12-21 13:27:40 <minimal> ok, main thing to think about is the SOA serial - do you keep it typical date-type format with simple increments
2020-12-21 13:29:06 <clandmeter> you dont have to manage it with linode
2020-12-21 13:29:39 <mps> khost -t soa alpinelinux.org ns1.linode.com.
2020-12-21 13:29:48 <minimal> as using "infrastructure-as-code" if you've lots of potential changes any day and you're using date-format (plus 1/2 digit extra) serial you could run into problems
2020-12-21 13:30:08 <ikke> We make like at most a couple of changes per month :P
2020-12-21 13:30:31 <mps> 2021000024
2020-12-21 13:30:42 <mps> 2021
2020-12-21 13:31:00 <minimal> welcome to the future :-)
2020-12-21 13:40:23 <ikke> https://www.linode.com/docs/api/domains
2020-12-21 13:46:24 <minimal> ikke: seems they take care of serial management, don't see if mentioned in that doc
2020-12-21 13:47:22 <ikke> Yes. You can just add records to the webif, they handle the zone files
2020-12-21 13:47:52 <minimal> makes life too easy ;-)
2020-12-21 16:21:28 <kunkku> I recently realized the aconf repo was lost on the gitlab migration
2020-12-21 16:21:54 <kunkku> https://git.alpinelinux.org/aports/tree/main/aconf/APKBUILD#n7
2020-12-21 16:22:20 <ikke> We still have everything
2020-12-21 16:22:49 <kunkku> should it be hosted on AL gitlab or shall I take it elsewhere?
2020-12-21 16:22:59 <ikke> I think we can host it on AL
2020-12-21 16:23:02 <ikke> gitlab
2020-12-21 16:26:55 <ikke> Do you still have the repo?
2020-12-21 16:43:11 <kunkku> yes, I have a local copy
2020-12-21 16:48:13 <ikke> kunkku: You could create a repo on gitlab and push it to there, then I can make it public
2020-12-21 16:48:56 <ikke> as it was a user repo before, that's the equivalent
2020-12-21 16:49:56 <kunkku> what should the path be?
2020-12-21 16:50:13 <ikke> /kunkku/aconf?
2020-12-21 16:50:53 <kunkku> would that be the only user repo?
2020-12-21 16:51:13 <ikke> What do you mean?
2020-12-21 16:51:50 <kunkku> of form /<user>/<project> ?
2020-12-21 16:52:48 <ikke> The core alpine projects live under /alpine
2020-12-21 16:52:56 <ikke> there are others who have projects hosted under their username
2020-12-21 16:54:02 <kunkku> are all of them private because I can't see them?
2020-12-21 16:56:07 <ikke> You should be able to see them here: https://gitlab.alpinelinux.org
2020-12-21 16:56:53 <kunkku> I can see 72 repos, all starting with alpine/
2020-12-21 16:57:12 <ikke> I guess because you are redirected to /alpine/ ?
2020-12-21 16:57:46 <ikke> https://gitlab.alpinelinux.org/explore
2020-12-21 16:58:57 <kunkku> no matter the filters, I can see 72 repos at most
2020-12-21 16:59:15 <ikke> And that last link?
2020-12-21 17:01:25 <kunkku> ok, now I can see
2020-12-21 17:05:10 <kunkku> now it's there
2020-12-21 17:05:48 <ikke> made it public
2020-12-21 17:08:17 <kunkku> thanks
2020-12-21 17:08:40 <ikke> Note that gitlab exposes archives that you can use as source for apkbuilds
2020-12-22 10:40:12 <clandmeter> ikke: ping
2020-12-22 10:40:15 <ikke> pong
2020-12-22 10:40:23 <clandmeter> damn you are fast
2020-12-22 10:40:27 <ikke> haha :D
2020-12-22 10:40:28 <clandmeter> is that a script?
2020-12-22 10:40:37 <ikke> I was just looking at the alert here
2020-12-22 10:40:52 <clandmeter> we had an email about a mirror not long ago
2020-12-22 10:41:04 <clandmeter> or was i the only one?
2020-12-22 10:41:06 <ikke> I handled all mirrors requests yesterday
2020-12-22 10:41:17 <ikke> the one about the rebranding?
2020-12-22 10:41:22 <clandmeter> yes
2020-12-22 10:42:50 <ikke> Was handled
2020-12-22 10:43:01 <ikke> https://mirrors.alpinelinux.org/#mirror29
2020-12-22 10:43:14 <clandmeter> nice
2020-12-22 10:43:17 <clandmeter> i was thinking
2020-12-22 10:43:23 <clandmeter> should we make that public?
2020-12-22 10:43:35 <clandmeter> is there really private info in that list?
2020-12-22 10:44:16 <clandmeter> im asking as we could allow operators to create MR's
2020-12-22 10:45:20 <ikke> I have no idea what is considered private
2020-12-22 10:46:01 <clandmeter> the only private thing would be the contact window
2020-12-22 10:46:25 <clandmeter> but i think 99 out of 100 those are public email addresses
2020-12-22 11:00:32 <ikke> I sent an e-mail about the speglar mirror yesterday, and now it seems they pulled it (but the mirror is still available under a different address)
2020-12-22 11:27:04 <HRio> aarch64 gitlab runner down, or just busy?
2020-12-22 11:28:10 <ikke> just busy
2020-12-22 11:29:10 <HRio> thx
2020-12-22 12:58:38 <clandmeter> ikke: wwwtest does not work anymore?
2020-12-22 12:59:09 <ikke> You tell me :)
2020-12-22 12:59:26 <clandmeter> not managed by me :)
2020-12-22 12:59:42 <ikke> I have the feeling it's not managed by anyone
2020-12-22 12:59:45 <clandmeter> i dont think i have access, its on old-git
2020-12-22 12:59:49 <ikke> aha
2020-12-22 13:00:54 <ikke> I don't think there is a reason to keep it there
2020-12-22 13:01:37 <clandmeter> why is it not working?
2020-12-22 13:01:43 <clandmeter> container not started? 
2020-12-22 13:02:31 <clandmeter> nginx is giving a tiemout
2020-12-22 13:02:36 <ikke> (the ipv6-test server is still running as well)
2020-12-22 13:02:42 <clandmeter> i guess its maybe an ip that has changed?
2020-12-22 13:03:33 <clandmeter> can you check the nginx container to see where it points to?
2020-12-22 13:03:43 <ikke> right, nginx still points to an old address
2020-12-22 13:04:57 <ikke> ok, working now
2020-12-22 13:15:25 <clandmeter> :(
2020-12-22 13:15:32 <clandmeter> www does not update automatically
2020-12-22 13:17:00 <ikke> yeah, ncopa already mentioned that
2020-12-22 13:18:03 <clandmeter> config looks correct
2020-12-22 13:26:16 <clandmeter> no idea why it does not work
2020-12-22 13:46:52 <ikke> clandmeter: restarted the build-3-13-x86_64 container, and now ns1 does no longer resolve it, but if I query nld9 directly, it resolves
2020-12-22 15:12:40 <ikke> hmm, now it works again :(
2020-12-22 18:31:40 <clandmeter> https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released
2020-12-22 18:32:03 <clandmeter> Review requests
2020-12-22 18:32:17 <ikke> Yes, I see, nice
2020-12-22 18:37:33 <ikke> The docker proxy is also interesting
2020-12-25 10:24:12 <ikke> clandmeter: fyi, I switched the records for git.a.o now to A/AAAA records
2020-12-25 12:20:53 <clandmeter> 👍
2020-12-25 13:26:17 <ikke> clandmeter: with this version of gitlab it seems there are still hanging processes
2020-12-25 15:11:31 <ikke> seems like the compose file documentation got removed from docs.docker.com and they just link to a github spec.md file
2020-12-25 18:22:45 <clandmeter> ikke: v2 still online
2020-12-25 18:25:07 <ikke> v2?
2020-12-25 18:25:11 <ikke> doc?
2020-12-25 18:26:37 <clandmeter> https://docs.docker.com/compose/compose-file/compose-file-v2/
2020-12-27 13:32:09 <cornfeedhobo> what happened to #alpine?
2020-12-27 14:10:35 <ikke> did something happen?
2020-12-27 14:11:36 <cornfeedhobo> it's cleared out. now bounces to ##namespace
2020-12-27 14:11:44 <cornfeedhobo> looks like someone at freenode booted it
2020-12-27 14:12:23 <cornfeedhobo> oh, maybe i'm just an idiot and it was changed to invite only
2020-12-27 14:12:57 <cornfeedhobo> but ... then it wouldn't be bouncing me
2020-12-27 19:03:03 <ikke> \o/
2020-12-28 13:16:27 <ikke> ncopa: nld9-dev1 disk is full (x86_64 builder)
2020-12-28 14:09:19 <ncopa> ok ill clean up
2020-12-28 15:28:02 <cornfeedhobo> :(
2020-12-28 15:34:15 <ncopa> i have free'd up a few gigs
2020-12-28 15:34:26 <ncopa> deleted some old build logs, and old distfiles
2020-12-29 01:59:09 <Ariadne> cornfeedhobo: the alpine e-mail client is at ##alpine, alpine linux is at #alpine-linux
2020-12-29 01:59:18 <Ariadne> this is intended to reduce confusion
2020-12-29 06:37:02 <mps> I'm banned from alpine project https://gitlab.alpinelinux.org/alpine/aports/-/project_members ?
2020-12-29 06:37:27 <mps> :)
2020-12-29 06:40:00 <ikke> https://gitlab.alpinelinux.org/groups/team/developers/-/group_members
2020-12-29 06:42:38 <mps> ah, not yet :P
2020-12-29 06:43:57 <mps> but little confusing because I'm not listed in previous url
2020-12-29 08:32:06 <cornfeedhobo> Ariadne: thanks. strange it didn't show up in /list
2020-12-29 08:33:41 <ikke> cornfeedhobo: because this channels are set to private, to reduce spam
2020-12-29 08:33:54 <cornfeedhobo> ah
2020-12-29 08:34:27 <ikke> There was a period last year where there was a lot of spam on freenode
2020-12-31 08:05:25 <ncopa> the 3.13 package are being uploaded on the master mirror. do we have enough space?
2020-12-31 08:05:55 <ncopa> 103G free