2020-06-01 07:27:30 <ikke> didstopia should resolve soon
2020-06-01 07:40:21 <clandmeter> ikke: morning
2020-06-01 07:40:59 <ikke> morning
2020-06-01 07:41:04 <clandmeter> ikke: im looking into gitlab issue
2020-06-01 07:41:07 <ikke> Ok
2020-06-01 07:41:18 <clandmeter> i think i found it
2020-06-01 07:41:20 <ikke> I wonder if it's some DB index issue
2020-06-01 07:41:21 <ikke> oh, ok
2020-06-01 07:41:22 <clandmeter> but i am not sure
2020-06-01 07:41:30 <clandmeter> try running this on both hosts
2020-06-01 07:41:36 <clandmeter> sync && time sh -c "dd if=/dev/zero of=testfile bs=100k count=10k && sync"
2020-06-01 07:41:48 <clandmeter> you will be surprised 
2020-06-01 07:41:59 <ikke> so IO difference
2020-06-01 07:42:13 <clandmeter> yes and not by a little diff
2020-06-01 07:42:32 <ikke> 1s vs 4s
2020-06-01 07:42:50 <ikke> now 1s vs 1s
2020-06-01 07:43:28 <clandmeter> https://tpaste.us/DDl0
2020-06-01 07:43:46 <clandmeter> but its a stupid test
2020-06-01 07:43:56 <mps> time dd if=/dev/zero of=/mnt/testf bs=1M count=1024 oflag=direct
2020-06-01 07:44:13 <clandmeter> i dont think dd in bb supports it
2020-06-01 07:44:18 <ikke> echo 1 >/proc/sys/vm/drop_caches
2020-06-01 07:44:34 <mps> i.e. I found  oflag=direct useful for such tests
2020-06-01 07:45:27 <mps> ah, true bb dd doesnt't support it
2020-06-01 07:45:34 <ikke> only skip_bytes
2020-06-01 07:45:40 <ikke> seek_bytes*
2020-06-01 07:45:49 <clandmeter> i think we could try some other io  test
2020-06-01 07:45:57 <clandmeter> ikke: check compose log
2020-06-01 07:46:06 <ikke> on test?
2020-06-01 07:46:07 <clandmeter> it also complains about op
2020-06-01 07:46:10 <clandmeter> io
2020-06-01 07:46:26 <clandmeter> redis_1     | 1:M 31 May 2020 14:44:19.098 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
2020-06-01 07:46:42 <ikke> ah, redis
2020-06-01 07:47:06 <clandmeter> the *real* instance is running on dedicated hw
2020-06-01 07:47:12 <ikke> nod
2020-06-01 07:47:17 <clandmeter> kind of :)
2020-06-01 07:47:28 <clandmeter> still vm ofc
2020-06-01 07:47:30 <ikke> But never noticed this before
2020-06-01 07:47:37 <ikke> At least, did not recall
2020-06-01 07:47:43 <clandmeter> maybe its also on the real one this notice
2020-06-01 07:47:56 <clandmeter> but it triggered my senses to try io
2020-06-01 07:48:12 <clandmeter> is there some tool we can try to run some tests?
2020-06-01 07:48:17 <ikke> yes, also on prod
2020-06-01 07:48:26 <clandmeter> mps: good morning to you too ;-)
2020-06-01 07:48:29 <ikke> redis_1     | 1:M 31 May 2020 03:18:42.040 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
2020-06-01 07:48:38 <clandmeter> right
2020-06-01 07:49:03 <ikke> clandmeter: what do you want to test?
2020-06-01 07:49:07 <mps> clandmeter: yes, goeden morgen (uh)
2020-06-01 07:49:26 <clandmeter> performance of the disk between the two
2020-06-01 07:49:31 <ikke> clandmeter: it's mostly postgres that is taking longer according to gl stats
2020-06-01 07:49:47 <clandmeter> well that could also be io bound?
2020-06-01 07:49:50 <ikke> sure
2020-06-01 07:50:01 <mps> hm, 'goede'
2020-06-01 07:50:08 <ikke> mps: correct :)
2020-06-01 07:50:16 <clandmeter> yes, its dutch
2020-06-01 07:50:22 <clandmeter> you will make mistakes all the time.
2020-06-01 07:50:43 <mps> isn't today holiday in NL?
2020-06-01 07:50:52 <clandmeter> yes it is
2020-06-01 07:51:00 <clandmeter> i wear sunglasses
2020-06-01 07:51:06 <ikke> :D
2020-06-01 07:51:32 <clandmeter> i want to copy those two disks/io
2020-06-01 07:51:33 <mps> so, go out somewhere you both, and enjoy holiday :)
2020-06-01 07:51:40 <clandmeter> if those simple stats really add up 
2020-06-01 07:52:05 <clandmeter> lol
2020-06-01 07:52:11 <ikke> clandmeter: why not apk add cmd:dd and test with io direct?
2020-06-01 07:52:12 <clandmeter> s/copy/compare
2020-06-01 07:52:12 <alpine-meetbot> clandmeter meant to say: i want to compare those two disks/io
2020-06-01 07:52:52 <clandmeter> cause dd is not the best io test for such cases.
2020-06-01 07:53:45 <mps> true, but is helpful as 'first aid'
2020-06-01 07:57:05 <ikke> clandmeter: we could also try 13.0 like you suggested
2020-06-01 07:59:09 <clandmeter> ikke: yes but i would prefer to first rule out vm issues
2020-06-01 07:59:17 <clandmeter> ikke: is this a new vm?
2020-06-01 07:59:21 <ikke> yes
2020-06-01 07:59:32 <clandmeter> and you delted the old one?
2020-06-01 07:59:35 <ikke> yes
2020-06-01 07:59:38 <clandmeter> ok
2020-06-01 07:59:43 <clandmeter> I also did some cleaning
2020-06-01 07:59:52 <clandmeter> i got an invoice and it was getting high
2020-06-01 07:59:58 <ikke> ok
2020-06-01 07:59:59 <clandmeter> close to our limit
2020-06-01 08:00:03 <ikke> nod
2020-06-01 08:06:29 <clandmeter> we could use someting like fio to run some tests?
2020-06-01 08:07:50 <mps> ehm, we don't bonnie+ in repo
2020-06-01 08:08:37 <mps> but fio is also nice, knowing who is author
2020-06-01 08:08:45 <ikke> clandmeter: we could check and see if 12.9 is also slower
2020-06-01 08:08:56 <ikke> heh, I was looking at fio as well :(
2020-06-01 08:08:57 <ikke> :)
2020-06-01 08:09:17 <clandmeter> there is also ioping
2020-06-01 08:11:04 <clandmeter> ikke: yeah we could try that
2020-06-01 08:11:15 <clandmeter> 12.9
2020-06-01 08:13:51 <clandmeter> ikke: when you restore you could afterwards make a tarball of srv/docker/gitlab, that would speedup doing some tests between versions.
2020-06-01 08:15:30 <clandmeter> ikke: did you compare results from gitlab performance bar with both versions?
2020-06-01 08:18:13 <clandmeter> i even get an exclamation mark on the bar
2020-06-01 08:33:10 <ikke> I compared prod vs testing
2020-06-01 08:36:41 <ikke> clandmeter: So recreate the test vm?
2020-06-01 08:36:55 <clandmeter> yes please
2020-06-01 08:37:21 <clandmeter> lets compare versions on the same host
2020-06-01 08:37:26 <clandmeter> makes more sense
2020-06-01 08:37:38 <ikke> nod
2020-06-01 08:37:46 <clandmeter> the bar is weird
2020-06-01 08:37:52 <clandmeter> it shows gitlab.a.o on both
2020-06-01 08:38:27 <ikke> test is gone :)
2020-06-01 08:38:29 <clandmeter> the sql queries are also very different
2020-06-01 08:39:02 <clandmeter> ikke: maybe select the same host spec?
2020-06-01 08:39:11 <clandmeter> dedi hw
2020-06-01 08:39:17 <clandmeter> just for the tests
2020-06-01 08:39:17 <ikke> ok, will do
2020-06-01 08:39:30 <clandmeter> so we shouldnt keep it running for too long.
2020-06-01 08:39:59 <clandmeter> ikke: did your issue get merged?
2020-06-01 08:40:03 <clandmeter> or solved
2020-06-01 08:40:07 <clandmeter> in gitlab
2020-06-01 08:40:20 <ikke> "Additional verification is required to add this service. Please open a support ticket.
2020-06-01 08:40:23 <ikke> "
2020-06-01 08:40:35 <ikke> clandmeter: no, did not see any activity on the issue
2020-06-01 08:40:47 <clandmeter> ah you cannot create such instance?
2020-06-01 08:40:52 <ikke> apparently not
2020-06-01 08:41:02 <clandmeter> maybe i can try.
2020-06-01 08:43:53 <clandmeter> same thing
2020-06-01 08:44:10 <ikke> ok, just use a normal instance then?
2020-06-01 08:44:10 <clandmeter> ok lets just try the other one
2020-06-01 08:44:16 <clandmeter> its just to compare 
2020-06-01 08:44:18 <ikke> you or me?
2020-06-01 08:44:23 <clandmeter> i can do it
2020-06-01 08:44:25 <ikke> ok
2020-06-01 08:44:37 <ikke> fyi: I created a gitlab-test A record 
2020-06-01 08:44:46 <ikke> so we can update that
2020-06-01 08:45:37 <clandmeter> cool
2020-06-01 08:45:40 <clandmeter> low ttl?
2020-06-01 08:45:43 <ikke> yes
2020-06-01 08:45:45 <ikke> 5m
2020-06-01 08:45:49 <clandmeter> nice
2020-06-01 08:45:54 <clandmeter> 172.105.64.166
2020-06-01 08:45:56 <clandmeter> :)
2020-06-01 08:47:10 <ikke> updated
2020-06-01 08:48:12 <clandmeter> the slowdown feels a lot like we had in the beginning
2020-06-01 08:50:48 <ikke> Now wait a few hours
2020-06-01 08:54:00 <clandmeter> :)
2020-06-01 08:54:29 <clandmeter> lets not forget to make that tarball to skip this step.
2020-06-01 09:13:26 <ikke> heh
2020-06-01 09:13:56 <ikke> reminds me, didn't you add a step to make a database export before migrating?
2020-06-01 09:20:55 <clandmeter> could be
2020-06-01 09:21:06 <clandmeter> ikke: also please remember i removed the backup volume
2020-06-01 09:21:14 <clandmeter> i dont think it was really used
2020-06-01 09:21:37 <ikke> I was not even aware of a backup-volume :)
2020-06-01 09:21:53 <clandmeter> it was mounted in the backup location
2020-06-01 09:22:05 <clandmeter> before we did linode backups
2020-06-01 09:22:12 <ikke> ok
2020-06-01 09:22:51 <ikke> btw, are the backups tied to the instance? What if we by accident (I sure hope it does not happen) delete the gitlab prod instance
2020-06-01 09:23:58 <clandmeter> good question
2020-06-01 09:24:12 <clandmeter> i dont see a backup option
2020-06-01 09:24:17 <clandmeter> only from within the instance
2020-06-01 09:24:21 <ikke> yeah
2020-06-01 09:25:45 <clandmeter> the upgrade step does not show a db export
2020-06-01 09:25:52 <clandmeter> or backup
2020-06-01 09:26:42 <clandmeter> we could add a gitlab backup just before the upgrade
2020-06-01 09:26:51 <clandmeter> but im not sure if that makes sense
2020-06-01 09:27:28 <ikke> I'm not sure we want to restore a full backup to roll-back
2020-06-01 09:27:33 <ikke> especially if it takes >1h
2020-06-01 09:27:50 <clandmeter> looks like  there is dump_db
2020-06-01 09:28:06 <clandmeter> you could do that manually
2020-06-01 09:28:35 <ikke> Yes, which is what I provisionally did
2020-06-01 09:28:47 <ikke> but then you said you were going to add that the script itself
2020-06-01 09:28:58 <clandmeter> haha
2020-06-01 09:29:00 <clandmeter> i did?
2020-06-01 09:29:02 <ikke> yes
2020-06-01 09:29:05 <clandmeter> you have such good memory?
2020-06-01 09:29:12 <clandmeter> or such good backlog?
2020-06-01 09:29:17 <ikke> memory
2020-06-01 09:29:20 <ikke> I guess :)
2020-06-01 09:29:54 <clandmeter> i guess making a dump doesnt hurt
2020-06-01 09:30:01 <clandmeter> its not really a backup
2020-06-01 09:31:23 <clandmeter> i think doing a periodic gitlab backup to something like minio could be usefull to work around massive stupidity by clicking delete this instance.
2020-06-01 09:43:23 <clandmeter> ikke: what is your advise to make an MR for gitlab?
2020-06-01 09:44:09 <ikke> We'd first have to find out where the issue is
2020-06-01 09:44:50 <clandmeter> hehe
2020-06-01 09:44:56 <clandmeter> i mean our project
2020-06-01 09:44:56 <ikke> And then find out what the policy is regarding MRs
2020-06-01 09:45:05 <ikke> hmm
2020-06-01 09:45:12 <clandmeter> i want to make an MR for our gitlab instance
2020-06-01 09:45:18 <clandmeter> should i fork?
2020-06-01 09:45:34 <ikke> I don't think forking is necessary
2020-06-01 09:45:46 <ikke> You can just push a branch and then create an MR against master
2020-06-01 09:45:48 <clandmeter> so it would introduce a feature branch?
2020-06-01 09:45:52 <ikke> yes
2020-06-01 09:46:11 <ikke> forking is only necessary when you don't have permissions to push to the original project, or aports, because we want to keep it clean
2020-06-01 10:02:26 <clandmeter> ok i did it from the webif :)
2020-06-01 10:03:47 <clandmeter> ah thats cool
2020-06-01 10:03:54 <clandmeter> its not a protected branch
2020-06-01 10:03:59 <clandmeter> so it does not push to github
2020-06-01 10:04:05 <clandmeter> so no trigger
2020-06-01 11:46:35 <ikke> The instance is available
2020-06-01 11:47:02 <ikke> It's booting
2020-06-01 12:14:42 <ikke> nice
2020-06-01 12:29:22 <ikke> ok, I have a tarbal from /srv/docker/gitlab
2020-06-01 12:29:38 <ikke> 20G atm
2020-06-01 12:52:35 <ikke> clandmeter: even with 12.9 it's slow, so I guess it has to do with the instance, and not a performance regression
2020-06-01 12:52:52 <ikke> It's higly variable
2020-06-01 16:47:14 <clandmeter> ikke: ok 
2020-06-01 16:47:32 <clandmeter> you feel safe to do the upgrade?
2020-06-01 16:53:36 <ikke> If we can roll-back, then yes
2020-06-01 18:33:46 <ikke> (I just added that)
2020-06-01 18:37:18 <ikke> funny that uk.a.o is hosted in NLD :)
2020-06-01 18:37:27 <ikke> Should we still list that as a mirror?
2020-06-01 18:37:28 <maxice8> not that far 
2020-06-01 18:37:43 <maxice8> The south american and specifically portuguese runescape servers are hosted in New Jersey :D
2020-06-01 18:39:02 <ikke> It used to be in the uk, but we decommissioned the mirror there
2020-06-01 19:10:02 <maxice8> I'll restrain myself from the comment in my mind
2020-06-01 20:00:36 <clandmeter> ikke: i think we should remove uk.a.o from list
2020-06-01 20:01:07 <clandmeter> seems mirrors is not updated anymore
2020-06-01 20:01:18 <ikke> Like more things :)
2020-06-01 20:01:31 <ikke> We changed mqtt topic, but did not verify everything was updated
2020-06-01 20:04:36 <clandmeter> mirrors is periodic i think
2020-06-01 20:05:05 <ikke> ok
2020-06-01 20:05:19 <ikke> That explains why it was showing ok
2020-06-01 20:05:29 <ikke> the status.json was is up-to-date I think
2020-06-02 07:16:49 <clandmeter> ikke: i think i know why mirrors didnt update
2020-06-02 07:17:12 <clandmeter> the status.json was okish
2020-06-02 07:17:40 <clandmeter> my guess is that one of the mirrors has a repo not existing on the master.
2020-06-02 09:38:47 <Cogitri> Could you maybe have a look at https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10692 so I can generate appstream data for 3.12? :) 
2020-06-02 11:51:06 <steveazz> Hello folks! Hopefully I'm in the right channel. I've started seeing failures when running `apk add` for example `ERROR: https://dl-4.alpinelinux.org/alpine/v3.10/main: temporary error (try again later)` it's been happen since around 8:00 UTC for me. I also sometimes see an error with a `Segmentation fault (core dumped)`. This only happens when I
2020-06-02 11:51:07 <steveazz> try to build a docker image that runs `apk add` and the build is running inside Docker in Docker. Are there any known issues at the moment, this seem to have been working fine yesterday.
2020-06-02 13:36:14 <steveazz> I was disconnected earlier, so apologise if you responded to me. But I've found out that use the `http` mirrors instead of the `https` ones seem to be working: 
2020-06-02 13:36:37 <steveazz> For example `http://dl-4.alpinelinux.org/alpine/v3.10/main/aarch64/APKINDEX.tar.gz`
2020-06-02 13:40:21 <mps> steveazz: these CDN mirrors use http, not https
2020-06-02 13:41:26 <mps> http://dl-cdn.alpinelinux.org/alpine/v3.12/main
2020-06-02 13:53:30 <steveazz> mps thank you for that information, it helps!
2020-06-02 13:57:56 <mps> np
2020-06-02 14:00:41 <steveazz> This might be a stupid question, but why would one prefer https over http for a mirror?
2020-06-02 14:02:56 <mps> well, privacy. people don't like that their ISP spy on them what packages they install
2020-06-02 14:31:46 <steveazz> And the reason why the CDN's don't have https is because of time constrains or is something that won't be solved?
2020-06-02 14:33:39 <Cogitri> You have to use a different URL for https
2020-06-02 14:33:46 <Cogitri> See mirrors.alpinelinux.org
2020-06-02 14:36:43 <steveazz> But when I use `https` I get `ERROR: https://uk.alpinelinux.org/alpine/v3.12/main: temporary error (try again later)` 
2020-06-02 14:37:05 <HRio> with bugs like this, you might not even be able to get the fix via a https mirror: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11607
2020-06-02 15:25:40 <mps> bind mount doesn't work in out developers lxc containers.? tried to setup armhf chroot but can't mount some needed dirs
2020-06-02 15:25:51 <mps> s/out/our/
2020-06-02 15:25:51 <alpine-meetbot> mps meant to say: bind mount doesn't work in our developers lxc containers.? tried to setup armhf chroot but can't mount some needed dirs
2020-06-02 15:46:05 <ncopa> where was ca-certificates supposed to move to? github?
2020-06-02 15:46:08 <ncopa> Fetching https://git.alpinelinux.org/ca-certificates/snapshot/ca-certificates-20191127.tar.xz
2020-06-02 15:46:14 <ncopa> curl: (22) The requested URL returned error: 404 
2020-06-02 16:57:13 <ikke> Who broke itC? ^ 
2020-06-02 16:58:46 <ikke> uuh
2020-06-02 16:59:16 <ikke> something is down
2020-06-02 16:59:18 <ikke> including netbox :-/
2020-06-02 16:59:25 <ikke> oh no, netbox is there
2020-06-02 17:04:44 <ikke> traefik is not accepting any connections again
2020-06-02 17:04:51 <ikke> disk is not full
2020-06-02 17:08:09 <ikke> restarting traefik fixed it..
2020-06-02 17:09:06 <ikke> strace showed constant operation timeouts, while after the restart, they are not there
2020-06-02 17:09:27 <ikke> well, they are there, but more other activity as well
2020-06-02 17:10:32 <ikke> ncopa: I think ca-certificate should be on gitlab as well, not?
2020-06-02 17:17:28 <clandmeter> ikke: maybe pull a new image
2020-06-02 17:22:37 <ikke> done
2020-06-02 17:25:05 <Cogitri> Can someone with the necessary bits mention on https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases that the 2 years of support only apply for main/ ?
2020-06-02 17:55:33 <Cogitri> And I think PPC64LE is stuck on OpenJDK, could someone kill it please?
2020-06-02 18:41:34 <ikke> Cogitri: re releases, done
2020-06-02 18:48:53 <Cogitri> Merci
2020-06-02 18:50:36 <ikke> killed openjdk as well :)
2020-06-02 20:05:50 <Ariadne> going to be taking down mips64 builder momentarily to try new build of BSP kernel to see if that fixes go
2020-06-02 20:05:59 <ikke> \(y)
2020-06-02 20:06:16 <Ariadne> i rebased the vendor config based on our linux-octeon package, so maybe with luck it will work
2020-06-02 20:06:42 <Ariadne> treefort:~/source/kernel$ grep OCTEON3_ETHERNET .config
2020-06-02 20:06:42 <Ariadne> # CONFIG_OCTEON3_ETHERNET is not set
2020-06-02 20:06:43 <Ariadne> ffs
2020-06-02 20:34:09 <Ariadne> ok
2020-06-02 20:34:12 <Ariadne> now doing the thing
2020-06-02 20:36:47 <Ariadne> reading vmlinux.64
2020-06-02 20:36:47 <Ariadne> ** Unable to read file vmlinux.64 **
2020-06-02 20:36:48 <Ariadne> hmm
2020-06-02 20:36:51 <Ariadne> one momento
2020-06-02 20:37:15 <Ariadne> well, thankfully i copied backup kernel
2020-06-02 20:37:33 <Ariadne> sooooo
2020-06-02 20:42:33 <Ariadne> haha
2020-06-02 20:42:39 <Ariadne> extracted to /root
2020-06-02 20:42:41 <Ariadne> instead of /
2020-06-02 20:42:49 <ikke> oh lol
2020-06-02 20:43:50 <ikke> extracted to the wrong root ;)
2020-06-02 20:44:53 <Ariadne> omg
2020-06-02 20:44:58 <Ariadne> i built it without CONFIG_NF_NAT
2020-06-02 20:45:20 <Ariadne> well before i waste more time on this
2020-06-02 20:45:24 <Ariadne> lets see if it fixes go
2020-06-02 20:45:54 <Ariadne> answer: no
2020-06-02 20:46:41 <ikke> sadface
2020-06-02 20:47:10 <Ariadne> so i am just putting back the original kernel
2020-06-02 20:49:40 <Ariadne> conclusion: we just keep go disabled until 5.4 kernel is working on this machine fully imo
2020-06-02 20:51:01 <ikke> ok, so disable docker and syncthing?
2020-06-02 20:57:01 <Ariadne> yeah for right now :/
2020-06-02 21:00:40 <Ariadne> oh, this is frustrating
2020-06-02 21:00:48 <Ariadne> we have the driver working for 10G ports
2020-06-02 21:00:51 <Ariadne> but not for 1G
2020-06-02 21:03:52 <Ariadne> oh well, the reverse engineering continues
2020-06-02 22:40:01 <Ariadne> So I had an idea.
2020-06-02 22:40:16 <Ariadne> Its pretty cursed but I think it can solve things
2020-06-02 22:40:27 <Ariadne> What if we run 5.4 kernel in kvm
2020-06-03 06:11:38 <Cogitri> Ugh, OpenJDK14 is now stuck on ppc64le (I think 13 was stuck before)
2020-06-03 08:59:04 <ncopa> script to purge latest-releases urls from cache: https://gitlab.alpinelinux.org/ncopa/purge-dlcdn-cache
2020-06-03 09:24:20 <ncopa> i have ran it from norway, brazil and usa now
2020-06-03 10:14:14 <clandmeter> ncopa: can you elaborate about this?
2020-06-03 10:14:29 <clandmeter> why not do a wildcard purge?
2020-06-03 10:15:08 <ncopa> a wildcard purge would purge everything, even if not needed?
2020-06-03 10:15:23 <clandmeter> why would it not be needed?
2020-06-03 10:15:41 <ncopa> because version changed since last release
2020-06-03 10:16:03 <clandmeter> yes so latest-stable would only have pkgs for 3.11?
2020-06-03 10:16:15 <clandmeter> but we it links now to 3.12
2020-06-03 10:17:01 <ncopa> if v3.11 had foo-1.0 and v3.12 updated it to foo-1.1, then latest-stable will have foo-1.1 and does not need to be purged from cache
2020-06-03 10:17:26 <ncopa> if both has foo-1.0, it needs to be purged
2020-06-03 10:17:53 <ncopa> i guess i could have used a wildcard purge, but i didnt
2020-06-03 10:25:50 <ikke> Does it matter from where you purge an entry?
2020-06-03 10:37:33 <clandmeter> i dont think it matters
2020-06-03 10:37:45 <clandmeter> at least i could not find any ref that it does
2020-06-03 10:43:10 <clandmeter> ncopa: on new release you could do something like this https://docs.fastly.com/en/guides/wildcard-purges#via-the-api
2020-06-03 11:01:11 <ncopa> clandmeter: thanks
2020-06-03 12:47:56 <clandmeter> ncopa: would it be an option to add a few of the release steps to gitlab CI?
2020-06-03 12:48:02 <clandmeter> not sure which ones you currently make
2020-06-03 12:59:48 <ncopa> the release steps are documented on wiki.alpin.pw
2020-06-03 12:59:58 <ncopa> http://wiki.alpin.pw/releng/release-checklist.html
2020-06-03 13:00:16 <ncopa> i guess it should move to somewhere
2020-06-03 13:09:51 <clandmeter> ok i think we can make an issue with that list and see what we can automate
2020-06-03 13:17:57 <clandmeter> ncopa: you normally tag from the new branch i guess?
2020-06-03 13:18:44 <clandmeter> i mean first branch than tag
2020-06-03 13:26:23 <ncopa> no. opposite
2020-06-03 14:55:05 <ncopa> hum.... we new location to store ca-certificates tarballs https://build.alpinelinux.org/buildlogs/build-3-10-aarch64/main/ca-certificates/ca-certificates-20191127-r2.log
2020-06-03 15:16:32 <ikke> ncopa: Can we move / host it on gitlab?
2020-06-03 15:19:04 <ikke> clandmeter: I created this based on that list: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11591
2020-06-03 15:23:39 <clandmeter> Nice
2020-06-03 17:15:41 <Cogitri> ppc64le is stuck again :c
2020-06-04 06:08:59 <clandmeter> ikke: do we have another not version copy of: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11591
2020-06-04 06:09:36 <clandmeter> It is not complete
2020-06-04 06:09:53 <clandmeter> We should add thinks like pkgs.a.o
2020-06-04 08:00:37 <ncopa> under what namespace should have host ca-certificates? alpine/ca-certificates? It is kind of alpine specific.
2020-06-04 08:00:56 <ikke> I would say so, yes
2020-06-04 08:09:57 <ncopa> moved
2020-06-04 08:10:02 <ikke> nice
2020-06-04 08:12:45 <ncopa> question about the download tarballs from  https://gitlab.alpinelinux.org/alpine/ca-certificates/-/tags
2020-06-04 08:12:52 <ncopa> will they have the same checksum?
2020-06-04 08:13:14 <ncopa> they seem to have same checksum now, but what heppens when gitlab is updated?
2020-06-04 08:16:49 <ncopa> i wonder if we should have a source archive like https://dev.alpinelinux.org/archive/ where we can upload dist tarballs
2020-06-04 08:17:11 <ncopa> or if gitlab's download by tag is enough
2020-06-04 08:32:11 <ikke> download by tag should be enough
2020-06-04 08:33:41 <ikke> We've been using that already
2020-06-04 08:34:19 <ncopa> ok
2020-06-04 08:34:36 <ikke> You can also create releases in gitlab
2020-06-04 09:02:27 <ncopa> ugh.. i ran out of diskspace on armv7
2020-06-04 09:05:41 <ncopa> i cleaned up distfiles
2020-06-04 10:34:50 <ikke> clandmeter: No, there is not, we should store a template somewhere
2020-06-04 11:36:17 <clandmeter> ncopa: I copied distfiles to another location
2020-06-04 11:36:45 <clandmeter> Should we move it and add logic for builders to use and update it?
2020-06-04 11:58:23 <ncopa> um
2020-06-04 11:58:38 <ncopa> i dont think its a priority
2020-06-04 11:59:07 <ncopa> the problem was on armv7 distfiles
2020-06-04 11:59:56 <clandmeter> ncopa: that is exactly what it would solve
2020-06-04 12:01:15 <ncopa> we would still want a local cache?
2020-06-04 12:01:34 <ncopa> a local shared cache
2020-06-04 12:02:06 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10658
2020-06-04 12:02:36 <clandmeter> i dont see the adv of having long lived cached distfiles on builders
2020-06-04 12:03:05 <clandmeter> storage is a limitation we are fighting all the time, bw is not.
2020-06-04 12:03:16 <ncopa> right, but we still want a short lived local cache
2020-06-04 12:03:22 <clandmeter> yes
2020-06-04 12:03:36 <clandmeter> a week or month is fine with me
2020-06-04 12:03:44 <clandmeter> its tuneable
2020-06-04 12:04:04 <ncopa> the problem is that we dont clean up current cache
2020-06-04 12:04:38 <clandmeter> i rsynced distfiles to a diff location. its around 250GB
2020-06-04 12:04:41 <clandmeter> the disk is 1TB
2020-06-04 12:05:29 <ncopa> we'd still need to clean it regularily
2020-06-04 12:05:41 <clandmeter> where? builders you mean?
2020-06-04 12:05:46 <ncopa> everywhere
2020-06-04 12:06:05 <clandmeter> builders would auto clean
2020-06-04 12:06:12 <clandmeter> keep cache for x days
2020-06-04 12:06:20 <clandmeter> weeks months whatever
2020-06-04 12:06:41 <clandmeter> not sure how fast distfiles would grow
2020-06-04 12:07:02 <clandmeter> but its only a single place we need to monitor.
2020-06-04 12:07:07 <ncopa> the immediate problem can be solved with a daily cron job on builders doing: find /var/cache/distfiles -mtime +7 -delete
2020-06-04 12:07:16 <ncopa> builder hosts
2020-06-04 12:07:27 <clandmeter> yes but we would lose some src tarballs
2020-06-04 12:07:33 <clandmeter> thats the idea of distfiles i guesS?
2020-06-04 12:09:21 <ncopa> brb. lunch
2020-06-04 12:45:20 <ncopa> on the builders it does not matter if they are deleted
2020-06-04 12:45:38 <ncopa> but we need to keep the sources for stable releases longterm
2020-06-04 12:46:01 <ncopa> we dont need to do that on every builder, so thats why we have distfiles.a.o
2020-06-04 12:46:11 <ncopa> but we dont need to keep everytying for edge
2020-06-04 12:46:51 <ncopa> but its more complicated to delete from distfiles.alpinelinux.org
2020-06-04 12:50:47 <ncopa> im more worried about the central release info
2020-06-04 12:50:52 <ncopa> how and where and what
2020-06-04 12:51:07 <ncopa> where do we want it?
2020-06-04 12:51:24 <ncopa> i guess either alpinelinux.org/releases.json or on the mirrors
2020-06-04 12:52:17 <ncopa> or both
2020-06-04 12:54:00 <ncopa> as i understand the major problem we want to solve is: what release branches are currently supported and what architectures do we have for each release branch
2020-06-04 12:56:57 <clandmeter> i think www would go good
2020-06-04 12:58:14 <clandmeter> regrading distfiles we could generate a list of files from aports and clean whatever is not used in $source anymore.
2020-06-04 12:58:31 <clandmeter> do that based on supported versions
2020-06-04 12:58:44 <ncopa> correct
2020-06-04 12:58:48 <clandmeter> which we fetch from releases.json ;-)
2020-06-04 13:02:14 <clandmeter> we also have mirrors.a.o on which you could host it, but i think i would prefer a.o
2020-06-04 14:10:46 <ncopa> i think a.o makes most sense
2020-06-04 19:06:58 <clandmeter> ikke: new patch releases out for gitlab
2020-06-04 19:07:18 <clandmeter> https://cloud.drone.io/alpinelinux/alpine-docker-gitlab/
2020-06-04 19:07:26 <ikke> clandmeter: ok!
2020-06-05 05:38:35 <ncopa> morning
2020-06-05 05:39:01 <ncopa> I got this in a private message: I was looking into secdb.alpinelinux.org lately - and its seems like it wan't updated for a while since the 7th of May.
2020-06-05 05:50:31 <ikke> the mqtt topic needs to be adjusted
2020-06-05 05:55:09 <ikke> i've adjusted it to gitlab/push/alpine/aports
2020-06-05 05:56:01 <ikke> hmm, but that's not enough
2020-06-05 05:56:06 <ikke> it expects the branchname in the topic
2020-06-05 06:49:40 <clandmeter> ikke: did you update the topic on secdb?
2020-06-05 07:22:39 <clandmeter> i think git.a.o is having issues
2020-06-05 07:22:54 <clandmeter> probably overloaded
2020-06-05 07:24:25 <clandmeter> aha
2020-06-05 07:24:41 <clandmeter> that explains things :)
2020-06-05 07:31:04 <ikke> I did, but then I noticed the script is looking for the branch name in the topic
2020-06-05 07:31:29 <clandmeter> the topic was correct
2020-06-05 07:31:45 <clandmeter> we kept that topic for bw compat
2020-06-05 07:32:02 <clandmeter> so the builders didnt need an update (and others)
2020-06-05 07:32:15 <clandmeter> the problem is it could not fetch from git.a.o
2020-06-05 07:32:25 <clandmeter> because cgit runs local...
2020-06-05 07:52:00 <ikke> Aha, ok
2020-06-05 07:54:43 <clandmeter> heh
2020-06-05 07:55:00 <clandmeter> ncopa: we really need that releases.json
2020-06-05 07:55:13 <clandmeter> seems your secdb scripts also needs manual fixes
2020-06-05 08:16:09 <clandmeter> ncopa: you can mention secdb is up2date again after some patching. 
2020-06-05 10:00:13 <ncopa> ok
2020-06-05 10:00:26 <ncopa> anyone knows what happened with https://alpinelinux.org/posts/Alpine-3.11.6-released.html in git repo?
2020-06-05 10:00:53 <ncopa> i cannot find any traces of the posts/Alpine-3.11.6-released.md
2020-06-05 10:01:58 <ncopa> i suspect there was a push --force or similar
2020-06-05 10:03:04 <ncopa> oh
2020-06-05 10:03:13 <ncopa> looks like i pushed to git-old
2020-06-05 10:03:16 <ncopa> https://git-old.alpinelinux.org/alpine-mksite/
2020-06-05 10:04:02 <ncopa> but commits are missing in gitlab: https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/commits/master
2020-06-05 11:29:56 <ncopa> hum
2020-06-05 11:30:03 <ncopa> wwwtest.a.o is not auto updated
2020-06-05 11:30:20 <ncopa> I suppose I need to fix the mqtt topic it listens to?
2020-06-05 11:30:46 <ncopa> I never updated it during 3.12 release
2020-06-05 11:39:40 <ikke> according to clandmeter the original topics should still be anounced
2020-06-05 11:42:19 <ncopa> wwwtest.alpinelinux.org/releases.json
2020-06-05 11:42:46 <ncopa> this should now be the source of truth
2020-06-05 11:55:20 <clandmeter> I can't see it on mobile
2020-06-05 16:34:11 <ncopa> i pushed to alpine/infra/alpine-mksite production branch
2020-06-05 16:34:24 <ncopa> but https://alpinleinux.org does not seem to be updated
2020-06-05 16:34:52 <ncopa> https://alpinelinux.org/downloads/ should have mips64 on minirootfs
2020-06-05 16:35:25 <ncopa> once that is fixed there should also be a https://alpinelinux.org/releases.json
2020-06-05 18:09:14 <ikke> clandmeter: neostrada is out-of-date for more then 14 days and no response on the e-mail
2020-06-05 18:29:09 <clandmeter> ncopa: i see mips but not releases.json
2020-06-05 18:29:23 <clandmeter> ikke: ok we can disable it i guess
2020-06-05 18:35:42 <clandmeter> ikke: we only have bw compat topic for aports
2020-06-05 18:35:50 <clandmeter> all others need to adjust to new format
2020-06-05 18:38:52 <ikke> clandmeter: aha, ok
2020-06-05 18:40:42 <clandmeter> i upated the topic on www to gitlab/push/alpine/infra/alpine-mksite
2020-06-05 18:40:59 <clandmeter> i dont think it matters that we dont listen for specific branch
2020-06-05 18:47:35 <clandmeter> looks like it was also missing cjson
2020-06-05 18:49:57 <clandmeter> looks like its working now
2020-06-05 18:50:29 <clandmeter> would be nice if darkhttp would accept json as mimetype
2020-06-05 18:52:35 <clandmeter> done
2020-06-05 18:53:22 <clandmeter> https://alpinelinux.org/releases.json
2020-06-05 18:54:03 <clandmeter> ncopa: Muito obrigado
2020-06-05 20:01:29 <Ariadne> so for the online conference, lets say we go with OpenCFP for that, should i prepare a docker image for that or what?  i can host it myself if that is easier
2020-06-05 20:50:09 <ikke> Hosting it should not be an issue
2020-06-05 20:51:04 <ikke> I was already thinking about making a docker compose project myself
2020-06-07 10:25:23 <ikke> clandmeter: ping
2020-06-07 12:13:23 <clandmeter> ikke: o/
2020-06-07 12:35:29 <ikke> I just pushed the db dump commit to 12.10
2020-06-07 12:37:07 <ikke> Going to run that on the test server
2020-06-07 12:37:14 <ikke> If that works, then deploying in prod
2020-06-07 12:51:59 <ikke> clandmeter: https://gitlab-test.alpinelinux.org/
2020-06-07 12:55:27 <clandmeter> Ok. I'm not home
2020-06-07 12:55:37 <ikke> ok
2020-06-07 12:55:51 <ikke> One test is failing, but might be related to an API change
2020-06-07 13:08:24 <ikke> aha, found a better way
2020-06-07 13:14:52 <ikke> clandmeter: do you agree upgrading gitlab tonight to 12.10?
2020-06-07 13:28:00 <clandmeter> Yes please
2020-06-07 16:06:17 <ikke> Done, succesfully upgraded
2020-06-07 16:06:24 <ikke> And response time is good
2020-06-07 16:22:20 <clandmeter> Nice :)
2020-06-07 16:23:15 <clandmeter> ikke: did you add the announcement?
2020-06-07 16:23:34 <ikke> yes
2020-06-07 16:23:48 <clandmeter> :)
2020-06-07 19:38:31 <clandmeter> oh this is the version with fancy fiel/folder icons
2020-06-07 19:39:19 <ikke> yes
2020-06-07 20:16:42 <ikke> clandmeter: I've just created a zabbix go agent plugin for openrc services
2020-06-07 20:17:22 <clandmeter> what does that mean?
2020-06-07 20:17:43 <ikke> https://tpaste.us/6Vlr 
2020-06-07 20:18:59 <ikke> clandmeter: It means you can add 'native' items to the agent without having to rely on scripts
2020-06-07 20:19:57 <ikke> Just wanted to see what it takes to create a plugin
2020-06-07 20:20:26 <ikke> There already exists a plugin for systemd, but not for openrc
2020-06-07 20:20:41 <ikke> I just created that (uses rc-status -f ini -a)
2020-06-07 20:27:14 <clandmeter> i still dont understand what it actually does?
2020-06-07 20:27:23 <clandmeter> what will you monitor?
2020-06-07 20:27:42 <clandmeter> the status of a service?
2020-06-07 20:27:48 <ikke> It can discover what services are present on a host and it can return the sate
2020-06-07 20:27:50 <ikke> state
2020-06-07 20:27:52 <ikke> yes
2020-06-07 20:28:15 <clandmeter> i didnt know about -f
2020-06-07 20:28:18 <ikke> https://tpaste.us/V4xJ 
2020-06-07 20:28:41 <ikke> I just ran rc-status -h :)
2020-06-07 20:28:55 <clandmeter> me2 but it didnt show -f :)
2020-06-07 20:29:09 <clandmeter> looks like a more recent addition 
2020-06-07 20:29:38 <ikke> aha, ok
2020-06-07 20:29:41 <ikke> what is recent?
2020-06-07 20:29:53 <clandmeter> well my local server is 3.8 :)
2020-06-07 20:29:56 <clandmeter> too lazy to upgrade
2020-06-07 20:32:22 <ikke> That is anoying though
2020-06-07 20:39:17 <clandmeter> https://github.com/OpenRC/openrc/commit/427a1ce2995b376ed6d112c5c5b422217f815fbb
2020-06-07 20:39:48 <clandmeter> 0.41
2020-06-07 20:44:33 <ikke> so >= 3.10
2020-06-08 07:35:46 <Cogitri> Did the host to SSH into for dev.a.o change?
2020-06-08 07:35:58 <Cogitri> Ah, it's git-old
2020-06-08 08:15:29 <clandmeter> ?
2020-06-08 08:44:23 <clandmeter> ikke: interesting stats: https://gitlab.alpinelinux.org/admin/dashboard/stats
2020-06-08 08:49:43 <Cogitri> clandmeter: I tried to ssh into git.a.o to get a tarball hosted on dev.a.o, but I had to ssh into git-old.a.o
2020-06-08 08:50:03 <clandmeter> why not ssh into dev.a.o?
2020-06-08 08:50:06 <ikke> You can use dev.a.o as host directly 
2020-06-08 08:53:56 <Cogitri> Oh :D
2020-06-08 08:54:04 <ikke> clandmeter: it shows 0 for me everywhere. For you as well? 
2020-06-08 08:54:15 <clandmeter> yup
2020-06-08 08:54:22 <clandmeter> it was sarcasm :)
2020-06-08 09:14:48 <ikke> Aha, my sarcasm detector is broken 
2020-06-08 10:18:30 <clandmeter> yeah sarcasm is difficult to detect properly. probably need strace or gdb. :)
2020-06-08 10:23:00 <mps> I learned to detect it by looking at nick ;)
2020-06-08 10:24:29 <ikke> aha, a machine learning model
2020-06-08 10:25:15 <clandmeter> mps: :D
2020-06-08 10:25:21 <mps> ikke: yes :D
2020-06-08 10:28:21 <clandmeter> mps: your methods are so old school 
2020-06-08 10:29:01 <ikke> clandmeter: You don't want him to deploy some kind of adverserial network :P
2020-06-08 10:29:26 <mps> hehe, again easy to detect sarcasm looking at nick :P
2020-06-08 10:29:41 <mps> clandmeter: ^
2020-06-08 10:30:13 <mps> you just confirmed my method works :)
2020-06-08 10:37:25 <clandmeter> ikke: if ppl complain they lost their account, i just removed a lot of spam
2020-06-08 10:38:20 <ikke> clandmeter: nod
2020-06-08 10:50:39 <danieli> I've been so exhausted lately my sarcasm-o-meter is out of order
2020-06-08 11:57:35 <ncopa> im adding a version or timestamp to generated secdb
2020-06-08 11:57:52 <ncopa> any opinion on the format?
2020-06-08 11:58:24 <ncopa> to make it simple we could let it be epoc (seconds since 1 jan 1970)
2020-06-08 11:58:27 <mps> timestamp, imo
2020-06-08 11:59:06 <ncopa> alternatively it could be "yyyy.mm.dd.ssss"
2020-06-08 11:59:26 <mps> ISO 8601
2020-06-08 11:59:38 <mps> yyyy-mm-dd
2020-06-08 11:59:53 <mps> but '-' could be problem for versioning
2020-06-08 12:00:08 <mps> yyyymmddssss
2020-06-08 12:00:09 <mps> ?
2020-06-08 12:00:31 <mps> just thinking aloud
2020-06-08 12:00:53 <ncopa> if it is for parsing by machine, i guess epoch is the simplest
2020-06-08 12:01:03 <ncopa> but it may be nice to make it somewhat human readable too
2020-06-08 12:01:10 <mps> sure
2020-06-08 12:01:35 <ncopa> 2020-06-08T09:08:54Z
2020-06-08 12:01:37 <mps> so, maybe your idea "yyyy.mm.dd.ssss" is best
2020-06-08 12:01:53 <ncopa> i think using an established standard is not bad idea at all
2020-06-08 12:02:26 <mps> this '2020-06-08T09:08:54Z' is parseable with some (most?) tool and libs
2020-06-08 12:02:42 <ncopa> yeah
2020-06-08 12:04:20 <mps> actually, I prefer this, but for 'show' it to humans usually add some script/method to display it in something like "yyyy.mm.dd.ssss" 
2020-06-08 12:10:13 <mps> and to add, unix timestamp is fine for internal use but not as fine for showing to humans
2020-06-08 12:39:46 <mps> hmm, '* Updated cty.dat 20200418 (cty-3007)
2020-06-08 12:39:59 <mps> http://xlog.nongnu.org/xlog.changelog
2020-06-08 12:40:13 <mps> about timestamp for sec db
2020-06-08 13:13:18 <clandmeter> oops
2020-06-08 13:13:22 <clandmeter> i pushed the wrong button
2020-06-08 13:22:57 <ikke> What did that wrong button do?
2020-06-08 13:24:54 <clandmeter> i have no idea
2020-06-08 13:25:07 <clandmeter> i pushed the spam report button for an entire issue
2020-06-08 13:25:12 <clandmeter> instead of the comment
2020-06-08 13:25:42 <clandmeter> i dont understand why it doesnt have a rn-rf button directly
2020-06-08 13:25:49 <clandmeter> for the comment that is
2020-06-08 13:26:08 <ikke> That used might now be bugged with captchas
2020-06-08 13:26:34 <clandmeter> could be
2020-06-08 13:26:47 <clandmeter> algitbot will have a headache
2020-06-08 13:27:43 <ikke> The spamlog is useless as well
2020-06-08 13:28:01 <ikke> no way to see the complete comment / content
2020-06-08 13:28:26 <clandmeter> yes its a mess
2020-06-08 13:28:33 <clandmeter> i tried looking for my report
2020-06-08 13:28:40 <clandmeter> if it would be included
2020-06-08 13:28:44 <clandmeter> but i gave up
2020-06-08 13:29:43 <clandmeter> tbh i dont even know what it really does, the topic spam button.
2020-06-08 13:29:58 <clandmeter> i guess its different than report this is spam feature
2020-06-08 15:29:13 <ncopa> I have added an MR that should use releases.json for secdb. https://gitlab.alpinelinux.org/alpine/infra/docker/secdb/-/merge_requests/1
2020-06-08 15:29:29 <ncopa> it will automatically add 'edge' and future new releases (given thye are added to releases.json)
2020-06-08 15:29:51 <ncopa> oh
2020-06-08 15:30:35 <ncopa> i forgot. we should add a README.md to that project that tells that downstream users should use https://secdb.alpinelinux.org
2020-06-08 15:39:19 <ikke> Do we know who is behind that secdb scanner? Maybe we can contact them to point that out
2020-06-09 03:28:15 <maldridge> I really need to setup certfp auth so I don't bounce off of channels like this one
2020-06-09 08:00:55 <ncopa> ikke I think it is this one: https://github.com/quay/clair
2020-06-09 08:01:55 <ncopa> they have a list of issues about alpine: https://github.com/quay/clair/issues?q=is%3Aissue+is%3Aopen+alpine
2020-06-09 08:03:16 <ikke> github search does not reveal that they clone secdb from git.a.o
2020-06-09 08:08:53 <ikke> ah: https://github.com/quay/clair/blob/c39101e9b8206401d8b9cb631f3aee47a24ab889/ext/vulnsrc/alpine/alpine.go#L38
2020-06-09 08:09:53 <ikke> So apparently they already use github, but I guess older versions still use git.a.o
2020-06-09 08:10:06 <ncopa> https://github.com/quay/clair/blob/master/ext/vulnsrc/alpine/alpine.go#L38
2020-06-09 08:10:40 <ikke> yeah, just fond it
2020-06-09 08:10:43 <ncopa> we should send them a feature request to switch to https://secdb.alpinelinux.org
2020-06-09 08:11:14 <ncopa> im working on the gnutls secfix now
2020-06-09 08:11:24 <ikke> The one about the tls session ticket?
2020-06-09 08:11:32 <ncopa> i think so yes
2020-06-09 08:11:44 <ncopa> https://gitlab.alpinelinux.org/alpine/aports/-/issues/11627
2020-06-09 10:46:25 <ikke> ncopa: question. I want to use the new zabbix go agent, which supports native plugins. But you have to build a dedicated version of the agent with those plugins added. Do so any issues adding a dedicated zabbix-agent2-alpine package for that?
2020-06-09 10:46:28 <ikke> to aports
2020-06-09 10:46:53 <ikke> it builds from the same source as zabbix itself, but just builds the agent with the plugins
2020-06-09 10:47:21 <ikke> https://tpaste.us/LYlK
2020-06-09 10:50:08 <clandmeter> ikke: are we no longer using 80 chars line limit for aports?
2020-06-09 10:51:31 <ikke> I think we still are, though, some proposed to either remove it, or make it more lenient
2020-06-09 10:52:19 <ncopa> what are thouse plugins?
2020-06-09 10:52:35 <ncopa> why cant we enable them in zabbix-agent package?
2020-06-09 10:53:38 <ikke> ncopa: You can add your own custom plugins. In this case for example, I've added a plugin to add native support for monitoring openrc services (requires >openrc-41.0).
2020-06-09 10:54:03 <ikke> The advantage is that you don't need to deploy external scripts with all necessary dependencies
2020-06-09 10:55:38 <ikke> We could build / host it internally, but then we need to make sure we can build it for all arches
2020-06-09 10:56:24 <ikke> We could add the plugins to the already existing agent, but then that agent is no longer 'vanilla'
2020-06-09 10:58:11 <ncopa> those plugins are build-time plugins?
2020-06-09 10:58:14 <ncopa> not runtiem lugins
2020-06-09 10:58:18 <ikke> yes, correct
2020-06-09 10:58:18 <ncopa> runtime*
2020-06-09 10:58:20 <ncopa> ok
2020-06-09 10:59:01 <ncopa> are there any reason to not include them in the "vanilla" package, as distro patch?
2020-06-09 10:59:46 <ncopa> i guess we shoudl shipe an zabbix-agent-alpine build, with the alpine specific plugins enabled
2020-06-09 11:00:34 <ikke> The plugins themselves will mostly be relevant to anyone who uses alpine, so in that sense, it would make sense to include them
2020-06-09 11:00:51 <ncopa> that is what im thinking
2020-06-09 11:01:10 <ncopa> meeting. brb
2020-06-09 11:01:30 <ikke> ok
2020-06-09 11:50:36 <ncopa> back
2020-06-09 11:51:43 <ncopa> so the only drawback i can see for including alpine specific plugins to zabbix is if we introduce any vulnerabilities
2020-06-09 12:00:20 <ikke> yes
2020-06-09 12:13:59 <ncopa> i think im fine with either way
2020-06-09 12:14:12 <ikke> ok
2020-06-09 12:14:33 <ncopa> the major benefit of keeping it separate apkbuild is that you can update the plugin without needing to rebuild everything
2020-06-09 12:14:59 <ncopa> but the cost is that you need to remember to update the plugin everytime zabbix is updated
2020-06-09 12:15:14 <ikke> yes, I'm well aware
2020-06-09 12:15:19 <ncopa> can be fixed with relgroups once that is implemented
2020-06-09 13:09:50 <ncopa> I haven an idea for another alpine zabbix plugin, that compares installed packages with secdb
2020-06-09 13:10:15 <ncopa> and generate alert or similar when tehre is something that needs to be updated
2020-06-09 15:01:53 <maldridge> ncopa: what updates secdb?  is it just a seperate repo that maintainers keep track of?
2020-06-09 15:03:45 <ncopa> maldridge: https://gitlab.alpinelinux.org/alpine/infra/docker/secdb
2020-06-09 15:22:22 <maldridge> ah, so it parses the APKBUILDs
2020-06-09 15:42:50 <ncopa> yes
2020-06-09 16:23:43 <ikke> ncopa: I think I will first add a separate agent to testing to get a bit more experience / feeling with it and it allows me to iterate more quickly (without having to push a lot of pkgrel bumps to zabbix)
2020-06-09 18:50:33 <HRio> is the gitlab runner for ppc64le down or just busy?
2020-06-09 18:51:04 <ikke> most likely busy
2020-06-09 18:51:09 <ikke> but it can be stuck at some jobs
2020-06-09 18:51:16 <HRio> it started now
2020-06-09 18:51:22 <ikke> ok, good
2020-06-09 18:51:34 <ikke> yeah, I don't see any jobs that are stuck
2020-06-09 18:51:43 <ikke> but it only handles 2 concurrent jobs
2020-06-09 18:53:20 <HRio> ah
2020-06-09 18:54:12 <ikke> 10 jobs running, 11 pending
2020-06-09 18:57:07 <HRio> backport MRs to stable (3.11 and 3.12) pushed as well for intal-ucode
2020-06-09 18:57:32 <HRio> O do not usually merge in stable... 
2020-06-09 18:57:34 <HRio> I
2020-06-09 18:58:19 <HRio> not sure what the rules are (I mostly merge in edge testing and commuity)
2020-06-09 18:58:40 <ikke> rules are quite simple actually :)
2020-06-09 19:00:52 <ikke> If it's just a security patch, then you just go ahead and push it
2020-06-09 19:01:28 <ikke> cpu firmware ofcourse is a bit different
2020-06-09 19:02:06 <HRio> yes its just blobs
2020-06-09 19:03:42 <HRio> should I do v3.10 as well
2020-06-09 19:03:57 <ikke> and I guess also 3.9
2020-06-09 19:03:58 <HRio> thats the first release we shipped intel-ucode
2020-06-09 19:04:01 <ikke> ah
2020-06-09 19:04:03 <ikke> ok
2020-06-09 19:06:16 <HRio> should we backport the package to v3.9?
2020-06-09 19:06:26 <ikke> I don't know
2020-06-09 19:06:26 <maxice8> is it in main/ ?
2020-06-09 19:06:41 <HRio> i.e. move from testing to community and then update?
2020-06-09 19:06:53 <HRio> no its just in community
2020-06-09 19:07:12 <ikke> sounds like something that belongs in main
2020-06-09 19:08:00 <HRio> yes I think so too
2020-06-09 19:09:26 <ikke> But users still need to do something to load it, or is having it in /boot enough?
2020-06-09 19:10:03 <HRio> I have patched grub to look for it and add it if installed
2020-06-09 19:10:20 <HRio> thats in the alpine grub package
2020-06-09 19:10:39 <ikke> ok
2020-06-09 19:11:03 <ikke> But I guess that was also added in v3.10
2020-06-09 19:11:38 <HRio> for xen 4.13 and later it can be loaded at runtime. for the older versions i add it first in initramfs
2020-06-09 19:13:12 <HRio> I added it "Thu May 16 2019" when was v3.9? 
2020-06-09 19:13:37 <ikke> 2019-01-29
2020-06-09 19:13:56 <HRio> git tag --contains gives v3.10.0
2020-06-09 19:14:12 <HRio> yes, so possibly to big for v3.9
2020-06-09 19:14:56 <HRio> I have no v3.9 left that boots with grub
2020-06-09 20:24:32 <Cogitri> Are there special accounts for bots in Gitlab?
2020-06-09 20:29:01 <ikke> Under user statics there is an item for bots, but I haven't found how to create them. 
2020-06-09 20:31:23 <ikke> Apparently according to an open ticket there is no special bot user type
2020-06-09 20:31:54 <Cogitri> Ah okie
2020-06-09 20:32:17 <Cogitri> I'll just make a new normal user for testing my bot thingie once I get to making that then
2020-06-10 06:52:09 <clandmeter> ncopa: did you recently remove 2.x from master mirror?
2020-06-10 06:53:22 <ncopa> not recently
2020-06-10 06:53:24 <ncopa> some time ago
2020-06-10 06:59:55 <clandmeter> ok
2020-06-10 07:01:39 <clandmeter> ikke: the ci templates, did something change? 
2020-06-10 07:02:29 <clandmeter> it should only work for the branch that is selected right?
2020-06-10 07:02:51 <clandmeter> branch as in ref
2020-06-10 09:42:50 <clandmeter> 
ACTION slaps ikke with a CI pipeline

2020-06-10 09:45:44 <ikke> ouch
2020-06-10 09:45:52 <ikke> "branch that is selected"?
2020-06-10 12:11:51 <clandmeter> ikke: yes in the yml
2020-06-10 12:11:58 <clandmeter> It set a ref
2020-06-10 12:12:10 <clandmeter> I guess it acts on that ref only
2020-06-10 12:12:35 <ikke> With something like  `only:` or `rules:`?
2020-06-10 12:37:40 <clandmeter> ikke: https://gitlab.alpinelinux.org/ncopa/secdb/-/tree/releases-json
2020-06-10 12:37:58 <clandmeter> why does it run ci when the yml targets master?
2020-06-10 12:39:35 <ikke> ref: master there means the master branch of alpine/infra/gitlab-ci-templates
2020-06-10 12:39:43 <ikke> meaning, that's where it will look for the file to include
2020-06-10 12:39:54 <ikke> It does not mean only run on master
2020-06-10 12:40:03 <ikke> It's part of the include statement
2020-06-10 12:40:11 <clandmeter> ah ok, sorry.
2020-06-10 13:06:51 <clandmeter> ikke: should be exclude PR's for that type of ci?
2020-06-10 14:06:36 <ikke> The default mode is to operate on branch pushes, which is what we want in most of the cases
2020-06-10 14:23:22 <clandmeter> so if we create a PR it will be triggered it seems.
2020-06-10 14:23:44 <ikke> Usually it's already triggered before the PR runs
2020-06-10 14:23:48 <ikke> ie, when you push the branch
2020-06-10 14:24:09 <clandmeter> ok
2020-06-10 14:24:12 <clandmeter> so can we limit that?
2020-06-10 14:24:26 <ikke> You only want it to happen on merge requests?
2020-06-10 14:24:37 <clandmeter> no only on pushes to master i guess
2020-06-10 14:24:47 <clandmeter> this ci is to generate a docker image
2020-06-10 14:25:05 <clandmeter> we could build it, but not push it.
2020-06-10 14:25:41 <ikke> I only see verify and build jobs
2020-06-10 14:25:45 <ikke> that's already what happens
2020-06-10 14:25:49 <ikke> push is limited to master
2020-06-10 14:26:16 <ikke> https://gitlab.alpinelinux.org/alpine/infra/gitlab-ci-templates/-/blob/master/docker-image.yml#L135
2020-06-10 14:26:28 <clandmeter> ah nice
2020-06-10 14:26:41 <clandmeter> so it only errors cause no runners are assigned 
2020-06-10 14:26:55 <ikke> yes
2020-06-10 14:27:12 <ikke> need to enable shared runners
2020-06-10 14:27:46 <ikke> https://gitlab.alpinelinux.org/ncopa/secdb/-/settings/ci_cd
2020-06-10 14:27:56 <ikke> Under runners, there is a green button: enable shared runners
2020-06-10 14:28:04 <clandmeter> nod
2020-06-10 14:40:40 <ikke> Should I enable that for secdb?
2020-06-10 18:54:06 <ikke> I just did :)
2020-06-11 14:41:02 <PureTryOut[m]> Is the s390x Gitlab CI runner down?
2020-06-11 14:41:47 <ikke> nope, just busy
2020-06-11 16:45:12 <mps> is something changed with dev.a.o, I cannot ssh there 'ps@dev.alpinelinux.org: Permission denied (publickey)'
2020-06-11 16:45:28 <mps> mps*
2020-06-11 18:12:17 <mps> anyone ^
2020-06-11 18:18:41 <ikke> should not
2020-06-11 18:18:48 <ikke> let me check
2020-06-11 18:20:41 <ikke> I only see these messages: "Connection closed by authenticating user mps <ip> port 43416 [preauth]"
2020-06-11 18:21:53 <mps> hmm, I use same key which I use for gitlab and lxc containers
2020-06-11 18:23:21 <ikke> The key there is different than you have on gitlab
2020-06-11 18:28:09 <ikke> all sorted out :)
2020-06-11 18:29:35 <mps> yes, thanks
2020-06-11 20:44:46 <clandmeter> ikke: do you smell gitlab?
2020-06-12 00:56:37 <Ariadne> ikke: can you make https://gitlab.alpinelinux.org/kaniini/autoconf-policy public
2020-06-12 04:48:30 <ikke> clandmeter: Yeah, i do think I smell something
2020-06-12 04:52:01 <ikke> Ariadne: done
2020-06-12 04:52:09 <Ariadne> thx
2020-06-12 06:17:36 <HRio> any problem with the builders for 3.10?
2020-06-12 06:19:04 <HRio> last upload to 3.10 seems to be nextcloud, but several commits after that https://gitlab.alpinelinux.org/alpine/aports/-/commits/3.10-stable/
2020-06-12 06:28:22 <ikke> algitbot: retry 3.10-stable
2020-06-12 06:48:24 <HRio> ikke: thx
2020-06-12 06:48:57 <ikke> Is it alright now?
2020-06-12 06:50:45 <HRio> looks like https://build.alpinelinux.org/buildlogs/build-3-10-x86_64/main/xen/
2020-06-12 06:51:03 <ikke> yeah, I saw it building xen
2020-06-12 06:51:58 <ikke> ok, the mirror is now copletely gone apparently
2020-06-12 06:52:01 <ikke> time to remove it
2020-06-12 06:54:10 <HRio> the "Read more" link here https://alpinelinux.org/ returns empty
2020-06-12 06:54:30 <HRio> under "ALPINE NEWS"
2020-06-12 07:27:16 <ikke> HRio: could you create an issue here? https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite
2020-06-12 07:43:29 <clandmeter> HRio: less is more ;-)
2020-06-12 07:55:38 <HRio> issue #1 :-)
2020-06-12 21:39:40 <Ariadne> can we get newedge builders uploading their build logs?
2020-06-13 05:28:50 <ikke> clandmeter: s390x-ci host appears down / unreachable
2020-06-13 05:29:23 <ikke> and the builder as well
2020-06-13 05:30:07 <ikke> tmhoang: ping
2020-06-13 12:38:29 <clandmeter> ikke: both are still down?
2020-06-13 12:39:03 <ikke> yes
2020-06-13 12:43:28 <clandmeter> email send
2020-06-13 12:43:39 <ikke> thanks
2020-06-13 13:33:47 <ikke> ah, great
2020-06-13 13:34:31 <ikke> The builder is not yet back, thoguh
2020-06-13 13:50:08 <clandmeter> ikke: ping
2020-06-13 13:51:45 <ikke> pong
2020-06-13 13:52:27 <ikke> clandmeter: ^
2020-06-13 13:53:50 <ikke> builder is still down
2020-06-13 13:54:06 <clandmeter> Did you see the mail?
2020-06-13 13:54:14 <clandmeter> Should be up
2020-06-13 13:54:16 <ikke> I have not seen a response
2020-06-13 13:54:25 <ikke> ah, spam..
2020-06-13 13:55:23 <ikke> --- s390x.alpinelinux.org ping statistics ---
2020-06-13 13:55:25 <ikke> 12 packets transmitted, 0 received, 100% packet loss, time 11137ms
2020-06-13 13:56:33 <clandmeter> Maybe fw?
2020-06-13 13:56:45 <clandmeter> Ssh also down?
2020-06-13 13:57:59 <clandmeter> I don't see a notification here about it being down
2020-06-13 13:58:09 <ikke> d'oh, I can ssh into it
2020-06-13 13:58:11 <ikke> just ping
2020-06-13 13:58:58 <clandmeter> :)
2020-06-13 13:59:15 <clandmeter> It happens to the best of us
2020-06-13 13:59:36 <ikke> But the builder is not active for some reason either
2020-06-13 13:59:45 <ikke> Oh, ok, I see it in the logs now
2020-06-13 14:00:50 <clandmeter> Ok I'll send email
2020-06-13 14:00:54 <ikke> I just did :)
2020-06-13 14:01:26 <clandmeter> Ok
2020-06-13 14:01:47 <clandmeter> Happy campers
2020-06-13 14:01:49 <ikke> wondering why it's not responding to ping though
2020-06-13 14:03:15 <clandmeter> Monitoring is bliss :)
2020-06-13 14:03:29 <ikke> Heh, I'm working on monitoring
2020-06-13 14:03:50 <ikke> but it would have reported s390x as down :)
2020-06-13 14:04:20 <ikke> I don't see anything being dropped in dmesg
2020-06-13 14:04:46 <ikke> But I see the packets arriving with tcpdump
2020-06-13 14:06:32 <ikke> It's using nflog
2020-06-13 14:07:30 <ikke> ah, type 0 and 8 are not allowed...
2020-06-13 14:15:50 <ikke> ok, fixed ping
2020-06-13 14:16:10 <ikke> So anoying when ping is blocked by firewalls :)
2020-06-13 14:19:34 <mps> yes
2020-06-13 14:20:04 <mps> admins don't know they can rate limit it instead of block
2020-06-13 14:57:52 <ikke> In this case someone most likely just forgot to allow it
2020-06-13 15:02:03 <ikke> But some times it's also because of a false sense of security
2020-06-13 15:25:03 <mps> this is last one is most often reasoning to not allow ICMP
2020-06-13 15:25:18 <ikke> yes
2020-06-13 15:28:30 <mps> in my professional work I meet not small number of admins with some diplomas who do not understand basics
2020-06-13 15:28:59 <mps> but we are OT (I fear maxice8 is lurking :) )
2020-06-13 15:29:27 <maxice8> I only moderate #alpine-linux atm
2020-06-13 15:30:13 <mps> anyway, for me is to be 'better safe than sorry'
2020-06-13 15:30:38 <mps> maxice8: sorry for joking, I hope you don't take this seriosly
2020-06-13 15:30:46 <ikke> Like I already said, we should not be too strict about being off-topic, as long as it does not disturb on-topic conversations
2020-06-13 15:31:41 <mps> and I think that is good maxice8 accepted that tedious and unwelcomed position
2020-06-13 15:32:41 <mps> working with bad people could be unpleasant task
2020-06-13 15:33:14 <mps> hmm, duty not task
2020-06-13 15:42:59 <maxice8> @ikke that is how I moderated a telegram group I had created for Gentoo 
2020-06-13 15:52:42 <ikke> huh
2020-06-13 16:11:48 <ikke> clandmeter: we lost s390x-ci again :(
2020-06-13 16:40:24 <clandmeter> Hmm
2020-06-13 16:40:45 <clandmeter> Do you want to mail?
2020-06-13 16:40:57 <clandmeter> I'm having a 🍻
2020-06-13 16:41:11 <clandmeter> That's not a good combo
2020-06-13 16:44:28 <ikke> I wonder if it has to do with workload
2020-06-13 18:30:38 <ikke> :(
2020-06-14 05:20:19 <ikke> s390x-ci: "ç11812631.616312Ÿ illegal operation: 0001 ilc:1 ç#1Ÿ SMP"
2020-06-14 05:25:48 <ikke> clandmeter: do we want to upgrade it to alpine 3.12?
2020-06-14 06:10:50 <maldridge> what is the hardware that's backing the s390x machine?
2020-06-14 06:13:36 <ikke> I'm not entirely sure
2020-06-14 06:14:17 <maldridge> I was mainly curious if its an actual System Z machine
2020-06-14 06:14:20 <ikke> IBM/s390 is all I know
2020-06-14 06:14:20 <maldridge> or something a bit more modest
2020-06-14 07:46:43 <mps> iirc, tmhoang told it is one of System Z machines, but I forgot which one
2020-06-14 08:06:43 <clandmeter> Morning
2020-06-14 08:07:02 <clandmeter> Looks like pho kills it?
2020-06-14 08:07:07 <clandmeter> Php
2020-06-14 08:08:43 <ikke> I had that suspicion as well 
2020-06-14 08:09:50 <clandmeter> Let's upgrade it
2020-06-14 08:12:34 <clandmeter> Hardware name: IBM 8561 LT1 400 (z/VM 6.4.0)
2020-06-14 08:12:45 <clandmeter> maldridge: ^
2020-06-14 09:00:59 <ikke> Who will do the upgrade? 
2020-06-14 09:20:44 <clandmeter> i have some time now, so i can do it.
2020-06-14 09:20:53 <clandmeter> did you turn of the runner?
2020-06-14 09:26:29 <ikke> No
2020-06-14 09:52:41 <clandmeter> ok stopping docker now
2020-06-14 09:55:35 <clandmeter> apk is being difficult. ive seen this behavior before.
2020-06-14 09:55:54 <clandmeter> cant  upgrade when having cmd:xxx in world
2020-06-14 09:56:11 <ikke> Oh yeah, that's anoying
2020-06-14 09:56:32 <clandmeter> is there an outstanding issue for htis that you may know about?
2020-06-14 10:00:20 <ikke> What cmd;
2020-06-14 10:00:28 <ikke> Is in there
2020-06-14 10:01:39 <clandmeter> sort
2020-06-14 10:01:45 <clandmeter> coreutils
2020-06-14 10:02:33 <clandmeter> the upgrade fails
2020-06-14 10:03:04 <clandmeter> interesting
2020-06-14 10:03:14 <clandmeter> (68/149) Replacing py3-cached-property (1.5.1-r1 -> 1.5.1-r1)
2020-06-14 10:03:14 <clandmeter> ERROR: py3-cached-property-1.5.1-r1: failed to rename usr/lib/python3.8/site-packages/.apk.2465692beef1da516c1ad2b814a994306f8c2fec7f49e628 to usr/lib/python3.8/site-packages/cached_property-1.5.1-py3.8.egg-info.
2020-06-14 10:03:56 <clandmeter> apk fix does fix it.
2020-06-14 10:05:44 <clandmeter> ok rebooting it
2020-06-14 10:05:58 <ikke> More users reported these kinds of issues 
2020-06-14 10:06:43 <clandmeter> ill try later to reproduce
2020-06-14 10:07:12 <clandmeter> cross fingers and hope it returns
2020-06-14 10:07:48 <ikke> I hope for you as well :-) 
2020-06-14 10:10:00 <clandmeter> Linux gitlab-runner-s390x 5.4.43-1-lts #2-Alpine SMP Thu, 28 May 2020 20:13:48 UTC s390x Linux
2020-06-14 10:16:32 <ikke> cool
2020-06-14 10:21:20 <ikke> So now what, try to run the php CI on s390x again and see if it crashes again?
2020-06-14 10:29:52 <clandmeter> i would wait till monday if thats possible
2020-06-14 10:29:57 <clandmeter> not sure we have support atm
2020-06-14 10:30:56 <clandmeter> ikke: when did you do the sqlite check for pkgs?
2020-06-14 10:32:20 <ikke> This morning
2020-06-14 10:32:35 <ikke> did you see it?
2020-06-14 10:33:03 <clandmeter> i saw the issue
2020-06-14 10:33:09 <clandmeter> and i saw its still broken
2020-06-14 10:33:16 <clandmeter> and i saw you did something in history
2020-06-14 10:33:20 <ikke> aha ok
2020-06-14 10:33:25 <clandmeter> thats about all i know
2020-06-14 10:33:33 <clandmeter> db seems corrupt
2020-06-14 10:33:50 <ikke> https://tpaste.us/KyYP 
2020-06-14 10:33:50 <clandmeter> database disk image is malformed
2020-06-14 10:33:54 <ikke> This is the output
2020-06-14 10:34:03 <clandmeter> yup
2020-06-14 10:35:01 <clandmeter> not sure how to read it probably
2020-06-14 10:35:07 <clandmeter> sounds like index is corrupt or so
2020-06-14 10:35:27 <ikke> So either find out how to repair the db, or regenerate it
2020-06-14 10:35:34 <ikke> which is going to take some time
2020-06-14 10:35:51 <clandmeter> yes but we have 2 instances 
2020-06-14 10:35:55 <clandmeter> so we could regen on the second
2020-06-14 10:36:08 <ikke> where is the 2nd instance?
2020-06-14 10:36:23 <clandmeter> its a bit obvious ;-)
2020-06-14 10:36:44 <clandmeter> aports-turbo-test
2020-06-14 10:36:46 <clandmeter> :)
2020-06-14 10:37:00 <clandmeter> but im not sure what the status is
2020-06-14 10:37:20 <clandmeter> i think we can just copy the db to the test, del edge and regen
2020-06-14 10:38:27 <ikke> and then copy back?
2020-06-14 10:38:32 <clandmeter> nod
2020-06-14 10:38:52 <clandmeter> but i need to check how the relation with flagged db is
2020-06-14 10:39:20 <ikke> I'm also kind of interested to see if this is fixable
2020-06-14 10:41:08 <clandmeter> right, thats also fine.
2020-06-14 10:41:13 <clandmeter> ill stay of it for now.
2020-06-14 10:41:25 <clandmeter> i copied the db to /root
2020-06-14 10:42:58 <clandmeter> you could regen the index
2020-06-14 10:43:18 <clandmeter> https://www.sqlite.org/lang_reindex.html
2020-06-14 10:43:41 <ikke> Let's try
2020-06-14 10:46:53 <ikke> just repsonds with the same error
2020-06-14 10:47:12 <ikke> https://stackoverflow.com/a/18260642/20261
2020-06-14 10:47:51 <clandmeter> heh
2020-06-14 10:47:56 <clandmeter> you are reading same page as me
2020-06-14 10:48:00 <clandmeter> im already dumping the db
2020-06-14 10:48:08 <ikke> ok
2020-06-14 10:48:27 <ikke> Then I'm doing nothing :)
2020-06-14 10:48:36 <clandmeter> go ahead :)
2020-06-14 10:48:51 <clandmeter> im just doing it on the one in /root
2020-06-14 10:49:10 <clandmeter> probably recreating the index is faster
2020-06-14 10:50:34 <ikke> you mean drop index + create index?
2020-06-14 10:51:12 <clandmeter> new.db is 0 bites
2020-06-14 10:51:17 <clandmeter> bytes
2020-06-14 10:51:19 <clandmeter> :)
2020-06-14 10:51:24 <ikke> at least it's not corrupt :P
2020-06-14 10:51:37 <ikke> what about `.recover`?
2020-06-14 10:52:06 <clandmeter> well i guess if you dump to an txt file and back to sqlite problem should be solved.
2020-06-14 10:52:19 <clandmeter> if its just the index
2020-06-14 10:53:16 <clandmeter> let me dump it to txt first
2020-06-14 10:56:35 <clandmeter> that does not look good :)
2020-06-14 10:56:51 <clandmeter> last line is ROLLBACK;
2020-06-14 10:56:55 <clandmeter> due to errors
2020-06-14 10:57:14 <clandmeter> but sqlite does not mention anything itself
2020-06-14 10:58:18 <clandmeter> heh
2020-06-14 10:58:30 <clandmeter> .recover is not available ;-)
2020-06-14 10:58:44 <clandmeter> With Sqlite 3.29.0
2020-06-14 10:58:50 <clandmeter> and ofc we have .28
2020-06-14 10:59:08 <ikke> Bummer 
2020-06-14 10:59:22 <clandmeter> 3.10
2020-06-14 10:59:26 <clandmeter> we could...
2020-06-14 10:59:38 <clandmeter> or i could just run a container :)
2020-06-14 11:00:59 <ikke> Yes, sounds a lot easier
2020-06-14 11:07:50 <clandmeter> ok now the txt is 8+ gb
2020-06-14 11:12:03 <clandmeter> doesnt look like its ok
2020-06-14 11:12:13 <clandmeter> lost and found is 8GB+
2020-06-14 11:15:54 <clandmeter> ikke, i dont think i can recover it, if you have ideas go ahead.
2020-06-14 11:16:38 <clandmeter> i have moved the tmp db into tmpdb dir for use in docker.
2020-06-14 11:40:20 <ikke> no, no other ideas, so lets just regen
2020-06-14 11:45:15 <clandmeter> its not the indexes that are the issue
2020-06-14 11:48:21 <ikke> I wonder what caused the corruption
2020-06-14 11:50:02 <clandmeter> i need to go soon, if you can regen thats nice, else i need to check it later.
2020-06-14 11:53:08 <ikke> I can
2020-06-14 11:53:18 <ikke> What was the plan for that?
2020-06-14 11:53:26 <ikke> In -test remove the edge db, regen, and copy back?
2020-06-14 12:23:12 <clandmeter> i would copy the current db dir to test
2020-06-14 12:23:16 <clandmeter> then remove edge
2020-06-14 12:23:31 <clandmeter> else it will need to add a lot of missing things
2020-06-14 12:24:17 <clandmeter> or comment out all other branches.
2020-06-14 12:25:28 <clandmeter> looks like its using a diff tag
2020-06-14 12:25:42 <clandmeter> and uses the live db
2020-06-14 12:27:32 <clandmeter> you could copy the db dir to some place and just run docker directly and mount that db in the container. rm edge and update.
2020-06-14 12:27:52 <clandmeter> you dont need http for updating
2020-06-14 12:30:09 <ikke> right
2020-06-14 12:30:15 <ikke> so just the update container
2020-06-14 12:30:41 <ikke> clandmeter: You did not start it again I suppose?
2020-06-14 13:02:53 <ikke> Running the update script now in a separate temporary container
2020-06-14 13:04:19 <ikke> (in a tmux session)
2020-06-14 14:09:09 <ikke> clandmeter: I can no longer start the runner on s390x with docker-compose
2020-06-14 14:09:18 <ikke> "Cannot start service gitlab-runner: read unix @->@/containerd-shim/3fedff5d13348b7e084ab6f4a5dd59a191d7d9b4fda6633b89ae1a7106d4eac1.sock: read: connection reset by peer: unknown"
2020-06-14 14:18:33 <ikke> Cannot start any container with docker
2020-06-14 14:21:42 <clandmeter> Docker is running?
2020-06-14 14:21:53 <ikke> yes
2020-06-14 14:22:30 <clandmeter> Anything in docker daemon log?
2020-06-14 14:22:56 <ikke> I'm checking /var/log/docker.log now
2020-06-14 14:23:02 <ikke> But more of the same
2020-06-14 14:23:12 <ikke> level=error msg="failed to kill shim" error="read unix @->@/containerd-shim/bb1a474ba0fd3e538e8e48b9e9884509c061f52a8a6bfaeca02edcb4b120c8a4.sock: read: connection reset by peer: unknown"
2020-06-14 14:23:18 <ikke> level=error msg="stream copy error: reading from a closed fifo"
2020-06-14 15:04:26 <clandmeter> ikke: do you know how to raise the log level of containerd?
2020-06-14 15:07:18 <ikke> probably in daemon.json
2020-06-14 15:07:20 <ikke> I guess?
2020-06-14 15:07:40 <clandmeter> cant find it
2020-06-14 15:08:19 <ikke> hmmm
2020-06-14 15:08:26 <clandmeter> i can only find https://success.docker.com/article/how-to-enable-debug-mode-containerd-without-docker-engine
2020-06-14 15:08:45 <ikke> You can specify -l to dockerd
2020-06-14 15:08:58 <ikke> in /etc/conf.d/docker
2020-06-14 15:09:13 <ikke> "-l, --log-level string                        Set the logging level ("debug"|"info"|"warn"|"error"|"fatal")
2020-06-14 15:09:15 <ikke> "
2020-06-14 15:09:38 <clandmeter> for docker?
2020-06-14 15:09:41 <clandmeter> or containerd
2020-06-14 15:11:09 <ikke> Don't know
2020-06-14 15:11:56 <ikke> I hope docker applies this to containerd as well 
2020-06-14 15:12:51 <clandmeter> yeah it does
2020-06-14 15:12:57 <clandmeter> but its not obbious
2020-06-14 15:13:18 <clandmeter> looks like not much improvement in info 
2020-06-14 15:15:40 <ikke> Hmm, ok
2020-06-14 15:19:06 <ikke> I've tried an strace, but there was a lot of output 
2020-06-14 15:19:38 <clandmeter> a strace on docker?
2020-06-14 15:19:40 <clandmeter> dockerd
2020-06-14 15:23:55 <ikke> dockerd yes
2020-06-14 15:23:59 <ikke> it's in the root dir
2020-06-14 15:24:10 <ikke> /root/docker-strace/
2020-06-14 15:27:11 <clandmeter> do you strace the pid?
2020-06-14 15:27:24 <clandmeter> it doesnt show much for me
2020-06-14 15:29:03 <ikke> yes, with -f
2020-06-14 15:29:43 <clandmeter> what do you start?
2020-06-14 15:30:07 <ikke> I attach to an existing process
2020-06-14 15:30:23 <clandmeter> i mean container
2020-06-14 15:30:26 <ikke> strace -f -p 6005
2020-06-14 15:30:26 <clandmeter> jsut docker run?
2020-06-14 15:30:34 <ikke> alpinelinux/alpine-gitlab-ci
2020-06-14 15:30:37 <ikke> yes
2020-06-14 15:30:39 <clandmeter>  docker run --rm -it alpine sh
2020-06-14 15:30:43 <ikke> should work
2020-06-14 15:31:00 <clandmeter> i tried to strace it but i got only a few lines
2020-06-14 15:31:08 <ikke> did you add -f?
2020-06-14 15:31:11 <clandmeter> does strace work correctly on golang?
2020-06-14 15:32:06 <clandmeter> no i didnt add -f
2020-06-14 15:32:45 <ikke> otherwise it will only show the process you attach to, not any subprocesses
2020-06-14 17:45:07 <ikke> Did you find anything (just back again)
2020-06-14 18:49:33 <clandmeter> no i didnt
2020-06-14 18:49:56 <ikke> hmm, and now?
2020-06-14 18:50:26 <clandmeter> im not sure where its happening
2020-06-14 18:50:47 <clandmeter> dockerd, containerd, or something different
2020-06-14 18:51:11 <ikke> yeah, hard to figure out
2020-06-14 18:51:28 <clandmeter> we could build an older version of docker and deps
2020-06-14 18:51:43 <clandmeter> but maybe its golang related...
2020-06-14 18:51:49 <clandmeter> so its a stab in the dark
2020-06-14 18:52:15 <ikke> yup, it certainly is, but you have to start somewhere
2020-06-14 18:52:19 <clandmeter> and we dont really have a place to test
2020-06-14 18:52:55 <ikke> s390x-ci.. :)
2020-06-14 18:52:58 <clandmeter> or downgrade
2020-06-14 18:53:01 <ikke> it's kind of doing nothign anyway
2020-06-14 18:53:09 <ikke> yup, that's also an option
2020-06-14 18:53:52 <clandmeter> but downgrading and making it work does not fix the long term problem
2020-06-14 18:54:51 <clandmeter> we could try play with qemu
2020-06-14 18:56:06 <clandmeter> do you remember on which alpine version we were?
2020-06-14 18:56:39 <ikke> I might still have it in the history
2020-06-14 18:57:04 <ikke> nope, docker-compose logs overwrote everything :-/
2020-06-14 18:57:31 <clandmeter> it was 10 or 11
2020-06-14 18:57:42 <ikke> I think 3.11
2020-06-14 18:57:52 <clandmeter> we had issues before
2020-06-14 18:58:00 <ikke> yes, so we decided to upgrade to 3.11
2020-06-14 18:58:02 <clandmeter> so i upgraded it not long ago
2020-06-14 18:58:05 <ikke> heh
2020-06-14 19:00:57 <clandmeter> ok im installing qemu and see if i can get something running
2020-06-14 19:01:21 <ikke> ok
2020-06-14 19:02:41 <clandmeter> is it me or is netbox slow
2020-06-14 19:03:53 <ikke> looks like it's you
2020-06-14 19:43:03 <clandmeter> looks like it does not like 3.11
2020-06-14 19:43:08 <clandmeter> at least not in qemu
2020-06-14 20:04:56 <clandmeter> ikke: 3.11 is ok
2020-06-14 20:04:59 <clandmeter> 3.12 not
2020-06-14 20:05:27 <ikke> ok, good to know
2020-06-14 20:05:37 <ikke> so should we downgrade the ci server again?
2020-06-14 20:06:39 <clandmeter> i think we should try to find out where the problem is
2020-06-14 20:06:42 <clandmeter> which part
2020-06-14 20:06:49 <clandmeter> so maybe build the packages
2020-06-14 20:07:04 <clandmeter> docker containerd and runc
2020-06-14 20:07:11 <clandmeter> i guess those are sep pkgs
2020-06-14 20:07:39 <clandmeter> build them on 3.12 based on 3.11 versions
2020-06-14 20:08:37 <ikke> zabbix upgraded to 5.0 :)
2020-06-14 20:09:00 <ikke> so latest version on older alpine, right?
2020-06-14 20:09:17 <ikke> 2https://pkgs.alpinelinux.org/package/edge/community/x86_64/containerd
2020-06-14 20:09:22 <ikke> https://pkgs.alpinelinux.org/package/edge/community/x86_64/runc
2020-06-14 20:09:55 <clandmeter> i think we could try with versions from 3.11 (which we know works)
2020-06-14 20:10:13 <clandmeter> maybe its possible to install them already
2020-06-14 20:10:14 <ikke> on 3.12?
2020-06-14 20:10:16 <ikke> right
2020-06-14 20:10:20 <clandmeter> yes
2020-06-14 20:10:31 <ikke> apk add -X <3.11-repo> docker?
2020-06-14 20:10:32 <clandmeter> its running in tmux
2020-06-14 20:10:47 <ikke> docker=<version>
2020-06-14 20:11:13 <ikke> clandmeter: this page is new in zabbix 5.0: https://zabbix.alpinelinux.org/zabbix.php?action=host.view
2020-06-14 20:11:27 <ikke> perhaps not perfect, but it's kind of what you were missing
2020-06-14 20:11:46 <clandmeter> ok nice
2020-06-14 20:12:05 <clandmeter> i need to fix something else now. you can try to run those pkgs in qemu
2020-06-14 20:12:11 <ikke> ok
2020-06-14 20:12:11 <clandmeter> there is also a script to restart it
2020-06-14 20:12:22 <ikke> restart the vm?
2020-06-14 20:12:26 <clandmeter> yes
2020-06-14 20:12:28 <clandmeter> well start
2020-06-14 20:12:29 <ikke> ok
2020-06-14 20:12:38 <clandmeter> shutdown -f :)
2020-06-14 20:12:43 <ikke> heh
2020-06-14 20:12:46 <clandmeter> in qemu ;-)
2020-06-14 20:12:52 <ikke> :D
2020-06-14 20:13:01 <ikke> would be painful if you run it on the host
2020-06-14 20:13:11 <ikke> (kind of funny, because the host is a vm on it's own)_
2020-06-14 20:17:14 <ikke> clandmeter: fyi, only patch-level version difference
2020-06-14 20:17:21 <ikke> 19.03.5 vs 19.03.11
2020-06-14 20:21:41 <ikke> Hmm, it keeps using the latest version
2020-06-14 20:22:05 <ikke> apk add -X <3.11-repo> docker=19.03.5-r1 does not work
2020-06-14 20:22:12 <ikke> and also using a pinned repo does not work
2020-06-14 20:40:08 <ikke> clandmeter: so it's containerd
2020-06-14 20:40:21 <ikke> 1.3.2 works, 1.3.4 is broken
2020-06-14 20:40:55 <ikke> Let's see if newer versions of docker work with 1.3.2
2020-06-14 20:41:55 <ikke> yes
2020-06-14 20:42:08 <ikke> so only using 3.11 version of containerd fixes the issue
2020-06-14 20:43:22 <ikke> fixed it on the host as well
2020-06-14 20:43:45 <ikke> Now the question if it's only s390x, or if it happes on other arches as well
2020-06-14 21:14:30 <maldridge> ikke: I believe it should be all arches that have this problem
2020-06-14 21:14:50 <maldridge> docker/containerd interface is mostly unspecified from my understanding.  I know there are specific version pairs for x86
2020-06-14 21:15:28 <ikke> maldridge: yes, that was my expectation, but strange that we did not get any reports about this yet though
2020-06-14 21:15:40 <ikke> it would mean docker is generally broken on 3.12
2020-06-14 21:17:25 <maldridge> this does mirror my observations
2020-06-14 21:18:01 <ikke> ok, so you have issues as well?
2020-06-14 21:19:21 <maldridge> yeah, at work we're running docker-in-docker, and we see similar issues with anything past 18.x
2020-06-14 21:20:23 <ikke> I wonder if 1.3.3 has this issue as well
2020-06-14 21:20:33 <clandmeter> Nice
2020-06-14 21:20:48 <clandmeter> That you found it
2020-06-14 21:21:00 <ikke> teamwork :)
2020-06-14 21:21:20 <clandmeter> It cross my mind it's all arches
2020-06-14 21:21:37 <ikke> if that's the case, we should revert containerd, I guess
2020-06-14 21:21:44 <ikke> but it's so strange that we did not get any complaints
2020-06-14 21:21:47 <clandmeter> But as you said. I would expect some reports
2020-06-14 21:21:51 <ikke> yea
2020-06-14 21:22:34 <maldridge> I think after bad experiences wiht docker in the past folks may be more cautious about updating
2020-06-14 21:22:51 <maldridge> I think work is actually still on 3.11, with some old hosts on 3.9 still
2020-06-14 21:34:23 <ikke> clandmeter: edge regen is stull running btw
2020-06-14 21:34:34 <ikke> edge/testing/aarch64
2020-06-14 21:34:36 <clandmeter> Yes
2020-06-14 21:34:51 <clandmeter> It's very slow via http
2020-06-14 21:35:14 <clandmeter> But portable
2020-06-14 21:35:29 <ikke> We just let it run
2020-06-14 21:35:40 <ikke> as long as it's stable
2020-06-15 04:38:43 <ikke> clandmeter: ok, it finished
2020-06-15 05:48:41 <clandmeter> Nice
2020-06-15 05:49:02 <maldridge> well done
2020-06-15 05:49:35 <ikke> should I just copy the edge db back?
2020-06-15 06:55:31 <clandmeter> i would first stop it :)
2020-06-15 06:55:58 <ikke> yes, sounds like a good plan
2020-06-15 06:56:21 <clandmeter> also remove releated files, i guess thats logical
2020-06-15 06:56:43 <ikke> the tmp files related to that db
2020-06-15 06:56:45 <ikke> indeed
2020-06-15 06:57:03 <clandmeter> yes i think those are wall
2020-06-15 08:05:04 <ikke> clandmeter: done
2020-06-15 08:05:31 <clandmeter> nice
2020-06-15 08:05:33 <clandmeter> all is ok?
2020-06-15 08:05:48 <ikke> it's working, I did not verify the corrupted queries yet
2020-06-15 08:07:30 <clandmeter> works
2020-06-15 08:07:33 <clandmeter> :)
2020-06-15 08:07:35 <ikke> cool
2020-06-15 08:07:47 <ikke> and flags?
2020-06-15 08:08:15 <clandmeter> its empty :(
2020-06-15 08:08:26 <ikke> hmm
2020-06-15 08:08:32 <clandmeter> weird
2020-06-15 08:08:42 <ikke> I kept a copy of the old db
2020-06-15 08:08:47 <clandmeter> i think it uses pkgname 
2020-06-15 08:09:09 <clandmeter> its not a big issue though
2020-06-15 08:10:19 <clandmeter> ah flagged is just a table
2020-06-15 08:10:25 <clandmeter> i thought it was a sep db
2020-06-15 08:10:33 <ikke> oh ok
2020-06-15 08:10:38 <ikke> so we would need to copy that
2020-06-15 08:11:46 <clandmeter> yeah i guess just export that table and insert again
2020-06-15 08:13:18 <ikke> Do you have time for that?
2020-06-15 08:42:13 <clandmeter> not now, im busy @work
2020-06-15 08:42:55 <ikke> me too
2020-06-15 10:58:07 <ikke> clandmeter: ping
2020-06-15 10:58:10 <clandmeter> pong
2020-06-15 10:58:20 <clandmeter> i just put focus on this screen :)
2020-06-15 10:58:23 <ikke> clandmeter: there seems to be also a table called flagged.flagged?
2020-06-15 10:58:25 <ikke> hah
2020-06-15 10:59:00 <ikke> but it's empty it seems
2020-06-15 11:00:56 <clandmeter> right
2020-06-15 11:01:02 <clandmeter> i just dumped flagged table
2020-06-15 11:01:17 <clandmeter> seems we both have some time availabe :)
2020-06-15 11:01:20 <ikke> flags are back
2020-06-15 11:01:48 <clandmeter> nice
2020-06-15 11:02:37 <clandmeter> case closed i guess
2020-06-15 11:03:41 <ikke> issue closed as well :P
2020-06-15 11:14:37 <clandmeter> ikke did you scale the app?
2020-06-15 11:15:22 <ikke> I just stopped the containers
2020-06-15 11:15:25 <ikke> and started them again
2020-06-15 11:15:53 <clandmeter> ok but with scale?
2020-06-15 11:16:25 <ikke> It just starts the existing containers again
2020-06-15 11:16:29 <ikke> so it will use the same 'scale'
2020-06-15 11:16:35 <clandmeter> ok
2020-06-15 11:16:42 <clandmeter> it was loading a bit slow
2020-06-15 11:17:12 <ikke> https://tpaste.us/b46e
2020-06-15 11:18:15 <ikke> seems like the DB is not complete yet
2020-06-15 11:18:39 <ikke> last update is from yesterday, even though I manually ran the update
2020-06-15 12:11:14 <clandmeter> ikke: looks like wal is not commited to db
2020-06-15 12:11:16 <clandmeter> not sure why
2020-06-15 12:11:38 <clandmeter> ah
2020-06-15 12:11:44 <clandmeter> i think i know why
2020-06-15 12:11:49 <ikke> permissions?
2020-06-15 12:12:11 <ikke> forgot -a with cp
2020-06-15 12:12:22 <clandmeter> yup
2020-06-15 12:12:25 <clandmeter> root:1000
2020-06-15 12:12:41 <ikke> I used rsync -a for the initial copy, so there it was good
2020-06-15 12:12:43 <clandmeter> but its weird it does now show an error
2020-06-15 12:12:46 <ikke> yup
2020-06-15 12:12:59 <clandmeter> next sync is in 3 min
2020-06-15 12:13:02 <ikke> ok
2020-06-15 12:13:29 <ikke> will sqlite automatically apply the wal?
2020-06-15 12:14:26 <clandmeter> lets wait and see :)
2020-06-15 12:14:35 <clandmeter> could be it just ignores whats inside atm
2020-06-15 12:14:37 <ikke> heh
2020-06-15 12:15:10 <ikke> .db-shm is now being written to
2020-06-15 12:15:18 <ikke> and the wal as well
2020-06-15 12:17:03 <ikke> ok. wal is empty now
2020-06-15 12:17:19 <ikke> success
2020-06-15 12:17:26 <clandmeter> its still running
2020-06-15 12:17:31 <clandmeter> now its ready :)
2020-06-15 12:17:46 <ikke> heh
2020-06-15 12:17:50 <clandmeter> looks ok
2020-06-15 12:18:59 <ikke> hmm, last push was for keepalived, but no updates from builders
2020-06-15 12:19:28 <ikke> I have the idea that the builders are always one update behind
2020-06-15 12:20:28 <ikke> not exactly like that, but they are not always getting the latest update
2020-06-15 12:26:24 <clandmeter> ikke: i think that sounds sane
2020-06-15 12:26:34 <clandmeter> i bet they are still pulling from git.a.o
2020-06-15 12:26:45 <clandmeter> and when the update comes git.a.o is not yet ready
2020-06-15 12:27:23 <ikke> but at 13:30 was the last update message from the builder, at 13:40 a new commit was pushed, but nothing happened until I manually retried
2020-06-15 12:28:05 <clandmeter> i dont follow
2020-06-15 12:28:15 <ikke> look back in the #alpine-commit logs
2020-06-15 12:28:38 <ikke> keepalived was pushed when the builders were (presumably) idle
2020-06-15 12:28:42 <ikke> but they didn't start building it
2020-06-15 12:28:50 <clandmeter> so, that matches my logic?
2020-06-15 12:29:11 <clandmeter> push to gitlab, builders pull but update is not yet pushed to git.a.o
2020-06-15 12:29:22 <clandmeter> so no update to build
2020-06-15 12:29:31 <ikke> ah, like that
2020-06-15 12:29:33 <clandmeter> 1 second later update is pushed to git.a.o
2020-06-15 12:29:38 <ikke> yeah
2020-06-15 12:29:44 <clandmeter> they need to pull from gitlab
2020-06-15 12:29:47 <ikke> yeah
2020-06-15 12:30:16 <clandmeter> best to update it in the pkg
2020-06-15 12:30:31 <ikke> aports-build?
2020-06-15 12:30:34 <clandmeter> else you need to update the config everywhere
2020-06-15 12:31:02 <clandmeter> ah you probably need to manually update it
2020-06-15 12:31:24 <clandmeter> i guess it needs a remote update
2020-06-15 12:32:00 <ikke> ok, and use https:// instead of git://
2020-06-15 17:49:01 <clandmeter> So what is the status of docker?
2020-06-15 17:49:17 <clandmeter> ikke: ^
2020-06-15 17:49:31 <ikke> good question
2020-06-15 17:49:49 <ikke> I'll install docker in my x86_64 vm and see if has the same issue
2020-06-15 17:50:56 <clandmeter> There is no reference to versions that work together?
2020-06-15 17:51:30 <clandmeter> Or is it not a compat issue?
2020-06-15 17:52:04 <ikke> runs fine on my VM
2020-06-15 17:52:14 <ikke> so I would say no
2020-06-15 17:52:38 <ikke> clandmeter: Maybe see if we can test containerd separately
2020-06-15 17:52:42 <clandmeter> Hmm
2020-06-15 17:52:51 <clandmeter> What did you test
2020-06-15 17:53:01 <clandmeter> 3.12¿
2020-06-15 17:53:03 <ikke> yes
2020-06-15 17:53:23 <ikke> containerd 1.3.4, which gave issues on s390x
2020-06-15 17:53:38 <clandmeter> Maybe check diff arches
2020-06-15 17:53:54 <clandmeter> mps: can you try on arm?
2020-06-15 17:53:56 <ikke> heh :D
2020-06-15 17:54:01 <ikke> Was thinking the same
2020-06-15 17:54:24 <clandmeter> Or did you run back to x86 ;-)
2020-06-15 17:56:22 <mps> clandmeter: you mean to run containerd on armv7?
2020-06-15 17:56:34 <mps> native
2020-06-15 17:56:35 <clandmeter> Yes
2020-06-15 17:56:47 <clandmeter> Docker
2020-06-15 17:57:00 <ikke> apk add docker; rc-service docker start; docker run -it --rm alpine
2020-06-15 17:57:01 <mps> ok, will try when I come back home, in about hour or two
2020-06-15 17:57:22 <clandmeter> Ok thx
2020-06-15 17:57:38 <mps> np
2020-06-15 17:57:41 <clandmeter> If it's trouble we can also use qemu
2020-06-15 17:58:16 <clandmeter> Although it's not working without kvm on s390x
2020-06-15 17:58:19 <mps> no, my arm boxes are ready, just now in shutdown state
2020-06-15 17:58:52 <mps> and I can't power-on them remotely
2020-06-15 18:00:42 <mps> though I can now do that on aarch64, is that also needed to test or only arm 32bit
2020-06-15 18:01:45 <ikke> no, aarch64 is fine
2020-06-15 18:01:58 <mps> ok
2020-06-15 18:50:00 <mps> clandmeter: ikke: looks like it works on 3.12
2020-06-15 18:50:19 <clandmeter> nice
2020-06-15 18:50:21 <clandmeter> thx
2020-06-15 18:50:27 <ikke> OK, so works on x86_64
2020-06-15 18:50:32 <ikke> On aarch64
2020-06-15 18:50:37 <ikke> Not on s390x
2020-06-15 18:50:54 <mps> how can I test it more
2020-06-15 18:51:15 <ikke> It's fairly self-evident
2020-06-15 18:51:29 <ikke> The issue is that we cannot start any container 
2020-06-15 18:51:49 <ikke> So if you successfully ran a container, it works
2020-06-15 18:51:55 <mps> that is what I see on console https://tpaste.us/evRW
2020-06-15 18:52:06 <mps> armv7
2020-06-15 18:52:15 <ikke> Looks good 
2020-06-15 18:53:00 <mps> hehe, this box have: 'apk version musl' => musl-1.2.0-r0
2020-06-15 18:53:29 <ikke> ncie
2020-06-15 19:16:21 <clandmeter> I guess we should open an issue on github
2020-06-15 19:16:30 <clandmeter> so they can help us debug it
2020-06-15 19:17:07 <ikke> yeah
2020-06-15 19:17:46 <ikke> there is an example on the containerd website how to interact with it (via go), might be worth a try to see if we can see if it's an issue with containerd 1.3.4 on s390x in general
2020-06-15 19:20:47 <clandmeter> if you are feeling lucky :)
2020-06-15 19:22:21 <ikke> I'm kind of trying to get all servers in Zabbix
2020-06-15 19:40:36 <clandmeter> that sounds like a good investment of spare time :)
2020-06-15 19:41:32 <ikke> It has been on my mental to-do list for ages :)
2020-06-15 19:41:58 <clandmeter> yes, i didnt want to mention it again ;-)
2020-06-15 21:26:18 <ikke> https://zabbix.alpinelinux.org/items.php?filter_set=1&filter_hostids%5B0%5D=10368
2020-06-15 21:27:01 <ikke> clandmeter: ^
2020-06-15 21:27:31 <ikke> Discovered by the new agent2 plugins
2020-06-16 04:45:00 <maldridge> clandmeter: ikke I have grown more competent at using ctr than I'd like
2020-06-16 04:45:06 <maldridge> what are you trying to do?
2020-06-16 04:46:09 <ikke> ctr?
2020-06-16 04:46:39 <clandmeter> exactly my question :)
2020-06-16 04:47:15 <maldridge> its the CLI toolset for containerd
2020-06-16 04:47:24 <ikke> ah
2020-06-16 04:47:40 <ikke> maldridge: we have an issue with containerd 1.3.4 on s390x
2020-06-16 04:47:50 <maldridge> I've noticed
2020-06-16 04:48:02 <ikke> where it seems to reset the socket connection when docker tries to talk on it
2020-06-16 04:48:27 <maldridge> hrm
2020-06-16 04:48:32 <ikke> so we wanted to see if we can reproduce it with containerd in isolation
2020-06-16 04:48:41 <maldridge> well, docker does a lot with containerd
2020-06-16 04:48:50 <maldridge> do you know where in the transaction its failing?
2020-06-16 04:48:59 <ikke> early on when starting containers
2020-06-16 04:50:04 <maldridge> early enough that its not even getting through setup?
2020-06-16 04:50:42 <ikke> "containerd-shim/bb1a474ba0fd3e538e8e48b9e9884509c061f52a8a6bfaeca02edcb4b120c8a4.sock: read: connection reset by peer: unknown"
2020-06-16 04:50:50 <ikke> sorry, that's incomplete
2020-06-16 04:51:08 <ikke> "read unix @->@/containerd-shim/3fedff5d13348b7e084ab6f4a5dd59a191d7d9b4fda6633b89ae1a7106d4eac1.sock: read: connection reset by peer: unknown"
2020-06-16 04:51:46 <ikke> maldridge: what do you consider setup?
2020-06-16 04:52:32 <maldridge> later than that
2020-06-16 04:52:44 <ikke> ok
2020-06-16 04:52:54 <maldridge> setup to me is where it begins to stream instructions to containerd for layering filesysetms
2020-06-16 04:52:57 <ikke> I have a vm on s390x with docker installed
2020-06-16 04:53:06 <maldridge> this looks like docker may be out of sync with containerd
2020-06-16 04:56:02 <ikke> clandmeter: the tmux session on s390x-ci should have the VM you created, right? Right now it seems to just have /bin/sh as pid1
2020-06-16 04:56:24 <ikke> ah, ofcourse
2020-06-16 04:56:34 <ikke> inception :P
2020-06-16 04:56:47 <clandmeter> ?
2020-06-16 04:57:00 <ikke> it was in a docker container
2020-06-16 04:57:31 <clandmeter> not following, docker runs in qemu kvm
2020-06-16 04:57:51 <ikke> when I attached to tmux, it was already running in a docker container
2020-06-16 04:57:59 <ikke> on the qemu vm
2020-06-16 04:58:05 <clandmeter> yup
2020-06-16 04:58:07 <ikke> on the alpine vm
2020-06-16 04:58:36 <ikke> maldridge: the same version seems to work on other arches
2020-06-16 04:58:49 <ikke> docker-19.03.11-r0
2020-06-16 05:00:13 <clandmeter> ikke: have you tried to build it locally?
2020-06-16 05:00:16 <clandmeter> i mean in the vm
2020-06-16 05:01:18 <ikke> no
2020-06-16 05:04:11 <clandmeter> would be nice to know if it happens with 1.3.3
2020-06-16 05:11:02 <ikke> I don't think I can connect to the VM via ssh, right? (the serial console is kind of limiting)
2020-06-16 05:13:55 <clandmeter> you can but not in this network setup
2020-06-16 05:14:15 <ikke> right
2020-06-16 05:14:52 <clandmeter> im trying to build containerd
2020-06-16 05:15:01 <clandmeter> with build-base
2020-06-16 05:15:03 <clandmeter> container
2020-06-16 05:15:15 <ikke> ah ok
2020-06-16 05:15:16 <clandmeter> but it seesm we dont ship stable versions of it
2020-06-16 05:16:47 <ikke> what do you mean?
2020-06-16 05:16:57 <clandmeter> there is only edge?
2020-06-16 05:17:16 <ikke> ah, for build-base
2020-06-16 05:17:44 <ikke> yeah, something todo still
2020-06-16 05:18:21 <ikke> the setup script for alpine-ci was already setup to downgrade
2020-06-16 05:19:48 <maldridge> ikke: I don't have solid advice then.  Usually containerd problems arise from mismatched versions since the API is somewhat considered internal
2020-06-16 05:20:16 <ikke> understood, thanks for your help
2020-06-16 05:20:42 <maldridge> I wouldn't be surprised though if on s390x there's some underlying primitive that isn't behaving as expected
2020-06-16 05:20:48 <maldridge> $work has these problems on power
2020-06-16 05:21:05 <clandmeter> it could also be golang update
2020-06-16 05:21:14 <ikke> yup
2020-06-16 05:21:15 <maldridge> which versions are you on?
2020-06-16 05:21:18 <maldridge> I can compare against void
2020-06-16 05:21:21 <ikke> 1.14
2020-06-16 05:21:41 <maldridge> .3?
2020-06-16 05:21:56 <ikke> for containerd?
2020-06-16 05:22:06 <maldridge> for go
2020-06-16 05:22:09 <ikke> 3.12. is go 1.13
2020-06-16 05:22:13 <maldridge> we noticed some build breakage at 1.14.3
2020-06-16 05:22:17 <ikke> .11
2020-06-16 05:22:42 <ikke> edge is 1.14.3
2020-06-16 05:22:49 <ikke> but this happened on 3.12
2020-06-16 05:22:50 <maldridge> ah
2020-06-16 05:23:14 <maldridge> we're at 1.3.4 (containerd); 19.03.9 (docker)
2020-06-16 05:23:20 <ikke> yes, same
2020-06-16 05:23:24 <maldridge> no observed instability, but we also don't build for s390x
2020-06-16 05:23:36 <ikke> 3.11 is 1.13.10 btw
2020-06-16 05:23:41 <ikke> so only one minor version bump
2020-06-16 05:23:47 <ikke> or patch rather
2020-06-16 05:29:28 <clandmeter> hmm maybe if i want to reproduce something i should start a container with --rm :D
2020-06-16 05:29:38 <clandmeter> *not*
2020-06-16 05:29:41 <ikke> heh
2020-06-16 05:36:23 <clandmeter> 1.3.3 same issue
2020-06-16 05:40:28 <clandmeter> 1.3.2 same...
2020-06-16 05:40:46 <clandmeter> lots of fun :)
2020-06-16 05:41:15 <ikke> heh
2020-06-16 05:41:28 <clandmeter> ok this makes it a bit more compicated
2020-06-16 05:41:55 <ikke> yup
2020-06-16 05:43:12 <clandmeter> are we actually sure its containerd?
2020-06-16 05:43:51 <ikke> I only pinned containerd from 3.11, and that fixed it
2020-06-16 05:45:12 <clandmeter> ok lets assume its containerd for now :)
2020-06-16 05:48:40 <clandmeter> git.a.o is pissing me off
2020-06-16 05:48:44 <clandmeter> its slow as...
2020-06-16 06:08:54 <clandmeter> its probably not golang
2020-06-16 06:14:54 <clandmeter> so docker works with containerd from 3.11, but it does not work with containerd build with same version as 3.11 with same version of golang.
2020-06-16 07:35:40 <ikke> maybe some other dependency
2020-06-16 07:36:31 <ikke> Can you downgrade completely to 3.11 and try to build then?
2020-06-16 07:36:41 <ikke> or did you already do that?
2020-06-16 08:03:27 <clandmeter> ncopa: ^
2020-06-16 08:03:32 <ncopa> hi
2020-06-16 08:04:00 <clandmeter> docker on 3.12 works with containerd from 3.11
2020-06-16 08:04:55 <clandmeter> i have tried to build the 3.11 version on 3.12 but that does not fix it
2020-06-16 08:05:35 <clandmeter> so i tried to build golang from 3.11 on 3.12 and build containerd with it, but does not solve it
2020-06-16 08:07:22 <clandmeter> i removed the libseccomp dependency  to rule it out
2020-06-16 08:08:25 <clandmeter> and i even saw some patches in musl so i build musl from 3.11
2020-06-16 08:09:16 <clandmeter> ikke: if i downgrade completely it will be just 3.11?
2020-06-16 08:09:24 <clandmeter> and it should probably work
2020-06-16 08:09:44 <ikke> good to rule out..
2020-06-16 08:10:01 <clandmeter> rule out what exactly?
2020-06-16 08:10:19 <ikke> that we have a known baseline that works
2020-06-16 08:10:43 <clandmeter> i already tried it on 3.11 i believe
2020-06-16 08:12:11 <clandmeter> i guess we need to debug the error.
2020-06-16 08:12:15 <clandmeter> but i have no clue how
2020-06-16 08:13:38 <ikke> maldridge: you said you had experience with containerd/ctr/
2020-06-16 08:13:42 <clandmeter> yes 3.11 is already tested. you can run the qemu script with 3.11 or 3.12
2020-06-16 08:14:50 <tmhoang> podman on alpine would be fun 
2020-06-16 08:15:19 <tmhoang> I like it more - especially the rootless feature
2020-06-16 08:16:31 <ikke> saidly that gitlab doesn't come with native podman support
2020-06-16 08:16:41 <tmhoang> :( 
2020-06-16 08:16:53 <ikke> sadly*
2020-06-16 08:17:33 <clandmeter> isnt podman a dropin replacement?
2020-06-16 08:18:13 <ikke> According to this: https://docs.gitlab.com/runner/executors/custom.html
2020-06-16 08:18:24 <ikke> yoneed to run a custom executor instead of the built-in docker executor
2020-06-16 08:18:50 <clandmeter> ah i remember again
2020-06-16 08:18:55 <clandmeter> there is some long stranding issue
2020-06-16 08:23:33 <ncopa> does this only affect s390x?
2020-06-16 08:23:40 <clandmeter> yes
2020-06-16 08:23:49 <clandmeter> we tried 3 other arches
2020-06-16 08:24:08 <ncopa> and mips64 does not work yet i guess
2020-06-16 08:24:15 <clandmeter> that i dont know
2020-06-16 08:24:17 <ncopa> which is the only other big-endian arch
2020-06-16 08:25:13 <ncopa> and this is the error? >
2020-06-16 08:25:14 <ncopa> "read unix @->@/containerd-shim/3fedff5d13348b7e084ab6f4a5dd59a191d7d9b4fda6633b89ae1a7106d4eac1.sock: read: connection reset by peer: unknown"
2020-06-16 08:25:19 <ikke> yes
2020-06-16 08:25:20 <clandmeter> yes sir
2020-06-16 08:26:08 <ncopa> and we have a stable reproducer?
2020-06-16 08:26:15 <ncopa> so it happens every time, not just occationally
2020-06-16 08:26:18 <ikke> yes
2020-06-16 08:26:21 <ncopa> good
2020-06-16 08:26:55 <ncopa> and if you copy containerd from 3.11 it works
2020-06-16 08:27:07 <ikke> We installed it from the 3.11 repo, and then it works
2020-06-16 08:27:42 <ikke> on s390x-ci.a.o itself, it's working now with containerd 1.3.2
2020-06-16 08:27:51 <ncopa> but not if you rebuild the same version on 3.12 it does not work
2020-06-16 08:28:06 <clandmeter> nope
2020-06-16 08:28:45 <clandmeter> so my idea was that it could be either the app itself, golang or anything other linked to it.
2020-06-16 08:29:46 <ncopa> makes sense
2020-06-16 08:30:36 <ncopa> oh
2020-06-16 08:30:58 <ncopa> one thing that changed from alpine 3.11 -> 3.12 was the GOFLAGS
2020-06-16 08:31:15 <clandmeter> ah yes
2020-06-16 08:31:21 <clandmeter> and its not defined in the apkbuild
2020-06-16 08:31:30 <clandmeter> its in abuild somewhere
2020-06-16 08:31:47 <ncopa> export GOFLAGS="-buildmode=pie"
2020-06-16 08:32:03 <ncopa> try build it on 3.12 but do: unset GOFLAGS
2020-06-16 08:32:30 <ncopa> does it happend with containerd built on edge as well?
2020-06-16 08:32:55 <ncopa> also, do we have the details recorded in an issue?
2020-06-16 08:33:08 <ncopa> may be good in case we need report something upstream
2020-06-16 08:33:20 <ncopa> so we dont need to re-read 500 lines of irc logs
2020-06-16 08:33:20 <ikke> Not yet, good idea
2020-06-16 08:33:37 <clandmeter> crap my container hangs
2020-06-16 08:33:45 <clandmeter> and its started with --rm :|
2020-06-16 08:34:03 <ikke> DOesn't seem to hang for me
2020-06-16 08:34:10 <clandmeter> thats not a container
2020-06-16 08:34:12 <clandmeter> thats a vm
2020-06-16 08:34:15 <ikke> ok
2020-06-16 08:34:18 <clandmeter> :p
2020-06-16 08:34:25 <ikke> try ctrl p + q
2020-06-16 08:35:12 <clandmeter> hmm
2020-06-16 08:35:57 <clandmeter> docker attach :D
2020-06-16 08:38:06 <ikke> yes
2020-06-16 08:40:57 <clandmeter> doesnt look like it works
2020-06-16 08:41:14 <ikke> if that doesn't work, you can try to exec another shell in the container
2020-06-16 08:41:32 <clandmeter> i mean new containerd
2020-06-16 08:41:39 <ikke> ah ok
2020-06-16 08:43:08 <clandmeter> ncopa: any way i can verify that the unset has worked?
2020-06-16 08:44:28 <clandmeter> https://tpaste.us/8Xky
2020-06-16 08:59:40 <ncopa> bah
2020-06-16 08:59:47 <ncopa> how do i build hello world in go
2020-06-16 09:01:57 <ncopa> clandmeter: you can use scanelf
2020-06-16 09:02:08 <ncopa> $ scanelf hello
2020-06-16 09:02:09 <ncopa>  TYPE   FILE 
2020-06-16 09:02:09 <ncopa> ET_DYN hello 
2020-06-16 09:02:25 <ncopa> if type is ET_DYN, then its a pie build
2020-06-16 09:02:55 <ncopa> $ scanelf hello
2020-06-16 09:02:55 <ncopa>  TYPE   FILE 
2020-06-16 09:02:55 <ncopa> ET_EXEC hello 
2020-06-16 09:03:08 <ncopa> ET_EXEC means its a non-pie build
2020-06-16 09:05:04 <clandmeter> ET_DYN /usr/bin/containerd
2020-06-16 09:06:06 <ncopa> containerd is linked to both libc and libseccomp
2020-06-16 09:06:25 <clandmeter> not mine
2020-06-16 09:06:34 <clandmeter> i removed libseccomp as my patch shows
2020-06-16 09:06:36 <ncopa> but its linked to libc, right?
2020-06-16 09:06:39 <clandmeter> yes
2020-06-16 09:06:45 <ncopa> so its still dynamically linked
2020-06-16 09:08:02 <ncopa> and it seems to be same version of runc
2020-06-16 09:10:59 <ncopa> someone had similar experience: https://github.com/moby/moby/issues/38742
2020-06-16 09:22:58 <clandmeter> ncopa: i dont understand what changed now?
2020-06-16 09:23:11 <clandmeter> if i uset it it should fall back to previous behaviour? 
2020-06-16 09:23:21 <clandmeter> btw, it hink we should move this to -devel
2020-06-16 11:08:36 <ikke> clandmeter: the gitlab host is still alpine 3.10. Do we want to upgrade it at some point?
2020-06-16 11:12:27 <clandmeter> maybe a good idea to do it with the v13 upgrade
2020-06-16 11:12:42 <ikke> I can do a test upgrade on test instance
2020-06-16 11:13:06 <ikke> Do we still want to wait for 13.1 before going to v13?
2020-06-16 11:16:52 <ikke> hmm
2020-06-16 11:24:09 <clandmeter> its not a rule, so you are free to upgrade
2020-06-16 11:24:24 <clandmeter> we are already on a few patch releases
2020-06-16 11:24:40 <clandmeter> next week it will be 13.1
2020-06-16 16:27:15 <clandmeter> mps: https://www.bloomberg.com/news/articles/2020-06-09/apple-plans-to-announce-move-to-its-own-mac-chips-at-wwdc
2020-06-16 16:27:41 <clandmeter> You can safely switch to macos now
2020-06-16 17:02:50 <mps> clandmeter: why do you think I'm hipster
2020-06-16 17:03:02 <mps> :)
2020-06-16 18:37:21 <clandmeter> i wanted to share it in offtopic but you are MIA
2020-06-16 18:48:05 <mps> yes, sorry. I left alpine-offtopic about week ago because it become more offtopic than alpine-offtopic and quite unpleasant for me to follow it
2020-06-16 19:02:41 <ikke> ppc64 runner network is slow :(
2020-06-17 10:36:53 <ikke> clandmeter: ok if I recreaste gitlab-test to get a fresh snapshot?
2020-06-17 10:37:01 <ikke> recreate*
2020-06-17 10:45:36 <clandmeter> Sure
2020-06-17 18:06:34 <maldridge> ikke: its probably best those jobs don't get picked up
2020-06-17 18:06:48 <maldridge> it appears as though this would use my branch to push to real repos without review
2020-06-17 18:08:47 <ikke> Yes, you are right. It's missing rules on when to run what job 
2020-06-17 18:09:32 <ikke> clandmeter: ^
2020-06-17 18:14:14 <clandmeter> ?
2020-06-17 18:15:48 <maldridge> clandmeter: I was able to send an MR that if it were to go through, would push containers for dabuild
2020-06-17 18:16:07 <maldridge> pipeline 23111
2020-06-17 18:16:23 <clandmeter> ci needs an update 
2020-06-17 18:18:19 <maldridge> anything I can do to assist with that update?  I'm trying to update a bunch of internal packages fro 3.12
2020-06-17 18:20:10 <maldridge> would very much like to remove the remaining 3.8 from my infrastructure
2020-06-17 23:37:25 <maldridge> oh, make_images has bashisms
2020-06-18 05:49:47 <ikke> removed the neostrada mirror
2020-06-18 12:05:27 <PureTryOut[m]> The Gitlab CI runners seem to be broken
2020-06-18 12:06:16 <ikke> PureTryOut[m]: I can look at it later
2020-06-18 12:07:44 <PureTryOut[m]> Thanks. I was working on the Qt 5.15 upgrade but broken CI makes it hard 😛
2020-06-18 12:08:30 <ikke> "fatal: unable to access 'https://gitlab.alpinelinux.org/alpine/aports/': error setting certificate verify locations:"
2020-06-18 12:09:30 <ikke> PureTryOut[m]: I think it's failing here: https://gitlab.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci/-/blob/master/overlay/usr/local/bin/build.sh#L141
2020-06-18 12:09:43 <ikke> PureTryOut[m]: can you see if you can reproduce it?
2020-06-18 12:09:53 <ikke> if you have time :)
2020-06-18 12:12:29 <PureTryOut[m]> Well I don't know what "CI_MERGE_REQUEST_PROJECT_URL" resolves to so no 😛
2020-06-18 12:12:54 <ikke> https://gitlab.alpinelinux.org/<username>/aports.git :)
2020-06-18 12:18:37 <PureTryOut[m]> Yeah seems to time out for me too
2020-06-18 12:24:40 <ikke> just timeout? Because in CI it complains about root ca bundles
2020-06-18 13:25:51 <PureTryOut[m]> Actually it got through after a few minutes
2020-06-18 13:27:11 <ikke> Can you try to run it in the alpinelinux/alpine-gitlab-ci docker container?
2020-06-18 13:33:30 <PureTryOut[m]> Uhh... That would be a first, no clue how to do that
2020-06-18 13:40:58 <ikke> docker run -it --rm alpinelinux/alpine-gitlab-ci
2020-06-18 13:41:58 <ikke> docker run -it --rm alpinelinux/alpine-gitlab-ci /bin/sh
2020-06-18 13:42:00 <ikke> to be sure
2020-06-18 13:55:28 <ncopa> does something complains about missing ca certficate bundles?
2020-06-18 13:55:39 <ncopa> i did a change in ca-certificates recently
2020-06-18 13:56:02 <ikke> yes
2020-06-18 13:56:24 <ikke> fatal: unable to access 'https://gitlab.alpinelinux.org/alpine/aports/': error setting certificate verify locations:
2020-06-18 13:56:26 <ikke>   CAfile: /etc/ssl/certs/ca-certificates.crt
2020-06-18 13:56:28 <ikke>   CApath: none
2020-06-18 13:57:21 <ncopa> using edge i suppose
2020-06-18 13:57:48 <ncopa> can you see what version of ca-certificates{,-bundle} is installed?
2020-06-18 13:59:01 <ikke> Purging ca-certificates (20191127-r3)
2020-06-18 13:59:12 <ikke> Purging ca-certificates (20191127-r3)
2020-06-18 13:59:14 <ikke> sorry
2020-06-18 13:59:16 <ikke> Upgrading ca-certificates-bundle (20191127-r3 -> 20191127-r4)
2020-06-18 14:18:11 <ikke> ncopa: so what should we do now?
2020-06-18 14:19:52 <ncopa> hum
2020-06-18 14:20:45 <ncopa> i suppose its curl that pulls it it in
2020-06-18 14:21:08 <ncopa> ca-certificates -r3 was broken because it purges the certificate when uninstalled
2020-06-18 14:21:17 <ncopa> it is fixed in ca-certificates -r4
2020-06-18 14:21:49 <ikke> but ca-certificates is still purged atm
2020-06-18 14:22:16 <ikke> meaning something does not depend on it anymore, right?
2020-06-18 14:36:41 <ncopa> i also did the change in curl to depend on ca-certificates-bundle
2020-06-18 14:36:59 <ncopa> so it got uninstalled, triggering the bug -r4 is fixing
2020-06-18 14:37:22 <ncopa> I suppose we are using a docker image?
2020-06-18 14:37:53 <ncopa> we could build new updated docker image
2020-06-18 14:38:09 <ncopa> but i suppose as quick fix I can revert curl update
2020-06-18 14:58:43 <ikke> so it's pure the upgrade that is breaking?
2020-06-18 15:05:24 <ncopa> yes
2020-06-18 15:05:32 <ncopa> well
2020-06-18 15:05:51 <ncopa> is is the removal that breaks it
2020-06-18 15:06:10 <ncopa> ca-certificates -r3 has a buggy post-install script
2020-06-18 15:06:14 <ncopa> post-deinstall*
2020-06-18 15:14:59 <ikke> ok
2020-06-18 15:15:05 <ikke> Then I will just regen the docker images
2020-06-18 15:16:39 <pyvpx> w/win 3
2020-06-18 15:16:43 <pyvpx> sorry
2020-06-18 16:02:54 <mps> what are our build arm hardware on packet
2020-06-18 16:03:40 <mps> one is Ampere iirc, but second one?
2020-06-18 16:04:02 <ikke> ThunderX
2020-06-18 16:04:15 <ikke> (funny, I was wondering the same, but could not remember Ampere :D)
2020-06-18 16:04:22 <mps> and Ampere run in 32bit mode?
2020-06-18 16:05:41 <mps> developer on #crystal-lang asked me, he is working on get sponsorship for crystal
2020-06-18 16:05:52 <ikke> checking
2020-06-18 16:10:24 <ikke> Ampere is aarch64
2020-06-18 16:10:36 <mps> and both runs aarch64 alpine as base OS? 
2020-06-18 16:11:29 <ikke> yes
2020-06-18 16:11:46 <ikke> armhf / armv7 is aarch64 in 32-bits mode
2020-06-18 16:11:51 <mps> thanks for answers
2020-06-18 16:20:35 <clandmeter> We have 2 diff ampere I think
2020-06-18 16:20:50 <ikke> usa4 seems to be hisilicon
2020-06-18 16:20:59 <ikke> at least, according to packet
2020-06-18 16:21:27 <mps> so we have more than 2 from packet?
2020-06-18 16:21:46 <clandmeter> Also x86
2020-06-18 16:22:03 <ikke> We have a bunch of servers from packet
2020-06-18 16:23:11 <mps> aha
2020-06-18 16:50:59 <ikke> PureTryOut[m]: I see CI is working again (I rebuilt the docker images)
2020-06-18 17:02:56 <PureTryOut[m]> Awesome!
2020-06-18 17:43:03 <ikke> ncopa: thanks for pointing that out :)
2020-06-18 19:49:51 <ikke> clandmeter: I've added nld3 and nld5 to zabbix :)
2020-06-18 20:24:38 <ikke> clandmeter: do you know what the dmeventd service is? It's crashed on both nld3 as nld5
2020-06-18 20:25:04 <ikke> devicemapper, something todo with lvm
2020-06-18 20:31:08 <mps> lvm2-dmeventd-2.02.186-r1 description: Device-mapper event daemon
2020-06-18 20:31:35 <ikke> I wonder why t consistently crashes
2020-06-18 20:32:00 <mps> is there any log
2020-06-18 20:32:50 <ikke> If there was, it's probably long gone
2020-06-18 20:35:09 <mps> iirc, it have option to log to stdout
2020-06-18 20:37:07 <mps> though I never used it in 'production', just tried for testing how it works and is it good for my use case
2020-06-18 20:37:39 <ikke> if I just run dmeventd, it exists with 0
2020-06-18 20:38:11 <ikke> ah, it forks
2020-06-18 20:38:16 <mps> hmm, try to look in man page, it should have option to run in foreground
2020-06-18 20:38:25 <ikke> yeah
2020-06-18 20:39:46 <ikke> It just runs, says it's ready
2020-06-18 20:39:52 <ikke> What was your usecase?
2020-06-18 20:40:44 <ikke> Just restarting it again
2020-06-18 20:41:48 <mps> I wanted to look can I use it as some kind of RAID for virtual machines in cloud
2020-06-18 20:43:38 <mps> but found that is easier to mount virtual disks under one of them, and more simple which was main reason to forget about device-mapper
2020-06-18 20:44:46 <mps> that was about 3-4 years ago
2020-06-18 20:45:50 <ikke> ok
2020-06-18 20:48:06 <mps> so, I can't help much with this, only remember it have different logging options
2020-06-18 21:03:33 <clandmeter> olla
2020-06-18 21:03:48 <clandmeter> ikke: yes dmeventd likes to crash
2020-06-18 21:04:04 <clandmeter> i never put any energy into it
2020-06-18 21:58:17 <clandmeter> Fastly TLS $0 for first 5 domains
2020-06-19 04:05:06 <maldridge> iirc its not necessary to keep dmeventd running if you use exclusively fixed disks
2020-06-19 04:56:58 <ikke> maldridge: ok, so if it crashes any all the time, we better can just stop it
2020-06-19 04:57:18 <ikke> clandmeter: does that mean we can provide our own certificates or something like that?
2020-06-19 05:34:14 <clandmeter> ikke: i already added one
2020-06-19 05:34:47 <ikke> ok
2020-06-19 05:34:49 <clandmeter> But I guess it need to be activated
2020-06-19 05:35:03 <clandmeter> Maybe you can help check
2020-06-19 05:35:27 <clandmeter> I looked a bit but could not find it that quickly
2020-06-19 07:00:44 <clandmeter> ikke: i think we need to update the cname to d.sni.global.fastly.net
2020-06-19 07:00:52 <clandmeter> if i understand the document correctly
2020-06-19 07:01:07 <clandmeter> but im not able to test cname locally
2020-06-19 07:13:45 <clandmeter> i set dl-cdn ttl to 5 min
2020-06-19 07:18:38 <clandmeter> will test in an hour or so
2020-06-19 18:26:35 <clandmeter> im updating dl-cdn
2020-06-19 18:26:42 <clandmeter> cname
2020-06-19 18:27:15 <clandmeter> if it doesnt work ill revert, but that means downtime of max 10min
2020-06-19 18:30:13 <ikke> k
2020-06-19 18:36:08 <clandmeter> ikke: it does not list ipv6 addresses
2020-06-19 18:36:13 <clandmeter> im not sure thats supported
2020-06-19 18:37:40 <clandmeter> ikke: please add a check for https cert expire for dl-cdn
2020-06-19 18:45:05 <clandmeter> ok looks like we a prepend dualstack to the hostname
2020-06-19 18:45:14 <clandmeter> lets see if that does not break things 
2020-06-19 18:48:25 <ikke> clandmeter: apparently I already have
2020-06-19 18:48:33 <ikke> Cert days left
2020-06-19 18:48:35 <ikke> 2020-06-19 18:45:3689 days
2020-06-19 18:48:38 <ikke> 2020-06-19 18:45:36 89 days
2020-06-19 18:48:50 <clandmeter> nice
2020-06-19 18:49:41 <ikke> How is that cert updated?
2020-06-19 18:51:04 <ikke> So after this ncopa could revert that change again for docker
2020-06-19 18:54:28 <clandmeter> not sure he already applied it for docker
2020-06-19 18:55:01 <clandmeter> fastly autoupdates it
2020-06-19 18:55:06 <clandmeter> if the record stays in dns
2020-06-19 18:55:19 <clandmeter> https://docs.fastly.com/en/guides/serving-https-traffic-using-fastly-managed-certificates#certificate-management-and-renewals
2020-06-19 18:56:11 <ikke> clandmeter: aha, ok, nice
2020-06-19 19:18:12 <ikke> I'm trying to get our infra into zabbix, but not all hosts have dmvpn
2020-06-19 20:45:39 <ikke> ouch, got bitten by a blocking outgoing firewall :(
2020-06-20 09:39:16 <clandmeter> ikke: im trying to do child pipelines for dynamic configs
2020-06-20 09:39:27 <clandmeter> im bumping to the same issue again
2020-06-20 09:39:37 <clandmeter> i assinged runners but it wont run :)
2020-06-20 09:39:45 <clandmeter> do i just tags?
2020-06-20 09:39:57 <clandmeter> mis tags 
2020-06-20 09:40:18 <ikke> can you show a job?
2020-06-20 09:41:22 <clandmeter> https://gitlab.alpinelinux.org/clandmeter/docker-abuild/-/jobs
2020-06-20 09:42:02 <clandmeter> i guess its because this is not under alpine it wont accept it
2020-06-20 09:42:54 <ikke> You are indeed missing tags
2020-06-20 09:43:06 <ikke> docker-alpine at minimum
2020-06-20 09:43:07 <clandmeter> which tags should i sue?
2020-06-20 09:43:14 <clandmeter> ok
2020-06-20 09:43:22 <clandmeter> for every job right?
2020-06-20 09:43:28 <ikke> Yes
2020-06-20 09:43:51 <ikke> The shared runners work for any project (otherwise aports MRs would not have any runners)
2020-06-20 09:45:26 <ikke> clandmeter: if you look here https://gitlab.alpinelinux.org/clandmeter/docker-abuild/-/settings/ci_cd
2020-06-20 09:45:28 <ikke> under runners
2020-06-20 09:45:33 <ikke> you see what tags each runner has
2020-06-20 09:46:40 <ikke> Oh, if you want to build docker images, then we need to assign the runners
2020-06-20 09:46:47 <ikke> they are now limited to the docker project
2020-06-20 09:47:00 <ikke> docker group*
2020-06-20 09:47:10 <clandmeter> vim is pissing me off with its auto tab crap in yml
2020-06-20 09:47:51 <ikke> I have :set sw=2 sts=2 ts=2 ai et
2020-06-20 09:47:53 <ikke> in my muscle memory
2020-06-20 09:57:10 <clandmeter> how can i assign that runner?
2020-06-20 09:59:10 <ikke> The runners are locked to a group, which means I cannot assign it to random projects :(
2020-06-20 10:00:55 <clandmeter> child pipelines are not very obvious...
2020-06-20 10:01:02 <clandmeter> https://gitlab.alpinelinux.org/clandmeter/docker-abuild/pipelines/23398
2020-06-20 10:01:57 <ikke> not obvious in what way?
2020-06-20 10:03:31 <clandmeter> why they wont start?
2020-06-20 10:05:01 <ikke> Ok, nothing happens when you click on them
2020-06-20 10:05:38 <clandmeter> i see that the child yml is not correct
2020-06-20 10:05:41 <clandmeter> maybe thats the issue
2020-06-20 10:09:03 <clandmeter> w00t
2020-06-20 10:09:18 <clandmeter> ikke: https://gitlab.alpinelinux.org/clandmeter/docker-abuild/pipelines/23401
2020-06-20 10:11:18 <clandmeter> still would be nice if it would mention something about the child if it fails.
2020-06-20 10:12:47 <clandmeter> hmm this whole child pipelines is weird
2020-06-20 10:13:01 <clandmeter> if the childs fail the main pipeline still shows green
2020-06-20 10:14:22 <clandmeter> it shows an error for downstream but when you click on it it will show its running (but its not) 
2020-06-20 10:14:47 <ikke> ah ok, I've not used it yet
2020-06-20 10:15:23 <ikke> Anoying that you cannot click on the child pipeline
2020-06-20 10:15:31 <clandmeter> you can now
2020-06-20 10:15:42 <clandmeter> oh yes that one
2020-06-20 10:15:44 <clandmeter> no idea
2020-06-20 10:16:17 <clandmeter> maybe with 13 it has some improivments
2020-06-20 10:16:27 <ikke> So the idea is that you dynamically generate the yml for all the different versions/
2020-06-20 10:16:28 <ikke> ?
2020-06-20 10:16:31 <clandmeter> they even mention that child pipes are broken in 12.9
2020-06-20 10:16:37 <clandmeter> yes
2020-06-20 10:16:39 <ikke> aha, ok
2020-06-20 10:16:43 <clandmeter> we now have releases.json
2020-06-20 10:16:51 <clandmeter> so i read that and generate the yml
2020-06-20 10:16:59 <ikke> aha, nice
2020-06-20 10:17:21 <clandmeter> it can be done much more simple
2020-06-20 10:18:08 <clandmeter> but i want to run all arches at ones
2020-06-20 10:18:15 <clandmeter> or it will take ages
2020-06-20 10:18:30 <ikke> yeah, makes sense
2020-06-20 10:18:49 <clandmeter> btw, 13.1 is out 
2020-06-20 10:18:51 <clandmeter> just the tag
2020-06-20 10:19:02 <clandmeter> so no official annoucement yet
2020-06-20 10:19:06 <ikke> ok
2020-06-20 10:19:18 <clandmeter> when we have time lets do v13
2020-06-20 10:19:20 <ikke> Will try the AL upgrade to 3.12 first
2020-06-20 10:19:34 <ikke> should not take too much time
2020-06-20 10:19:36 <clandmeter> ok i need to run to the sligro :)
2020-06-20 10:19:37 <ikke> (on the test instance)
2020-06-20 10:19:46 <clandmeter> well, bike :)
2020-06-20 11:14:37 <ikke> clandmeter: I have now a project image builder registered for x86_64 which I can assign to individual projects
2020-06-20 11:14:51 <ikke> If that's working, I can do it for the other arches as well
2020-06-20 11:54:38 <ikke> clandmeter: any idea what this mount means? "/dev/sda on /var/lib/docker type ext4 (rw,relatime)"
2020-06-20 11:55:01 <ikke> (note that /dev/sda is mounted on / as well)
2020-06-20 12:38:10 <clandmeter> nope, which host?
2020-06-20 12:39:01 <ikke> deu1-dev1
2020-06-20 12:47:44 <clandmeter> ikke: other docker hosts seems to have the same
2020-06-20 12:47:54 <ikke> yeah
2020-06-20 12:47:58 <ikke> just wondering what kind of mount it is
2020-06-20 12:48:14 <ikke> or what the purpose is
2020-06-20 12:48:27 <clandmeter> its not done by us i guess, something from within docker.
2020-06-20 12:49:58 <ikke> I'll just ignore it in Zabbix
2020-06-20 12:50:05 <ikke> otherwise we get double alerts 
2020-06-20 13:52:09 <Ariadne> at some point soon i should have a working docker-compose for opencfp
2020-06-20 13:59:22 <ikke> Ariadne: ok, nice
2020-06-20 15:29:17 <ikke> clandmeter: upgrade to 3.12 seems to work without issue
2020-06-20 17:14:45 <ikke> clandmeter: https://gitlab-test.alpinelinux.org/admin
2020-06-20 18:00:11 <clandmeter> Nice
2020-06-20 18:00:22 <clandmeter> I'm not home ATM
2020-06-20 19:13:53 <ikke> clandmeter: upgrading tomorrow?
2020-06-20 21:28:45 <clandmeter> ikke: ok, but im not home whole day, fathers day tomorrow.
2020-06-20 21:29:39 <ikke> aha, ok
2020-06-20 21:29:51 <ikke> armhf does seem stuck though
2020-06-20 21:34:06 <ikke> clandmeter: armhf builder seems stuck on sync (uniterruptable sleep)
2020-06-20 21:34:30 <ikke> same with armv7
2020-06-20 21:34:46 <clandmeter> git?
2020-06-20 21:35:06 <ikke> no, just `sync`
2020-06-20 21:35:10 <ikke> after apk add
2020-06-20 21:35:18 <ikke> at least, subprocess of apk add
2020-06-20 21:35:31 <mps> are they on usa4 ?
2020-06-20 21:35:34 <clandmeter> disk issues?
2020-06-20 21:35:55 <ikke> var/cache/misc/shared-mime-info-1.15-r0.trigger /usr/share/mime
2020-06-20 21:35:57 <ikke> mps: yes
2020-06-20 21:36:26 <mps> I'm building aarch64 linux-edge right now there
2020-06-20 21:36:39 <ikke> clandmeter: seems so
2020-06-20 21:36:41 <clandmeter> ikke: which host is that?
2020-06-20 21:36:50 <ikke> usa4-dev1
2020-06-20 21:37:04 <ikke> sd 3:0:0:0: [sdc] tag#32 timing out command, waited 180s
2020-06-20 21:37:23 <ikke> blk_update_request: I/O error, dev sdc, sector 202196808 op 0x1:(WRITE) flags 0x4a00 phys_seg 2 prio class 0
2020-06-20 21:37:36 <ikke> doesn't seem healthy
2020-06-20 21:37:50 <clandmeter> usa4-dev1 is not in netbox
2020-06-20 21:37:54 <clandmeter> only dev3
2020-06-20 21:38:00 <ikke> just usa4
2020-06-20 21:38:09 <ikke> usa4.alpin.pw
2020-06-20 21:38:36 <mps> multipath issue?
2020-06-20 21:39:13 <ikke> connection1:0: ping timeout of 3 secs expired, recv timeout 3, last rx 6400330503, last ping 6400331456, now 6400332416
2020-06-20 21:39:17 <ikke> I guess so
2020-06-20 21:42:33 <clandmeter> but it also errors on local apk?
2020-06-20 21:42:55 <ikke> local apk?
2020-06-20 21:43:12 <clandmeter> apk not in qemu
2020-06-20 21:43:27 <ikke> yes
2020-06-20 21:43:29 <ikke> it's not ci
2020-06-20 21:43:32 <ikke> the actual builders
2020-06-20 21:43:54 <ikke> I stopped the armhf container, and now lxc is confused
2020-06-20 21:44:11 <clandmeter> i tried to use lvs
2020-06-20 21:44:14 <clandmeter> but it hangs
2020-06-20 21:44:46 <clandmeter> i think we only use multipath (iscsi) for qemu
2020-06-20 21:47:59 <clandmeter> should we try a force reboot?
2020-06-20 21:48:17 <clandmeter> dmesg complains about sdc, but i think thats iscsi
2020-06-20 21:52:26 <clandmeter> i think we got bit by https://status.packet.com/incidents/2vf6gk94b3mh
2020-06-20 21:53:16 <clandmeter> ive done a reboot
2020-06-20 21:56:18 <ikke> ok
2020-06-20 21:56:46 <ikke> reboot was my suggestion as well
2020-06-20 21:56:57 <ikke> it's not rebooted yet
2020-06-20 21:57:03 <ikke> my ssh session is still alive
2020-06-20 21:57:10 <ikke> ok, now it's gone
2020-06-20 21:58:10 <clandmeter> i had to reboot from control panel
2020-06-20 21:58:19 <clandmeter> its posting
2020-06-20 21:58:32 <clandmeter> but its arm, so will take half a day
2020-06-20 21:59:02 <clandmeter> i think ncopa should have gotten an update about this
2020-06-20 21:59:07 <mps> hope my last downloaded files are saved
2020-06-20 21:59:19 <clandmeter> he is the owner
2020-06-20 21:59:33 <clandmeter> i moved ownership long time ago
2020-06-20 21:59:48 <clandmeter> but it seems he does not really put attention to these email notifications
2020-06-20 22:01:37 <clandmeter> ikke: its up again
2020-06-20 22:03:20 <clandmeter> i dont see dmvpn routes
2020-06-20 22:06:59 <clandmeter> ok i restarted nhrpd
2020-06-20 22:07:02 <clandmeter> seems ok now
2020-06-20 22:08:08 <clandmeter> its going to try to build missing pkgs now, first up linux-lts
2020-06-21 14:04:58 <clandmeter> ikke: still issues with the host?
2020-06-21 14:14:46 <ikke> clandmeter: unsure
2020-06-21 14:14:51 <ikke> I still see those messages in dmesg
2020-06-21 14:16:39 <clandmeter> hmm
2020-06-21 14:23:45 <clandmeter> I raised a ticket at packet
2020-06-21 14:24:26 <clandmeter> haha, it seems not to honor line breaks.
2020-06-21 14:24:33 <clandmeter> thats rough when you paste a dmesg
2020-06-21 14:55:47 <clandmeter> looks like they are aware of the iscsi issues
2020-06-21 15:13:33 <clandmeter> ikke: ack?
2020-06-21 15:14:07 <clandmeter> please follow https://status.packet.com/
2020-06-21 15:14:12 <clandmeter> should be added soon
2020-06-21 15:16:57 <clandmeter> Maybe turn off the runner
2020-06-21 15:17:14 <clandmeter> And turn off iscsi for now
2020-06-21 15:22:40 <ikke> clandmeter: What do we use iscsi fore?
2020-06-21 15:22:44 <ikke> the qemu runner?
2020-06-21 15:22:58 <clandmeter> Yes
2020-06-21 15:24:09 <ikke> ok
2020-06-21 15:24:24 <ikke> do we need to reboot again?
2020-06-21 15:42:54 <ikke> Yup, it's now on the status page
2020-06-21 15:42:58 <ikke> "Block Storage connectivity issue in EWR1"
2020-06-21 15:58:57 <clandmeter> Yup
2020-06-21 15:59:15 <clandmeter> If lxc is ok no need to reboot
2020-06-21 15:59:22 <ikke> ok
2020-06-22 05:34:34 <clandmeter> ikke: got mail about iscsi
2020-06-22 05:34:45 <clandmeter> Please restart services and try
2020-06-22 05:36:06 <ikke> What services? The entire server?
2020-06-22 06:00:45 <clandmeter> If you like
2020-06-22 06:00:54 <clandmeter> But I mean iscsi
2020-06-22 06:26:30 <ncopa> morning
2020-06-22 07:56:07 <clandmeter> ncopa: morning
2020-06-22 08:49:50 <ikke> clandmeter: How do I start the runner machine again? rc-service qemu-runner start says "`/usr/bin/qemu-system-x86_64' is not readable"
2020-06-22 08:50:44 <clandmeter> Just restart system
2020-06-22 08:50:58 <clandmeter> I have no time to check atm
2020-06-22 08:51:27 <ikke> ah, there is a runner.start in /etc/local.d
2020-06-22 08:55:45 <clandmeter> yes it starts via local service
2020-06-22 08:57:04 <ikke> I just restarted it
2020-06-22 08:57:09 <ikke> running that script didn't do a lot
2020-06-22 09:34:08 <ikke> it's running again
2020-06-22 09:37:41 <clandmeter> nice
2020-06-22 09:37:42 <clandmeter> thx
2020-06-22 09:37:58 <clandmeter> we still need to do 13.0.x
2020-06-22 09:38:05 <ikke> yes
2020-06-22 16:20:55 <clandmeter> https://www.linode.com/spotlight/alpine-linux/
2020-06-22 16:37:44 <maxice8> Nice
2020-06-22 17:03:48 <mps> :)
2020-06-22 17:08:53 <ikke> :)
2020-06-22 17:15:30 <clandmeter> https://about.gitlab.com/releases/2020/06/22/gitlab-13-1-released/
2020-06-22 17:17:10 <ikke> "Merge request reviews moved to core"
2020-06-22 17:17:19 <ikke> nice
2020-06-22 20:39:36 <mps> clandmeter: is that behind your new backyard?
2020-06-22 20:42:25 <maldridge> congrats on the limelight
2020-06-22 22:00:52 <clandmeter> mps: no my parents garden
2020-06-24 07:58:24 <clandmeter> ikke: seems they want to re-write docker-compose in golang.
2020-06-24 08:00:50 <ikke> aha, ok. Is a nice indication that they intend to keep support docker-compose
2020-06-24 08:01:32 <clandmeter> did github uii change?
2020-06-24 08:01:39 <ikke> yes
2020-06-24 08:01:55 <ikke> I already had it for a couple of days (opt-in)
2020-06-24 08:02:22 <clandmeter> i feels changed 
2020-06-24 08:02:31 <ikke> They moved your cheese
2020-06-24 08:02:35 <clandmeter> but cant really see much functional difference
2020-06-24 08:03:40 <ikke> no, mainly just design change
2020-06-24 08:04:27 <clandmeter> i dont see releases anymore
2020-06-24 08:04:36 <ikke> on the right
2020-06-24 08:04:40 <ikke> (I was looking for them as well)
2020-06-24 08:04:49 <ikke> "Latest release"
2020-06-24 08:05:00 <clandmeter> sigh
2020-06-24 08:05:15 <clandmeter> its one of the most used items for me
2020-06-24 08:07:51 <mps> releases are tags now, I think
2020-06-24 08:09:03 <ikke> That never changed
2020-06-24 08:09:36 <mps> hmm, tags then there is releases button
2020-06-24 09:50:03 <Cogitri> > ERROR: docker-compose-fish-completion-1.26.0-r0: BAD signature
2020-06-24 09:50:03 <Cogitri> Seems like the CDN does things again?
2020-06-24 09:55:16 <ikke> Cogitri: docker-compose was upgraded, downgraded and upgraded again
2020-06-24 09:55:31 <ikke> so the cdn still has the previously upgraded package in cache
2020-06-24 09:55:34 <Cogitri> Ah yes, we should enforce pkgrel bumps in that case
2020-06-24 09:55:36 <ikke> just a matter of purging it
2020-06-24 09:55:52 <Cogitri> Seems to work now, thanks
2020-06-24 15:14:34 <ncopa> seems like pushing to alpine-mksite does not update https://alpinelinux.org can somebody please regenereate the page manually?
2020-06-24 15:14:51 <ncopa> neither does wwwtest, so i only tested it locally and pushed to prod
2020-06-24 15:15:35 <ncopa> i fixed https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/issues/1
2020-06-24 21:55:17 <mps> in last days whenever I push something directly to origin (skipping MR) I have tell algitbot to retry
2020-06-25 04:45:30 <ikke> mps: yes, we have to update origin on the builders to point to gitlab.a.o instead of git.a.o
2020-06-25 08:18:15 <Cogitri> Can I somehow get the test-suite.log from s390x CI? https://gitlab.alpinelinux.org/Cogitri/aports/-/jobs/150914
2020-06-25 08:21:25 <ikke> the containers are destroyed again afterwards. 
2020-06-25 08:23:05 <Cogitri> Ah yes, on some Gitlabs the workspace is saved (but that needs a lot of storage, I guess)
2020-06-25 08:23:22 <Cogitri> I'll just see if I can print the test-suite.log to the CLI then
2020-06-25 08:24:01 <ikke> One option is to find some kind of general pattern that we can supply these things as artifacts
2020-06-25 08:26:38 <ikke> Cogitri: I guess these projects don't use docker runners
2020-06-25 08:26:48 <Cogitri> *.log ?
2020-06-25 08:27:06 <Cogitri> I think they save the workspace as artifact
2020-06-25 08:31:41 <ikke> It doesn't seem you supply recursive patterns
2020-06-25 08:32:13 <ikke> s/you/you can/
2020-06-25 08:32:13 <alpine-meetbot> ikke meant to say: It doesn't seem you can supply recursive patterns
2020-06-25 08:38:47 <Cogitri> Ah, that's unfortunate
2020-06-25 08:39:20 <ikke> maybe something like */*/src/*/*.log works :P
2020-06-25 08:53:46 <clandmeter> ikke: do we have an issue for that?
2020-06-25 08:53:51 <clandmeter> reg origin
2020-06-25 08:53:57 <ikke> clandmeter: no
2020-06-25 08:54:18 <clandmeter> i guess we need one.
2020-06-25 08:54:23 <ikke> yes
2020-06-25 08:54:27 <clandmeter> need to reserve some time to fix that
2020-06-25 08:54:31 <clandmeter> and update gitlab.
2020-06-25 08:54:37 <ikke> I have time tonight
2020-06-25 08:54:47 <ikke> also for gitlab
2020-06-25 08:54:56 <clandmeter> let me check my agenda
2020-06-25 08:55:25 <clandmeter> i think i can after dinner
2020-06-25 09:37:43 <ikke> clandmeter: ^
2020-06-25 09:37:58 <clandmeter> crash?
2020-06-25 09:38:55 <ikke> I cannot ping / ssh the builder either
2020-06-25 09:39:08 <clandmeter> probably network
2020-06-25 09:43:17 <ikke> ok, back
2020-06-25 11:07:32 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10697
2020-06-25 14:14:54 <clandmeter> ikke: nice
2020-06-25 14:15:04 <clandmeter> think we could script it to run it from the lxc host itself
2020-06-25 14:15:12 <clandmeter> else we get crazy :)
2020-06-25 14:15:22 <ikke> yup
2020-06-25 14:15:25 <clandmeter> btw, i just got invited for a BBQ
2020-06-25 14:15:39 <clandmeter> so i think i cannot make it tonight
2020-06-25 14:15:41 <ikke> Oh nice
2020-06-25 14:15:42 <ikke> right
2020-06-25 14:15:49 <clandmeter> i will be skinny dipping ;-)
2020-06-25 14:16:03 <clandmeter> no i wont post pictures
2020-06-25 14:16:11 <clandmeter> sorry mps
2020-06-25 14:16:13 <clandmeter> :p
2020-06-25 16:08:47 <mps> clandmeter: I don't need photos, but translation of 'skinny dipping' would be good to get
2020-06-25 16:13:41 <ikke> mps: You'd rather not know
2020-06-25 16:14:59 <mps> ah, some of the 'games' for big modern people not yet grown, thanks ikke. better if I don't know what is it :)
2020-06-25 17:28:43 <clandmeter> mps: naked swimming
2020-06-25 17:32:47 <mps> nothing else ?
2020-06-25 17:36:13 <mps> we are born naked, I tend to think :)
2020-06-26 09:55:21 <clandmeter> ikke, i updated the remotes on armv7 and aarch64 builders
2020-06-26 09:57:35 <clandmeter> with something like this https://tpaste.us/YpB4
2020-06-26 10:01:51 <ikke> Ok, thanks
2020-06-26 10:02:01 <ikke> will do the other ones as well
2020-06-26 10:48:18 <ikke> clandmeter: ugh, windows line endings :/
2020-06-26 10:49:31 <clandmeter> Huh?
2020-06-26 10:49:49 <ikke> or mac line endings, one of both
2020-06-26 10:49:53 <ikke> in that paste
2020-06-26 10:50:23 <clandmeter> tpaste was broeken in that host
2020-06-26 10:50:38 <clandmeter> Buy ik was om Linux
2020-06-26 10:50:42 <ikke> lol
2020-06-26 10:50:48 <ikke> nice autocorrect
2020-06-26 10:51:07 <clandmeter> It did not go to English :(
2020-06-26 10:51:19 <clandmeter> I was on Linux
2020-06-26 10:51:29 <clandmeter> Not sure where it came from
2020-06-26 10:52:04 <ikke> so I assume armhf was also fixed, right?
2020-06-26 10:53:07 <clandmeter> I'm not home atm
2020-06-26 10:53:17 <clandmeter> Should have the script in root
2020-06-26 10:53:39 <ikke> I assume because armhf and armv7 are one the same host
2020-06-26 11:00:53 <ikke> ok, done
2020-06-26 11:29:54 <clandmeter> nice
2020-06-26 11:30:01 <clandmeter> all hosts done now?
2020-06-26 11:30:07 <ikke> yes
2020-06-26 11:30:14 <clandmeter> cool
2020-06-26 11:30:24 <ikke> I did not verify armhf yet though
2020-06-26 11:30:29 <clandmeter> well its not that cool, so lets call it nice :)
2020-06-26 11:30:44 <ikke> yes, done as well
2020-06-26 11:31:08 <ikke> clandmeter: the heat is affecting you :P
2020-06-26 11:32:15 <clandmeter> i dont think its the heat, the effect is even in winter times ;-)
2020-06-26 11:33:21 <clandmeter> mps: your commits should be instant now.
2020-06-26 11:52:27 <mps> clandmeter: thanks, and thanks for info
2020-06-26 16:54:26 <ikke> clandmeter: ping
2020-06-26 17:02:09 <clandmeter> ikke: pang
2020-06-26 17:13:09 <ikke> pung
2020-06-26 17:14:01 <ikke> clandmeter: do you have time to look at the gitlab upgrade?
2020-06-26 17:14:16 <mps> :)
2020-06-26 17:14:34 <clandmeter> I'm not home
2020-06-26 17:14:38 <ikke> k
2020-06-26 17:14:38 <clandmeter> Later maybe
2020-06-26 17:14:47 <clandmeter> I pushed latest versions
2020-06-26 17:15:06 <ikke> I can do it on my own, as long as nothing breaks :)
2020-06-26 17:15:16 <ikke> Which I don't necessarily expect (we already tested it)
2020-06-26 17:15:30 <mps> (Lee Peng)
2020-06-26 17:15:38 <mps> https://en.wikipedia.org/wiki/Li_Peng
2020-06-26 17:15:39 <alpine-meetbot> [WIKIPEDIA] Li Peng | "Li Peng (Chinese: 李鹏; pinyin: Lǐ Péng; 20 October 1928 – 22 July 2019) was a Chinese politician. Known as the "Butcher of Beijing" for his role in the Tiananmen Square Massacre, Li served as the fourth Premier of the People's Republic of China from 1987 to 1998, and as the Chairman of the Standing Committee..."
2020-06-26 17:15:54 <mps> sorry 
2020-06-26 17:15:57 <clandmeter> Go for it
2020-06-26 17:16:27 <clandmeter> Is it breaks it will also break with me around
2020-06-26 19:03:11 <clandmeter> ikke: ping
2020-06-26 19:06:20 <ikke> pong
2020-06-26 19:06:39 <clandmeter> we want to give it a shot?
2020-06-26 19:09:19 <ikke> sure
2020-06-26 19:10:13 <ikke> Do we want to upgrade to 3.12 at the same time? 
2020-06-26 19:10:23 <ikke> The test instance went without issue, but not sure if there are differences
2020-06-26 19:10:43 <clandmeter> i think we can
2020-06-26 19:10:56 <clandmeter> should not really affect docker that much
2020-06-26 19:11:01 <clandmeter> it runs or it doesnt :)
2020-06-26 19:11:03 <ikke> heh
2020-06-26 19:11:07 <ikke> We noticed that :P
2020-06-26 19:11:58 <clandmeter> worse case scenario we revert to a backup
2020-06-26 19:12:09 <ikke> I hope not :)
2020-06-26 19:14:18 <ikke> I'll run the system upgrade
2020-06-26 19:16:28 <ikke> ok, done
2020-06-26 19:16:42 <clandmeter> should we reboot it first?
2020-06-26 19:16:55 <ikke> I think that's a good idea
2020-06-26 19:17:04 <clandmeter> i guess there is a kernel  update
2020-06-26 19:17:10 <ikke> No suprises next time we reboot
2020-06-26 19:17:18 <clandmeter> did y ou update-conf?
2020-06-26 19:17:24 <ikke> no, not yet
2020-06-26 19:17:27 <clandmeter> k
2020-06-26 19:18:15 <ikke> rc_cgroup_memory_use_hierarchy="YES
2020-06-26 19:18:23 <ikke> Did we set that?
2020-06-26 19:18:34 <clandmeter> uhm
2020-06-26 19:18:40 <ikke> in rc.conf
2020-06-26 19:18:47 <clandmeter> i know that setting
2020-06-26 19:18:52 <clandmeter> i just dont know why i do
2020-06-26 19:19:03 <ikke> I'll just leave it
2020-06-26 19:19:12 <clandmeter> i think its because of a boot warning
2020-06-26 19:20:23 <ikke> -xen_opts=dom0_mem=256M
2020-06-26 19:20:27 <ikke> +xen_opts=dom0_mem=384M
2020-06-26 19:20:33 <clandmeter> huh
2020-06-26 19:20:44 <clandmeter> that i dont know
2020-06-26 19:20:47 <ikke> seems like a default change
2020-06-26 19:20:51 <clandmeter> k
2020-06-26 19:21:34 <ikke> would be nice if update-conf could to a 3-way diff
2020-06-26 19:21:45 <ikke> but that would require us to keep the original somewhere
2020-06-26 19:22:49 <ikke> -features="ata base ide scsi usb virtio ext4"
2020-06-26 19:22:51 <ikke> +features="ata base cdrom ext4 keymap kms mmc raid scsi usb virtio"
2020-06-26 19:22:57 <ikke> mkinitfs.conf
2020-06-26 19:23:02 <clandmeter> keep the default
2020-06-26 19:23:06 <clandmeter> err
2020-06-26 19:23:10 <clandmeter> keep original
2020-06-26 19:23:19 <ikke> nod
2020-06-26 19:23:25 <clandmeter> if it boots it boots :)
2020-06-26 19:24:04 <ikke> /etc/update-extlinux.conf
2020-06-26 19:24:18 <ikke> default=lts
2020-06-26 19:24:39 <clandmeter> its not yet on lts?
2020-06-26 19:24:52 <clandmeter> which version were we on?
2020-06-26 19:25:08 <ikke> 4.19.78-0-virt
2020-06-26 19:25:15 <ikke> virt apparently
2020-06-26 19:25:18 <clandmeter> ah right, keep virt
2020-06-26 19:25:21 <clandmeter> its kvm 
2020-06-26 19:25:23 <ikke> But it's 'hardened' now
2020-06-26 19:25:41 <clandmeter> ok make it virt
2020-06-26 19:25:43 <ikke> ok
2020-06-26 19:27:12 <ikke> I think that's it
2020-06-26 19:27:27 <ikke> run update-extlinux I guess?
2020-06-26 19:28:39 <ikke> Ok, do we shutdown gitlab first? (docker-compose down)
2020-06-26 19:28:57 <clandmeter> ok you can
2020-06-26 19:29:02 <clandmeter> i guess docker will stop the containers
2020-06-26 19:31:42 <ikke> ok, done
2020-06-26 19:31:49 <ikke> rebooting the system now
2020-06-26 19:31:59 <clandmeter> cross fingers
2020-06-26 19:33:40 <ikke> ping is back
2020-06-26 19:33:45 <ikke> and I'm in
2020-06-26 19:34:01 <ikke> now to upgrade gitlab
2020-06-26 19:34:22 <ikke> pulling new image for 13.0
2020-06-26 19:34:30 <clandmeter> did gitlab start?
2020-06-26 19:34:33 <ikke> no
2020-06-26 19:34:35 <clandmeter> i guess not
2020-06-26 19:34:40 <ikke> The containers are not there
2020-06-26 19:34:41 <clandmeter> due to docker-compose down
2020-06-26 19:34:45 <ikke> (which was my intention)
2020-06-26 19:35:00 <clandmeter> would be nice to know if it would work without an upgrade
2020-06-26 19:35:20 <ikke> I can change the version back to test
2020-06-26 19:35:34 <ikke> I did the same on the test instance
2020-06-26 19:35:46 <clandmeter> i let you decide
2020-06-26 19:35:55 <clandmeter> its ok for me
2020-06-26 19:37:17 <ikke> started 12.10
2020-06-26 19:39:35 <ikke> currently waiting on 'gitlab_1    | Updating directories...'
2020-06-26 19:39:40 <ikke> for a while
2020-06-26 19:40:03 <ikke> ok, continuing
2020-06-26 19:40:54 <ikke> 'gitlab_1    | s6-svwait: fatal: unable to subscribe to events for /run/s6/web: No such file or directory'
2020-06-26 19:41:13 <ikke> It's up now
2020-06-26 19:41:21 <ikke> going for 13.0 now
2020-06-26 19:45:45 <ikke> Ok, up, 13.0.6
2020-06-26 19:45:50 <ikke> 13.0.7 
2020-06-26 19:45:52 <clandmeter> hmm
2020-06-26 19:45:53 <clandmeter> ok
2020-06-26 19:45:54 <clandmeter> :)
2020-06-26 19:56:37 <clandmeter> looks good
2020-06-26 19:57:02 <clandmeter> its raining cats and dogs here
2020-06-26 19:57:29 <ikke> No cloud to find here
2020-06-26 19:57:58 <clandmeter> looking at the radar, its coming your way
2020-06-26 19:58:37 <ikke> quite slowly
2020-06-26 20:02:56 <Cogitri> Hey, can you maybe give me the IP for my container cogitri-edge-aarch64.usa1.alpin.pw ? Seems like the name doesn't resolve for me even though I have alpine's DNS set
2020-06-26 20:29:02 <Cogitri> Ah nevermind, somehow dig can resolve it but ssh not
2020-06-26 20:30:00 <ikke> strange
2020-06-26 20:55:50 <ikke> clandmeter: ping
2020-06-26 20:58:17 <clandmeter> pang
2020-06-26 20:58:37 <clandmeter> did we break it?
2020-06-26 20:58:37 <ikke> clandmeter: does e-mail to ticket only work for alpine/aports?
2020-06-26 20:58:40 <ikke> no
2020-06-26 20:58:43 <ikke> afaik not
2020-06-26 20:58:53 <clandmeter> no its a general thing afaik
2020-06-26 20:59:10 <ikke> maybe it only shows up after there already is a ticket
2020-06-26 20:59:50 <ikke> https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/issues does not show the e-mail address to send it to
2020-06-26 21:00:02 <ikke> or perhaps because it's private
2020-06-26 21:00:13 <clandmeter> that would make sense
2020-06-26 21:01:30 <ikke> Would be nice to be able to forward mirror requests
2020-06-26 21:01:36 <clandmeter> https://gitlab.alpinelinux.org/alpine/infra/docker/ansible/-/issues/2
2020-06-26 21:01:37 <clandmeter> heh
2020-06-26 21:01:59 <ikke> why on the ansible project :D
2020-06-26 21:02:31 <ikke> first one listed or so?
2020-06-26 21:02:50 <clandmeter> i have no idea
2020-06-26 21:02:57 <clandmeter> brain fart i suspect
2020-06-26 21:03:43 <clandmeter> moving it to aports
2020-06-26 21:26:23 <maldridge> question: are all of alpine's build boxes in roughly the same physical place?
2020-06-26 21:27:25 <ikke> no
2020-06-26 21:27:35 <ikke> They are all over the world :)
2020-06-26 21:27:43 <maldridge> how do you handle bandwidth issues then?
2020-06-26 21:27:51 <maldridge> this is a problem I've recently started thinking more about with Void
2020-06-26 21:28:32 <ikke> Honestly we (I) haven't noticed any major bandwidth issues with our builders to be honest
2020-06-26 21:32:31 <maldridge> hmm, Ok
2020-06-26 21:33:06 <ikke> We have one CI builder in south-america that has issues from time to time
2020-06-26 21:36:19 <maldridge> s390x?
2020-06-26 21:36:33 <ikke> No, that one has a different issue
2020-06-26 21:36:46 <maldridge> like being an s390 machine :P
2020-06-26 21:37:00 <ikke> heh
2020-06-27 06:31:55 <Cogitri> Sorry, but could you maybe take a look at https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10692 ? pmOS wants to finish their image for the Pinephone pmOS community edition so it'd be nice if they had fresh appstream data on that
2020-06-27 07:17:30 <ikke> clandmeter: Where do you want to host this ^?
2020-06-27 07:26:45 <clandmeter> I don't mind
2020-06-27 07:27:11 <clandmeter> We have 2or3 docker boxes
2020-06-27 07:27:14 <ikke> yeah
2020-06-27 07:27:20 <ikke> deu1 or gbr2 I guess?
2020-06-27 07:27:41 <clandmeter> I don't know from my head
2020-06-27 07:28:10 <clandmeter> The cgit one had high cpu
2020-06-27 07:28:23 <ikke> Cogitri: I left one remark regarding the docker-compose config
2020-06-27 07:28:53 <ikke> ah, I see it's already fixed
2020-06-27 07:29:07 <clandmeter> Or you can try adding it in ansible ;)
2020-06-27 07:31:55 <ikke> I have no clue how it add it for just a single host, the current ansible code only contains generic definitions
2020-06-27 07:32:22 <ikke> Probably add a site entry for a single host?
2020-06-27 07:32:49 <maldridge> generally your ansible should be written to take variables 
2020-06-27 07:33:24 <clandmeter> The one now is very simplistic
2020-06-27 07:33:48 <ikke> clandmeter: right, deu1 has high cpu indeed
2020-06-27 07:44:24 <ikke> Do we want to move this project under the alpine infra group btw?
2020-06-27 08:03:35 <ikke> Cogitri: do you mind me moving the project under alpine/infra/compose? fyi I'll make you maintainer and make the project public then
2020-06-27 08:07:55 <Cogitri> Ah sure, no problem with that. Thanks for looking into this :)
2020-06-27 08:13:40 <ikke> ok, done
2020-06-27 09:58:25 <ikke> Cogitri: I get a segfault when building the image :-/
2020-06-27 10:02:03 <ikke> https://tpaste.us/5Eb9 
2020-06-27 10:10:11 <Cogitri> Oof
2020-06-27 10:10:33 <Cogitri> On x86_64? I can look into it in a bit
2020-06-27 10:10:37 <ikke> yes
2020-06-27 10:11:18 <ikke> Cogitri: maybe it's better to separate the docker image from the compose project, like we do for other projects. then the image is built and uploaded by CI and we don't need to build it on the hosts
2020-06-27 10:11:26 <ikke> (not that it would solve this problem necessarily)
2020-06-27 11:00:43 <Cogitri> ikke: Huh, I just built the thing locally and that worked just fine on my x86_64 machine
2020-06-27 11:01:06 <Cogitri> I did docker build -t appstream .
2020-06-27 11:41:45 <Cogitri> As for the splitting: Sure, I don't mind yhat but not sure what's necessary for that
2020-06-27 11:42:32 <ikke> Another project on gitlab, move the Dockerfile + a simple gitlab-ci.yml file there
2020-06-27 11:42:46 <Cogitri> And maybe it only SEGFAULTs with the many buildjobs we have (ninja defaults to -j$(nproc)), so maybe try with ninja -j1 in the Dockerfile?
2020-06-27 11:42:50 <ikke> And I need to create a repo on docker hub
2020-06-27 11:42:58 <ikke> I can try
2020-06-27 11:43:18 <ikke> though nproc is 4 on that host
2020-06-27 11:43:57 <Cogitri> Oh, I built with my laptop (so 4 cores/8 threads)
2020-06-27 11:51:22 <Cogitri> Also, is pkgs.a.o down?
2020-06-27 11:52:07 <ikke> no
2020-06-27 12:17:11 <Cogitri> Ah weird, seems to work again for me, not sure what was that
2020-06-27 12:27:18 <ikke> Now it's raining cats and dogs here
2020-06-27 12:30:34 <ikke> Cogitri: https://gitlab.alpinelinux.org/alpine/infra/docker/appstream-generator
2020-06-27 12:31:29 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/alpine-gitlab-ci/-/blob/master/.gitlab-ci.yml
2020-06-27 12:39:06 <Cogitri> Ah nice, thanks
2020-06-27 12:39:25 <Cogitri> What do I put it in the 1st repo? 😅
2020-06-27 12:39:40 <Cogitri> The Dockerfile?
2020-06-27 12:44:49 <ikke> yes
2020-06-27 16:47:16 <Cogitri> ikke: The dockerfile builds fine now (thanks for the help so far! :) but seems like it's missing some credentials? https://gitlab.alpinelinux.org/alpine/infra/docker/appstream-generator/-/jobs/152549#L32
2020-06-27 17:10:31 <ikke> Cogitri: ah, forgot to give algitbot access to that docker repo
2020-06-27 17:12:14 <ikke> Cogitri: job succeded
2020-06-27 17:12:32 <ikke> strange that it fails on the host itself
2020-06-27 17:13:12 <Cogitri> Yup, strange indeed
2020-06-27 17:14:44 <Cogitri> Ok, pushed an updated docker-compose.yml
2020-06-27 17:51:47 <ikke> Cogitri: it's running now
2020-06-27 17:52:03 <ikke> the appstream generator
2020-06-27 17:54:11 <Cogitri> Great, thanks :)
2020-06-27 17:55:04 <ikke> Once it resolves: https://appstream.alpinelinux.org
2020-06-27 18:17:44 <ikke> it's working for me now
2020-06-27 18:40:16 <ikke> Cogitri: it's finished, but the log shows some errors
2020-06-27 18:41:04 <ikke> not sure if that's expected
2020-06-27 18:42:30 <Cogitri> Can you send me the log?
2020-06-27 18:42:36 <Cogitri> Seems like 3.12 wasn't generated
2020-06-27 18:47:01 <ikke> Cogitri: last 200 lines: https://tpaste.us/QnYQ
2020-06-27 18:47:45 <ikke> let me know if you need more
2020-06-27 18:48:14 <Cogitri> That should be fine, just some invalid desktop files
2020-06-27 18:49:05 <ikke> ok, in the end it says it could not generate the html due to a template missing
2020-06-27 18:55:10 <Cogitri> Will look into that in a bit, but that's just an overview of what packages got generated, so not fatal that it's missing
2020-06-27 18:56:14 <ikke> nod
2020-06-27 19:19:07 <Cogitri> Hm, does the log mention something about 3.12?
2020-06-27 19:19:48 <ikke> no, only edge
2020-06-27 19:22:40 <Cogitri> Weird
2020-06-27 19:24:04 <ikke> the exit code is 1
2020-06-27 19:32:52 <Cogitri> Ah, I guess I messed something up in the config then
2020-06-27 19:33:14 <Cogitri> Is the job automatically run?
2020-06-27 19:34:20 <ikke> No
2020-06-27 19:35:31 <ikke> If you want it scheduled, you should add cron in the appstream-generator container that automaticall runs the job
2020-06-27 19:35:39 <ikke> same as we do for pkgs.a.o
2020-06-27 19:36:40 <Cogitri> Oh, somehow I had expected one would schedule the container to be run as cron and not do cron inside the container
2020-06-27 19:36:46 <Cogitri> I'll look into that, thanks!
2020-06-27 22:02:38 <Cogitri> Ok. pushed a new version which should generate for 3.12 as well now (and put files into a dir that's versioned by date)
2020-06-27 22:06:33 <Cogitri> Would it be possible to clean up the current dir and run it again? It should run with cron afterwards
2020-06-27 22:14:54 <ikke> "appstream-generator_appstream-generator_1 exited with code 0"
2020-06-27 22:15:31 <ikke> needed to update the compose project
2020-06-27 22:17:04 <ikke> ok, should be running now
2020-06-27 22:25:51 <Cogitri> Great, thanks
2020-06-27 22:27:51 <Cogitri> Seems to 404 now, but maybe it just needs a little?
2020-06-27 22:31:12 <ikke> 2020/06/27 22:30:11 [error] 20#20: *3 "/static/export/index.html" is not found (2: No such file or directory), client: 172.23.0.15, server: localhost, request: "GET / HTTP/1.1", host: "appstream.alpinelinux.org"
2020-06-27 22:32:15 <Cogitri> Maybe the cronjob didn't run yet (or I messed that up?)
2020-06-27 22:32:31 <Cogitri> Oh right, it generates to /static/$date now
2020-06-27 22:33:55 <Cogitri> Updated the nginx config
2020-06-27 22:43:32 <ikke> ok, no more 404
2020-06-27 22:47:03 <Cogitri> Okay, patched appstream-generator into shape so it can export to a certain dir, I think with my current setup caching wouldn't work properly. Once the new docker image is done building it should hopefully work. Sorry for the trouble :)
2020-06-28 06:50:55 <Cogitri> https://build.alpinelinux.org/buildlogs/build-edge-x86_64/community/firefox/firefox-77.0.1-r3.log seems like x86_64 has no storage left?
2020-06-28 09:52:11 <Cogitri> ikke: ^
2020-06-28 10:02:49 <ikke> Cogitri: yeah, I already freed up a bit of space
2020-06-28 10:06:54 <Cogitri> Ah okie
2020-06-28 10:10:20 <Cogitri> Can you also purge the static volume and restart the appstream container? Should export to the correct things then and properly cache
2020-06-28 10:11:40 <ikke> ok, note that I already purged it last night
2020-06-28 10:11:43 <ikke> should I do it again
2020-06-28 10:11:51 <ikke> (but before your latest image)
2020-06-28 11:06:14 <ikke> Cogitri: recreated the containers and volumes from the latest image
2020-06-28 11:19:14 <Cogitri> Okie, thanks!
2020-06-28 11:19:17 <Cogitri> 
ACTION hopes it just works now :)

2020-06-28 15:14:21 <ikke> clandmeter: ping (wrongly pinged you in #alpine-linux)
2020-06-28 15:14:34 <clandmeter> ok
2020-06-28 15:15:35 <ikke> I did a sys upgrade on gbr2-dev1 last night (frankly just to get an up-to-date tpaste)
2020-06-28 15:15:45 <ikke> but that also installed a new kernel (with kernel modules)
2020-06-28 15:16:18 <ikke> I had to restart docker as well
2020-06-28 15:18:38 <clandmeter> ok
2020-06-28 15:18:45 <clandmeter> thats packet i guess?
2020-06-28 15:18:47 <clandmeter> err
2020-06-28 15:18:54 <clandmeter> linode
2020-06-28 15:19:00 <ikke> yeah
2020-06-28 15:20:04 <clandmeter> and all ok?
2020-06-28 15:20:15 <ikke> It just cannot load new kernel modules
2020-06-28 15:20:27 <clandmeter> did you reboot?
2020-06-28 15:20:30 <ikke> no
2020-06-28 15:20:36 <clandmeter> you need to
2020-06-28 15:20:41 <clandmeter> with new kernel
2020-06-28 15:20:55 <clandmeter> alpine does not keep old modules/kernel
2020-06-28 15:20:58 <ikke> Yeah, I understand
2020-06-28 15:21:02 <ikke> same with archlinux
2020-06-28 15:21:21 <clandmeter> ok, so you have a question or issue?
2020-06-28 15:21:35 <ikke> Just wanted to discuss with you what the best option was
2020-06-28 15:21:40 <ikke> just reboot it is :)
2020-06-28 15:21:49 <clandmeter> yeah :)
2020-06-28 15:21:56 <ikke> but your irc session is on that box
2020-06-28 15:22:02 <clandmeter> oh ok
2020-06-28 15:22:14 <ikke> and pkgs.a.o
2020-06-28 15:22:16 <clandmeter> not only mine
2020-06-28 15:22:19 <ikke> yup
2020-06-28 15:22:25 <ikke> ncopa and danieli as well
2020-06-28 15:22:40 <clandmeter> danieli was angry with me last time ;-)
2020-06-28 15:22:48 <danieli> haha, not angry, but it was a bit unexpected
2020-06-28 15:22:54 <ikke> :P
2020-06-28 15:23:02 <clandmeter> reboot is ok for me
2020-06-28 15:23:08 <danieli> ack
2020-06-28 15:23:13 <clandmeter> i dont have friends anyway on irc ;-)
2020-06-28 15:26:52 <ikke> ok, all seems right
2020-06-28 15:26:54 <danieli> it's alive, it's aliveeee!
2020-06-28 17:05:00 <Cogitri> ikke: Sorry, did another small change to the Dockerfile and docker-compose, seems to work locally now
2020-06-28 17:06:24 <ikke> ok
2020-06-28 17:07:13 <ikke> purge everything again?
2020-06-28 17:07:56 <Cogitri> Shouldn't be necessary I think
2020-06-28 17:07:58 <ikke> ok
2020-06-28 17:08:23 <ikke> restarted the container
2020-06-28 17:09:36 <Cogitri> Thanks
2020-06-28 17:10:33 <ikke> Note that the cron will run at 2am
2020-06-28 17:12:16 <Cogitri> Ah, thanks for mentioning, somehow I had expect the cronjob would start immediately on start of the container and then wait 24hrs
2020-06-28 17:15:34 <ikke> the daily periodic cron is just hardcoded to run at 2am
2020-06-28 17:23:45 <Cogitri> Oh okie :)
2020-06-28 17:24:56 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/alpine-baselayout/crontab#L5
2020-06-28 17:28:17 <ikke> running it manually now
2020-06-28 17:28:44 <ikke> ugh
2020-06-28 17:28:48 <ikke> segfault :-/
2020-06-28 17:29:01 <Cogitri> Huh what, how does it SEGFAULT now
2020-06-28 17:29:12 <ikke> 2020-06-28 17:28:37 - INFO: Processed gcr/3.36.0-r0/aarch64, components: 0, hints: 0
2020-06-28 17:29:15 <ikke> 2020-06-28 17:28:37 - DEBUG: Looking for icon 'lftp-icon' for 'lftp/4.9.1-r0/aarch64::lftp.desktop' (XDG)
2020-06-28 17:29:16 <ikke> Segmentation fault
2020-06-28 17:34:27 <ikke> [ 7353.411268] appstream-gener[24624]: segfault at 0 ip 00007fa08c0127fe sp 00007f80870753d0 error 4 in ld-musl-x86_64.so.1[7fa08bfdd000+47000]
2020-06-28 17:50:21 <Cogitri> How weird, the container seems to work just fine on my laptop and vps 🤔
2020-06-28 17:51:00 <Cogitri> I guess maybe purge the volume for good measure, but this sure is weird
2020-06-28 17:56:15 <Cogitri> Ah, seems like it SEGFAULTed for me now as well, time to debug, thanks for the message
2020-06-28 18:00:48 <ikke> np
2020-06-28 18:53:53 <Cogitri> Okie, pushed a fix for the asgen-config.json that should mitigate that
2020-06-28 18:54:31 <ikke> ok
2020-06-28 18:55:21 <ikke> running it again
2020-06-28 19:13:02 <Cogitri> Thanks
2020-06-28 19:14:30 <ikke> Cogitri: one potential fix, add -p to the mkdir of the current day so that you can run it more then once per day
2020-06-28 19:14:44 <ikke> ok, it just finished
2020-06-28 19:15:08 <ikke> did v3.12 now as well
2020-06-28 19:15:47 <Cogitri> Nice
2020-06-28 19:16:20 <Cogitri> Ah, I didn't really think it'd be relevant since we only run daily (and I think appstream-generator complains about putting data into a non-empty dir)
2020-06-28 19:16:52 <ikke> yeah
2020-06-28 19:17:00 <ikke> just noticed it when I tried to run it again after the segfault
2020-06-28 19:54:06 <Cogitri> Ah yes, hopefully won't happen again though :)
2020-06-28 19:54:18 <Cogitri> Seems like everything works now, thanks for the help!
2020-06-28 19:55:17 <Cogitri> Ah, another thing - did you get to making the "make aports forks public" script a cronjob yet or do we still have to ping you for that?
2020-06-28 20:49:44 <ikke> still manually :)
2020-06-28 21:09:56 <clandmeter> ikke: thx for fixing the appstream generator. Cogitri: sorry it took some time (im not that active lately...)
2020-06-28 21:42:37 <Cogitri> Ah no worries about that :)
2020-06-28 21:52:36 <Cogitri> Thanks for all the work infra does, we could hardly do anything without that :)
2020-06-29 06:40:26 <ncopa> morning. the alpine-mirrors package. do we still need it? It fails to build:
2020-06-29 06:40:33 <ncopa> >>> ERROR: alpine-mirrors: Following mirrors failed:
2020-06-29 06:40:33 <ncopa> http://ftp.yzu.edu.tw/Linux/alpine/
2020-06-29 06:40:33 <ncopa> http://nl.alpinelinux.org/alpine/
2020-06-29 06:40:33 <ncopa> http://dl-4.alpinelinux.org/alpine/
2020-06-29 06:40:33 <ncopa> http://dl-5.alpinelinux.org/alpine/
2020-06-29 06:40:34 <ncopa> http://mirror.rise.ph/alpine
2020-06-29 06:40:56 <ncopa> and i suspect it is not maintained?
2020-06-29 06:58:24 <ikke> I think mirrors.a.o is now covering that?
2020-06-29 06:58:31 <ikke> which is maintained
2020-06-29 06:59:07 <ikke> is setup-alpine still using `/usr/share/alpine-mirrors/*`?
2020-06-29 07:13:06 <maldridge> what is backing mirrors.a.o?
2020-06-29 07:13:54 <ikke> A repo on gitlab
2020-06-29 07:17:23 <maldridge> interesting
2020-06-29 10:38:45 <ikke> Cogitri: it ran succesfully tonight :)
2020-06-29 11:36:04 <Cogitri> Great :)
2020-06-29 21:08:26 <ikke> clandmeter: TIL https://docs.docker.com/compose/extends/
2020-06-29 21:10:54 <clandmeter> nice, never seen it.
2020-06-29 21:15:58 <ikke> kde is moving to gitlab as well
2020-06-29 21:16:12 <ikke> https://about.gitlab.com/blog/2020/06/29/welcome-kde/
2020-06-29 21:17:50 <ikke> not sure if they are self-hosting or using gitlab.com
2020-06-29 21:18:12 <ikke> ah, community edition, so self-hosted
2020-06-30 11:16:53 <mps> ikke: yes, https://dot.kde.org/2020/06/30/kdes-gitlab-now-live