2021-08-01 15:13:11 <ikke> clandmeter: we cannot just set a cname for dl-2 to dl-cdn, because dl-cdn does not recognize dl-2.a.o
2021-08-01 15:18:46 <clandmeter> You mean for https
2021-08-01 15:19:00 <ikke> No
2021-08-01 15:19:03 <ikke> in general
2021-08-01 15:19:11 <ikke> <title>Fastly error: unknown domain dl-2.alpinelinux.org</title>
2021-08-01 15:19:28 <ikke> I'll just use a redirect from nld3
2021-08-01 15:19:39 <ikke> curl -I --resolve dl-2.alpinelinux.org:80:147.75.101.119 http://dl-2.alpinelinux.org/alpine
2021-08-01 15:19:45 <ikke> HTTP/1.1 301 Moved Permanently
2021-08-01 15:19:53 <ikke> Location: http://dl-cdn.alpinelinux.org/alpine
2021-08-01 15:21:34 <ikke> clandmeter: agree: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/7
2021-08-01 15:21:43 <ikke> o wait
2021-08-01 15:21:49 <ikke> can remove -orig now
2021-08-01 15:21:57 <clandmeter> You can add it to the backend
2021-08-01 15:22:24 <ikke> clandmeter: right
2021-08-01 15:22:27 <ikke> do we want to do that?
2021-08-01 15:22:39 <clandmeter> Redirect is probably better
2021-08-01 15:22:43 <ikke> yeah
2021-08-01 16:03:42 <ikke> w00t
2021-08-01 16:03:47 <ikke> pushing via ssh works!
2021-08-01 16:42:34 <ikke> Well, it's expected :P
2021-08-01 20:44:48 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/merge_requests/8
2021-08-01 21:00:10 <ikke> Would be nice if someone could review it.
2021-08-02 09:04:27 <ncopa> good morning
2021-08-02 09:04:45 <ncopa> anyone want to work on the new ppc64le machine, and try get alpine installed on it?
2021-08-02 09:04:48 <clandmeter> o/
2021-08-02 09:05:04 <clandmeter> do we have oob?
2021-08-02 09:05:48 <ncopa> i dont think we do. did we have that with the old ppc64le?
2021-08-02 09:06:03 <clandmeter> i dont recall installing it
2021-08-02 09:06:09 <clandmeter> maybe you did?
2021-08-02 09:07:36 <ncopa> i don tthink we had on the old one
2021-08-02 09:08:18 <clandmeter> well, who wants to burn himself on installing alpine baremetal without oob? 
2021-08-02 09:09:11 <clandmeter> im not even sure how the boot procedure on those ppc machines work
2021-08-02 09:09:57 <ncopa> i wonder if it was roberto oliveria that did it last time
2021-08-02 09:10:03 <ncopa> i guess we could ask him how he did it
2021-08-02 09:10:24 <clandmeter> i bet they have physical access :)
2021-08-02 09:10:55 <ncopa> in either case, i think we should get started with it sooner than later, since they will power of the ppc64le vms at the end of this week
2021-08-02 09:11:17 <clandmeter> i think we should ask them to wait a bit with that.
2021-08-02 09:11:32 <clandmeter> for sure if you want to run baremetal
2021-08-02 09:12:04 <clandmeter> the previous baremetal we were using, was it the same version?
2021-08-02 09:17:39 <ncopa> it was power8 and this is power8 too, so yes, i think it was the same version of the hardware
2021-08-02 09:18:04 <ncopa> but this one seems to have a CPU less (10 less cores)
2021-08-02 09:18:41 <ncopa> seems like this one has 10 cores, but only 9 working
2021-08-02 10:40:13 <ikke> Without oob access it seems not feasible for us to install alpine on i
2021-08-02 10:40:15 <ikke> it*
2021-08-02 10:41:22 <ikke> ncopa: I do want to work on it, but like clandmeter not really sure how to get ppc64le booting
2021-08-02 10:42:29 <clandmeter> i think we should ask them, 1, can we have a bit more time to migrate, 2. is it possible to install alpine on it (like we had before)
2021-08-02 10:43:08 <clandmeter> ikke: that foobar mirrors is really unstable isnt it?
2021-08-02 10:43:12 <ikke> yes
2021-08-02 10:43:15 <clandmeter> s/mirrors/mirror
2021-08-02 10:43:15 <alpine-meetbot> clandmeter meant to say: ikke: that foobar mirror is really unstable isnt it?
2021-08-02 10:43:30 <clandmeter> i think we should just remove it
2021-08-02 10:44:08 <ikke> https://zabbix.alpinelinux.org/report2.php?triggerid=16160&filter_templateid=0
2021-08-02 10:44:16 <clandmeter> i think upcoming 5 weeks I will have more time available
2021-08-02 10:44:29 <ikke> Ok
2021-08-02 10:44:32 <ikke> linorg as well btw
2021-08-02 10:44:37 <ikke> linorg.usp.br
2021-08-02 10:45:20 <clandmeter> looks like it
2021-08-02 10:46:09 <ikke> Did you see the gitlab MR?
2021-08-02 10:46:37 <clandmeter> ive seen a lot of MR's
2021-08-02 10:46:47 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/merge_requests/8
2021-08-02 10:46:58 <clandmeter> yes
2021-08-02 10:47:19 <ikke> Would like to get your feedback
2021-08-02 10:47:25 <clandmeter> it will take half my holiday to review it ;-)
2021-08-02 10:47:28 <ikke> :D
2021-08-02 10:47:54 <clandmeter> np i will look at it.
2021-08-02 10:48:12 <clandmeter> i guess this means it actually works? :)
2021-08-02 10:48:26 <ikke> It seems to work
2021-08-02 10:48:46 <ikke> but I suppose we should test it more thoroughly
2021-08-02 10:49:06 <clandmeter> we should extend the test suite
2021-08-02 10:49:10 <ikke> yes
2021-08-02 10:49:24 <clandmeter> look into history what we bump into and add a test case for it
2021-08-02 10:49:40 <ikke> * can we clone a project
2021-08-02 10:49:52 <ikke> * can we push via ssh
2021-08-02 10:49:56 <ikke> * can we push via http
2021-08-02 10:50:09 <clandmeter> do ppl actually push via http?
2021-08-02 10:50:14 <clandmeter> or you mean via webif?
2021-08-02 10:50:14 <ikke> I assume they do
2021-08-02 10:50:51 <clandmeter> do these cli tools push via http?
2021-08-02 10:51:04 <ikke> It uses the remote that you confiugred
2021-08-02 10:51:06 <ikke> configured*
2021-08-02 10:51:51 <clandmeter> but http is possible i guess
2021-08-02 10:52:10 <clandmeter> that would need no external tools except the cli tool
2021-08-02 10:54:16 <ikke> clandmeter: one challenge I have with docker-compose is that only the .env file can be used to provide global environment variables
2021-08-02 10:56:04 <clandmeter> ikke: this switches to pg12?
2021-08-02 10:56:14 <ikke> clandmeter: we already use it
2021-08-02 10:56:28 <clandmeter> we do?
2021-08-02 10:56:30 <ikke> yes
2021-08-02 10:56:34 <clandmeter> thats the update we did last time?
2021-08-02 10:57:00 <ikke> yes
2021-08-02 10:57:04 <clandmeter> oh we have local changes to compose i guess
2021-08-02 10:57:18 <ikke> even better, it's not a git repo
2021-08-02 10:57:30 <clandmeter> or worse :)
2021-08-02 10:57:55 <ikke> ;-)
2021-08-02 10:58:54 <ncopa> re ppc64le, i think that roberto probably knows better than current IBM how to do it, so i was thinking that we could email roberto directly
2021-08-02 10:59:10 <ncopa> some hints here: https://wiki.alpinelinux.org/wiki/Ppc64le
2021-08-02 11:56:33 <ikke> whoops
2021-08-02 11:56:36 <ikke> we have a large CI backlog
2021-08-02 14:51:37 <nmeum> how does the invitation mail process for the tsc work? should we just create a meeting link to the bbb instance clandmeter has setup and give that to the person sending the invation mail?
2021-08-02 14:51:41 <nmeum> https://gitlab.alpinelinux.org/alpine/tsc/-/issues/5#note_171407 etc.
2021-08-02 18:55:13 <danieli> is 172.16.23.17 / my x86_64 build container dead?
2021-08-02 18:55:29 <danieli> debug1: Local version string SSH-2.0-OpenSSH_8.5
2021-08-02 18:55:29 <danieli> kex_exchange_identification: read: Connection reset by peer
2021-08-02 18:55:29 <danieli> Connection reset by 172.16.23.17 port 22
2021-08-02 18:56:39 <ikke> let me check
2021-08-02 18:58:29 <ikke> can you try again?
2021-08-02 18:58:51 <danieli> works flawlessly
2021-08-02 18:58:54 <danieli> what was it? did you restart it?
2021-08-02 18:58:59 <ikke> restart sshd
2021-08-02 18:59:08 <danieli> i see, wonder what caused it to freeze
2021-08-02 18:59:08 <ikke> After some update, sshd needed a restart
2021-08-02 18:59:13 <danieli> ah right, thanks
2021-08-02 19:55:48 <clandmeter> Ariadne: liske is where?
2021-08-02 19:56:12 <ikke> #alpine-devel
2021-08-02 19:56:13 <Ariadne> in -devel usually
2021-08-02 19:56:14 <clandmeter> ah in devel
2021-08-02 19:56:18 <clandmeter> he is from bbb?
2021-08-02 19:56:29 <ikke> He gave a presentation on the conf
2021-08-02 19:56:31 <Ariadne> not from but admins a very large cluster
2021-08-02 19:56:52 <ikke> ifstate
2021-08-02 19:59:42 <clandmeter> ikke: so those certs are auto created as they are still pointing to our IP address
2021-08-02 19:59:55 <clandmeter> they are created on first hit
2021-08-02 20:00:18 <ikke> But how do they even find out those domains are still pointing to our IP?
2021-08-02 20:01:03 <clandmeter> somebody/.something access those domains
2021-08-02 20:01:55 <clandmeter> its another bug i think in this docker setup, it should only generate certs for the domain set.
2021-08-03 15:00:31 <nmeum> did you manage to fix the turnserver issues?
2021-08-03 16:17:07 <clandmeter> nmeum: i guess you are talking to me?
2021-08-03 18:41:05 <nmeum> yes, but it seems like it worked out in any case :)
2021-08-04 09:12:04 <clandmeter> nmeum: i added two fixes for bbb-docker https://github.com/bigbluebutton/docker/pulls
2021-08-04 09:12:41 <nmeum> nice
2021-08-04 09:13:48 <ikke> I still wonder how "generates certificates for any domain that points to the application" works
2021-08-04 09:15:39 <clandmeter> ikke: its automatic based on SNI
2021-08-04 09:17:01 <clandmeter> the certs are generated on the first http connection by a browser (or similar) sending a hostname.
2021-08-04 09:17:36 <ikke> So people are still hitting those other domains, causing the certificates to be generated?
2021-08-04 09:17:42 <clandmeter> it also had a gazzilion other certs based on ips, probably from bots who dont send a hostname
2021-08-04 09:18:02 <clandmeter> yes it looks like that
2021-08-04 09:18:28 <ikke> people / bots :)
2021-08-04 09:18:29 <clandmeter> and I guess thats kind of normal in VPS world where ipv4 addresses get recycled
2021-08-04 10:55:27 <ikke> clandmeter: pushed a new upgrade for gitlab 3.12, a security upgrade
2021-08-04 11:22:21 <clandmeter> Thanks
2021-08-04 11:36:36 <ikke> clandmeter: do you have time to modify algitbot to not anounce commits from branches starting with 'pathces/*'?
2021-08-04 13:30:29 <clandmeter> ikke: how can we reproduce it?
2021-08-04 13:48:36 <ikke> clandmeter: rebase a an MR that was created as a mailing list patch
2021-08-04 15:23:46 <clandmeter> Can you try?
2021-08-04 15:28:06 <ikke> Just rebased an MR
2021-08-04 15:28:15 <ikke> Seems to work
2021-08-04 16:07:26 <ikke> clandmeter: thanks :-)
2021-08-05 04:39:27 <ikke> 🤨
2021-08-05 04:56:58 <danieli> it was up for me all the times i checked, can it be a bad path between zabbix and the mirror?
2021-08-05 04:57:05 <danieli> or could be some timeout be too low?
2021-08-05 04:57:28 <ikke> hmm
2021-08-05 04:58:10 <ikke> Timeout is 5 seconds
2021-08-05 05:01:26 <ikke> Loads quite slow for me atm
2021-08-05 05:24:30 <ikke> https://imgur.com/a/XaeHhW4
2021-08-05 05:32:52 <danieli> for me, it takes 30ms to load /alpine/MIRRORS.txt in my browser
2021-08-05 05:32:56 <danieli> what are the traffic stats on the box like?
2021-08-05 05:33:15 <danieli> found it under /.stats, i initially forgot it was there
2021-08-05 05:33:46 <danieli> is the uplink saturated when the latency increases by any chance?
2021-08-05 05:46:51 <ikke> 2.4gbps max on 2x10gbps links
2021-08-05 06:20:54 <danieli> 2x10gbps is no worries then
2021-08-05 07:55:42 <clandmeter> ikke: did i miss the taskfile concept introduction?
2021-08-05 07:56:23 <clandmeter> is it only used in gitlab project?
2021-08-05 08:03:44 <ikke> Yeah, was experimenting with it.
2021-08-05 08:04:12 <ikke> The MR does mention why
2021-08-05 09:19:01 <ncopa> did anyone started to look at installing alpin eon the new ppc64le host?
2021-08-05 09:25:54 <ikke> I did not
2021-08-05 09:38:40 <clandmeter> i think we agree its not possible without oob
2021-08-05 09:38:57 <clandmeter> i can sent an email to ask them
2021-08-05 09:50:00 <danieli> was the tsc sent details about it?
2021-08-05 09:50:05 <danieli> i don't think i've even seen talk about it
2021-08-05 09:52:46 <clandmeter> ?
2021-08-05 10:29:25 <clandmeter> i added infra keys to the ppc box and deleted the login user they provided
2021-08-05 10:33:35 <ikke> danieli: we've been in contact with IBM about it
2021-08-05 10:33:57 <danieli> I see, makes sense
2021-08-05 10:34:01 <danieli> wait, they actually respond? :D
2021-08-05 10:34:19 <ikke> yes, especially since Ariadne managed to get their attention
2021-08-05 10:34:55 <danieli> poke them enough in the right places and i guess they'll turn over
2021-08-05 10:38:27 <Ariadne> i think it’s because i have a giant baseball bat
2021-08-05 10:40:03 <danieli> what, did you beat up a mainframe or something?
2021-08-05 12:20:56 <ncopa> ppc64le is bothering me again
2021-08-05 12:21:16 <ncopa> i cannot git clone to my ncopa-edge-ppc64le so i cannot test if new release in include kernel for ppc64le
2021-08-05 12:24:43 <ikke> Not sure if it matters, but can you try to set mtu to 1450 on your container?
2021-08-05 12:44:48 <danieli> sub-1500 is usually used as a last ditch effort, is their network really broken that badly?
2021-08-05 12:45:02 <danieli> makes sense to try it either way
2021-08-05 12:46:50 <ikke> Yeah, there is something block pmtu, not really clear what
2021-08-05 12:47:10 <ikke> But they did say that the mtu should be 1450 (even though we should not need to have to set it to 1450 manually)
2021-08-05 12:50:28 <danieli> PMTUD might be blocked because of restrictive defaults in firewalls... or that the software that sets MTU in the first place has some sort of flaw
2021-08-05 14:37:19 <clandmeter> ncopa:  you are having issues on the new host?
2021-08-05 14:37:37 <ncopa> no
2021-08-05 14:37:39 <ncopa> the old vm
2021-08-05 14:37:53 <ncopa> i think i have not set the MTU for the lxc containers
2021-08-05 14:38:06 <ikke> I did it for the builder containers
2021-08-05 14:38:39 <ncopa> i don tthink i did it for my ncopa-edge-ppc64le
2021-08-05 14:38:56 <ncopa> re the new ppc64le i think we should email roberto and ask how he did it last time
2021-08-05 14:39:10 <ncopa> how he installed alpine on the machine
2021-08-05 14:40:00 <ncopa> ught the rebuild of alpine-netboot-3.14.1-ppc64le.tar.gz: FAILED
2021-08-05 14:41:39 <danieli> it shouldn't be necessary to set settings like that in containers when traffic is masqueraded, should it?
2021-08-05 14:41:57 <ncopa> i think we need set up clampmss
2021-08-05 14:42:54 <danieli> how is that going to help if MTU is the issue?
2021-08-05 14:43:29 <danieli> MSS goes in the TCP headers
2021-08-05 14:43:50 <ncopa> the underlying issue is that PMTU does not work, likely due to the icmp need to frag packets gets blocked somewhere
2021-08-05 14:44:39 <danieli> simpler switches and routers usually only care about MTU and don't check MSS
2021-08-05 14:46:09 <danieli> i'd suggest sniffing some traffic and digging into the pcap but that's too vague to be useful advice :(
2021-08-05 14:50:07 <ikke> MSS is TCP only
2021-08-05 14:50:30 <ikke> setting  the MTU manually is just a work-around
2021-08-05 14:52:31 <danieli> MSS clamping is a workaround, properly adjusting MTU on the intermediary network gear is often a better solution
2021-08-05 14:53:24 <danieli> oh, and not blindly blocking all ICMP
2021-08-05 14:56:31 <ikke> It's using openstack or something like that
2021-08-05 14:56:49 <ikke> We already let them know we have issues, but they don't seem to have a proper solution
2021-08-05 14:58:30 <clandmeter> the problems are set to be fixed in 2-3 weeks
2021-08-05 15:00:30 <danieli> one of the easier short term solutions will probably be to be less restrictive on ICMP traffic to allow some of the messages used for PMTUD
2021-08-05 15:01:37 <danieli> MTU in general is a mess in my org's DC sites so i've gotten a bit too used to it :(
2021-08-05 15:10:42 <ikke> danieli: the issue is we (and I suppose they as well) don't know what is restricting it
2021-08-05 15:10:51 <danieli> ah, then it's a bit worse :D
2021-08-05 15:11:09 <ikke> yes
2021-08-05 15:11:30 <danieli> pcaps all the way
2021-08-05 15:11:49 <ikke> but like clandmeter said, we are moving away from this platform
2021-08-05 15:12:15 <clandmeter> i just send an email to roberto, lets see if he can help us.
2021-08-05 15:52:47 <minimal> ikke: can't you use ping with "-s" and gradually reduce the packet size until it stops working to figure out the value?
2021-08-05 15:53:37 <ikke> yes, I did multiple times
2021-08-05 15:54:03 <ikke> But it all depends on the destination
2021-08-05 15:54:38 <ikke> and I think, at least for now, with lowering the MTU on the interface, we did solve that issue
2021-08-05 15:55:06 <ikke> before we could not fetch https://httpbin.org/headers
2021-08-05 15:55:09 <ikke> after, we  could
2021-08-05 15:55:32 <minimal> hopefully then you won't have any packets with DF set that are larger than the now reduced MTU
2021-08-05 15:55:33 <ncopa> could it be that the return packet gets mtu black-holed?
2021-08-05 15:56:21 <minimal> like DNSSEC replies which can be quite large...
2021-08-05 15:57:14 <ncopa> have a nice evening. im done with computers for today
2021-08-05 15:57:28 <ikke> normally, at least with ipv4, it should automatically fragment packets that are too large
2021-08-05 15:57:38 <ikke> but apparently there is a hop that just drops it
2021-08-05 15:57:47 <minimal> not if the package have DF (Don't fragment) bit set
2021-08-05 15:57:49 <ikke> yes
2021-08-05 15:57:52 <minimal> s/package/packets/
2021-08-05 15:57:52 <alpine-meetbot> minimal meant to say: not if the packets have DF (Don't fragment) bit set
2021-08-05 15:59:18 <ikke> Which seems to be the case for all packets I have here in a tcpdump
2021-08-05 16:00:08 <minimal> as your example was a https util then (haven't checked) if the size of the reply containing the TLS certs chains is big and also if DF is set in that reply then the package would get dropped
2021-08-05 16:00:49 <minimal> s/util/url/
2021-08-05 16:00:49 <alpine-meetbot> minimal meant to say: as your example was a https url then (haven't checked) if the size of the reply containing the TLS certs chains is big and also if DF is set in that reply then the package would get dropped
2021-08-05 16:01:16 <ikke> minimal: yes
2021-08-05 16:01:36 <ikke> it was not the certificate
2021-08-05 16:03:20 <minimal> PDMTU issues unfortunately have been around for a very long time - years ago there used to be a website that listed "well known" site that didn't work due to their firewall misconfigurations
2021-08-05 16:10:52 <minimal> I'm assuming that "normal" SSH sends lots of small packets go give a good interactive response 'feeling' to users whereas rsync-over-ssh is using larger packets
2021-08-05 17:43:41 <ikke> minimal: yes
2021-08-08 09:14:29 <ikke> "uwsgi_proto_uwsgi_parser(): No error information [proto/uwsgi.c line 40]"
2021-08-09 02:04:41 <Thalheim> gitlab of course has dropped support for "older" browsers; have y'all received any complaints about this or is it just par for the course?
2021-08-09 02:06:23 <Thalheim> cgit satisfies repo access but not issue tracking, but I am curious how many people feel "left out" if any?
2021-08-09 06:00:31 <danieli> haven't heard anything about that
2021-08-09 06:05:16 <danieli> either way, at this point there's probably no other choice than leaving old (and barely used) browsers behind for dead
2021-08-09 06:56:07 <clandmeter> Thalheim: isn't there some cli tool to access issues?
2021-08-09 06:56:15 <clandmeter> or was that for github
2021-08-09 07:02:22 <clandmeter> https://glab.readthedocs.io/en/latest/
2021-08-09 08:31:03 <danieli> there is one for github but i'm not sure how good the one for gitlab is
2021-08-09 16:14:38 <Thalheim> clandmeter: thanks! that looks pretty nifty
2021-08-10 16:02:17 <lemontree> https://lists.alpinelinux.org/ is down
2021-08-10 16:02:43 <ikke> nOT FOR ME
2021-08-10 16:02:48 <ikke> sorry, caps
2021-08-10 16:03:39 <lemontree> hm, ok then its probably a local problem
2021-08-10 16:04:13 <lemontree> the thing is that alpinelinux.org is reachable here
2021-08-10 16:05:17 <ikke> What do you get?
2021-08-10 16:57:09 <lemontree> a 404 error in firefox, but in curl it would work. and now it is also reachable in firefox again
2021-08-10 16:57:29 <ikke> Hmm, strange, we get reports more often about this, wonder what is happening
2021-08-10 17:09:19 <lemontree> yes i also have these errors regulary for both docs.alpinelinux and lists.alpinelinux
2021-08-10 17:14:36 <danieli> looks like those are vhosts on the same IP so it might be related to issues with the configuration
2021-08-10 17:14:58 <ikke> hosted on nld3
2021-08-10 17:15:03 <ikke> but so is alpinelinux.org
2021-08-11 14:18:56 <rnalrd> sigh I command to wrong repo
2021-08-11 14:19:11 <rnalrd> ikke, I do I delete the branch created?
2021-08-11 14:21:34 <ikke> git push --delete <remote> <branchname>
2021-08-11 14:21:41 <rnalrd> git push origin --delete larena/linux-54139
2021-08-11 14:21:48 <rnalrd> error: unable to delete 'larena/linux-54139': remote ref does not exist
2021-08-11 14:21:49 <ikke> right
2021-08-11 14:22:04 <ikke> it;s just linux-54139
2021-08-11 14:22:07 <rnalrd> k
2021-08-11 14:22:19 <rnalrd> thanks
2021-08-11 14:22:57 <ikke> clandmeter: aha, ibm says they are going to install alpine on the ppc64le machine
2021-08-11 14:29:16 <mps> hehe, they read my complaints on other channel (not related to alpine) :)
2021-08-11 14:41:35 <clandmeter> Yes saw it 
2021-08-12 19:03:21 <ikke> clandmeter: upgraded the proxy container on nld3 (and updated the nginx config)
2021-08-12 19:50:17 <clandmeter> 👍
2021-08-12 19:50:42 <clandmeter>  Proxy is nginx?
2021-08-12 19:50:52 <ikke> yes
2021-08-12 19:51:26 <ikke> I still have a copy of the old files in case something did not work
2021-08-15 08:10:54 <lemontree> ikke: danieli hey i had again the issue with lists.alpinelinux.org being 404, but i fixed it by enabling  DNS over HTTPS in firefox
2021-08-15 08:11:17 <lemontree> so i am not sure how things work, but i am sure it has to do with dns but i don't understand things enough
2021-08-15 08:20:40 <lemontree> i recently had a similar issue with nixnet.xyz where i could not reach a service, this was also fixed by restarting the DNS thing, i think (don't know the tecorrect technical terms here, therefore i don't use any to avoid confusion)
2021-08-15 08:21:56 <ikke> lemontree: I've seen the issue as well. I tried to upgrade our nginx proxy to see if that solved it, but apparently not
2021-08-15 08:23:11 <lemontree> when did you update the nginx proxy?
2021-08-15 08:23:43 <lemontree> if it was in the last 1 hour or so, it might been the cause for me having a working lists.alpinelinux and not dns
2021-08-15 08:23:47 <ikke> Thursday
2021-08-15 08:23:55 <lemontree> ah ok
2021-08-16 10:45:41 <ikke> ppc64le CI network is acting up again 🤔
2021-08-18 14:47:56 <nmeum> would it be possible to get access to https://dev.alpinelinux.org/~nmeum/? would like to upload a pre-built image for the hifive unmatched
2021-08-18 14:49:40 <ncopa> i think you may have access already? ssh nmeum@dev.alpinelinux.org
2021-08-18 14:51:08 <nmeum> > nmeum@dev.alpinelinux.org: Permission denied (publickey).
2021-08-18 14:51:12 <nmeum> doesn't look like it
2021-08-18 14:51:35 <ncopa> and nmeum does not exis in /etc/passwd
2021-08-18 14:52:58 <nmeum> could that be changed? ;)
2021-08-18 14:53:24 <ncopa> nmeum: try now
2021-08-18 14:54:54 <nmeum> ncopa: permission denied still
2021-08-19 06:07:45 <ikke> Ariadne: mips builder seems to be awol again
2021-08-19 06:24:22 <Ariadne> kk
2021-08-19 08:43:28 <nmeum> would it possible to allow me to create public git repository under my namespace in the alpine gitlab? would allow me to host my build script for the hifive unmatched there
2021-08-19 08:44:34 <ikke> nmeum: we can make it public after you created it
2021-08-19 09:01:08 <nmeum> ikke: https://gitlab.alpinelinux.org/nmeum/alpine-unmatched thanks!
2021-08-19 09:01:45 <ikke> ✔️
2021-08-19 09:04:28 <nmeum> \o/
2021-08-19 09:04:30 <nmeum> thanks again
2021-08-19 09:04:40 <clandmeter> nmeum: you got a board?
2021-08-19 09:04:50 <nmeum> three actually!
2021-08-19 09:05:01 <clandmeter> lol
2021-08-19 09:05:21 <clandmeter> personal usage?
2021-08-19 09:06:07 <nmeum> technically they belong to my employeer but we need to boot a linux distro on them and alpine was the logical choice
2021-08-19 09:06:28 <ikke> nice
2021-08-19 09:06:36 <clandmeter> how does it perform?
2021-08-19 09:07:06 <clandmeter> we are trying to get a few from riscv, but seems they only have limited qty
2021-08-19 09:09:04 <nmeum> I haven't done any performance critical computations yet, still working on improving alpine support currently (adjusting the linux-edge kernel config and such to support all peripherals)
2021-08-19 09:23:58 <mps> nmeum: if you need it I have linux-rc branch in my local aport, last was 5.14-rc4
2021-08-19 09:30:58 <clandmeter> nmeum: we currently ship minirootfs for edge
2021-08-19 09:31:15 <clandmeter> not sure you were aware
2021-08-19 09:31:16 <nmeum> clandmeter: oh, I wasn't able to find one
2021-08-19 09:31:18 <nmeum> link?
2021-08-19 09:31:32 <clandmeter> https://dl-cdn.alpinelinux.org/alpine/edge/releases/riscv64/
2021-08-19 09:31:40 <nmeum> aahh
2021-08-19 09:31:49 <clandmeter> the usual suspect :)
2021-08-19 09:32:06 <nmeum> I wasn't aware that we create miniroot images for edge snapshots
2021-08-19 09:32:16 <nmeum> nice, that should allow me to simplify everything a bit
2021-08-19 09:32:24 <clandmeter> i guess so
2021-08-19 09:34:16 <clandmeter> we also use it for docker images
2021-08-19 09:38:01 <mps> clandmeter: ah, why didn't you told that two weeks ago. thanks anyway
2021-08-19 09:39:06 <clandmeter> well, we should have a marketing team who can add pretty news items on our www :)
2021-08-19 09:39:50 <nmeum> hehe
2021-08-19 09:40:16 <mps> well, that is my mistake actually because I know where are these files, but I didn't looked
2021-08-19 09:40:50 <clandmeter> of course its your mistake, i just hide it so nobody here knows.
2021-08-19 09:41:20 <mps> :D
2021-08-19 09:41:22 <clandmeter> its what friends do :p
2021-08-19 09:42:29 <mps> yesterday I told nmeum we don't have them. sorry nmeum 
2021-08-19 09:43:56 <nmeum> no problem
2021-08-19 10:06:01 <nmeum> the "problem" with the miniroot tarball is though that it comes without openrc (which makes sense if it is intended for chroots, containers, …)
2021-08-19 10:07:29 <nmeum> I think ideally we would need something like the "generic arm" image for riscv
2021-08-19 12:04:46 <ikke> clandmeter: https://gitlab.com/stavros/harbormaster
2021-08-20 11:51:48 <ikke> Cogitri: ^
2021-08-20 12:05:23 <Cogitri> Should work now (hopefully)
2021-08-20 12:06:44 <Cogitri> yay
2021-08-20 13:48:29 <ikke> Anyone heard / has experience with nginx unit?
2021-08-20 20:19:13 <danieli> ikke: some - what about it?
2021-08-20 20:20:03 <ikke> netbox switched from gunicorn to nginx unit
2021-08-20 20:20:12 <ikke> stating that it prevents them from having a separate nginx container
2021-08-20 20:20:26 <ikke> well, it avoids the need
2021-08-20 20:21:59 <danieli> I see, I would have expected them to move to something like ASGI
2021-08-20 20:26:27 <ikke> danieli: does ASGI also handle static content?
2021-08-20 20:26:58 <danieli> not on its own, but there are probably asgi app servers that can handle that on their own
2021-08-20 20:28:09 <danieli> once again that was poorly phrased, but you get the point :)
2021-08-20 20:28:13 <ikke> yup
2021-08-22 08:33:26 <ikke> netboot container on nld3 upgraded to 3.14.1
2021-08-24 13:02:50 <sircmpwn> how large is a full alpine mirror?
2021-08-24 13:02:54 <sircmpwn> s/full/complete/
2021-08-24 13:19:50 <ikke> 1.3T
2021-08-25 09:56:29 <ikke> uhoh ^
2021-08-25 09:56:35 <ikke> lost our s390x infra
2021-08-25 10:00:59 <danieli> spoke too soon ^
2021-08-25 10:05:42 <ikke> ahuh
2021-08-25 10:05:47 <ikke> luckily :)
2021-08-26 10:34:30 <ikke> does arch="noarch arch1 arch2" work? This is for an akms package, where the source itself is noarch, but the modules will only build on certain arches
2021-08-27 06:00:35 <ikke> clandmeter: tpaste is acting up again
2021-08-27 06:11:10 <clandmeter> Did you try to restart it?
2021-08-27 06:12:34 <ikke> I did 2 days ago
2021-08-27 06:47:10 <ikke> (for the record, I just did it again)
2021-08-27 07:05:30 <clandmeter> hmm
2021-08-27 07:06:06 <clandmeter> maybe we need to debug whats going on
2021-08-29 15:03:05 <nmeum> would it possible to give me temporary access to a riscv64 lxc? I want to test a fix for #12948 but don't have any potent buildserver (at least none where I can use qemu-binfmt) that is capabale of building gcc in a timely manner
2021-08-29 15:11:36 <ikke> We do not have riscv64 lxc containers yet
2021-08-29 15:20:31 <mps> nmeum: maybe this can help https://arvanta.net/alpine/install-alpine-riscv64-qemu/
2021-08-29 15:21:23 <mps> I forgot to add 'accel=tcg,thread=multi' there
2021-08-29 15:22:34 <mps> and I expect tomorrow kernel 5.14 which should be faster
2021-08-29 15:24:14 <mps> I've set lxc on i5 with 8GB ram. no it is not fast but can build kernel and other things on it. just have to be patient
2021-08-29 15:24:38 <mps> ssd disk is must
2021-08-29 17:11:29 <clandmeter> ikke: we do
2021-08-29 17:11:41 <ikke> oh
2021-08-29 17:11:58 <ikke> That's news to me :)
2021-08-29 17:12:21 <clandmeter> I think mps and ncopa have one
2021-08-29 17:12:31 <clandmeter> And myself
2021-08-29 17:12:39 <ikke> hosted, or local?
2021-08-29 17:13:09 <clandmeter> On our infra
2021-08-29 17:13:18 <ikke> ah ok
2021-08-29 17:13:38 <ikke> I see on nld5
2021-08-29 17:15:39 <clandmeter> Can you create one for nmeum ?
2021-08-29 17:15:55 <ikke> Was already ony it
2021-08-29 17:16:08 <clandmeter> :)
2021-08-29 17:16:10 <ikke> it does not get an IP yet for some reason
2021-08-29 17:17:02 <ikke> was that a known issue?
2021-08-29 17:17:16 <clandmeter> Yes
2021-08-29 17:17:21 <clandmeter> Set one static
2021-08-29 17:17:32 <ikke> ok
2021-08-29 17:17:33 <clandmeter> Think I did it in lxc config
2021-08-29 17:17:47 <clandmeter> Or else on the host
2021-08-29 17:18:20 <clandmeter> Also profile is broken
2021-08-29 17:18:31 <ikke> indeed through lxc config
2021-08-29 17:18:40 <clandmeter> Due to qemu user
2021-08-29 17:19:08 <clandmeter> nmeum: ^
2021-08-29 17:20:59 <ikke> nmeum: did you have vpn access?
2021-08-29 17:25:51 <ikke> if so, can you try to access 172.16.4.204
2021-08-29 18:52:30 <mps> clandmeter: yes, I have it but don't using it much, only for kernel, u-boot and opensbi building (for testing rv64 booting in qemu)
2021-08-29 18:52:43 <mps> rest I test in my local lxc
2021-08-29 19:00:20 <nmeum> ikke clandmeter: just returned to my computer, the build on my server actually just finished
2021-08-29 19:00:52 <nmeum> my gcc patch also seems to work so I guess I don't need access to the lxc anymore
2021-08-29 19:01:16 <nmeum> sorry for not reaching out sooner, but thanks anyhow for setting it up
2021-08-29 19:01:20 <nmeum> (I don't have vpn access though)
2021-08-29 19:11:00 <ikke> nmeum: try nld5-dev1.alpinelinux.org port 22204
2021-08-29 19:12:20 <nmeum> nmeum@? doesn't seem to like my pubkey but prompts for a password
2021-08-29 19:12:23 <ikke> root@
2021-08-29 19:12:32 <nmeum> ah yes that works
2021-08-29 19:12:47 <nmeum> sweet, I can use this next time
2021-08-29 19:12:48 <nmeum> thanks a bunch!
2021-08-29 19:12:50 <ikke> np
2021-08-29 19:15:04 <mps> when we are talking about riscv, yesterday someone told me that some (company?) preparing riscv farm with 'a lot' of boxes for linux distro developers
2021-08-29 19:15:24 <mps> on #linux-riscv
2021-08-29 19:15:47 <mps> are we interested in it
2021-08-29 19:30:17 <ikke> clandmeter: fyi, I've setup a new container for postgres called postgres-13. It replaces postgres (postgres 10), which afaik only zabbix actively used. I've stopped the old container
2021-08-29 19:30:31 <ikke> on nld3
2021-08-29 21:02:11 <clandmeter> Ok
2021-08-29 21:02:58 <clandmeter> mps: on this network?
2021-08-29 21:09:25 <mps> clandmeter: #linux-riscv is on libera
2021-08-29 21:10:43 <mps> '16:43 ........ palmer| have you seen the board farm that's getting set up?'
2021-08-29 21:11:03 <mps> that is from this channel
2021-08-29 21:11:53 <mps> '16:45 ........ palmer| ask Wei Wu <lazyparser@gmail.com>, IIRC he's involved in setting it up'
2021-08-29 21:12:25 <mps> '16:46 ........ palmer| they're planning on having a bunch of board remotely accessible for open source folks'
2021-08-30 07:05:31 <clandmeter> mps: how is that channel different than #riscv?
2021-08-30 07:06:26 <mps> clandmeter: very small number of people are there, which I didn't knew when joined
2021-08-30 07:07:38 <mps> I think I will leave this channel and stay only on #riscv
2021-08-30 07:08:24 <mps> clandmeter: do you have real rv64 box
2021-08-30 07:09:35 <clandmeter> nope
2021-08-30 07:09:47 <mps> aha
2021-08-30 07:11:26 <mps> I thought to ask for one box on this 'farm' for me personally but maybe I should tell them that I'm alpine dev and they will give me better one in this case :)
2021-08-30 07:12:36 <mps> but would like if we will request something officially, to not appear as we dezorganized 'bunch' of devs
2021-08-30 07:12:59 <mps> would like to know*
2021-08-30 07:13:11 <clandmeter> we are still looking for hw
2021-08-30 07:13:21 <clandmeter> so whatever you come up with is good :)
2021-08-30 07:13:38 <clandmeter> i think rv64 builder is still acting up?
2021-08-30 07:14:08 <mps> I don't have access to it so idk
2021-08-30 07:14:22 <clandmeter> last time  you asked right?
2021-08-30 07:14:34 <mps> what?
2021-08-30 07:19:08 <clandmeter> <mps> so, upload of riscv64 doesn't works?
2021-08-30 07:33:51 <mps> builder shows it is idle but packages aren't uploaded to mirrors
2021-08-30 08:27:47 <clandmeter> im looking at the rv64 builder
2021-08-30 08:27:54 <clandmeter> not sure whats going on
2021-08-30 08:28:30 <clandmeter> looks like mqtt is not exec aports-build
2021-08-30 08:29:29 <clandmeter> ikke: did you do any changes t o usa5?
2021-08-30 08:31:50 <ikke> clandmeter: no
2021-08-30 08:32:08 <clandmeter> so it magically stopped working :)
2021-08-30 08:32:54 <clandmeter> ikke: how did you restart it?
2021-08-30 08:33:08 <clandmeter> docker-compose restart abuild?
2021-08-30 08:35:38 <ikke> docker-compose restart
2021-08-30 08:40:58 <clandmeter> nmeum: how did you last time debug qemu-user?
2021-08-30 08:41:03 <clandmeter> you did it on the host right?
2021-08-30 09:17:40 <nmeum> clandmeter: yes with the qemu-user gdb stub, see the -g flag
2021-08-30 09:17:59 <nmeum> basically you just start the thing manually with qemu-user pass the g flag and then connect to the gdb stub with gdb-multiarch
2021-08-30 09:22:26 <clandmeter> nmeum: ok, but what if qemu-user is not available on the system?
2021-08-30 09:23:19 <clandmeter> normally qemu-user is not available on the guest
2021-08-30 09:26:19 <nmeum> I mostly use qemu-user with chroots, for debugging purposes I started it from outside the chroot
2021-08-30 09:26:23 <nmeum> not sure if there is a better way
2021-08-30 09:56:58 <clandmeter> nmeum: im confused, you run gdb-mutliarch from the chroot?
2021-08-30 10:51:32 <nmeum> clandmeter: no, I run qemu-user and gdb-multiarch from outside the chroot e.g. qemu-riscv64 -g 1234 /opt/alpine-chroot-edge-riscv64/usr/bin/foobar and then outside the chroot gdb-multiarch, target remote :1234, set sysroot /opt/alpine-chroot-edge-riscv64, et cetera
2021-08-30 21:57:54 <backbox> test
2021-08-31 07:27:49 <ikke> We need to clean up usa2 a bit
2021-08-31 07:39:57 <nmeum> is the riscv64 builder really not gone? it hasn't uploaded in #alpine-commits for a while
2021-08-31 07:40:05 <nmeum> for example, it has also not build busybox 1.34.0 yet
2021-08-31 07:40:24 <nmeum> and I can't see the current build status on build.alpinelinux.org 🤔
2021-08-31 07:40:35 <ikke> It's not gone, just not fully functional for some reason
2021-08-31 07:41:08 <ikke> clandmeter was looking at it, but I suppose he did not find the cause yet
2021-08-31 07:42:45 <clandmeter> yes i looked at it brielfly
2021-08-31 07:42:50 <clandmeter> been bit busy
2021-08-31 07:42:59 <clandmeter> it looks like it builds
2021-08-31 07:43:07 <clandmeter> it juist does not trigger for some reason
2021-08-31 07:43:26 <clandmeter> so we need to check whats going on with mqtt-exec
2021-08-31 09:30:47 <ncopa> im cleaning up a bit on the arm  machine
2021-08-31 09:31:24 <ncopa> oh.. usa2-dev1 is not the arm
2021-08-31 09:31:27 <ikke> ahuh
2021-08-31 09:31:29 <ikke> it's s390x
2021-08-31 09:37:27 <ncopa> cleaned up distfiles a bit
2021-08-31 09:39:00 <ikke> thanks!
2021-08-31 16:42:40 <ikke> Now it's arm/aarch :P
2021-08-31 19:19:52 <clandmeter> re rv64, seems aports-build pid is stale and so it does not build again
2021-08-31 19:27:00 <Thalheim> > Today we are releasing versions 14.2.2, 14.1.4, and 14.0.9 for GitLab Community Edition (CE) and Enterprise Edition (EE). These versions contain important security fixes and we strongly recommend that all GitLab installations be upgraded to one of these versions immediately.
2021-08-31 19:41:00 <ikke> We are not on 14.x yet
2021-08-31 20:06:40 <ncopa> ppc64le is killing me
2021-08-31 20:07:20 <ncopa> if its not the build of the release images that breaks, then its the upload of it
2021-08-31 20:07:36 <ncopa> seems impossible to get ppc64le releases out the door
2021-08-31 20:08:00 <ikke> Network issues, or not?'
2021-08-31 20:09:14 <ncopa> well, it was a stupid bug in the release script that make things fail if the abuild signing pub key is not in ~/.abuild
2021-08-31 20:09:25 <ncopa> and i tagged and pushed before fixing that
2021-08-31 20:09:33 <ncopa> so i fixed it and rebuilt the images
2021-08-31 20:09:42 <ncopa> all fine i though
2021-08-31 20:10:05 <ncopa> later it turns out that the latest-releases.yaml was not updated
2021-08-31 20:10:12 <ncopa> so the docker image build failed
2021-08-31 20:10:47 <ncopa> so i deleted all the release/ppc64le/*3.12.8*
2021-08-31 20:11:00 <ncopa> and are now recreating te release for the third time
2021-08-31 20:11:14 <ncopa> the build of the release seems to be ok
2021-08-31 20:11:25 <ncopa> but the upload hangs due to network issues
2021-08-31 20:12:10 <ncopa> i guess the problem is that the release scripts are buggy and does not catch unexpected things to happen
2021-08-31 20:12:34 <ncopa> because for the last 10 years unexpected things has happened rarely
2021-08-31 20:12:48 <ncopa> with ppc64le the game has changed
2021-08-31 20:16:00 <ncopa> now im trying to manually fix up the things
2021-08-31 20:16:04 <ncopa> and im tired
2021-08-31 20:16:08 <ncopa> so i will make mistakes
2021-08-31 20:16:22 <ncopa> i think the last release someone complained bokren signatures or similar
2021-08-31 20:16:30 <ikke> right
2021-08-31 20:16:34 <ncopa> or borken checksums
2021-08-31 20:16:49 <ikke> We should form a release engineering SIG
2021-08-31 20:17:03 <ikke> You talked about it before
2021-08-31 20:17:29 <ncopa> yah, it happens when the automatted process fails due to the "unexpected" and manual process takes over and manual worker is tired and wants throw computer out the window and move to the forest
2021-08-31 20:18:07 <ikke> yes, so one thing the release engineering team could do is work on more robust tooling
2021-08-31 20:18:37 <ncopa> interesting thing with the network issue is that rsync seems to work if i add --bwlimit=1000k to rsync
2021-08-31 20:18:50 <ikke> interesting
2021-08-31 20:18:58 <ikke> Is it rsync that is hanging midway?
2021-08-31 20:19:03 <ncopa> yup
2021-08-31 20:19:17 <ncopa> not sure if its midway or when it happens
2021-08-31 20:19:22 <ncopa> but its definitively rsync that hangs
2021-08-31 20:19:38 <ikke> That's exactly what I'v encountered with syncing to dl-master
2021-08-31 20:19:44 <ncopa> yes
2021-08-31 20:19:51 <ikke> but I could not reproduce it when syncing to gbr1
2021-08-31 20:20:33 <ncopa> i was able to save tcpdump on both ends
2021-08-31 20:20:52 <ikke> Anything interesting?
2021-08-31 20:21:04 <ncopa> could not figure anything out
2021-08-31 20:21:16 <ikke> It's kind of tricky to follow with so much data
2021-08-31 20:21:41 <ncopa> and i was kinda busy and wanted the 3.14.2 release out so i didnt spend too much time on it
2021-08-31 20:21:48 <ikke> understood
2021-08-31 20:22:43 <ncopa> i suspect that setting up clam-mss in iptables on the ubuntu host might work around it
2021-08-31 20:22:50 <ncopa> but i nver figured out how to do so
2021-08-31 20:22:55 <ncopa> on ubuntu