2021-12-01 05:29:33 <ikke> clandmeter: yes, but I don't think we have to build for 3 arches + 3*CI on ppc64le :P
2021-12-01 15:58:49 <clandmeter> im looking at git.a.o
2021-12-01 15:59:00 <clandmeter> uwsgi_proto_uwsgi_parser(): No error information [proto/uwsgi.c line 40]
2021-12-01 16:01:57 <clandmeter> 	if (wsgi_req->proto_parser_pos > 0) {
2021-12-01 16:01:57 <clandmeter> 	}
2021-12-01 16:01:57 <clandmeter> 		uwsgi_error("uwsgi_proto_uwsgi_parser()");
2021-12-01 16:11:04 <clandmeter> ikke: that code is related to async, so for now i disabled that for uwsgi
2021-12-01 16:11:36 <ikke> clandmeter: ok, let's see if it helps
2021-12-01 16:11:54 <clandmeter> there is also a note about fds
2021-12-01 16:12:08 <clandmeter> and i think i added it
2021-12-01 16:12:13 <clandmeter> but no comment added
2021-12-01 16:12:22 <clandmeter> well no useful one :)
2021-12-01 16:12:41 <clandmeter> temporary set max-fd until upsteam is fixed
2021-12-01 16:14:47 <ikke> upsteam :)
2021-12-01 16:16:53 <Hello71> choo choo?
2021-12-01 16:23:06 <ikke> clandmeter: only 3.14 and 3.15 builders still need to be moved (and 2 dev containers) for ppc64le
2021-12-01 21:59:30 <ikke> moved all builders now to the new host
2021-12-01 22:00:06 <ikke> ncopa: ^
2021-12-02 08:36:36 <ncopa> good morning! great! thanks!
2021-12-02 08:43:31 <ikke> fyi, developer containers (yours and mine) still need to be moved
2021-12-02 08:45:10 <ncopa> i think he said we could keep the vms if we wanted. I dont know if we want?
2021-12-02 09:29:32 <ikke> ncopa: oh, before I forget, do you have a backup of the new signing keys?
2021-12-02 09:31:08 <ikke> ncopa: The e-mail said we would have to return the VMs
2021-12-02 09:54:12 <clandmeter> ikke: maybe we can keep one for CI?
2021-12-02 09:58:23 <ikke> clandmeter: We have a dedicated CI machine
2021-12-02 09:58:54 <ikke> ppc64le-ci.alpinelinux.org
2021-12-02 09:59:15 <clandmeter> oh right, forgot.
2021-12-02 10:00:03 <clandmeter> we talked about a visual infra overview long time ago, i think it was with danieli. 
2021-12-02 10:00:18 <clandmeter> i could really use that to train my brain 
2021-12-02 10:17:25 <ikke> I think netbox would be a good source for that
2021-12-02 10:18:41 <clandmeter> right, but a little bit of extra meta would be nice.
2021-12-02 10:19:23 <ikke> like what?
2021-12-02 10:20:45 <clandmeter> i dont have a clear list, just so its obvious what which box does, what is the general use case or mixed if thats the case.
2021-12-02 10:30:52 <clandmeter> reminds me about visual things, I'm playing with python for the first time and i cannot get used to not closing functions... my brain is slow in visual adjustment :)
2021-12-02 10:31:32 <ikke> So you haven't worked with yaml enough before? :P
2021-12-02 10:33:56 <clandmeter> why do you think i stay away from our CI? :p
2021-12-02 10:35:41 <ikke> :D
2021-12-02 15:01:11 <danieli> clandmeter: oh yeah, i remember that, at one point i made a thing to visualize VM hosts and the VMs on them
2021-12-03 09:30:45 <clandmeter> ikke: ping
2021-12-03 09:33:35 <ikke> pong
2021-12-03 09:34:26 <clandmeter> gitlab disk almost full?
2021-12-03 09:34:32 <ikke> yes, 14G left
2021-12-03 09:34:50 <ikke> Was looking at what was taking up the space
2021-12-03 09:34:59 <clandmeter> let me guess
2021-12-03 09:35:04 <clandmeter> it starts with a A
2021-12-03 09:36:23 <clandmeter> we dont set docker logs limits on that host?
2021-12-03 09:43:39 <ikke> 145G    shared/artifacts
2021-12-03 09:43:55 <clandmeter> 51G logs
2021-12-03 09:44:16 <ikke> right, but those are not docker logs, correct?
2021-12-03 09:44:23 <clandmeter> nope gitlab
2021-12-03 09:44:24 <ikke> just log files we write to disk
2021-12-03 09:44:58 <clandmeter> but try to tail docker compose log for traefik :)
2021-12-03 09:53:01 <ikke> many traffic
2021-12-03 11:32:01 <ikke> production_json.log and production.log are 40G together
2021-12-03 11:41:58 <ikke> clandmeter: what should run logrotate in gitlab?
2021-12-03 11:56:45 <ikke> running entrypoint.sh logrotate
2021-12-04 09:53:32 <nmeum> is there an existing issue regarding CI for riscv64 in the gitlab somewhere?
2021-12-04 09:57:09 <ikke> I thought there was, but apparently there is not
2021-12-04 09:58:19 <nmeum> I will create one then (:
2021-12-04 09:58:48 <nmeum> currently trying to collect issues which should be addressed for official riscv64 support in 3.16
2021-12-04 10:00:58 <nmeum> #10738
2021-12-04 10:01:03 <nmeum> https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10738
2021-12-04 13:09:24 <clandmeter> nmeum: gcc-gnat got uninstalled somehow
2021-12-04 13:09:33 <clandmeter> i think we needto boostrap again to get it
2021-12-04 13:11:20 <nmeum> clandmeter: huh, why did it get uninstalled?
2021-12-04 13:11:25 <clandmeter> i think its related to docker
2021-12-04 13:11:35 <clandmeter> when we restart the container it gets removed
2021-12-04 13:11:41 <peer> wierd
2021-12-04 13:11:52 <clandmeter> then on next run of gcc build its build without it
2021-12-04 13:30:05 <ikke> Anything preventing us from removing edge/mips64?
2021-12-04 17:12:06 <ikke> clandmeter: ping
2021-12-04 18:23:54 <ikke> clandmeter: Any idea why bundle exec rake gettext:compile RAILS_ENV=production
2021-12-04 18:24:30 <ikke> would also do things like generating the .gitlab_shell_secret? 
2021-12-04 18:42:46 <ikke> It feels to me similar to make, where when running make install, it would also build the project
2021-12-04 19:52:25 <clandmeter> Not sure
2021-12-04 19:52:43 <clandmeter> Why are you asking?
2021-12-04 19:55:19 <ikke> I was fixing another issue (warning) and it failed running that command when it tried to write to the .gitlab_shell_secret file
2021-12-04 19:56:09 <ikke> Just found it strange that it would give that error when running a gettext:compile task
2021-12-04 20:09:28 <ikke> clandmeter: fyi, I've pushed gitlab 14.4
2021-12-04 20:30:47 <clandmeter> looks like im having trouble importing edge into apkbrowser
2021-12-05 11:42:10 <ikke> clandmeter: I've cleaned up buildlogs older then one month on the biggest aports projects, currently 120G free again
2021-12-05 12:10:23 <ikke> https://tpaste.us/DXQd?hl=true
2021-12-05 17:21:14 <clandmeter> Nice
2021-12-05 17:25:50 <ikke> There's still 80G in shared files
2021-12-05 17:26:05 <ikke> I think we need to run this regularly to keep it in check
2021-12-06 11:48:24 <clandmeter> ikke: busy?
2021-12-06 11:48:44 <ikke> Have about 10 minutes
2021-12-06 11:49:17 <clandmeter> correction
2021-12-06 11:49:20 <clandmeter> seems you are busy
2021-12-06 11:49:30 <ikke> heh
2021-12-06 11:49:32 <clandmeter> im getting emials
2021-12-06 11:49:34 <clandmeter> emails even
2021-12-06 11:49:43 <ikke> Yes, restoring a new copy of gitlab-test
2021-12-06 11:50:15 <clandmeter> 👍
2021-12-06 11:50:29 <ikke> "Account Limit reached. Please open a support ticket." 😐
2021-12-06 11:50:41 <clandmeter> wut?
2021-12-06 11:50:56 <clandmeter> limit of what?
2021-12-06 11:52:10 <ikke> I don't know, happens when I want to restore a backup
2021-12-06 11:52:41 <ikke> $77 dollar acrued charges
2021-12-06 19:23:25 <clandmeter> ikke: found the issue?
2021-12-06 19:23:43 <ikke> Regarding linode? 
2021-12-06 19:41:20 <clandmeter> yup
2021-12-06 19:48:05 <ikke> No
2021-12-06 19:48:29 <ikke> I suppose we need to contact them again 
2021-12-06 19:49:19 <clandmeter> blaboon: hi, is there an issue with our account?
2021-12-06 20:55:07 <blaboon> clandmeter: taking a look...
2021-12-06 21:02:57 <blaboon> clandmeter ikke: should be fixed up now. i forgot to clean up one of the account values when looking at the billing issue from a month or so ago
2021-12-06 21:03:17 <clandmeter> thanks!
2021-12-06 21:03:24 <ikke> 👍
2021-12-06 21:03:52 <ikke> It's working now, thanks!
2021-12-06 21:04:02 <MY-R> 
ACTION :D

2021-12-06 21:04:36 <MY-R> blaboon: please add it to cron job ;)
2021-12-06 21:07:04 <blaboon> hehehe
2021-12-06 21:08:52 <MY-R> :)
2021-12-07 19:10:33 <ikke> clandmeter: have gitlab 14.4 running on gitlab-test
2021-12-07 19:35:18 <clandmeter> nice
2021-12-08 11:34:00 <ncopa> http://distfiles.alpinelinux.org/distfiles/v3.12/ncurses-6.2-20200523.tgz
2021-12-08 11:34:15 <ncopa> I have a problem. it gives a 404. I cannot fix the CVE
2021-12-08 11:34:25 <ncopa> do we have that archice somewhere else?
2021-12-08 11:34:37 <ikke> yes
2021-12-08 11:36:07 <ikke>  buildozer@172.105.82.32:/srv/distfiles/
2021-12-08 11:36:55 <ncopa> is it available from some public http? can we copy it to http://distfiles.alpinelinux.org/distfiles/v3.12/ncurses-6.2-20200523.tgz ?
2021-12-08 11:37:00 <ikke> Yes, will do that
2021-12-08 11:37:06 <ncopa> thank you!
2021-12-08 11:39:04 <ikke> ncopa: it's there now
2021-12-08 11:46:21 <ncopa> thank you very much!
2021-12-09 18:25:28 <clandmeter> ikke: heh that e-mail
2021-12-09 18:25:34 <ikke> :)
2021-12-09 18:26:36 <ikke> clandmeter: planning to upgrade gitlab
2021-12-09 18:26:52 <clandmeter> Oops I just found one under my pillow :)
2021-12-09 18:27:06 <ikke> :D
2021-12-09 18:27:07 <clandmeter> Ok nice
2021-12-09 18:27:40 <ikke> heh, conflicting responses :D
2021-12-09 18:35:55 <Ariadne> haha oops
2021-12-09 18:36:06 <Ariadne> i did not know our CI was at OSUOSL, i thought it was at unicamp :)
2021-12-09 18:56:55 <ikke> clandmeter: not sure where it's comming from, but when I commit in gitlab-test via the web-interface, it tries to retrieve pipelines in the background, which returns a 301, where the location is http://, not https://, which gets blocked
2021-12-09 18:57:11 <ikke> Don't get that in our production environment
2021-12-10 11:55:10 <ikke> OOM killed :(
2021-12-10 14:24:27 <clandmeter> ikke: did you find the issue?
2021-12-10 14:24:38 <clandmeter> reg gitlab 
2021-12-10 14:25:22 <clandmeter> Sorry I went to bed early yesterday… bit tired 
2021-12-10 14:29:26 <ikke> No, I did not
2021-12-10 14:29:41 <ikke> the configuration says gitlab should use https
2021-12-10 14:31:48 <ikke> Smells like a bug in GL somewhere 
2021-12-10 14:37:12 <ikke> Do you have time to test gitlab-test a bit? 
2021-12-10 14:37:32 <ikke> This so far does not look like a blocking issue 
2021-12-10 14:38:04 <ikke> It just shows an error banner, but did not prevent committing. 
2021-12-10 16:57:45 <clandmeter> ikke: it seems i cannot login
2021-12-10 16:58:16 <clandmeter> 422
2021-12-10 16:58:16 <clandmeter> The change you requested was rejected.
2021-12-10 17:03:25 <ikke> Hmm, I think I could login
2021-12-10 17:32:41 <Ariadne> i just received a spam mail from pkgs.a.o
2021-12-10 17:32:53 <Ariadne> https://usercontent.irccloud-cdn.com/file/ZnDALHyR/Screen%20Shot%202021-12-10%20at%2011.32.22%20AM.png
2021-12-10 17:53:20 <clandmeter> Spam?
2021-12-10 17:54:07 <clandmeter> Ah the message is bs
2021-12-10 18:00:15 <clandmeter> ikke: i cannot login, not even incognito
2021-12-10 18:04:06 <ikke> hmm
2021-12-10 18:05:04 <ikke> right, I get the same now
2021-12-10 18:08:23 <clandmeter> i think ive seen this before
2021-12-10 18:08:26 <clandmeter> in the beginning
2021-12-10 18:08:39 <clandmeter> something with rev proxy and some config option in nginx
2021-12-10 18:08:46 <clandmeter> not sure if its related
2021-12-10 18:08:48 <ikke> https://tpaste.us/ypde 
2021-12-10 18:09:49 <ikke> ActionController::InvalidAuthenticityToken
2021-12-10 20:58:35 <ikke> clandmeter: seems like a session id is missing on gitlab-test
2021-12-10 20:59:22 <ikke> _gitlab_session
2021-12-11 08:52:44 <clandmeter> ikke: upgrade time again ;-)
2021-12-11 08:52:48 <ikke> yes
2021-12-13 09:10:04 <clandmeter> ikke: i guess we are safe from infra perspective reg Log4j? i don't remember we are using any java or related applications?
2021-12-13 09:10:31 <ikke> Not that I'm aware of
2021-12-13 12:18:46 <ikke> clandmeter: bigbluebutton? 
2021-12-13 12:31:01 <danieli> ikke: https://github.com/bigbluebutton/bigbluebutton/issues/13897
2021-12-13 12:31:43 <danieli> looks like there's only 1.x
2021-12-13 13:05:13 <ikke> danieli: thanks 
2021-12-13 14:30:03 <clandmeter> ikke: i got some free time atm, are there any blockers for setting distfiles to the new location?
2021-12-13 14:34:43 <ikke> What do we do for buildlogs? 
2021-12-13 14:36:12 <clandmeter> send it to distfiles
2021-12-13 14:36:25 <clandmeter> im adding build.a.o to it
2021-12-13 14:36:32 <ikke> Ok
2021-12-13 14:53:26 <ikke> I don't see any blockers then 
2021-12-13 16:07:41 <ikke> clandmeter: did you change any DNS yet?
2021-12-13 16:17:07 <clandmeter> nope
2021-12-13 16:17:12 <clandmeter> im still busy with it
2021-12-13 16:17:58 <ikke> np, created an MR for the DNS change
2021-12-13 16:18:00 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/14/
2021-12-13 16:25:40 <clandmeter> ok good
2021-12-13 16:25:42 <clandmeter> lets push that
2021-12-13 19:11:30 <clandmeter> worden riscv64 build logs wel geupload uberhaubt 
2021-12-13 19:11:53 <ikke> this is #alpine-infr ;-)
2021-12-13 19:12:05 <ikke> clandmeter: they have always been uploaded
2021-12-13 19:12:31 <clandmeter> dit kanaal is nu officieel nederlands
2021-12-13 19:14:32 <mps> nearly, though I understand last two sentences in dutch ;)
2021-12-13 19:20:12 <clandmeter> ikke: i wonder how it did that
2021-12-13 19:20:29 <clandmeter> as the container does not have access to dmvpn (i think)
2021-12-13 19:20:34 <ikke> I suppose you sit it up
2021-12-13 19:21:04 <clandmeter> sit it up?
2021-12-13 19:21:07 <clandmeter> :)
2021-12-13 19:21:19 <ikke> yet, sit it up :)
2021-12-13 19:21:31 <clandmeter> my *ss hurts now
2021-12-13 19:21:46 <clandmeter> but i dont see the network in the container
2021-12-13 19:22:00 <clandmeter> and if i click a build log on pkgs i get 404
2021-12-13 19:24:14 <ikke> This url used to work: https://build.alpinelinux.org/buildlogs/build-edge-riscv64/testing/eolie/eolie-0.9.101-r1.log
2021-12-13 19:25:01 <ikke> clandmeter: did you sync logs from old distfiles to new distfiles?
2021-12-13 19:25:10 <clandmeter> no
2021-12-13 19:25:18 <ikke> I guess we need to do that then
2021-12-13 19:25:20 <clandmeter> i thoguht you would do that :)
2021-12-13 19:29:07 <ikke> clandmeter: I guess the last sync was tonight, and that log was from this morning
2021-12-13 19:29:59 <ikke> although /var/cache/distfiles/buildlogs contains older files
2021-12-13 19:31:42 <clandmeter> ikke: was there no other way to reach distfiles? some port foward?
2021-12-13 19:32:41 <ikke> port 22006
2021-12-13 19:33:31 <clandmeter> logtarget = "distfiles.alpinelinux.org:/var/cache/distfiles/buildlogs"
2021-12-13 19:34:19 <ikke> nothing in .ssh/config?
2021-12-13 19:35:05 <clandmeter> i think i was on the wrong host
2021-12-13 19:35:06 <clandmeter> :)
2021-12-13 19:35:16 <ikke> heh
2021-12-13 19:35:25 <ikke> deu5 instead of usa5?
2021-12-13 19:35:40 <clandmeter> i have a ssh config for riscv64
2021-12-13 19:35:44 <clandmeter> but thats the dev box
2021-12-13 19:36:23 <ikke> ah ok
2021-12-13 19:36:28 <clandmeter> the container can ssh just fine 
2021-12-13 19:36:34 <clandmeter> so it should sync the logs
2021-12-13 19:36:58 <clandmeter> i have no idea how it worked before
2021-12-13 19:37:10 <clandmeter> ah yes
2021-12-13 19:37:15 <clandmeter> it does have a config :D
2021-12-13 19:37:43 <clandmeter> done
2021-12-13 19:37:47 <clandmeter> should be fixed
2021-12-13 19:37:50 <ikke> ok
2021-12-13 22:22:08 <clandmeter> ikke: did you check the other builders?
2021-12-13 22:22:41 <ikke> sorry, no
2021-12-13 22:27:17 <clandmeter> done s390x
2021-12-13 22:31:22 <clandmeter> ikke: did you update the A record for ppc64le?
2021-12-13 22:31:29 <clandmeter> i mean is that the builder now?
2021-12-13 22:31:41 <ikke> The ip never changed
2021-12-13 22:31:55 <ikke> and the A record was never changed either
2021-12-13 22:31:55 <clandmeter> but you moved them
2021-12-13 22:32:00 <ikke> no, created new ones
2021-12-13 22:32:06 <ikke> gbr3-vm[123]
2021-12-13 22:32:09 <clandmeter> to vm's?
2021-12-13 22:32:23 <clandmeter> ok so ppc64le.a.o is the builder now?
2021-12-13 22:32:30 <ikke> yes
2021-12-13 22:34:38 <clandmeter> then i guess it should be done now
2021-12-14 16:25:05 <clandmeter> ikke: looks like builders are not uploading
2021-12-14 16:25:13 <clandmeter> logs
2021-12-14 16:25:59 <ikke> yes
2021-12-14 16:26:04 <clandmeter> oh
2021-12-14 16:26:08 <clandmeter> i think its the path
2021-12-14 16:27:56 <clandmeter> how was it handled on the original host?
2021-12-14 16:28:09 <clandmeter> a symlink or managed by a nginx rule?
2021-12-14 16:31:35 <clandmeter> i made a symlink in the build.a.o dir to buildlogs
2021-12-14 16:31:47 <clandmeter> so that kind of works, but still the logs are missing
2021-12-14 16:33:08 <clandmeter> i remember we had issues before uploading logs due to scp and ssh firewall limits
2021-12-14 16:33:35 <ikke> Don't recall
2021-12-14 16:34:04 <clandmeter> i think it wanted to make x connections and after x it stopped due to fw limits
2021-12-14 16:34:41 <clandmeter> but this is probably something else
2021-12-14 16:39:57 <ikke> Oh, ssh connection limitting
2021-12-14 16:43:08 <clandmeter> looks like the path to distfiles is different
2021-12-14 16:43:23 <clandmeter> i added a symlink, need to see if it iworks now
2021-12-14 16:46:16 <clandmeter> ikke: can you check x86 builder?
2021-12-14 16:46:22 <clandmeter> in the buildlog folder
2021-12-14 16:46:50 <ikke> What should I check for?
2021-12-14 16:46:55 <clandmeter>  /var/cache/distfiles/buildlogs
2021-12-14 16:47:00 <clandmeter> check the latest file
2021-12-14 16:47:06 <clandmeter> it should metnion an error at the end
2021-12-14 16:47:22 <clandmeter> food
2021-12-14 16:47:28 <ikke> build-edge-x86.v3.15.0-853-g2b71913c23.log
2021-12-14 16:47:40 <ikke> Host key verification failed
2021-12-14 16:48:58 <ikke> Seems like the hostkey was not properly added to known_hosts
2021-12-14 16:50:20 <ikke> oh, you only added it by IP
2021-12-14 16:53:02 <ikke> The key seems different
2021-12-14 17:12:46 <ikke> fixed the known_hosts on all x86* builders
2021-12-14 17:18:12 <ikke> It's uploading now, but with the wrong content type
2021-12-14 17:18:45 <ikke> Or I mean, the upload is going alright, but serves them with the incorrect content-type
2021-12-14 17:19:54 <ikke> fixed
2021-12-14 17:35:53 <clandmeter> so what is the issue with the key?
2021-12-14 17:36:17 <clandmeter> i can ssh just fine from riscv64
2021-12-14 17:38:48 <clandmeter> ikke: i think we need to use local_addrs for rspamd
2021-12-14 17:48:18 <ikke> I have no experience with rspamd
2021-12-14 18:21:39 <ikke> clandmeter: I don't know what the issue is, but it still asks to verify the host, and when I choose yes, it adds a different key (id-ed25519).
2021-12-14 18:41:04 <clandmeter> maybe it depends on the keys on the client? i have no idea.
2021-12-14 18:43:34 <clandmeter> regarding rspamd, we currently bypass rspamd for our infra servers to relay.
2021-12-14 18:45:09 <clandmeter> but rspamd also sets dkim, so this will be skipped for our infra servers.
2021-12-14 18:51:44 <clandmeter> ikke: im trying ssh-keyscan , but it fails due ssh connection limiting.
2021-12-14 19:29:25 <ikke> clandmeter: so we need to adjust the ratelimitting?
2021-12-14 19:29:45 <clandmeter> maybe add all the keys
2021-12-14 19:30:02 <clandmeter> https://tpaste.us/Lm7W
2021-12-14 19:31:26 <rails> could use dns SSHFP as an alternative
2021-12-14 19:34:15 <clandmeter> is that widely supported? 
2021-12-14 19:36:20 <rails> openssh has it, but not by default =\
2021-12-14 19:36:47 <clandmeter> we have our dns in linode
2021-12-14 19:36:53 <clandmeter> not sure they support it
2021-12-14 19:37:01 <rails> i.. dont know if linode dns can do dnssec
2021-12-14 19:37:47 <rails> nop
2021-12-14 19:38:33 <rails> https://www.linode.com/docs/guides/dns-manager/#dnssec-limitations
2021-12-14 19:38:35 <rails> thats a sad
2021-12-15 08:45:09 <nmeum> would it be possible to give me temporary access to an s390x machine? I would like to debug https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27944#note_198509 further but unfourtunatly, the gdb stub provided by s390x qemu-user does not allow setting breakpoints on memory read/writes using awatch/rwatch which makes this memory corruption somewhat diffcult to debug
2021-12-15 08:53:47 <ikke> I think we can give you a container 
2021-12-15 08:54:01 <ikke> ncopa: any objection? 
2021-12-15 09:03:28 <nmeum> don't even necessarily need root and you can delete it after a few days/weeks
2021-12-15 09:10:13 <ncopa> no objection. ofcourse can nmeum get a container
2021-12-15 09:15:01 <ikke> 👍
2021-12-15 09:57:32 <nmeum> sweet, should I send you my ssh key or can you just fetch that from gitlab?
2021-12-15 10:05:02 <ikke> I'll use the one from gitlab 
2021-12-15 10:14:37 <nmeum> ty
2021-12-15 11:12:58 <ikke> nmeum: did you already have VPN access?
2021-12-15 11:30:55 <ikke> I created a container
2021-12-15 12:03:11 <nmeum> ikke: no, I don't but I do have ssh access to existing containers without VPN. for example nld5-dev1.alpinelinux.org
2021-12-15 12:03:42 <nmeum> s/for example/via/
2021-12-15 12:03:42 <alpine-meetbot> nmeum meant to say: ikke: no, I don't but I do have ssh access to existing containers without VPN. via nld5-dev1.alpinelinux.org
2021-12-15 12:05:48 <ikke> We have a wireguard setup. Would that work for you? 
2021-12-15 12:06:27 <ikke> Otherwise I can create a port forward 
2021-12-15 12:24:28 <nmeum> ikke: port forward would be easier for me atm
2021-12-15 12:31:08 <ikke> Ok
2021-12-15 15:31:15 <ncopa> my ncopa-edge-ppc64le cannot git clone gitlab.a.o/alpine/aports. no idea whats wrong
2021-12-15 15:36:23 <ikke> ncopa: I suppose it's still on the vms 
2021-12-15 15:36:27 <ncopa> yes
2021-12-15 15:36:54 <ncopa> curl https://alpinelinux.org hangs
2021-12-15 15:37:28 <ncopa> did we do soemthing with it after reboot? change MTU or similar?
2021-12-15 15:50:57 <ikke> I recall setting it on the vm in /etc/netplan
2021-12-15 16:02:16 <ncopa> my ncopa-edge-ppc64le is useless. dunno what the problem is and I don't have the energy to troubleshoot it
2021-12-15 16:02:29 <ncopa> is it possible to disable the CI for a specific MR?
2021-12-15 16:05:28 <ikke> ncopa: I can move the container for you
2021-12-15 16:08:24 <ncopa> would be nice. thank you!
2021-12-15 16:21:06 <ikke> It's now on the new host (172.16.11.60)
2021-12-15 16:28:17 <ikke> nmeum: you should be able to connect to s390x.alpinelinux.org -p22009
2021-12-15 16:28:50 <nmeum> ikke: as root or as nmeum?
2021-12-15 16:28:54 <ikke> root
2021-12-15 16:29:03 <nmeum> yes, works!
2021-12-15 16:29:07 <nmeum> thanks a lot!
2021-12-15 16:29:08 <ikke> cool
2021-12-15 16:30:00 <nmeum> let's hope I can figure out this gcc-go issue on s390x now 🤗
2021-12-15 16:58:29 <ncopa> thank you ikke! I owe you another beer
2021-12-16 11:44:01 <ncopa> ikke: how do I connect to ncopa-edge-ppc64le now? 
2021-12-16 11:44:07 <ncopa> ssh: Could not resolve hostname ncopa-edge-ppc64le.bra1-dev1.alpin.pw: Name does not resolve
2021-12-16 11:44:43 <ikke> Need to check why dns is not working
2021-12-16 11:44:55 <ikke> 172.16.11.60 should owrk
2021-12-16 11:45:26 <ncopa> maybe there is no NS for bra1-dev1.alpin.pw
2021-12-16 11:46:10 <ikke> it should be bra1
2021-12-16 11:46:17 <ikke> but that wasn't working either the last time I tried
2021-12-16 11:46:52 <ikke> there is an NS record for bra1 pointing to 172.16.11.2
2021-12-16 11:47:33 <ncopa> $ dig ncopa-edge-ppc64le.bra1-dev1.alpin.pw @172.16.11.2
2021-12-16 11:47:33 <ncopa> ; <<>> DiG 9.16.22 <<>> ncopa-edge-ppc64le.bra1-dev1.alpin.pw @172.16.11.2
2021-12-16 11:47:33 <ncopa> ;; global options: +cmd
2021-12-16 11:47:33 <ncopa> ;; connection timed out; no servers could be reached
2021-12-16 11:48:29 <ncopa> there is nothing listening on .2
2021-12-16 11:50:21 <ikke> I see a setting missing
2021-12-16 11:52:23 <ikke> Now there is something listening, but no answer yet
2021-12-16 11:56:15 <ncopa> i had to renew the dhcp lease
2021-12-16 11:56:18 <ncopa> it works now i think
2021-12-16 11:58:22 <ncopa> whee!!! git pull works now! \o/
2021-12-16 11:58:27 <ikke> ncopa: cool
2021-12-16 11:58:29 <ncopa> ikke: thank you!
2021-12-16 11:58:47 <ikke> You're welcome
2021-12-18 11:58:52 <nmeum> ikke: would it be possible to also give me access to an ppc64le container? it's the only architecture where my gcc-go based go bootstrap still fails
2021-12-18 11:59:24 <ikke> sure
2021-12-18 11:59:32 <nmeum> great, ty!
2021-12-18 12:12:25 <ikke> root@ppc64le.alpinelinux.org -p 22061
2021-12-18 12:23:53 <nmeum> > debug1: Connecting to ppc64le.alpinelinux.org [143.106.167.125] port 22061.
2021-12-18 12:23:54 <nmeum> hangs there
2021-12-18 12:23:57 <nmeum> is the port really open?
2021-12-18 12:27:34 <ikke> sshd is running and I did add the portforward. Let me check
2021-12-18 12:30:18 <ikke> I do see the nat rule being hit
2021-12-18 12:31:55 <nmeum> nmap also claims that the port is filtered
2021-12-18 12:32:59 <ikke> I'm missing a MASQUERADE rule
2021-12-18 12:45:46 <ikke> Hmm, not sure yet what is blocking it
2021-12-18 12:50:44 <nmeum> I don't know iptables unfourtunatly
2021-12-18 12:52:14 <ikke> comparing it to the s390x builder
2021-12-18 13:39:05 <ikke> nmeum: it works now, I've manually removed a rule that blocks the traffic
2021-12-18 13:39:11 <ikke> (we use awall to generate the rules)
2021-12-18 13:48:44 <nmeum> yes, does work indeed. thanks!
2021-12-18 13:51:36 <ikke> nice
2021-12-18 14:07:17 <nmeum> hehe, gcc-go works perfectly fine on the ppc64le container :D
2021-12-18 14:07:37 <nmeum> does the gitlab ppc64le ci also use POWER8E or is a different machine?
2021-12-18 14:08:41 <ikke> Different machine
2021-12-18 14:09:17 <nmeum> hm, maybe it is also a docker/seccomp issue
2021-12-18 14:09:48 <nmeum> libucontext on ppc64le (contrary to all architectures) uses SYS_swapcontext maybe that's not prohibited by the seccomp profile and setcontext returns with EPERM/EINVAL?
2021-12-18 14:10:01 <nmeum> s/not prohibited/not allowed/
2021-12-18 14:10:01 <alpine-meetbot> nmeum meant to say: libucontext on ppc64le (contrary to all architectures) uses SYS_swapcontext maybe that's not allowed by the seccomp profile and setcontext returns with EPERM/EINVAL?
2021-12-18 14:13:37 <ikke> nmeum: I see we still have a custom seccomp profile
2021-12-18 14:13:48 <ikke> once the runner is idle, I'll restart docker
2021-12-18 14:13:55 <nmeum> I don't know if that's really the case
2021-12-18 14:13:57 <nmeum> just a theory
2021-12-18 14:14:02 <ikke> nod
2021-12-18 14:14:07 <ikke> easy enough to test though
2021-12-18 14:14:36 <nmeum> how? (:
2021-12-18 14:14:51 <ikke> I mean, removing the custom seccomp profile and restarting docker
2021-12-18 14:15:24 <nmeum> what does the seccomp profile look like?
2021-12-18 14:15:45 <ikke> https://tpaste.us/ZjD6
2021-12-18 14:15:58 <ikke> https://tpaste.us/ZjD6?hl=true
2021-12-18 14:16:30 <nmeum> the stuff in syscalls is a whitelist, not a blacklist, right?
2021-12-18 14:16:49 <ikke> yes
2021-12-18 14:16:56 <nmeum> because it does not contain swapcontext(2)
2021-12-18 14:17:08 <nmeum> where can I find the default seccomp profile?
2021-12-18 14:17:12 <nmeum> i.e. the non-custom one
2021-12-18 14:17:36 <ikke> https://github.com/moby/moby/blob/master/profiles/seccomp/default.json
2021-12-18 14:17:40 <ikke> but there it's not either
2021-12-18 14:18:16 <nmeum> looks like it
2021-12-18 14:18:32 <ikke> I could add it
2021-12-18 14:18:42 <nmeum> maybe there is a technical reason why it isn't allowed?
2021-12-18 14:18:44 <ikke> How sensitive is it?
2021-12-18 14:19:20 <nmeum> I don't know
2021-12-18 14:19:25 <nmeum> https://github.com/systemd/systemd/pull/9487
2021-12-18 14:19:43 <nmeum> looks like systemd nspawn does have it in its seccomp whitelist, also for ppc reasons
2021-12-18 14:20:32 <nmeum> looks like this system call is also only available on powerpc
2021-12-18 14:20:41 <ikke> ok
2021-12-18 14:20:44 <nmeum> which could explain why it isn't in the default seccomp profile
2021-12-18 14:21:34 <nmeum> disclaimer: I am neither an expert on powerpc nor do I know much about the specifics swapcontext syscall
2021-12-18 14:21:42 <ikke> you and me both :P
2021-12-18 14:22:13 <nmeum> unfourtunatly the go error output also does not include the errno for why swapcontext failed but it does look very likely that this might due to a seccomp violation
2021-12-18 14:22:22 <nmeum> I think Ariadne is the expert on all things swapcontext
2021-12-18 14:22:39 <ikke> ahuh
2021-12-18 14:22:41 <nmeum> but if you don't mind we could just change the seccomp profile for ppc64le, include that system call and restart the go build to see if it fixes the problem
2021-12-18 14:23:01 <ikke> sure
2021-12-18 14:23:25 <ikke> draining the runner so that I can restart docker
2021-12-18 14:24:58 <nmeum> from the systemd nspawn PR (linked above) I would assume that we just need to add "swapcontext" to the syscall whitelist
2021-12-18 14:28:13 <nmeum> I will be briefly afk for ~45 min, the MR with the failing ppc64le CI job is https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27944
2021-12-18 14:28:23 <ikke> ok
2021-12-18 15:14:35 <nmeum> I am back, did you restart docker already? (:
2021-12-18 15:15:13 <ikke> yes
2021-12-18 15:15:40 <ikke> But jobs hang at uploading packages atm
2021-12-18 15:15:57 <nmeum> oh
2021-12-18 15:16:02 <nmeum> but that means the build has passed
2021-12-18 15:16:02 <nmeum> \o/
2021-12-18 15:16:07 <ikke> cool
2021-12-18 15:16:21 <nmeum> great
2021-12-18 15:16:27 <ikke> sorry
2021-12-18 15:16:33 <ikke> I did not mean your job specifically
2021-12-18 15:16:39 <ikke> I meant in general
2021-12-18 15:16:48 <nmeum> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/567051
2021-12-18 15:16:54 <nmeum> yeah, but my job hangs at uploading packages too
2021-12-18 15:17:04 <ikke> ok
2021-12-18 15:17:23 <nmeum> I think I will propose a PR to the moby repo for adding swapcontext to the default seccomp profile
2021-12-18 15:17:28 <nmeum> I think it makes sense to have that enabled by default
2021-12-18 15:17:34 <ikke> nod
2021-12-18 15:18:32 <nmeum> but sweet, now gcc-go works on all CI architectures. I think I just need to do some separate testing with riscv64 and than it could be used to boostrap community/go without using recursive dependencies
2021-12-18 15:19:12 <ikke> That's nice
2021-12-18 15:32:28 <nmeum> https://github.com/moby/moby/pull/43092
2021-12-18 15:44:21 <Ariadne> yes on ppc64le we use a syscall for swapcontext
2021-12-18 15:44:27 <Ariadne> so seccomp seems likely
2021-12-18 15:44:38 <Ariadne> we could change that probably
2021-12-18 15:51:09 <ikke> adding swapcontext to the allowed syscalls on ppc64le seems to have worked
2021-12-18 16:34:21 <nmeum> Ariadne: yes, it was the seccmop thing and it was fixed by updating our seccomp profile, I am trying to get docker/moby upstream to include that syscall in the default profile
2021-12-18 16:35:17 <Ariadne> i think it should be changed in libucontext
2021-12-18 16:35:48 <Ariadne> it is likely surprising that libucontext uses kernel syscall on ppc where it doesn't anywhere else
2021-12-18 16:35:55 <nmeum> I think glibc uses the syscall too, don't they?
2021-12-18 16:36:10 <Ariadne> no
2021-12-18 16:36:18 <nmeum> oh
2021-12-18 16:36:53 <Ariadne> i used the syscall because it was there, and i did not want to sort out the details of ELFv2 ABI
2021-12-18 16:37:00 <Ariadne> but then i had to do it for MIPS
2021-12-18 16:41:14 <nmeum> why does linux even offer the swapcontext syscall on ppc? compared to all other architectures i mean
2021-12-18 16:41:28 <nmeum> or to rephrase that: why is the system call not available on other architectures?
2021-12-18 16:41:49 <Ariadne> dunno
2021-12-18 16:42:07 <Ariadne> my guess: IBM wanted it
2021-12-18 16:48:46 <nmeum> meh, the kernel git history doesn't seem to fully track the history on this system call because it has been present for so long that it has been added with the initial git checkin as part of Linux 2.6.12
2021-12-18 17:40:27 <Ariadne> hmmph
2021-12-18 17:40:41 <Ariadne> i wrote a getcontext that should be equivalent, dropped it in, and the whole thing blows up
2021-12-18 17:52:43 <Hello71> there are several bitkeeper to git converted repos
2021-12-18 18:42:57 <nmeum> yeah, but I was to lazy to lookup those
2021-12-20 01:41:20 <Hello71> is it possible to enable zram for ci and builders to enable qt5-qtwebengine to build? i can't think of a better solution
2021-12-20 01:42:56 <Hello71> it is waste of resources to keep rebuilding from scratch because of failure to build job 20000/21000. it will keep using huge ram unless debug symbols are turned off or something else changes
2021-12-20 01:43:10 <Hello71> i thought of reducing jobs but can't think of a clean way to do it
2021-12-20 07:30:53 <clandmeter> Hello71: which arch?
2021-12-20 14:17:05 <Hello71> mainly x86_64 i think
2021-12-20 16:19:25 <ikke> Hello71: how would zram work?
2021-12-20 16:20:20 <ikke> clandmeter: I've changed the alerts in Zabbix to repeat only once per week, not once per day
2021-12-20 16:26:41 <mps> zram helped me 2-3 years ago to build rust and firefox on my chromebook with 4GB ram
2021-12-20 16:27:30 <mps> without it was not possible because it crashed/segfaults
2021-12-20 19:43:55 <ikke> clandmeter: I've restored the database on gitlab-test and redid the upgrade and now I can login again
2021-12-20 19:44:00 <ikke> Not sure what happened
2021-12-20 19:51:10 <ikke> clandmeter: sorry, spoke to soon, accidentally looked at gitlab.a.o, not gitlab-test.a.o :(
2021-12-20 20:24:58 <ikke> 14.3 fails as well :(
2021-12-20 21:35:54 <clandmeter> ikke: do you know the issue?
2021-12-20 21:36:14 <clandmeter> i have a feeling its related to the proxy setup. 
2021-12-20 21:36:17 <Hello71> ikke: do you mean what does zram do or how would it address the issue
2021-12-21 05:31:00 <ikke> clandmeter: I don't know what the issue is. I did read that if rails doesn't think it's ssl, it won't send the session cookie
2021-12-21 05:37:07 <ikke> Hello71: Is it the idea to use it as compressed swap space?
2021-12-21 06:28:01 <clandmeter> ikke: how can i access test setup?
2021-12-21 06:34:31 <clandmeter> ah got it, the ip is still the same
2021-12-21 06:47:06 <clandmeter> ikke: whats up with the double config for nginx?
2021-12-21 06:47:22 <clandmeter> diff http.d/default.conf conf.d/default.conf
2021-12-21 06:49:30 <clandmeter> looks like its switch back to using original upstream config?
2021-12-21 06:54:12 <ikke> Hmm
2021-12-21 06:57:18 <clandmeter> ikke: fixed
2021-12-21 06:57:26 <clandmeter> please try
2021-12-21 06:58:25 <clandmeter> im not sure when you introduced that new config?
2021-12-21 06:58:29 <clandmeter> with the latest update?
2021-12-21 06:58:54 <ikke> I don't recall changing that part
2021-12-21 07:02:20 <clandmeter> can it be related to https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/commit/155e8cabddaa0a90904e5400441a34c60495fbbb
2021-12-21 07:03:30 <clandmeter> i think you changed the location
2021-12-21 07:03:45 <clandmeter> and the scripts think its a new setup copying original config from upsteam
2021-12-21 07:05:46 <clandmeter> the missing item here is: proxy_set_header    X-Forwarded-Ssl     on;
2021-12-21 07:06:05 <clandmeter> and it looks like this is not added to the nginx patch
2021-12-21 07:06:18 <clandmeter> so a new setup should always give this error
2021-12-21 07:17:00 <ikke> OK, good to know 
2021-12-21 07:18:02 <ikke> Thanks! 
2021-12-21 11:20:46 <ikke> clandmeter: what script were you referring to?
2021-12-21 11:21:23 <clandmeter> entrypoint
2021-12-21 11:21:50 <ikke> I did change the location there
2021-12-21 11:22:44 <clandmeter> yes
2021-12-21 11:23:03 <clandmeter> and im not sure, but i think it did not detect a config, so it installed a new one
2021-12-21 11:27:48 <ikke> Ok, right. It looks for the config in /etc/gitlab/nginx/http.d/default.conf, while the existing config is located at /etc/gitlab/nginx/conf.d/default.conf
2021-12-21 14:20:02 <Hello71> ikke: yes
2021-12-21 17:08:05 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/docker/gitlab/-/merge_requests/9
2021-12-21 20:10:20 <clandmeter> ok good, but why are the other patches in this mr?
2021-12-21 20:13:22 <ikke> sudo -> doas because the build failed
2021-12-21 20:13:46 <ikke> apply_patch related changes to make it easier to see in the build what patch file actually fails to apply
2021-12-21 20:14:12 <ikke> (patch does not mentions the name of the patch file>
2021-12-21 20:18:37 <clandmeter> ah ok
2021-12-21 20:18:39 <clandmeter> lgtm
2021-12-21 20:26:45 <ikke> clandmeter: if all is working, I'm planning to upgrade gitlab to 14.4 tomorrow
2021-12-21 20:27:00 <clandmeter> all tests are ok?
2021-12-21 20:27:30 <ikke> except anonymously getting gpg / ssh keys :)
2021-12-21 20:27:40 <clandmeter> did you add more tests since recent issues?
2021-12-21 20:27:45 <clandmeter> or tested it manually
2021-12-21 20:27:49 <ikke> manually
2021-12-21 20:27:53 <clandmeter> nod
2021-12-21 20:28:19 <clandmeter> yes the ssh keys thingy is something i bump into regularly 
2021-12-21 20:28:36 <ikke> We can add basic integration tests for gitlab-shell / gitally to catch those being broken earlier
2021-12-21 20:28:52 <clandmeter> but i can use github instead
2021-12-21 20:34:24 <mps> ikke: and disable Approve button :)
2021-12-22 11:43:18 <ikke> clandmeter: did you create a symlink from conf.d to http.d on gitlab-test?
2021-12-22 11:44:30 <ikke> hmm, that's how it's on production as well
2021-12-22 16:13:15 <ikke> clandmeter: cgit process is running 100% cpu, but it just has one proces
2021-12-22 16:13:20 <ikke> I don't see a lot of traffic
2021-12-22 16:13:24 <rails> morning
2021-12-22 16:13:31 <ikke> rails: morning
2021-12-22 16:13:49 <rails> gitlab being wonky?
2021-12-22 16:14:00 <ikke> no, git.a.o
2021-12-22 16:14:04 <ikke> gitlab is fine
2021-12-22 17:09:21 <ikke> clandmeter: any reason you disabled ugreen on cgit uwsgi?
2021-12-22 17:12:58 <Cogitri[m]> ikke: Hey, the asgen config has been changed, could you pull the newest compose and restart? :)
2021-12-22 17:14:07 <ikke> done
2021-12-22 17:14:48 <Cogitri[m]> Thanks 🤣
2021-12-22 17:14:52 <Cogitri[m]> * Thanks 👍
2021-12-22 18:25:14 <clandmeter> Yes I mention it to you last time
2021-12-22 18:25:18 <clandmeter>  iirc
2021-12-22 18:26:04 <clandmeter>  the issue is with threads
2021-12-22 18:56:23 <ikke> Well, with threads we still have issues :(
2021-12-22 18:56:30 <ikke> without*
2021-12-22 20:39:27 <ikke> upgraded gitlab to 14.4
2021-12-22 21:05:36 <clandmeter> we can switch it back to threads
2021-12-22 21:05:50 <clandmeter> and add a check to restart if needed
2021-12-22 21:06:15 <clandmeter> not really a fix, but should keep it going.
2021-12-23 11:32:58 <ikke> Anything left for https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10733 ?
2021-12-23 14:03:06 <clandmeter> ikke: it looks like all is working?
2021-12-23 15:29:21 <blaboon> 35
2021-12-23 15:29:35 <blaboon> sorry, wrong window :D
2021-12-23 15:31:04 <ikke> ;-)
2021-12-24 17:18:05 <ikke> something something ceph something something
2021-12-24 19:15:49 <ikke> clandmeter: there still is 22G of logfiles on usa9. What should I do with that?
2021-12-26 18:29:54 <ikke> clandmeter: I've migrated traefik on gitlab-test to v2
2021-12-26 19:53:19 <ikke> maxice8: ping
2021-12-27 08:37:44 <clandmeter> morning ikke 
2021-12-27 08:37:54 <clandmeter> regarding logs, not sure what they are? buildlogs?
2021-12-27 08:38:29 <clandmeter> i think we should create a cron task to purge all logs older than x
2021-12-27 08:38:48 <clandmeter> we sync them to distfiles anyways iirc
2021-12-27 11:31:59 <ikke> clandmeter: should that sync job not remove them?
2021-12-27 12:40:00 <clandmeter> No I think it excludes the log files 
2021-12-27 12:51:37 <ikke> Ah ok
2021-12-27 18:33:42 <ikke> clandmeter: ping
2021-12-27 18:36:07 <ikke> 0
2021-12-27 18:37:38 <ikke> clandmeter: I've setup traefik on gitlab-test to forward ssh on ipv6 to gitlab-shell. 
2021-12-27 20:39:58 <clandmeter> Heh
2021-12-27 20:40:47 <ikke> Now gitlab ipv6 points to traefik, so ssh on gitlab.a.o does not work with ipv6
2021-12-27 20:41:22 <clandmeter> Yes I noticed that when I had ipv6 enabled
2021-12-27 20:41:26 <ikke> aha
2021-12-27 20:41:53 <clandmeter>  but ipv6 gives me a lot of troubles
2021-12-27 20:41:57 <ikke> I work mostly on my lxc container, so no ipv6 there :-p
2021-12-27 20:42:04 <ikke> Hmm, generally not for me
2021-12-27 20:42:39 <clandmeter>  you router hands out ipv6 addresses to all clients?
2021-12-27 20:43:01 <clandmeter>  Oh my desktop is fine
2021-12-27 20:43:49 <clandmeter> But stuff like smartphones and chrome casts not
2021-12-27 20:52:07 <ikke> clandmeter: so should I copy that setup to gitlab production?
2021-12-28 19:01:51 <Hello71> ikke: did you delete moby seccomp profile on all ci?
2021-12-28 19:02:17 <ikke> I have not
2021-12-28 19:02:22 <ikke> not explicitly
2021-12-28 19:03:16 <Hello71> https://gitlab.alpinelinux.org/alxu/aports/-/jobs/577209#L1165
2021-12-28 19:03:22 <Hello71> by delete i mean stop overriding
2021-12-28 19:03:32 <ikke> Hello71: yes, understood
2021-12-28 19:04:23 <Hello71> it is only failing for that reason on ppc64le. is that ubuntu now?
2021-12-28 19:04:42 <ikke> no
2021-12-28 19:04:47 <Hello71> s390x failed due to git fetch timeout https://gitlab.alpinelinux.org/alxu/aports/-/jobs/577208, i guess this one is known
2021-12-28 19:05:26 <ikke> The ppc64le builders are back on alpine, CI was always on alpine / bare-metal
2021-12-28 19:07:08 <Hello71> i see. anyways, do you know what the issue is? my understanding was that we stopped overriding docker profile so unshare should return eperm
2021-12-28 19:07:56 <ikke> It was there on ppc64le
2021-12-28 19:09:46 <ikke> yes, that was the only one left
2021-12-28 19:09:48 <Hello71> not sure what you mean. are you saying i should restart ppc64le job and it will work now?
2021-12-28 19:09:58 <ikke> I just restarted the docker daemon
2021-12-28 19:10:00 <Hello71> s390x also just failed (after restart) with same reason: https://gitlab.alpinelinux.org/alxu/aports/-/jobs/577249#L1218
2021-12-28 19:10:37 <ikke> Isn't unshare restricted anyway?
2021-12-28 19:10:39 <Hello71> aarch64 has some other issue i need to investigate. runtime test passes
2021-12-28 19:11:03 <Hello71> yes but it needs to return EPERM for upstream. we have a patch to allow it to return ENOSYS but i want to remove it
2021-12-28 19:11:31 <Hello71> i guess it will fail if you set CONFIG_NAMESPACES=n but i don't think they care about that
2021-12-28 19:12:41 <ikke> Hello71: What might be a factor is that the CI hosts are using an older version for alpinelinux for s390x and ppc64le
2021-12-28 19:12:46 <Hello71> mmhmm
2021-12-28 19:13:12 <ikke> I want to upgrade, but I need assistance from IBM
2021-12-28 19:14:02 <Hello71> i think it should work with alpine 3.12+ though? alpine 3.11 is eol so hopefully we aren't still using that
2021-12-28 19:27:42 <Hello71> so does that mean it is safe to ignore these issues and merge anyways because the builders are on newer versions?
2021-12-28 19:28:05 <Hello71> hm, i guess it would mask other issues from this or future MRs
2021-12-28 19:51:33 <ikke> I'll ask IBM if they can assist us in upgrading these hosts
2021-12-28 19:51:48 <ikke> I don't have OOB access, so doing it myself is tricky
2021-12-28 19:57:00 <Hello71> ok
2021-12-28 20:57:54 <ikke> ncopa: clandmeter all ppc64le containers have been copied to the new build host. Safe to delete the old containers and let IBM know?
2021-12-28 21:07:54 <clandmeter> Lgtm
2021-12-29 08:06:45 <ncopa> ikke: sgtm
2021-12-29 08:14:17 <ikke> sgtm? 
2021-12-29 10:39:27 <ncopa> sounds good to me
2021-12-29 11:18:14 <ikke> ah
2021-12-29 11:31:29 <ikke> Ok, all containers and data is wiped
2021-12-31 04:02:36 <zv> how do y'all manage package signing keys getting to builders securely and making sure no unauthorized folks or software vulns can lead to improper access?
2021-12-31 14:51:21 <ikke> zv: We generate the keys on the builders, so we don't have to ship them there. Keeping them secure is a challenge, and this is something we have ideas to improve.
2021-12-31 14:53:35 <ikke> clandmeter: I've upgraded netbox to 3.1, nld5 to Alpine 3.15
2021-12-31 15:55:25 <ikke> nld8/9 upgraded to alpine 3.15
2021-12-31 17:13:27 <ikke> nld3 upgraded to alpine 3.15
2021-12-31 17:38:28 <ikke> gbr2 upgraded to alpine 3.15
2021-12-31 17:48:59 <ikke> dmvpn1 upgraded to alpine 3.15
2021-12-31 17:57:01 <ikke> dmvpn2 upgraded to alpine 3.15
2021-12-31 20:02:57 <danieli> that's odd, oftc won't let me join -devel, -linux and -offtopic now all of a sudden
2021-12-31 20:04:04 <danieli> it worked after doing a captcha lol
2021-12-31 21:33:55 <ikke> danieli: we set +R due to spam 
2021-12-31 21:34:02 <danieli> makes sense