2021-01-01 15:55:13 <ncopa> i have deleted release candidates from dl-master. there are 128G free now
2021-01-01 15:57:23 <ikke> 👍
2021-01-03 17:19:11 <Cogitri> ikke: Do you happen to know if I can generate custom tarballs and set those as release in Gitlab CI?
2021-01-03 17:19:33 <Cogitri> For apk-polkit-rs I need to make custom tarballs for each tag to include the cargo dependencies so it can be built w/o network
2021-01-03 17:19:49 <Cogitri> Would be neat if gitlab-ci automatically generated that tarball on a tag
2021-01-03 17:20:03 <ikke> Cogitri: afaik we need a newer version of gitlab
2021-01-03 17:20:14 <Cogitri> If found https://docs.gitlab.com/ee/ci/yaml/README.html#release but AFAICS it can only set description?
2021-01-03 17:20:59 <ikke> gitlab 13.5
2021-01-03 17:21:08 <ikke> https://about.gitlab.com/releases/2020/10/22/gitlab-13-5-released/#attach-binary-assets-to-releases
2021-01-03 17:23:18 <Cogitri> Ah thanks
2021-01-03 17:24:20 <ikke> Not clear from the documentation how it works, though
2021-01-03 17:25:19 <Cogitri> https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package/ I guess
2021-01-03 17:26:20 <ikke> Yes, reading that
2021-01-03 17:33:19 <ikke> I'm not sure where to store these assets
2021-01-03 17:33:42 <ikke> I guess you'd need to use artifacts that are stored forever
2021-01-03 18:50:40 <Cogitri> Seems like there's some package registry thingie I can upload to with Gitlab
2021-01-03 18:51:06 <Cogitri> Ohhhh, seems like I can even get download paths without that sha512 sum in it so it's as easy as changing $pkgver in APKBUILDs
2021-01-03 18:57:24 <Cogitri> Tested it on GNOME's gitlab since they have a more recent Gitlab version and uploading seems to work but the download on https://gitlab.gnome.org/Cogitri/test-gtk4/-/releases/0.12.0 404's, great :D
2021-01-03 22:01:31 <Cogitri> ikke: fwiw works now, seems like the admins had to toggle some thingie to enable the package registry. Here's the registry: https://gitlab.gnome.org/Cogitri/test-gtk4/-/packages and here's the .gitlab-ci.yml: https://gitlab.gnome.org/Cogitri/test-gtk4/-/blob/master/.gitlab-ci.yml in case you're curious :)
2021-01-03 22:05:16 <ikke> ok, I think we would need to do that as well
2021-01-03 22:06:08 <ikke> Don't they have support for the `release:` section yet?
2021-01-03 23:07:04 <Cogitri> https://gitlab.com/groups/gitlab-org/-/epics/2510 doesn't seem like they do
2021-01-03 23:07:53 <Cogitri> Huh, seems like I can't download from a package registry as unauthorised user?
2021-01-03 23:10:42 <Cogitri> https://docs.gitlab.com/ee/user/packages/generic_packages/#download-package-file : Prerequisites: You need to authenticate with the API. 
2021-01-03 23:10:43 <Cogitri> Yikes
2021-01-03 23:12:05 <Cogitri> https://gitlab.com/gitlab-org/gitlab/-/issues/271534
2021-01-04 00:04:50 <Cogitri> https://gitlab.gnome.org/Cogitri/gnome-health/-/commit/03073ec2bf7a2f23d3a62021f9586d264ecc1952 with this "neat" workaround it works, hooray
2021-01-04 05:29:29 <ikke> it is an ugly hack indeed
2021-01-04 07:38:38 <ncopa> looks like we now have 3.0T total disk space on dl-master. 2.1T available
2021-01-04 08:23:32 <ikke> Oh, nice, the storage has increased?
2021-01-04 08:56:58 <ncopa> apparently it has
2021-01-04 09:38:41 <clandmeter> the emails im seeing are weird
2021-01-04 09:38:54 <clandmeter> ncopa: you also receive them, but i guess you dont notice them.
2021-01-04 09:39:05 <ikke> What e-mails are you receiving?
2021-01-04 09:39:14 <clandmeter> freespace ones
2021-01-04 09:39:57 <clandmeter> Disk size:
2021-01-04 09:39:57 <clandmeter>   1500000.0 GiB
2021-01-04 09:40:07 <clandmeter> what is that?
2021-01-04 09:40:25 <ncopa> i think they gave us more diskspace
2021-01-04 09:40:30 <ncopa> we now have 3TB
2021-01-04 09:40:46 <ncopa> i suspect its thanks to jirutka
2021-01-04 09:40:48 <clandmeter> Free space:
2021-01-04 09:40:48 <clandmeter>   2131.01 GiB
2021-01-04 09:40:54 <clandmeter> thats whats in the email
2021-01-04 09:40:56 <clandmeter> from 11h ago
2021-01-04 09:41:19 <clandmeter> subject: VPS #8185: 0.14 % of disk space left
2021-01-04 09:41:46 <ncopa> i think someone increased the storage
2021-01-04 09:42:09 <clandmeter> i think jiruka also receives them, could be he asked for more space.
2021-01-04 09:43:42 <ncopa> i suspect that is what happened
2021-01-04 09:43:50 <clandmeter> im checking the panel
2021-01-04 09:43:54 <clandmeter> but it does not show more space
2021-01-04 09:44:21 <ncopa> i logged in to the machine and it shows 3TB disk space
2021-01-04 09:44:34 <clandmeter> as a single disk?
2021-01-04 09:44:39 <ncopa> yes
2021-01-04 09:45:09 <clandmeter> i beleive you, i just dont underrstand this panel :)
2021-01-04 09:45:23 <clandmeter> looks like the os gets 920GB
2021-01-04 09:45:33 <clandmeter> and there is another 2.1 dataset
2021-01-04 09:45:44 <ncopa> https://tpaste.us/W74M
2021-01-04 09:46:01 <clandmeter> simfs?
2021-01-04 09:46:19 <ncopa> it could also be that it they temporarily gave us more diskspace so we dont get locked out, i dunno
2021-01-04 09:46:30 <ncopa> maybe we should ask jirutka if he knows anything about it
2021-01-04 09:46:57 <clandmeter> if i have some free time i can prepare uk.a.o
2021-01-04 09:47:03 <clandmeter> the other servers are ready 
2021-01-04 09:47:35 <clandmeter> https://kb.vpsfree.org/manuals/vps/vpsadminos
2021-01-04 09:47:46 <clandmeter> The VPS is running on OpenVZ Legacy. This virtualization platform is old and deprecated. We recommend an upgrade to vpsAdminOS, our new virtualization platform. Please see the knowledge base English / Czech for more information.
2021-01-05 12:10:20 <ncopa> can anyone check what this is about? https://github.com/alpinelinux/docker-alpine/issues/122
2021-01-05 12:12:29 <ikke> I'm already working with tynor88 here https://gitlab.alpinelinux.org/alpine/aports/-/issues/12268
2021-01-05 12:25:02 <MY-R> ncopa: guy just using some weird not official mirror? "fetch https://alpine.pkgs.org/edge/alpine-main-x86_64/x86_64/APKINDEX.tar.gz"
2021-01-05 12:26:22 <MY-R> and APKINDEX.tar.gz "Error 404: Not Found"
2021-01-05 12:54:03 <ncopa> yeah, i wonder what that is
2021-01-05 13:31:57 <TBK[m]> Any special reason for https://github.com/alpinelinux/docker-alpine not being hosted on gl.a.o?
2021-01-05 13:57:26 <ncopa> nobody cared enough to move it
2021-01-05 14:42:55 <TBK[m]> k :( Would be nice to have it under the same roof so issues could be delegated and moved to the relevant project.
2021-01-05 14:42:55 <TBK[m]> A "soft migration" could be done by disabling GH issues and add a couple of lines to the readme.
2021-01-06 14:57:36 <TBK[m]> build-3-13-mips64 is missing on build.a.o
2021-01-06 14:57:56 <ikke> yea
2021-01-06 14:58:02 <ikke> the host seems to be awol
2021-01-06 16:10:04 <ncopa> Ariadne: seems like mips64 builder is gone again
2021-01-06 16:15:59 <Ariadne> ok
2021-01-06 16:16:05 <Ariadne> i'll check on it
2021-01-06 17:25:21 <ncopa> I'd like to tag rc3
2021-01-06 23:50:55 <TBK[m]> ikke: if you are around could I get you to check where the aarch64 ci runner has wandered of to?
2021-01-07 05:33:48 <ikke> TBK[m]: seem to be all accounted for
2021-01-07 05:38:52 <ikke> 
ACTION must learn to read

2021-01-07 05:39:06 <ikke> booted the ci runner vm again
2021-01-07 14:35:44 <ncopa> Ariadne: what happened to the mips64 builder?
2021-01-07 14:35:57 <ncopa> should we drop mips64?
2021-01-07 14:36:01 <ncopa> or should we drop 3.13 release?
2021-01-07 15:07:14 <ncopa> ugh. i should fix the setup-disk to not install syslinux on non x86*
2021-01-07 18:16:44 <blawiz> tried following this install buide for raspi, https://wiki.alpinelinux.org/wiki/Raspberry_Pi_4_-_Persistent_system_acting_as_a_NAS_and_Time_Machine ... getting "filesystems couldnt be fixed" after last rebooot
2021-01-07 18:17:46 <blawiz> doesnt mount the vfat during boot, to /media/mmcblk0p1 which has "boot" whic /boot is symlinked to
2021-01-07 18:18:15 <blawiz> also after every reboot i have to re-run setup-alpine to get network and hostname etc going
2021-01-07 18:21:35 <mps> blawiz: we have #alpine-linux channel for user help
2021-01-07 18:26:31 <mps> blawiz: do you install in sys mode or 'run from ram', i.e. default
2021-01-07 18:31:56 <blawiz> not sure tbh 
2021-01-07 18:35:14 <mps> blawiz: but please join #alpine-linux channel and ask there
2021-01-07 18:35:40 <blawiz> okok
2021-01-07 18:36:23 <blawiz> weird the channel is not listed in https://kiwiirc.com/search?q=alpine&network=*
2021-01-08 05:40:15 <Ariadne> ncopa: its internet connection is down, we are working to fix it asap
2021-01-08 05:40:25 <Ariadne> i think the modem needs to be rebooted again
2021-01-08 05:40:53 <Ariadne> i'm planning to get another builder and put it in my colo to replace the current one :)
2021-01-08 08:10:56 <ncopa> would be good
2021-01-08 12:47:19 <Ariadne> connectivity should be restored
2021-01-08 12:48:11 <ikke> ack
2021-01-08 12:48:20 <Ariadne> we kinda got banned
2021-01-08 12:48:22 <Ariadne> lol
2021-01-08 12:48:36 <ikke> oof
2021-01-08 12:48:46 <ikke> traffic?
2021-01-08 12:49:03 <Ariadne> yeah tripped some alarm
2021-01-08 14:49:49 <ncopa> ugh
2021-01-09 01:53:56 <TBK[m]> No space left on device. -> build-edge-aarch64: failed to build ceph: https://build.alpinelinux.org/buildlogs/build-edge-aarch64/community/ceph/ceph-15.2.8-r1.log
2021-01-09 08:06:55 <Ariadne> yes, ARM builders seem out of disk
2021-01-09 08:18:46 <ikke> ok, the cron that cleans up docker stuff is missing there
2021-01-09 08:19:04 <ikke> oh
2021-01-09 08:19:08 <ikke> not docker :P
2021-01-09 08:34:48 <mps> builder is '/dev/mapper/vg0-lv_root 455795016 432572156         0 100% /'
2021-01-09 08:36:30 <ikke> dedupping reclaimed 16G
2021-01-09 08:42:27 <ikke> NOW you mention it? :P
2021-01-09 13:17:36 <Cogitri> Seems like armv7 is full again, Rust is too huge :D
2021-01-09 13:18:05 <ikke> yes, trying to figure out how to get more space
2021-01-09 13:18:10 <ikke> or rather, clean things up
2021-01-09 13:19:22 <Cogitri> Cleaning up my aarch64 container if that helps
2021-01-09 13:19:41 <ikke> if it's on usa4, it would help
2021-01-09 13:21:06 <Cogitri> I think it is
2021-01-09 13:21:13 <Cogitri> Cleaned it up, not a bad idea either way :)
2021-01-09 13:22:17 <ikke> <500M :P
2021-01-09 13:49:37 <mps> huh, I'm using 6-7GB
2021-01-09 13:54:27 <mps> now about 3GB
2021-01-09 13:58:32 <mps> now 2GB, can't go down more
2021-01-09 13:58:58 <ikke> mps: thanks
2021-01-09 13:59:18 <ikke> 6G free atm
2021-01-09 13:59:43 <ikke> There are a lot of older distfiles for edge that I suppose can be removed, but I'd need something to identify them
2021-01-09 14:35:08 <ddevault> hi, can I have my gitlab account updated to allow me to create public repos under my personal namespace
2021-01-09 14:35:18 <ddevault> I want to experiment with some API integrations
2021-01-09 14:47:21 <ikke> ddevault: this is a global setting, we cannot enable it per user (we can make repos public per user / namespace, but not give them permissions to make public repos)
2021-01-09 14:50:33 <ddevault> okay, thanks ikke
2021-01-09 14:50:38 <ddevault> I think I can get what I need with internal repos
2021-01-09 14:51:03 <ddevault> I'll ask again if not and we can look for something else to do, thanks
2021-01-09 21:11:47 <ddevault> okay, I have completed an experimental version of this concept
2021-01-09 21:12:13 <ddevault> I have some code which can take patches emailed to the mailing list and open gitlab MRs for them, then bidirectionally forward feedback as appropriate
2021-01-09 21:12:38 <ddevault> I will need someone with admin access to aports on gitlab.a.o to help get it up and running
2021-01-09 21:12:50 <ddevault> <discuss>
2021-01-09 21:13:03 <ikke> I was thinking about something like that
2021-01-09 21:13:08 <ddevault> example here: https://gitlab.alpinelinux.org/ddevault/scdoc/-/merge_requests/6
2021-01-09 21:31:24 <maxice8> ddevault: that sounds amazing
2021-01-09 21:33:11 <ddevault> I'm not interested in spending 5 minutes to do what could take 30 seconds because no one wants to review the mailing list anymore
2021-01-09 22:42:59 <MY-R> Oo
2021-01-09 23:59:26 <ddevault> I'm going to take lists.a.o briefly offline to do that side of the deployment work, presuming no one objects
2021-01-10 00:09:31 <ddevault> back online
2021-01-10 00:22:28 <ddevault> summary of the necessary changes: https://paste.sr.ht/~sircmpwn/16d4cfa18614a75f762771a701c8544c8c7c9f7e
2021-01-10 04:34:11 <Ariadne> nice work
2021-01-10 12:09:56 <ikke> clandmeter: any opinion on ^^
2021-01-10 20:04:13 <clandmeter> ikke: about sr.ht?
2021-01-10 22:03:37 <TBK[m]> is the mips64 box AWOL again?
2021-01-10 22:45:56 <ikke> no
2021-01-11 08:01:49 <clandmeter> ikke: ?
2021-01-11 08:02:00 <clandmeter> the mirror?
2021-01-11 08:02:17 <ikke> clandmeter: the proposal from ddevault 
2021-01-11 09:28:38 <clandmeter> ddevault: hi, what is the current status about your cooperation with our infra team?
2021-01-11 12:47:46 <ddevault> clandmeter: I will commit to as much work as is necessary to maintain my interests in alpine
2021-01-11 12:48:01 <ddevault> namely keeping the list running and now, it seems, some secondary software to forward patches to gitlab
2021-01-11 12:48:11 <ddevault> so I can keep my packages up to date
2021-01-11 12:53:30 <clandmeter> ddevault: understand, will you keep lists.a.o updated?
2021-01-11 12:53:51 <ddevault> sure
2021-01-11 12:54:08 <ddevault> it takes care of itself for the most part, hasn't crashed since I first deployed it
2021-01-11 12:54:30 <clandmeter> i think its still running an older version?
2021-01-11 12:54:35 <ddevault> I updated it yesterday
2021-01-11 12:54:41 <clandmeter> nod
2021-01-11 12:54:45 <clandmeter> good
2021-01-11 12:54:53 <clandmeter> thanks for that
2021-01-11 12:55:09 <clandmeter> i have a concern about your request
2021-01-11 12:55:42 <clandmeter> i guess it needs admin perms to impersonate users?
2021-01-11 12:55:58 <ddevault> no, that's not how it works
2021-01-11 12:56:13 <ddevault> it would just use one account to submit MRs on behalf of others, and clarifies the original author in the MR description
2021-01-11 12:56:34 <ddevault> it needs admin perms to set up webhooks, and normal perms to use the API to open MRs, leave comments, etc
2021-01-11 12:57:01 <clandmeter> ok
2021-01-11 13:01:11 <clandmeter> maybe im not getting it, but the MR's will be created against branches on the main aports tree? not like we do now when the user uses its own branch?
2021-01-11 13:01:56 <ddevault> it pushes a branch called list-patch-%d or something similar, to the main aports repository
2021-01-11 13:02:01 <ddevault> and sets the flag which deletes the branch after merge
2021-01-11 13:02:17 <ikke> We could make a dedicated fork for this, I guess?
2021-01-11 13:02:29 <ddevault> it would complicate things
2021-01-11 13:02:32 <ddevault> is there any reason to?
2021-01-11 13:02:52 <ddevault> gitlab already makes references in the upstream repo for every MR opened
2021-01-11 13:02:54 <ddevault> similarly to github
2021-01-11 13:04:06 <ikke> https://tpaste.us/vZWO
2021-01-11 13:04:21 <ikke> refs/heads namespace is clean now
2021-01-11 13:05:15 <ddevault> git ls-remote | wc -c gives 4.5MiB from where I'm standing
2021-01-11 13:05:33 <ddevault> and in any case, it cleans up after itself, the branches are removed after merge
2021-01-11 13:07:34 <Cogitri> Would mean all users fetch those temporary branches though
2021-01-11 13:07:42 <Cogitri> Would be a lot nicer to have it in a fork imho
2021-01-11 13:15:01 <ddevault> putting it in a fork makes it a lot less useful as a re-usable appliance
2021-01-11 13:15:11 <ddevault> part of the appeal of writing this is that it might be useful to other sr.ht users elsewhere
2021-01-11 13:15:22 <ddevault> and I don't want to maintain a patched codebase for alpine's sake
2021-01-11 13:16:13 <ikke> I can imagine there are more projects that work with a fork + MR workflow
2021-01-11 13:16:44 <ddevault> and many who also commit to branches and MR from those
2021-01-11 13:16:52 <ikke> sure
2021-01-11 13:16:52 <ddevault> it's not easy to tell which is being used just by observing MRs
2021-01-11 13:17:02 <ikke> enough projects who pus hdirectly to master as well
2021-01-11 13:17:11 <ddevault> in any case, using a fork requires asking the user to make some error-prone judgement calls
2021-01-11 13:17:24 <ikke> who is the user in this case?
2021-01-11 13:17:25 <ddevault> where does the fork go? who owns it? what is it named? The fork name would normally clash with the upstream repo
2021-01-11 13:17:39 <ddevault> many people might not like to have two versions of the repo in their profile, upstream and upstream.for-ml-patches
2021-01-11 13:18:07 <ddevault> the user -> whoever is in charge of the integration
2021-01-11 13:18:30 <ddevault> right now the integratino is two steps: choose gitlab repo, choose mailing list
2021-01-11 13:18:38 <ddevault> integration*
2021-01-11 13:19:06 <ikke> I imagine what we ask would add just one more parameter? What project to make the MR against?
2021-01-11 13:19:21 <ikke> btw, what about patches aginst stable branches?
2021-01-11 13:19:51 <ddevault> the implicit step would be setting up a fork beforehand
2021-01-11 13:20:07 <ikke> sure, but that can be optional
2021-01-11 13:20:19 <ddevault> patches against stable branches wouldn't work correctly with this approach
2021-01-11 13:20:33 <ddevault> I'll have to come up with something later
2021-01-11 13:20:50 <ddevault> [PATCH v3.12] or something would probably work fine
2021-01-11 13:21:06 <ddevault> anyway, a few temporary branches seems like a reasonable price to me
2021-01-11 13:21:28 <ddevault> despite maxice8's best efforts, several people are still submitting patches via the mailing list, and I would rather not see them get shafted
2021-01-11 13:21:50 <ddevault> 7 people in the last month
2021-01-11 13:22:09 <ikke> yes, I see all these patches
2021-01-11 16:41:16 <ikke> clandmeter: ncopa suggested trying to upgrade musl in gitlab from edge to see if gitaly-ssh still hangs. Anything against that? Reverting that should be easy enough.
2021-01-11 16:41:52 <clandmeter> Np
2021-01-11 16:42:08 <clandmeter> I would do it from the current container
2021-01-11 16:42:43 <ikke> done
2021-01-11 16:42:46 <ikke> yes
2021-01-11 16:42:54 <clandmeter> Easy to drop it
2021-01-11 16:43:15 <ikke> apk add -X https://dl-cdn.alpinelinux.org/alpine/edge/main -u musl
2021-01-11 16:44:35 <clandmeter> No missing symbols?
2021-01-11 16:44:36 <ikke> Will keep an eye to see if we still get hanging gitaly-ssh processes
2021-01-11 16:45:02 <ikke> I think for x86_64 it was ABI compatible?
2021-01-11 16:45:37 <ikke> https://abi-laboratory.pro/index.php?view=timeline&l=musl
2021-01-11 16:45:40 <ikke> yes
2021-01-11 16:45:55 <ikke> No removed symbols
2021-01-11 16:47:03 <ikke> rebasing still works
2021-01-11 17:18:44 <ncopa> upgrading musl should always work, downgrading may not
2021-01-11 17:19:06 <ikke> ok
2021-01-11 17:22:17 <ddevault> so are we at an impasse with the gitlab bot here
2021-01-11 17:26:57 <ikke> ncopa: any opinion on this? ^ (summary: the integration that ddevault made requires topic branches to be created directly on alpine/aports
2021-01-11 17:30:55 <ncopa> ikke: tbh, i don't want get involved in that. my head is more than enough busy with other things. I trust you and clandmeter on this.
2021-01-11 17:31:24 <ncopa> you know gitlab much better than i do
2021-01-11 17:31:36 <ikke> This is not a gitlab specific thing
2021-01-11 17:31:52 <ikke> just whether you mind topic branches created directly on alpine/aports
2021-01-11 17:31:56 <ncopa> oh
2021-01-11 17:32:00 <ncopa> right
2021-01-11 17:32:22 <ncopa> so every time i do git pull i'll get those branches locally i suppose?
2021-01-11 17:32:37 <ncopa> will they get deleted once they are merged?
2021-01-11 17:32:38 <ikke> Yes, for as long as they are still there
2021-01-11 17:32:40 <ikke> yes
2021-01-11 17:33:06 <ikke> you need to set fetch.prune to true to automatically remove remote tracking branches
2021-01-11 17:33:48 <ncopa> will I have to clean up references to the deleted branch locally too?
2021-01-11 17:34:01 <ikke> not if you use that config setting
2021-01-11 17:34:15 <ikke> then after git fetch (or git pull) from alpine/aports, they are pruned
2021-01-11 17:34:59 <ncopa> and fetch.prune is false by default?
2021-01-11 17:35:02 <ikke> yes
2021-01-11 17:35:18 <ncopa> so i need to reconfig every single place i have checked out
2021-01-11 17:35:31 <ncopa> and on the build servers too i suppose
2021-01-11 17:36:13 <ikke> yes
2021-01-11 17:36:57 <ncopa> git fetch --prune
2021-01-11 17:37:04 <ncopa> or git pull --prune
2021-01-11 17:37:07 <ncopa> i guess
2021-01-11 17:37:44 <ncopa> i guess i can fix the build script to deal with it
2021-01-11 17:37:56 <ncopa> how will it work for stable branches?
2021-01-11 17:38:20 <ikke> fetch --prune?
2021-01-11 17:38:54 <ncopa> yeah, instead of setting the config i can change the script to do `git pull --prune` 
2021-01-11 17:38:59 <ikke> yes
2021-01-11 17:39:10 <ncopa> so i dont need remeber set that every time i set up new build server
2021-01-11 17:39:15 <ikke> it's a safe option if you don't care about remote tracking branches for removed branches
2021-01-11 17:39:40 <ncopa> what happens if an MR never gets merged?
2021-01-11 17:39:47 <ncopa> the the feature branch will stay there forever?
2021-01-11 17:39:49 <ikke> Then we need to clean it up at some point
2021-01-11 17:39:52 <ikke> yes
2021-01-11 17:40:20 <ncopa> given that we currently have 174 open MRs
2021-01-11 17:40:41 <ncopa> we can expect to have  ~150 -> 200 feature branches at all times
2021-01-11 17:40:51 <ncopa> i dont like it tbh
2021-01-11 17:40:54 <ikke> This is only for patches sent to the ML
2021-01-11 17:41:28 <ncopa> i suppose we can live with that
2021-01-11 17:41:33 <ncopa> test and see how it goes
2021-01-11 17:41:39 <ncopa> how are those features branches named?
2021-01-11 17:41:48 <ncopa> who/what creates the feature branch?
2021-01-11 17:42:09 <ikke> list-patch-<nr>
2021-01-11 17:42:20 <ddevault> mailed-patch-<nr>, actually
2021-01-11 17:42:36 <ncopa> and what happens if there already exists a branch with that name?
2021-01-11 17:42:43 <ddevault> and they are created by a sr.ht service which has hitherto not been in use for alpine linux, and is designed to facilitate linkage between sr.ht services and third-party software
2021-01-11 17:43:01 <ddevault> it will fail, but this is an unlikely situation
2021-01-11 17:43:13 <ddevault> someone would probably have to arrange for that deliberately with the intention of breaking it
2021-01-11 17:43:16 <ikke> ddevault: I guess it does not handle rerolls for patch series either?
2021-01-11 17:43:16 <ncopa> im just want make sure so not someone sends a crafted email and delete 3.12-stable for example
2021-01-11 17:43:27 <ddevault> that's not possible
2021-01-11 17:43:44 <ddevault> it's mailed-patch-<int>, where the integer is filled in from the database and increments upwards from one
2021-01-11 17:44:00 <ddevault> unless we name an important branch mailed-patch-<n> where n is a future ID, it won't be an issue
2021-01-11 17:44:07 <ikke> ddevault: just a suggestion, maybe use a branch namespace?
2021-01-11 17:44:12 <ncopa> ddevault: that is my concern. that someone intentionally overwrites an already existing branch
2021-01-11 17:44:31 <ikke> refs/heads/ml/*
2021-01-11 17:44:34 <ddevault> ikke: presently yes, but some plans upstream will determine the relationship between versions in a patch series, and this will make it possible to expand on this feature in the gitlab integration
2021-01-11 17:44:34 <ikke> or whatever
2021-01-11 17:44:44 <ddevault> as for namespaces, mailed-patch- is a namespace, it just doesn't use slashes
2021-01-11 17:44:59 <ikke> ddevault: right, so you cannot use gits built-in support for namespaces
2021-01-11 17:45:15 <ddevault> git has built-in support for namespaces?
2021-01-11 17:45:54 <ikke> / is a native namespace divider, but there is also some (rarely used) namespace support)
2021-01-11 17:46:08 <ddevault> I don't think that's true
2021-01-11 17:46:17 <ddevault> I have never heard of such a thing and a cursory review of some relevant man pages turns up nothing
2021-01-11 17:46:24 <ddevault> namespaces in branch names are by convention only
2021-01-11 17:47:01 <ddevault> slashes seem more brittle to me, because git actually does use them for refs/heads/etc
2021-01-11 17:47:01 <ncopa> in either case, right now i want get 3.13 out. anything that slows that down or delays the release is unpopular
2021-01-11 17:47:15 <ddevault> I was told that from 3.13, patches sent to the mailing list will not be reviewd
2021-01-11 17:47:21 <ddevault> so I consider this a blocker unless those plans change
2021-01-11 17:47:28 <ddevault> reviewed*
2021-01-11 17:47:36 <ikke> I have not heard such a thing
2021-01-11 17:48:05 <ddevault> maxice8 said so much on the mailing list
2021-01-11 17:48:10 <ddevault> and has been telling everyone who sends in a patch to go to gitlab instead
2021-01-11 17:48:22 <ncopa> and if ikke and clandmeter want me to make decision, tha answer is going to be: "is it needed for the 3.13 release? will it help us get the 3.13 out? can it wait til after 3.13?"
2021-01-11 17:48:41 <ddevault> it can wait if we can agree not to ignore the list after 3.13 comes out
2021-01-11 17:48:56 <ikke> ddevault: well, he said he won't be reviewing it anymore
2021-01-11 17:49:02 <ddevault> is anyone else?
2021-01-11 17:49:04 <ncopa> right now everything that does not helps getting 3.13 out we get a "no" from me :)
2021-01-11 17:49:14 <ikke> ddevault: I am, but I don't have a lot of bandwidth
2021-01-11 17:49:31 <ddevault> abandonment through reviewer attrition is abandonment all the same
2021-01-11 17:49:38 <ikke> sure
2021-01-11 17:49:52 <ddevault> can we not make some kind of compromise here, commit to reviewing from the list until the release is behind us and we can get this tool in place?
2021-01-11 17:50:12 <ikke> ddevault: We cannot make commitments on behalve of others
2021-01-11 17:50:21 <ddevault> sure, but can anyone commit on behalf of themselves
2021-01-11 17:50:30 <ikke> I'll try to still apply patches on a best-effort basis
2021-01-11 17:50:56 <maxice8> I will review until 3.13.0 is out, if you want I can also keep reviewing until the tool is implemented. 
2021-01-11 17:51:05 <ddevault> that would be agreeable, thank you
2021-01-11 17:51:20 <ddevault> in that case, we can defer the tool discussion until post-release
2021-01-11 17:57:18 <ncopa> ddevault: i currently ignore any MR or patch from mailinglist that does not help 3.13 get out
2021-01-11 17:57:45 <ddevault> yeah, no complaints here
2021-01-11 17:57:51 <ddevault> just concerned about the post-release
2021-01-12 12:32:36 <Cogitri> Not that important, but is it just me or is the DKIM signature missing for every message on the ~devel list? 
2021-01-12 12:32:52 <Cogitri> Just curious if that's an oddity of the ML or if I didn't setup my mail server correctly
2021-01-12 12:35:12 <mps> Cogitri: I see it in your last mail
2021-01-12 12:37:11 <Cogitri> Hm, clicking on "Details" in my message in https://lists.alpinelinux.org/~alpine/devel/%3CCAGP1gyPexhACLxkTfmqVYX%2BDg9awd0LqwjnnSvHaTWc%3Dvp1XUg%40mail.gmail.com%3E says that the DKIM signature is missing, but seems like it works for other things so I guess that's just the ML
2021-01-12 12:37:15 <Cogitri> Thanks mps :)
2021-01-12 12:39:28 <mps> hmm, yes. interesting
2021-01-12 14:41:42 <clandmeter> ikke: any difference now with new musl?
2021-01-12 14:42:28 <ikke> 7 gitaly-ssh processes hanging around
2021-01-12 14:44:40 <ikke> clandmeter: s390x builder suffers from long dns delays (causing timeout issues_
2021-01-12 14:46:22 <ikke> clandmeter: queries to unknown hosts seem to consistently take 5 seconds
2021-01-12 15:10:30 <clandmeter> ikke: this is all related?
2021-01-12 15:10:44 <ikke> 2 separate issues
2021-01-12 15:10:50 <clandmeter> nod
2021-01-12 15:11:05 <ikke> But builds are failing on s390x atm due to it
2021-01-12 15:11:16 <clandmeter> so both new gitlab and musl does not solve the hanging processes 
2021-01-12 15:11:28 <ikke> 1lookup proxy.golang.org on 172.16.10.1:53: read udp 172.16.10.4:53948->172.16.10.1:53: i/o timeout
2021-01-12 15:11:31 <ikke> clandmeter: seems like it
2021-01-12 15:51:39 <MY-R> proxy.golang.org got address 172.217.16.17, maybe somebody filter it out as bogons ip (by setting up wrong netmask or something) :D
2021-01-12 15:51:57 <ikke> MY-R: it's not a spefic hostname
2021-01-12 15:52:12 <ikke> one time it's proxy.golang.org, next time it's a different name
2021-01-12 15:52:29 <MY-R> oh, ok then ignore me :)
2021-01-12 22:40:33 <mps> usa4 disk is nearly full
2021-01-13 14:32:52 <ikke> still dns issues on the s390x builder :-/
2021-01-13 16:10:54 <MY-R> no dns cache? :<
2021-01-13 16:11:38 <ikke> dnsmasq does caching
2021-01-13 16:12:02 <ikke> But not that long apparently
2021-01-13 16:12:07 <MY-R> but it doesnt server expired caches?
2021-01-13 16:12:39 <ncopa> maybe we should set up unbound for the build servers
2021-01-13 16:12:39 <MY-R> unbound and kresd/knot cache resolver got that option
2021-01-13 16:13:03 <ikke> ncopa: But I think that still does not solve the underlying issue
2021-01-13 16:13:23 <ikke> at least what I saw, for some reason dns queries take 5 seconds
2021-01-13 16:13:28 <ncopa> packetloss?
2021-01-13 16:13:49 <ikke> Looks like it, but hard to verify
2021-01-13 16:14:16 <ncopa> i worked around it last time by doing ping
2021-01-13 16:14:22 <ncopa> so the name got cached
2021-01-13 16:14:23 <ikke> I did try a packet dump. but for some reason I never saw the response from the external dns server, even though the query went through
2021-01-13 16:14:34 <MY-R> even for me unbound without enabled expired cache got some long delays with resolving randomly some names what was rly annoying because I have stable enough network
2021-01-13 16:14:49 <ncopa> might be that the response packet got lost
2021-01-13 16:15:10 <MY-R> and knot cache by default dump caches on disk and can serve expired too
2021-01-13 16:16:06 <ikke> hmm, might it be that dns uses a response from cache when there is no response? (like MY-R was alluding to I guess)
2021-01-13 16:17:16 <MY-R> it still will try resolve name after timeout and if got some learning function with predicting and refresh caches
2021-01-13 16:17:25 <MY-R> it got*
2021-01-13 16:17:59 <MY-R> but ye, better got response than dont get it at all
2021-01-13 16:18:04 <ikke> right, but that might explain why I still got a result on the CLI, even though I never saw a packet coming back
2021-01-13 16:18:51 <MY-R> use few dns hosts like 1.1.1.1 and 8.8.8.8 and 9.9.9.9 and it will pick up the fastest one
2021-01-13 16:19:02 <ikke> I tried
2021-01-13 16:19:07 <ikke> without resolve :P
2021-01-13 16:20:00 <MY-R> or use recursive mode so wont care about any google/cloudflare stuff which tend to block some requests from time to time or got some rate limits
2021-01-13 16:20:37 <MY-R> but ye that with unbound or knot cache at least
2021-01-13 16:25:22 <MY-R> ikke: did you put that option in dnsmasq.conf? all-servers
2021-01-13 16:25:42 <MY-R> without it it will resolve one by one, with that option it will try all servers at once
2021-01-13 16:31:00 <ikke> MY-R: I haven't verified, will check
2021-01-13 16:31:50 <MY-R> because ye is weird that it couldnt resolve anything from multiple servers, so 'all-servers' would help for sure
2021-01-13 16:40:00 <MY-R> ikke: ahh didnt see your meassge, so you always got response but with some big delay?
2021-01-13 16:43:08 <ikke> MY-R: for uncached results, it seemed to always take 5 seconds, which looks suspicious to me
2021-01-13 16:43:19 <ikke> but it can be a default timeout in dnsmasq
2021-01-13 16:47:11 <MY-R> ikke: so ye all-servers is first option to try
2021-01-13 17:09:44 <ikke> MY-R: I guess all-servers helps 
2021-01-13 17:10:08 <ikke> shotgun approach :P
2021-01-13 17:17:07 <MY-R> ikke: ye with single forward server everything can happen, it is UDP :P
2021-01-13 17:17:38 <MY-R> can lost somewhere between, then dnsmasq was waiting 5 seconds to time out and pickup another, and now not waiting at all
2021-01-13 17:17:46 <ikke> sure, normally it's not an issue, but apparently our s390x builder does not have a reliable connection
2021-01-13 17:19:57 <MY-R> since few years I have to use at least two forwarders or just recursive because those weird random delays
2021-01-13 17:20:27 <MY-R> so wouldnt call it shotgun but necessary evil :P
2021-01-13 17:51:55 <minimal> MY-R: if you remember back a few years ago when Alpine first started to be used as a Docker base image by many people, there were complains about Alpine's then behaviour being "broken" as people were relying on DNS lookups to go sequentially through the list of servers (i.e. 1st would point to Consul, 2nd to Internet) in /etc/resolv.conf whereas as Alpine did lookups in parallel (as per the DNS RFCs) if upstream server responding 
2021-01-13 17:51:55 <minimal> quicker than Consul then it would cause lookup failures for .consul domain
2021-01-13 18:00:27 <MY-R> minimal: ye I think I read that but for me those people were broken ;) many tools already ignoring order in resolv.conf file
2021-01-13 18:02:08 <minimal> yupe, that's why I pointed out Alpine was following RFCs
2021-01-14 07:52:48 <Cogitri> Built new Docker image for aports-qa-bot with autolabeling support and the new mentors group
2021-01-14 07:59:19 <ikke> Cogitri: alright
2021-01-14 07:59:32 <ikke> so only making sure it uses the latest image, right?
2021-01-14 08:00:29 <Cogitri> Yup, pulling new image and restarting should do
2021-01-14 08:01:03 <ikke> ✔️
2021-01-14 08:04:18 <Cogitri> Thanks :)
2021-01-14 10:04:17 <Cogitri> Pushed another version which doesn't do an API call per commit to add/remove commits, could you pull and restart again? 😅
2021-01-14 10:05:34 <ikke> yes
2021-01-14 10:05:48 <ikke> done
2021-01-14 10:08:54 <Cogitri> Thanks
2021-01-14 10:27:48 <ncopa> the https://dl-master.alpinelinux.org/ has a cert failure due to hostname is cz.a.o. can we update the cert to allow both dl-master and cz.a.o?
2021-01-14 10:29:58 <ikke> it's also referred to by master.a.o
2021-01-14 10:30:12 <ikke> I don't have access, so I cannot arrange it
2021-01-14 10:33:27 <ikke> Not sure if we want our regular wildcard cert there?
2021-01-14 11:51:47 <clandmeter> what is the reason to properly support it?
2021-01-14 11:51:54 <clandmeter> we do not advertise it do we?
2021-01-14 11:54:49 <ikke> dl-master?
2021-01-14 12:02:49 <clandmeter> yes
2021-01-14 15:04:12 <ncopa> the alpine-mksite downloads things from there
2021-01-14 15:04:17 <ncopa> currently using http
2021-01-14 15:04:22 <ncopa> but it would be nice to use https
2021-01-14 15:43:41 <ncopa> reasong why we want https for dl-master: https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/issues/3
2021-01-15 10:40:59 <clandmeter> ikke: pkgs is not smart enough to handle updates like that
2021-01-15 10:41:29 <ikke> like what?
2021-01-15 10:41:46 <clandmeter> like what you did
2021-01-15 10:42:01 <clandmeter> i guess you added 3.13?
2021-01-15 10:42:38 <ikke> yes, I did, indeed
2021-01-15 10:42:47 <ikke> I tried to recall what I did last time
2021-01-15 10:42:54 <ikke> (And I ran the import command
2021-01-15 10:42:59 <ikke> which seems to import everything)
2021-01-15 10:43:28 <clandmeter> it runs the update every 15 min
2021-01-15 10:43:34 <ikke> yes
2021-01-15 10:43:34 <clandmeter> which it does now a lot of times
2021-01-15 10:43:46 <clandmeter> grinding to a halt i guess
2021-01-15 10:44:33 <clandmeter> https://tpaste.us/zx6a
2021-01-15 10:53:16 <clandmeter> ikke: i turned of the update container and run the update in tmux
2021-01-15 10:55:16 <clandmeter> need to add some flock logic to the update script
2021-01-15 10:58:29 <clandmeter> ikke: looks like it finished
2021-01-15 10:58:37 <clandmeter> seems most of 3.13 was already added
2021-01-15 20:52:39 <mcrute> is there a way that I can trigger the website to be rebuilt and deployed?
2021-01-15 20:56:15 <ikke> good question
2021-01-15 20:59:28 <ikke> It's updated either by commits from the mksite repo, or from aports
2021-01-15 21:07:56 <danieli> it used to be commits to the private mksite
2021-01-15 21:08:42 <ikke> it's now on gitlab
2021-01-15 21:09:00 <danieli> figured as much, hence the 'it used to be' :)
2021-01-15 21:09:10 <danieli> it might as well use webhooks and other fancy stuff now
2021-01-15 21:09:16 <ikke> mcrute already contributed to it
2021-01-15 21:33:08 <maxice8> armhf has the wrong distfile for libtorrent-rasterbar
2021-01-16 00:56:49 <mcrute> ikke: sorry had to step away... yeah I knew that contributing would re-deploy it but was hoping to not have to make a dummy commit
2021-01-16 00:57:02 <mcrute> maybe there's a crank somewhere I can turn to do a deploy?
2021-01-16 00:57:16 <mcrute> we just pushed an updated YAML to the cloud images repo and need the page re-generated
2021-01-16 14:09:31 <ikke> clandmeter: ping
2021-01-16 14:10:30 <clandmeter> pong
2021-01-16 14:10:39 <ikke> I did a thing
2021-01-16 14:11:12 <ikke> latest musl adds a new syscal that gives issues with docker/seccomp
2021-01-16 14:11:25 <ikke> so I thought, lets try to upgrade usa7 to alpine 3.13 :P
2021-01-16 14:11:43 <clandmeter> ok, and it breaks?
2021-01-16 14:11:53 <ikke> well, in the oob console, it now asks for a "user password"
2021-01-16 14:12:06 <ikke> and mentions after failed tries "hdd is locked"
2021-01-16 14:12:15 <clandmeter> huh
2021-01-16 14:12:25 <ikke> yes
2021-01-16 14:12:38 <clandmeter> that does not sound like an os issue?
2021-01-16 14:12:41 <ikke> no
2021-01-16 14:12:51 <ikke> It happens before it boots the OS
2021-01-16 14:13:12 <clandmeter> whats on usa7?
2021-01-16 14:13:17 <ikke> x86_64 ci host
2021-01-16 14:13:23 <clandmeter> ok
2021-01-16 14:15:42 <ikke> but it does not ring a bell for you either then
2021-01-16 14:16:22 <clandmeter> vaguely 
2021-01-16 14:16:42 <clandmeter> the interface has changed a bit
2021-01-16 14:19:15 <clandmeter> ikke: did you make a screenshot?
2021-01-16 14:19:19 <clandmeter> or similar
2021-01-16 14:19:45 <ikke> no
2021-01-16 14:19:58 <ikke> rebooting it
2021-01-16 14:20:08 <clandmeter> i just made it run into resque os
2021-01-16 14:20:43 <clandmeter> i want to see if we get the same issue
2021-01-16 14:20:52 <ikke> https://ibb.co/KqPHHNZ
2021-01-16 14:20:55 <ikke> from my console history
2021-01-16 14:21:32 <clandmeter> lets keep that around
2021-01-16 14:21:38 <clandmeter> in case we need to email packet
2021-01-16 14:21:57 <clandmeter> sorry, i will keep using the old name until i get my head around the new one :)
2021-01-16 14:22:16 <clandmeter> ok so i get the same issue
2021-01-16 14:22:30 <clandmeter> we need to report it
2021-01-16 14:22:52 <ikke> yea
2021-01-16 14:23:03 <ikke> THat was my plan, but want to verify with you that it's not something we did
2021-01-16 14:23:17 <ikke> Not that I found that very likely
2021-01-16 14:23:20 <clandmeter> ok, will you report it?
2021-01-16 14:23:33 <ikke> I'll try
2021-01-16 14:23:48 <clandmeter> I can also, but i need to run in 30 min
2021-01-16 14:23:56 <ikke> np, I'll do it
2021-01-16 14:24:02 <clandmeter> i normally first start a chat
2021-01-16 14:24:15 <clandmeter> but im not sure if thats still avaible. 
2021-01-16 14:24:39 <ikke> There is a community slack
2021-01-16 14:24:51 <clandmeter> i used the chat on the website
2021-01-16 14:24:55 <clandmeter> but i dont see it anymore
2021-01-16 14:25:13 <ikke> There is a 'contact us' button
2021-01-16 14:26:01 <clandmeter> Yes
2021-01-16 14:26:09 <clandmeter> Use that
2021-01-16 14:26:23 <clandmeter> I think they prefer email
2021-01-16 14:30:12 <ikke> ok, submitted something
2021-01-16 14:30:36 <clandmeter> ok cross fingers :)
2021-01-16 14:30:47 <clandmeter> at least it has no important data right?
2021-01-16 14:31:01 <ikke> not that I;m aware of
2021-01-16 14:31:52 <ikke> It's a very bare metal server :P-
2021-01-16 14:32:31 <ikke> But my plan is to get our servers upgraded to alpine 13
2021-01-16 14:32:32 <MY-R> another good reason to reboot servers from time to time :P 
2021-01-16 14:32:35 <ikke> MY-R: nod
2021-01-16 14:32:56 <ikke> our CI servers have little state, so I'm not that afraid to start with those
2021-01-16 14:33:24 <clandmeter>  its still a weird issue
2021-01-16 14:33:29 <clandmeter> i wonder how this happend
2021-01-16 14:33:43 <MY-R> remember old debian times when it was always lotery if server start or not because everyone was messing something around and didnt make notes :P
2021-01-16 14:34:05 <ikke> to alpine 3.13*
2021-01-16 14:34:12 <ikke> alpine 13 will take a while
2021-01-16 14:35:57 <ikke> apparently those are sata ssds
2021-01-16 14:36:02 <ikke> https://www.micron.com/products/ssd/product-lines/5200
2021-01-16 14:36:36 <ikke> https://media-www.micron.com/-/media/client/global/documents/products/technical-marketing-brief/5200_sed_tcg-e_tech_brief.pdf?la=zh-tw&rev=14cf6a54b6d54139a075ca8823df984c
2021-01-16 14:36:42 <ikke> self encrypting drives
2021-01-16 14:36:59 <clandmeter> yes its a sata
2021-01-16 14:37:05 <clandmeter> https://tweakers.net/pricewatch/1183355/micron-5200-max-480gb/specificaties/
2021-01-16 14:37:28 <clandmeter> whats funny is that this config was always weird
2021-01-16 14:37:33 <ikke> "self encrypting"
2021-01-16 14:37:46 <ikke> sounds like that happened
2021-01-16 14:37:55 <clandmeter> the cpu is an engineering sample
2021-01-16 14:38:02 <ikke> right, the 128 core one
2021-01-16 14:38:27 <clandmeter>  DXE--ACPI Initialization..Welcome to GRUB!                                                        
2021-01-16 14:38:27 <clandmeter> error: failure reading sector 0x0 from `hd2'.
2021-01-16 14:38:32 <ikke> yes, had that too
2021-01-16 14:38:56 <ikke> probably because the disks are still locked?
2021-01-16 14:39:03 <clandmeter> i reported it ones, but never got a reply
2021-01-16 14:39:15 <clandmeter> yes, but i wonder what the boot drive is
2021-01-16 14:39:24 <clandmeter> it should not be the 480 drives
2021-01-16 14:39:48 <ikke> the 240G nvme/
2021-01-16 14:39:51 <ikke> ?
2021-01-16 14:39:52 <clandmeter> but maybe grub is unhappy it cant read it
2021-01-16 14:40:13 <clandmeter> i think the nvme should be boot yes
2021-01-16 14:40:22 <clandmeter> but its been a long  time since we installed it
2021-01-16 14:40:53 <clandmeter> could be packet has some master password to unlock t he drive
2021-01-16 14:43:39 <clandmeter> the vague remembrance was about when i wanted to do a secure erase on another packet server before
2021-01-16 14:43:50 <ikke> aha, ok
2021-01-16 14:43:57 <clandmeter> when decommissioning 
2021-01-16 14:44:04 <clandmeter> so in the end i did it sw based
2021-01-16 14:45:48 <ikke> But we need some kind of solution for our ci hosts and faccesat2
2021-01-16 14:46:23 <clandmeter> faccewhat?
2021-01-16 14:46:28 <ikke> new syscall
2021-01-16 14:46:36 <ikke> added in the latest musl
2021-01-16 14:46:38 <clandmeter> google yields no results 
2021-01-16 14:46:50 <clandmeter> msybe stat?
2021-01-16 14:46:54 <ikke> https://lwn.net/Articles/820410/
2021-01-16 14:47:11 <ikke> https://github.com/moby/moby/pull/41353
2021-01-16 14:47:14 <clandmeter> double s :)
2021-01-16 14:47:42 <ikke> but apparently an up-to-date seccomp profile is not enough
2021-01-16 14:47:59 <ikke> libseccomp needs to be 2.4.8 or newer
2021-01-16 14:48:08 <ikke> (according to Hello71)
2021-01-16 14:48:33 <ikke> I tried running docker with the latest seccomp profile, but it was still denied
2021-01-16 14:48:40 <clandmeter> i guess we first wait for packet to responde?
2021-01-16 14:48:50 <ikke> I guess so
2021-01-16 14:49:24 <clandmeter> ok i will check back later, need to run now. or ping me tomorrow morning
2021-01-16 14:49:46 <ikke> o.
2021-01-16 14:50:03 <ikke> I can setup a temporary (smaller) host somewhere to hande x86_64 ci?
2021-01-16 14:50:58 <clandmeter> sure
2021-01-16 14:51:04 <clandmeter> packet should be ok
2021-01-16 14:51:16 <clandmeter> maybe remove the temp servers
2021-01-16 14:51:24 <clandmeter> err
2021-01-16 14:51:31 <ikke> linode?
2021-01-16 14:51:40 <clandmeter> linode ys :)
2021-01-16 14:52:41 <ikke> mps: do we still need the mailman server?
2021-01-16 14:53:56 <clandmeter> ikke: lets keep them
2021-01-16 14:54:03 <clandmeter> we can run the ci for x time
2021-01-16 14:54:10 <clandmeter> decide later what to kill
2021-01-16 14:54:19 <clandmeter> its charged per hour iirc
2021-01-16 14:54:37 <clandmeter> i have some other stuff also 
2021-01-16 14:54:41 <clandmeter> i will clean it up
2021-01-16 14:54:44 <clandmeter> was some test
2021-01-16 14:58:18 <mps> ikke: could you wait half an hour, I'm not at home now but would like to make backup. then it could be removed
2021-01-16 14:59:23 <ikke> mps: sure
2021-01-16 15:29:39 <mps> ikke: ready. you can remove it
2021-01-16 15:29:43 <ikke> ok
2021-01-16 15:56:34 <ikke> chatting with equinix support atm
2021-01-16 16:06:06 <ikke> clandmeter: so something caused the disks to be encrypted, but not clue what. We are most likely going to have to reinstall that server
2021-01-16 16:09:27 <ikke> clandmeter: will reinstall the server
2021-01-16 16:15:17 <clandmeter> Wow
2021-01-16 16:15:25 <clandmeter> That's crazy
2021-01-16 16:15:27 <ikke> yes
2021-01-16 16:15:48 <clandmeter> Can it be us?
2021-01-16 16:16:02 <ikke> I have no clue
2021-01-16 16:16:44 <ikke> What could have set the encryption password?
2021-01-16 16:38:23 <ikke> clandmeter: ok, the locked disks cause issues, even for reinstalling, they recommend that we provision a new instance
2021-01-16 16:42:25 <MY-R> maybe it isnt encryption password but drive is locked, something must have set password/lock flag directly to disk or in uefi/bios
2021-01-16 16:45:25 <ikke> Sounds like it
2021-01-16 17:23:59 <ikke> clandmeter: equinix rescue os is now alpine 3.12 :D
2021-01-16 17:39:52 <ikke> yeah, just rub it in
2021-01-16 17:56:34 <ikke> Anyone experience with LACP bonding? Previously we didn't have any special bonding setup, but now when I setup bonding, I have packetloss
2021-01-16 17:59:47 <ikke> ok, bond-mode 802.3ad was enough
2021-01-16 19:03:55 <ikke> clandmeter: I've re-installed usa7, but somehow cannot seem to get it to boot from disks
2021-01-16 19:04:58 <ikke> It keeps going to ipxe, which seems to fail, but it keeps overwriting it's own output
2021-01-16 19:23:07 <ikke> ok, progress
2021-01-16 20:07:49 <ikke> I'm seeing "Welcome to grub" now, but nothing much else :(
2021-01-16 20:35:39 <ikke> sigh
2021-01-16 20:36:07 <ikke> takes ages to reboot every time
2021-01-16 20:37:50 <mps> sorry i can't help
2021-01-16 20:37:58 <ikke> apprecies the gesture
2021-01-16 20:38:03 <ikke> trying extlinux
2021-01-16 20:38:17 <ikke> appreciate*
2021-01-16 20:38:36 <mps> is it uefi or old bios
2021-01-16 20:38:50 <ikke> Seems right no legacy bios
2021-01-16 20:38:53 <ikke> right now*
2021-01-16 20:39:08 <ikke> I guess, not sure
2021-01-16 20:40:15 <mps> few days ago I made script to install syslinux on uefi boxes, tested it on macbook and older lenovo notebook
2021-01-16 20:40:54 <mps> if you think it could help I can paste it
2021-01-16 20:42:38 <MY-R> ikke: maybe you missed something in /etc/default/grub or you didnt add 'nvme' to mkinitfs dunno :\
2021-01-16 20:43:02 <ikke> MY-R: probably something like that
2021-01-16 20:43:16 <ikke> But hard to see the forest throug the trees
2021-01-16 20:43:41 <mps> if the module is missing it should at least try to start kernel and show first few messages
2021-01-16 20:43:44 <ikke> MY-R: THat _is_ something I missed from the old setup
2021-01-16 20:43:53 <ikke> unless serial is broken as well
2021-01-16 20:44:29 <mps> if you got boot loader menu it is not probably
2021-01-16 20:44:42 <ikke> no, I didn't see the menu
2021-01-16 20:44:44 <ikke> just the header
2021-01-16 20:44:51 <mps> ah
2021-01-16 20:45:55 <mps> but even "Welcome to grub" says that grub sent this over serial
2021-01-16 20:46:09 <ikke> hm
2021-01-16 20:47:00 <MY-R> no documentation how things are working on packet.net?
2021-01-16 20:47:24 <ikke> https://metal.equinix.com/developers/docs/guides/
2021-01-16 20:48:02 <ikke> sadly they don't have alpine images, so I'm using an alpine rescue OS to install alpine
2021-01-16 20:48:58 <ikke> There is another server there that I installed in a similar way, so maybe copy things from there
2021-01-16 20:49:08 <maxice8> Was the CI fixed ?
2021-01-16 20:49:37 <ikke> no, guess what I'm working on
2021-01-16 20:50:04 <ikke> (obviously yak shaving)
2021-01-16 20:50:25 <maxice8> huh
2021-01-16 20:50:49 <ikke> I decided to be couragious and tried to upgrade our x86_64 CI host to alpine 3.13
2021-01-16 20:50:59 <ikke> that kinda went south
2021-01-16 20:51:25 <maxice8> oof
2021-01-16 20:51:44 <maxice8> Doesn't that mean we are shipping a potentially broken docker setup on 3.13 or is it just related to our CI ?
2021-01-16 20:51:58 <ikke> it's a seccomp issue
2021-01-16 20:52:12 <ikke> faccessat2 is blocked, but shouldn't
2021-01-16 20:52:39 <ikke> the CI hosts were not on 3.13
2021-01-16 20:52:46 <ikke> and most  still are not
2021-01-16 20:52:52 <ikke> well, none of them is even atm
2021-01-16 21:01:01 <ikke> the other server is running in efi mode
2021-01-16 21:01:07 <ikke> so let me check the bios
2021-01-16 21:05:44 <ikke> switching to uefi now
2021-01-16 21:35:17 <clandmeter> hi
2021-01-16 21:35:20 <ikke> hey
2021-01-16 21:35:29 <clandmeter> ikke: i think you need uefi 
2021-01-16 21:35:34 <ikke> clandmeter: I tried uefi
2021-01-16 21:35:40 <ikke> but then it does not find any boot devices
2021-01-16 21:35:47 <ikke> and rescue OS is not working either
2021-01-16 21:36:12 <clandmeter> rescue is 3.12 you mentioned?
2021-01-16 21:36:15 <ikke> yes
2021-01-16 21:36:32 <clandmeter> so they recently upgraded
2021-01-16 21:36:35 <ikke> yup
2021-01-16 21:37:20 <clandmeter> how do you switch to eufi and back?
2021-01-16 21:37:22 <ikke> I'm now switching back to bios and trying extlinux
2021-01-16 21:37:27 <clandmeter> enter the bios?
2021-01-16 21:37:28 <ikke> yes
2021-01-16 21:37:56 <clandmeter> i think uefi should work in rescue os, if not its a bug.
2021-01-16 21:38:16 <ikke> It's not even managing to get there
2021-01-16 21:39:19 <ikke> clandmeter: It's now booting from the hard drive, but stopping at "Welcome to GRUB!"
2021-01-16 21:39:27 <ikke> any idea?
2021-01-16 21:39:44 <clandmeter> are you using the correct serial?
2021-01-16 21:39:54 <ikke> I copied the settings from the previous installation
2021-01-16 21:40:00 <ikke> (I still had the output from update-conf)
2021-01-16 21:40:18 <clandmeter> but this is another server?
2021-01-16 21:40:22 <ikke> yes
2021-01-16 21:40:58 <clandmeter> there was a page mentioning the right settings
2021-01-16 21:41:08 <clandmeter> but the rebranding does not help saerching
2021-01-16 21:41:54 <ikke> Rebooting into rescue os now
2021-01-16 21:42:27 <clandmeter> at least its not an arm 
2021-01-16 21:42:42 <ikke> is that even worse?
2021-01-16 21:42:55 <ikke> (hard to imagine)
2021-01-16 21:43:07 <clandmeter> yes it takes minutes to boot
2021-01-16 21:43:19 <clandmeter> and fails every x times
2021-01-16 21:43:21 <ikke> well, this also takes minutes to boot
2021-01-16 21:43:41 <clandmeter> also in bios mode?
2021-01-16 21:43:50 <ikke> yes
2021-01-16 21:43:57 <ikke> in uefi mode it fails quite fast
2021-01-16 21:44:31 <clandmeter> so we moved from 128 cores to 24?
2021-01-16 21:44:45 <ikke> I guess 48?
2021-01-16 21:45:00 <clandmeter> 1 x AMD EPYC 7402P 24-Core Processor @ 2.8GHz
2021-01-16 21:45:40 <clandmeter> if it boots into rescue, you can check which serial its using
2021-01-16 21:45:49 <ikke> through dmesg?
2021-01-16 21:46:02 <ikke> was thinking about how to find out
2021-01-16 21:46:09 <ikke> /proc/cmdline is not mentioning it
2021-01-16 21:46:26 <clandmeter> the module is not loaded i think?
2021-01-16 21:46:39 <ikke> what module?
2021-01-16 21:46:47 <clandmeter> oh cmdline 
2021-01-16 21:46:53 <clandmeter> sorry i was mixing config.tz
2021-01-16 21:46:55 <clandmeter> gz
2021-01-16 21:46:56 <ikke> he
2021-01-16 21:47:20 <ikke> ok, booted
2021-01-16 21:47:24 <clandmeter> does it boot in rescue?
2021-01-16 21:47:38 <clandmeter> cmdline should show afaik
2021-01-16 21:47:39 <ikke> yes
2021-01-16 21:47:58 <ikke> oh, it was truncated
2021-01-16 21:48:35 <ikke> console=tty0 console=ttyS1,115200]
2021-01-16 21:48:58 <ikke> nproc -> 48
2021-01-16 21:49:44 <clandmeter> SMT
2021-01-16 21:49:57 <ikke> yes
2021-01-16 21:50:06 <ikke> but wasn't 128 smt as well?
2021-01-16 21:50:13 <clandmeter> yes
2021-01-16 21:50:21 <clandmeter> i htink it was also single socket
2021-01-16 21:50:22 <ikke> so 64 -> 48 cores
2021-01-16 21:50:28 <ikke> sorry
2021-01-16 21:50:31 <ikke> 64 -> 24
2021-01-16 21:50:36 <clandmeter> yup
2021-01-16 21:50:46 <clandmeter> but hey, whos complaining :)
2021-01-16 21:50:49 <ikke> :)
2021-01-16 21:51:06 <ikke> the chat support was very nice btw
2021-01-16 21:51:06 <clandmeter> the serial settings are different then yours?
2021-01-16 21:52:05 <mps> console=ttyS1
2021-01-16 21:53:13 <ikke> https://tpaste.us/E1a5 this is what I had
2021-01-16 21:53:19 <ikke> But I guess it should be S1
2021-01-16 21:53:49 <mps> yes -unit=1
2021-01-16 21:54:16 <ikke> mps: ah, thanks
2021-01-16 21:54:50 <mps> also for kernel cmdline console=ttyS1
2021-01-16 21:54:52 <ikke> yes
2021-01-16 21:54:57 <clandmeter> we should report the rescue os does not boot in efi mode.
2021-01-16 21:55:22 <ikke> clandmeter: it does not even get to booting via ipxe
2021-01-16 21:57:03 <clandmeter> is the server still supermicro?
2021-01-16 21:58:33 <ikke> product: PowerEdge R6515 (SKU=NotProvided;ModelName=PowerEdge R6515)
2021-01-16 21:58:50 <clandmeter> ok
2021-01-16 21:59:22 <ikke> Ok, adjusted the serial device
2021-01-16 21:59:24 <ikke> will try to reboot now
2021-01-16 21:59:32 <ikke> (ran grub-mkconfig -o ...)
2021-01-16 21:59:50 <clandmeter> grub is so much fun
2021-01-16 22:00:27 <mps> it even have its script lang :)
2021-01-16 22:00:28 <ikke> heh
2021-01-16 22:00:35 <ikke> clandmeter: we could switch to extlinux?
2021-01-16 22:01:08 <clandmeter> as long as it boots i dont care that much
2021-01-16 22:01:12 <clandmeter> at least it supports efi
2021-01-16 22:01:21 <ikke> i'll reboot first
2021-01-16 22:01:35 <clandmeter> i know for our servers we mostly use efi now, or things like gpu's have issues.
2021-01-16 22:01:43 <mps> clandmeter: syslinux also support efi
2021-01-16 22:02:02 <clandmeter> only on x86 i think
2021-01-16 22:02:46 <mps> didn't tried on arm
2021-01-16 22:03:07 <clandmeter> https://wiki.syslinux.org/wiki/index.php?title=Install#UEFI
2021-01-16 22:05:15 <ikke> Booting from Hard drive C:
2021-01-16 22:05:18 <ikke> GRUB loading.
2021-01-16 22:05:20 <ikke> Welcome to GRUB!
2021-01-16 22:05:24 <ikke> and then a lot of silence
2021-01-16 22:05:57 <ikke> ok, extlinux it is
2021-01-16 22:06:03 <ikke> hoping that goes better
2021-01-16 22:06:41 <clandmeter> i had many of such fights with those servers.
2021-01-16 22:06:45 <ikke> Attempt <I lost count>
2021-01-16 22:06:49 <clandmeter> for sure the aarch64
2021-01-16 22:07:07 <clandmeter> and tianocore takes forever to load.
2021-01-16 22:07:14 <clandmeter> like the matrix on repeat 
2021-01-16 22:09:02 <ikke> Maybe to do with ifupdown-ng, but I had to set bond-mode now as well
2021-01-16 22:09:09 <ikke> 802.3a
2021-01-16 22:09:31 <ikke> Otherwise quite a bit of packetloss
2021-01-16 22:09:57 <clandmeter> i think i always set bonding mode
2021-01-16 22:10:07 <ikke> ah ok
2021-01-16 22:10:11 <ikke> I never noticed it
2021-01-16 22:10:11 <clandmeter> when loading the module
2021-01-16 22:10:15 <ikke> ah
2021-01-16 22:10:25 <clandmeter> iirc
2021-01-16 22:10:31 <ikke> I noticed it during setup-alpine in the recue os
2021-01-16 22:11:50 <ikke> This is the last attempt for today
2021-01-16 22:12:42 <ikke> huh
2021-01-16 22:12:54 <ikke> now it suddenly switched to sdb instead of sda
2021-01-16 22:13:11 <ikke> (and no sda)
2021-01-16 22:13:34 <clandmeter> ok i will also call it a day, i can check it tomorrow if you want.
2021-01-16 22:14:08 <clandmeter> thanks for spending the time already.
2021-01-16 22:14:21 <clandmeter> its frustrating...
2021-01-16 22:14:39 <clandmeter> from experience :)
2021-01-16 22:14:43 <ikke> hehe
2021-01-16 22:17:10 <ikke> Ok, I think i've installed extlinux now
2021-01-16 22:18:23 <ikke> added console=ttyS1 to the cmdline
2021-01-16 22:21:43 <ikke> hmm, grub is still installed :-/
2021-01-16 22:35:51 <ikke> Progress
2021-01-16 22:35:58 <ikke> mount: mounting UUID=adbfa266-4535-44a8-b5a5-92a2539ba8d4 on /sysroot failed: No such file or directory
2021-01-16 22:37:12 <ikke> so I guess now it's a matter of getting the modules in
2021-01-16 22:37:18 <ikke> but that's for tomorrow
2021-01-17 07:50:26 <ikke> clandmeter: it's now booted in an emergency shell becuase it cannot find the root partition. I can manually active the lvm volume group though
2021-01-17 08:28:41 <clandmeter> Morning
2021-01-17 08:28:52 <ikke> hi
2021-01-17 08:29:02 <clandmeter> Did you fix it?
2021-01-17 08:29:04 <ikke> no
2021-01-17 08:29:14 <ikke> Not sure what I'm missing
2021-01-17 08:29:19 <ikke> but lvm does not get activates somehow
2021-01-17 08:29:44 <clandmeter> Nvme is enabled?
2021-01-17 08:33:29 <ikke> I've added it to the mkninitfs features
2021-01-17 08:34:08 <ikke> in the emergency shell, I can do lvm vgscan; lvm vgchange --sysinit --activate y, and then the volumes are available
2021-01-17 08:36:11 <clandmeter> Can you boot with debug?
2021-01-17 08:42:28 <ikke> I have time again later
2021-01-17 08:45:56 <mps> I have good experience with lvm, i.e. never use it :)
2021-01-17 11:03:34 <ikke> clandmeter: booted with quiet removed and debug aded
2021-01-17 11:05:01 <mps> and I have to repeat that 'quiet' should be removed by default from alpine kernel/bootloader cmdline
2021-01-17 11:06:17 <ikke> https://tpaste.us/0Evz 
2021-01-17 11:07:50 <mps> dev mapper not started in initramfs?
2021-01-17 11:08:26 <ikke> sounds like it, but what should start it?
2021-01-17 11:08:52 <mps> I really have no idea
2021-01-17 11:09:31 <mps> ask on -devel or -linux maybe someone there know
2021-01-17 11:09:57 <ikke> I did
2021-01-17 11:10:01 <ikke> no responses yet
2021-01-17 11:10:32 <mps> btw, why is lvm on these machines
2021-01-17 11:11:05 <ikke> modules=sd-mod,usb-storage,ext4
2021-01-17 11:11:13 <ikke> We use lvm by default everywhere
2021-01-17 11:11:52 <ikke> Makes it easier to deal with partitions
2021-01-17 11:12:02 <mps> hmm
2021-01-17 11:12:38 <ikke> and it works normally just fine
2021-01-17 11:12:52 <clandmeter> hi
2021-01-17 11:12:55 <ikke> hey
2021-01-17 11:13:00 <clandmeter> i think i know what should start it
2021-01-17 11:13:12 <ikke> please enlighten me
2021-01-17 11:13:28 <clandmeter> we have a small c based tool, i dont remember the name
2021-01-17 11:13:33 <ikke> ifplugd?
2021-01-17 11:13:42 <ikke> or something like that
2021-01-17 11:14:03 <ikke> nlplug-findfs
2021-01-17 11:14:08 <clandmeter> yes 
2021-01-17 11:14:12 <clandmeter> that should be it
2021-01-17 11:15:14 <clandmeter> if you set debug, it will spit a lot of data
2021-01-17 11:16:09 <ikke> did not see anythign
2021-01-17 11:16:44 <ikke> I'll try to run it manuall
2021-01-17 11:16:47 <ikke> manually*
2021-01-17 11:18:00 <clandmeter> did you set debug_init?
2021-01-17 11:19:34 <ikke> ah, no, just debug
2021-01-17 11:19:54 <clandmeter> prepare for lots of debug info
2021-01-17 11:20:11 <ikke> basically dmesg output directly to the console
2021-01-17 11:22:49 <ikke> running nlplug-findfs -p /sbin/mdev -d now
2021-01-17 11:23:10 <ikke> taking a bit of time
2021-01-17 11:25:56 <ikke> exit due to timeout
2021-01-17 11:26:11 <clandmeter> how did you install alpine?
2021-01-17 11:26:16 <clandmeter> via rescue os?
2021-01-17 11:26:31 <ikke> yes
2021-01-17 11:26:53 <ikke> But I also upgraded to 3.13
2021-01-17 11:27:32 <clandmeter> so you boot 3.12 and upgrade then reboot
2021-01-17 11:27:40 <ikke> no
2021-01-17 11:27:48 <clandmeter> boot 3.12 install, upgrade, reboot?
2021-01-17 11:28:11 <ikke> boot rescue os; setup-alpine, skip disk
2021-01-17 11:28:14 <ikke> upgrade
2021-01-17 11:28:26 <ikke> setup-disk <..> 
2021-01-17 11:29:23 <clandmeter> did you run update-conf?
2021-01-17 11:29:44 <ikke> hmm,n o
2021-01-17 11:30:11 <clandmeter> maybe there is something that packet has changed in its rescue os that is biting you
2021-01-17 11:31:19 <clandmeter> whats in mkinitfs.conf?
2021-01-17 11:31:34 <clandmeter> which options are enabled?
2021-01-17 11:31:58 <clandmeter> rescue os is network based, so its missing disk based modules (probably)
2021-01-17 11:56:46 <ikke> /mnt/etc/mkinitfs # cat mkinitfs.conf
2021-01-17 11:56:46 <ikke> features="ata base ide scsi usb virtio ext4 lvm nvme"
2021-01-17 12:01:02 <clandmeter> looks good
2021-01-17 12:01:46 <clandmeter> is the bootable flag set?, not sure that makes a difference.
2021-01-17 12:01:57 <ikke> Otherwise it would not even boot iirc
2021-01-17 12:02:05 <ikke> it already gets to initramfs
2021-01-17 12:02:55 <clandmeter> did you get the debug log?
2021-01-17 12:03:05 <ikke> from nlplug?
2021-01-17 12:03:46 <ikke> I want to write the output to a file
2021-01-17 12:03:54 <ikke> looks like outputting to a console delays it too much
2021-01-17 12:07:09 <ikke> hmm, seems it does not output to stdout / stderr
2021-01-17 12:10:01 <clandmeter> https://gitlab.alpinelinux.org/alpine/mkinitfs/-/blob/master/initramfs-init.in
2021-01-17 12:10:36 <ikke> yes, I'm looking at that
2021-01-17 12:10:49 <ikke> https://gitlab.alpinelinux.org/alpine/mkinitfs/-/blob/master/initramfs-init.in#L507
2021-01-17 12:11:05 <ikke> oh, I did not pass a root
2021-01-17 12:11:35 <ikke> hmmmm
2021-01-17 12:11:52 <ikke> let me reboot again
2021-01-17 12:19:14 <ikke> clandmeter: If I manually run it, it seems to work :-.
2021-01-17 12:19:16 <ikke> :-/
2021-01-17 12:20:02 <ikke> clandmeter: https://tpaste.us/axqE
2021-01-17 12:20:05 <clandmeter> does it boot without lvm?
2021-01-17 12:20:21 <ikke> I have not tried
2021-01-17 12:23:01 <ikke> hmm, I found a type
2021-01-17 12:23:03 <ikke> typo
2021-01-17 12:23:09 <ikke> /dev/mapper/vg0-lv_root on /sysroot failed: No such file or directory
2021-01-17 12:23:16 <ikke> nlplug-findfs -p /sbin/mdev /dev/mapper/vg0_lvroot
2021-01-17 12:32:25 <ikke> sight
2021-01-17 12:32:30 <ikke> sigh*
2021-01-17 12:33:07 <ikke> sounds almost like a timing issue or something like that
2021-01-17 12:37:40 <ikke> Ok, i've now added debug_init to the cmdline
2021-01-17 12:43:22 <ikke> + nlplug-findfs -p /sbin/mdev -d /dev/mapper/vg0_lvroot
2021-01-17 12:47:26 <ikke> clandmeter: https://tpaste.us/qgJ5 
2021-01-17 12:52:38 <ikke> send time I execute the exact same command, and it works :(
2021-01-17 12:52:43 <ikke> send/second
2021-01-17 12:54:49 <ikke> lesigh
2021-01-17 12:55:37 <ikke> clandmeter: can you please take a look?
2021-01-17 13:35:22 <ikke> I'm trying a hack now
2021-01-17 13:35:42 <ikke> add an additional sleep before nlplug-finfs
2021-01-17 13:35:45 <ikke> findfs
2021-01-17 14:49:01 <ikke> equinix console is now unavailable :/
2021-01-17 16:31:44 <ikke> I decided to do a reinstall
2021-01-17 16:31:47 <ikke> more cleanly
2021-01-17 16:31:56 <ikke> And I also noticed that I selected the 480G drivers
2021-01-17 16:32:02 <ikke> before
2021-01-17 16:45:50 <ikke> w00t
2021-01-17 16:45:53 <ikke> w00t
2021-01-17 16:45:54 <ikke> w00t
2021-01-17 17:59:48 <ikke> clandmeter: so confirmed, it seems to be an issue with alpine 3.13
2021-01-17 17:59:52 <ikke> installing 3.12 -> works
2021-01-17 17:59:58 <ikke> ugprading to 3.13 -> broken
2021-01-17 18:00:02 <ikke> downgrading to 3.12 -> works
2021-01-17 18:17:43 <clandmeter> Hmm
2021-01-17 18:18:06 <clandmeter> So there is a change in mkinitfs repo?
2021-01-17 18:18:11 <ikke> not sure
2021-01-17 18:18:13 <ikke> maybe kernel?
2021-01-17 18:18:36 <ikke> it's nlplug-findfs that somehow does not find the fs
2021-01-17 18:18:41 <clandmeter> Try edge kernel and reboot :)
2021-01-17 18:18:51 <ikke> so I don't think it's mkinitfs-init itself
2021-01-17 18:18:56 <clandmeter> Or just 3.13 kernel
2021-01-17 18:19:08 <ikke> Yeah, can try that
2021-01-17 18:28:38 <ikke> clandmeter: 3.13 kernel does not work
2021-01-17 18:32:55 <clandmeter> Ah ok
2021-01-17 18:33:17 <clandmeter> Lvm related?
2021-01-17 18:33:41 <ikke> yes
2021-01-17 18:33:50 <ikke> it worked when it didn't use lvm
2021-01-17 18:34:00 <clandmeter> Interesting
2021-01-17 18:34:08 <ikke> but I guess that's because it does not relies on nlplug-findfs
2021-01-17 18:34:20 <clandmeter> Good work, at least we know where to look now
2021-01-17 18:34:53 <clandmeter> I guess more setups are broken
2021-01-17 19:21:38 <clandmeter> ikke: i wonder what is causing this. 
2021-01-17 19:21:48 <clandmeter> if iuse qemu it works just fine
2021-01-17 19:22:26 <ikke> clandmeter: If I compare the output of nlplug-findfs, for some reason there are not events for the block devices
2021-01-17 19:22:30 <ikke> /dev/sd*
2021-01-17 19:22:44 <ikke> but if I run it manually afterwards, it's present
2021-01-17 19:23:12 <clandmeter> yes i also noticed missing block devices in your paste
2021-01-17 19:23:48 <ikke> nlplug-findfs: uevent: action='add' subsystem='block' devname='sdc2' devpath='/devices/pci0000:00/0000:00:03.2/0000:01:00.0/host12/port-12:0/end_device-12:0/target12:0:0/12:0:0:0/block/sdc/sdc2'                                        
2021-01-17 19:23:51 <ikke> this one
2021-01-17 19:40:09 <ikke> clandmeter: I'm reinstalling x86 as well
2021-01-17 19:40:23 <clandmeter> reinstalling?
2021-01-17 19:40:34 <ikke> Wanted to upgrade it to 3.12
2021-01-17 19:41:01 <ikke> https://tpaste.us/K5aP 
2021-01-17 19:41:18 <ikke> I messed something up when I installed it previously with raid
2021-01-17 19:41:37 <ikke> (and maybe we did some lvm raid-like setup as well)
2021-01-17 20:32:01 <ikke> clandmeter: testing 3.13 kernel on nld7 now (with efi, testing if it has the same issue)
2021-01-17 20:34:17 <ikke> seems like the same issue
2021-01-17 20:34:21 <ikke> nope
2021-01-17 20:34:25 <ikke> (was impatient)
2021-01-17 20:42:28 <ikke> so nld7 is now 3.13
2021-01-17 20:42:40 <ikke> so seems to only affect legacy boot
2021-01-17 20:55:51 <ikke> mps: lets continue here
2021-01-17 20:56:43 <mps> I don't have to add anything more and don't want to annoy you when you are working on servers
2021-01-17 20:56:53 <ikke> np
2021-01-17 21:01:41 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/issues/12325
2021-01-17 21:03:05 <mps> just pushed linux-edge-5.10.8 :)
2021-01-17 21:03:50 <ikke> heh
2021-01-17 21:04:40 <mps> but I don't know could it fix this problem
2021-01-17 21:05:25 <ikke> Do you think it would be worth to report this upstream?
2021-01-17 21:05:47 <ikke> Not sure what information to provide exactly
2021-01-17 21:06:00 <mps> what? kernel or nl....
2021-01-17 21:06:06 <mps> find
2021-01-17 21:06:46 <mps> I don't have box with nvme where I could test this
2021-01-17 21:06:55 <ikke> not sure if this is related to nvme
2021-01-17 21:07:15 <mps> also I'm not
2021-01-17 21:07:29 <mps> though 5.10.8 have one nvme fix
2021-01-17 21:07:47 <mps> let me look again
2021-01-17 21:08:23 <mps> https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.10.8
2021-01-17 21:08:36 <mps> nvme-tcp: Fix possible race of io_work and direct send
2021-01-17 21:09:06 <ikke> yes, no clue if it is related at all
2021-01-17 21:09:07 <mps> 'tcp' for nvme? don't understand this
2021-01-17 21:09:23 <ikke> nvme over network?
2021-01-17 21:10:27 <mps> who knows, maybe nvme uses tcp to communicate with devices. never looked at this
2021-01-17 21:11:56 <mps> https://blogs.oracle.com/linux/nvme-over-tcp
2021-01-17 21:12:07 <mps> interesting
2021-01-17 21:47:54 <minimal> yupe its been around for a little while, basically and alternative to iSCSI over FibreChannel/Ethernet/whatever
2021-01-18 08:15:08 <clandmeter> mps: its still rather new, i had a talk with them last year in the beginning of the pandemic.
2021-01-18 08:15:43 <clandmeter> ikke: im bumping into a few lua pkg checksum errors
2021-01-18 08:33:21 <mps> 5.11-rc4 have this changelog 'nvme: don't intialize hwmon for discovery controllers'
2021-01-18 08:33:55 <mps> maybe (probably?)  it will be backported to 5.10.9
2021-01-18 10:21:37 <ikke> clandmeter: on what url?
2021-01-18 11:16:18 <ikke> clandmeter: I could try to upgrade usa7 to 3.13 but pin the kernel
2021-01-18 14:57:15 <clandmeter> ikke: ok
2021-01-18 14:57:25 <clandmeter> do we know what is the actual issue?
2021-01-18 14:58:16 <clandmeter> i already cleared the pkgs that have checksum errors
2021-01-18 16:15:11 <ikke> clandmeter: No, I have no idea what the actual issue is, or how to track it down
2021-01-18 16:15:32 <ikke> it looks some kind of race condition, but adding sleep before nlplug-findfs does not appear to help
2021-01-19 12:55:55 <ncopa> can someone from infra team help set up PureTryOut https://gitlab.alpinelinux.org/PureTryOut to have git push access to testing and community? Thanks!
2021-01-19 13:11:03 <ikke> I'll do it later today
2021-01-19 13:18:11 <TBK[m]> s390x buildozer (edge/3.13/3.12) do not appear to work at all and the same applies to ppc64le edge.
2021-01-19 17:13:14 <Ariadne> i'm confuzzled on something involving our mailing lists
2021-01-19 17:13:43 <Ariadne> i am sending messages to our mailing list, with what appear to be valid dkim-signature, but the archives do not show a dkim-signature under 'details'
2021-01-19 17:14:09 <ikke> I think Cogitri mentioned something similar\
2021-01-19 17:14:11 <Ariadne> i am, however, operating my mailer on an alpine s390x VM, which is big-endian, so it may be that i am generating broken signatures
2021-01-19 17:14:57 <ikke> yes, he mentioned the same
2021-01-19 17:16:23 <Cogitri> Yes, my DKIM signature apparently goes to /dev/null on the Ml
2021-01-19 17:16:35 <Cogitri> But seems to work just fine on other things apparently
2021-01-19 17:16:57 <ikke> ddevault: ^
2021-01-19 17:16:58 <Ariadne> i have no idea if my signatures are working or not :D
2021-01-19 17:17:03 <Cogitri> And some random dkim validators on the net say it's fine so yeah 🤷‍♂️
2021-01-19 17:17:24 <ddevault> it relies on the mail server to add an authentication header
2021-01-19 17:18:06 <ikke> How would we do that with postfix
2021-01-19 17:18:30 <ddevault> run it through a DKIM verification milter
2021-01-19 17:18:37 <ddevault> in sr.ht production we use go-msgauth
2021-01-19 17:18:42 <ddevault> https://github.com/emersion/go-msgauth
2021-01-19 17:19:02 <Ariadne> i presently use opendkim, but opendkim seems not so great.  how does go-msgauth compare?
2021-01-19 17:19:15 <ddevault> we wrote go-msgauth for the express purpose of replacing opendkim
2021-01-19 17:19:20 <Ariadne> haha
2021-01-19 17:19:31 <ddevault> here's our alpine package https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/item/sr.ht/go-msgauth
2021-01-19 17:19:39 <ddevault> never cleaned it up for upstreaming into aports, though
2021-01-19 17:19:47 <ddevault> see the confd file, which is where all the configuration occurs
2021-01-19 17:20:18 <ddevault> note, we use this both for verifying signatures for incoming emails and signing outgoing emails, if you don't want to do the latter it may require some tweaking
2021-01-19 17:20:38 <Ariadne> query: if dkim_sign_domains= is left blank, will it sign for any domain?
2021-01-19 17:20:44 <ddevault> I think so
2021-01-19 17:20:46 <ddevault> err, no
2021-01-19 17:20:52 <ddevault> it won't sign anything, but it will verify signatures
2021-01-19 17:21:07 <ddevault> I think - this package hasn't been tested outside of our use-case
2021-01-19 17:21:09 <Ariadne> would be nice to have an sign anything outbound
2021-01-19 17:21:51 <Ariadne> i've been moving my email usecases onto my own infrastructure now that i can afford to get a /24 toasted if it goes to shit
2021-01-19 17:22:08 <ddevault> IPv4 reputation issues with email are overstated
2021-01-19 17:22:17 <Ariadne> hmm, true :)
2021-01-19 17:23:26 <Ariadne> looks like a nice software
2021-01-19 17:26:01 <Ariadne> ddevault: would you have any objection to my cleanng up your APKBUILD and including it into alpine testing?
2021-01-19 17:26:10 <ddevault> no objection
2021-01-19 17:26:39 <ddevault> if you make breaking changes, please let me know so that I can migrate if it eventually makes its way into community
2021-01-19 17:26:49 <Ariadne> cool, i'll work on it this weekend
2021-01-19 17:27:17 <Ariadne> i don't envision making any breaking changes :)
2021-01-22 00:48:03 <MY-R> just in case I will leave it this here:
2021-01-22 00:48:10 <MY-R> "Scheduled - While performing a scheduled maintenance to replace a faulty switch, we ran into an issue that caused a brief outage in the EU-Central (Frankfurt) data center. At this time all services are restored. We have scheduled an emergency maintenance on Thursday, January 21, 2021 from 23:00 UTC until January 22, 2021 02:00 UTC to complete the switch replacement. While we do not expect
2021-01-22 00:48:13 <MY-R> any downtime, there may be a period of brief packet loss or latency."
2021-01-23 15:02:39 <ikke> We need to cleanup s390x builder, root is full
2021-01-23 15:02:56 <ikke> /var I mean
2021-01-23 15:15:11 <ikke> I've deleted edge distfiles older then 30 days for now
2021-01-23 15:15:14 <ikke> 10G free
2021-01-24 14:17:23 <ikke> clandmeter: fyi, usa7 (x86_64 CI host) is now running alpine 3.13 with an older kernel
2021-01-24 14:27:48 <ikke> gitlab-runner-aarch64 is running 3.13 as well now
2021-01-24 14:34:41 <ikke> and gitlab-runner-armv7
2021-01-24 14:38:06 <ikke> those are VMs, so upgrade is fairly trivial
2021-01-24 14:38:29 <ikke> I guess ppc64le and s390x are more tricky
2021-01-24 21:14:35 <clandmeter> ikke: ok nice. 
2021-01-24 21:15:24 <ikke> clandmeter: any idea how we should to ppc64le / s390x? We do not really have an oob console to fall back to
2021-01-24 21:15:45 <ikke> In the bast I guess we just did it
2021-01-24 21:15:51 <clandmeter> good question
2021-01-24 21:16:07 <clandmeter> we should probably first check if we have contact 
2021-01-24 21:16:19 <ikke> nod
2021-01-24 21:16:27 <clandmeter> and mention the upgrade plan
2021-01-24 21:16:46 <clandmeter> ive not talked to anyone much
2021-01-24 21:17:16 <clandmeter> i guess we can ask tmhoang?
2021-01-24 21:18:48 <ikke> I guess
2021-01-25 14:28:10 <Cogitri> ikke: Could you pull appstream-generator (both the image and the compose) and restart it?
2021-01-25 16:20:03 <ikke> Cogitri: done
2021-01-25 16:21:38 <Cogitri> Thanks
2021-01-25 16:31:28 <mcrute> hey ikke is there a way I can manually rebuild and re-deploy the website without pushing a fake bump commit? the AMI page really needs rebuilt
2021-01-25 16:32:11 <ikke> mcrute: not right now
2021-01-25 16:32:24 <ikke> I can see if I can trigger the build script
2021-01-25 16:32:53 <mcrute> thanks... I'm working on a better way to do this but it's not going to be ready for a little bit
2021-01-25 16:33:15 <mcrute> I can send you SSH keys if you can give me access to the right thing so I don't have to bother you in the meantime
2021-01-25 16:34:55 <ikke> I think we'd rather make sure it's not necessary to do it manually
2021-01-25 16:35:41 <mcrute> that would certainly be ideal
2021-01-25 16:36:03 <mcrute> okay, well if there's anything I can do to help give me a shout
2021-01-25 16:37:54 <ikke> mcrute: shouldn't there be a commit against https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite?
2021-01-25 16:39:36 <mcrute> no the builder just loads the yaml we publish in the AMI repo
2021-01-25 16:39:47 <mcrute> similar to how the releases page works
2021-01-25 16:40:24 <ikke> mcrute: it uses make to build the site, but atm it does not see any changes
2021-01-25 16:41:14 <mcrute> the make target for the cloud pages wgets the yaml we publish so the build would need run but no new commits would exist
2021-01-25 16:43:40 <ikke> there is an update-release target that removes the releases.yaml file and runs make again
2021-01-25 16:43:47 <ikke> that is missing for the cloud images
2021-01-25 16:43:53 <ikke> I can do it manually for now
2021-01-25 16:44:12 <mcrute> okay I'll fix that in the mksite scripts
2021-01-25 16:44:23 <ikke> can you check it's up-to-date now?
2021-01-25 16:45:10 <mcrute> it is not yet
2021-01-25 16:46:03 <ikke> hmm, it did not do the curl part
2021-01-25 16:46:56 <ikke> ok, fixed
2021-01-25 16:47:46 <ikke> mcrute: we have a webhooks-to-mqtt service, I guess we could use that to trigger rebuilding the cloud images part
2021-01-25 16:48:11 <mcrute> looks good now
2021-01-25 16:48:20 <ikke> ok, good
2021-01-25 16:48:26 <mcrute> thanks
2021-01-25 16:48:44 <ikke> and, I don't mind being bothered about this, just had to figure out how to properly trigger this
2021-01-25 16:48:48 <mcrute> how do I trigger that webhook? i can add it to our inages repo
2021-01-25 16:48:58 <ikke> mcrute: let me get the details
2021-01-25 16:49:27 <mcrute> I just don't want to bother you with trivial things we could script :-)
2021-01-25 16:49:36 <ikke> nod
2021-01-25 16:52:25 <ikke> mcrute: so we first need a propert make target for this
2021-01-25 16:52:38 <ikke> like update-release
2021-01-25 16:54:44 <mcrute> ok
2021-01-25 18:59:40 <ikke> https://status.linode.com/incidents/dm9mc14kv2t4
2021-01-25 20:10:12 <ikke> zabbix has been upgraded to 5.2
2021-01-25 20:20:36 <ikke> \o/
2021-01-25 21:07:00 <mps> also I had outage at linode
2021-01-25 21:07:51 <mps> interesting thing is that one linode VM in Frakfurt worked fine but important one was cut of the net
2021-01-26 05:28:28 <ikke> mps: yes, it was a network outage, the servers themself were still running
2021-01-26 12:52:35 <clandmeter> ikke: hi
2021-01-26 12:52:40 <clandmeter> sorry i am swamped 
2021-01-26 12:53:18 <clandmeter> could not respond your msgs on time :(
2021-01-26 12:53:31 <clandmeter> and i dont think it will improve coming weeks
2021-01-26 12:54:23 <ikke> No problem, just trying to keep you updated
2021-01-26 12:55:45 <ikke> Could we extend webhooks so that mcrute can post to it and trigger the clouds page to be updated? 
2021-01-26 12:56:09 <ikke> It's currently geared towards gitlab events 
2021-01-26 13:06:42 <clandmeter> ikke: i think its trivial to add something
2021-01-26 13:07:21 <clandmeter> what is the cloud page?
2021-01-26 13:07:25 <clandmeter> the one of www?
2021-01-26 13:09:41 <clandmeter> ikke: how does mcrute want to talk to it, via gitlab?
2021-01-26 13:11:54 <ikke> The process is similar to how the releases are updated
2021-01-26 13:12:22 <ikke> Which is triggered via mqtt
2021-01-26 13:13:29 <clandmeter> currently everything is triggered via mqtt iirc
2021-01-26 13:13:53 <ikke> yes
2021-01-26 13:14:07 <clandmeter> so webhooks.a.o is just an interface to mqtt
2021-01-26 13:14:10 <ikke> yup
2021-01-26 13:14:23 <ikke> So my idea was to provide a webhook that mcrute could trigger
2021-01-26 13:14:32 <ikke> from his side, when the images are updated
2021-01-26 13:15:01 <clandmeter> right
2021-01-26 13:15:08 <clandmeter> but it can also be done from the other side (i think)
2021-01-26 13:15:23 <clandmeter> his changes are in gitlab?
2021-01-26 13:15:39 <ikke> not at the moment
2021-01-26 13:15:46 <clandmeter> ah ok
2021-01-26 13:15:50 <ikke> then it would be easy :)
2021-01-26 13:16:17 <clandmeter> its never easy, only seems so ;-)
2021-01-26 13:16:41 <clandmeter> if its from ie github we could add another entrypoint
2021-01-26 13:17:34 <clandmeter> so you would have to analyze the payload and run some script
2021-01-26 13:21:16 <clandmeter> ikke: if you are feeling adventurous, you could try to build www via a pipeline
2021-01-26 13:22:17 <clandmeter> which could be triggered via a webhook
2021-01-26 13:31:04 <ikke> clandmeter: Yeah, I was thinkg about that as well
2021-01-26 13:31:12 <ikke> but the trigger sounded like an easier interrim solution
2021-01-26 13:31:26 <clandmeter> yes maybe
2021-01-26 13:32:05 <clandmeter> we could also start doing MR's on www changes
2021-01-26 13:32:20 <clandmeter> push them to wwwtest
2021-01-26 13:32:22 <ikke> I already created an MR for the changelog
2021-01-26 13:32:43 <ikke> https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged
2021-01-26 13:33:09 <ikke> the harder part are the changes outside of source control
2021-01-26 13:33:18 <ikke> git commits, releases, cloud images
2021-01-26 13:37:47 <clandmeter> changes outside of source control?
2021-01-26 13:37:56 <clandmeter> like aports git commits?
2021-01-26 13:41:38 <clandmeter> ah we have /releases now :)
2021-01-26 13:44:10 <ikke> All of those
2021-01-26 13:56:05 <clandmeter> ikke: i think we can trigger ci from another project?
2021-01-26 13:57:41 <clandmeter> mcrute: maybe an idea to combine arches on a single line at https://alpinelinux.org/cloud/ ?
2021-01-26 13:58:22 <clandmeter> and have a launch button per arch
2021-01-26 14:01:34 <ikke> what generates the latest-releases.yaml on the mirrors?
2021-01-26 14:49:27 <clandmeter> i think ncopas release scripts
2021-01-26 14:58:46 <ikke> ah ok
2021-01-26 14:59:28 <ikke> it gets updated on rsync/* events from mqtt
2021-01-26 15:59:16 <clandmeter> what gets updated?
2021-01-26 16:02:07 <ikke> https://www.alpinelinux.org/releases/
2021-01-26 16:10:29 <mcrute> clandmeter: it would be nice to be able to trigger the webhook from GitHub for now... we'll move to GitLab soon but we're still publishing image updates in the interim
2021-01-26 16:11:05 <mcrute> clandmeter: do you mean combine arches in a single table?
2021-01-26 16:11:29 <mcrute> with more cloud providers the page size becomes nProviders * nRegions * nArchs
2021-01-26 16:11:54 <clandmeter> hi
2021-01-26 16:12:01 <mcrute> was thinking about making it searchable/filterable like the Ubuntu one https://cloud-images.ubuntu.com/locator/
2021-01-26 16:12:24 <mcrute> hi :-)
2021-01-26 16:12:47 <clandmeter> i have tried to limit the use of js, but i know its not always possible.
2021-01-26 16:13:56 <mcrute> I'm not a huge fan of JS but don't have a great idea to make such a big list manageble otherwise, do you have a better idea?
2021-01-26 16:22:49 <clandmeter> i guess provider+region will have multiple arches, these could be combines in the launch column?
2021-01-26 16:23:12 <clandmeter> that would cut a single table in half 
2021-01-26 16:24:23 <clandmeter> there is a lot of duplicated data atm in the table.
2021-01-26 17:34:23 <mcrute> I like that... I'll bake it into V2 (which is coming soon(TM))
2021-01-27 19:29:12 <ncopa> looks like the CI for secdb is broken? https://gitlab.alpinelinux.org/alpine/infra/docker/secdb/-/merge_requests/3
2021-01-27 19:29:40 <ncopa> jobs are getting stuck
2021-01-27 20:13:13 <ikke> ncopa: shared runners were not enabled
2021-01-27 20:13:16 <ikke> on the project
2021-01-27 20:15:34 <ikke> hmm, no, that's not enough
2021-01-27 20:15:55 <ikke> ah, right
2021-01-27 20:16:09 <ikke> that's something quite anoying with gitlab
2021-01-27 20:16:29 <ikke> We deliberately limited the docker image runners to not be public shared runners
2021-01-27 20:16:43 <ikke> because they give direct access to the docker socket on the host
2021-01-27 20:42:16 <ikke> but that also prevents them from being used by forks :-/
2021-01-27 20:44:31 <Hello71> sounds like a good reason to minimize how often MRs are required
2021-01-27 20:45:24 <ikke> What do you mean?
2021-01-27 20:46:00 <ikke> I guess a better solution would be to look into one of the daemonless alternatives
2021-01-27 20:47:26 <ikke> But it's kind of anoying that you cannot easily share runners to forks
2021-01-27 20:47:37 <ikke> except for creating globally shared runners
2021-01-27 20:49:24 <Hello71> i mean try to move as much code as possible to separate repos
2021-01-27 20:50:04 <ikke> Isn't https://gitlab.alpinelinux.org/alpine/infra/docker/secdb/ a separate repo?
2021-01-27 20:50:14 <ikke> and how would that avoid MRs being made?
2021-01-27 20:50:22 <Hello71> eh... never mind
2021-01-27 20:50:35 <ikke> sorry, I'm not following you :)
2021-01-27 21:09:29 <ikke> I guess podman would work for building images in CI
2021-01-28 22:26:14 <ncopa> how do i trigger a rebuild of the https://alpinelinux.org/ site so the 3.13.1 gets published?
2021-01-28 22:31:55 <ncopa> pushing anything to aports git master was the answer
2021-01-29 08:29:00 <clandmeter> ncopa, ikke: we have news from arm.
2021-01-29 08:31:02 <clandmeter> we can swap out the current boxes for single or dual socket ampere machines
2021-01-29 08:33:28 <ncopa> morning clandmeter. sounds good. what is the disk size of those?
2021-01-29 08:33:41 <clandmeter> check your email ;-)
2021-01-29 08:33:53 <clandmeter> Mt Snow: 128GB (8x 16GB), 1TB SSD U.2, 1x 25GbE NIC
2021-01-29 08:33:53 <clandmeter> Mt Jade: 16x 16GB per socket, 2x SSD U.2 NVMe, 1x 25GbE NIC
2021-01-29 08:37:14 <clandmeter> jade is 512G with 160 cores
2021-01-29 09:16:13 <ncopa> so we could have 2 jade machines?
2021-01-29 09:17:31 <clandmeter> :)
2021-01-29 09:17:51 <ncopa> doesnt say how big disks are on the jade machines
2021-01-29 09:18:15 <clandmeter> i have no clue why somebody would choose snow... but maybe its a gigabyte machine :)
2021-01-29 09:23:16 <ncopa> maybe we just go ahead with our usa1-dev machine? its the aarch64 builder
2021-01-29 09:23:45 <ncopa> then we can verify that it can actually run 32 bit containers before we move the armv7 builder
2021-01-29 09:24:46 <clandmeter> I could ask upfront about it
2021-01-29 09:24:58 <clandmeter> i also wonder how it will perform in qemu
2021-01-29 09:26:30 <ncopa> LBlaboon: hey! I pushed alpine 3.13.1 yesterday. maybe update the linode images?
2021-01-29 10:00:01 <clandmeter> ncopa: i found a presentation which mentions it can run 32bit applications
2021-01-29 10:01:18 <ncopa> great
2021-01-29 11:12:17 <clandmeter> ncopa: if we do the builder first, it could mean we run in trouble
2021-01-29 11:14:56 <clandmeter> maybe better to firstly try with CI?
2021-01-29 11:22:53 <ikke> I did not receive that e-mail
2021-01-29 11:22:56 <ikke> but nice
2021-01-29 11:26:01 <ncopa> what do we have for the CI?
2021-01-29 11:27:01 <ikke> Currently our arm CI runs as a VM one the builders
2021-01-29 11:27:16 <ikke> on the*
2021-01-29 11:28:28 <ikke> and armhf is missing
2021-01-29 11:31:32 <ncopa> its usa4, isnt it?
2021-01-29 11:31:44 <ikke> for armv7 and armhf
2021-01-29 11:31:47 <ikke> and armv7 ci
2021-01-29 11:32:02 <ncopa> the aarch64 ci?
2021-01-29 11:32:13 <ikke> aarch64 ci is on usa1
2021-01-29 11:32:44 <ncopa> ok, so it shoudlnt matter which we do first in other words
2021-01-29 11:33:03 <ncopa> usa1 is the weakest so it is the one that will benefit most i suppose
2021-01-29 11:33:31 <ncopa> but I don't mind if you prefer do usa4, dont let me stand in the way :)
2021-01-29 11:33:54 <ikke> for me it does not matter
2021-01-29 11:45:36 <clandmeter> ok good
2021-01-29 11:45:40 <clandmeter> for me it also doesnt matter
2021-01-29 11:45:52 <clandmeter> we need to find out how much time we have to migrate
2021-01-29 11:46:46 <clandmeter> let me reply the email and cc you
2021-01-29 12:02:27 <clandmeter> done
2021-01-29 15:35:06 <LBlaboon> ncopa: yep, we're already working on it. taking us a little longer because we're doing rebuilds of everything to patch the sudo CVE from earlier this week
2021-01-29 15:49:54 <ikke> That one is fun 
2021-01-29 15:50:09 <ikke> Are you building it yourself? 
2021-01-29 15:52:08 <LBlaboon> yep
2021-01-29 15:52:47 <ikke> Aha ook 
2021-01-29 15:52:52 <ikke> Ok*
2021-01-30 00:31:13 <LBlaboon> forgot to mention this earlier, but 3.13.1 is out on linode
2021-01-30 18:53:12 <TBK[m]> aarch64, armv7 and s390x edge buildozers all appear according to build.a.o to be stuck on community/telepathy-glib 0.24.2-r0