2022-08-01 11:14:25 <mps> linux 5.19 is probably last in 5.x series, Linus announced that next will be 6.0 probably
2022-08-01 22:02:39 <ptrc> ah
2022-08-01 22:02:40 <ptrc> https://twitter.com/Linode_Status/status/1554223973696606210
2022-08-01 22:02:47 <ptrc> > Our team is investigating a connectivity issue in our London data center.
2022-08-01 22:03:15 <panekj> pkgs.a.o dead :C
2022-08-01 22:03:43 <ptrc> yeah, that's the host i'm talking about
2022-08-02 03:38:28 <ikke> Fun times 
2022-08-02 10:53:11 <ikke> cleaned up .cache/yarn and .cache/go on all arm*/aarch64 builders for 3.15, 3.16 and edge
2022-08-02 10:53:15 <ikke> sorry
2022-08-02 10:53:20 <ikke> .cargo, not .cache/go
2022-08-02 10:53:26 <ikke> since go is building atm
2022-08-02 20:00:17 <ptrc> ikke: could you deploy aports-turbo changes when you're not too busy? https://ptrc.gay/ffDBrVKE
2022-08-02 20:02:58 <ikke> ptrc: can you remind me thursday, have more time then
2022-08-02 20:03:03 <ptrc> sure thing :)
2022-08-04 09:02:25 <ikke> Lesigh 
2022-08-04 11:29:59 <ikke> 67G free atm
2022-08-04 14:02:23 <ikke> ptrc: fyi, I'm working on aports-turbo at
2022-08-04 14:02:25 <ikke> atm
2022-08-04 14:03:12 <ptrc> was about to remind you a while ago, but saw that you also merged some mrs, thanks :)
2022-08-04 14:03:59 <ikke> np, I just noticed that the branch drop-down first item is still disbaled
2022-08-04 14:04:00 <ikke> disabled*
2022-08-04 14:04:17 <ikke> Fixing that
2022-08-04 14:12:30 <ptrc> doesn't it default to edge?
2022-08-04 14:12:42 <ptrc> as in, when there's no param in the url
2022-08-04 14:15:01 <ptrc> yeah, https://gitlab.alpinelinux.org/alpine/infra/aports-turbo/-/blob/a7ce1829/aports.lua#L18
2022-08-04 14:15:36 <ptrc> i mean, i don't know if there's a good reason to even allow showing packages from all branches
2022-08-04 14:15:38 <ikke> Yes, but wasn't the idea that you can go back to that state?
2022-08-04 14:15:54 <ikke> oh, right
2022-08-04 14:16:02 <ikke> It used to be that way
2022-08-04 14:16:08 <ikke> was too heavy on the db
2022-08-04 14:16:35 <ikke> so the dbs were split per version
2022-08-04 17:05:43 <ikke> clandmeter: apparently `deploy: replicas: 4` does work with docker compose, unlike what the documentation mentions
2022-08-04 17:06:06 <ikke> (as the replacement for scale: 4)
2022-08-05 04:41:36 <clandmeter> ikke: yes i think it depends on compose version?
2022-08-05 04:41:48 <ikke> clandmeter: probably
2022-08-05 04:41:58 <clandmeter> as set in yml
2022-08-05 04:41:59 <ikke> I noticed that docker compose config translated it to that
2022-08-05 04:42:10 <clandmeter> not the actual binary
2022-08-05 04:42:28 <ikke> even with version: "2" it translated it internally
2022-08-05 04:42:31 <clandmeter> 2.4 vs 3.x
2022-08-05 04:42:36 <ikke> "2.4"
2022-08-05 04:42:51 <ikke> and complained about that it was deprecated :)
2022-08-05 04:43:08 <clandmeter> 2.4 is deprecated? 
2022-08-05 04:43:20 <ikke> no
2022-08-05 04:43:24 <ikke> the scale setting
2022-08-05 04:44:14 <clandmeter> it mentions this on 3.x i guess?
2022-08-05 04:44:17 <clandmeter> or also on 2.4?
2022-08-05 04:44:22 <ikke> also when you use 2.4
2022-08-05 04:44:41 <clandmeter> that is weird
2022-08-05 04:44:52 <clandmeter> as scale is specific to 2.2 and higher iirc
2022-08-05 04:44:57 <ikke> https://gitlab.alpinelinux.org/alpine/infra/docker/aports-turbo/-/blob/master/docker-compose.prod.yml
2022-08-05 04:45:48 <clandmeter> we should probably also update configs on traefik
2022-08-05 04:46:25 <clandmeter> i did some modifying at work to get better ssl and header support
2022-08-05 04:46:29 <clandmeter> secure headers 
2022-08-05 04:46:32 <ikke> ah ok
2022-08-05 04:46:48 <clandmeter> i can share it with you if you like
2022-08-05 04:47:03 <ikke> clandmeter: this is partial docker compose config output with that set: https://tpaste.us/Pjeb 
2022-08-05 04:47:36 <ikke> clandmeter: it might be that it's docker compose 2 that started doing that
2022-08-05 04:47:45 <clandmeter> looks ok to me if it works
2022-08-05 04:47:52 <ikke> it works
2022-08-05 04:48:04 <ikke> but with version: "3" and setting replicas also works
2022-08-05 04:48:07 <clandmeter> so it will spin 4 replicas ?
2022-08-05 04:48:10 <ikke> yes
2022-08-05 04:48:16 <ikke> both work
2022-08-05 04:48:16 <clandmeter> good
2022-08-05 04:48:27 <ikke> the latter doesn't give you warnings
2022-08-05 04:48:31 <clandmeter> i think i used the old config as else i needed to pass it on cmdline
2022-08-05 04:48:43 <ikke> yes, but now it doesn't anymore
2022-08-05 04:49:08 <clandmeter> most of the 3.x stuff is not needed for me anyways so i always stick to 2.4
2022-08-05 04:49:31 <ikke> https://tpaste.us/6E6B
2022-08-05 04:49:36 <ikke> I just don't like the warnings :P
2022-08-05 04:49:44 <ikke> this is running atm
2022-08-05 04:49:47 <clandmeter> im ok with 3
2022-08-05 04:49:55 <clandmeter> sooner or later we need to migrate anyways
2022-08-05 04:50:02 <clandmeter> if it work
2022-08-05 04:50:02 <clandmeter> s
2022-08-05 04:50:07 <ikke> yup
2022-08-05 04:50:18 <ikke> The documentation says docker compose would ignore that part
2022-08-05 04:50:21 <ikke> but it apparently doesn't
2022-08-05 04:50:25 <clandmeter> i just got back from schiphol :)
2022-08-05 04:50:30 <clandmeter> what a mess :)
2022-08-05 04:50:56 <ikke> https://docs.docker.com/compose/compose-file/compose-file-v3/#deploy
2022-08-05 04:51:20 <ikke> 'The following
2022-08-05 04:51:22 <ikke> sub-options only takes effect when deploying to a swarm with docker stack deploy, and is ignored by docker-compose up and docker-compose run, except for resources."
2022-08-05 04:51:34 <clandmeter> nod
2022-08-05 04:52:28 <clandmeter> is there any pref/need to switch to podman?
2022-08-05 04:54:12 <ikke> I was thinking about that
2022-08-05 04:54:27 <ikke> security would be one reason
2022-08-05 06:47:06 <ncopa> your problem is that you dont get replicas? maybe consider run kubernetes?
2022-08-05 07:16:35 <ptrc> 
ACTION taps the sign

2022-08-05 07:16:36 <ptrc> https://xeiaso.net/blog/do-i-need-kubernetes
2022-08-05 07:56:15 <ikke> ncopa: no, we do get replicas 
2022-08-05 07:56:49 <ikke> ncopa: we use that for aports - turbo because it's just a single process 
2022-08-05 11:28:12 <rnalrd> ikke https://lists.alpinelinux.org/lists/%7Ealpine returns 404
2022-08-05 11:36:37 <ikke> Try to refresh. For some reason nginx likes to return 404 from time to time 
2022-08-05 12:14:57 <rnalrd> ok thanks
2022-08-05 18:01:57 <ikke> ok, gitlab 15.1 image got built :)
2022-08-05 18:32:57 <ikke> https://gitlab-test.alpinelinux.org
2022-08-08 07:00:45 <nmeum> ikke: mosquitto upstream stopped ignoring sigpipe in the client library \o/ https://github.com/eclipse/mosquitto/commit/0c9d9f21633c5dbb482893a9d6bdf40111829925
2022-08-08 07:01:02 <nmeum> so after the next release we can remove the workarounds from mqtt-exec etc
2022-08-08 07:01:21 <ikke> Nice! 
2022-08-08 09:02:35 <ncopa> congrats! I don't mind backporting that patch
2022-08-08 10:00:03 <clandmeter> nmeum: nice work, seems like the smallest things take the most time (years in this case) to get fixed :)
2022-08-09 08:53:18 <ddevault> this gitlab mailing list integration upgrade is rough
2022-08-09 08:53:25 <ddevault> I'm not sure I want to finish reimplementing it or commit to long-term maintenance
2022-08-09 08:53:30 <ddevault> let's discuss possible solutions
2022-08-09 08:55:37 <panekj> option 1) get rid of gitlab (:
2022-08-09 08:55:44 <ddevault> god, I wish
2022-08-09 08:57:08 <ddevault> some ideas: (2) get some volunteers to help finish & maintain it; (3) get some maintainers to review patches on the lists; (4) deprecate submitting patches via email
2022-08-09 08:57:49 <ikke> What kind of project is it? Maybe I can devote a bit of time on it 
2022-08-09 08:58:00 <panekj> what kind of language
2022-08-09 08:58:39 <ddevault> Go
2022-08-09 08:58:48 <ddevault> https://git.sr.ht/~sircmpwn/hashiru
2022-08-09 08:59:06 <ddevault> starting to get burnt out on it so I just yeeted a broken WIP commit onto master for posterity
2022-08-09 09:00:30 <ddevault> regarding option (4), I'm pretty tired of pushing against gitlab and I'm prepared to accept defeat, but it will likely mean that I step down as the maintainer of many packages
2022-08-09 09:01:07 <ikke> 3 is hard
2022-08-09 09:01:30 <ddevault> though lots of people still use the mailing lists for submitting patches, I can't continue being the only one spending days maintaining an integration between the two systems
2022-08-09 09:01:43 <ikke> I'd like to try 2
2022-08-09 09:02:38 <ddevault> aight
2022-08-09 09:02:57 <ddevault> you can get an idea of the current thrust of the work by reading over this: https://git.sr.ht/~sircmpwn/hashiru/commit/750b1f1afabb387e90ed51ddade7c3605d7c916f
2022-08-09 09:03:10 <ddevault> I was working on the webhook which submits a gitlab MR when a patch comes in
2022-08-09 09:09:28 <ddevault> also, re: (3) being difficult
2022-08-09 09:09:59 <ikke> I mean, finding people willing to do it who also have push access 
2022-08-09 09:10:07 <ddevault> does not feel great where a minor inconvenience of reviewing patches on a mailing list is held in higher regard than a major inconvenience of tasking one person with the development and maintenance of a complex software band-aid
2022-08-09 09:10:33 <ddevault> it would be nice if we were able to identify any kind of compromise
2022-08-09 10:20:51 <ikke> ddevault: It's not a nice experience to act as a humane bridge between 2 systems
2022-08-09 10:20:58 <ikke> human*
2022-08-09 10:21:27 <ddevault> it doesn't have to be two systems
2022-08-09 10:21:37 <ddevault> no need to manually copy patches to gitlab, just review them on the list
2022-08-09 10:21:49 <ikke> It's not just reviewing, it's also CI
2022-08-09 10:22:04 <ikke> And the feedback CI gives
2022-08-09 10:23:35 <ddevault> well, the CI can probably be run on builds.sr.ht as well
2022-08-09 10:24:19 <ptrc> then alpine would need to run 2 separate workflows, two different CI images / systems and all that
2022-08-09 10:24:24 <ddevault> same scripts
2022-08-09 10:24:25 <ikke> That means either 2 systems, or moving to build.sr.ht completely, which is another discussion
2022-08-09 10:24:44 <ddevault> or, it may be possible to write a simpler API integration which submits gitlab pipelines when patches come in
2022-08-09 10:24:54 <ddevault> rather than the whole merge request syncronization dance
2022-08-09 10:25:36 <ptrc> pipelines are tied to commits, so it would need to push to gitlab either way
2022-08-09 10:25:44 <ddevault> well, maybe
2022-08-09 10:25:52 <ptrc> imo it breaks workflow if half the stuff is on gitlab and the other half somewhere else
2022-08-09 10:26:01 <ddevault> could make a few small edits to the build scripts wherein the pipeline runs against master, then fetches and applys the patch from the mailing list before testing it
2022-08-09 10:26:07 <ptrc> and i don't think alpine would want to move to a mail-patches-only workflow
2022-08-09 10:26:27 <ddevault> how does this "break the workflow"
2022-08-09 10:26:54 <ddevault> the mailing lists are still in use, abandoning them breaks people's workflows to a much more significant extent
2022-08-09 10:27:13 <ddevault> but I'm sick of fighting this gitlab crap so if that's how it has to be then so be it
2022-08-09 10:27:50 <ddevault> if this discussion goes even slightly in the direction of "mailing list patches aren't worth supporting" then I'm going to concede that point immediately and take my leave
2022-08-09 10:28:10 <ddevault> I'm not rehashing that fight again
2022-08-09 10:28:16 <ptrc> if you're an alpine developer who reviews stuff, you can't easily go through the whole list, mark as duplicate, reference in a comment easily, etc.
2022-08-09 10:28:25 <ikke> ddevault: I have always been a proponent of keeping the MLs alive
2022-08-09 10:28:28 <ddevault> sure you can.
2022-08-09 10:28:31 <ikke> and supported the effort
2022-08-09 10:28:34 <ptrc> i'm not saying that mailing lists are bad / not worth it, the opposite actually
2022-08-09 10:29:04 <ptrc> i'm saying that having two *separate* systems would be an inconvenience
2022-08-09 10:29:38 <ddevault> right now the "inconvenience" is signficantly greater than what you describe and is entirely borne by me
2022-08-09 10:30:31 <ptrc> fwiw, if there's a way i could help support the gitlab<->ml integration, i'd be more than happy to help
2022-08-09 10:30:41 <ikke> And I also offered my help
2022-08-09 10:30:46 <ddevault> sure
2022-08-09 10:30:50 <ddevault> send some patches
2022-08-09 10:30:54 <ddevault> via email, please :)
2022-08-09 10:30:59 <ptrc> which project?
2022-08-09 10:31:05 <ikke> Linked earlier
2022-08-09 10:31:06 <ddevault> see links above
2022-08-09 10:31:16 <ptrc> ah, sure
2022-08-09 10:34:41 <ptrc> hm, maybe it would be possible to simplify this even further and use the existing email gitlab integration? it would be just a daemon that's forwarding any incoming new patches to the aports mr inbox and forwarding the responses back to the appropriate thread (if it wouldn't be possible to just CC: back the ML in the patch itself)
2022-08-09 10:35:07 <ddevault> if I recall correctly, I don't think it forwards responses back, and it expects patches as attachments, not the standard git send-email format
2022-08-09 10:35:13 <ddevault> but it would be better than nothing
2022-08-09 10:37:49 <ptrc> ah
2022-08-09 10:38:03 <ptrc> i got a response "Unfortunately, your email message to GitLab could not be processed. You are not allowed to perform this action. If you believe this is in error, contact a staff member."
2022-08-09 10:38:05 <ptrc> ikke: ?
2022-08-09 10:38:26 <ikke> hmm, not sure
2022-08-09 10:38:34 <ikke> Need to check
2022-08-09 10:38:47 <ikke> ptrc: did you try to send it to the main aports project/
2022-08-09 10:39:01 <ptrc> this is from the "Create merge request" button
2022-08-09 10:39:17 <ptrc> mhm, i thought it's gonna choose my fork automatically, maybe not?
2022-08-09 10:39:46 <ptrc> wouldn't be an issue with the official bridge though
2022-08-09 10:48:21 <ptrc> ddevault: what would be the email format for git.sr.ht repos? the clone section only says "You can also use your local clone with git send-email." :/
2022-08-09 10:50:40 <ddevault> https://git-send-email.io
2022-08-09 10:51:03 <ptrc> sorry, email *address* format
2022-08-09 10:51:28 <ptrc> but i guess the one from the guide with repo name replaced should work
2022-08-09 10:52:23 <ddevault> ~sircmpwn/public-inbox@lists.sr.ht is fine
2022-08-09 10:52:26 <ddevault> or just sir@cmpwn.com
2022-08-09 16:32:41 <ikke> 0..
2022-08-09 16:34:35 <psykose> 0?
2022-08-09 16:35:19 <ikke> that was still in the chat buffer :P
2022-08-09 16:35:37 <psykose> :)
2022-08-09 16:36:01 <ikke> wondering what is taking so much space on usa9
2022-08-09 16:36:05 <psykose> hovering 95 since release
2022-08-09 16:36:10 <psykose> just slowly reached it
2022-08-09 16:36:46 <ikke> Still 400G free in the vgs
2022-08-09 16:36:58 <ikke> sadly usa10 is not usable
2022-08-09 16:37:06 <ikke> still not resolved the reboots
2022-08-09 16:38:59 <psykose> what was the issue again
2022-08-09 16:39:05 <psykose> (yes, my memory sucks)
2022-08-09 16:39:28 <psykose> i assume it reboots at random with the above usa10 down/up things
2022-08-09 16:40:19 <ikke> yes, and even more often when it does not alert
2022-08-09 16:40:54 <ikke> average uptime is 7h
2022-08-09 16:41:03 <ikke> over last 30 days
2022-08-09 16:41:13 <ikke> reboots without any indication why
2022-08-09 16:42:05 <ikke> already tried to switch to linux-edge
2022-08-09 16:44:49 <ikke> I _can_ try to stop the ci container see if that helps
2022-08-09 16:45:07 <psykose> sure
2022-08-09 16:47:20 <ikke> any clue why in general there is only just a couple of hours of logs in /var/log/messages(.0)?
2022-08-09 16:48:02 <psykose> for me it's a lot, so i assume something deletes them on the server?
2022-08-09 16:48:28 <ikke> I've noticed that everywhere
2022-08-09 16:48:56 <psykose> the default is 200KB rotation
2022-08-09 16:49:04 <psykose> i pass -s0
2022-08-09 16:49:20 <ikke> where?
2022-08-09 16:49:24 <psykose> conf.d/syslog
2022-08-09 16:49:29 <ikke> right
2022-08-09 16:49:36 <psykose> `busybox syslogd -h` has the options
2022-08-09 16:49:54 <psykose> all the logging infrastructure is horrific
2022-08-09 16:50:03 <psykose> but sadly good logging integration requires good service management
2022-08-09 16:50:10 <psykose> so there's not much to fix
2022-08-09 16:51:20 <psykose> the choices are generally 'unbounded unrotated logs' or 'no logs', but at least for /var/log/messages you can raise that or pass -b10 or something or both
2022-08-09 16:51:45 <ikke> yes, set -s 1024 -b 14 now
2022-08-09 18:58:52 <ikke> clandmeter: What is left for setting up the new mirror infra? Is there something I can do?
2022-08-09 18:59:09 <ikke> each releases puts more pressure on NLD3 storage
2022-08-09 19:07:28 <psykose> what are the changes
2022-08-09 19:08:37 <ikke> 1. We get a newer generation set of servers
2022-08-09 19:08:52 <ikke> 2. We get at least one server with lots of storage for our mirror infra
2022-08-09 19:09:12 <psykose> i mean to the mirror setup so they don't all have to be on one server
2022-08-09 19:09:31 <ikke> The idea was to use anycast
2022-08-09 19:09:47 <ikke> but that depends on if we can get more nodes
2022-08-09 19:09:53 <psykose> hmm
2022-08-09 19:10:04 <psykose> my head doesn't understand how that would allow splitting the files
2022-08-09 19:10:14 <psykose> the anycast would just 404 if they're not there
2022-08-09 19:10:26 <ikke> That would not be about splitting any files
2022-08-09 19:10:30 <ikke> those would be complete mirrors
2022-08-09 19:10:35 <ikke> what NLD3 now provides
2022-08-09 19:10:40 <psykose> well.. then all the things are still on the same server
2022-08-09 19:10:42 <psykose> doesn't gain much
2022-08-09 19:10:52 <ikke> This is separate from the build infra
2022-08-09 19:10:57 <psykose> aware
2022-08-09 19:11:07 <psykose> i mean not having to store everything of everywhere in one place
2022-08-09 19:11:18 <ikke> Well, we are stuck with that for the time being
2022-08-09 19:11:26 <ikke> everyone syncs from us using rsync
2022-08-09 19:11:31 <ikke> from rsync.a.o
2022-08-09 19:11:35 <psykose> mhm
2022-08-09 19:11:42 <ikke> That's not something you easily refactor
2022-08-09 19:12:47 <ikke> at some point, we _can_ think about moving older versions somewhere else (like we had ancient.a.o)
2022-08-09 19:13:01 <ikke> but as older versions are a lot smaller, that does not add up to a lot
2022-08-09 19:14:00 <ikke> But as we seem to have at least one large-storage node from equinix, that should not matter
2022-08-09 19:14:13 <ikke> Solving this for the builders is another issue
2022-08-09 19:14:34 <ikke> But that's something we can change while still aggregating everything on the mirror infra
2022-08-09 19:42:21 <ikke> psykose: I think one thing we will need in the future is someting that can accept packages from the builders (signed), and will resign it with cannonical keys and generate apk indexes
2022-08-09 19:43:02 <psykose> yeah
2022-08-09 19:43:05 <psykose> makes sense
2022-08-09 19:43:32 <psykose> (with the canonical ubuntu keys? /s)
2022-08-09 19:43:36 <ikke> ncopa mentioned that in the last conference
2022-08-09 19:43:41 <ikke> psykose: :D
2022-08-09 19:43:59 <psykose> :)
2022-08-09 22:30:13 <psykose> ikke: could u upgrade/fix/whatever the special zabbix plugin and move it to community and also backport it, since otherwise it's impossible to upgrade it without adding openssl3 to deu1
2022-08-10 04:26:42 <ikke> psykose: It's deprecated, since that zabbix now supports proper plugins for agent2 that do not need to be compiled directly with the agent. I've tried to convert it, but it was not working yet atm.
2022-08-10 06:49:13 <clandmeter> ikke: all 3 servers are already running
2022-08-10 06:49:23 <ikke> Right 
2022-08-10 06:49:27 <clandmeter> but thats it
2022-08-10 06:49:29 <clandmeter> running :)
2022-08-10 06:49:42 <ikke> Did we get confirmation that we could keep using all 3?
2022-08-10 06:49:49 <clandmeter> i think so
2022-08-10 06:49:52 <ikke> Ok
2022-08-10 06:50:00 <clandmeter> i think we also need to add more servers to replace the others
2022-08-10 06:50:04 <ikke> Yes 
2022-08-10 06:58:10 <clandmeter> looks like the nld one has zfs configured
2022-08-10 07:00:29 <clandmeter> ikke: i guess we need to think about how we want to set them up
2022-08-10 07:00:49 <clandmeter> do we want to do it based on containers or just plain os
2022-08-10 07:01:35 <clandmeter> or do you want to use some config mgnt
2022-08-10 07:39:46 <clandmeter> ikke: anycast ip is added to the 3 servers
2022-08-10 07:40:10 <clandmeter> but im unsure how to test it :)
2022-08-10 07:56:17 <ikke> Setup a webserver where each server has some different content 
2022-08-10 07:56:51 <ikke> And then try to access the anycast ip from different locations 
2022-08-10 08:15:25 <clandmeter> yes thats the thing
2022-08-10 08:15:28 <clandmeter> one is in asia
2022-08-10 08:15:35 <clandmeter> i dont have access to something in asia
2022-08-10 08:15:58 <clandmeter> but i think its working correctly
2022-08-10 08:19:02 <clandmeter> not*
2022-08-10 08:20:35 <clandmeter> if is ssh to the elastic ip from usa5, it will end up on nld
2022-08-10 08:21:06 <clandmeter> i have a feeling we will need to setup bgp
2022-08-10 08:28:36 <ikke> Hmm, ok
2022-08-10 08:29:06 <ikke> To setup BGP, we need to know where to connect to
2022-08-10 08:32:04 <clandmeter> let me first try to figure out if it really works or not
2022-08-10 08:32:22 <clandmeter> accessuing it from a dallas server does seem to work
2022-08-10 08:32:28 <clandmeter> but that could be related to the same dc
2022-08-10 08:33:08 <clandmeter> ok
2022-08-10 08:33:15 <clandmeter> it looks like it works this way
2022-08-10 08:38:31 <clandmeter> yup, i spinned some linodes in the same region and they all connect the the related server
2022-08-10 08:38:43 <clandmeter> to*
2022-08-10 09:49:16 <clandmeter> that is crude
2022-08-10 09:49:30 <clandmeter> apk upgrade just broke my systems kernel
2022-08-10 09:53:35 <ikke> How so? 
2022-08-10 09:59:31 <clandmeter> not sure what happend, but the running kernel is not the installed kernel
2022-08-10 10:01:21 <panekj> that's normal?
2022-08-10 10:02:55 <panekj> when you upgrade, you won't have same kernel until you reboot
2022-08-10 11:29:23 <ikke> panekj: I don't think clandmeter is unaware of that, so wonder what he means
2022-08-10 21:59:55 <clandmeter> what i mean is, the kernel located in boot is still the old kernel
2022-08-10 22:00:00 <clandmeter> but the modules are updated
2022-08-11 18:51:37 <ikke> psykose: making progress with the zabbix agent2 plugin
2022-08-11 18:51:58 <psykose> ah, right, that thing
2022-08-11 18:51:59 <psykose> thanks!
2022-08-11 18:52:06 <psykose> let me know what you've done and how it goes
2022-08-11 18:52:43 <ikke> https://gitlab.alpinelinux.org/alpine/infra/zabbix-agent2-plugins
2022-08-11 18:52:53 <ikke> that's still the old code that needs to be built in-tree
2022-08-11 18:55:49 <psykose> how does it work now
2022-08-11 18:56:23 <ikke> a separate binary that reads from a socket
2022-08-11 18:56:32 <ikke> and is started by the agent (after you add it to the config)
2022-08-11 18:58:16 <psykose> mmhmm
2022-08-11 18:58:43 <ikke> It's a common strategy, as dynload is not really a thing
2022-08-11 18:58:50 <ikke> /dlopen
2022-08-11 19:01:19 <ikke> I suppose it's not kept running, just executed when needing data
2022-08-11 19:01:25 <ikke> I don't see it in the processlist
2022-08-11 19:01:48 <psykose> makes sense
2022-08-11 20:09:11 <ikke> https://gitlab.alpinelinux.org/alpine/infra/zabbix-agent2-plugins/-/merge_requests/1
2022-08-11 20:13:37 <psykose> looks alright at a glance
2022-08-11 20:20:10 <ikke> mergedf
2022-08-11 20:20:20 <ikke> Now I need to update that package
2022-08-11 20:51:20 <ikke> psykose: !37516
2022-08-11 21:03:42 <psykose> lookin good
2022-08-12 07:08:17 <clandmeter> ikke: how do i know if i would add a conflicting a record to gitlab tf?
2022-08-12 08:17:03 <clandmeter> for now i just added the hosts to linode dns directly, if you have time please add them to tf
2022-08-12 15:23:23 <psykose> ikke: how big are the aports turbo .sqlite databases
2022-08-12 15:24:31 <ikke> 23G in total
2022-08-12 17:23:52 <clandmeter> 
ACTION sends a big hug to ikke 

2022-08-12 17:24:27 <ikke> 😊
2022-08-12 17:25:20 <clandmeter> nld should be in sync i believe
2022-08-12 17:25:28 <clandmeter> the other 2 are ssllooww
2022-08-12 17:28:44 <ikke> clandmeter: you need help adding the dns records to linode tf?
2022-08-12 17:29:00 <clandmeter> yes i think that makes sense
2022-08-12 17:29:07 <clandmeter> ill config the other two traefiks
2022-08-12 17:29:35 <ikke> Note that, because you already added them to linode, we need to manually import them
2022-08-12 17:29:44 <ikke> in the tf state
2022-08-12 17:29:56 <clandmeter> i was a bit unsure about how to get it done
2022-08-12 17:30:38 <ikke> https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/blob/master/domains_mirrors.tf
2022-08-12 17:30:48 <ikke> Just copy resource for an A record and adjust
2022-08-12 17:33:20 <ikke> Do these machines also have IPv6 setup?
2022-08-12 17:38:51 <clandmeter> yes they do
2022-08-12 17:39:21 <ikke> ok, nice
2022-08-12 17:39:22 <clandmeter> but im not sure about the anycast ip and how that would work
2022-08-12 17:39:34 <ikke> I would expect it would work somewhat similar
2022-08-12 17:40:19 <ikke> How did you setup the anycast ip for ipv4?
2022-08-12 17:40:28 <clandmeter> add it to lo
2022-08-12 17:40:49 <ikke> aha, interesting
2022-08-12 17:41:10 <clandmeter> i guess the routing is via the other ipv4 address
2022-08-12 17:42:20 <ikke> I don't see it in the route table
2022-08-12 17:51:09 <clandmeter> ok the setups should be ok
2022-08-12 17:51:15 <clandmeter> although i cannot test it :)
2022-08-12 17:56:03 <clandmeter> ikke: i modified the sync script on us and sg
2022-08-12 17:56:11 <clandmeter> removed the delayed updates
2022-08-12 17:56:22 <clandmeter> cause i think that will break if the sync is not complete
2022-08-12 17:56:39 <clandmeter> so for init sync let it run with that switch
2022-08-12 17:56:50 <clandmeter> without*
2022-08-12 17:57:39 <clandmeter> and we need to monitor if the mqtt-exec thingy actually works :)
2022-08-12 17:57:43 <clandmeter> i need to go now
2022-08-12 18:25:22 <ikke> alright
2022-08-13 07:55:30 <ikke> gbr2 upgraded to alpine 3.16
2022-08-13 09:04:17 <clandmeter> ikke: did or will you add the entries to tf?
2022-08-13 09:09:15 <ikke> I did not do it yet 
2022-08-13 09:16:39 <clandmeter> ikke: are you ok with the hostname setup i choose? 
2022-08-13 09:27:46 <ikke> Didn't check it in detail yet
2022-08-13 09:41:27 <ikke> I'll look at it in a bit
2022-08-13 11:41:06 <ikke> psykose: I've fixed the agent2 setup on deu1
2022-08-13 13:23:43 <mps> ikke: sorry to annoy you but I don't have anyone except you to ask. why this doesn't skip arm64 `[[ $arch = */${_carch}/ ]] && continue`
2022-08-13 13:23:51 <mps> in ash script
2022-08-13 13:24:28 <mps> in APKBUILD actually
2022-08-13 13:25:06 <panekj> isn't it $CARCH
2022-08-13 13:25:17 <mps> test script is https://tpaste.us/Z4xl
2022-08-13 13:25:41 <mps> panekj: ^
2022-08-13 13:27:20 <mps> panekj: for kernels we set _carch differently from $CARCH
2022-08-13 13:28:03 <mps> on arm64 CARCH is aarch64 though actual ARCH is arm64
2022-08-13 13:28:58 <ikke> Is [[ even supported by ash? Maybe with the bash compat enabled 
2022-08-13 13:28:59 <panekj> */ doesn't expand
2022-08-13 13:29:04 <ikke> But we avoid it 
2022-08-13 13:29:31 <panekj> mps: add set -o xtrace under #!
2022-08-13 13:29:35 <mps> hah, I thought it is supported
2022-08-13 13:29:57 <ikke> Use case / esac
2022-08-13 13:30:09 <mps> ok, will recode it
2022-08-13 13:33:01 <mps> `case "$arch" in $_carch) continue ;; esac`
2022-08-13 13:33:57 <mps> uhm, also this doesn't work
2022-08-13 13:35:37 <panekj> because $arch is not equal to $_carch
2022-08-13 13:35:57 <panekj> if you add the xtrace opt and run it, it will show you what it actually evaluates
2022-08-13 13:36:22 <mps> oh, think I found bug
2022-08-13 13:36:34 <mps> panekj: yes
2022-08-13 13:37:05 <mps> arch is actually ./arch/arm64/
2022-08-13 13:39:58 <mps> btw, we don't use linux-edge4virt on any infra machine?
2022-08-13 15:24:06 <ikke> mps: not that I'm aware of
2022-08-13 15:24:13 <ikke> most use lts, and in a few cases edge
2022-08-13 15:38:36 <psykose> i'll restart deu1/7 in a bit
2022-08-13 15:42:01 <mps> ikke: ok, fine. I'm working on 'removal' of the linux-edge4virt. will try to make linux-edge to work fine in VM
2022-08-13 20:24:49 <clandmeter> ikke: did you check?
2022-08-13 20:25:11 <ikke> clandmeter: I've been kinda busy today
2022-08-13 20:25:18 <ikke> Just back
2022-08-13 20:27:33 <clandmeter> np, just checking
2022-08-13 20:39:11 <ikke> so nld.t1, usa.t1 and sgp.t1?
2022-08-13 20:40:42 <ikke> and t1.a.o
2022-08-13 20:40:53 <ikke> Looks fine to me
2022-08-13 20:41:36 <ikke> not sure if it makes sense to host everything under *.mirror.a.o?:
2022-08-13 20:50:42 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/linode-tf/-/merge_requests/26
2022-08-13 20:59:21 <ikke> Note that the pipeline now says they will be created, as they have not been imported yet.
2022-08-13 21:06:08 <clandmeter> depends on what you want to use it for?
2022-08-13 21:16:22 <ikke> This is fine
2022-08-13 21:16:33 <ikke> I just like to overly group things
2022-08-14 09:02:41 <clandmeter> ikke: so this mr will overwrite what is in linode?
2022-08-14 09:03:33 <ikke> It depends on linode handles it exactly, but right now, it would try to add new records, as it's not aware of the existing records
2022-08-14 09:05:24 <clandmeter> lets try?
2022-08-14 09:06:04 <ikke> Sure. Note that after it's merged, you still need to run the final job manually, it will not start automatically
2022-08-14 09:06:43 <clandmeter> which final job?
2022-08-14 09:06:50 <ikke> In the pipeline for master
2022-08-14 09:07:13 <ikke> Branch pipelines only show what will change, the pipeline for master contains the job that actually applies it
2022-08-14 09:07:46 <clandmeter> you mean the deploy part?
2022-08-14 09:08:32 <ikke> yes
2022-08-14 09:08:56 <ikke> If you click on the play button, it will execute that job
2022-08-14 09:09:03 <ikke> Which you just did :)
2022-08-14 09:09:28 <clandmeter> yes i was just reviewing previous steps
2022-08-14 09:09:52 <ikke> interesting, it succeeded  :)
2022-08-14 09:10:20 <ikke> It just created additional records
2022-08-14 09:10:57 <ikke> So we now have those duplicate
2022-08-14 09:11:24 <clandmeter> yup
2022-08-14 09:11:27 <clandmeter> i see them, :)
2022-08-14 09:11:55 <ikke> The ones with a default TTL are the ones that you created manually
2022-08-14 09:12:06 <clandmeter> nod
2022-08-14 09:12:11 <clandmeter> let me delete those
2022-08-14 09:13:12 <clandmeter> gone
2022-08-14 09:13:31 <clandmeter> interesting it accepts it with the same value
2022-08-14 09:13:39 <clandmeter> just different ttl
2022-08-14 09:13:44 <clandmeter> which is actually the same
2022-08-14 09:15:00 <ikke> We should also clean up old records at some point
2022-08-14 09:15:04 <ikke> a lot of stale once
2022-08-14 09:15:06 <ikke> ones*
2022-08-14 10:07:05 <clandmeter> there is an MR for it right?
2022-08-14 10:07:15 <ikke> Some
2022-08-14 10:07:19 <ikke> For some*
2022-08-14 10:33:18 <PureTryOut> I've been on holiday for a while so I might have missed things. The riscv64 repos seem quite out of date, is the builder just slow or did something happen to it?
2022-08-14 10:33:35 <PureTryOut> Glad there is CI for it now though, even if it's emulated and just slow
2022-08-14 10:36:54 <ikke> PureTryOut: it's blocking / hanging on rust all the time
2022-08-14 10:37:14 <ikke> maybe now on gomplate
2022-08-14 10:38:22 <ikke> but it being emulated does not help
2022-08-14 10:40:25 <ikke> gomplate is still building
2022-08-14 10:46:32 <PureTryOut> yeah ok
2022-08-14 10:47:01 <PureTryOut> any chance on getting hardware at some point? some vendor that might want it?
2022-08-14 10:47:06 <ikke> In other news, rv64 now supports rust
2022-08-14 10:47:26 <ikke> PureTryOut: from what I understood, there simply does not exist any suitable hw yet
2022-08-14 10:47:35 <ikke> everything is like rpi-grade hw
2022-08-14 10:47:38 <PureTryOut> darn
2022-08-14 10:47:45 <ikke> I have one here
2022-08-14 10:50:42 <PureTryOut> oh, which one?
2022-08-14 10:50:54 <ikke> starfive
2022-08-14 10:51:21 <ikke> https://rvspace.org/
2022-08-14 10:52:40 <ikke> PureTryOut: btw, there is also #alpine-riscv64 if you're interested
2022-08-14 10:53:00 <PureTryOut> oh yes, definitely am. Thanks!
2022-08-14 10:53:50 <PureTryOut> Are there more arch specific channels like that?
2022-08-14 10:54:39 <mps> I think roma riscv notebook is on horizon
2022-08-14 10:54:48 <ikke> PureTryOut: not that I'm aware of
2022-08-14 10:56:33 <mps> server grade riscv could be expected next year
2022-08-14 10:58:47 <PureTryOut> ok thanks
2022-08-14 10:59:12 <PureTryOut> mps: https://www.theregister.com/2022/07/01/riscv_roma_laptop/ this one>?
2022-08-14 10:59:42 <mps> yes
2022-08-14 11:00:08 <PureTryOut> ok interesting
2022-08-14 11:16:21 <ikke> mps: For some reason, my wg tunnels never work after boot, I always have to restart them
2022-08-14 11:16:26 <ikke> did you notice that?
2022-08-14 11:16:39 <ikke> (apparently due to it not able to resolve wg.a.o on boot)
2022-08-14 11:16:49 <mps> hm
2022-08-14 11:17:19 <mps> I usually start it from /etc/local.d/
2022-08-14 11:18:00 <mps> but on my other machines it works from /etc/network/interfaces, not related to alpine infra
2022-08-14 11:19:04 <mps> for alpine infra I keep keys and config in 'secure' place, not on rootFS
2022-08-14 12:55:29 <mps> do 'we' have x86_64 qemu VM on infra which is booted with ovmf -bios parameter
2022-08-14 13:24:08 <ikke> I don't think we have any x86_64 vms
2022-08-14 13:34:23 <mps> ikke: ok, thanks for info
2022-08-14 13:35:04 <mps> I can't start alpine x86_64 virt iso with ovmf
2022-08-14 13:35:27 <mps> armv7 and aarch64 works fine
2022-08-14 19:57:44 <psykose> ikke: the aarch64 ci seems to take like 5 minutes just to self-upgrade or install makedepends
2022-08-14 19:58:34 <ikke> hmm
2022-08-14 20:03:42 <ikke> ci vm is not busy or something
2022-08-14 20:44:19 <psykose> same goes for the other arm ones really
2022-08-14 20:44:24 <psykose> just looks like some 10kb/s networking
2022-08-14 20:46:28 <ikke> https://tpaste.us/4oZX 
2022-08-14 20:46:49 <ikke> that's on the aarch64 runner host
2022-08-14 20:48:02 <psykose> dunno what you want me to say :) https://gitlab.alpinelinux.org/alpine/aports/-/jobs/802238
2022-08-14 20:50:59 <ikke> psykose: I saw it
2022-08-14 20:51:17 <ikke> Just trying to figure out what's going on
2022-08-14 20:53:15 <ikke> Mind if I restart docker on the host?
2022-08-14 20:56:52 <psykose> go for it
2022-08-14 20:56:58 <psykose> nothing important rn
2022-08-14 23:45:26 <psykose> still the same if you changed anything
2022-08-15 08:40:42 <mps> linux-tools fails on some arches because there is eudev-dev in cache, iiuc
2022-08-15 08:55:51 <PureTryOut> nah eudev-dev is pulled in through pcsc-lite. That one however is in cache. Remind me, when are we switching to rootbld again? 😅
2022-08-15 09:02:07 <ikke> PureTryOut: when it is actually suitable to run on builders 
2022-08-15 09:02:56 <PureTryOut> in what way is it not?
2022-08-15 11:29:49 <panekj> we're missing "net" on probably almost all go/rust aports
2022-08-15 11:30:13 <panekj> (or proper way of fetching stuff)
2022-08-15 11:52:56 <PureTryOut> Is that a problem though? Easy enough to add if a builder fails on it
2022-08-15 11:53:01 <PureTryOut> And CI could be changed first to catch it
2022-08-15 13:32:47 <clandmeter> ikke i think the new t1's are ready
2022-08-15 13:32:51 <clandmeter> can you add them to monitoring?
2022-08-15 13:33:52 <clandmeter> when you think they are healty, we can decomission the old mirrors
2022-08-15 13:35:05 <clandmeter> we probably need to reconfig fastly
2022-08-15 13:35:35 <clandmeter> or maybe just point to the current hostnames via cname to new one
2022-08-15 13:36:03 <clandmeter> and we need to update rsync.a.o to point to t1.a.o
2022-08-15 13:39:34 <clandmeter> I think we can just remove all the *.alpinelinux.org mirrors and keep only dl-cdn and rsync.a.o
2022-08-15 13:47:42 <ikke> They are still used 
2022-08-15 13:48:02 <ikke> nl.a.o dl-*.a.o
2022-08-15 13:48:11 <ikke> I see them regularly popping up
2022-08-15 14:02:36 <ikke> clandmeter: maybe we should make an announcement somewhere that these mirrors are deprecated? 
2022-08-15 14:03:14 <clandmeter> lets just keep the records, but remove them from resources?
2022-08-15 14:04:18 <ikke> From what resources? 
2022-08-15 14:04:31 <clandmeter> mirrors.a.o ie
2022-08-15 14:04:35 <ikke> Yes
2022-08-15 14:04:46 <clandmeter> thats it mostly i guess
2022-08-15 14:04:51 <ikke> Ahuh 
2022-08-15 14:05:07 <clandmeter> and whenever we bump into it we replace it with cdn
2022-08-15 14:06:17 <clandmeter> i think this was already our procedure, so not much changes regarding that.
2022-08-15 14:06:20 <clandmeter> just mirrors for now
2022-08-15 14:07:01 <clandmeter> would be nice to get some stats per region
2022-08-15 14:18:35 <ikke> Install vnstat? 
2022-08-15 14:29:33 <clandmeter> what about zabbix?
2022-08-15 14:37:38 <panekj> be sure to sweep the wiki for those subdomains
2022-08-15 14:41:12 <ikke> clandmeter: I'll add it
2022-08-15 15:32:12 <psykose> ikke: you could set CI to run rootbld
2022-08-15 15:45:34 <ikke> Not how it works currently 
2022-08-15 17:36:29 <ikke> clandmeter: any idea why /dev/null is only readable by root on usa.t1 and sgp.t1?
2022-08-15 17:36:50 <psykose> is it even a character device?
2022-08-15 17:37:01 <psykose> i've had some weird stuff where something deleted devnull before
2022-08-15 17:37:37 <ikke> yes, it is
2022-08-15 17:39:24 <psykose> strange
2022-08-15 17:39:32 <psykose> something probably just chmod'd it
2022-08-15 17:40:39 <ikke> same for /dev/full
2022-08-15 17:50:52 <ikke> clandmeter: I've added the t1 servers to Zabbix
2022-08-15 17:51:05 <ikke> clandmeter: Is there anything specific you wanted to monitor?
2022-08-15 19:33:15 <clandmeter> ikke: some regular stats i guess?
2022-08-15 19:33:19 <clandmeter> like bw 
2022-08-15 19:33:22 <ikke> Yes, I have the basic linux monitoring
2022-08-15 19:33:24 <clandmeter> disk usage should not be an issue
2022-08-15 19:33:27 <clandmeter> :)
2022-08-15 19:33:32 <ikke> clandmeter: I may hope not
2022-08-15 19:34:40 <clandmeter> i was just wondering how well the bw would be distributed
2022-08-15 19:35:23 <ikke> Probably still lopsided
2022-08-15 19:35:26 <ikke> But we'll see
2022-08-15 19:36:38 <ikke> clandmeter: do you have an idea why permissions in /dev may be messed up slightly?
2022-08-15 19:36:49 <ikke> zabbix agent failed because it could not read /dev/null
2022-08-15 19:36:52 <ikke> on sgp and usa
2022-08-15 19:37:14 <clandmeter> interesting
2022-08-15 19:37:26 <clandmeter> well these servers are generated from beta images
2022-08-15 19:37:50 <clandmeter> so anything special you bump into we should add to a ticket
2022-08-15 19:37:58 <clandmeter> then i will follow that up
2022-08-15 19:38:11 <ikke> But it's not consistent
2022-08-15 19:38:23 <clandmeter> what is the real problem?
2022-08-15 19:38:34 <clandmeter> i mean what permissions does it have?
2022-08-15 19:38:39 <clandmeter> and what is exepected
2022-08-15 19:38:44 <ikke> 660 root:root
2022-08-15 19:38:50 <ikke> 666 root:root
2022-08-15 19:39:01 <clandmeter> and nld does not have this issue?
2022-08-15 19:39:04 <ikke> No
2022-08-15 19:39:09 <clandmeter> hmm
2022-08-15 19:39:31 <ikke> I also noticed that /dev/console has group tty on nld, but not on the others
2022-08-15 19:39:37 <ikke> and probably more differences
2022-08-15 19:40:29 <ikke> clandmeter: what was your idea to use as backend for dl-cdn?
2022-08-15 19:40:33 <ikke> all 3 servers?
2022-08-15 19:40:55 <clandmeter> ok can you write them down somewhere? then i will do a fresh install of alpine on a new machine and see if its from the image or post install
2022-08-15 19:40:57 <ikke> or just t1.a.o and let anycast do the rest
2022-08-15 19:41:06 <clandmeter> good question
2022-08-15 19:41:21 <clandmeter> not sure if anycast can cause issues
2022-08-15 19:41:32 <clandmeter> what happens if one of them is out of sync
2022-08-15 19:41:56 <ikke> I suppose that is something we should monitor
2022-08-15 19:42:24 <clandmeter> we could try the anycast hostname and see what happens
2022-08-15 19:42:33 <ikke> https://zabbix.alpinelinux.org/zabbix.php?action=map.view&sysmapid=7
2022-08-15 19:42:53 <ikke> I didn't adjust the bottom part yet
2022-08-15 19:43:04 <clandmeter> is it master or dl-master?
2022-08-15 19:43:12 <ikke> Both records exist
2022-08-15 19:43:25 <clandmeter> i thought dl-master was the "official" naming
2022-08-15 19:43:45 <clandmeter> as master could be.... confusing in other concepts
2022-08-15 19:43:53 <clandmeter> and master is bad naming ;
2022-08-15 19:43:56 <clandmeter> ;-)
2022-08-15 19:44:26 <clandmeter> anyways just keep it, master should be working.
2022-08-15 19:44:57 <clandmeter> I guess we cname rsync to t1.
2022-08-15 19:45:10 <ikke> Yes, I think so
2022-08-15 19:46:09 <clandmeter> remove the other hosts from mirrors and cname them to t1 also or if they didnt support rsync we could cname them to cdn
2022-08-15 19:46:52 <ikke> Maybe we start with moving rsync
2022-08-15 19:50:41 <clandmeter> before we do that, first let them sync for a day or 2
2022-08-15 19:50:46 <clandmeter> then i will check if all is ok
2022-08-15 19:51:08 <clandmeter> i made some modifications to the compose setup for mirror
2022-08-15 19:51:19 <clandmeter> i hope i didnt break anything
2022-08-15 19:51:28 <clandmeter> i think it was semi broken
2022-08-15 19:53:58 <ikke> Ok
2022-08-15 19:54:10 <ikke> I just added standard mirror monitoring to the hosts in zabbix
2022-08-15 19:54:18 <ikke> :D
2022-08-15 19:54:42 <ikke> Need to set a macro
2022-08-15 20:04:42 <ikke> clandmeter: nld.t1 was last updated 7h ago apparently
2022-08-15 20:05:00 <clandmeter> ok let me check
2022-08-15 20:05:50 <ikke> same with the other 2
2022-08-15 20:06:02 <ikke> 15:00 utc was last time
2022-08-15 20:07:09 <clandmeter> that is weird
2022-08-15 20:07:13 <clandmeter> i see it has new files
2022-08-15 20:07:37 <clandmeter> what do you reference for time?
2022-08-15 20:07:42 <ikke> last-update file
2022-08-15 20:07:49 <ikke> last-updated*
2022-08-15 20:08:10 <clandmeter> what is the uptime time on that one?
2022-08-15 20:08:17 <clandmeter> updatetimer*
2022-08-15 20:08:28 <clandmeter> every hour or so?
2022-08-15 20:08:34 <ikke> 5m
2022-08-15 20:08:50 <clandmeter> when the file gets updated?
2022-08-15 20:09:02 <clandmeter> or when you check it?
2022-08-15 20:09:07 <ikke> when I check it
2022-08-15 20:09:26 <clandmeter> last updated is 1300
2022-08-15 20:11:01 <clandmeter> and its set to 20:00 atm
2022-08-15 20:11:08 <clandmeter> so i guess it gets updated every hour
2022-08-15 20:11:19 <ikke> there is an hourly cron indeed on dl-master
2022-08-15 20:11:37 <clandmeter> i have a cron that does a complete sync every hour
2022-08-15 20:11:43 <clandmeter> i guess that is failing
2022-08-15 20:11:51 <clandmeter> currently it specifally updates the repo that is updated
2022-08-15 20:11:56 <ikke> right
2022-08-15 20:11:59 <clandmeter> not the whole mirror
2022-08-15 20:12:05 <ikke> based on msg.a.o anouncements?
2022-08-15 20:12:09 <clandmeter> yup
2022-08-15 20:12:23 <clandmeter> just need to figure out why it does not update every hour
2022-08-15 20:12:47 <clandmeter> i would prefer to have an additional container that could trigger a full sync in the mqtt-exec container
2022-08-15 20:13:36 <ikke> Not easy without network, I suppose
2022-08-15 20:14:04 <clandmeter> interesting, it does work when i run run-parts manually
2022-08-15 20:14:15 <clandmeter> maybe it fails due to missing tty or something
2022-08-15 20:14:41 <ikke> What would require a tty?
2022-08-15 20:14:53 <clandmeter> docker compose exec
2022-08-15 20:15:01 <ikke> ah, you need -T
2022-08-15 20:15:08 <clandmeter> thats what i have :)
2022-08-15 20:15:37 <clandmeter> heh
2022-08-15 20:15:45 <clandmeter> crond is not running
2022-08-15 20:15:50 <ikke> ok, that does not help
2022-08-15 20:16:04 <ikke> but what _is_ running then?
2022-08-15 20:16:32 <clandmeter> what do you mean what is running?
2022-08-15 20:16:50 <ikke> oh, you run this on the host I guess
2022-08-15 20:17:05 <clandmeter> yes
2022-08-15 20:17:18 <ikke> btw, do you know tmux command set synchronize-panes?
2022-08-15 20:17:18 <clandmeter> its the only interface to run commands in a container
2022-08-15 20:17:49 <ikke> 
ACTION mumbles something about a socket :P

2022-08-15 20:19:10 <clandmeter> did you start crond on usa already?
2022-08-15 20:19:27 <ikke> I did not
2022-08-15 20:19:43 <ikke> I'm fumbling with Zabbix
2022-08-15 20:19:52 <clandmeter> zabbix is not in default runlevel
2022-08-15 20:20:16 <ikke> oh, right
2022-08-15 20:20:18 <ikke> forgot that
2022-08-15 20:21:37 <clandmeter> ok they all run crond now :)
2022-08-15 20:21:49 <clandmeter> need to add that to my list of fixes
2022-08-15 20:22:00 <clandmeter> i guess we enable that on default installs iirc
2022-08-15 20:23:50 <ikke> https://zabbix.alpinelinux.org/zabbix.php?action=dashboard.view&dashboardid=12
2022-08-15 20:24:01 <ikke> clandmeter: yes, we do
2022-08-15 20:26:04 <ikke> clandmeter: should we make a ticket on gitlab to keep track of these issues?>
2022-08-15 20:26:21 <clandmeter> which issues?
2022-08-15 20:26:48 <ikke> equinix alpine installation issues
2022-08-15 20:27:54 <clandmeter> yes
2022-08-15 20:27:58 <clandmeter> i just created the first one :)
2022-08-15 20:28:06 <clandmeter> https://gitlab.alpinelinux.org/clandmeter/alpine-disk-image/-/issues
2022-08-15 20:28:51 <clandmeter> if you can add the permission issues, then ill check that later.
2022-08-15 20:29:38 <ikke> Yes, was working on it
2022-08-15 20:31:25 <clandmeter> there is another strange behaviour is see different from nld and the other two
2022-08-15 20:31:49 <clandmeter> when i do docker compose exec, the output on nld is nice with correct line breaks
2022-08-15 20:32:03 <clandmeter> on the other two its scrolling from left to right without linebreaks
2022-08-15 20:32:57 <clandmeter> ah the version is different
2022-08-15 20:33:22 <clandmeter> ah and i upgraded nld to latest alpine version
2022-08-15 20:44:26 <clandmeter> ikke: are you ok if I upgrade those other two boxes?
2022-08-15 20:44:46 <ikke> clandmeter: fine with me
2022-08-15 20:44:59 <clandmeter> you only changed dev/null right?
2022-08-15 20:45:02 <ikke> yes
2022-08-15 20:46:33 <clandmeter> i think i see whats going on
2022-08-15 20:46:37 <clandmeter> there is a new mdev.conf
2022-08-15 20:48:05 <clandmeter> they replace the whole file with just one line
2022-08-15 21:03:04 <ikke> clandmeter: btw, I'm thinking about setting up our own registry
2022-08-15 21:03:15 <ikke> just for infra images initially
2022-08-15 21:03:19 <clandmeter> you mentioned it before yes
2022-08-15 22:05:52 <ikke> clandmeter: did you do that?
2022-08-15 22:06:02 <clandmeter> yup
2022-08-15 22:06:06 <ikke> ok
2022-08-15 22:06:38 <clandmeter> equinix has a bug with mdev.conf, issue is filled.
2022-08-15 22:06:43 <clandmeter> i fixed it manually
2022-08-15 22:06:48 <clandmeter> i hope
2022-08-15 22:06:52 <ikke> ok
2022-08-16 02:08:01 <ptrc> ikke: was the anitya-watch script from aports-turbo also updated? it flagged libsoup3 as outdated, even though that version is pre-release
2022-08-16 02:19:06 <ptrc> https://apps.fedoraproject.org/datagrepper/v2/id?id=4673d17c-4791-4719-bfcc-ee0a3ff17bf1&is_raw=true&size=extra-large
2022-08-16 02:19:07 <ptrc> https://pkgs.alpinelinux.org/flagged?origin=libsoup3
2022-08-16 07:30:52 <clandmeter> ikke: usa5 crash again?
2022-08-16 07:31:25 <ikke> Looks like it
2022-08-16 07:31:47 <ikke> I was just thinking how it has been stable lately 🤔
2022-08-16 07:32:05 <psykose> wonder what we should do about riscv rust
2022-08-16 07:33:12 <clandmeter> we should move the builder to lxc
2022-08-16 07:33:17 <clandmeter> and updated qemu
2022-08-16 07:33:33 <clandmeter> see if that makes a difference
2022-08-16 07:33:44 <clandmeter> i guess the issue is qemu related?
2022-08-16 07:50:01 <clandmeter> looks like usa5 is still working
2022-08-16 07:52:02 <clandmeter> something is wrong with network
2022-08-16 07:54:33 <ikke> clandmeter: it gets in some deadlocked state. Often I can still login via console, but then certain commands make it completely blocked 
2022-08-16 07:54:50 <clandmeter> i dont think thzats the case now
2022-08-16 07:54:53 <ikke> Ok
2022-08-16 07:55:08 <clandmeter> see some weird msg in ringbuffer
2022-08-16 07:55:14 <clandmeter> something with the bond
2022-08-16 07:55:30 <clandmeter> and its not related to james
2022-08-16 07:55:49 <clandmeter> jikes now the kernel is acting up
2022-08-16 07:56:49 <clandmeter> interesting, ringbuffer initially didnt mention anything
2022-08-16 07:56:56 <clandmeter> but trying some stuff make it crash
2022-08-16 07:57:47 <ikke> Heh 
2022-08-16 09:49:51 <clandmeter> ikke: now that we got those 3 t1 boxes up and running
2022-08-16 09:49:57 <clandmeter> we need to think about migrating the rest of the infra
2022-08-16 09:50:04 <ikke> Indeed
2022-08-16 09:50:37 <clandmeter> i guess the mirrors are up to date now
2022-08-16 09:50:44 <clandmeter> i dont see any red flags
2022-08-16 10:04:19 <clandmeter> ikke: did you verify if all devices are ok now?
2022-08-16 10:20:16 <ikke> clandmeter: yes, that looks better
2022-08-16 10:21:25 <ikke> https://zabbix.alpinelinux.org/zabbix.php?action=dashboard.view&dashboardid=12
2022-08-16 13:01:59 <mps> `+NAME = Hurr durr I'ma ninja sloth` https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.0-rc1 :D
2022-08-16 13:19:56 <ikke> Heh
2022-08-16 15:37:34 <ikke> ptrc: it should be all deployed at the same time, part of the same source
2022-08-16 19:56:11 <ikke> clandmeter: https://gitlab.alpinelinux.org/alpine/infra/mirrors/-/merge_requests/3
2022-08-16 21:21:22 <clandmeter> looks like already gone :)
2022-08-16 21:26:10 <ikke> ?
2022-08-16 21:27:03 <ikke> aah
2022-08-17 07:53:31 <clandmeter> ikke: https://feedback.equinixmetal.com/networking-features/p/global-ipv6-anycast-support
2022-08-17 08:09:53 <ikke> Dus nog geen support
2022-08-17 08:10:12 <clandmeter> yes no support yet
2022-08-17 08:10:27 <ikke> 😉
2022-08-17 13:55:17 <PureTryOut> uh CI on https://gitlab.alpinelinux.org/alpine/infra/docker/build-base isn't running for me
2022-08-17 13:58:15 <PureTryOut> for example, https://gitlab.alpinelinux.org/PureTryOut/build-base/-/jobs/806045
2022-08-17 13:58:26 <PureTryOut> > This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch , or no runners that match all of the job's tags: docker-alpine x86_64 
2022-08-17 15:37:49 <ncopa> i think we may need alpinelinux/alpine-gitlab-ci:3.15-x86
2022-08-17 15:38:07 <ikke> Hmm, ok. Specific reason?
2022-08-17 15:38:21 <ncopa> seems like it fails to downgrade? https://gitlab.alpinelinux.org/J0WI/aports/-/jobs/806043
2022-08-17 15:38:39 <ncopa> ERROR: libcrypto1.1-1.1.1q-r0: trying to overwrite etc/ssl1.1/cert.pem owned by ca-certificates-bundle-20220614-r2.
2022-08-17 15:39:49 <ikke> Ok, not specific to x86
2022-08-17 15:41:04 <ncopa> i think we need alpine-gitlab-ci:3.[345]-*
2022-08-17 15:41:18 <ncopa> docker images for version branch
2022-08-17 15:41:48 <ikke> Right, I've been trying to avoid that if possible :)
2022-08-17 15:42:17 <ikke> But if necessary, I'll work on that
2022-08-17 15:42:34 <ncopa> other option is to add a replaces in libcrypto1.1
2022-08-17 15:44:28 <ncopa> PureTryOut: you need to enable shared runners here: https://gitlab.alpinelinux.org/PureTryOut/build-base/-/settings/ci_cd
2022-08-17 15:44:32 <ncopa> I can do that for you
2022-08-17 15:57:12 <ncopa> ikke: do you have other ideas how to solve it? I think the proper thing is to have docker images per release branch
2022-08-17 15:57:15 <PureTryOut> ncopa: oh thanks, that's a TIL. That wasn't needed for aports 🤔
2022-08-17 15:57:33 <ikke> PureTryOut: was this an old fork?
2022-08-17 15:57:35 <ncopa> PureTryOut: someone probably enabled it for you
2022-08-17 15:57:48 <ikke> New forks should have it enabled by default
2022-08-17 15:58:20 <PureTryOut> aports? Yeah think so, from first day we switched to Gitlab from Github
2022-08-17 15:58:26 <ikke> ncopa: well, it was nice being able to properly downgrade
2022-08-17 15:58:34 <ikke> PureTryOut: build-base
2022-08-17 15:58:42 <PureTryOut> Oh no that was newly made
2022-08-17 15:58:47 <ikke> jhmm, ok
2022-08-17 15:59:41 <ikke> oh, it's because projects under the docker namespace use group runners
2022-08-17 15:59:50 <ikke> not shared runners
2022-08-17 16:00:44 <ikke> ncopa: having images per release complicates things a bit, making sure it's supported for every new release etc
2022-08-17 16:00:55 <ncopa> understand
2022-08-17 16:01:26 <ikke> But if necessary, we can make it happen
2022-08-17 16:01:49 <ncopa> do you ahve other ideas who to fix it?
2022-08-17 16:02:07 <ikke> other than the replaces=?
2022-08-17 16:02:16 <ncopa> we just add a replaces=ca-certificates-bundle in libcrypto1.1?
2022-08-17 16:06:11 <ncopa> i think we may need a replaces in ca-certificates-bundle as well
2022-08-17 16:07:04 <ikke> It's specically an issue between edge and 3.15
2022-08-17 16:07:17 <ncopa> oh, its edge..
2022-08-17 16:07:22 <ikke> yes
2022-08-17 16:07:31 <ikke> 3.16 does not have /etc/ssl1.1
2022-08-17 16:07:32 <ncopa> i saw alpine....:latest-x86
2022-08-17 16:07:39 <ikke> https://pkgs.alpinelinux.org/contents?file=certs&path=%2Fetc%2Fssl1.1&name=&branch=v3.16
2022-08-17 16:07:49 <ncopa> and thought it was 3.16
2022-08-17 16:11:10 <ncopa> ERROR: ca-certificates-bundle-20220614-r2: trying to overwrite etc/ssl1.1/cert.pem owned by libcrypto1.1-1.1.1q-r0.
2022-08-17 16:11:10 <ncopa> ERROR: ca-certificates-bundle-20220614-r2: trying to overwrite etc/ssl1.1/certs owned by libcrypto1.1-1.1.1q-r0.
2022-08-17 16:11:31 <ncopa> yup. we need a replaces
2022-08-17 16:12:22 <ikke> ncopa: I think alpine is the exception where :latest does not mean actual latest :P
2022-08-17 16:18:18 <ncopa> ok, i have another workaround: apk add --upgrade ..... || apk fix
2022-08-17 16:19:28 <ncopa> i think we can do that in our alpine-gitlab-ci image
2022-08-17 16:20:33 <ikke> right
2022-08-18 00:12:20 <psykose> that's not a very good workaround because it's useful the other 95% of the time on edge to find things that actually conflict
2022-08-18 00:12:31 <psykose> and fix them
2022-08-18 15:25:52 <ikke> cleanup time ^ :)
2022-08-20 07:25:21 <mps> what happened with abuild rootpkg. I always get `make: *** No rule to make target 'install'.  Stop.`
2022-08-20 07:26:52 <ikke> nothing happened to it
2022-08-20 07:27:43 <mps> ok, what is cause then
2022-08-20 07:28:08 <ikke> No idea?
2022-08-20 07:28:32 <mps> just did 'apk upgrade' nothing else
2022-08-20 07:54:47 <panekj> that sounds like broken package()
2022-08-20 07:55:26 <panekj> what you are trying to rootpkg doesn't have 'install' in Makefile
2022-08-20 07:55:27 <psykose> or builddir
2022-08-20 08:11:29 <mps> panekj: tried with more pkgs
2022-08-20 08:12:46 <psykose> what package
2022-08-20 08:13:08 <mps> haproxy for example
2022-08-20 08:14:02 <psykose> https://img.ayaya.dev/f30EO4qfqXDM
2022-08-20 08:16:28 <mps> I see, but doesn't work in my dev lxc
2022-08-20 08:17:10 <ikke> something in your environment?
2022-08-20 08:17:38 <mps> nothing changed for long
2022-08-20 11:08:48 <mps> Newbyte: ping
2022-08-20 11:09:54 <mps> #13877
2022-08-20 11:10:55 <psykose> ping for what
2022-08-20 11:11:43 <mps> psykose: ^
2022-08-20 11:11:53 <mps> issue above
2022-08-20 11:11:56 <psykose> you can respond in the issue page?
2022-08-20 11:12:18 <mps> and you think I don't know that? ;p
2022-08-20 11:12:22 <psykose> apparently?
2022-08-20 22:40:00 <ptrc> mps: fyi, you forgot to change the filename of the asahi tarball 
2022-08-21 07:27:50 <mps> ptrc: I see. forgot to copy local changes to dev lxc, thank you for noticing
2022-08-25 18:06:51 <ikke> gitlab security announcement 
2022-08-25 18:08:07 <ikke> https://about.gitlab.com/releases/2022/08/22/critical-security-release-gitlab-15-3-1-released
2022-08-25 18:10:41 <psykose> time for a bunch of upgrades
2022-08-25 18:14:46 <ikke> I don't have access to a pc :-) 
2022-08-25 18:16:07 <psykose> if only i did :p
2022-08-25 18:16:47 <mps> what is 'pc'?
2022-08-26 07:56:50 <ikke> Linode had issues again in gbr
2022-08-26 07:58:55 <psykose> yep
2022-08-26 07:58:59 <psykose> didnt last too long
2022-08-26 11:34:46 <PureTryOut> uh, CI seems broken. It can't find any changes and thus all arches are failing. https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/38246/pipelines
2022-08-26 11:35:56 <psykose> no, someone just pushed a syntax error without checking it
2022-08-26 11:36:20 <psykose> (is fixed)
2022-08-26 11:39:38 <PureTryOut> ah ok
2022-08-31 14:29:58 <ikke> Hmm, we now have a deadline for the equnix legacy bare-metal servers. So we need to make effort to migrate our infra 
2022-08-31 14:30:27 <ikke> November 30th
2022-08-31 14:30:53 <ikke> clandmeter: ^