2025-03-02 19:06:17 <ikke> lesigh
2025-03-02 19:54:03 <clandmeter> whats happening? very colorful here
2025-03-02 19:55:26 <ikke> Lots of scraping, but I'm not sure what exactly is causing the load issues
2025-03-02 19:56:03 <ikke> Tried increasing the amount of workers, but that just increases the amount requests being able to be handled
2025-03-02 19:56:25 <ikke> Not sure if it's just plain crawling that's the issue, or that there are specific requests which are heavy
2025-03-03 09:37:04 <qaqland> oh, git.a.o is 502
2025-03-03 09:38:41 <pj> ikke: any particular ip ranges that are guilty?
2025-03-03 10:07:13 <ikke> pj: I see some ranges making the most requests, but I'm not entirely sure those requests cause most of the load 
2025-03-03 10:07:28 <ikke> I need to quantify that somehow 
2025-03-03 10:14:39 <pj> there's been an ongoing problem for gitea/forgejo instances where bots keep scraping git archive tarballs which blows up their git archive cache
2025-03-03 10:16:14 <pj> an example from admin of my fedi instance: https://donotsta.re/notice/AreSNZlRlJv73AW7tI
2025-03-03 10:23:42 <ikke> I may start sending the access logs to loki, may help to make it easier to analyse the requests
2025-03-03 11:10:40 <ncopa> not sure if we could do something like this? https://fosstodon.org/@dalias@hachyderm.io/114055514402836782
2025-03-03 11:16:55 <ikke> I'm already doing something like that with nginx
2025-03-03 11:17:10 <ikke> well, not a download speed, but more rate limit delay
2025-03-03 11:17:57 <ikke> But, understanding where the slowness comes from is step 1
2025-03-03 15:17:10 <invoked> if you find out it's from crawlers, it's a bit of whack-a-mole listing ips, but even where i see site operators doing that, they drop requests instead of returning something, which probably just gets the crawl job moved to a different queue/ip/asn. i personally favor returning 402.
2025-03-03 15:17:31 <invoked> in any case site operators need to get better about working together because this is war
2025-03-03 15:27:54 <ikke> invoked: agree
2025-03-04 00:33:51 <francis[m]> From Trusted and Vouched Dealers... (full message at <https://matrix.org/oftc/media/v1/media/download/Ab25Tr5I39SorVcgwfZVX9KjvK5hzFkXPqaq6zPoStn6FhS0eryRf22tO0285mYsT2Q5DC1Fzo_9Fxdcfs8ItX9CeVpqvJpwAG1hdHJpeC5vcmcvVmJqenFjUFRnSlRQT0VWQnBvQkZ5UmNs>)
2025-03-04 00:34:26 <fossdd[m]> pj
2025-03-04 00:34:34 <fossdd[m]> :3
2025-03-04 00:34:44 <pj> i can't do anything, no op here
2025-03-04 00:34:58 <fossdd[m]> oh damn
2025-03-04 00:38:23 <pj> maybe I could reactive my mjolnir instance that I used for lapce
2025-03-04 00:38:39 <pj> would make it easier to ban across all channels
2025-03-04 00:39:31 <fossdd[m]> they are all coming from :matrix.org, if they could just spend their money right and ban them immediatley
2025-03-04 06:38:18 <ikke> pj: seems to be lots of addresses from brazil
2025-03-04 06:38:21 <ikke> different ISPs
2025-03-04 06:38:46 <ikke> 45.x.y.z
2025-03-04 06:39:42 <ikke> many requests from unique addresses
2025-03-04 06:43:46 <pj> is that like consistent scraping?
2025-03-04 06:43:52 <ikke> seems to be
2025-03-04 06:44:03 <ikke> That was just the last 5000 requests
2025-03-04 06:44:11 <ikke> and I've seen that before as well
2025-03-04 12:31:09 <ikke> Took some more drastic measures, see how that helps
2025-03-04 12:32:15 <ikke> Regarding git.a.o that is (deu1-dev1)
2025-03-04 12:32:50 <ikke> I'm not that impressed with loki at the moment
2025-03-04 12:33:31 <ikke> Something very simple like getting the amount of requests per IP address proves extremely difficult
2025-03-04 12:40:46 <ikke> Put a lot of ASNs in a single rate-limit bucket
2025-03-04 12:41:29 <ikke> I now see an average of 7 429 responses per second
2025-03-04 12:42:08 <vkrishn> git.a.o is gbr-app-1. here
2025-03-04 12:42:36 <pj> opensearch dashboards is much nicer but it's heavy to host
2025-03-04 12:42:52 <ikke> vkrishn: not for me..
2025-03-04 12:43:50 <vkrishn> some routing geo- based? gbr-app-1. is a.o owned?
2025-03-04 12:44:04 <ikke> yes, it is
2025-03-04 12:44:10 <ikke> but it does not host git.a.o
2025-03-04 12:46:54 <vkrishn> ok, seems it redirected from it, as both gbr-app-1. (1st), then deu1-dev1. (2nd)
2025-03-04 12:48:26 <ikke> The load on deu2-dev1 decreased significantly
2025-03-04 12:59:15 <vkrishn> got it, https://wiki.alpinelinux.org/images/alogo.svg on git.a.o webpage connects to gbr-app-1.
2025-03-04 13:07:58 <vkrishn> general rule for modern search engine A reviting same commit-id on git repo should be considered bad(dos attach like)
2025-03-04 13:08:15 <vkrishn> revisiting^
2025-03-04 13:09:17 <vkrishn> search engine should learn its a git repo and not visit same commit
2025-03-04 14:09:58 <ikke> The current traffic comes from many residential ip addresses from many ISPs. Sounds like a botnet or some vpn abusing customers 
2025-03-04 14:14:12 <vkrishn> if i had lots of slurps (brain freeze) with pizza, I would live publish those logs (with some parts scrubbed)
2025-03-04 14:17:25 <vkrishn> i also create lists of IPs that concoct urls(forge) for pages/services
2025-03-04 14:18:01 <vkrishn> i found quite a few from MS
2025-03-04 14:25:22 <vkrishn> also look for IPs that do POSTs on pages/services that don't exist
2025-03-04 16:14:01 <ikke> Similar to gitlab
2025-03-04 16:14:14 <ikke> a long tail of ip addresses from different countries all making a single request
2025-03-04 16:15:35 <ikke> But even from apple, and microsoft apparently
2025-03-04 16:27:08 <vkrishn> been there, follow the trails aka breadcrumbs ;)
2025-03-04 16:27:49 <vkrishn> interesting events unfold
2025-03-04 16:29:23 <vkrishn> make sure to keep the ip list trails or if possible temporarily backup those logs
2025-03-04 16:35:41 <vkrishn> these are few reasons to have only IPv6 rolled globally, and every living human have 2fs(ff) ip assigned as digital/path ID
2025-03-04 16:36:39 <iggy> maybe time to switch to allowlist vs denylist? (I'm only partially kidding)
2025-03-04 16:38:51 <iggy> wonder how hard it'd be to engineer some sort of system of IP web-of-trust or maybe rate limit everyone and then non-abusers get extra tokens in their bucket
2025-03-04 16:41:56 <vkrishn> there were some methods/suggestion in hachyderm url posted yesterday, 
2025-03-04 16:42:42 <vkrishn> i.e deny them and invoke/revoke based on some rule(maybe time)
2025-03-04 16:43:48 <ikke> Would exclude too many legitimate users
2025-03-04 16:45:15 <vkrishn> i noticed quite a lot of crawlers are crond so would visit your IP segment at perticular time
2025-03-04 16:46:03 <vkrishn> that's when it should be locked out
2025-03-04 16:46:18 <vkrishn> its not easy initially
2025-03-04 16:46:49 <ikke> The hard part is determining the traffic is legitimate or harmful
2025-03-04 16:47:06 <ikke> And mostly, what traffic is causing the issues
2025-03-04 16:47:36 <vkrishn> but gradually a foggy IP map gets somewhere in your mind
2025-03-04 16:47:49 <vkrishn> start small
2025-03-04 16:48:21 <ikke> Yes, that's what I'm already doing
2025-03-04 16:53:38 <vkrishn> isn't there a nginx module that slurp streams all css/js into single request?
2025-03-04 16:55:34 <vkrishn> add that to cache(varnish), might smoothen out some more
2025-03-04 17:18:20 <ikke> I must say I really like the geo module of nginx
2025-03-04 17:29:52 <ikke> I think I have at least shedded some of the load now
2025-03-04 17:29:55 <gplessis> Hi there! I was investigating an issue we - as a security business - are experiencing while pulling information from secdb.alpinelinux.org (originally every 6 hours, descreased it to once a day). I noticed logs from this channel saying that you are experiencing some load problems because of heavy crawling / scraping. We are hosted on Azure and may have been caught in your most recent protection 
2025-03-04 17:30:01 <gplessis> actions, likely because of noisy and not very mindful neighbors using the same IP ranges.
2025-03-04 17:30:22 <gplessis> Is there anything we can do to help, and to get our connectivity to secdeb restored?
2025-03-04 17:30:25 <ikke> gplessis: Ok, would you mind sending the ip adres in a dM?
2025-03-04 17:30:31 <ikke> / subnet
2025-03-04 17:31:24 <gplessis> sure
2025-03-04 17:32:40 <ikke> Ok, I've removed the block
2025-03-04 17:33:14 <gplessis> thank you very much for your help!
2025-03-04 20:28:24 <ikke> Seems like the rate limiting does have effect
2025-03-04 20:44:14 <vkrishn> how would you describe it from "crawlers vantage point" ;)
2025-03-04 20:46:50 <ikke> From their vantage point, they are certainly affected
2025-03-04 20:51:36 <vkrishn> :), i mean what would gptAI say to itself when visiting a.o sites
2025-03-04 20:52:06 <ikke> This tastes rather bland
2025-03-04 20:52:11 <vkrishn> nm, offtopic
2025-03-04 20:52:14 <vkrishn> :)
2025-03-04 21:01:22 <vkrishn> looks like your allowlist has got its first entry
2025-03-04 22:19:32 <sertonix[m]> I think I got ratelimited too. Loading any gitlab.a.o page results in 429 for some requests which cases the page to be broken.
2025-03-05 05:45:43 <ikke> sertonix[m]: should be fixed now
2025-03-05 09:24:30 <qaqland> i inquired with sjtug and they are changing and working on architecture migration.
2025-03-05 15:41:30 <ikke> qaqland: thanks
2025-03-06 18:51:46 <vkrishn> ikke: an example of IPs that fall in not good list, https://tpaste.us/Ro9Y 
2025-03-06 18:52:12 <ikke> That's Alibaba Cloud
2025-03-06 18:52:14 <ikke> already covered
2025-03-06 18:52:42 <ikke> Oh no, this one is different
2025-03-06 18:54:40 <vkrishn> its trying to get, GET /static../.git/config , something that don't exist,
2025-03-06 18:54:57 <ikke> Yeah, trying to find accidentally exposed credentials
2025-03-06 18:54:59 <vkrishn> sometimes i sit and scan looking for similar entry in logs
2025-03-06 18:55:21 <ikke> But right now, that kind of traffic is not what I'm focussing on
2025-03-06 18:56:00 <ikke> And, at least for now, it appears things are under control
2025-03-06 18:56:57 <vkrishn> yes, thanks for the effort, noticed some improvement
2025-03-07 09:53:21 <darim> Hi, I'm trying to register an account on the Alpine GitLab, but I'm not receiving the confirmation email. I've tried different email providers, but none of them worked. I also tried signing up with GitHub, but that didn’t work either. Is there an issue?
2025-03-07 10:00:05 <ikke> darim: I can check in an hour 
2025-03-07 10:00:59 <ikke> The only issue I'm aware of is outlook.com blacklisting us
2025-03-07 10:04:01 <darim> Okay, that might be the issue. My email is hosted on Outlook.com, and my GitHub account is also linked to an Outlook.com email address. I will try it with gmail.
2025-03-07 15:05:09 <mio> is it normal to have a ~25 s wait between new page loads to gitlab.a.o sometimes? this has improved over the past few days, just wondered if my ip was caught in some filtering
2025-03-07 15:37:27 <ikke> mio: are you opening a lot of pages at the same time?
2025-03-07 15:38:51 <mio> 2-3 tabs?
2025-03-07 15:39:40 <mio> it occurs on the first load usually, or refreshing same tab after some time
2025-03-07 16:09:48 <ikke> mio: that should not trigger it
2025-03-07 16:10:00 <ikke> The IP address I see for you is not rate limited
2025-03-07 16:11:17 <ikke> mio: if you open the network tab, you should see 429 responses if you're rate limited
2025-03-07 16:13:41 <mio> ikke: okay, thanks!
2025-03-07 16:15:27 <mio> have had 1 tab open of the merge_requests page that is refreshed a few times an hour or a few hours, and 1-2 new tabs potentially opened from that
2025-03-07 16:16:35 <mio> the list refresh seems to have the wait, then reading MRs after that initial period is fine for a while
2025-03-07 16:18:05 <mio> will try to keep the number of open tabs low and see if that helps, thanks
2025-03-07 16:18:18 <mio> (based on what you mentioned)
2025-03-07 17:16:48 <vkrishn_> i use mouse, but noticed that there are key-bindings to pages too, it slightly faster that F5(full refresh)
2025-03-07 17:44:07 <mio> interesting, will try that as well, thanks
2025-03-07 18:23:52 <ikke> mio: I sometimes open 8 tabs in sequence without issue
2025-03-07 18:41:23 <mio> ikke: okay, good to know. just switched to firefox, page loading seems to be fine so far
2025-03-07 18:42:55 <mio> maybe my other browser is having some issue or cached some old setting
2025-03-10 19:50:41 <ikke> I have cleaned up distfiles a bit
2025-03-12 01:53:45 <vkrishn> ikke: around? can i lock-discussion to a TSC issue?
2025-03-12 01:54:02 <vkrishn> pls lock-it, https://gitlab.alpinelinux.org/alpine/tsc/-/issues/93
2025-03-12 01:54:07 <vkrishn> seems i cannot
2025-03-12 01:54:10 <vkrishn> thanks
2025-03-12 01:55:18 <vkrishn> finally, really wanted to get off irc, now got a reason ;)
2025-03-12 02:23:14 <vkrishn> seems there is no export to pdf to take a backup of the issue, so just copy-pasting the page for backup
2025-03-12 05:17:56 <vkrishn> nice, the brower gave a nice screenshot too - for the TSC issue page
2025-03-12 14:10:08 <fossdd[m]> looks like gitlab is sending me 5x the same email again
2025-03-12 14:10:15 <fossdd[m]> i think i noticed that yesterday already
2025-03-12 14:30:20 <vkrishn> psykose would be _alice here?
2025-03-12 14:30:32 <vkrishn> https://d.insteps.net/scr/screenshot-2025-03-12-at-06-59-25-TSC-93.png, pls see there is not reply btn
2025-03-12 14:32:18 <durrendal> fossdd[m]: I just noticed that as well, I just assumed gitlab REALLY wanted me to know my CI job failed
2025-03-12 14:32:30 <fossdd[m]> haha
2025-03-12 14:32:43 <vkrishn> i am planning 2 more TSC, please enable them this time, play nice pls (i have been jotting the points for all 3 for yrs), they have no linking to current irc postings
2025-03-12 14:33:46 <vkrishn> durrendal: wanted to say but missed, a tester has can be independant role and not associated with AL
2025-03-12 14:33:54 <ikke> Please keep the scope reduced 
2025-03-12 14:33:55 <vkrishn> a tester can^
2025-03-12 14:34:03 <vkrishn> ok sorry
2025-03-12 14:35:29 <vkrishn> in img, seems all participants said all at once and were talking to themselves
2025-03-12 14:36:13 <vkrishn> i can also formulate "TSC: Rules off engagement" if gitlab is buggy
2025-03-12 14:36:20 <durrendal> vkrishn: yes that makes sense, I only meant yesterday that by participating in some of the current maintenance workflow you might be better able to emphasize that difference.
2025-03-12 14:37:12 <fossdd[m]> gitlab is not buggy? you know that comments happen in threads, right? https://docs.gitlab.com/user/discussions/
2025-03-12 14:38:17 <vkrishn> yes, but if you reply , new reply btn should be on it, so that sub-reply can be a hierarchy
2025-03-12 14:39:00 <vkrishn> but if 2 reply happens on same orig posts, the replay field is in the end
2025-03-12 14:41:23 <vkrishn> which means, the 2nd replier has to wait(rules of enagement)
2025-03-12 14:41:40 <vkrishn> reading the docs now
2025-03-12 14:43:05 <vkrishn> or I can leave the page unlocked for 12hrs, get reply, lock it, do my reply, unlock
2025-03-12 14:43:20 <vkrishn> cycle repeat
2025-03-12 14:43:42 <vkrishn> but need privilege to do locking/unlocking
2025-03-12 14:48:51 <vkrishn> just fun, but why use "hearts up" instead 'thumbs up', sounded like social forum (no hard feelings as its a feature)
2025-03-12 14:52:28 <vkrishn> Formulated rules like these, will always feel like an 'outsider thing', it cannot have partialiaty in it
2025-03-12 14:53:20 <vkrishn> fight it at its tech point
2025-03-12 14:53:43 <vkrishn> partiality^
2025-03-12 14:56:36 <vkrishn> its targetted towards a running system, not one person
2025-03-12 15:00:04 <vkrishn> if AL survives the next generation(AI infested era) in its principals, your kids would thank you
2025-03-12 15:07:52 <invoked> vkrishn: my suggestion would be to focus (constrain) what you're proposing to things you've worked out some basic concept how it would be implemented (with gitlab or a tool, that ideally adds no additional steps to what everyone is currently doing). cut out everything else (or set it aside for later)
2025-03-12 15:08:51 <invoked> this process is about getting buy-in from others
2025-03-12 15:09:39 <invoked> i could be wrong, but anything that drastically changes -- or adds steps to -- what everyone currently does, is likely to face a bunch of resistance
2025-03-12 15:12:19 <vkrishn> why does Formulated rules always look like corporate/business :), coz they are never implemented at lower level, need costs to do it
2025-03-12 15:12:44 <invoked> the reasons don't matter. it's your job to figure out how to get the buy-in, that's all i'm saying
2025-03-12 15:14:11 <invoked> if enough people are giving you a clear negative signal on something, you need to rework it or maybe cut it out.
2025-03-12 15:17:17 <vkrishn> tech feasibility or not, I am open to ikke's suggestions, can only suggest when tools have been selected, based on available funds
2025-03-12 15:19:18 <invoked> there are enough tools. it's probably more about "how would $foo be done with the tools we already have"
2025-03-12 15:21:08 <invoked> if you can answer that question on each point of what you're proposing, and it doesn't much impact what people are currently doing, you'll probably face less resistance
2025-03-12 15:21:11 <vkrishn> can you pick one small $foo and its tools if possible, let me think on it
2025-03-12 15:22:08 <vkrishn> meaning answer with tech codes/implimented details?
2025-03-12 15:23:42 <invoked> just start with gitlab. for example: https://gitlab.alpinelinux.org/alpine/infra/aports-testing-ttl
2025-03-12 15:24:08 <vkrishn> that part is already in effect with TSC #2
2025-03-12 15:24:13 <invoked> look at past issues and how the team has solved them in the past
2025-03-12 15:24:26 <vkrishn> wait, sorry, let me check
2025-03-12 15:25:02 <vkrishn> yup
2025-03-12 15:25:59 <ikke> Ftr, it's not in effect yet
2025-03-12 15:27:26 <vkrishn> yes, thanks need to know, but that is not something new added in
2025-03-12 15:33:50 <vkrishn> ikke: just pick any contigous part and say - feasible or not feasible
2025-03-12 15:34:14 <vkrishn> will start on it
2025-03-12 15:35:09 <vkrishn> but then i think core TSC team has to ad-hoc pass it first
2025-03-12 15:46:34 <invoked> most people (i assume) prefer to see code. like: "here's a subset of these initial ideas, and here's a script that implements & automates those".  even if you do that (as you can see with the link i sent) it will take time to be accepted and then made active
2025-03-12 15:47:47 <invoked> even then some may want stuff taken out...
2025-03-12 15:48:37 <vkrishn> yes, that second part is what is closer to #93:)
2025-03-12 15:48:47 <vkrishn> AH!
2025-03-12 15:50:23 <invoked> so just... mentally: think about how each thing works, with code, and what you're asking of people who are already used to working the current way. summarize succinctly how this helps anything, and be clear about how it would change how people currently work (if it does)
2025-03-12 15:50:52 <invoked> right or wrong, people get annoyed by big proposals, especially when there's no code
2025-03-12 15:53:14 <vkrishn> summarize it, will do, just need to pick aleady replied into one section
2025-03-12 15:54:05 <vkrishn> already^
2025-03-12 15:55:34 <vkrishn> will do in context of current running infra
2025-03-12 15:56:33 <vkrishn> ikke: would need just small info helps, was thinking to put in https://gitlab.alpinelinux.org/alpine/tsc/-/issues/82, but ask here if ok?
2025-03-12 16:02:56 <durrendal> I don't think rehashing issue 93 in issue 82 is the way to go. 82 is focused on finding a technical solution to our build infrastructure, 93 is more documentation/process oriented.
2025-03-12 16:03:16 <durrendal> thanks for trying algitbot
2025-03-12 16:04:35 <vkrishn> no, was going to ask  to give a rough diagramatic sketch (nothing fancy) of current infra
2025-03-12 16:04:58 <vkrishn> current bump codes that maintainers use
2025-03-12 16:07:02 <durrendal> I don't believe we currently have any topology diagrams that are accessible generally. But I can speak somewhat to the infrastructure, what do you want to know?
2025-03-12 16:07:58 <vkrishn> yes, its probably need to be done. Void linux has a webpage like that
2025-03-12 16:08:18 <vkrishn> but just rough sketch
2025-03-12 16:09:01 <vkrishn> i need to understand the IO flow mostly, thats all
2025-03-12 16:09:47 <vkrishn> I think i mentioned it here, before
2025-03-12 16:10:16 <vkrishn> yes, when i asked for signed key-flow
2025-03-12 16:11:37 <vkrishn> gtg, need to check on my other router, seems not ok, will check logs for link
2025-03-12 16:12:12 <durrendal> we're currently migrating away from Equinix metal, so our infra is in flux currently.
2025-03-12 16:13:10 <durrendal> anyways, since you'll check the logs and see this, I'll continue
2025-03-12 16:14:04 <durrendal> Someone makes an MR making a change to an aport, this could be anything from a new package to a metadata change. Lets say I need to upgrade sbcl from 2.5.1 to 2.5.2
2025-03-12 16:14:34 <durrendal> I'd bump pkgver in the APKBUILD in my MR, it would then queue up a series of CI jobs across all the enabled cpu arches supported by that package, build it and test it
2025-03-12 16:15:19 <durrendal> then someone with developer access would come along and review the MR, this would be folks like fossdd[m], they'd let me know if I made a mistake or did something incorrectly, advise on best practice
2025-03-12 16:15:48 <durrendal> once ready, the MR would be merged. Only a few key folks have merge access. Those people are vetted before they get this level of access.
2025-03-12 16:16:11 <durrendal> Once the MR is merged it is queued to the builders via an MQTT pub/sub broker.
2025-03-12 16:16:43 <durrendal> When the builder sees a new package is available it attempts to build it, upon success it signs and publishes the built artifact using rsync.
2025-03-12 16:17:24 <durrendal> https://build.alpinelinux.org/ <- you can watch that process here if you'd like
2025-03-12 16:18:06 <durrendal> and if you happen to know the name/version of a package, what repo it comes from, and what version of Alpine it's being built for you can see the log from the build process
2025-03-12 16:18:08 <durrendal> https://build.alpinelinux.org/buildlogs/build-3-21-x86_64/community/woodpecker/woodpecker-2.8.0-r2.log
2025-03-12 16:18:48 <durrendal> this information is accessible via the MQTT broker, you can sub to it if you'd like and watch the build process.
2025-03-12 16:19:35 <ikke> ftr, #alpine-infra is not the best channel for these discussions
2025-03-12 16:21:08 <durrendal> that's an extremely valid point, this would be better in #-linux or #-devel since we're only tangentially talking about infrastructure here. Normally people would field "how does this work" questions there.
2025-03-12 16:23:51 <durrendal> sorry for the noise ikke, if anything else comes up and I'm around I'll try to redirect it elsewhere.
2025-03-12 17:28:51 <vkrishn> durrendal: thanks, will chew on it
2025-03-12 17:29:27 <vkrishn> btw the way, try a analytics done from MQTT , https://b.infosight.in/alpine
2025-03-12 17:31:06 <vkrishn> after password input, re-hit on the url input area
2025-03-12 17:31:59 <vkrishn> open->mqtt build edge
2025-03-12 23:25:10 <invoked> ikke: there are not, as far as i can tell, any bang-bang ways to convert antora playbooks to mkdocs.
2025-03-12 23:25:44 <invoked> minor styling issues aside, i don't think it would be a lot of work to convert them manually, and i can do that.
2025-03-12 23:26:22 <invoked> but i'll hold off on that for now, let me know what you want to be done
2025-03-12 23:28:44 <invoked> my opinion is that mkdocs would probably be easier to own. but i'm not a tech writer.
2025-03-13 16:14:58 <mio> totally non-urgent question, can someone confirm whether a certain ip address was caught in filtering for gitlab.a.o? did some troubleshooting on my connection issue, think might have accidentally gotten an ip rate-limited early on and hadn't realised
2025-03-13 16:16:49 <ikke> mio: can you DM the address?
2025-03-13 16:19:01 <mio> dm-ed ...  found a workaround to the load delay was seeing earlier, but would be good to know for future reference
2025-03-13 16:27:22 <mio> issue resolved over dm, thanks ikke
2025-03-13 17:14:53 <durrendal> ikke: speaking of rate limiting and mio's connection, had a couple of questions spring to mind while mio and I were troubleshooting their connection issues yesterday. Is gitlab on k0s now? Do we have a dualstack ipv4/v6 load balancer in front of it, and do is the VPC network also dualstack? All just curiosity on my part.
2025-03-13 17:15:27 <ikke> No, gitlab is still runing in docker
2025-03-13 17:15:38 <ikke> No VPC or anything, just plain connectivity
2025-03-13 17:17:55 <durrendal> Ah so our AAAA points directly to the server running the gitlab container then, that makes sense.
2025-03-13 17:18:23 <ikke> It points directly to the ip address of the reverse proxy even ;-)
2025-03-13 17:18:31 <ikke> (traefik container)
2025-03-13 17:20:11 <durrendal> haha of course, that makes a lot more sense :)
2025-03-14 10:31:19 <ikke> Hmm..
2025-03-14 11:37:25 <pj> bots?
2025-03-14 11:38:07 <ikke> Yes, most likely
2025-03-14 11:38:19 <ikke> But all coming from different IPs
2025-03-14 11:38:29 <ikke> max 2 requests per IP
2025-03-14 17:24:03 <ikke> This must be some crazy botnet
2025-03-14 17:42:12 <pj> I've seen some mentions of LLM scraping getting worse
2025-03-14 17:56:49 <ikke> Yeah, but this seems like someone is purchasing botnets for scraping
2025-03-14 20:32:00 <clandmeter> ikke: why does it return 502?
2025-03-14 20:32:36 <clandmeter> just now the cpu load was not that high
2025-03-14 20:32:46 <ikke> Because the workers are busy
2025-03-14 20:34:22 <ikke> I've been rate limiting quite some subnets already
2025-03-14 20:35:21 <clandmeter> should we try to put cloudflare in front if it?
2025-03-14 20:35:27 <clandmeter> or wouldnt that matter
2025-03-14 20:37:49 <ikke> It wouldn't be transparent (ie, getting "Verifying you're a human preloaders")
2025-03-14 23:19:42 <sertonix[m]> The gnome gitlab has added something against AI scrapers. Is that something that could be used?
2025-03-15 03:59:05 <ikke> sertonix[m]: reference?
2025-03-15 03:59:31 <ikke> I guess I saw it, some javascript
2025-03-15 07:43:34 <sertonix[m]> ikke: Yes. If you are fast enough (or the browser is slow enough) you get to this link: https://xeiaso.net/blog/2025/anubis/
2025-03-15 15:21:17 <invoked> unfortunate it's come to that. (which consequently prevents simple browsers/browsing without js -- ie even if that is a small % of users, it's collateral damage)
2025-03-15 15:59:51 <sertonix[m]> When you change your user agent to not contain "Mozilla" anubis would let you pass without needing js.
2025-03-15 16:30:11 <fossdd[m]> really? i think gnome infra itself has some heavy limitations on user agents (has to be the latest chrome etc) but didnt think that anubis itself would restrict that
2025-03-15 16:32:50 <sertonix[m]> anubis doesn't restrict user agents. If you want to know how it works just read the blog post I linked
2025-03-15 17:28:53 <invoked> seems like an obvious hole, which assumes people working on the crawlers won't read the same blog post
2025-03-15 17:32:33 <invoked> it's been ~14 years since i've written a crawler (i did write one that successfully crawled over a billion pages), but then again i wrote one that respected robots.txt. the ai crawlers don't respect boundaries, and i assume they'd quite readily switch user agent if they knew.
2025-03-15 17:44:57 <ikke> I wonder if it works in this case though
2025-03-15 17:45:03 <ikke> single page requests from arbitrary IPs
2025-03-15 21:07:30 <sertonix[m]> The blog post mentions that the AI crawlers need to look like a browser to get around normal bot protections. Which means that they probably wouldn't switch user agents just cause of anubis.
2025-03-16 06:55:53 <f_> sertonix[m]: looks like gnome badly configured their anubis deployment
2025-03-16 06:56:05 <f_> Either that or I'm apparently an AI bot
2025-03-16 06:57:15 <f_> I saw anubis being used in other git instances too
2025-03-16 06:57:59 <f_> invoked: yeah there's colateral damage of using Anubis.. but if it's just for the gitlab there shouldn't be any colateral damage at all .. because gitlab requires JS anyway
2025-03-16 14:42:44 <mio> 3.21 armv7 builder has been on php84-pecl-imagick-3.8.0_rc2-r0 for 10+ hours, looks like the build might be stuck
2025-03-16 15:32:05 <invoked> f_: true. i was making commentary on how this is an arms race. to quote the post: "This is a tradeoff that I am not happy about, but it is the world we live in now."
2025-03-17 07:25:52 <macmpi> it seems like edge/community packages do not get propagated: last is go 1.24.1-r0 update yesterday 1pm
2025-03-17 07:55:11 <ikke> macmpi: builders are stuck building rebuilding go packages 
2025-03-17 07:56:10 <ikke> https://build.alpinelinux.org
2025-03-17 08:03:50 <macmpi> already built packages are not available in community repo (but are in main): https://pkgs.alpinelinux.org/packages?name=&branch=edge&repo=community&arch=x86_64&origin=&flagged=&maintainer=
2025-03-17 08:07:17 <ikke> It only uploads packages after the whole repo is completely built for consistency 
2025-03-17 08:38:21 <macmpi> ok, thanks
2025-03-17 15:27:36 <ncopa> we are running out of diskspace on build-edge-aarch64
2025-03-17 15:28:57 <ncopa> the machine has a 2TB nvme, right? I wonder if we should replace it with a 4TB
2025-03-17 15:41:08 <ikke> I was thinking something similar
2025-03-17 15:41:48 <ikke> s390x iirc is also very constrained 
2025-03-17 16:08:45 <invoked> are you talking physical nvme drives or vm slice? if physical, beware those qlc drives
2025-03-17 16:11:46 <ikke> physical
2025-03-17 16:35:17 <invoked> ok. yes, i urge you to hard pass any and all qlc devices. absolute rubbish
2025-03-17 16:35:39 <invoked> even if they're free, more trouble than they are worth for write-heavy anything
2025-03-17 16:41:09 <ikke> We do have some funds now, so we do not need to cheap out
2025-03-17 17:11:48 <invoked> at 4tb density in a single m2 i think the good options are rather limited (mainly like samsung 990 pro, which are just okay in terms of endurance, but substantially better than any qlc)
2025-03-17 17:12:24 <invoked> if you have multiple m2s, or u2s, then your options open up
2025-03-17 17:14:44 <invoked> (also pci, of course)
2025-03-17 17:17:58 <invoked> i recently stuck a few samsung 990 pros in my personal machines. i've not used the samsung ssds prior to this. they seem to be decent... but my write load is well below that of a builder.
2025-03-18 01:43:23 <mio> edge s390x builder looks to be low on disk space, `mkdir: can't create directory '/home/buildozer/aports/testing/lomiri-indicator-location/src': No space left on device`
2025-03-18 07:46:07 <ncopa> mio: I have cleaned it up a bit now
2025-03-18 07:55:08 <ikke> ncopa: i cleaned it up yesterday as well
2025-03-18 07:57:00 <ncopa> we should probably delete some of the older versions
2025-03-18 11:49:17 <ncopa> this is waiting for available runners. https://gitlab.alpinelinux.org/alpine/abuild/-/merge_requests/361
2025-03-18 11:49:49 <ncopa> we have had various providers offered CI power. are there any of them we could accept and add CI runners sooner than later?
2025-03-18 12:00:24 <ikke> We are in contact with some, though, we have been focussing first on getting T1 mirror servers
2025-03-18 13:03:38 <mio> ncopa: thanks!
2025-03-23 19:24:25 <ptrc> not sure why, but i can't write to dev.a.o/archive/ncftp/
2025-03-23 19:24:41 <ptrc> the permissions seem alright (g+w, and i'm in the correct group)
2025-03-23 19:25:15 <ptrc> actually no
2025-03-23 19:25:20 <ptrc> it doesn't have g+w on the directory
2025-03-23 19:25:45 <ptrc> well, it's nothing urgent
2025-03-23 19:36:16 <ikke> ptrc: fixed
2025-03-23 19:36:27 <ptrc> thanks!
2025-03-23 19:37:50 <ptrc> fwiw, i guess all directories should have group write permissions, in case another dev needs to place an archive/patch there
2025-03-23 19:38:47 <ikke> fixed
2025-03-24 12:54:53 <mio> could someone maybe check on the edge armhf builder to see if py3-pebble is progressing? it looks to be hanging, it's been over 10 h
2025-03-24 12:57:46 <mio> thanks
2025-03-24 16:37:57 <ncopa> mio: I restarted build-edge-armhf
2025-03-24 16:38:15 <ncopa> it appeared to be deadlocked in the pebble testsuite
2025-03-24 16:38:46 <ikke> ncopa: I've received details for 1 (of 4 to come) VMs for CI
2025-03-24 17:09:42 <mio> ncopa: thanks!
2025-03-24 18:15:50 <ikke> ncopa: I've added a new x86_64 CI runner now
2025-03-25 10:08:02 <ncopa> awesome! thanks ikke!
2025-03-25 15:42:17 <PureTryOut> Could dev.alpinelinux.org perhaps get some extra storage? It currently only has 3.1G available, sadly not enough for the full source of a package I'm making
2025-03-26 13:43:56 <nu_> would you like to have a bigger storage medium for che-bld-1? i see a lot of 95%+ used alerts
2025-03-26 15:02:08 <ncopa> yeah, I think thats a good idea. Not sure what ikke and clandmeter thinks
2025-03-26 16:07:11 <ikke> ncopa: I think it's a good idea as well. I was thinking we might be able to use the opencollective funds for that
2025-03-26 16:57:22 <clandmeter> +1
2025-03-26 17:16:40 <ncopa> yeah
2025-03-26 17:19:16 <ncopa> nu_: see if you find a good 4TB nvme. you can expense it to our opencollective
2025-03-26 18:47:21 <f_> The joys of OpenCollective :)
2025-03-27 16:40:47 <ikke> durrendal: We will be receiving quite some infra structue in the coming time, which needs to be installed. We were thinking about starting to use Ansible to automate it
2025-03-27 16:42:17 <ikke> One question we need to answer is what execution model for ansible we will use
2025-03-27 16:55:53 <durrendal> ikke: ooh exciting! There's a few different ways to go about it that's for sure. You could pre-seed ansible playbooks using cloud init, git clone them and then execute them that way. Or you could seed them through terraform (maybe opentofu?) and execute them from a controlling system or directly on the system. If you could assume SSH access to a system you could tie it into CI
2025-03-27 16:55:55 <durrendal> and ssh into the system to apply the playbook. Probably so many more
2025-03-27 16:56:50 <durrendal> I have to step off for a work meeting, but I'm super curious to hear your thoughts
2025-03-27 17:03:10 <ikke> I'd like to start small, first focussing on those new systems. We cannot rely on a single automation system since we work with many vendors. We could perhaps use netbox for the inventory, but I wouldn't mind manually keeping track of the inventory either
2025-03-27 17:17:22 <ikke> One thing I saw is that you also have a pull base workflow
2025-03-27 18:21:58 <durrendal> ikke: a pull model is an interesting idea, then each node would essentially manage itself. Maybe a base image could be made that self bootstraps and then clones a repo, applies the state from it
2025-03-27 18:22:35 <durrendal> though I'm assuming a lot about initial configuration, dunno if these are servers that already have alpine installed ot them and then we just get our ssh key distributed to them?
2025-03-27 18:23:05 <ikke> durrendal: Not necessarily
2025-03-27 18:23:33 <ikke> We received 4 VMs for CI this week that we needed to do the installation on
2025-03-27 18:27:01 <durrendal> Oh excellent so a custom ISO could be an option. Or maybe an initial push bases playbook that sets up pull based state orchestration
2025-03-27 18:27:45 <ikke> yeah, was thinking about the latter
2025-03-27 18:29:01 <durrendal> It would keep things unified. Simple. All that we'd to ensure is a base install, ssh key access, and that python was installed. Easy stuff
2025-03-28 07:31:01 <clandmeter> durrendal: thanks for thinking with us. Maybe we could plan a meeting just to sync ideas. I have a bit more time the coming weeks. ok with you ikke?
2025-03-28 11:21:53 <ikke> clandmeter: fine with me
2025-03-28 11:29:58 <ikke> I'm testing having gitlab-runners in a kubernetes cluster
2025-03-28 11:31:50 <ikke> sadly kubernetes does not support image platforms (when using manifest images) to download a different arch image
2025-03-28 11:31:56 <ikke> so we have to rely on dedicated tagged images
2025-03-28 11:57:09 <ncopa> https://gitlab.alpinelinux.org/alpine/aports/-/jobs/1786470 has been waiting a while for a risc-v runner. I think I should set up another one. I have a p550 under my desk
2025-03-28 11:57:24 <ncopa> or I could setup a couple on scaleway
2025-03-28 11:58:03 <ikke> I see that nld-bld-2 is unreachable
2025-03-28 11:58:45 <ikke> fyi, there are MRs with some large builds running
2025-03-28 11:59:12 <ikke> s390x is also pending
2025-03-28 11:59:19 <ncopa> yeah
2025-03-28 11:59:25 <ncopa> now sure what we can do about s390x
2025-03-28 11:59:34 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/82026 has been running for 13 hours
2025-03-28 11:59:35 <ncopa> would be nice with another runner
2025-03-28 12:00:01 <ikke> maybe that could've been split in multiple MRs
2025-03-28 12:00:42 <ncopa> i suppose my push of icu-76 rebuilds didnt help
2025-03-28 12:00:59 <ncopa> chromium, firefox, libreoffice etc
2025-03-28 12:01:13 <ncopa> and 3 copies of webkit
2025-03-28 12:03:46 <ncopa> ikke: did you have the docs to set up a new runner somewhere?
2025-03-28 12:14:39 <ncopa> i think i found it. the setup-runners.sh script
2025-03-28 12:23:46 <ncopa> but im holding it wrong
2025-03-28 12:23:51 <ncopa> curl: (22) The requested URL returned error: 401
2025-03-28 12:23:51 <ncopa> ERROR: could not create runner
2025-03-28 12:39:51 <durrendal> clandmeter: that sounds like a great idea to me. I think the only blocking item I have on my calendar is travel for a wedding next month, but I'd rather relay the specifics on that privately.
2025-03-28 15:46:32 <ncopa> git branch
2025-03-28 17:47:32 <Kladky> Ah, so that's why riscv64 pipelines are queuing up
2025-03-28 17:48:43 <ikke> ncopa: curious, I'm using it constantly without issue. Did you use a personal access token with the create runner scope?
2025-03-29 09:57:34 <ikke> ncopa: I've added my banapi f3
2025-03-29 14:39:37 <ikke> I've enabled a new gitlab-runner running on a kubernetes cluster for x86_64 now
2025-03-29 14:39:41 <ikke> Let's see how it works
2025-03-29 14:44:46 <durrendal> \o/ cool addition! I assume that means light weight stuff like linting will just scale out across the kubernetes cluster
2025-03-29 14:46:52 <ikke> durrendal: not yet, I'd need another tag for that, but can do that
2025-03-29 14:47:51 <ikke> But I guess I need to tell kubernetes these jobs require less resources
2025-03-29 14:55:20 <durrendal> Ah so it's just to scale put jobs for a specific CPU architecture then? I think if the experiment works well it might be good to make the resource constraint changes, then you may even be able yo gate the creation of the actual build steps behind lint passing successfully
2025-03-29 14:56:40 <ikke> We used to only run the build jobs after linting, but that was deemed too anoying
2025-03-29 14:57:09 <ikke> Right now, I'm setting cpu / memory limits for the jobs, but these are quite large, which means only a limited amount of jobs per node can be executed
2025-03-29 14:57:39 <ikke> I'm trying to figure out how to override it per job, but the documentation is hard to find
2025-03-29 14:58:45 <ikke> Ah, here: https://docs.gitlab.com/runner/executors/kubernetes/#overwrite-container-resources
2025-03-29 15:03:19 <durrendal> Ah yeah I guess I can get that. And it ignores the very real scenario that the linting might falsely flag an issue.
2025-03-29 15:04:38 <durrendal> Per job limitations sound excellent addition though
2025-03-29 15:05:18 <ikke> durrendal: linting is a warning anyway right now, so it would not prevent the pipeline from continuing
2025-03-29 15:10:26 <durrendal> Great point, and it really should be a warning anyways.
2025-03-29 15:11:09 <ikke> do you have experience with kubernetes?
2025-03-29 15:14:45 <ikke> https://gitlab.alpinelinux.org/alpine/aports/-/commit/69d3c798586b8a4253de010660dfdaef15e9d6c1
2025-03-29 15:26:33 <durrendal> Only in a homelab setting. I started experimenting with it when you mentioned you were testing k0s recently. 
2025-03-29 15:26:54 <ikke> ah ok
2025-03-29 15:27:16 <durrendal> What kind of specs do the systems running the kubernetes cluster have? 
2025-03-29 15:27:38 <ikke> The ones I've just talkes about: 48 cores, 128G memory :)
2025-03-29 15:27:53 <durrendal> It's one of those technologies I've been telling myself to learn for years and haven't had a good impetus until now
2025-03-29 15:27:54 <ikke> They are qemu VMs, so no idea how fast the cores are
2025-03-29 15:28:13 <durrendal> Those are seriously nice specs though! All donated hosting?
2025-03-29 15:28:25 <ikke> Yes
2025-03-29 15:29:39 <ikke> I need to figure out how we want to manage the deployments for our clusters
2025-03-29 15:29:56 <ikke> I do have a solid idea how to manage deployments themselves
2025-03-29 15:30:24 <ikke> More like how to organize it in repositories
2025-03-29 15:31:50 <ikke> basically: you have N clusters, you have M deployments.
2025-03-29 15:32:45 <ikke> Do we want to create a repository for each cluster that somehow imports deployments, while still allowing for customization
2025-03-29 15:33:01 <ikke> And maybe I've just answered my question
2025-03-29 15:33:29 <ikke> kustomize allows you to pull in deployments from other repositories
2025-03-29 15:41:23 <durrendal> Love to see it :) was a bit worried after Equinix fell through.
2025-03-29 15:42:47 <ikke> https://imgur.com/a/d7iI7c8
2025-03-29 15:42:49 <durrendal> That's a great idea actually, then everything is under version control, every part of the deployment configuration can be audited and contributed to. Maybe even have per cluster/deployment CI
2025-03-29 15:43:28 <ikke> gitlab recommends fluxcd
2025-03-29 15:43:37 <ikke> there is an integration
2025-03-29 15:43:55 <ikke> but you also need to deploy an agent both in the k8s cluster as in gitlab
2025-03-29 15:44:43 <ikke> fluxcd pulls from a repo and then deploys what's defined there
2025-03-29 15:51:20 <ikke> rubber ducking is really usefull :-)
2025-03-29 15:52:53 <durrendal> I am happy to play the proverbial rubber duck haha. It helps that this is objectively cool infrastructure
2025-03-29 15:53:55 <durrendal> I suppose given that design. The gitlab instance would need to stay it's own service to host the configuration for the fluxcd agent to pull from
2025-03-29 15:57:12 <ikke> Yeah, probably not managed by fluxcd at least
2025-03-29 15:57:24 <ikke> But not sure if we will run gitlab on kubernetes soon
2025-03-29 15:57:34 <durrendal> Are we also using the control plane node as a worker node? The difference in resources seems super large
2025-03-29 15:57:48 <ikke> On that particular cluster, yes
2025-03-29 15:59:03 <ikke> I wanted to run a controller+worker setup on all nodes, but apparently you need to have an loadbalancer to each controller if you want to have multiple controllers
2025-03-29 16:20:54 <durrendal> That's what I thought seeing the pods running on it. The multi controller configuration might make sense if you had two smaller nodes to use as dedicated control systems with larger more robust nodes to scale out across
2025-03-29 16:21:40 <durrendal> I would be curious to see if moving gitlab to the kubernetes node helps with some of the scraping issues, I imagine it'd be more resilient
2025-03-29 16:22:49 <ikke> ftr, this cluster is not suitable for moving web applications to
2025-03-29 16:23:27 <ikke> 1) there is limited storage
2025-03-29 16:23:45 <ikke> 2) We cannot connect to it externally except for some forwarded ssh port
2025-03-29 16:24:35 <ikke> To properly scale gitlab out, we would need to move to gitaly cluster, which is a whole project on its own
2025-03-29 16:25:03 <ikke> And also comes with questions like how to manage backups in a way that they remain consistent
2025-03-29 16:25:04 <durrendal> Ah this is purely compute for ephemeral CI then? The limited storage seems like that would make it unsuitable for the current builders 
2025-03-29 16:25:17 <ikke> durrendal: exactly, CI is the goal
2025-03-29 16:25:43 <ikke> which is also what we're losing from equinix
2025-03-29 16:25:49 <durrendal> Well that is a super powered CI cluster in that case, wicked cool
2025-03-29 16:26:28 <durrendal> Was Equinix only providing x86_64 compute? 
2025-03-29 16:28:25 <ikke> yes
2025-03-29 16:28:48 <ikke> We used to have aarch64 there as well, but the HW was provided by ARM. The hosting part for that stopped and we moved that to somewhere else
2025-03-29 16:31:30 <ikke> durrendal: A deployment I created earlier for anoter k8s cluster: https://gitlab.alpinelinux.org/alpine/infra/k8s/update-certs
2025-03-29 17:46:44 <vkrishn> ikke: there is also collaborative https://web.drawpile.net/ (session), if wanting to have fun with new infra design sketches
2025-03-29 17:47:13 <vkrishn> nice to read the current ongoing efforts
2025-03-29 17:47:14 <ikke> ❌ Incompatible browser: Firefox on Linux does not support pressure-sensitive pens. Consider using a different browser
2025-03-29 17:47:16 <ikke> :(
2025-03-29 17:47:23 <vkrishn> yes, just say ok
2025-03-29 17:47:26 <ikke> Not that I have one
2025-03-29 17:47:37 <vkrishn> use touchpad or mouse
2025-03-29 17:47:43 <vkrishn> touchpad is better
2025-03-29 17:48:45 <fossdd[m]> drawpile is also in the testing repo
2025-03-29 17:49:24 <vkrishn> yes, wanted so say earlier, may wanna consider running  have an al instance
2025-03-29 17:49:47 <vkrishn> it is currently to go in community/
2025-03-29 17:51:16 <vkrishn> we have currentlysome hw dormant
2025-03-29 18:01:08 <vkrishn> hmmm, annotations does seems to work
2025-03-29 19:20:46 <durrendal> ikke: that's a great example of what you were talking out. I can definitely see the benefit of building modular deployments that can be reused between clusters
2025-03-29 19:21:44 <durrendal> I'm not super versed in it, but could we then take these deployments and mesh them together into a helm chart that defines the totality of a deployed service/services
2025-03-29 19:28:00 <ikke> durrendal: I use kustomize instead of helm charts
2025-03-29 19:28:08 <ikke> It can consume helm charts though
2025-03-29 19:30:42 <ikke> kustomize is less yaml templates and more declarative resources
2025-03-29 19:31:03 <ikke> only thing I really miss so far is something like values.yaml, but you can get quite far without it
2025-03-30 03:38:32 <mio> the libxml2 upgrade was reverted but the edge builders may still have the newer version cached, e.g. gettext-0.22.5-r2 (-r2 being the pkgrel in the revert commit) still selects the newer version according to the build log. maybe if a newer version is still cached it could be removed before retrying the builds?
2025-03-30 16:44:02 <invoked> didn't see an error fire here for it, but mirrors.ocf.berkeley.edu isn't working for me (https://ocf.berkeley.edu is reachable though)
2025-03-30 16:48:59 <ikke> Zabbix sees it as well
2025-03-30 17:49:36 <ikke> ..
2025-03-30 18:10:06 <ptrc> can we get a "project: rootbld in production use" milestone on the alpine/ gitlab group? ( wrt https://gitlab.alpinelinux.org/alpine/tsc/-/issues/33, and to categorize issues that fail only in rootbld )