2025-03-02 19:06:17 <ikke> lesigh
2025-03-02 19:54:03 <clandmeter> whats happening? very colorful here
2025-03-02 19:55:26 <ikke> Lots of scraping, but I'm not sure what exactly is causing the load issues
2025-03-02 19:56:03 <ikke> Tried increasing the amount of workers, but that just increases the amount requests being able to be handled
2025-03-02 19:56:25 <ikke> Not sure if it's just plain crawling that's the issue, or that there are specific requests which are heavy
2025-03-03 09:37:04 <qaqland> oh, git.a.o is 502
2025-03-03 09:38:41 <pj> ikke: any particular ip ranges that are guilty?
2025-03-03 10:07:13 <ikke> pj: I see some ranges making the most requests, but I'm not entirely sure those requests cause most of the load 
2025-03-03 10:07:28 <ikke> I need to quantify that somehow 
2025-03-03 10:14:39 <pj> there's been an ongoing problem for gitea/forgejo instances where bots keep scraping git archive tarballs which blows up their git archive cache
2025-03-03 10:16:14 <pj> an example from admin of my fedi instance: https://donotsta.re/notice/AreSNZlRlJv73AW7tI
2025-03-03 10:23:42 <ikke> I may start sending the access logs to loki, may help to make it easier to analyse the requests
2025-03-03 11:10:40 <ncopa> not sure if we could do something like this? https://fosstodon.org/@dalias@hachyderm.io/114055514402836782
2025-03-03 11:16:55 <ikke> I'm already doing something like that with nginx
2025-03-03 11:17:10 <ikke> well, not a download speed, but more rate limit delay
2025-03-03 11:17:57 <ikke> But, understanding where the slowness comes from is step 1
2025-03-03 15:17:10 <invoked> if you find out it's from crawlers, it's a bit of whack-a-mole listing ips, but even where i see site operators doing that, they drop requests instead of returning something, which probably just gets the crawl job moved to a different queue/ip/asn. i personally favor returning 402.
2025-03-03 15:17:31 <invoked> in any case site operators need to get better about working together because this is war
2025-03-03 15:27:54 <ikke> invoked: agree
2025-03-04 00:33:51 <francis[m]> From Trusted and Vouched Dealers... (full message at <https://matrix.org/oftc/media/v1/media/download/Ab25Tr5I39SorVcgwfZVX9KjvK5hzFkXPqaq6zPoStn6FhS0eryRf22tO0285mYsT2Q5DC1Fzo_9Fxdcfs8ItX9CeVpqvJpwAG1hdHJpeC5vcmcvVmJqenFjUFRnSlRQT0VWQnBvQkZ5UmNs>)
2025-03-04 00:34:26 <fossdd[m]> pj
2025-03-04 00:34:34 <fossdd[m]> :3
2025-03-04 00:34:44 <pj> i can't do anything, no op here
2025-03-04 00:34:58 <fossdd[m]> oh damn
2025-03-04 00:38:23 <pj> maybe I could reactive my mjolnir instance that I used for lapce
2025-03-04 00:38:39 <pj> would make it easier to ban across all channels
2025-03-04 00:39:31 <fossdd[m]> they are all coming from :matrix.org, if they could just spend their money right and ban them immediatley
2025-03-04 06:38:18 <ikke> pj: seems to be lots of addresses from brazil
2025-03-04 06:38:21 <ikke> different ISPs
2025-03-04 06:38:46 <ikke> 45.x.y.z
2025-03-04 06:39:42 <ikke> many requests from unique addresses
2025-03-04 06:43:46 <pj> is that like consistent scraping?
2025-03-04 06:43:52 <ikke> seems to be
2025-03-04 06:44:03 <ikke> That was just the last 5000 requests
2025-03-04 06:44:11 <ikke> and I've seen that before as well
2025-03-04 12:31:09 <ikke> Took some more drastic measures, see how that helps
2025-03-04 12:32:15 <ikke> Regarding git.a.o that is (deu1-dev1)
2025-03-04 12:32:50 <ikke> I'm not that impressed with loki at the moment
2025-03-04 12:33:31 <ikke> Something very simple like getting the amount of requests per IP address proves extremely difficult
2025-03-04 12:40:46 <ikke> Put a lot of ASNs in a single rate-limit bucket
2025-03-04 12:41:29 <ikke> I now see an average of 7 429 responses per second
2025-03-04 12:42:08 <vkrishn> git.a.o is gbr-app-1. here
2025-03-04 12:42:36 <pj> opensearch dashboards is much nicer but it's heavy to host
2025-03-04 12:42:52 <ikke> vkrishn: not for me..
2025-03-04 12:43:50 <vkrishn> some routing geo- based? gbr-app-1. is a.o owned?
2025-03-04 12:44:04 <ikke> yes, it is
2025-03-04 12:44:10 <ikke> but it does not host git.a.o
2025-03-04 12:46:54 <vkrishn> ok, seems it redirected from it, as both gbr-app-1. (1st), then deu1-dev1. (2nd)
2025-03-04 12:48:26 <ikke> The load on deu2-dev1 decreased significantly
2025-03-04 12:59:15 <vkrishn> got it, https://wiki.alpinelinux.org/images/alogo.svg on git.a.o webpage connects to gbr-app-1.
2025-03-04 13:07:58 <vkrishn> general rule for modern search engine A reviting same commit-id on git repo should be considered bad(dos attach like)
2025-03-04 13:08:15 <vkrishn> revisiting^
2025-03-04 13:09:17 <vkrishn> search engine should learn its a git repo and not visit same commit
2025-03-04 14:09:58 <ikke> The current traffic comes from many residential ip addresses from many ISPs. Sounds like a botnet or some vpn abusing customers 
2025-03-04 14:14:12 <vkrishn> if i had lots of slurps (brain freeze) with pizza, I would live publish those logs (with some parts scrubbed)
2025-03-04 14:17:25 <vkrishn> i also create lists of IPs that concoct urls(forge) for pages/services
2025-03-04 14:18:01 <vkrishn> i found quite a few from MS
2025-03-04 14:25:22 <vkrishn> also look for IPs that do POSTs on pages/services that don't exist
2025-03-04 16:14:01 <ikke> Similar to gitlab
2025-03-04 16:14:14 <ikke> a long tail of ip addresses from different countries all making a single request
2025-03-04 16:15:35 <ikke> But even from apple, and microsoft apparently
2025-03-04 16:27:08 <vkrishn> been there, follow the trails aka breadcrumbs ;)
2025-03-04 16:27:49 <vkrishn> interesting events unfold
2025-03-04 16:29:23 <vkrishn> make sure to keep the ip list trails or if possible temporarily backup those logs
2025-03-04 16:35:41 <vkrishn> these are few reasons to have only IPv6 rolled globally, and every living human have 2fs(ff) ip assigned as digital/path ID
2025-03-04 16:36:39 <iggy> maybe time to switch to allowlist vs denylist? (I'm only partially kidding)
2025-03-04 16:38:51 <iggy> wonder how hard it'd be to engineer some sort of system of IP web-of-trust or maybe rate limit everyone and then non-abusers get extra tokens in their bucket
2025-03-04 16:41:56 <vkrishn> there were some methods/suggestion in hachyderm url posted yesterday, 
2025-03-04 16:42:42 <vkrishn> i.e deny them and invoke/revoke based on some rule(maybe time)
2025-03-04 16:43:48 <ikke> Would exclude too many legitimate users
2025-03-04 16:45:15 <vkrishn> i noticed quite a lot of crawlers are crond so would visit your IP segment at perticular time
2025-03-04 16:46:03 <vkrishn> that's when it should be locked out
2025-03-04 16:46:18 <vkrishn> its not easy initially
2025-03-04 16:46:49 <ikke> The hard part is determining the traffic is legitimate or harmful
2025-03-04 16:47:06 <ikke> And mostly, what traffic is causing the issues
2025-03-04 16:47:36 <vkrishn> but gradually a foggy IP map gets somewhere in your mind
2025-03-04 16:47:49 <vkrishn> start small
2025-03-04 16:48:21 <ikke> Yes, that's what I'm already doing
2025-03-04 16:53:38 <vkrishn> isn't there a nginx module that slurp streams all css/js into single request?
2025-03-04 16:55:34 <vkrishn> add that to cache(varnish), might smoothen out some more
2025-03-04 17:18:20 <ikke> I must say I really like the geo module of nginx
2025-03-04 17:29:52 <ikke> I think I have at least shedded some of the load now
2025-03-04 17:29:55 <gplessis> Hi there! I was investigating an issue we - as a security business - are experiencing while pulling information from secdb.alpinelinux.org (originally every 6 hours, descreased it to once a day). I noticed logs from this channel saying that you are experiencing some load problems because of heavy crawling / scraping. We are hosted on Azure and may have been caught in your most recent protection 
2025-03-04 17:30:01 <gplessis> actions, likely because of noisy and not very mindful neighbors using the same IP ranges.
2025-03-04 17:30:22 <gplessis> Is there anything we can do to help, and to get our connectivity to secdeb restored?
2025-03-04 17:30:25 <ikke> gplessis: Ok, would you mind sending the ip adres in a dM?
2025-03-04 17:30:31 <ikke> / subnet
2025-03-04 17:31:24 <gplessis> sure
2025-03-04 17:32:40 <ikke> Ok, I've removed the block
2025-03-04 17:33:14 <gplessis> thank you very much for your help!
2025-03-04 20:28:24 <ikke> Seems like the rate limiting does have effect
2025-03-04 20:44:14 <vkrishn> how would you describe it from "crawlers vantage point" ;)
2025-03-04 20:46:50 <ikke> From their vantage point, they are certainly affected
2025-03-04 20:51:36 <vkrishn> :), i mean what would gptAI say to itself when visiting a.o sites
2025-03-04 20:52:06 <ikke> This tastes rather bland
2025-03-04 20:52:11 <vkrishn> nm, offtopic
2025-03-04 20:52:14 <vkrishn> :)
2025-03-04 21:01:22 <vkrishn> looks like your allowlist has got its first entry
2025-03-04 22:19:32 <sertonix[m]> I think I got ratelimited too. Loading any gitlab.a.o page results in 429 for some requests which cases the page to be broken.
2025-03-05 05:45:43 <ikke> sertonix[m]: should be fixed now
2025-03-05 09:24:30 <qaqland> i inquired with sjtug and they are changing and working on architecture migration.
2025-03-05 15:41:30 <ikke> qaqland: thanks
2025-03-06 18:51:46 <vkrishn> ikke: an example of IPs that fall in not good list, https://tpaste.us/Ro9Y 
2025-03-06 18:52:12 <ikke> That's Alibaba Cloud
2025-03-06 18:52:14 <ikke> already covered
2025-03-06 18:52:42 <ikke> Oh no, this one is different
2025-03-06 18:54:40 <vkrishn> its trying to get, GET /static../.git/config , something that don't exist,
2025-03-06 18:54:57 <ikke> Yeah, trying to find accidentally exposed credentials
2025-03-06 18:54:59 <vkrishn> sometimes i sit and scan looking for similar entry in logs
2025-03-06 18:55:21 <ikke> But right now, that kind of traffic is not what I'm focussing on
2025-03-06 18:56:00 <ikke> And, at least for now, it appears things are under control
2025-03-06 18:56:57 <vkrishn> yes, thanks for the effort, noticed some improvement
2025-03-07 09:53:21 <darim> Hi, I'm trying to register an account on the Alpine GitLab, but I'm not receiving the confirmation email. I've tried different email providers, but none of them worked. I also tried signing up with GitHub, but that didn’t work either. Is there an issue?
2025-03-07 10:00:05 <ikke> darim: I can check in an hour 
2025-03-07 10:00:59 <ikke> The only issue I'm aware of is outlook.com blacklisting us
2025-03-07 10:04:01 <darim> Okay, that might be the issue. My email is hosted on Outlook.com, and my GitHub account is also linked to an Outlook.com email address. I will try it with gmail.
2025-03-07 15:05:09 <mio> is it normal to have a ~25 s wait between new page loads to gitlab.a.o sometimes? this has improved over the past few days, just wondered if my ip was caught in some filtering
2025-03-07 15:37:27 <ikke> mio: are you opening a lot of pages at the same time?
2025-03-07 15:38:51 <mio> 2-3 tabs?
2025-03-07 15:39:40 <mio> it occurs on the first load usually, or refreshing same tab after some time
2025-03-07 16:09:48 <ikke> mio: that should not trigger it
2025-03-07 16:10:00 <ikke> The IP address I see for you is not rate limited
2025-03-07 16:11:17 <ikke> mio: if you open the network tab, you should see 429 responses if you're rate limited
2025-03-07 16:13:41 <mio> ikke: okay, thanks!
2025-03-07 16:15:27 <mio> have had 1 tab open of the merge_requests page that is refreshed a few times an hour or a few hours, and 1-2 new tabs potentially opened from that
2025-03-07 16:16:35 <mio> the list refresh seems to have the wait, then reading MRs after that initial period is fine for a while
2025-03-07 16:18:05 <mio> will try to keep the number of open tabs low and see if that helps, thanks
2025-03-07 16:18:18 <mio> (based on what you mentioned)
2025-03-07 17:16:48 <vkrishn_> i use mouse, but noticed that there are key-bindings to pages too, it slightly faster that F5(full refresh)
2025-03-07 17:44:07 <mio> interesting, will try that as well, thanks
2025-03-07 18:23:52 <ikke> mio: I sometimes open 8 tabs in sequence without issue
2025-03-07 18:41:23 <mio> ikke: okay, good to know. just switched to firefox, page loading seems to be fine so far
2025-03-07 18:42:55 <mio> maybe my other browser is having some issue or cached some old setting
2025-03-10 19:50:41 <ikke> I have cleaned up distfiles a bit
2025-03-12 01:53:45 <vkrishn> ikke: around? can i lock-discussion to a TSC issue?
2025-03-12 01:54:02 <vkrishn> pls lock-it, https://gitlab.alpinelinux.org/alpine/tsc/-/issues/93
2025-03-12 01:54:07 <vkrishn> seems i cannot
2025-03-12 01:54:10 <vkrishn> thanks
2025-03-12 01:55:18 <vkrishn> finally, really wanted to get off irc, now got a reason ;)
2025-03-12 02:23:14 <vkrishn> seems there is no export to pdf to take a backup of the issue, so just copy-pasting the page for backup
2025-03-12 05:17:56 <vkrishn> nice, the brower gave a nice screenshot too - for the TSC issue page
2025-03-12 14:10:08 <fossdd[m]> looks like gitlab is sending me 5x the same email again
2025-03-12 14:10:15 <fossdd[m]> i think i noticed that yesterday already
2025-03-12 14:30:20 <vkrishn> psykose would be _alice here?
2025-03-12 14:30:32 <vkrishn> https://d.insteps.net/scr/screenshot-2025-03-12-at-06-59-25-TSC-93.png, pls see there is not reply btn
2025-03-12 14:32:18 <durrendal> fossdd[m]: I just noticed that as well, I just assumed gitlab REALLY wanted me to know my CI job failed
2025-03-12 14:32:30 <fossdd[m]> haha
2025-03-12 14:32:43 <vkrishn> i am planning 2 more TSC, please enable them this time, play nice pls (i have been jotting the points for all 3 for yrs), they have no linking to current irc postings
2025-03-12 14:33:46 <vkrishn> durrendal: wanted to say but missed, a tester has can be independant role and not associated with AL
2025-03-12 14:33:54 <ikke> Please keep the scope reduced 
2025-03-12 14:33:55 <vkrishn> a tester can^
2025-03-12 14:34:03 <vkrishn> ok sorry
2025-03-12 14:35:29 <vkrishn> in img, seems all participants said all at once and were talking to themselves
2025-03-12 14:36:13 <vkrishn> i can also formulate "TSC: Rules off engagement" if gitlab is buggy
2025-03-12 14:36:20 <durrendal> vkrishn: yes that makes sense, I only meant yesterday that by participating in some of the current maintenance workflow you might be better able to emphasize that difference.
2025-03-12 14:37:12 <fossdd[m]> gitlab is not buggy? you know that comments happen in threads, right? https://docs.gitlab.com/user/discussions/
2025-03-12 14:38:17 <vkrishn> yes, but if you reply , new reply btn should be on it, so that sub-reply can be a hierarchy
2025-03-12 14:39:00 <vkrishn> but if 2 reply happens on same orig posts, the replay field is in the end
2025-03-12 14:41:23 <vkrishn> which means, the 2nd replier has to wait(rules of enagement)
2025-03-12 14:41:40 <vkrishn> reading the docs now
2025-03-12 14:43:05 <vkrishn> or I can leave the page unlocked for 12hrs, get reply, lock it, do my reply, unlock
2025-03-12 14:43:20 <vkrishn> cycle repeat
2025-03-12 14:43:42 <vkrishn> but need privilege to do locking/unlocking
2025-03-12 14:48:51 <vkrishn> just fun, but why use "hearts up" instead 'thumbs up', sounded like social forum (no hard feelings as its a feature)
2025-03-12 14:52:28 <vkrishn> Formulated rules like these, will always feel like an 'outsider thing', it cannot have partialiaty in it
2025-03-12 14:53:20 <vkrishn> fight it at its tech point
2025-03-12 14:53:43 <vkrishn> partiality^
2025-03-12 14:56:36 <vkrishn> its targetted towards a running system, not one person
2025-03-12 15:00:04 <vkrishn> if AL survives the next generation(AI infested era) in its principals, your kids would thank you
2025-03-12 15:07:52 <invoked> vkrishn: my suggestion would be to focus (constrain) what you're proposing to things you've worked out some basic concept how it would be implemented (with gitlab or a tool, that ideally adds no additional steps to what everyone is currently doing). cut out everything else (or set it aside for later)
2025-03-12 15:08:51 <invoked> this process is about getting buy-in from others
2025-03-12 15:09:39 <invoked> i could be wrong, but anything that drastically changes -- or adds steps to -- what everyone currently does, is likely to face a bunch of resistance
2025-03-12 15:12:19 <vkrishn> why does Formulated rules always look like corporate/business :), coz they are never implemented at lower level, need costs to do it
2025-03-12 15:12:44 <invoked> the reasons don't matter. it's your job to figure out how to get the buy-in, that's all i'm saying
2025-03-12 15:14:11 <invoked> if enough people are giving you a clear negative signal on something, you need to rework it or maybe cut it out.
2025-03-12 15:17:17 <vkrishn> tech feasibility or not, I am open to ikke's suggestions, can only suggest when tools have been selected, based on available funds
2025-03-12 15:19:18 <invoked> there are enough tools. it's probably more about "how would $foo be done with the tools we already have"
2025-03-12 15:21:08 <invoked> if you can answer that question on each point of what you're proposing, and it doesn't much impact what people are currently doing, you'll probably face less resistance
2025-03-12 15:21:11 <vkrishn> can you pick one small $foo and its tools if possible, let me think on it
2025-03-12 15:22:08 <vkrishn> meaning answer with tech codes/implimented details?
2025-03-12 15:23:42 <invoked> just start with gitlab. for example: https://gitlab.alpinelinux.org/alpine/infra/aports-testing-ttl
2025-03-12 15:24:08 <vkrishn> that part is already in effect with TSC #2
2025-03-12 15:24:13 <invoked> look at past issues and how the team has solved them in the past
2025-03-12 15:24:26 <vkrishn> wait, sorry, let me check
2025-03-12 15:25:02 <vkrishn> yup
2025-03-12 15:25:59 <ikke> Ftr, it's not in effect yet
2025-03-12 15:27:26 <vkrishn> yes, thanks need to know, but that is not something new added in
2025-03-12 15:33:50 <vkrishn> ikke: just pick any contigous part and say - feasible or not feasible
2025-03-12 15:34:14 <vkrishn> will start on it
2025-03-12 15:35:09 <vkrishn> but then i think core TSC team has to ad-hoc pass it first
2025-03-12 15:46:34 <invoked> most people (i assume) prefer to see code. like: "here's a subset of these initial ideas, and here's a script that implements & automates those".  even if you do that (as you can see with the link i sent) it will take time to be accepted and then made active
2025-03-12 15:47:47 <invoked> even then some may want stuff taken out...
2025-03-12 15:48:37 <vkrishn> yes, that second part is what is closer to #93:)
2025-03-12 15:48:47 <vkrishn> AH!
2025-03-12 15:50:23 <invoked> so just... mentally: think about how each thing works, with code, and what you're asking of people who are already used to working the current way. summarize succinctly how this helps anything, and be clear about how it would change how people currently work (if it does)
2025-03-12 15:50:52 <invoked> right or wrong, people get annoyed by big proposals, especially when there's no code
2025-03-12 15:53:14 <vkrishn> summarize it, will do, just need to pick aleady replied into one section
2025-03-12 15:54:05 <vkrishn> already^
2025-03-12 15:55:34 <vkrishn> will do in context of current running infra
2025-03-12 15:56:33 <vkrishn> ikke: would need just small info helps, was thinking to put in https://gitlab.alpinelinux.org/alpine/tsc/-/issues/82, but ask here if ok?
2025-03-12 16:02:56 <durrendal> I don't think rehashing issue 93 in issue 82 is the way to go. 82 is focused on finding a technical solution to our build infrastructure, 93 is more documentation/process oriented.
2025-03-12 16:03:16 <durrendal> thanks for trying algitbot
2025-03-12 16:04:35 <vkrishn> no, was going to ask  to give a rough diagramatic sketch (nothing fancy) of current infra
2025-03-12 16:04:58 <vkrishn> current bump codes that maintainers use
2025-03-12 16:07:02 <durrendal> I don't believe we currently have any topology diagrams that are accessible generally. But I can speak somewhat to the infrastructure, what do you want to know?
2025-03-12 16:07:58 <vkrishn> yes, its probably need to be done. Void linux has a webpage like that
2025-03-12 16:08:18 <vkrishn> but just rough sketch
2025-03-12 16:09:01 <vkrishn> i need to understand the IO flow mostly, thats all
2025-03-12 16:09:47 <vkrishn> I think i mentioned it here, before
2025-03-12 16:10:16 <vkrishn> yes, when i asked for signed key-flow
2025-03-12 16:11:37 <vkrishn> gtg, need to check on my other router, seems not ok, will check logs for link
2025-03-12 16:12:12 <durrendal> we're currently migrating away from Equinix metal, so our infra is in flux currently.
2025-03-12 16:13:10 <durrendal> anyways, since you'll check the logs and see this, I'll continue
2025-03-12 16:14:04 <durrendal> Someone makes an MR making a change to an aport, this could be anything from a new package to a metadata change. Lets say I need to upgrade sbcl from 2.5.1 to 2.5.2
2025-03-12 16:14:34 <durrendal> I'd bump pkgver in the APKBUILD in my MR, it would then queue up a series of CI jobs across all the enabled cpu arches supported by that package, build it and test it
2025-03-12 16:15:19 <durrendal> then someone with developer access would come along and review the MR, this would be folks like fossdd[m], they'd let me know if I made a mistake or did something incorrectly, advise on best practice
2025-03-12 16:15:48 <durrendal> once ready, the MR would be merged. Only a few key folks have merge access. Those people are vetted before they get this level of access.
2025-03-12 16:16:11 <durrendal> Once the MR is merged it is queued to the builders via an MQTT pub/sub broker.
2025-03-12 16:16:43 <durrendal> When the builder sees a new package is available it attempts to build it, upon success it signs and publishes the built artifact using rsync.
2025-03-12 16:17:24 <durrendal> https://build.alpinelinux.org/ <- you can watch that process here if you'd like
2025-03-12 16:18:06 <durrendal> and if you happen to know the name/version of a package, what repo it comes from, and what version of Alpine it's being built for you can see the log from the build process
2025-03-12 16:18:08 <durrendal> https://build.alpinelinux.org/buildlogs/build-3-21-x86_64/community/woodpecker/woodpecker-2.8.0-r2.log
2025-03-12 16:18:48 <durrendal> this information is accessible via the MQTT broker, you can sub to it if you'd like and watch the build process.
2025-03-12 16:19:35 <ikke> ftr, #alpine-infra is not the best channel for these discussions
2025-03-12 16:21:08 <durrendal> that's an extremely valid point, this would be better in #-linux or #-devel since we're only tangentially talking about infrastructure here. Normally people would field "how does this work" questions there.
2025-03-12 16:23:51 <durrendal> sorry for the noise ikke, if anything else comes up and I'm around I'll try to redirect it elsewhere.
2025-03-12 17:28:51 <vkrishn> durrendal: thanks, will chew on it
2025-03-12 17:29:27 <vkrishn> btw the way, try a analytics done from MQTT , https://b.infosight.in/alpine
2025-03-12 17:31:06 <vkrishn> after password input, re-hit on the url input area
2025-03-12 17:31:59 <vkrishn> open->mqtt build edge
2025-03-12 23:25:10 <invoked> ikke: there are not, as far as i can tell, any bang-bang ways to convert antora playbooks to mkdocs.
2025-03-12 23:25:44 <invoked> minor styling issues aside, i don't think it would be a lot of work to convert them manually, and i can do that.
2025-03-12 23:26:22 <invoked> but i'll hold off on that for now, let me know what you want to be done
2025-03-12 23:28:44 <invoked> my opinion is that mkdocs would probably be easier to own. but i'm not a tech writer.
2025-03-13 16:14:58 <mio> totally non-urgent question, can someone confirm whether a certain ip address was caught in filtering for gitlab.a.o? did some troubleshooting on my connection issue, think might have accidentally gotten an ip rate-limited early on and hadn't realised
2025-03-13 16:16:49 <ikke> mio: can you DM the address?
2025-03-13 16:19:01 <mio> dm-ed ...  found a workaround to the load delay was seeing earlier, but would be good to know for future reference
2025-03-13 16:27:22 <mio> issue resolved over dm, thanks ikke
2025-03-13 17:14:53 <durrendal> ikke: speaking of rate limiting and mio's connection, had a couple of questions spring to mind while mio and I were troubleshooting their connection issues yesterday. Is gitlab on k0s now? Do we have a dualstack ipv4/v6 load balancer in front of it, and do is the VPC network also dualstack? All just curiosity on my part.
2025-03-13 17:15:27 <ikke> No, gitlab is still runing in docker
2025-03-13 17:15:38 <ikke> No VPC or anything, just plain connectivity
2025-03-13 17:17:55 <durrendal> Ah so our AAAA points directly to the server running the gitlab container then, that makes sense.
2025-03-13 17:18:23 <ikke> It points directly to the ip address of the reverse proxy even ;-)
2025-03-13 17:18:31 <ikke> (traefik container)
2025-03-13 17:20:11 <durrendal> haha of course, that makes a lot more sense :)
2025-03-14 10:31:19 <ikke> Hmm..
2025-03-14 11:37:25 <pj> bots?
2025-03-14 11:38:07 <ikke> Yes, most likely
2025-03-14 11:38:19 <ikke> But all coming from different IPs
2025-03-14 11:38:29 <ikke> max 2 requests per IP
2025-03-14 17:24:03 <ikke> This must be some crazy botnet
2025-03-14 17:42:12 <pj> I've seen some mentions of LLM scraping getting worse
2025-03-14 17:56:49 <ikke> Yeah, but this seems like someone is purchasing botnets for scraping
2025-03-14 20:32:00 <clandmeter> ikke: why does it return 502?
2025-03-14 20:32:36 <clandmeter> just now the cpu load was not that high
2025-03-14 20:32:46 <ikke> Because the workers are busy
2025-03-14 20:34:22 <ikke> I've been rate limiting quite some subnets already
2025-03-14 20:35:21 <clandmeter> should we try to put cloudflare in front if it?
2025-03-14 20:35:27 <clandmeter> or wouldnt that matter
2025-03-14 20:37:49 <ikke> It wouldn't be transparent (ie, getting "Verifying you're a human preloaders")
2025-03-14 23:19:42 <sertonix[m]> The gnome gitlab has added something against AI scrapers. Is that something that could be used?
2025-03-15 03:59:05 <ikke> sertonix[m]: reference?
2025-03-15 03:59:31 <ikke> I guess I saw it, some javascript
2025-03-15 07:43:34 <sertonix[m]> ikke: Yes. If you are fast enough (or the browser is slow enough) you get to this link: https://xeiaso.net/blog/2025/anubis/
2025-03-15 15:21:17 <invoked> unfortunate it's come to that. (which consequently prevents simple browsers/browsing without js -- ie even if that is a small % of users, it's collateral damage)
2025-03-15 15:59:51 <sertonix[m]> When you change your user agent to not contain "Mozilla" anubis would let you pass without needing js.
2025-03-15 16:30:11 <fossdd[m]> really? i think gnome infra itself has some heavy limitations on user agents (has to be the latest chrome etc) but didnt think that anubis itself would restrict that
2025-03-15 16:32:50 <sertonix[m]> anubis doesn't restrict user agents. If you want to know how it works just read the blog post I linked
2025-03-15 17:28:53 <invoked> seems like an obvious hole, which assumes people working on the crawlers won't read the same blog post
2025-03-15 17:32:33 <invoked> it's been ~14 years since i've written a crawler (i did write one that successfully crawled over a billion pages), but then again i wrote one that respected robots.txt. the ai crawlers don't respect boundaries, and i assume they'd quite readily switch user agent if they knew.
2025-03-15 17:44:57 <ikke> I wonder if it works in this case though
2025-03-15 17:45:03 <ikke> single page requests from arbitrary IPs
2025-03-15 21:07:30 <sertonix[m]> The blog post mentions that the AI crawlers need to look like a browser to get around normal bot protections. Which means that they probably wouldn't switch user agents just cause of anubis.
2025-03-16 06:55:53 <f_> sertonix[m]: looks like gnome badly configured their anubis deployment
2025-03-16 06:56:05 <f_> Either that or I'm apparently an AI bot
2025-03-16 06:57:15 <f_> I saw anubis being used in other git instances too
2025-03-16 06:57:59 <f_> invoked: yeah there's colateral damage of using Anubis.. but if it's just for the gitlab there shouldn't be any colateral damage at all .. because gitlab requires JS anyway
2025-03-16 14:42:44 <mio> 3.21 armv7 builder has been on php84-pecl-imagick-3.8.0_rc2-r0 for 10+ hours, looks like the build might be stuck
2025-03-16 15:32:05 <invoked> f_: true. i was making commentary on how this is an arms race. to quote the post: "This is a tradeoff that I am not happy about, but it is the world we live in now."
2025-03-17 07:25:52 <macmpi> it seems like edge/community packages do not get propagated: last is go 1.24.1-r0 update yesterday 1pm
2025-03-17 07:55:11 <ikke> macmpi: builders are stuck building rebuilding go packages 
2025-03-17 07:56:10 <ikke> https://build.alpinelinux.org
2025-03-17 08:03:50 <macmpi> already built packages are not available in community repo (but are in main): https://pkgs.alpinelinux.org/packages?name=&branch=edge&repo=community&arch=x86_64&origin=&flagged=&maintainer=
2025-03-17 08:07:17 <ikke> It only uploads packages after the whole repo is completely built for consistency 
2025-03-17 08:38:21 <macmpi> ok, thanks
2025-03-17 15:27:36 <ncopa> we are running out of diskspace on build-edge-aarch64
2025-03-17 15:28:57 <ncopa> the machine has a 2TB nvme, right? I wonder if we should replace it with a 4TB
2025-03-17 15:41:08 <ikke> I was thinking something similar
2025-03-17 15:41:48 <ikke> s390x iirc is also very constrained 
2025-03-17 16:08:45 <invoked> are you talking physical nvme drives or vm slice? if physical, beware those qlc drives
2025-03-17 16:11:46 <ikke> physical
2025-03-17 16:35:17 <invoked> ok. yes, i urge you to hard pass any and all qlc devices. absolute rubbish
2025-03-17 16:35:39 <invoked> even if they're free, more trouble than they are worth for write-heavy anything
2025-03-17 16:41:09 <ikke> We do have some funds now, so we do not need to cheap out
2025-03-17 17:11:48 <invoked> at 4tb density in a single m2 i think the good options are rather limited (mainly like samsung 990 pro, which are just okay in terms of endurance, but substantially better than any qlc)
2025-03-17 17:12:24 <invoked> if you have multiple m2s, or u2s, then your options open up
2025-03-17 17:14:44 <invoked> (also pci, of course)
2025-03-17 17:17:58 <invoked> i recently stuck a few samsung 990 pros in my personal machines. i've not used the samsung ssds prior to this. they seem to be decent... but my write load is well below that of a builder.
2025-03-18 01:43:23 <mio> edge s390x builder looks to be low on disk space, `mkdir: can't create directory '/home/buildozer/aports/testing/lomiri-indicator-location/src': No space left on device`
2025-03-18 07:46:07 <ncopa> mio: I have cleaned it up a bit now
2025-03-18 07:55:08 <ikke> ncopa: i cleaned it up yesterday as well
2025-03-18 07:57:00 <ncopa> we should probably delete some of the older versions
2025-03-18 11:49:17 <ncopa> this is waiting for available runners. https://gitlab.alpinelinux.org/alpine/abuild/-/merge_requests/361
2025-03-18 11:49:49 <ncopa> we have had various providers offered CI power. are there any of them we could accept and add CI runners sooner than later?
2025-03-18 12:00:24 <ikke> We are in contact with some, though, we have been focussing first on getting T1 mirror servers
2025-03-18 13:03:38 <mio> ncopa: thanks!