2025-03-02 19:06:17 <ikke> lesigh
2025-03-02 19:54:03 <clandmeter> whats happening? very colorful here
2025-03-02 19:55:26 <ikke> Lots of scraping, but I'm not sure what exactly is causing the load issues
2025-03-02 19:56:03 <ikke> Tried increasing the amount of workers, but that just increases the amount requests being able to be handled
2025-03-02 19:56:25 <ikke> Not sure if it's just plain crawling that's the issue, or that there are specific requests which are heavy
2025-03-03 09:37:04 <qaqland> oh, git.a.o is 502
2025-03-03 09:38:41 <pj> ikke: any particular ip ranges that are guilty?
2025-03-03 10:07:13 <ikke> pj: I see some ranges making the most requests, but I'm not entirely sure those requests cause most of the load 
2025-03-03 10:07:28 <ikke> I need to quantify that somehow 
2025-03-03 10:14:39 <pj> there's been an ongoing problem for gitea/forgejo instances where bots keep scraping git archive tarballs which blows up their git archive cache
2025-03-03 10:16:14 <pj> an example from admin of my fedi instance: https://donotsta.re/notice/AreSNZlRlJv73AW7tI
2025-03-03 10:23:42 <ikke> I may start sending the access logs to loki, may help to make it easier to analyse the requests
2025-03-03 11:10:40 <ncopa> not sure if we could do something like this? https://fosstodon.org/@dalias@hachyderm.io/114055514402836782
2025-03-03 11:16:55 <ikke> I'm already doing something like that with nginx
2025-03-03 11:17:10 <ikke> well, not a download speed, but more rate limit delay
2025-03-03 11:17:57 <ikke> But, understanding where the slowness comes from is step 1
2025-03-03 15:17:10 <invoked> if you find out it's from crawlers, it's a bit of whack-a-mole listing ips, but even where i see site operators doing that, they drop requests instead of returning something, which probably just gets the crawl job moved to a different queue/ip/asn. i personally favor returning 402.
2025-03-03 15:17:31 <invoked> in any case site operators need to get better about working together because this is war
2025-03-03 15:27:54 <ikke> invoked: agree
2025-03-04 00:33:51 <francis[m]> From Trusted and Vouched Dealers... (full message at <https://matrix.org/oftc/media/v1/media/download/Ab25Tr5I39SorVcgwfZVX9KjvK5hzFkXPqaq6zPoStn6FhS0eryRf22tO0285mYsT2Q5DC1Fzo_9Fxdcfs8ItX9CeVpqvJpwAG1hdHJpeC5vcmcvVmJqenFjUFRnSlRQT0VWQnBvQkZ5UmNs>)
2025-03-04 00:34:26 <fossdd[m]> pj
2025-03-04 00:34:34 <fossdd[m]> :3
2025-03-04 00:34:44 <pj> i can't do anything, no op here
2025-03-04 00:34:58 <fossdd[m]> oh damn
2025-03-04 00:38:23 <pj> maybe I could reactive my mjolnir instance that I used for lapce
2025-03-04 00:38:39 <pj> would make it easier to ban across all channels
2025-03-04 00:39:31 <fossdd[m]> they are all coming from :matrix.org, if they could just spend their money right and ban them immediatley