2026-06-01 04:49:38 <ikke> FYI, I've added the -k argument for buildrepo on the riscv64 3.24 builder to speed up the process a bit. 
2026-06-01 05:03:54 <ncopa> thanks!
2026-06-01 05:03:59 <ncopa> that was a good idea
2026-06-01 19:48:46 <ikke> So sad to see a single threaded build process on a 64 core machine
2026-06-02 04:55:57 <ikke> riscv64 is still building vector
2026-06-02 04:57:27 <ikke> killing it, last log update was yesterday
2026-06-03 05:51:22 <ncopa> gitlab is sluggish
2026-06-03 05:58:07 <ikke> load is low
2026-06-03 05:58:29 <ikke> Was a brief load spike, memory usage is relatively high
2026-06-03 20:20:40 <ncopa> i killed the build of vector on build-3-24-riscv64
2026-06-03 20:20:46 <ncopa> its building on the p550
2026-06-03 20:21:02 <ncopa> i intend to manually copy it over when its done
2026-06-03 20:21:37 <ikke> ack
2026-06-03 20:24:47 <ncopa> looks like the poor p550 also has two heavy CI jobs
2026-06-03 20:30:21 <ncopa> i stopped build-edge-riscv64
2026-06-03 20:30:37 <ncopa> at least for now, so it can complete vector build and the 2 heavy CI jobs
2026-06-05 05:46:09 <ncopa> i think riscv64 builder died again
2026-06-05 05:46:13 <ncopa> ssh: connect to host 172.16.30.3 port 22: Host is unreachable
2026-06-05 05:47:18 <ncopa> hm... its responding on ping
2026-06-05 05:55:40 <ncopa> it is back
2026-06-05 06:20:39 <ikke> ok, good
2026-06-05 06:21:03 <ikke> my ssh session survived
2026-06-05 06:21:18 <ikke> it's getting closer
2026-06-05 07:17:00 <ncopa> 20 left. it does not seem to be many that failed either
2026-06-05 07:17:15 <ncopa> maybe we can get it over the finish line today
2026-06-05 14:03:54 <ncopa> build-3-24-riscv64 now builds updates in main again, but it does not appear to have uploaded the current community packages
2026-06-05 14:04:08 <ikke> perhaps due to -k?
2026-06-05 14:04:38 <ncopa> probably, it continuted to build but exited with fail
2026-06-05 18:14:34 <ncopa> i think we can remove the -k now from build-3-24-riscv64
2026-06-05 18:14:59 <ncopa> i removed it
2026-06-05 18:15:34 <ncopa> hopefully we can do rleease next week
2026-06-05 18:15:41 <ncopa> now its weekend
2026-06-06 07:59:58 <cotto> greetings! I sent a mail to the list on May 28, but it doesn't show on the website archive. Could you please check?
2026-06-06 08:00:21 <cotto> (and I haven't received a response, yet)
2026-06-06 08:00:40 <cotto> https://lists.alpinelinux.org/~alpine/infra
2026-06-06 09:07:59 <omni> something is slow
2026-06-06 09:09:09 <omni> I opened an MR fifteen minutes ago and there are still no pipelines
2026-06-06 09:54:27 <omni> looks like things are starting to work again
2026-06-06 10:53:43 <ikke> cotto: Can you send it to infra@alpinelinux.org instead?
2026-06-06 12:00:22 <cotto> can do.
2026-06-06 12:10:21 <ikke> cotto: thanks, received it
2026-06-06 12:37:14 <ikke> omni: load has been quite high for a bit
2026-06-06 12:50:02 <ikke> We received about 120r/s (normally it's below 20)
2026-06-07 06:47:52 <ncopa> the riscv64 builder died again
2026-06-07 06:47:59 <ikke> yes
2026-06-07 06:48:12 <ikke> I mentioned it to you and carlo
2026-06-07 11:40:17 <omni> something with timestamps (?) in CI logs makes them less parsable by syntax highlighters
2026-06-07 15:06:06 <ikke> omni: can you elaborate?
2026-06-07 20:17:05 <omni> ikke: I thought I saw it in several different logs, timestamps to the left plus something else and it did something to the readability of logs both in the web interface and when you piped them (jobs/1234/raw) through bat --lang log
2026-06-07 20:17:17 <omni> I don't see it in any logs I'm currently looking at
2026-06-07 20:17:29 <ikke> perhaps depending on the runner?
2026-06-07 20:17:39 <ikke> may be a new "feature"
2026-06-08 07:48:47 <ncopa> something is wrong with the rpi image generation i think
2026-06-08 07:52:19 <ncopa> nope just the local script that has issues
2026-06-09 04:45:39 <ikke> ncopa: if you have time, could you check the riscv64 edge builder? mariadb fails due to some issue with /var/tmp
2026-06-09 08:19:51 <omni> the sshfs build has some (/var/cache?) issue on build-3-20-riscv64
2026-06-09 10:43:03 <ncopa> I'll have a look at it.
2026-06-09 10:43:17 <ncopa> https://alpinelinux.org/sponsors/ should we remove equinix metal now?
2026-06-09 10:58:46 <ikke> yes
2026-06-09 13:59:47 <ncopa> fatal: remote error: GitLab is currently unable to handle this request due to load (ID 01KTPAWHGPMESWSBASYQ15STGS).
2026-06-09 13:59:57 <ikke> :(
2026-06-09 14:00:28 <ikke> I guess that's some form of rate limiting. Server load is not high atm
2026-06-09 14:01:00 <ncopa> yup. I did sync panes in tmux so all builders did something exactly at the same time
2026-06-09 14:01:33 <ikke> ah
2026-06-10 07:37:06 <omni> we're missing a v3.24 label in gitlab
2026-06-10 08:06:32 <ikke> Ack, will create it
2026-06-10 18:34:38 <f_> is the aarch64 runner down or overloaded?
2026-06-10 18:35:02 <f_> a CI job for it was queued for 2 hours and then it gave up
2026-06-10 18:35:18 <f_> rerunning it says "no active runners online"
2026-06-10 18:36:02 <ikke> They appear to be offline
2026-06-10 18:36:23 <ikke> It is running a job
2026-06-10 18:38:25 <ikke> kicked that job
2026-06-10 18:39:52 <f_> okay it now says pending, thank you!
2026-06-11 11:52:27 <ncopa> someone pinged me in private message on IRC. "just want to be new mirror"
2026-06-11 11:52:27 <ncopa> "nobody replied me at alpine-mirrors@alpinelinux.org"
2026-06-11 12:09:36 <ikke> el-brujo?
2026-06-11 16:12:22 <ncopa> Yes
2026-06-11 20:36:55 <Habbie> mio directed me here for questions about the loongarch64 builder
2026-06-11 20:37:05 <Habbie> https://build.alpinelinux.org/buildlogs/build-edge-loongarch64/community/pdns-recursor/pdns-recursor-5.4.2-r0.log
2026-06-11 20:37:16 <Habbie> this MR passed before merge
2026-06-11 20:37:21 <Habbie> it is not passing now
2026-06-11 20:37:30 <Habbie> my coworker points out that that failing test relies on /tmp working
2026-06-11 20:37:41 <Habbie> is there any interesting difference between /tmp on MR builders and this builder?
2026-06-11 20:37:52 <Habbie> (mio shared the suspicion that MR builds are in docker and this builds was in lxc)
2026-06-12 02:24:00 <pj> gitlab CI runs in docker
2026-06-12 03:02:16 <lotheac> ...or kubernetes
2026-06-12 05:30:09 <ikke> Habbie: in both cases /tmp is just a directory, nothing special
2026-06-12 08:15:12 <achill> is gitlab just for me this slow?
2026-06-12 08:18:55 <qaqland> +1
2026-06-12 09:42:32 <Habbie> ikke, thanks
2026-06-12 09:57:54 <ikke> achill: There was some scraping going on this morning
2026-06-12 09:58:00 <ikke> I did block it
2026-06-12 10:02:47 <achill> thanks!
2026-06-13 09:24:26 <omni> is build-3-22-riscv64 stuck or slowly chugging along?
2026-06-13 09:24:57 <omni> and is build-3-21-riscv64 lost?
2026-06-13 10:09:11 <Habbie> mio and i excluded loong from pdns-recursor testing for now
2026-06-15 13:42:55 <ncopa> looking at the fastly metrics. we have like 5k request per sec now
2026-06-15 13:43:18 <ncopa> thats 2-3 x since last time I checked
2026-06-15 13:43:36 <ncopa> peaks on 11k req/sec
2026-06-15 14:17:51 <clandmeter> we are their highest non paying customer i guess :)
2026-06-15 14:21:40 <achill> damn, i suppose its because we just released a new version?
2026-06-15 14:27:53 <ncopa> i think its more related to it is Monday
2026-06-15 14:28:03 <ncopa> looke like there are dips saturday/sunday
2026-06-15 14:30:05 <ncopa> last month has been around 330M req/day
2026-06-15 14:30:39 <ncopa> on week days
2026-06-15 14:30:53 <ncopa> that is ~3.8k req/s
2026-06-15 14:31:35 <ncopa> oh, thats cache hitss. there are another 4-5M cache misses
2026-06-15 14:32:40 <ncopa> in weekends last month it has been ~160-180M cache hits per day
2026-06-15 14:35:34 <ncopa> the diff of weekdays and weekends is interesting
2026-06-15 14:35:37 <ncopa> per region
2026-06-15 14:36:05 <ncopa> in southamerica there is a visible diff, but a bit less
2026-06-15 14:36:19 <ncopa> in africa you almost not see any diff
2026-06-15 14:36:40 <ncopa> in india it is very visible diff
2026-06-15 14:37:06 <ncopa> australia/newzeeland even less than south amerca
2026-06-15 14:38:02 <ncopa> usa it is also very large diff of weekdays and weekends. the diff is less in Europe
2026-06-15 14:38:16 <ncopa> but it looks like there are more cache misses in Europe
2026-06-17 12:12:29 <ikke> OOM Killer
2026-06-17 15:10:41 <ncopa> im currently holding the build-3-22-riscv64 off til build-3-21-riscv64 has completed the kernel build. I think if there are multiple parallel heavy jobs the probability for deadlocks increase
2026-06-17 15:11:30 <ikke> ack
2026-06-17 16:40:04 <Sertonix[m]> Why is there both https://gitlab.alpinelinux.org/RenovateBot and https://gitlab.alpinelinux.org/renovate-bot ?
2026-06-17 18:13:07 <ikke> Sertonix[m]: One is a bot account on the alpine group, the other is a regular user
2026-06-17 18:13:20 <ikke> I needed the latter to make sure it could access all projects
2026-06-18 04:41:33 <ikke> Hmm, what is eating that space
2026-06-18 09:17:42 <achill> gitlab seems to have a "automatic rebase prior merge" option for the fast-farward merge, would be really cool to have that in aports
2026-06-18 09:17:44 <achill> https://matrix-media-share.postmarketos.org/achill/8YER1xHDSS/clipboard.png
2026-06-18 09:18:34 <ikke> Hmm, interesting
2026-06-18 09:18:53 <achill> just tested it on another repo and "it works" 🤷
2026-06-18 09:20:17 <ikke> I don't see that option. Did you test it on pmos gitlab or alpine gitlab?
2026-06-18 09:20:52 <achill> oh its on pmos which seems to be on gitlab 19
2026-06-18 09:22:26 <omni> ncopa: build-3-21-riscv64 is done now, right? could build-3-22-riscv64 and build-edge-riscv64 come back online? the latter hasn't gotten to upload to community since late may
2026-06-18 09:26:17 <ncopa> trying to prioritizing the stable builders so we could get stable releases out. will have a look
2026-06-18 09:27:08 <ncopa> re gitlab. I have a rebase-merge-mr script that uses glab. very convenient
2026-06-18 09:27:42 <omni> sure, so build-3-22-riscv64 first, then?
2026-06-18 09:27:49 <achill> sure, but when i merge most times from my phone where i dont have that :P
2026-06-18 09:27:49 <ncopa> having it in gitlab ui would be awesome
2026-06-18 09:28:22 <ncopa> yeah. would be awesome to be able to do rebase-merge from phone
2026-06-18 09:30:41 <ncopa> omni: yeah im working on build-3-22-riscv64 now. the hw makes cc1 deadlock
2026-06-18 09:30:54 <ncopa> im manually trying to force it throught the build
2026-06-18 09:32:10 <omni> <3
2026-06-18 09:39:13 <omni> "Job has been erased by root 1 day ago" I haven't seen before
2026-06-18 09:39:28 <omni> (CI pipelines)
2026-06-18 10:20:18 <ikke> omni: I have to periodically clean things up because they take a lot of diskspace
2026-06-18 10:24:57 <omni> ikke: I guessed it was something like that, just hadn't seen it before
2026-06-18 10:43:34 <ncopa> im strugling with the nginx build. it is someetime ifficult to know if it is deadlocked or simply just slow
2026-06-18 11:55:08 <omni> build-3-22-armhf is lost?
2026-06-18 11:55:22 <ncopa> probably not
2026-06-18 11:57:15 <ncopa> will check in a bit
2026-06-18 12:19:47 <ncopa> i finally managed to dance around configure when building nginx on riscv64 builder. it is finally building now. with -j1
2026-06-18 12:25:31 <ncopa> build-3-22-armhf was not lost. it was just a design flaw which is solved with mqtt-exec 0.7.1. (auto publish state on (re)connect)
2026-06-18 14:27:12 <ncopa> i think we need to decommission one of the pioneer machines. it is non-sustainable
2026-06-18 14:35:02 <ikke> What's the alternative?
2026-06-18 14:44:30 <ncopa> manually hand holdign for building basically every package. (login and do manual `abuild build` til it passes)
2026-06-18 14:44:51 <ncopa> we could also drop support for older stable releases for riscv64
2026-06-18 14:45:30 <ncopa> drop 3.21 and 3.22, and move the builders we do maintain to the p550
2026-06-18 14:46:09 <ncopa> i do have a startpro64 here which I still havent been able to get up and running due to lack of a working maintained kernel
2026-06-18 14:47:14 <ncopa> I could try get it running with a vendor kernel or entire vendor distro and run the builders on debian or ubuntu or whatever vendor distro there is
2026-06-18 15:12:47 <lotheac> given that there was an issue on one of the rv64 machines with golang earlier that i traced to being a missing kernel patch that only failed on cpus with vector support... just wondering if the builders have been patched as well
2026-06-18 15:13:12 <lotheac> kernel-wise
2026-06-18 15:37:05 <ncopa> i think they are. this is unreliable hardware. segfaults and deadlocks  at random places.
2026-06-18 15:37:39 <ncopa> I have finally success of passing the configure script for zfs-lts after 5-6 retries
2026-06-18 15:38:05 <lotheac> i had that thought, but honestly it might be outdated kernels equally likely
2026-06-18 15:39:18 <ncopa> the 6.18 kernel *should* have everything needed. I also backported some patches so it should all be there
2026-06-18 15:39:35 <ncopa> but we were never able to get any output from serial console when booting the kernel
2026-06-18 15:39:47 <lotheac> ok, then you're probably right :)
2026-06-18 15:40:04 <lotheac> i think the one i debugged earlier had a pretty old kernel
2026-06-18 15:40:43 <ncopa> i remember we looked at the weird vector support. we did have that problem, but I think we solved it in the kernel
2026-06-18 15:41:25 <lotheac> right, that's why i reacted here -- it was equally weird, it manifested "randomly" (context switches between specific vector insn execution)
2026-06-18 15:41:42 <ncopa> i have no idea why the 6.18 kernel didnt work. it could be we need other boot loader
2026-06-18 15:42:02 <ncopa> or it could be that the new kernel reorder the serial consoles
2026-06-18 15:43:16 <lotheac> given all that, i say let's try replacing the hw
2026-06-18 15:43:27 <lotheac> (i'll chip in if needed)
2026-06-18 15:45:49 <ncopa> i have (pre)-ordered a titan machine in december something. I doubt we ever see it.
2026-06-18 15:46:04 <ncopa> I also ordered a sipeed k3 pico
2026-06-18 15:46:13 <ncopa> we have funding in opencollective
2026-06-18 15:46:36 <ncopa> it has been difficult to get access to the hardware
2026-06-18 15:50:18 <lotheac> oof.
2026-06-18 15:50:28 <ncopa> I'd like this one: https://www.epicsemi.com/products/contrail/contrail-compute/
2026-06-18 15:51:01 <lotheac> if we can't get hardware, then we can't do anything. well, we can decom the old one if it's unreliable, but :D
2026-06-18 16:02:48 <ncopa> lotheac: I wonder if you could investigate what it would take to get one of those machines for Alpine builder use? if it is ok with ikke and clandmeter. if it is possible to get one of those, where could it be physically hosted?
2026-06-18 16:03:38 <lotheac> yeah, sure, i can help out. i can't physically have one though
2026-06-18 16:04:17 <ncopa> thats understandable :)
2026-06-18 16:10:46 <ncopa> looks like the current 3.22 riscv64 build is almost done
2026-06-19 10:17:20 <Sertonix[m]> ncopa: looks like I can't create release tags for abuild. Where should that be changed?
2026-06-19 10:17:40 <Sertonix[m]> (due to protected tags)
2026-06-19 10:18:18 <ikke> Sertonix[m]: I can make you maintainer 
2026-06-19 10:25:11 <Sertonix[m]> Ah, that makes sense. Would be nice
2026-06-19 10:47:07 <ikke> Sertonix[m]: done
2026-06-19 15:25:45 <ikke> sent an email to Marist
2026-06-19 22:35:17 <Advertising[m]> https://t.me/+Ad0IXGnDQFllYmE5
2026-06-20 12:05:51 <omni> are some builders not building?
2026-06-20 12:35:49 <ikke> Like?
2026-06-20 12:37:09 <omni> several of the stable builders
2026-06-20 12:39:20 <omni> if you look from the expat security backports, in #alpine-commits, you'll see that only a few uploaded
2026-06-20 12:40:45 <omni> and looking at build.a.o several seems to be at "pulling git"
2026-06-20 12:48:51 <ikke> Happens when we ask 30+ builders to all pull aports around the same time
2026-06-20 12:51:14 <omni> mhm, so I should pause a bit longer in between?
2026-06-20 12:51:28 <omni> what is blocking where?
2026-06-20 12:59:15 <ikke> Literally `git pull`
2026-06-20 12:59:20 <ikke> on the builders
2026-06-20 13:00:45 <achill> yeah, them pulling all on the same time is where its blocking i guess. maybe some offset, like sleeping a random value between 0 and 15 secs before git pulling could help them?
2026-06-20 13:03:00 <omni> I mean, are they pulling from gitlab and it is gitlab or whatever is fronting that that is blocking? because I don't understand how it would be blocking on the builder side of things
2026-06-20 13:03:55 <ikke> It times out or something like that
2026-06-20 13:04:09 <ikke> the builders are idle when I check them
2026-06-20 13:06:43 <omni> and that sounds a bit like it's the remote side, from the builder pow, that is refusing or failing to respond
2026-06-20 13:07:00 <ikke> yes, it's a DoS
2026-06-20 13:28:30 <omni> but if we're in control over both ends, we should be able to make excepmtions or have different rate limits on where the traffic comes from?
2026-06-20 13:29:31 <ikke> It's not ratelimitting, it's exhausting resources
2026-06-23 03:35:34 <qaqland> gitlab is so slow
2026-06-23 05:10:29 <ikke> qaqland: different from normal?
2026-06-23 05:14:30 <qaqland> it's back now
2026-06-23 05:16:07 <ikke> Monitoring doesn't show any issues when you reported it
2026-06-24 21:22:37 <iggy> gitlab having issues? https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/104182 just took over 2 minutes to fully load. Just the first request for the initial html took 20 secs
2026-06-24 21:28:10 <Sertonix[m]> Yes, it's known
2026-06-25 00:12:12 <Sertonix[m]> Has https://gitlab.alpinelinux.org/alpine/infra/apkbrowser/-/merge_requests/13 been considered for the real infra? Being able to see the commit in less than 30 seconds is quite nice
2026-06-25 00:45:23 <jvvv> The docker-compose.yml has the config.ini as a volume, but I am having trouble finding what that file actually is. I was expecting to see it copy and/or filter the config.ini.example, but that does not appear to be the case.
2026-06-25 00:47:08 <jvvv> Well, not 'what the file actually is' but more 'where that file is actually coming from'
2026-06-25 05:02:41 <ikke> I don't know what happened, but gitaly is no longer able to keep up
2026-06-25 05:04:24 <ikke> We are certainly getting more traffic
2026-06-25 05:42:53 <ikke> Adjust the go-away config to ask for a PoW challenge now in most cases
2026-06-25 05:42:58 <ikke> Adjusted*
2026-06-25 05:47:11 <lotheac> seems anecdotally much faster now
2026-06-25 05:48:49 <ikke> yes
2026-06-25 06:51:22 <omni> <3
2026-06-25 10:37:18 <ikke> cgit has also been gaining traffic
2026-06-25 10:54:56 <f_> :(
2026-06-25 10:55:48 <ikke> Adjusted the go-away config there too
2026-06-25 10:57:23 <f_> Can the difficulty be less high?
2026-06-25 10:57:45 <f_> Difficulty 20 seems too high
2026-06-25 10:58:44 <f_> it takes about 2-3 minutes to complete on my poor 2012 i7
2026-06-25 10:59:30 <dne> some 3.24 builders appear to be stuck in "pulling git", perhaps also affected by scrapers
2026-06-25 12:23:32 <jvoisin> ikke: sorry you have to deal with alpine's infra being crawled to death :<
2026-06-25 12:51:27 <ncopa> it is painful indeed
2026-06-25 12:51:36 <ncopa> things I think we could do:
2026-06-25 12:53:08 <ncopa> - clean up keep-around refs. seems to be complicated and comes with some downsides (broken URLs in gitlab UI). but I think we should delete refs older than 1 year or so.
2026-06-25 12:54:01 <ncopa> - long term: move gitaly to separate dedicated host. probably makes sense to move to Kubernetes?
2026-06-25 12:55:55 <ncopa> - first step towards kubernetes: move postgresql and redis to separate host. this will offload the gitlab instance a bit, and the gitaly problems will not contaminate the db access. downside is that backup/restore gets complicated
2026-06-25 12:56:32 <ikke> f_: will look into it 
2026-06-25 12:57:01 <ikke> ncopa: first step: work out backup strategy 
2026-06-25 12:57:05 <ncopa> - second step would be to move gitlab instance to Kubernetes. we could still continue with gitaly on same node, or we could now move gitaly to their own node(s)
2026-06-25 12:57:27 <ncopa> yeah. backup/restore gets complicated
2026-06-25 12:57:53 <ncopa> compared to what we do today
2026-06-25 13:21:51 <ikke> gitlab does have a backup solution, so we will have to test that out
2026-06-25 14:34:03 <lotheac> cnpg works pretty nicely these days btw. i'm using it
2026-06-25 15:05:46 <ikke> ok, good to know
2026-06-25 15:34:32 <f_> ikke: still says 20 for me on git.a.o
2026-06-25 15:35:33 <ikke> oh, changed it for gitlab, not git.a.o
2026-06-25 15:35:37 <f_> ah
2026-06-25 15:36:25 <f_> oh I see difficulty 15
2026-06-25 15:36:59 <f_> it's quite faster, about 5-6 seconds
2026-06-25 15:37:34 <f_> Maybe difficulty 10 would accomodate slightly lower-end devices?
2026-06-25 15:45:18 <ikke> Lowered it now, hopefully it's enough
2026-06-25 15:46:39 <f_> thank you ikke, it's instant now
2026-06-25 16:34:19 <iggy> step 1. determine if it's worth it to run your own gitlab instance? I think I've seen a solid reason for it in the past, but just making sure the right questions are being asked
2026-06-25 16:36:16 <f_> :(
2026-06-25 18:16:35 <ikke> :-(
2026-06-25 20:25:45 <ikke> f_: I switched it back to 15
2026-06-25 20:59:42 <f_> ikke: scrapers ruining all the fun :( thanks for the heads up
2026-06-25 21:00:27 <ikke> Yes, they do
2026-06-26 05:49:44 <ikke> It's getting worse
2026-06-26 08:40:36 <dne> poor gitlab
2026-06-26 10:03:13 <f_> it's already so frustrating as a contributor, I can't imagine how frustrating it must be for the infra team
2026-06-26 10:51:42 <ikke> They were targeting source archives, which I had bypassing the PoW checks
2026-06-26 10:54:27 <ikke> Wasn't ready to anounce it yet, but I deployed a status page: https://status.alpinelinux.org/
2026-06-26 10:54:32 <ikke> Only available on ipv6
2026-06-26 10:55:09 <ikke> https://status.alpinelinux.org/incidents/c3726dbf-1633-4b59-8966-fa7b06c4717f
2026-06-26 11:08:04 <PureTryOut> Shit, can't view it from home and work then 😢
2026-06-26 11:12:14 <f_> PureTryOut: use tor for free IPv4=>IPv6 /s
2026-06-26 11:12:46 <f_> but all it has for now is what we already know
2026-06-26 11:13:28 <f_> https://ircb.dersco.re/uploads/funderscore/7316d25a-f_-paste.txt
2026-06-26 11:21:59 <PureTryOut> I didn't expect otherwise now, but it would be nice to be able to view the page in the future 😅
2026-06-26 11:39:50 <f_> I'd imagine that IPv4 would work in the future PureTryOut 
2026-06-26 11:42:52 <f_> ikke: are you able to reduce difficulty to 15 again or do these manage to get past the PoW that way?
2026-06-26 12:05:06 <ikke> f_: are you getting difficulty 20? Because it should only do that for specific old browser UAs
2026-06-26 12:09:33 <f_> yes I am
2026-06-26 12:09:51 <f_> Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0
2026-06-26 12:10:00 <ikke> That should not match
2026-06-26 12:11:05 <f_> on gitlab
2026-06-26 12:11:18 <f_> On cgit I get difficulty 15
2026-06-26 12:52:23 <Sertonix[m]> build.a.o shows some riscv64 builders as lost, does something need to be changed?
2026-06-26 14:08:45 <ikke> Sertonix[m]: mqtt-exec needs to be updated so that it shows back as online when the connection recoveres
2026-06-26 14:18:06 <ikke> f_: could you make another request and show the request id?
2026-06-26 14:22:36 <f_> d74c2f4b2a9a8a536073482803042fe7
2026-06-26 14:25:09 <ikke> oh, I apparently increased that one as well
2026-06-26 14:25:38 <ikke> f_: should be lowered again now
2026-06-26 14:26:46 <f_> thanks!
2026-06-26 15:06:48 <ikke> It's quite ironic. They do their best (I suppose) to mask the source by using residential proxies, and then anounce themselves by using ancient browser UAs
2026-06-26 16:17:36 <iggy> they probably are legitimately ancient browsers on the backend
2026-06-26 16:17:56 <iggy> using some 2 year old version of puppeteer or something
2026-06-27 11:25:06 <ptrc> hm, not sure if someone mentioned this, but it seems like the @alpinelinux.org emails that get redirected to Gmail simply bounce everything
2026-06-27 11:25:20 <ptrc> https://ptrc.gay/yLqlgFaH.txt
2026-06-27 11:25:34 <ptrc> getting this from gbr-app-1.alpinelinux.org
2026-06-27 11:26:54 <ptrc> i believe that there should be something in place like https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme
2026-06-27 11:28:30 <ptrc> because otherwise my emails will just be dropped by all servers that enforce email security
2026-06-27 11:37:03 <ikke> Right, I use postsrsd personally 
2026-06-27 11:37:05 <ptrc> seems like it affects clandmeter, fcolista, and rnalrd through Alpine relay, and fabled through painokone.iki.fi which is doing the same thing
2026-06-27 21:52:54 <Sertonix[m]> Does the edge-loongarch64 builder have anything wierd at /tmp/pytest-of-builder? There are test failures that I can't reproduce locally
2026-06-27 21:58:05 <Habbie> Sertonix[m], i recently had a test fail -only- on loongarch, also involving /tmp. i do not know why
2026-06-27 21:58:15 <Habbie> 05:30Z <@ikke> Habbie: in both cases /tmp is just a directory, nothing special
2026-06-27 21:58:18 <Habbie> you can scroll back to this i bet
2026-06-28 00:46:15 <Sertonix[m]> Maybe the permission bits are not set correctly?
2026-06-28 00:59:51 <Habbie> i trust ikke to have checked things like that
2026-06-28 01:00:00 <Habbie> Sertonix[m], tell me about your failures :)
2026-06-28 01:05:05 <Sertonix[m]> https://build.alpinelinux.org/buildlogs/build-edge-loongarch64/community/py3-port-for/py3-port-for-1.0.0-r0.log
2026-06-28 01:08:38 <Habbie> "Directory not empty" does not -feel- like permission failure
2026-06-28 01:16:23 <Sertonix[m]> I think "error removing" is before the "Directory not empty" error.
2026-06-28 01:16:51 <Habbie> i think those two lines are one error
2026-06-28 01:17:01 <Habbie> also, if it managed to make that dir, then it should be able to remove things from it
2026-06-28 01:17:26 <Habbie> but all i have is "i see the error and this one explanation does not fit", i do not have explanations that do fit
2026-06-28 01:18:06 <Sertonix[m]> The "During handling of the above exception, another exception occurred:" part makes me think that they are seperate errors.
2026-06-28 01:18:57 <jvvv> Can this be built single threaded?
2026-06-28 01:19:19 <Habbie> Sertonix[m], ah, i see what you mean. I can't judge that without digging into that codebase or emulating the error
2026-06-28 01:19:26 <Sertonix[m]> It's possible to make a non-empty directory with permission bits set to 0 that would cause "rm -rf" to fail. But that should not happen on loongarch64 specifically.
2026-06-28 01:19:27 <Habbie> it could be separate, but it also really could be not
2026-06-28 01:19:58 <Habbie> Sertonix[m], yes, but that would not be a /tmp permission issue - it would be something pytest did
2026-06-28 01:22:21 <Habbie> (i think - happy to be proven wrong)
2026-06-28 01:22:50 <Habbie> in any case it's interesting that we now have two reports for the loongarch builder involving /tmp
2026-06-29 05:53:10 <ikke> I noticed
2026-06-30 08:25:05 <ikke> If you look carefully, you'll notice where I decided to block an entire AS: https://www.pasteboard.co/378eCw__O_LG.png
2026-06-30 08:38:32 <dne> lovely
2026-06-30 09:51:23 <jvoisin> do you have a small "am I doing something racist?" moment when you do it? I know I do :<
2026-06-30 10:48:24 <ikke> not when the're big companies