2025-11-01 08:24:47 <ncopa> kwidgetsaddons tests was deadlocked on build-3-23-s390x. I restarted it
2025-11-02 12:48:18 <omni> =(
2025-11-02 13:40:35 <omni> \:D/
2025-11-02 13:53:54 <omni> but are things still stuck somehow?
2025-11-02 14:13:31 <ikke> anything I've missed?
2025-11-02 14:16:55 <omni> things are moving again
2025-11-02 14:24:13 <omni> ikke: <3
2025-11-03 06:22:20 <lindsay> Unable to access https://build.alpinelinux.org/buildlogs/ , 404 Not Found
2025-11-03 07:08:01 <qaqland> same
2025-11-03 08:30:31 <clandmeter> we have issues with the server 
2025-11-04 17:17:13 <Ariadne> clandmeter: is there anything i can do to help resolve these problems with build.a.o?
2025-11-04 18:31:32 <vkrishn> and no-one has spoken about it in 24hrs+, and its not even Friday
2025-11-04 19:04:20 <ikke> Ariadne: do you have experience with zfs? We regularly get kernel panics. The data is stored on a linode volume, no idea if that's a single disk that can become corrupt.
2025-11-04 19:04:49 <Ariadne> i do have some experience with zfs
2025-11-04 19:05:25 <Ariadne> do you have an example of the kernel panic?
2025-11-04 19:10:20 <ikke> WARNING: zfs: adding existent segment to range tree (offset=2365157c00 size=800)
2025-11-04 19:12:28 <ikke> zpool status -v showed a bunch of files that could not be restored 
2025-11-04 19:14:10 <Ariadne> in that case, the volume is cooked
2025-11-04 19:14:39 <ikke> Yeah, I figured 
2025-11-04 20:31:40 <omni> ok, I'll say it, why not mirror two volumes?
2025-11-04 20:35:30 <ikke> Cost versus available budget 
2025-11-04 20:36:19 <ikke> Disk space, contrary to popular believe, is not cheap 
2025-11-05 01:27:37 <lotheac> generally i would expect cloud block storage to be resilient against corruption. my point being that if it was corrupted due to software we run and not any external factor, then mirroring would not even have helped
2025-11-05 01:28:48 <lotheac> but i agree with the ariadne. probably best to throw away that volume, salvage the data that is salvageable, and make a new one
2025-11-05 01:29:02 <ikke> working on that as we speak
2025-11-05 01:30:23 <lotheac> cool :)
2025-11-05 23:25:26 <ikke> I'm trying to get as much data as possible copied over to the new volume
2025-11-05 23:32:20 <vkrishn> these are just distfiles?
2025-11-05 23:37:08 <vkrishn> restoring only for edge, then last 1 release should get things started
2025-11-05 23:38:48 <vkrishn> rest can be done later, in passive/pull way
2025-11-05 23:44:13 <ikke> The crashes happen while fetching edge and v3.23
2025-11-05 23:45:00 <ikke> The other data hardly caused issues and has been synced already 
2025-11-05 23:47:19 <vkrishn> ok, build.a.o seems back :)
2025-11-06 00:01:41 <ikke> Yes, I'm keeping an eye on it
2025-11-06 00:21:22 <mio> could someone check on the 3.23 loongarch64, x86 and x86_64 builders?
2025-11-06 00:21:48 <ikke> mio: heh, I was just checking all builders :P
2025-11-06 00:22:03 <mio> great timing then, thanks
2025-11-06 00:23:20 <mio> for polari, the 2nd command in check() is unfortunately causing the build to not exit, still looking into the reason
2025-11-06 00:25:08 <mio> py3-anyio should be okay next retry
2025-11-06 00:25:12 <ikke> ok
2025-11-06 00:30:06 <mio> is it possible to request upload access to dev.a.o please? 1-2 aports have source archive missing/no longer available
2025-11-06 01:02:56 <ikke> I have switched distfiles to use the new volume
2025-11-06 01:07:02 <mio> thanks!
2025-11-06 01:52:14 <ikke> bootstrapping openjdk17 on x86_64
2025-11-06 01:54:37 <ikke> mio: can you try to ssh into mio@dev.alpinelinux.org?
2025-11-06 01:55:49 <mio> thanks! ah, not yet, permission denied
2025-11-06 01:56:14 <mio> sorry, meant just tried to do so and got an error
2025-11-06 01:56:44 <mio> (polari should be okay now as well)
2025-11-06 01:58:28 <ikke> I used the ssh keys from gitlab
2025-11-06 01:58:48 <mio> ah, one moment
2025-11-06 02:00:09 <mio> works, thanks!
2025-11-06 02:00:32 <ikke> There's a symlink in your homedir to the archive directory
2025-11-06 02:01:10 <mio> fantastic, thanks very much!
2025-11-06 02:19:52 <ikke> bootstrapping openjdk11 on aarch64
2025-11-06 02:34:40 <ikke> bootstrapping openjdk8 on s390x
2025-11-06 07:41:08 <omni> has anything changed wrt aarch64 CI in the past few days?
2025-11-06 07:41:52 <omni> I'm trying to run for !92407 and it is slower than before and consistently runnig out of memory
2025-11-06 12:27:18 <ikke> omni: nothing has changed
2025-11-06 12:43:20 <ikke> omni: perhaps another heavy job that was running?
2025-11-06 19:16:03 <omni> I've been retrying now and then the past couple of days
2025-11-06 19:17:23 <omni> last run wasn't as slow, but still running out of memory like it didn't last week
2025-11-06 23:27:45 <vkrishn> build-edge-x86_64 seems down, but alert is not getting triggered
2025-11-06 23:28:12 <mio> probably hanging, not necessarily offline
2025-11-06 23:28:44 <mio> as in hanging build, which may explain no notification
2025-11-06 23:31:35 <vkrishn> maybe a regular interval heartbeat
2025-11-06 23:36:46 <vkrishn> i put 2min cron sending devices thermal data to redis/mqtt
2025-11-06 23:41:35 <vkrishn> nope, currently its just battery status
2025-11-07 13:36:52 <ikke> ..
2025-11-07 13:37:09 <ikke> Still working here 
2025-11-09 00:01:30 <ikke> Upgrading gitlab
2025-11-09 00:45:19 <ikke> bootstrapping ghc on x86_64
2025-11-10 16:44:04 <ncopa> bootstrapping openjdk21 on s390x
2025-11-10 20:26:54 <ncopa> im bootstrapping openjdk21 on build-3-23-riscv64
2025-11-10 20:28:56 <ikke> 👍
2025-11-11 08:12:17 <ncopa> ghc was bootstrapped on aarch64
2025-11-11 08:17:35 <ikke> 👍
2025-11-11 14:25:56 <ncopa> im bootstrapping openjdk21 on build-3-23-x86_64
2025-11-11 14:29:30 <mio> could more space be added to build-3-23-x86 please for rust build? "No space left on device (os error 28)"
2025-11-11 14:34:47 <ncopa> added 200G
2025-11-11 14:40:07 <mio> thanks!
2025-11-12 07:18:56 <angeloverlain[m]> uh, can someone take a look at this? https://gitlab.alpinelinux.org/alpine/infra/apkbrowser/-/merge_requests/22 would be plenty useful to make sure we don't have outdated pkgs in aports