2025-11-01 08:24:47 <ncopa> kwidgetsaddons tests was deadlocked on build-3-23-s390x. I restarted it
2025-11-02 12:48:18 <omni> =(
2025-11-02 13:40:35 <omni> \:D/
2025-11-02 13:53:54 <omni> but are things still stuck somehow?
2025-11-02 14:13:31 <ikke> anything I've missed?
2025-11-02 14:16:55 <omni> things are moving again
2025-11-02 14:24:13 <omni> ikke: <3
2025-11-03 06:22:20 <lindsay> Unable to access https://build.alpinelinux.org/buildlogs/ , 404 Not Found
2025-11-03 07:08:01 <qaqland> same
2025-11-03 08:30:31 <clandmeter> we have issues with the server 
2025-11-04 17:17:13 <Ariadne> clandmeter: is there anything i can do to help resolve these problems with build.a.o?
2025-11-04 18:31:32 <vkrishn> and no-one has spoken about it in 24hrs+, and its not even Friday
2025-11-04 19:04:20 <ikke> Ariadne: do you have experience with zfs? We regularly get kernel panics. The data is stored on a linode volume, no idea if that's a single disk that can become corrupt.
2025-11-04 19:04:49 <Ariadne> i do have some experience with zfs
2025-11-04 19:05:25 <Ariadne> do you have an example of the kernel panic?
2025-11-04 19:10:20 <ikke> WARNING: zfs: adding existent segment to range tree (offset=2365157c00 size=800)
2025-11-04 19:12:28 <ikke> zpool status -v showed a bunch of files that could not be restored 
2025-11-04 19:14:10 <Ariadne> in that case, the volume is cooked
2025-11-04 19:14:39 <ikke> Yeah, I figured 
2025-11-04 20:31:40 <omni> ok, I'll say it, why not mirror two volumes?
2025-11-04 20:35:30 <ikke> Cost versus available budget 
2025-11-04 20:36:19 <ikke> Disk space, contrary to popular believe, is not cheap 
2025-11-05 01:27:37 <lotheac> generally i would expect cloud block storage to be resilient against corruption. my point being that if it was corrupted due to software we run and not any external factor, then mirroring would not even have helped
2025-11-05 01:28:48 <lotheac> but i agree with the ariadne. probably best to throw away that volume, salvage the data that is salvageable, and make a new one
2025-11-05 01:29:02 <ikke> working on that as we speak
2025-11-05 01:30:23 <lotheac> cool :)