2026-03-01 00:48:14 <lotheac> i added some debugging code to my pipeline... on the (siLj_NXx) shared-runner nor-ci-1, which is crashing with SIGILL on a vector instruction, cpuinfo does claim vector support: model name: Spacemit(R) X60 isa: rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt
2026-03-01 00:50:35 <lotheac> and, even if go is compiled with GORISCV64=rva20u64 (which is also the default), it will check at runtime whether the CPU supports RVV 1.0 (vector extensions). and it seems this CPU does, but regardless crashes with SIGILL at internal/bytealg/indexbyte_riscv64.s:83
2026-03-01 00:54:21 <lotheac> i am thinking, maybe we patch out the runtime check and always use the non-vector impls (which is what go1.25 does) and report this to the go people to let them sort it out? i'm not familiar with riscv64 cpu extensions
2026-03-01 06:54:50 <lotheac> the weird thing here is that the commits which implement the vector support in go specifically mention Spacemit X60, which is ostensibly the same CPU as is failing here
2026-03-01 07:29:33 <lotheac> ikke: you around? i could really use a rv64 shell for this instead of trying to hope my CI jobs get scheduled on the correct box :p
2026-03-01 07:32:22 <ikke> Only on milk-v pioneer hw. nor-ci-1 is different
2026-03-01 07:32:53 <lotheac> that doesn't help unfortunately, shared-runner pioneer1 cpu does not support the vector insns and thus does not exhibit the issue
2026-03-01 07:34:56 <lotheac> is nor-ci-1 ncopa's perchance?
2026-03-01 07:37:13 <ikke> Yes
2026-03-01 07:38:02 <lotheac> so i guess i need to talk to him unless i get lucky with CI scheduling
2026-03-01 07:38:46 <lotheac> ncopa: would you mind giving me a temp shell on nor-ci-1 for debugging a vector instruction issue on that specific cpu?
2026-03-01 22:45:11 <aelin> the s390x runner is idle... but has not picked up any of the spirv rebuilds yet. maybe it's broken?
2026-03-03 10:23:26 <aelin> gentle poke, what's the state of the s390x builder?
2026-03-03 10:26:29 <ikke> Will check later.
2026-03-03 10:48:23 <ncopa> something happened with the s390x builder
2026-03-03 10:49:18 <ncopa> it rebooted 3 days ago and it appears that /var was not mounted, so no lxc
2026-03-03 10:52:00 <ncopa> vgdisplay shows scary messages
2026-03-03 10:52:06 <ncopa> vg1 is gone apparently
2026-03-03 10:53:39 <ncopa>   WARNING: VG vg1 is missing PV fKw74P-69tj-NEHO-jWLi-KTCg-ycFF-mfykG6 (last written to /dev/dasdg1).
2026-03-03 10:53:39 <ncopa>   WARNING: VG vg1 is missing PV jHErDq-RYC7-w9u1-iaQo-lCfR-P4mP-WLCra3 (last written to /dev/dasdh1)
2026-03-03 11:17:17 <ncopa> i think this effectively means that we lost the data
2026-03-03 11:23:35 <ncopa> I think I will reboot it again and see if the disks comes back
2026-03-03 11:24:02 <ncopa> usa2-dev1 [~]# uptime
2026-03-03 11:24:02 <ncopa>  11:23:54 up 3 days, 18:32,  0 users,  load average: 0.00, 0.00, 0.00
2026-03-03 11:25:00 <ncopa> nope
2026-03-03 11:25:50 <ikke> :(
2026-03-03 11:26:26 <ncopa> i think we lost  /dev/dasdg1 and /dev/dasdh1 for some reason
2026-03-03 11:26:52 <ncopa> I dont know who we should ask
2026-03-03 11:27:59 <ikke> Martha
2026-03-03 11:28:35 <ncopa> I think it was rebooted Friday 27 February and after that we lost those two disks
2026-03-03 11:34:15 <ikke> Sent an email
2026-03-03 11:38:08 <ncopa> Thanks!
2026-03-03 16:06:31 <ncopa> lotheac: Sorry I can give you a login to nor-ci-1, but not right now. Please ping me tomorrow
2026-03-03 16:44:34 <ikke> ncopa: ping
2026-03-03 16:45:28 <ikke> I got reaction from Marist. They services and rebooted the machine. They expect the new disk configuration was not made persistent. I found that the disks are listed in /etc/zipl.conf, and added the suggested disk numbers there
2026-03-03 16:48:52 <ikke> A ran `zipl`. Rebooting now and see if it comes back
2026-03-03 16:51:50 <ikke> ok, it's back, including the volumes and containers.
2026-03-04 06:14:15 <ikke> aelin: thanks for reporting btw
2026-03-04 06:14:53 <lotheac> ncopa: good morning. ping re: nor-ci-1
2026-03-04 07:35:52 <ncopa> lotheac: good morning! do you have access to the dmvpn network? can you route to 172.21.1.3?
2026-03-04 07:43:07 <lotheac> i do not
2026-03-04 07:43:47 <ncopa> ok. let me try set up some portforward after i eat breakfast
2026-03-04 07:43:51 <lotheac> cheers
2026-03-04 16:35:40 <ikke> ^ is expected
2026-03-05 11:51:41 <ikke> Tweaked this trigger to wait slightly longer
2026-03-06 06:01:48 <huajingyun> jujutsu build has been stuck on build-edge-riscv64 for at least 6 hours, or even longer
2026-03-06 06:03:39 <huajingyun> ikke: do you have time to take a look? thanks
2026-03-06 06:09:41 <ikke> huajingyun: I've killed the build
2026-03-06 06:16:29 <huajingyun> ikke: Okay, thank you. hope it will pass this time
2026-03-08 23:09:12 <omni> looks like build-edge-riscv64 is stuck on jujutsu again
2026-03-09 01:27:17 <huajingyun> omni: yeah, the second build also got stuck. mio has already disabled checks for riscv64, so it might still require ikke to manually kill the build that has entered the 3rd build of jujutsu
2026-03-09 06:50:23 <ikke> huajingyun: killed it again
2026-03-09 06:52:18 <huajingyun> ikke: thanks a lot