2026-09-17 19:37:42 tomalok: new stable releases are out 2026-09-17 21:32:18 hi folks! I'm trying to get Alpine Cloud (generic) images to grab CIDATA from an ISO image at /dev/vdb but it doesn't seem to like the idea 2026-09-17 21:32:29 am I doing something obviously wrong? 2026-09-17 21:53:30 and it's the python cloud-init version, not the tiny-cloud-init one? 2026-09-17 21:54:04 I got the "generic" one from https://www.alpinelinux.org/cloud/ 2026-09-17 21:54:11 oh 2026-09-17 21:54:15 oh I understand everything now 2026-09-17 22:08:25 okay no I tried the python cloud-init images and there doesn't seem to be any difference 2026-09-17 22:08:33 it also doesn't seem to pick up the network interface for some reason 2026-09-17 22:08:41 just sits there at login prompt 2026-09-17 22:09:30 no, sorry, it does configue the network interface after all. just doesn't run cloud-init 2026-09-17 22:16:27 oh, I see what happened; it picked up the wrong terminal to print output to in the middle of the boot process 2026-09-17 23:17:00 ncopa thanks, i'll start working on building the new images 2026-09-17 23:20:16 figured it all out, many thanks to minimal 2026-09-17 23:20:46 turns out i needed to pass UsePAM yes to sshd, and the tty selection in initramfs is wrong for this virtual hardware configuration 2026-09-19 22:49:40 ncopa, ikke: new cloud releases are staged and ready for CDN sync 2026-09-19 22:55:08 ikke: also https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/136 once CDN sync is complete 2026-09-21 05:49:07 tomalok: images are syncing now 2026-09-21 10:23:58 syncing is done 2026-09-21 15:00:55 ikke: thanks, looks like ncopa's merged the MR, but it's not live yet. 2026-09-21 16:57:43 merged now 2026-09-21 16:57:56 we need to merge to branch 'production' 2026-09-21 18:27:46 aha! thanks! 2026-09-23 02:48:04 hello! I'm seeing alpine-3.23 images are taking a long time to boot up now (aarch64 takes ~10 minutes, amd64 a bit less). I'm using e.g. alpine-3.23.6-aarch64-uefi-cloudinit-r0, ami-08255352ef3c448f0 in eu-central-1 2026-09-23 02:51:44 I see this in cloud init log: https://gist.github.com/thresheek/5c0417731df360fdcc21ee619a702460 2026-09-23 02:52:34 This seems to happen in two different AWS accounts, different regions, using only 3.23 - seemingly other versions (I test 3.17 to 3.24) are fine 2026-09-23 02:53:27 Any idea what might cause dhcp to be that bad only on 3.23? 2026-09-23 02:54:03 Nothing about the accounts/regions is shared btw, and dhcpd is just whatever AWS has 2026-09-23 03:48:29 thresh_ - please create an issue with as much details as you can over at https://gitlab.alpinelinux.org/alpine/cloud/alpine-cloud-images/-/work_items 2026-09-23 03:52:11 cloudinit didn't like dhcpcd until 3.21, used dhclient before that, but 3.21+ it's been dhcpcd. 2026-09-23 19:35:13 thanks tomalok, there you go: https://gitlab.alpinelinux.org/alpine/cloud/alpine-cloud-images/-/work_items/194 - I believe 3.23 lacks a specific datasource for EC2 for some reason 2026-09-23 19:57:23 thresh_: yes if you're missing the datasource definition that would very likely cause something like this to happen. 2026-09-23 19:57:59 But the definition of that datasource hasn't changed in tomalok's image system for a couple of years I think, so it's odd that it's absent 2026-09-23 19:58:12 I agree, but that's what I get when I boot up that AMI 2026-09-23 20:16:50 the the cloud-init datasource should be auto-detected. there is no longer any difference between images that run on GCP, Azure, AWS, etc. 2026-09-23 20:19:13 I think i've been trying an older Alpine 3.24 image where it worked, it seems the naming scheme changed (-uefi- got dropped). Testing with a new one now 2026-09-23 20:19:34 tomalok, it does try to autodetect, and this means two timeouts each five minutes now 2026-09-23 20:21:43 yeah, the same issue on alpine-3.24.2-aarch64-cloudinit-r0, timeouts. works on alpine-3.24.1-aarch64-uefi-cloudinit-r0 2026-09-23 20:23:47 no autodetect should ever take that long. Tiny Cloud should figure it out almost instantaneously. 2026-09-23 20:24:28 s/Cloud should/Cloud/ 2026-09-23 20:25:32 I would rather use cloud-init so the image bootup process would be similar to the rest of my fleet (lots of other OSes) 2026-09-23 20:28:27 i get that... though tiny cloud's #cloud-config compatibility's improved quite a bit -- dunno what specific #cloud-config you use. 2026-09-23 20:29:26 but ultimately, "what happened" is what needs to be answered. have you tried the latest 3.22 release (.6, i think)? 2026-09-23 20:32:36 booted up alpine-3.22.6-aarch64-uefi-cloudinit-r0 / ami-0c528d25c2ccc0c7d (eu-central-1), works fine 2026-09-23 20:32:56 I suppose it's a dhcpcd bug then 2026-09-23 20:34:37 it it's a dhcpcd bug then it'd also be apparent with the tiny cloud images 2026-09-23 20:35:09 what version of cloudinit is in 3.22.6 vs 3.23.6 vs 3.24.1 vs. 3.24.2? 2026-09-23 20:38:26 would tiny cloud launch dhcpcd the same way as cloud-init does? 2026-09-23 20:39:35 yeah, pretty much. 2026-09-23 20:48:04 3.24.1 - not an issue, cloud-init 26.1 -- 3.24.2 - an issue, also cloud-init 26.1 2026-09-23 20:49:14 3.22.6: cloud-init-24.3.1-r2, 3.23.6: cloud-init-25.3-r0, 3.24.1: 3.24.1 and 3.24.2: cloud-init-26.1-r3 2026-09-23 20:53:26 okay, i see a potential issue. latest round of images are "generic" so they're supposed to support AllTheClouds™. There is an auxiliary setup-cloudinit script that runs when the image gets built that used to set datasource according to the cloud -- but would leave it to autodetection if cloud was "generic". i'll review that part to see if 2026-09-23 20:53:26 there's a better way to explicitly say "autodetect" -- but there's no way autodetection should take 5m every time it needs to figure out what cloud it's on. 2026-09-23 21:01:45 might be a missing init script that's not put in place to pre-identify the datasource. 2026-09-23 21:03:18 autodetection works as intented though, it's just dhcpcd that's blocking for 300 seconds for some reason 2026-09-23 21:05:46 yeah, but there's an init script that figures it out more efficiently and saves the result. there's a setup-cloud-init script in our aports where that additional init script should be put in place. 2026-09-23 21:11:13 our cloud-init package breaks datasources into individual subpackages, is there not a way to just install what's needed for a specific provider instead? 2026-09-23 21:11:31 though, I guess in that case you'd want to statically set the datasource config. 2026-09-23 21:23:43 we don't want to hardcode the datasource per image -- we've been there and we already have too many images :) 2026-09-23 21:23:43 i'm taking a look at enabling /etc/init.d/cloud-init-ds-identify to run before cloud-init-local 2026-09-23 21:24:27 before i enable it on the test machine, i'm going to reboot it and watch again and see if it takes a while to come up 2026-09-23 21:25:54 much quicker on the reboot. though i don't know if that's because i ran cloud-init-ds-identify by hand 2026-09-23 21:29:51 nm, that was 3.24.1, not 3.24.2 2026-09-23 21:37:20 unfortunately I cant get 3.24.2 tiny to accept my user with a proper, but the networking does indeed come up quite fast 2026-09-23 21:37:30 proper userdata that works on cloud-init that is 2026-09-23 21:43:33 as long as you're not doing anything too crazy, submit an issue vs. the tiny-cloud project -- either there's something unsupported or out of alignment 2026-09-23 21:45:16 3.24.2 does indeed stall after a reboot too 2026-09-23 21:46:18 yep, will do re userdata 2026-09-23 21:56:57 reboot also had the pause, i added cloud-init-ds-identify to boot runlevel... 2026-09-23 21:58:02 ...still seeing a pause... 2026-09-23 21:58:26 I've booted up alpine-3.24.1-aarch64-uefi-cloudinit-r0 (which works fine), upgraded just dhcpcd via `apk add --upgrade dhcpcd), rebooted, and now it wont come up 2026-09-23 21:58:32 waiting for it to boot up now 2026-09-23 22:00:05 yep, same issue even with "datasource_list: ["Ec2"]" in the cloud.cfg 2026-09-23 22:00:57 so the problem is in dhcpcd that was upgraded from 10.3.2 to 10.5.2 2026-09-23 22:01:33 cloud-init autodetection just amplifies the issue - it hangs twice for 300 seconds, while without autodetection it hangs only once 2026-09-23 22:10:02 adding ds-identify seems to have improved it from 10m to 5m 2026-09-23 22:10:21 what changed with dhcpcd, and why doesn't it affect tiny-cloud? 2026-09-23 22:11:42 re format that remains the question, but re latter it seems tiny-cloud would use a different invocation of dhcpcd 2026-09-23 22:11:47 re former 2026-09-23 22:12:24 Sep 23 22:02:04 ip-172-31-9-101 daemon.info dhcpcd[1889]: eth0: changing route to 2600:1f13:71a:6f02::/64 2026-09-23 22:12:36 getting a lot of these in /var/log/messages for some reason 2026-09-23 22:13:29 i'm bisecting dhcpcd releases, trying out 10.5.0 now 2026-09-23 22:17:39 did we drop dhclient? last in 3.20 2026-09-23 22:46:01 If you set the datasource directly does the 5m delay drop to a few seconds? 2026-09-23 22:47:48 i can try that and reboot -- where do i set it? 2026-09-23 22:48:08 it's set directly in the case of 3.24.1 + dhcpcd 10.5.2, and no, it's five minutes 2026-09-23 22:49:05 tomalok: /etc/cloud/cloud.cfg 2026-09-23 22:49:27 but this looks like competition between normal networking and cloud-init -- i'm seeing one ipv4 dhcpcd started by cloud-init persisting while the normal networking sets up a different one for ipv4 & ipv6 2026-09-23 22:49:45 thresh_: I checked a couple of my aws boxes earlier today and saw the opposite, they're x64's though and were deploy in July 2026-09-23 22:50:29 Not to say the issue isn't there, but maybe the timing helps narrow down where the issue came from 2026-09-23 22:51:56 thresh_: on the boxes you're seeing the delay on what does the setup look like? Bog standard alpine ami with no modifications? 2026-09-23 22:52:50 pretty much, just the ssh key injected via ec2 userdata 2026-09-23 22:53:48 switching to one datasource is still getting the 300s pause 2026-09-23 22:54:21 good to see you reproduced it, appreciate taking a look 2026-09-23 22:55:19 need to test this on alpine 3.22 -- earlier version of dhcpcd. 3.23 & 3.24 & edge use the same 2026-09-23 22:59:21 yeah, I'm manually bisecting the dhcpcd git repo to figure out the offending commit now 2026-09-23 23:08:22 ok, one last test for me today -- i'm going to tell cloud-init to prefer udhcpc instead of dhclient 2026-09-23 23:26:36 there was a reason we used to use dhclient -- udhcpc didn't work. 2026-09-23 23:36:01 using dhcpcd git master, reverted https://github.com/NetworkConfiguration/dhcpcd/commit/620188969138523473a390c6a69166d4587673e6 and https://github.com/NetworkConfiguration/dhcpcd/commit/6b8d4e11ffedd1d2e02c987a94c2c8569ac28be1 and there is no 300s timeout now 2026-09-24 00:24:40 this looks very related: https://github.com/NetworkConfiguration/dhcpcd/pull/733 2026-09-24 00:32:54 the patch in that PR fixes the issue for me, too 2026-09-24 02:54:29 thanks for hunting that down thresh_ 2026-09-24 15:20:28 dhcpcd patched in aports: !108713 2026-09-24 16:20:49 backported to 3.24-stable and 3.23-stable too. 2026-09-24 16:23:53 i will build some revised 3.23.6 & 3.24.2 cloudinit images once i see 10.5.2-r1 reach pkgs.alpinelinux.org 2026-09-24 16:39:56 thanks for taking care of it! 2026-09-24 16:40:00 niiice! 2026-09-24 17:40:47 tomalok: please don't merge changes to cloud-init without at least notifying me. The change you made to call ds-identify will fail on some platforms if we don't add additional dependencies to the package. 2026-09-24 18:00:23 https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/108718 <- I've opened an MR to add what's missing 2026-09-24 18:19:54 I think dhcpcd-10.5.2-r1 is in 3.23 and 3.24 repos now, at least aarch64 gets that here. Guess I can start the rebuilds 2026-09-24 18:22:12 durrendal -- apologies. 2026-09-24 18:24:05 thresh_ - do you roll your own images, or do you use the ones from alpinelinux.org/cloud? 2026-09-24 18:25:22 I'm using the base images you publish to AWS, and then I create my own stuff on top of them 2026-09-24 18:26:54 it may be some hours before they're imported, published, and released. i will do a quick test after the import 2026-09-24 18:27:28 By "start the rebuilds" I meant I'm taking a current (broken) AMI, upgrade the packages in it (I always do that anyway), apply my other modifications, etc. So the resulting image will not have a boot delay 2026-09-24 18:27:54 Yeah, it's fine, those things take time, especially when copying between the regions etc 2026-09-24 18:37:00 tomalok: it's okay, I appreciate the help, I just want to be involved so that I can also be helpful. 2026-09-24 18:37:43 I depend on cloud-init and your alpine images, so insofar as I can help ensure they work reliably and well I want to :) helping with ci is at least one of those ways 2026-09-24 18:42:37 durrendal: 👍 2026-09-24 18:58:44 (confirmed the old image + dhcpcd upgrade == no issue) 2026-09-24 19:20:14 okay, now let's try importing using the right AWS account this time 🤦 2026-09-24 21:16:39 verified fresh 3.24.2 aarch64 cloudinit r1 AMI no longer has the long timeouts. working on publishing them... 2026-09-25 14:28:17 ikke: we have -r1 cloudinit images for 3.24 and 3.23 ready for CDN sync -- an update to alpine-mksite will follow once those are out the door 2026-09-25 17:16:22 ikke: MR https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/137 once CDN's synced with the updated cloudinit cloud images