2023-05-01 01:21:43 ncopa: some more structural changes to tiny-cloud went in today -- did not get to auto-install related features -- before doing that wanted to test and see if cloud images were still working first. images build and get imported, but looks like tiny-cloud-early is firing off before the rw remount (and it's currently trying to ensure /var/lib/cloud 2023-05-01 01:21:43 dir is there) but it does succeed later, and then tiny-cloud-main and tiny-cloud-final... though i'm not seeing the individual steps being taken (or not?) because they're no longer wrapped in ebegin/eend? but i'm no longer able to SSH in, so there's something not working. 2023-05-01 01:25:12 fwiw tiny-cloud-3.0.0_rc0 apk is in edge, will do more troubleshooting during the coming week as i can (more time for this next weekend) 2023-05-01 12:49:40 tomalok: thank you very much! I will see if I can simulate aws meta data server in a vm here and try debug it 2023-05-01 14:42:26 ncopa: i'm probably going to switch from defining early() main() final() functions to list of functions in VARs that get iterated over -- can keep the "doing this thing..." output out of the action functions -- and take care of the associated SKIP_ACTIONS in one place 2023-05-01 14:47:22 yeah, i think thats a good idea 2023-05-01 14:47:40 TINY_CLOUD_EARLY_ACTIONS="... ... ..." 2023-05-01 14:47:42 etc 2023-05-01 14:49:48 and the case statement could simply just set ACTIONS="$ACTIONS_EARLY" etc. 2023-05-01 14:50:21 but alas it's time for that DayJob™ 2023-05-01 15:28:56 have a good work day! 2023-05-01 15:29:26 im setting up a test letwork here in my libvirt. got uefi http boot working 2023-05-02 04:37:16 ncopa: got a MR for switching from early/main/final functions to iterating over the contents of a variable, and fixed output for during openrc startup -- but just realized i need to make some fixes to tests now too. will do that, and test some more tomorrow. also looks like the first 'assemble-interfaces' is unable to do its job during early 2023-05-02 04:37:16 because /etc/network isn't readwrite when the mdev hotplug gets installed -- maybe we don't need to do that, or maybe we need to ship a default /etc/network/interfaces -- anyways, that's all for today. 2023-05-02 08:15:34 I’ll try have a look at it later today or tomorrow 2023-05-02 15:42:30 tomalok: I added a commit which fixes the tests in your MR 2023-05-02 17:44:55 ncopa: yep, saw that, thanks! back to the DayJob™ for now, will work more on this tonight. 2023-05-03 04:04:15 ncopa: tiny-cloud-3.0.0_rc1 APK is out -- haven't tested the upgrade scripts yet -- but it's working with a freshly minted alpine-cloud-images AWS edge image (logs and console output). one thing i noticed though is that the AWS NVMe symlinks didn't show up... 2023-05-03 04:07:37 mdev -s puts them in place -- both sd and xvd? looks like the nvme_ebs_liinks hotplug got put into place (but after the devices already showed up) -- this might have been why i originally put "early" in sysinit runlevel... though i thought that mdev-conf-4.4 should have also done the symlinking? 2023-05-03 04:16:40 disabled the nvme_ebs_links hotplug, set the tiny-cloud bootstrap to incomplete, and rebooted -- yeah, those nvme symlinks aren't being created (even after manually doing an mdev -s 2023-05-03 12:43:09 tomalok: what does `grep nvme /etc/mdev.conf` show? 2023-05-03 12:43:16 nvme.* root:disk 0660 */lib/mdev/persistent-storage 2023-05-03 12:44:15 apk will not overwrite mdev.conf it is modified. it will install new mdev.conf as mdev.conf.apk-new 2023-05-03 14:38:59 ncopa: mdev-conf gets installed before the tiny-cloud-early adds... 2023-05-03 14:39:05 -nvme[0-9]+n.* root:disk 0660 */lib/mdev/nvme-ebs-links 2023-05-03 14:39:10 ...above... 2023-05-03 14:39:22 nvme.* root:disk 0660 */lib/mdev/persistent-storage 2023-05-03 14:40:35 ... but even after removing the nvme-ebs-links line and doing "mdev -s" the device links still do not appear 2023-05-03 14:51:59 ncopa: persistent-storage is looping trying to do the "nvme id-ctrl /dev/nvme0n1" but not getting anything set for ebs_alias 2023-05-03 14:55:16 ncopa: i see what's missing. you need to use nvme id-ctrl -b  -- without it you'll get truncated at the first NUL and never reach where it says "xvda" 2023-05-03 14:56:51 ...and when I add that to /lib/mdev/persistent-storage we get the symlinks :) 2023-05-03 15:11:10 it is the -b option that is missing? 2023-05-03 15:11:14 yep 2023-05-03 15:11:52 hum, my testcase does not catch that either 2023-05-03 15:11:57 will fix it 2023-05-03 15:12:07 👍 2023-05-03 15:12:53 i'm building tiny-cloud-3.0.0_rc1-r1 to not install nvme-ebs-links for AWS, and then it's DayJob™ time 2023-05-03 15:14:51 how can i test it? 2023-05-03 15:15:25 the the nvme id-ctrl -b? 2023-05-03 15:15:40 the aws cloud image 2023-05-03 15:15:50 i suppose I need to create an image using packer myself? 2023-05-03 15:15:59 and upload it to aws somehow 2023-05-03 15:16:13 i can get this into my AWS account and make it public for testing 2023-05-03 15:16:57 i pushed mdev-conf-4.4-r2 with the ebs fix 2023-05-03 15:18:02 thanks. 3.0.0_rc1-r1 is merged, not sure how long before APKs arrive in CDN 2023-05-03 15:18:25 max 15 mins usually 2023-05-03 15:18:35 i think its synced each 15 min 2023-05-03 15:18:49 do you have a preferred arch for the image? 2023-05-03 15:18:56 nah 2023-05-03 15:19:07 it'll be aarch64 then 2023-05-03 15:19:17 but i'd like to be able to build the image myself, so I can experiment with things, and test what I break 2023-05-03 15:20:12 once the APKs are in place I'll ./build --clean import --only edge aws uefi tiny aarch64 and toggle the public switch. 2023-05-03 15:28:32 for testing i've been pasting in updates to /sbin/tiny-cloud or /lib/tiny-cloud/... on an instance, running from command line, marking bootstrap incomplete, rerunning, watching on console and rebooting, etc. 2023-05-03 15:31:05 👍 2023-05-03 15:31:36 mdev-conf-4.4-r2 just appeared... and there's the new Tiny Cloud too 2023-05-03 15:33:10 heh, apparently the last edge image i built was also 20230503 (utc), needed to add --reviseto build a "-r1" image 2023-05-03 15:35:22 confirmed the new APKs are going into the image 2023-05-03 15:36:33 (ah, didn't get a -r1 because it wasn't officially published) 2023-05-03 15:40:49 and if I want use v3.18 repos? 2023-05-03 15:42:45 well, you could modify /etc/apk/repositories & downgrade? -- not sure if 3.18 diverges a lot from edge yet? 2023-05-03 15:42:56 they should be identical 2023-05-03 15:43:04 good 2023-05-03 15:43:07 v3.18 is just a complete rebuild 2023-05-03 15:43:40 ah, even the pkgs that didn't change get rebuilt (because gcc, etc has changed) 2023-05-03 15:43:48 yes 2023-05-03 15:45:40 i get this error: $ ./build --clean import --only edge aws uefi tiny x86_64 2>&1 | tpaste 2023-05-03 15:45:40 https://tpaste.us/lgOg 2023-05-03 15:45:46 it's in us-west-2 -- ami-0229e85697c0d2332 2023-05-03 15:46:23 ah, yes, i have a local fix for that that i haven't pushed yet. i can do that in a moment 2023-05-03 15:49:59 https://gitlab.alpinelinux.org/alpine/cloud/alpine-cloud-images/-/merge_requests/151 merged 2023-05-03 15:55:31 thanks 2023-05-03 15:55:50 also +3.18 and -3.14 2023-05-03 15:58:22 can i build a local image? which I can boot in qemu? 2023-05-03 15:59:24 yeah, you should be able to do that. if you do ./build local instead of import you'll get .qcow2 images 2023-05-03 16:01:25 actually you probably don't want to go beyond the "local" target anyways -- the default "upload" target is my web space on dev.alpinelinux.org 2023-05-03 16:02:00 configs -> state -> local -> upload -> import -> publish -> release (the various build target steps) 2023-05-03 16:02:34 but i think local should also create a .vhd suitable for importing to AWS 2023-05-03 16:02:50 $ ./build --clean local --only 3.18 aws uefi tiny x86_64 2>&1 | tpaste 2023-05-03 16:02:50 https://tpaste.us/d1Zo 2023-05-03 16:03:09 botocore.exceptions.NoRegionError: You must specify a region. 2023-05-03 16:03:51 maybe i should run it on my macbook instead 2023-05-03 16:05:33 ah, so there's an old dependency for figuring out what the latest stuff / whether an image has been published for AWS -- you'll need to set up your AWS configs... ~/.aws/ at least minimally 2023-05-03 16:05:47 i want to get rid of that and just use the uploaded metadata as the source of truth 2023-05-03 16:07:28 you might be able to also export AWS_REGION=... but i think it's going to also want creds to do the check 2023-05-03 16:09:13 i have creds. need to figure out how to configure ~/.aws 2023-05-03 16:09:18 :) 2023-05-03 16:09:32 i'll check back again a little later... 2023-05-03 16:22:43 got it built with this diff: https://tpaste.us/4aOO 2023-05-03 16:27:35 and it boots in qemu 2023-05-03 18:52:58 nocloud does not work at all with an alpine release iso, for ex alpine-virt 2023-05-03 18:53:10 problem is that there is nothing that creates /etc/network/interfaces 2023-05-03 18:53:26 I think it would make sense to run assemble-interfaces in early phase 2023-05-03 20:49:46 assemble-interfaces is a part of tiny-cloud-network and that's only pulled in as a dep for AWS 2023-05-03 20:50:19 at the moment 2023-05-03 21:20:01 doesn't the base image (or ifupdown-ng) provide some kind of default /etc/network/interfaces file? if there was no tiny-cloud involved, how does that get there? 2023-05-03 22:53:26 tomalok: setup-interfaces AFAIK 2023-05-04 00:31:03 but there's nothing pre-set in place. 2023-05-04 00:39:10 I don't believe so 2023-05-04 02:15:27 so for tiny-cloud maybe we pull tiny-cloud-network back into the main package and have a post-install script that ensures that a basic /etc/network/interfaces gets put into place... so the images have that well before the first time they boot. 2023-05-04 04:58:36 base image does not provide any default interfaces file because there is no way to know what network hardware will be there at build time 2023-05-04 04:59:24 we cannot do it from post-install either because network driver may not be loaded at install time 2023-05-04 05:00:06 I'll create an issue with the long answer, but basically, we need something that configures thing at runtime, during boot 2023-05-04 05:01:51 which is why i'm looking at tiny-cloud in the first place 2023-05-04 07:25:38 details here: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/issues/37 2023-05-04 14:19:31 ncopa: haven't looked yet -- but could we populate a very basic interfaces file with just loopback at install? so that file, at least, is guaranteed to be there, and assemble any additional ones actually there during firstboot? 2023-05-04 14:19:56 s/firstboot/firstboot tiny-cloud-early/ 2023-05-04 14:28:25 I already have a proof of concept implemented, which I tested works. If the interfaces file exists, it does nothing 2023-05-04 14:28:47 https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/merge_requests/52 2023-05-04 14:45:51 i hope to have some time at lunch to at least glance at these, and be able to review more in depth this evening 2023-05-04 14:48:56 start from the bottom (ignoring the draft) 2023-05-04 14:50:52 use gitlab "rebase" if you merge the first ones. the two last MR are on top of the previous, so they should be easier to review once they are rebased 2023-05-04 15:07:12 i took the liberty to add my today's tiny-cloud work as downstream patches in alpine and included them in alpine 3.18 rc3 2023-05-04 18:09:39 ncopa: what sort of schedule am i on for review / tweaking with regards to 3.18 final? 2023-05-04 19:05:25 I am hoping for do the 3.18.0 on Monday or Thursday. But I’ll take what we get. We can possibly cheat a bit and add stuff to 3.18.1 as long as it is 100% backwards compatible and don’t introduce regressions 2023-05-04 19:06:00 i don’t want push you. I know you are busy 2023-05-06 04:43:55 ncopa: i should have some time this weekend to look at the remaining tiny-cloud PRs in more detail. will keep you posted 2023-05-06 05:41:44 thanks!:q 2023-05-06 05:42:02 s/:q// 2023-05-07 07:54:13 tomalok: i suppose its unrealistic to have the nocloud stuff included in 3.18 release on Monday/Tuesday? 2023-05-07 08:34:29 maybe we should have a call 2023-05-07 08:38:14 i also wonder if it would make more sense to move my stuff out of tiny-cloud to a `alpine-autoinstall` script instead? 2023-05-07 14:39:14 tomalok: I had a call with clandmeter and ikke earlier today, and we came up with an idea that I think is a good solution to move forward in time, without stressing you out 2023-05-07 14:39:46 basically, we can implement the alpine installer as a separate cloud provider 2023-05-07 14:40:11 and we can ship it as tiny-cloud-alpine package (for example) 2023-05-07 14:40:21 or tiny-cloud-autoinstall or whatever 2023-05-07 14:41:39 then we can re-use the need stuff from nocloud provider, and re-use the tiny-cloud framework 2023-05-07 14:42:10 but are free to implement whatever we want without depending on what nocloud does or needs, or what any of the other cloud providers needs 2023-05-07 14:46:07 i was thinkign we could use `#alpine-config` instead of `#cloud-config` for userdata yaml 2023-05-07 14:46:16 then we are free to do whatever we want 2023-05-07 15:17:28 still waking up and brain is a bit fuzzy, but that's an interesting idea, ncopa -- also frees you up to do whatever you like with #alpine-config that doesn't necessarily have to match what cloud-init does. 2023-05-07 15:57:19 ncopa: another thought -- it's possible to write all this as a userdata handler, and not have to duplicate the nocloud stuff. in the `init` at the end there's a TODO to pull in more functions from the detected userdata -- if "#alpine-config" is detected there, source the userdata/alpine-config with the right functions -- even possible to set the 2023-05-07 15:57:19 ACTIONS_EARLY (etc) the way they need to be 2023-05-07 15:57:28 (afk for a bit) 2023-05-08 21:12:49 tomalok: i took the liberty to merge some things I thought was unproblematic directly to main 2023-05-08 21:13:06 there are now only two merge requets open. i closed the old ones 2023-05-08 21:13:18 and i did alpine 3.18 rc5 2023-05-08 23:45:47 ncopa: will take a look, would like to get another rc out of tiny-cloud itself and test an edge image again, but it appears i must work late tonight for DayJob™ -- will try 2023-05-08 23:52:34 ncopa: i think the alpine-config userdata handler could entirely contain what you need for alpine autoinstall -- without defining a separate cloud. for auto-install just stick with CLOUD=nocloud and supply userdata with #alpine-config content type. 2023-05-09 05:56:39 no. I need to run stuff during main phase (and maybe also early) 2023-05-09 05:57:34 the userdata_type thing only works in final, as the driver can not be loaded until after userdata is saved 2023-05-09 05:57:48 before user data is save we dont know the type 2023-05-09 05:58:35 that why I had to do this: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/merge_requests/53/diffs?commit_id=0ca24bbc22324e90c873d1196d274d21a50da42f 2023-05-09 05:59:18 I still have an unresolved issue, even with a separate alpine cloud provider 2023-05-09 10:53:19 tomalok: i rebased it, added another tiny-cloud phase (called `net`, but not sure if thats good name). everything except meta-data (eg network config) is done from alpine-config user-data handler now 2023-05-09 10:54:10 another release candidate of tiny-cloud would be helpful 2023-05-09 14:17:38 ncopa: adding the 'net' phase has helped move this forward. i've merged & tagged 3.0.0_rc2 2023-05-09 14:18:42 awesome! 2023-05-09 14:18:49 just in time! phew 2023-05-09 14:19:08 man... this release has been WILD... 2023-05-09 14:19:09 :D 2023-05-09 17:29:45 haven't updated the APK in aports though 2023-05-09 17:55:21 i did 2023-05-09 21:23:57 alpine 3.18.0 was released 2023-05-10 00:07:05 ncopa, ikke, clandmeter: i plan to build/release cloud images tomorrow morning, there's likely some tweaks that need doing for tiny-cloud (and some testing). any word on an image/metadata upload / CDN download home? 2023-05-10 06:15:30 i haven't heard anything about CDN for alpine cloud. its technically just to upload stuff in a subdir on dl-master. 2023-05-10 06:15:49 tomalok: do you have on open ticket for infra about it? 2023-05-10 06:16:13 I just learned that there is another tiny-cloud project: https://github.com/jirutka/virt-init 2023-05-10 06:16:27 solves same problem as tiny-cloud 2023-05-10 07:38:36 interesting, didnt see this one before 2023-05-10 07:45:03 ncopa: i was thinking to bypass dl-master and push those directly to the t1 infra 2023-05-10 07:45:57 The t1 infra has a lot more storage 2023-05-10 07:52:32 you will need to push it to all of them? 2023-05-10 10:54:57 we are using 2.1T of 2.9T on dl-master 2023-05-10 10:55:02 806G free 2023-05-10 10:55:29 Yes, they have recently increased the amount of storage we have 2023-05-10 10:59:43 i deleted edge and v3.17 releases/*_rc* 2023-05-10 11:00:00 now we have 830G free 2023-05-10 11:02:36 $ du -s edge/releases/ 2023-05-10 11:02:36 57.4G edge/releases/ 2023-05-10 11:02:47 we shoudl have some routine to delete old edge releases 2023-05-10 11:02:55 i dont think we need keep them more than a year 2023-05-10 11:02:58 or even half year 2023-05-10 11:03:44 there was a request to do edge release images 2023-05-10 11:04:10 and add rpi disk images 2023-05-10 12:56:15 ncopa: yes I was aware of virt-init, there are lots of cloud-init type projects out there 2023-05-10 12:56:41 the vast majority of them only support very specific situations 2023-05-10 15:21:42 ncopa, ikke, clandmeter: nope i haven't opened a ticket about it yet. was interested in knowing more about how it works -- sounds like instead of SSHing the images to dev.alpinelinux.org/~tomalok/... they'd instead be SSH'd to a directory on dl-master (or some other host) and then the CDN magic happens? 2023-05-10 15:22:22 tomalok: cdn is backed by t1.alpinelinux.org, which is an anycast address to 3 servers 2023-05-10 15:23:09 currently the t1 servers rsync from dl-master 2023-05-10 15:23:18 while the builders rsync to dl-master 2023-05-10 15:24:08 But we need to think about the structure as we do not want to need to store the images on the builders and also not want the builders to remove them 2023-05-10 15:24:20 right 2023-05-10 15:24:26 (ie, rsync deleting everything not on the builder from that arch) 2023-05-10 15:25:36 simple workaround for that would be to stop delete things not on the builder, for releases 2023-05-10 15:26:09 the place where these get uploaded would also need to be an authoritative data store for image builds (the metadata files) to keep track of how far along the image is -- whether it't officially released or not 2023-05-10 15:26:22 or we can create a new subdirectory. v3.18/cloudimages/ instead of adding them under v3.18/releases/$arch 2023-05-10 15:26:56 one release is 230G 2023-05-10 15:27:11 or we could add them to v3.18/releases/cloud/$arch/ 2023-05-10 15:27:15 so the path i had in mind was https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/ 2023-05-10 15:27:22 do we want to store those images on dl-master as well? 2023-05-10 15:27:47 and filling up dl-master 2023-05-10 15:27:54 so /cloud// 2023-05-10 15:28:34 we're filling up the mirrors anyway, arent we? 2023-05-10 15:28:55 we could as well fill up the master then, i think 2023-05-10 15:29:02 not sure how big problem it is 2023-05-10 15:29:23 maybe we should ask jirutka, who has the contact with the dl-master ppl 2023-05-10 15:29:37 That's a good idea 2023-05-10 15:29:50 fwiw cloudinit images are the largest at about 240 MiB -- and then we have all the different variant combineations 2023-05-10 15:30:16 brb need to rearrange stuff in construction zone 2023-05-10 15:38:40 i think we should set a policy for retention of these images before they start piling up... their metadata has an EOL date 2023-05-10 15:40:05 And also think about the infrastructure that builds them 2023-05-10 15:40:08 edge images currently have an EOL date of "tomorrow", so maybe we should have a little leeway there? 2023-05-10 15:41:08 we get a datastore for metadata that's not in the target cloud itself, and i think the build stuff could be run in gitlab-ci 2023-05-10 15:42:45 ikke: will you ask jirutka 2023-05-10 15:42:50 building on ci sounds like a good idea 2023-05-10 15:43:48 the build script is flexible enough to do "the whole shebang" or a specific slice of variants, all the way through or a step at a time. 2023-05-10 15:46:46 though importing/publishing _inside_ the cloud (like is done for AWS currently) would need to be done differently -- mcrute had some ideas about a self-service tool to import/publish to a user's cloud, but unless the cloud providers have some kind of assistance for wide distribution, the days of "click-to-launch" might be numbered. 2023-05-10 15:47:26 s/user's cloud/user's cloud account/ 2023-05-10 15:47:45 and might give rise to the $$$$ alpine images againa 2023-05-10 15:49:31 tomalok: most cloud providers provide per-user import functionality, usually from a S3 (or equivalent) bucket 2023-05-10 16:03:30 minimal: that's essentially how we do it... S3 -> snapshot -> AMI 2023-05-10 16:05:06 i will see if i can find a contact at OCI and see what might be involved with providing official Alpine images there (it's the DayJob™) I'm about a decade out of AWS, so most of my contacts there arem 2023-05-10 16:05:14 *aren't there anymore 2023-05-10 16:06:53 but with all the clouds' region expansion -- and "realms" that normal people can't touch anyways (i.e. gov) some kind of "here's our official image, make it available" would be nice 2023-05-10 17:05:51 looks like there's a little bit to do yet with regards to tiny-cloud on clouds -- tiny-cloud-net wants to start before networking, and the main phase won't start because net didn't run. 2023-05-10 17:19:45 Installing tiny-cloud (3.0.0_rc2-r1) - the latest according to APKBUILD is r3 2023-05-10 17:24:34 hm, ok. tiny-cloud-net has "need net" and "before *" -- does "before" beat "need"? -- it's trying to start before it has net. 2023-05-10 17:24:58 need is a dependency (pulls in something else) 2023-05-10 17:25:02 while before is ordering 2023-05-10 17:26:03 tomalok: but 'need net' is tricky 2023-05-10 17:26:11 "need net" and "before tiny-cloud" (or tiny-cloud says "after tiny-cloud-net") maybe 2023-05-10 17:26:34 tomalok: https://github.com/OpenRC/openrc/blob/master/service-script-guide.md#be-wary-of-need-net-dependencies 2023-05-10 17:28:25 either way the main phase should have at least tried even if tiny-cloud-net failed 2023-05-10 17:32:21 back in boot runlevel set_default_interfaces runs, which, while it autodetects interfaces... 2023-05-10 17:32:33  $MOCK ip link set dev $i down >/dev/null 2023-05-10 17:33:18 wrong line -- the line above is where it sets it back down again -- except for the one it found 2023-05-10 17:33:39 that could be enough to satisfy "need net"? 2023-05-10 17:35:28 1) see what devices you can bring up, and then 2) bring all of them down again again, except for the first one found. 2023-05-10 17:35:53 Probably yes 2023-05-10 17:37:03 hm, should probably check with ncopa if setting _all_ interfaces back down again after autodetection is ok 2023-05-10 17:38:49 Hi 2023-05-10 17:39:16 I’m afk. What’s up? 2023-05-10 17:40:37 iirc aws needs net to fetch meta-data (early) 2023-05-10 17:40:58 i think that the autodetect of interfaces is causing "use net" to be prematurely true -- we bring up link on the eths and take the first one that's up -- but then it's not put back down again 2023-05-10 17:41:24 no, openrc doesn’t know that 2023-05-10 17:41:48 "early" does no net, "net" is the first phase that needs net -- but in early set_default_interfaces leaves one non-loopback interface up 2023-05-10 17:42:27 so tiny-cloud-net's "need net; before *" happens before networking comes up 2023-05-10 17:43:27 its just a link that is up. It has no ip addr 2023-05-10 17:43:44 i don’t think that is causing it 2023-05-10 17:44:30 ncopa: does openrc make that distinction? 2023-05-10 17:44:31 per the link ikke shared ""need net" requires some other interface to be up" - not necessarily with an IP 2023-05-10 17:45:21 openrc does not check any link status or network interface at all 2023-05-10 17:45:49 ncopa: why does that document imply it does then? 2023-05-10 17:46:30 it only checks if there any service that provides “net” 2023-05-10 17:47:08 it's probably written in mind with gentoo netifrc which has the net.xyz service block until it's actually up iirc 2023-05-10 17:47:10 well 2023-05-10 17:47:21 i'm pretty sure networking does block until that anyway 2023-05-10 17:47:39 ight 2023-05-10 17:47:43 right 2023-05-10 17:47:56 need x before * sounds like a really bad hack tho 2023-05-10 17:48:48 Yeah, we can likely drop the before * 2023-05-10 17:49:25 make it before tiny-cloud, and/or have tiny-cloud say after tiny-cloud-net 2023-05-10 17:51:24 will give that a go and see how "need net" fares 2023-05-10 17:59:12 oh, and only the yash test fails now -- i think i saw something about that psykose? 2023-05-10 18:00:04 the new yash version forbids local in posixlycorrect 2023-05-10 18:02:18 i wonder if we just s/local //g if there'd be any unexpected consequences 2023-05-10 18:02:55 if there's no duplicate names there wouldn't be 2023-05-10 18:03:04 you could also just disable the yash test for now and work on it later 2023-05-10 18:11:22 is there a right way to disable a test? (commenting it all out seems a bit heavyhanded) 2023-05-10 18:11:41 commenting out 4 lines seems pretty ez 2023-05-10 18:12:36 or adding "when: manual" or never? 2023-05-10 18:12:48 just put a dot 2023-05-10 18:13:13 .test-yash: 2023-05-10 18:13:15 it'll get skipped 2023-05-10 18:13:27 👍 2023-05-10 18:13:28 but that is much less obvious than some comments you could have just pushed by now and called a day :p 2023-05-10 18:21:40 I think we should accept local 2023-05-10 18:23:23 I suspect that the issue is that early stage does imds but fails due to no network and exits before creating interfaces 2023-05-10 18:25:44 I can try help debug when I’m back home 2023-05-10 18:27:25 i don't believe there's an 'imds' being done in early phase (but for nocloud, that first call brings in files from the CIDATA volume) 2023-05-10 18:28:36 currently scratching my head as to why rootbld is throwing fails for create_default_user and no_userdata_main on my dev box 2023-05-10 18:30:06 Hm ok 2023-05-10 18:30:46 for alpine cloud you have an imds call in early so you have all that stuff available early 2023-05-10 18:31:14 yes 2023-05-10 18:31:59 is the interfaces file created at all? 2023-05-10 18:32:14 in rootbld somewhere? 2023-05-10 18:32:23 in aws 2023-05-10 18:32:45 no way to tell. the main phase fails because net phase fails (need to soften that dep) so no SSH keys 2023-05-10 18:34:15 and no way to get serial console output? 2023-05-10 18:34:37 i think i have a fix in https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/releases/3.0.0_rc3 but abuild rootbld on my local box is failing two tests (complaining about stderr) when i'm trying to make the APK 2023-05-10 18:35:17 you can watch serial console (if you're quick enough with the clicks) or you can wait for system console log to be populated 2023-05-10 18:35:42 But that’s my main suspicion, that interfaces file fails to get created for some reason 2023-05-10 18:36:28 i'll give it another go later -- the instance i spun up, i already terminated so all that output's gone 2023-05-10 18:36:44 but now, lunch 2023-05-10 18:37:23 hm, i have a 3.17.3 instance running maybe i can upgrade and see what wants to do 2023-05-10 18:38:08 I’m pretty sure that last commit will break alpine boot, because there is nothing pulling in -net now 2023-05-10 18:38:34 (should we rename it to init?) 2023-05-10 18:39:01 let's think about names after it works again :) 2023-05-10 18:39:54 so what i pushed in rc3 does still have "need net" but it has "before tiny-cloud" 2023-05-10 18:40:04 (that's tiny-cloud-net) 2023-05-10 18:41:20 tiny-cloud main just has "need net" and "before sshd" and relies on -net's "before tiny-cloud" 2023-05-10 18:41:27 but, lunch. 2023-05-10 18:41:41 Bon appetite 2023-05-10 18:41:51 oh and appt. may be a while longer 2023-05-10 18:44:24 there is nothing pulling in tiny-cloud-net now 2023-05-10 19:09:12 maybe i'm misunerstanding -- openrc dependencies? 2023-05-10 19:09:54 you went from 'need tiny-cloud-net' to not having it 2023-05-10 19:10:06 or tiny-cloud --setup doesn't put it in place like it does the others? 2023-05-10 19:10:07 so unless something adds it somewhere nothing depends on it 2023-05-10 19:11:13 yeah it's missing in tiny-cloud --setup 2023-05-10 19:17:57 my initramfs init does not add tiny-cloud-net. it got pulled in by tiny-cloud's `need tiny-cloud-net` 2023-05-10 19:20:03 we've always been explicit about adding each phase. the "need tiny-cloud-net" meant that if that failed, then main fails too... would rather each phase try to do what it can independent of the success/phase of what came before 2023-05-10 19:20:47 ok 2023-05-10 19:20:55 (so maybe it fails to pull in userdata in net, but maybe it can at least install ssh keys in main) 2023-05-10 19:21:34 im not able to reproduce the test fail in rootbld here 2023-05-10 19:21:37 weird 2023-05-10 19:21:51 never got the updated APK out, so i'm going to un-release r3 and update with net in --setup 2023-05-10 19:22:12 maybe my build env is too old or some other weirdness going on there. 2023-05-10 19:22:50 its containerized, so its weird. really weird 2023-05-10 19:30:20 i ran `./build --clean local --only 3.18 aws uefi tiny x86_64` and booted it in local qemu 2023-05-10 19:30:38 the output: https://tpaste.us/kXE4 2023-05-10 19:31:15 ++ set_default_interfaces already set up 2023-05-10 19:31:50 assemble-interfaces happens because aws has the vnic-eth-hotplug module 2023-05-10 19:32:44 network comes up 2023-05-10 19:32:50 that's not the order i saw at all. before i dig into this weird rootbld thing any more going to spin up that edge image 2023-05-10 19:33:20 did you upload the image to dev.a.o? 2023-05-10 19:33:47 yeah it should be in there 2023-05-10 19:34:01 aarch64 2023-05-10 19:35:30 just caught the tail end on the console... 2023-05-10 19:35:45  * ERROR: cannot start tiny-cloud as tiny-cloud-net would not start 2023-05-10 19:35:57 it'll be a while before i can see the rest of it... 2023-05-10 19:36:05 ...but will leave this instance up 2023-05-10 19:37:44 https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/edge/cloud/aws/aarch64/alpine-20230510-aarch64-uefi-tiny-r0.vhd is the image 2023-05-10 19:38:59 the only thing tiny-cloud-net does is save userdata. which is an imds -e @userdata > "$tmpfile" 2023-05-10 19:39:20 we use set -e, right? 2023-05-10 19:40:17 yeah "set -e" is in tiny-cloud 2023-05-10 19:40:59 but... it should have network running at that point 2023-05-10 19:41:14 and i dont think imds fails with error 2023-05-10 19:41:28 in the console logs the first time, i saw dhcp get the Ip address AFTER tiny-cloud-net tried to run 2023-05-10 19:42:05 with dhcp? 2023-05-10 19:42:15 udhcpc or dhcpcd? 2023-05-10 19:42:19 starting to see some system logs... 2023-05-10 19:42:43 also, do you have more than a single network interface? 2023-05-10 19:43:06 just one network interface 2023-05-10 19:43:11 * ERROR: cannot start tiny-cloud-net as networking would not start 2023-05-10 19:43:11 * Starting networking ... * lo ... [ ok ] 2023-05-10 19:43:11 * eth0 ...dhcpcd-10.0.1 starting 2023-05-10 19:43:57 hum.. i didnt do git pull here locally 2023-05-10 19:43:57 early ran just fine 2023-05-10 19:45:20 so the aws image is 3.0.0_rc2 haven't gotten an rc3 built locally yet -- i may just push it for an MR and see if it works on gitlab 2023-05-10 19:45:42 yeah 2023-05-10 19:47:23 tiny-cloud-3.0.0_rc2-r1 2023-05-10 19:49:16 https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/47047 2023-05-10 19:49:25 need to run be back in a bit 2023-05-10 19:57:55 hmpf... serial console is disabled on the image 2023-05-10 20:02:29 and no grub menu to edit boot cmd line 2023-05-10 20:02:42 ok, i dont know how to debug this 2023-05-10 20:02:56 on aarch64 2023-05-10 20:09:39 i give up for today. will continue tomorrow 2023-05-10 20:09:50 i wonder how I can reproduce it on x86_64 2023-05-11 02:41:08 fwiw tiny-cloud-3.0.0_rc4 aports is merged for edge, but not seeing any APKs yet, 30m later 2023-05-11 06:53:15 i think the arm builder was (is?) down 2023-05-11 06:55:08 Still is 2023-05-11 06:55:48 The network is unreachable 2023-05-11 23:11:23 hello. when can we expect Amazon EC2 3.18 release images to be published? 2023-05-11 23:11:55 Also, looks like https://gitlab.alpinelinux.org/alpine/cloud/alpine-cloud-images/-/commit/2ca06efb11ddb256f2eba5369ca4bd888260db97#b60fc2a049041eb7aed51c5f81deda81b1941451_58_58 should have referenced 3.17 file 2023-05-11 23:12:05 err 3.18.conf file of course 2023-05-11 23:14:18 thresh: tomalok will likely be along shortly to update on their status 2023-05-11 23:14:47 thresh: are you using the tiny-cloud or cloud-init images? 2023-05-11 23:27:03 I'm using tiny-cloud I guess, the filter is "alpine-3.17.*-x86_64-uefi-tiny-r*" 2023-05-11 23:28:27 yeah that's tiny-cloud 2023-05-12 01:26:30 thresh: we're working through some issues that were discovered with the new tiny-cloud-3.0.0_rc, and possibly an issue with the ovmf apk for x86_64 uefi for our build process... also aarch64 gitlab runners are "awol" which is also slowing down debugging/testing/fixing. -- i hope that issues will be resolved by weekend's end and the fresh batch of 2023-05-12 01:26:30 images will be published after that. 2023-05-12 01:38:39 thanks! 2023-05-12 04:03:20 tomalok: we've heard back from ungleich, they have an issue with a router that affects the arm servers 2023-05-12 04:04:03 No eta for fixing it, though 2023-05-12 14:41:48 in the meantime, trying to figure out why x86_64 uefi images aren't building -- ncopa: anything significant change with the ovmfpackage between 3.17 and 3.18? 2023-05-12 15:24:02 not that i know 2023-05-12 15:24:40 i think i enabled secure boot at some point if it wasn't, but i dont think it should make any difference 2023-05-12 15:25:00 the edk2 secure bundled with qemu is broken. that i know for sure 2023-05-12 15:26:07 i experimented today with 2 qemu on macos. the idea was to set up a fake aws metadata server. but user network is a bit complicated 2023-05-12 15:27:02 anything I can do to help troubleshoot x86_64? 2023-05-12 17:28:37 i'm hoping to attach a VNC connection to see what's going on during boot -- packer sends keypresses via console and _should_ get sshd set up for the rest of the build stuff -- works fine with x86_64 bios and aarch64 uefi -- but not x86_64 uefi -- and until i see what's happening ovmf is the only logical suspect 2023-05-12 17:29:25 it might be i need to provide some additional qemu parameters that i didn't need before 2023-05-12 18:32:55 tomalok: aarch64 is back fyi 2023-05-12 18:33:02 (and arm in general) 2023-05-12 18:43:21 ikke: thx! 2023-05-13 17:43:54 i need to double check i'm building the right image variant -- but after doing a brew upgrade, it seems like x86_64 uefi image is building again... 2023-05-13 17:45:18 sounds like good news 2023-05-13 17:47:36 maybe qemu 8 is more forgiving with regards the latest ovmf 2023-05-13 17:58:57 3.0.0_rc4 init ordering is fixed -- network comes up before tiny-cloud-net -- theory is that "before *" is mightier than "need net" 2023-05-13 17:59:30 should probably also check aarch64 just in case 2023-05-13 18:29:38 hm, final phase didn't run_userdata, that needs fixing 2023-05-13 18:33:06 ikke: where do you prefer i open an issue to sort out the upload / CDN download for cloud images and metadata? 2023-05-13 18:34:28 alpine/infra/infra suffices 2023-05-13 18:39:44 👍 2023-05-13 19:02:16 ikke: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10798 2023-05-13 19:02:24 yes, got the notification 2023-05-13 19:02:31 heh 2023-05-13 19:03:13 figured out why run_userdata didn't work -- it's not packaged in the APK.. :( 2023-05-13 19:17:19 alpine installer's alpine-config userdata isn't packaged either 2023-05-14 04:15:37 tiny-cloud-3.0.0_rc5 is out -- the APKs should include appropriate /lib/tiny-cloud/user-data/... in them now, and tiny-cloud-alpine APK is dependent on tiny-cloud-nocloud 2023-05-14 04:16:49 some more stuff on the 3.0.0 checklist to work on tomorrow -- https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/issues/40 -- ncopa if you're not spending the weekend in the forest, what's your opinion on yast compatibility and tabs-vs-spaces? 2023-05-16 10:43:40 tomalok: i prefer `local` over yash. I prefer tabs over spaces, due to embedded yaml as here-docs. 2023-05-16 10:44:40 i commented the details in the issue 2023-05-16 10:44:56 i also think we can merge the write_files as is 2023-05-16 10:45:24 I'd like to add the missing alpine stuff too, and its nice to avoid the merge conflict 2023-05-16 10:47:37 also tabs vs spaces needs to be fixed other places anyway so we do all that in one go anyway. so I think we should merge write_files first 2023-05-16 10:51:47 i think we can move most of the alpine-config to cloud-config and let alpine-config source cloud-config and extend it. But I only want to do that after write_files is merged too 2023-05-16 14:03:53 ncopa: thanks for weighing in -- i've taken a quick peek at the write_files MR -- would like t have a test to see what it does when there's no `write_files` content in user-data 2023-05-16 15:12:22 i think we have a test for that tests/tiny-cloud.test::no_userdata_main 2023-05-16 15:12:40 but i think the test is broken currently 2023-05-18 13:37:19 tomalok: thank you for merging! 2023-05-18 14:41:05 ncoopa: thank you for the next MR -- that's almost ready i think 2023-05-18 14:41:15 s/ncoopa/ncopa/ 2023-05-22 08:47:47 tomalok: i was tinking how we organize the tests. the tests/init-* are sort of leftovers from when we had /lib/tiny-cloud/init-{early,main,final}. I think it might make sense to have a single tests/init.test for testing everythign in lib/tiny-cloud/init 2023-05-22 08:48:17 i.e merge tests/init-*.test into a single tests/init.test or maybe etc/init-functions.test 2023-05-22 08:49:47 i also have a challenge with the move of tiny-cloud-early from boot -> default runlevel. But I think I will create a ticket for that with the details and possible solutions 2023-05-22 10:05:46 a couple of more thoughts. I think we should create the default user in main phase, so it can be overridden with `user:` keyword in user-data. https://cloudinit.readthedocs.io/en/latest/reference/modules.html#users-and-groups 2023-05-22 12:49:47 one thing to point out about the "default" user in cloud-init, if you specify a "users:" section in your user-data then if you do NOT specify "- default" as the 1st entry then the default user is NOT created 2023-05-22 12:50:09 That is stated in the docs you linked to but some people miss that. 2023-05-22 14:08:35 yes, i noticed that 2023-05-22 15:20:20 ncopa: thanks for the test reorg and sorry the phase renaming impacted the initramfs - but i think it's clearer now and shouldn't change again. 2023-05-22 15:20:42 yes, yes, I agree 2023-05-22 15:20:51 and I knew it was a risk 2023-05-22 15:21:10 and I'm ok to break it (once or twice) 2023-05-22 15:21:24 we just should try protect us from doing too often 2023-05-22 15:21:48 (nod) 2023-05-22 15:22:38 im currently talking with debian-cloud people. apparently it is amazon who pays for the debian AMIs 2023-05-22 15:24:22 ikke: we did have some arrangement with azure, right? they are willing to sponsor alpine images? 2023-05-22 15:24:43 ncopa: We still need to setup a meeting with them 2023-05-22 15:24:58 ok 2023-05-22 15:25:28 im thinking that it would be good to have some sponsorship from them before we start provide official images 2023-05-22 16:50:37 good morning, may I ask how far are EC2 images are from being released? 2023-05-22 16:52:07 (for 3.18 that is) 2023-05-22 18:19:34 thresh: better ask in #alpine-cloud. I would guess a week or two 2023-05-22 18:20:16 ncopa: we are in #alpine-cloud ;) 2023-05-22 18:20:17 would actually want to solve the sponsorship of the ami storage first 2023-05-22 18:20:41 lol, its time for calling it a day 2023-05-22 18:20:58 i though i was in #debian-cloud 2023-05-22 18:21:04 it's time for lunch and topping off my energy drink levels 2023-05-22 18:21:43 i think tiny-cloud 3.0.0 is getting much closer, and when that happens i think we're good for a new set of cloud images... 2023-05-22 18:22:27 ncopa ikke clandmeter - i would like to sort out a permanent store for golden images & metadata though we can roll forward to the next point releases if we need to 2023-05-22 18:23:07 https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10798 2023-05-23 08:01:48 i wonder if we should ask for an amazon contact on twitter 2023-05-23 08:02:08 amazon is sponsoring debian AMIs. maybe they can help us as well 2023-05-23 13:54:37 ncopa: i've never tweet'd so i dunno the prospects of that -- but irregardless of that, alpine/infra/infra#10798 will need sorting ;) 2023-05-24 15:03:41 tomalok: fyi, im pushing a change in the dependencies for tiny-cloud-alpine 2023-05-24 15:24:24 er, was in the middle of trying to build rc6 from the previous branch i had -- what was the dep? 2023-05-24 15:25:13 (reads logs) ifupdown-ng? 2023-05-24 15:25:17 yup 2023-05-24 15:25:42 will double check to make sure that survived 2023-05-24 15:25:52 i booted it here and it works (after fixing initramfs init) 2023-05-24 15:26:03 there are some error messages though, that could be hidden 2023-05-24 15:26:23 oh look merge conflicts... ;P 2023-05-24 15:26:37 well, that must be why it's not bulding ... will fix.. 2023-05-24 15:28:09 ERROR: unable to find map key 'network-interfaces' 2023-05-24 15:28:21 you tagged it as _rc6 i guess? 2023-05-24 15:28:30 yeah, retagged 2023-05-24 15:28:54 it never got fully released 2023-05-24 15:28:55 abump tiny-cloud-3.0.0_rc6 should do it 2023-05-24 15:32:04 that went by pretty quick (and no scrollback on the iPad) -- what'd it do? 2023-05-24 15:32:47 set pkgver=3.0.0_rc6 and pkgrel=0, build it and git commit 2023-05-24 15:32:52 git show should show the exact thing 2023-05-24 15:34:41 create default user test still not happy on this dev box (already has the default user), but otherwise looks like it's ok 2023-05-24 15:35:47 hum 2023-05-24 15:40:01 let's see if i can capture its complaint 2023-05-24 15:40:29 it shouldn't complain, due to fake getent 2023-05-24 15:40:47 oh, maybe we need the fake getent in tests/bin/ 2023-05-24 15:40:53 might be other test that needs it 2023-05-24 15:41:03 Fail: regexp adduser.*alpine not in stdout 2023-05-24 15:41:21 which test is it? 2023-05-24 15:41:33 userdata_create_default_user 2023-05-24 15:42:25 hum, im pretty sure i fixed that 2023-05-24 15:43:48 + fake_bin getent <<-EOF 2023-05-24 15:43:48 + #!bin/sh 2023-05-24 15:43:48 + EOF 2023-05-24 15:43:54 should do it 2023-05-24 15:44:32 40d15db038958fd821000399420b212c2174198c 2023-05-24 15:45:03 thats how i fixed it, but it was needed other place as well apparently 2023-05-24 15:45:52 it also needs exit 2 2023-05-24 15:46:33 looks like we found something for "* [ ] ?" ;) 2023-05-24 15:47:27 i'll build some edge cloud images a little later and test, but for now it's time for DayJob™ 2023-05-24 15:51:08 https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/merge_requests/88 2023-05-24 15:51:09 ok 2023-05-30 04:30:59 ncopa: about this --><-- close to releasing tiny-cloud 3.0.0 (waiting for rc7-r1 to land for at least aarch64 or x86_64 for one more round of testing) -- figured i should check in with you in advavnce if there's anything you'd need to worry about backporting it to 3.18-stable? 2023-05-31 00:16:04 https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/releases/3.0.0 - working on updating aports (master/edge) - holding off on backport to 3.18-stable 'til ncopa can weigh in on how that might affect the alpine auto-installer. 2023-05-31 04:16:39 Just fyi, ARM is unreachable again 2023-05-31 04:17:31 i think we even ran out of jokes by now 2023-05-31 04:17:52 this hosting is worse than my basement at this point 2023-05-31 04:18:56 networking is hard though so it's expected :-)