2023-03-06 19:30:22 FYIhttps://aws.amazon.com/about-aws/whats-new/2023/03/amazon-ec2-create-amis-boot-uefi-legacy-bios/ 2023-03-06 20:53:21 thanks minimal -- will take a look at that later -- vending a hybrid uefi/bios image has been on the backlog for a while now, hopefully there will be some good info here on how exactly to build such an image 2023-03-06 20:54:34 at the very least this simplifies things when we import/register the AMI 2023-03-06 21:33:40 tomalok: Grub supports "hybrid" mode for UEFI and BIOS booting, I've used it in the past for USB sticks 2023-03-06 21:35:28 Limini also supprts that but I've not yet tried it with Limine 2023-03-08 08:18:19 hi 2023-03-08 08:18:32 so i have been working on nocloud provider 2023-03-08 08:18:52 i think we need to implement "local-hostname", which is set before network 2023-03-08 08:19:10 so we can use the hostname when requesting ip address from dhcp 2023-03-08 08:19:26 which means that we need to run it in the tiny-cloud-early 2023-03-08 08:20:00 however, -eraly runs before mdev, which means it runs before we have remounted / as rw 2023-03-08 08:20:16 s/-eraly/-early/ 2023-03-08 08:21:16 i looked at how cloud-init does it, and it has the stages 'local', 'config' and 'final' 2023-03-08 08:21:40 'local' corresponds to tiny-cloud-early, but it is run as soon / is remounted rw 2023-03-08 08:22:15 and before network is started 2023-03-08 08:22:55 any ideas how we solve this? 2023-03-08 08:23:24 im thinking that we could move the mdev stuff 2023-03-08 08:24:12 we could move it to mdev-conf, which means we enable it by default for everyone 2023-03-08 09:57:46 what are the EBS /dev/xvd* symlinks needed for? 2023-03-08 11:13:50 I have added support for EBS aliases in mdev-conf: https://gitlab.alpinelinux.org/alpine/mdev-conf/-/merge_requests/4 2023-03-08 16:41:14 ncopa: re "local-hostname", I guess it depends how exactly/how close to cloud-init behaviour you want to be, for cloud-init NoCloud you can either specify "hostname: value" in meta-data file or else specify "hostname: value" in user-data 2023-03-08 16:42:12 likewise network config can either be included in meta-data, via "network-interfaces:" section or in network-config file 2023-03-08 16:55:28 well, what we need to do is set hostname before network is up 2023-03-08 16:55:46 so we can pass the hostname to the dhcp client 2023-03-08 16:57:15 i also notice that the expand_root does a remout -o rw 2023-03-08 16:58:01 i think tiny-cloud-early should happen after the /etc/init.d/root has executed and remounted root to rw 2023-03-08 17:00:32 ncopa: right, well NoCloud is before network is up, unless you're using NoClout "seedfrom" to pull the config via URLs 2023-03-08 17:00:54 speaking from a cloud-init perspective 2023-03-08 17:03:35 https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html#permitted-keys only mention "local-hostname" 2023-03-08 17:06:16 tomalok: the MRs that are not "Draft:" can be merged: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/merge_requests 2023-03-08 17:10:42 ncopa: oh, you're using "ds=" on cmdline, I was thinking mainly in terms of CIDATA ISO/FAT fs 2023-03-08 17:11:12 so are you also using "seedfrom" via "ds=" to provide the meta-data and user-data? 2023-03-08 17:30:48 i think it accepts both LABEL=CIDATA and ds= on command line 2023-03-08 18:00:49 ncopa: the "cidata"/"CIDATA" labels are automatically search for, I don't think they can be specified on cmdline 2023-03-08 18:02:05 ncopa: for what you are working on are you obtaining meta-data/user-data/network-config from a filesystem (ISO/FAT) or from a seed url? 2023-03-08 18:03:25 currently im only adding tests for tiny-cloud, making sure things works as expected when I start refactor things 2023-03-08 18:04:02 the plan is to support filesystem cidata initally and later on seed url 2023-03-08 18:04:25 naturally, a seed url needs to be fetched after network is configured 2023-03-08 18:05:45 i think we should let tiny-cloud-early execute after root is remounted rw 2023-03-08 18:05:55 ncopa: for cloud-init with seed-url a ephimeral network is brought up to contact the seed-url, the "normal" network should then be brought up afterwards 2023-03-08 18:06:17 right 2023-03-08 18:06:18 as network config can be obtained via the seed url(s) 2023-03-08 18:06:35 i guess tiny-cloud could support that as well 2023-03-08 18:06:49 the same way that things work for e.g. AWS to get to metadata server 2023-03-08 18:06:59 yeah, i figured 2023-03-08 18:08:03 i wonder what is setting up the ephimeral network for alpine aws images currently? 2023-03-08 18:08:10 though I seem to remember when I tested seedurls on Alpine with cloud-init I hit a problem with ifupdown-ng's ifup that the attempt to bring up the "proper" network didn't actually happen as ifupdown-ng (ifstate?) seemed to remember the previous settings rather than the modified (by cloud-init) /etc/network/interfaces file 2023-03-08 18:09:52 does not surprise me 2023-03-08 18:10:39 it was approx 6 months ago I last tested this stuff with cloud-init 2023-03-08 18:11:13 unfortunately ifupdown-ng development in general seems to have "paused" 2023-03-08 18:13:43 ephemeral network could be set up by something else than ifupdown? 2023-03-08 18:14:27 from memory cloud-init uses dhclient (certainly for AWS, etc) 2023-03-08 18:15:14 that's the only DHCP client that cloud-init officially uses (though udhcpc and dhcpcd work in scenarios where ephemerial DHCP is not needed) 2023-03-08 18:16:39 169.254.169 is link local address, i guess something configures link-local ipv4 before network is "started" 2023-03-08 18:17:10 are you talking about tiny-cloud or cloud-init? 2023-03-08 18:18:22 cloud-init 2023-03-08 18:18:38 just looking at the DS code currently 2023-03-08 18:19:11 will continue tomorrow. its late. have a nice evening! 2023-03-08 18:19:55 ncopa: yeah, I'll start up a VM which uses seedfrom to refresh my memory 2023-03-09 02:40:39 ncopa: non-draft MRs are merged 2023-03-09 07:55:56 awesome! thanks! 2023-03-09 16:37:55 10:10 AM it was approx 6 months ago I last tested this stuff with cloud-init 2023-03-09 16:37:55 10:11 AM unfortunately ifupdown-ng development in general seems to have "paused" 2023-03-09 16:38:11 not paused intentionally. present maintainers are just busy with other projects 2023-03-09 16:38:16 we can add more maintainers :) 2023-03-09 16:59:54 Ariadne: understood. It has been almost 1 year since the last release. I'm not a C coder, otherwise I would volunteer 2023-03-10 14:13:25 ncopa: FYI I've noticed when using "ds=" on cmdline that, certainly with Grub, I have to enclose the ds value in quotes as otherwise the cmdline passed by Grub to the kernel is truncated at the 1st semicolon 2023-03-24 16:11:52 tomalok: I have added EBS link support to mdev-conf, so we should be able to let tiny-cloud early run after coldplug (kernel modules loaded), after rootfs is remouted rw, but before network 2023-03-24 16:12:30 as I understand it had to run before coldplug so the mdev handler was installed before kernel modules were loaded 2023-03-24 16:25:06 ncopa that was a while ago when that was decided but without looking too close it sounds right. i think tiny-cloud sets up aliases for xvdX* as well as sdX*, does that also hold true for what's in mdev-conf? 2023-03-24 16:26:10 i think it only sets what nvme-cli shows, so either xvdX* or sdX* but not both 2023-03-24 16:26:14 also was the mdev-conf update in place for 3.17? 2023-03-24 16:26:22 no, not 3.17 2023-03-24 16:26:25 this is for 3.18 2023-03-24 16:26:47 what are those symlinks needed for? 2023-03-24 16:28:46 they may have both been put into place because someone else did it that way and/or non-NVMe disks may have also had both... or EC2 is/was ambiguous... like i said, been a while. 2023-03-24 16:30:37 logic is here: https://gitlab.alpinelinux.org/alpine/mdev-conf/-/commit/87c664016d5e4ae8131857e3fcab425f6f97bb0b 2023-03-24 16:31:06 at the moment i'm trying to figure out some EC2 ENI eth1+ issues which (i think) are affecting 3.17 -- so after that's solved and backported, we could in theory drop the mdev thing for 3.18+ 2023-03-24 16:31:29 do you mind if we only create a single symlink and not both? and if it turns out we need both we add that later? 2023-03-24 16:31:49 iirc the AWS docs only mentions how to create a single symlink 2023-03-24 16:31:56 i don't mind, but we'd need to probably have a release note because behavior change 2023-03-24 16:32:01 yeah 2023-03-24 16:32:12 and maybe bump major version number 2023-03-24 16:32:27 at least minor version 2023-03-24 16:33:14 it is a potentially breaking change. we also move the time where init-early is executed, so i think it justifies a major version number 2023-03-24 16:33:20 i mean, its just a number 2023-03-24 16:33:56 what is the issue with EC2 ENI eth1 issue? 2023-03-24 16:34:07 heh 2023-03-24 16:34:55 when i'm attaching an ENI as eth1, it's not routing properly 2023-03-24 16:34:55 i didnt quite understand how that was supposed to work. it look like it tries to configure eth0 (eth*) when it shows up in kernel uevent (cold or hotplug) 2023-03-24 16:35:23 but the code looked broken 2023-03-24 16:36:31 im currently working on tests for init-main, and i just realized it will be complicated with a generic set_fake_meta 2023-03-24 16:36:51 mdev triggers a (potential) rebuild of assemble interfaces and (currently) sets up another udhcpc on eth1, and then goes to look for more IP/IPv6 in imds metadata 2023-03-24 16:37:18 does the mdev trigger actually work? 2023-03-24 16:37:25 is the trigger executed? 2023-03-24 16:37:27 yeah, that works fine 2023-03-24 16:37:58 the problem is once it's done things aren't routing properly and there's two default routes. 2023-03-24 16:38:28 so i'm exploring _not_ running DHCP and just getting imds metadata and setting it up that way 2023-03-24 16:38:40 (no dhcp on eth1) 2023-03-24 16:40:27 two different default routes? 2023-03-24 16:43:47 https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System using this as a reference to help detangle -- previously i was using the table/rule stuff for associating secondary IPs on the same interface (though I didn't have both "ip rule add from/to" 2023-03-24 16:45:57 I thought the xvdX names was because AWS used those originally for disks (back when they used Xen) rather than sdX or vdX 2023-03-24 16:47:31 i'm currently working on setting up static /etc/network/interfaces.d/eth1 to see if going that route solves the problem -- this may end up reworking (maybe simplifying) the hotplug for ENIs 2023-03-24 16:48:25 the things i do on vacation... :P 2023-03-24 16:49:37 heh 2023-03-24 16:50:13 sounds like pretty advanced networking setup 2023-03-24 16:50:32 how does that actually work from aws? Or, how is it supposed to work? 2023-03-24 16:50:42 tomalok: I thought AWS used their own ethernet "tool/package" to set up all networking from the metadata server info? or perhaps all-but-eth0's-main-IP 2023-03-24 16:51:06 you get a second interface, eth1, with a separate subnet and separate default gw? 2023-03-24 16:51:11 really i just want to float static IPv4 & IPv6 between nodes... 2023-03-24 16:51:44 "float static ip...?" 2023-03-24 16:52:31 tomalok: can you do that on AWS? don't you have to rely on the IPs they give the machine on each boot (and then use an elastic IP for "floating") 2023-03-24 16:53:19 multiple default gw may lead to assymetric routing which is why you need a separate routing table which is selected based on source ip 2023-03-24 16:53:26 so i have a docker swarm, and when it's time to refresh each node, in theory i can just launch a new one with all the shiny, and it joins the swarm on the private network. when it's up, i simply detach the ENI from an old node, and attach it to the new node. the attaching generates an event that mdev picks up on, and brings up eth1 (per 2023-03-24 16:53:26 ifupdown-ng's configs) 2023-03-24 16:53:57 ncopa: yeah which is why i'm seeing connections only getting half-made 2023-03-24 16:54:40 i create a tool called pingu a dcade ago, that did multi-isp failover (and load-balancing) 2023-03-24 16:54:49 i'm sure that dockerd isn't helping the situation, but for now i'm just tying to establish a connection to SSH port, which should be listening on all interfaces 2023-03-24 16:55:21 so basically, you want to "move" an ip from one host to another? 2023-03-24 16:56:00 yeah, but you have to move detach/attach an AWS reesource so it's not exactly like it can all be done on the hosts 2023-03-24 16:56:40 tomalok: does the interface keep the mac address? 2023-03-24 16:56:57 at best the hosts detect that the ENI is gone (and ifdown eth1) and the ENI is there (and ifup eth1) -- there are two other static IPs so, the cluster is fine in the interim 2023-03-24 16:57:10 ikke: good question, haven't checked 2023-03-24 16:57:50 right, so AWS assigns an ip on an iface to host-a. you deassociate it with host-a (it gets removed htoplug style on the host) and then you assign it to other host which gets a hotplug event? 2023-03-24 16:58:10 yep 2023-03-24 16:58:25 and the host is supposed to fetch the metadata for that interface from aws 2023-03-24 16:58:43 that's what "use imds" does in the ifupdown-ng configs 2023-03-24 16:58:50 so you are effectively moving an interface with ip and all from host-a to host-b 2023-03-24 16:59:44 on new host: 0e:64:30:45:c5:05 2023-03-24 17:00:03 yup, so i figured, imds is used to fetch the network config via a link local address 2023-03-24 17:00:13 on old host: 0e:64:30:45:c5:05 2023-03-24 17:00:43 there's some networky things that don't come along on the DHCP ride 2023-03-24 17:00:51 thats pretty cool actually 2023-03-24 17:01:13 its like if you unplug an usb NIC and plug it into another host 2023-03-24 17:01:44 tomalok: so how are you updating the interface info in the metadata server? 2023-03-24 17:01:56 udhcpc can't do both ipv4 and ipv6 for example, and then there's secondary IPs that can be on an interface... and it's also possible to put a whole CIDR on an ENI 2023-03-24 17:02:07 i assume aws handles the metadata 2023-03-24 17:02:14 yeah, udhcpc is pretty limited 2023-03-24 17:03:10 minimal: on the hosts i'm just reading what's there and making it happen (well, whole CIDRs aren't implemented yet) -- altering the ENIs themselves is done via AWS console or API 2023-03-24 17:04:04 tomalok: that's why AWS have the amazon-ec2-net-utils on their distro (and used by some other distros) to handle other interfaces and secondary addresses 2023-03-24 17:04:08 AWS also recently made it possible for IPv6-only networking -- and this doesn't work with that either. if we can sort that out, THEN i think that may justify a major version number bump ;) 2023-03-24 17:05:45 yes, and they obviously have a IPv6 address for the metadata server now 2023-03-24 17:05:50 minimal: yes, but it's only really friendly to Amazon Linux... i did read the code to figure out what it was doing before implementing in tiny-cloud (well, its predecessor) but it has evolved further by then 2023-03-24 17:06:19 yes, latest version of that (v2) appears to have been rewritten to use systemd-network lol 2023-03-24 17:06:23 so, for this to work properly, we'd need declarative config and set the configs when the uevent arrives (ala systemd) 2023-03-24 17:06:47 yeah the IPv6 metadata is not a problem... getting a single DHCP client to be configured to handle either-or-both IPv4 & IPv6 is. 2023-03-24 17:07:02 can dhcpcd do that? I think it may 2023-03-24 17:07:43 I believe the AWS "model" is - configure primary IP address via DHCP and then get all other network setting from metadata server (whether immediately, i.e. for secondary address(es)) or via hotplug event for additional interfaces 2023-03-24 17:07:57 that might be a reason to switch the cloud images to use that instead 2023-03-24 17:07:59 dhcpcd can do both v4 and v6 in a single daemon 2023-03-24 17:08:32 and i think it also does link-local 2023-03-24 17:09:01 but i don't think it can get an ip form dhcp and then append more addresses to the iface 2023-03-24 17:09:16 i dont think dhcp protocol is designed for that 2023-03-24 17:10:07 so in my mind its either: aws provide your stuff via dhcp, or you get the network config from imds and set it statically 2023-03-24 17:10:24 but getting network config from imds and then set dhcp does not make sense in my mind 2023-03-24 17:11:46 with dhcp you can set "static" ip (a single ip) on server side by connect the ip with mac address 2023-03-24 17:11:54 so there's where ifupdown-ng assists the DHCP client -- after DHCP is done the "use imds" kicks in and fills in the gaps 2023-03-24 17:12:39 "the gaps" here are additional ip addresses and routes? 2023-03-24 17:12:44 cloud-init sets up an ephemeral address so it can get to the metadata server in order to retrieve network config 2023-03-24 17:12:55 and then downs that epheremal IP afterwards 2023-03-24 17:13:06 ncopa: yep 2023-03-24 17:13:24 eth0: dhcp + imds 2023-03-24 17:13:28 eth1+: imds 2023-03-24 17:13:37 maybe 2023-03-24 17:13:52 minimal: and that epheremal ip is a link-local i assume? 2023-03-24 17:13:59 ncopa: those "gaps" are what I was trying to point out to you about tiny-cloud for unsupported cloud providers yesterday 2023-03-24 17:14:35 tomalok: i wonder if we really need dhcp? since we get the network config from imds 2023-03-24 17:15:19 ncopa: you can see it handled here: https://github.com/canonical/cloud-init/blob/main/cloudinit/net/ephemeral.py 2023-03-24 17:15:31 we do dhcp on eth0 so we that that interface up to be able to grab the metadata to set up the rest 2023-03-24 17:16:17 tomalok: for ephemeral cloud-init is just doing a "ip addr add" rather than using a dhcp client 2023-03-24 17:16:49 e.g line 120 of that link I sent 2023-03-24 17:17:38 ncopa: cloud-init is using link-local for IPv6 and a specific IP address for IPv4 2023-03-24 17:18:30 minimal: does cloud-init use any sort of dhcp later? 2023-03-24 17:18:50 looks like cloud-init does not use ipv4LL 2023-03-24 17:19:02 unless that is an implementation of ipv4LL 2023-03-24 17:19:40 actually I'm confused as there is both EphemeralIPv4Network (line 20) and EphemeralDHCPv4 (line 296) lol 2023-03-24 17:20:00 https://en.wikipedia.org/wiki/Link-local_address#IPv4 2023-03-24 17:20:20 "The simultaneous use of IPv4 addresses of different scope on the same interface, such as configuring link-local addresses as well as globally routable addresses, may lead to confusion and increased complexity.[1]: 1.9  Therefore, hosts search for a DHCP server on the network before assigning link-local addresses." 2023-03-24 17:20:46 i think that is how dhcpcd works, it will fall back to link-local if it fails to get dhcp 2023-03-24 17:20:49 ncopa: yes the AWS metadata server's IPv4 address is in the IPv4 local-local address block 2023-03-24 17:21:10 169.254.169.254 2023-03-24 17:21:36 can you reach that metadata after you have got your ipv4 from dhcp? i assume you can, via default route 2023-03-24 17:23:08 tomalok: i might be able to help with the networking bits, but after i have the cloud-config support done and nocloud 2023-03-24 17:23:22 i need the nocloud in for alpine v3.18 2023-03-24 17:24:19 tbh, i dont have that much respect from cloud-init code. I'm not sure its a good reference 2023-03-24 17:24:36 ncopa: you understand that nocloud won't help for cloud providers other than AWS/Azure/Oracle/Google that tiny-cloud supports? 2023-03-24 17:24:48 would be interesting to see what they do in amazon linux net utils 2023-03-24 17:25:01 minimal: yes 2023-03-24 17:25:09 https://github.com/amazonlinux/amazon-ec2-net-utils 2023-03-24 17:25:43 ncopa: i can create/publish nocloud images -- at the very least a golden image would be pushed to dev.alpinelinux.org/~tomalok/... 2023-03-24 17:26:47 ncopa: I don't think there is really any other reference point from a non-EC2-specific perspective 2023-03-24 17:26:49 i do eventually want a set of golden cloud images to be downloadable in all flavors 2023-03-24 17:27:11 need to go afk for a while 2023-03-24 17:29:43 tomalok: you're using Packer "natively" on each cloud provider? 2023-03-24 17:30:43 tomalok: yeah, thats why i am working on the cloud-config support in tiny-cloud 2023-03-24 17:31:42 i may also enable tiny-cloud on all alpine iso images so you can do unattended installs of alpine by simply provide a nocloud config on a volume with LABEL=cidata 2023-03-24 17:32:05 ncopa: can build you a 3.17 and/or edge one in short order if you like 2023-03-24 17:32:41 minimal: no longer using cloud providers directly. images are built for each set of dimensions with QEMU 2023-03-24 17:32:44 i sort of built my own some weeks ago 2023-03-24 17:33:27 tomalok: ah, ok, that's good as it would be a pain needing resources in each provider in order to create images 2023-03-24 17:33:28 but i may ask about a nocloud image soonish from edge 2023-03-24 17:33:34 if i can get a stable upload site (for stable downloading) we can start offering different images for a whole lotta clouds without having to import them to those clouds 2023-03-24 17:34:12 tomalok: "for a whole lotta clouds"? how would that work without support for various metadata servers? 2023-03-24 17:34:12 we might be able to ship them via dl-cdn.alpinelinux.org at some point 2023-03-24 17:34:19 that was my idea 2023-03-24 17:34:26 I was thinking about that as well 2023-03-24 17:34:35 need to go now as well 2023-03-24 17:34:37 its late 2023-03-24 17:34:39 o/ 2023-03-24 17:35:20 ncopa: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/v3.17/cloud/aws/aarch64/ 2023-03-24 17:35:23 for example 2023-03-24 17:35:49 Need to think about images for azure as well soon 2023-03-24 17:36:38 if we can figure out where the builder can upload and where alpinelinux.org/cloud can provide downloads we're mostly there. 2023-03-24 17:36:48 gotta run for reals now 2023-03-27 14:30:07 ncopa: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/edge/cloud/nocloud/ - if there's need for other arch's we can sort out QEMU for that 2023-03-27 14:30:42 ikke: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/edge/cloud/azure/ - probably "alpha" quality and still some tweaking to do 2023-03-27 14:32:16 edge/3.18 cloud images also switch to using dhcpcd so that one DHCP client handles both/either ipv4/ipv6 as necessary 2023-03-27 15:30:57 nice! 2023-03-27 16:08:38 Ah azure images nice! 2023-03-27 16:11:08 also oci and gcp 2023-03-27 16:11:39 all need testing (and probably tweaking) 2023-03-27 18:18:33 tomalok: nice