2023-03-06 19:30:22 <minimal> FYIhttps://aws.amazon.com/about-aws/whats-new/2023/03/amazon-ec2-create-amis-boot-uefi-legacy-bios/
2023-03-06 20:53:21 <tomalok> thanks minimal -- will take a look at that later -- vending a hybrid uefi/bios image has been on the backlog for a while now, hopefully there will be some good info here on how exactly to build such an image
2023-03-06 20:54:34 <tomalok> at the very least this simplifies things when we import/register the AMI
2023-03-06 21:33:40 <minimal> tomalok: Grub supports "hybrid" mode for UEFI and BIOS booting, I've used it in the past for USB sticks
2023-03-06 21:35:28 <minimal> Limini also supprts that but I've not yet tried it with Limine
2023-03-08 08:18:19 <ncopa> hi
2023-03-08 08:18:32 <ncopa> so i have been working on nocloud provider
2023-03-08 08:18:52 <ncopa> i think we need to implement "local-hostname", which is set before network
2023-03-08 08:19:10 <ncopa> so we can use the hostname when requesting ip address from dhcp
2023-03-08 08:19:26 <ncopa> which means that we need to run it in the tiny-cloud-early
2023-03-08 08:20:00 <ncopa> however, -eraly runs before mdev, which means it runs before we have remounted / as rw
2023-03-08 08:20:16 <ncopa> s/-eraly/-early/
2023-03-08 08:21:16 <ncopa> i looked at how cloud-init does it, and it has the stages 'local', 'config' and 'final'
2023-03-08 08:21:40 <ncopa> 'local' corresponds to tiny-cloud-early, but it is run as soon / is remounted rw
2023-03-08 08:22:15 <ncopa> and before network is started
2023-03-08 08:22:55 <ncopa> any ideas how we solve this?
2023-03-08 08:23:24 <ncopa> im thinking that we could move the mdev stuff
2023-03-08 08:24:12 <ncopa> we could move it to mdev-conf, which means we enable it by default for everyone
2023-03-08 09:57:46 <ncopa> what are the EBS /dev/xvd* symlinks needed for?
2023-03-08 11:13:50 <ncopa> I have added support for EBS aliases in mdev-conf: https://gitlab.alpinelinux.org/alpine/mdev-conf/-/merge_requests/4
2023-03-08 16:41:14 <minimal> ncopa: re "local-hostname", I guess it depends how exactly/how close to cloud-init behaviour you want to be, for cloud-init NoCloud you can either specify "hostname: value" in meta-data file or else specify "hostname: value" in user-data
2023-03-08 16:42:12 <minimal> likewise network config can either be included in meta-data, via "network-interfaces:" section or in network-config file
2023-03-08 16:55:28 <ncopa> well, what we need to do is set hostname before network is up
2023-03-08 16:55:46 <ncopa> so we can pass the hostname to the dhcp client
2023-03-08 16:57:15 <ncopa> i also notice that the expand_root does a remout -o rw
2023-03-08 16:58:01 <ncopa> i think tiny-cloud-early should happen after the /etc/init.d/root has executed and remounted root to rw
2023-03-08 17:00:32 <minimal> ncopa: right, well NoCloud is before network is up, unless you're using NoClout "seedfrom" to pull the config via URLs
2023-03-08 17:00:54 <minimal> speaking from a cloud-init perspective
2023-03-08 17:03:35 <ncopa> https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html#permitted-keys only mention "local-hostname"
2023-03-08 17:06:16 <ncopa> tomalok: the MRs that are not "Draft:" can be merged: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud/-/merge_requests
2023-03-08 17:10:42 <minimal> ncopa: oh, you're using "ds=" on cmdline, I was thinking mainly in terms of CIDATA ISO/FAT fs
2023-03-08 17:11:12 <minimal> so are you also using "seedfrom" via "ds=" to provide the meta-data and user-data?
2023-03-08 17:30:48 <ncopa> i think it accepts both LABEL=CIDATA and ds= on command line
2023-03-08 18:00:49 <minimal> ncopa: the "cidata"/"CIDATA" labels are automatically search for, I don't think they can be specified on cmdline
2023-03-08 18:02:05 <minimal> ncopa: for what you are working on are you obtaining meta-data/user-data/network-config from a filesystem (ISO/FAT) or from a seed url?
2023-03-08 18:03:25 <ncopa> currently im only adding tests for tiny-cloud, making sure things works as expected when I start refactor things
2023-03-08 18:04:02 <ncopa> the plan is to support filesystem cidata initally and later on seed url
2023-03-08 18:04:25 <ncopa> naturally,  a seed url needs to be fetched after network is configured
2023-03-08 18:05:45 <ncopa> i think we should let tiny-cloud-early execute after root is remounted rw
2023-03-08 18:05:55 <minimal> ncopa: for cloud-init with seed-url a ephimeral network is brought up to contact the seed-url, the "normal" network should then be brought up afterwards
2023-03-08 18:06:17 <ncopa> right
2023-03-08 18:06:18 <minimal> as network config can be obtained via the seed url(s)
2023-03-08 18:06:35 <ncopa> i guess tiny-cloud could support that as well
2023-03-08 18:06:49 <minimal> the same way that things work for e.g. AWS to get to metadata server
2023-03-08 18:06:59 <ncopa> yeah, i figured
2023-03-08 18:08:03 <ncopa> i wonder what is setting up the ephimeral network for alpine aws images currently?
2023-03-08 18:08:10 <minimal> though I seem to remember when I tested seedurls on Alpine with cloud-init I hit a problem with ifupdown-ng's ifup that the attempt to bring up the "proper" network didn't actually happen as ifupdown-ng (ifstate?) seemed to remember the previous settings rather than the modified (by cloud-init) /etc/network/interfaces file
2023-03-08 18:09:52 <ncopa> does not surprise me
2023-03-08 18:10:39 <minimal> it was approx 6 months ago I last tested this stuff with cloud-init
2023-03-08 18:11:13 <minimal> unfortunately ifupdown-ng development in general seems to have "paused"
2023-03-08 18:13:43 <ncopa> ephemeral network could be set up by something else than ifupdown?
2023-03-08 18:14:27 <minimal> from memory cloud-init uses dhclient (certainly for AWS, etc)
2023-03-08 18:15:14 <minimal> that's the only DHCP client that cloud-init officially uses (though udhcpc and dhcpcd work in scenarios where ephemerial DHCP is not needed)
2023-03-08 18:16:39 <ncopa> 169.254.169 is link local address, i guess something configures link-local ipv4 before network is "started"
2023-03-08 18:17:10 <minimal> are you talking about tiny-cloud or cloud-init?
2023-03-08 18:18:22 <ncopa> cloud-init
2023-03-08 18:18:38 <minimal> just looking at the DS code currently
2023-03-08 18:19:11 <ncopa> will continue tomorrow. its late. have a nice evening!
2023-03-08 18:19:55 <minimal> ncopa: yeah, I'll start up a VM which uses seedfrom to refresh my memory
2023-03-09 02:40:39 <tomalok> ncopa: non-draft MRs are merged
2023-03-09 07:55:56 <ncopa> awesome! thanks!
2023-03-09 16:37:55 <Ariadne> 10:10 AM <minimal> it was approx 6 months ago I last tested this stuff with cloud-init
2023-03-09 16:37:55 <Ariadne> 10:11 AM <minimal> unfortunately ifupdown-ng development in general seems to have "paused"
2023-03-09 16:38:11 <Ariadne> not paused intentionally.  present maintainers are just busy with other projects
2023-03-09 16:38:16 <Ariadne> we can add more maintainers :)
2023-03-09 16:59:54 <minimal> Ariadne: understood. It has been almost 1 year since the last release. I'm not a C coder, otherwise I would volunteer
2023-03-10 14:13:25 <minimal> ncopa: FYI I've noticed when using "ds=" on cmdline that, certainly with Grub, I have to enclose the ds value in quotes as otherwise the cmdline passed by Grub to the kernel is truncated at the 1st semicolon
2023-03-24 16:11:52 <ncopa> tomalok: I have added EBS link support to mdev-conf, so we should be able to let tiny-cloud early run after coldplug (kernel modules loaded), after rootfs is remouted rw, but before network
2023-03-24 16:12:30 <ncopa> as I understand it had to run before coldplug so the mdev handler was installed before kernel modules were loaded
2023-03-24 16:25:06 <tomalok> ncopa that was a while ago when that was decided but without looking too close it sounds right.  i think tiny-cloud sets up aliases for xvdX* as well as sdX*, does that also hold true for what's in mdev-conf?
2023-03-24 16:26:10 <ncopa> i think it only sets what nvme-cli shows, so either xvdX* or sdX* but not both
2023-03-24 16:26:14 <tomalok> also was the mdev-conf update in place for 3.17?
2023-03-24 16:26:22 <ncopa> no, not 3.17
2023-03-24 16:26:25 <ncopa> this is for 3.18
2023-03-24 16:26:47 <ncopa> what are those symlinks needed for?
2023-03-24 16:28:46 <tomalok> they may have both been put into place because someone else did it that way and/or non-NVMe disks may have also had both...  or EC2 is/was ambiguous... like i said, been a while.
2023-03-24 16:30:37 <ncopa> logic is here: https://gitlab.alpinelinux.org/alpine/mdev-conf/-/commit/87c664016d5e4ae8131857e3fcab425f6f97bb0b
2023-03-24 16:31:06 <tomalok> at the moment i'm trying to figure out some EC2 ENI eth1+ issues which (i think) are affecting 3.17 -- so after that's solved and backported, we could in theory drop the mdev thing for 3.18+
2023-03-24 16:31:29 <ncopa> do you mind if we only create a single symlink and not both? and if it turns out we need both we add that later?
2023-03-24 16:31:49 <ncopa> iirc the AWS docs only mentions how to create a single symlink
2023-03-24 16:31:56 <tomalok> i don't mind, but we'd need to probably have a release note because behavior change
2023-03-24 16:32:01 <ncopa> yeah
2023-03-24 16:32:12 <ncopa> and maybe bump major version number
2023-03-24 16:32:27 <tomalok> at least minor version
2023-03-24 16:33:14 <ncopa> it is a potentially breaking change. we also move the time where init-early is executed, so i think it justifies a major version number
2023-03-24 16:33:20 <ncopa> i mean, its just a number
2023-03-24 16:33:56 <ncopa> what is the issue with EC2 ENI eth1 issue?
2023-03-24 16:34:07 <ikke> heh
2023-03-24 16:34:55 <tomalok> when i'm attaching an ENI as eth1, it's not routing properly
2023-03-24 16:34:55 <ncopa> i didnt quite understand how that was supposed to work. it look like it tries to configure eth0 (eth*) when it shows up in kernel uevent (cold or hotplug)
2023-03-24 16:35:23 <ncopa> but the code looked broken
2023-03-24 16:36:31 <ncopa> im currently working on tests for init-main, and i just realized it will be complicated with a generic set_fake_meta
2023-03-24 16:36:51 <tomalok> mdev triggers a (potential) rebuild of assemble interfaces and (currently) sets up another udhcpc on eth1, and then goes to look for more IP/IPv6 in imds metadata
2023-03-24 16:37:18 <ncopa> does the mdev trigger actually work?
2023-03-24 16:37:25 <ncopa> is the trigger executed?
2023-03-24 16:37:27 <tomalok> yeah, that works fine
2023-03-24 16:37:58 <tomalok> the problem is once it's done things aren't routing properly and there's two default routes.
2023-03-24 16:38:28 <tomalok> so i'm exploring _not_ running DHCP and just getting imds metadata and setting it up that way
2023-03-24 16:38:40 <tomalok> (no dhcp on eth1)
2023-03-24 16:40:27 <ncopa> two different default routes?
2023-03-24 16:43:47 <tomalok> https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System using this as a reference to help detangle -- previously i was using the table/rule stuff for associating secondary IPs on the same interface (though I didn't have both "ip rule add from/to"
2023-03-24 16:45:57 <minimal> I thought the xvdX names was because AWS used those originally for disks (back when they used Xen) rather than sdX or vdX
2023-03-24 16:47:31 <tomalok> i'm currently working on  setting up static /etc/network/interfaces.d/eth1 to see if going that route solves the problem -- this may end up reworking (maybe simplifying) the hotplug for ENIs
2023-03-24 16:48:25 <tomalok> the things i do on vacation... :P
2023-03-24 16:49:37 <ncopa> heh
2023-03-24 16:50:13 <ncopa> sounds like pretty advanced networking setup
2023-03-24 16:50:32 <ncopa> how does that actually work from aws? Or, how is it supposed to work?
2023-03-24 16:50:42 <minimal> tomalok: I thought AWS used their own ethernet "tool/package" to set up all networking from the metadata server info? or perhaps all-but-eth0's-main-IP
2023-03-24 16:51:06 <ncopa> you get a second interface, eth1, with a separate subnet and separate default gw?
2023-03-24 16:51:11 <tomalok> really i just want to float static IPv4 & IPv6 between nodes...  
2023-03-24 16:51:44 <ncopa> "float static ip...?"
2023-03-24 16:52:31 <minimal> tomalok: can you do that on AWS? don't you have to rely on the IPs they give the machine on each boot (and then use an elastic IP for "floating")
2023-03-24 16:53:19 <ncopa> multiple default gw may lead to assymetric routing which is why you need a separate routing table which is selected based on source ip
2023-03-24 16:53:26 <tomalok> so i have a docker swarm, and when it's time to refresh each node, in theory i can just launch a new one with all the shiny, and it joins the swarm on the private network.  when it's up, i simply detach the ENI from an old node, and attach it to the new node.  the attaching generates an event that mdev picks up on, and brings up eth1 (per
2023-03-24 16:53:26 <tomalok> ifupdown-ng's configs)
2023-03-24 16:53:57 <tomalok> ncopa: yeah which is why i'm seeing connections only getting half-made
2023-03-24 16:54:40 <ncopa> i create a tool called pingu a dcade ago, that did multi-isp failover (and load-balancing)
2023-03-24 16:54:49 <tomalok> i'm sure that dockerd isn't helping the situation, but for now i'm just tying to establish a connection to SSH port, which should be listening on all interfaces
2023-03-24 16:55:21 <ncopa> so basically, you want to "move" an ip from one host to another?
2023-03-24 16:56:00 <tomalok> yeah, but you have to move detach/attach an AWS reesource so it's not exactly like it can all be done on the hosts
2023-03-24 16:56:40 <ikke> tomalok: does the interface keep the mac address?
2023-03-24 16:56:57 <tomalok> at best the hosts detect that the ENI is gone (and ifdown eth1) and the ENI is there (and ifup eth1) -- there are two other static IPs so, the cluster is fine in the interim
2023-03-24 16:57:10 <tomalok> ikke: good question, haven't checked
2023-03-24 16:57:50 <ncopa> right, so AWS assigns an ip on an iface to host-a. you deassociate it with host-a (it gets removed htoplug style on the host) and then you assign it to other host which gets a hotplug event?
2023-03-24 16:58:10 <tomalok> yep
2023-03-24 16:58:25 <ncopa> and the host is supposed to fetch the metadata for that interface from aws
2023-03-24 16:58:43 <tomalok> that's what "use imds" does in the ifupdown-ng configs
2023-03-24 16:58:50 <ncopa> so you are effectively moving an interface with ip and all from host-a to host-b
2023-03-24 16:59:44 <tomalok> on new host: 0e:64:30:45:c5:05
2023-03-24 17:00:03 <ncopa> yup, so i figured, imds is used to fetch the network config via a link local address
2023-03-24 17:00:13 <tomalok> on old host: 0e:64:30:45:c5:05
2023-03-24 17:00:43 <tomalok> there's some networky things that don't come along on the DHCP ride
2023-03-24 17:00:51 <ncopa> thats pretty cool actually
2023-03-24 17:01:13 <ncopa> its like if you unplug an usb NIC and plug it into another host
2023-03-24 17:01:44 <minimal> tomalok: so how are you updating the interface info in the metadata server?
2023-03-24 17:01:56 <tomalok> udhcpc can't do both ipv4 and ipv6 for example, and then there's secondary IPs that can be on an interface...  and it's also possible to put a whole CIDR on an ENI
2023-03-24 17:02:07 <ncopa> i assume aws handles the metadata
2023-03-24 17:02:14 <ncopa> yeah, udhcpc is pretty limited
2023-03-24 17:03:10 <tomalok> minimal: on the hosts i'm just reading what's there and making it happen (well, whole CIDRs aren't implemented yet) -- altering the ENIs themselves is done via AWS console or API
2023-03-24 17:04:04 <minimal> tomalok: that's why AWS have the amazon-ec2-net-utils on their distro (and used by some other distros) to handle other interfaces and secondary addresses
2023-03-24 17:04:08 <tomalok> AWS also recently made it possible for IPv6-only networking -- and this doesn't work with that either.  if we can sort that out, THEN i think that may justify a major version number bump ;)
2023-03-24 17:05:45 <minimal> yes, and they obviously have a IPv6 address for the metadata server now
2023-03-24 17:05:50 <tomalok> minimal: yes, but it's only really friendly to Amazon Linux...  i did read the code to figure out what it was doing before implementing in tiny-cloud (well, its predecessor) but it has evolved further by then
2023-03-24 17:06:19 <minimal> yes, latest version of that (v2) appears to have been rewritten to use systemd-network lol
2023-03-24 17:06:23 <ncopa> so, for this to work properly, we'd need declarative config and set the configs when the uevent arrives (ala systemd)
2023-03-24 17:06:47 <tomalok> yeah the IPv6 metadata is not a problem...  getting a single DHCP client to be configured to handle either-or-both IPv4 & IPv6 is.
2023-03-24 17:07:02 <ncopa> can dhcpcd do that? I think it may
2023-03-24 17:07:43 <minimal> I believe the AWS "model" is - configure primary IP address via DHCP and then get all other network setting from metadata server (whether immediately, i.e. for secondary address(es)) or via hotplug event for additional interfaces
2023-03-24 17:07:57 <tomalok> that might be a reason to switch the cloud images to use that instead
2023-03-24 17:07:59 <minimal> dhcpcd can do both v4 and v6 in a single daemon
2023-03-24 17:08:32 <ncopa> and i think it also does link-local
2023-03-24 17:09:01 <ncopa> but i don't think it can get an ip form dhcp and then append more addresses to the iface
2023-03-24 17:09:16 <ncopa> i dont think dhcp protocol is designed for that
2023-03-24 17:10:07 <ncopa> so in my mind its either: aws provide your stuff via dhcp, or you get the network config from imds and set it statically
2023-03-24 17:10:24 <ncopa> but getting network config from imds and then set dhcp does not make sense in my mind
2023-03-24 17:11:46 <ncopa> with dhcp you can set "static" ip (a single ip) on server side by connect the ip with mac address
2023-03-24 17:11:54 <tomalok> so there's where ifupdown-ng assists the DHCP client -- after DHCP is done the "use imds" kicks in and fills in the gaps
2023-03-24 17:12:39 <ncopa> "the gaps" here are additional ip addresses and routes?
2023-03-24 17:12:44 <minimal> cloud-init sets up an ephemeral address so it can get to the metadata server in order to retrieve network config
2023-03-24 17:12:55 <minimal> and then downs that epheremal IP afterwards
2023-03-24 17:13:06 <tomalok> ncopa: yep
2023-03-24 17:13:24 <tomalok> eth0: dhcp + imds
2023-03-24 17:13:28 <tomalok> eth1+: imds
2023-03-24 17:13:37 <tomalok> maybe
2023-03-24 17:13:52 <ncopa> minimal: and that epheremal ip is a link-local i assume?
2023-03-24 17:13:59 <minimal> ncopa: those "gaps" are what I was trying to point out to you about tiny-cloud for unsupported cloud providers yesterday
2023-03-24 17:14:35 <ncopa> tomalok: i wonder if we really need dhcp? since we get the network config from imds
2023-03-24 17:15:19 <minimal> ncopa: you can see it handled here: https://github.com/canonical/cloud-init/blob/main/cloudinit/net/ephemeral.py
2023-03-24 17:15:31 <tomalok> we do dhcp on eth0 so we that that interface up to be able to grab the metadata to set up the rest
2023-03-24 17:16:17 <minimal> tomalok: for ephemeral cloud-init is just doing a "ip addr add" rather than using a dhcp client
2023-03-24 17:16:49 <minimal> e.g line 120 of that link I sent
2023-03-24 17:17:38 <minimal> ncopa: cloud-init is using link-local for IPv6 and a specific IP address for IPv4
2023-03-24 17:18:30 <tomalok> minimal: does cloud-init use any sort of dhcp later?
2023-03-24 17:18:50 <ncopa> looks like cloud-init does not use ipv4LL
2023-03-24 17:19:02 <ncopa> unless that is an implementation of ipv4LL
2023-03-24 17:19:40 <minimal> actually I'm confused as there is both EphemeralIPv4Network (line 20) and EphemeralDHCPv4 (line 296) lol
2023-03-24 17:20:00 <ncopa> https://en.wikipedia.org/wiki/Link-local_address#IPv4
2023-03-24 17:20:20 <ncopa> "The simultaneous use of IPv4 addresses of different scope on the same interface, such as configuring link-local addresses as well as globally routable addresses, may lead to confusion and increased complexity.[1]: 1.9  Therefore, hosts search for a DHCP server on the network before assigning link-local addresses."
2023-03-24 17:20:46 <ncopa> i think that is how dhcpcd works, it will fall back to link-local if it fails to get dhcp
2023-03-24 17:20:49 <minimal> ncopa: yes the AWS metadata server's IPv4 address is in the IPv4 local-local address block
2023-03-24 17:21:10 <minimal> 169.254.169.254
2023-03-24 17:21:36 <ncopa> can you reach that metadata after you have got your ipv4 from dhcp? i assume you can, via default route
2023-03-24 17:23:08 <ncopa> tomalok: i might be able to help with the networking bits, but after i have the cloud-config support done and nocloud
2023-03-24 17:23:22 <ncopa> i need the nocloud in for alpine v3.18
2023-03-24 17:24:19 <ncopa> tbh, i dont have that much respect from cloud-init code. I'm not sure its a good reference
2023-03-24 17:24:36 <minimal> ncopa: you understand that nocloud won't help for cloud providers other than AWS/Azure/Oracle/Google that tiny-cloud supports?
2023-03-24 17:24:48 <ncopa> would be interesting to see what they do in amazon linux net utils
2023-03-24 17:25:01 <ncopa> minimal: yes
2023-03-24 17:25:09 <minimal> https://github.com/amazonlinux/amazon-ec2-net-utils
2023-03-24 17:25:43 <tomalok> ncopa: i can create/publish nocloud images -- at the very least a golden image would be pushed to dev.alpinelinux.org/~tomalok/...
2023-03-24 17:26:47 <minimal> ncopa: I don't think there is really any other reference point from a non-EC2-specific perspective
2023-03-24 17:26:49 <tomalok> i do eventually want a set of golden cloud images to be downloadable in all flavors
2023-03-24 17:27:11 <tomalok> need to go afk for a while
2023-03-24 17:29:43 <minimal> tomalok: you're using Packer "natively" on each cloud provider?
2023-03-24 17:30:43 <ncopa> tomalok: yeah, thats why i am working on the cloud-config support in tiny-cloud
2023-03-24 17:31:42 <ncopa> i may also enable tiny-cloud on all alpine iso images so you can do unattended installs of alpine by simply provide a nocloud config on a volume with LABEL=cidata
2023-03-24 17:32:05 <tomalok> ncopa: can build you a 3.17 and/or edge one in short order if you like
2023-03-24 17:32:41 <tomalok> minimal: no longer using cloud providers directly.  images are built for each set of dimensions with QEMU
2023-03-24 17:32:44 <ncopa> i sort of built my own some weeks ago
2023-03-24 17:33:27 <minimal> tomalok: ah, ok, that's good as it would be a pain needing resources in each provider in order to create images
2023-03-24 17:33:28 <ncopa> but i may ask about a nocloud image soonish from edge
2023-03-24 17:33:34 <tomalok> if i can get a stable upload site (for stable downloading) we can start offering different images for a whole lotta clouds without having to import them to those clouds
2023-03-24 17:34:12 <minimal> tomalok: "for a whole lotta clouds"? how would that work without support for various metadata servers?
2023-03-24 17:34:12 <ncopa> we might be able to ship them via dl-cdn.alpinelinux.org at some point
2023-03-24 17:34:19 <tomalok> that was my idea
2023-03-24 17:34:26 <ikke> I was thinking about that as well
2023-03-24 17:34:35 <ncopa> need to go now as well
2023-03-24 17:34:37 <ncopa> its late
2023-03-24 17:34:39 <ikke> o/
2023-03-24 17:35:20 <tomalok> ncopa: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/v3.17/cloud/aws/aarch64/ 
2023-03-24 17:35:23 <tomalok> for example
2023-03-24 17:35:49 <ikke> Need to think about images for azure as well soon
2023-03-24 17:36:38 <tomalok> if we can figure out where the builder can upload and where alpinelinux.org/cloud can provide downloads we're mostly there.
2023-03-24 17:36:48 <tomalok> gotta run for reals now
2023-03-27 14:30:07 <tomalok> ncopa: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/edge/cloud/nocloud/ - if there's need for other arch's we can sort out QEMU for that
2023-03-27 14:30:42 <tomalok> ikke: https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/edge/cloud/azure/ - probably "alpha" quality and still some tweaking to do
2023-03-27 14:32:16 <tomalok> edge/3.18 cloud images also switch to using dhcpcd so that one DHCP client handles both/either ipv4/ipv6 as necessary
2023-03-27 15:30:57 <ncopa> nice!
2023-03-27 16:08:38 <clandmeter> Ah azure images nice!
2023-03-27 16:11:08 <tomalok> also oci and gcp
2023-03-27 16:11:39 <tomalok> all need testing (and probably tweaking)
2023-03-27 18:18:33 <ikke> tomalok: nice
2023-03-28 10:30:33 <ncopa> tomalok: imds' ssh-key is the authorized keys for default user and not host keys, right?
2023-03-28 13:41:17 <tomalok> ncopa: correct
2023-03-28 14:21:27 <tomalok> that is, imds @ssh-keys
2023-03-29 10:27:07 <clandmeter> hi
2023-03-29 10:27:59 <clandmeter> I just talked to ncopa, and i think we want to setup a meeting with everybody involved in cloud (and want to be involved).
2023-03-29 10:29:43 <clandmeter> ncopa, tomalok, mcrute, minimal, ikke (psykose?) and myself. If anybody else here would like to join please let us know. 
2023-03-29 15:29:47 <mcrute> clandmeter: sounds good! Any thoughts on scheduling yet?
2023-03-29 17:36:47 <tomalok> mcrute: hi!  looks like we reached the max limit of 148 public images in at least one of the regions...
2023-03-29 17:37:13 <tomalok> 2023-03-29T17:34:55Z: ==> alpine.null.3.16-x86_64-uefi-cloudinit-aws (shell-local): botocore.exceptions.ClientError: An error occurred (ResourceLimitExceeded) when calling the ModifyImageAttribute operation: You have reached your quota of 148 for the number of public images allowed in this Region. Deregister unused public images or make them
2023-03-29 17:37:13 <tomalok> private, or request an increase in your public AMIs quota.
2023-03-29 17:37:43 <tomalok> probably us-west-2, but i'll take a look after the publish job is complete
2023-03-29 18:32:52 <mcrute> tomalok: will get a QIR in right now... I need to go back to shaking the tree over there to get our images added to the main catalog
2023-03-29 18:33:14 <mcrute> my best contact has since left the company and my second best contact is being flighty at the moment :-/
2023-03-29 18:34:52 <mcrute> huh, I dunno where that 148 number comes from... the quota is set at 316
2023-03-29 18:36:30 <mcrute> sigh... I have to do it on a region-by region basis... I'm going to have to script this but should be able do it tonight with a quick hack
2023-03-29 18:38:34 <mcrute> tomalok: what number do you think makes sense? looks like we're around 259 in west-2 and east-1, so maybe 500?
2023-03-29 19:07:30 <tomalok> mcrute: this is an old question...  how long do we want to hang on to old AMIs?  ;)  the latest images do have an EOL deprecation set but I don't think that makes them non-public, they're still accessible, but otherwise unlisted
2023-03-29 19:32:52 <mcrute> tomalok: I think that's a good topic for our group meeting... my original though was we keep them forever *waves hand* mainly because people tend to encode public image IDs into things like CloudFormation templates of ECS cluster or ASGs and then if the image ever disappears those constructs stop working when they try to scale or replace or change infrastructure which leads to either the mutation failing...
2023-03-29 19:33:02 <mcrute> or in the worst case breaking people's infrastructure
2023-03-29 19:34:11 <mcrute> that being said it's actually rather personally financially expensive to do that and I'm keen to not have that bill continue to grow (it's grown by 3x since we started this project)
2023-03-29 19:35:17 <tomalok> if we make old images available downloadable, they wouldn't necessarily need to be _our_ AMIs
2023-03-29 19:35:21 <mcrute> so we need to figure out as a project what we want to do here, if there's some other way to fund this then I think it's worthwhile to keep them all for stability and ease-of-use, if it's not then maybe we should make a hard pivot to your import idea plus a rotating cast of n-1 images globally and publish advice on how people can ensure stability
2023-03-29 19:35:34 <mcrute> thanks for attending my TED talk
2023-03-29 19:35:50 <tomalok> afk for a bit
2023-03-29 19:36:26 <mcrute> tomalok: yeah totally and CDN cost is covered in different ways... I'm all on-board with this but want it to be a group consensus that we advertise publicly before we pull support for old images (maybe give a 6-month notice)
2023-03-29 19:37:03 <mcrute> I'll be in an out today too... will try to pay some attention as I can
2023-03-29 22:51:13 <tomalok> mcrute: 500 public per region would hold us for a little while
2023-03-29 22:52:37 <psykose> isn't an AMI just an image? why are there 500?
2023-03-29 22:54:50 <psykose> i can see by https://alpinelinux.org/cloud/ most of the duplication is having one per region (is there no way to just have a global image?), but aside from that it's nowhere near 500 by image type+version matrix (no region duplication)
2023-03-29 22:56:00 <tomalok> there is a pending change to cloud/ where we do create various flavors of "golden" image variants for download.  however, we have historically also imported and published into AWS regions directly to make it convenient to use.
2023-03-29 22:58:23 <tomalok> we have the images over under https://dev.alpinelinux.org/~tomalok/alpine-cloud-images and the cloud page actually has the commented out code ready for when we decide where it gets uploaded for reals, and where everyone downloads it from (probably cdn, i'd imagine)
2023-03-29 22:58:47 <tomalok> we have to do somethingn like this anyways if we want to vend actual nocloud images -- there's no cloud provider to import/publish to :)
2023-03-29 22:59:24 <tomalok> and it allows us to get the other clouds "out there" before we do any coding for their import/publish stuff
2023-03-29 23:01:11 <tomalok> (the 500 is per region, and apparently not all regions had their limit increased beyond 148 (an odd number, but probably 150 somewhere)
2023-03-29 23:12:36 <psykose> what i mean is that https://dev.alpinelinux.org/~tomalok/alpine-cloud-images/v3.17/cloud/aws/x86_64/ is 4 images (bios,uefi tiny,cloudinit) per version per architecture
2023-03-29 23:12:46 <psykose> if you don't ship anything except the latest version this is not very many images
2023-03-29 23:12:58 <psykose> and i see zero reason to ship 3.17.0 instead of only .3 for instance
2023-03-29 23:14:34 <psykose> maybe that's a hot take? :p
2023-03-29 23:19:12 <minimal> how would 3.17.0 or 3.17.2 be shipped? there's no separate repos for 3.17.x so an image of 3.17.x built at a point in time is an image of that point in time - if 2 images are built a week apart then they're different images (assuming some package included in them has changed)
2023-03-29 23:20:05 <minimal> unless they're only built within a few minutes of the particular 3.17.x release and never changed
2023-03-29 23:20:48 <psykose> i assume it basically is that, they're just built at the same time as release and that's all, same as the .iso's already
2023-03-29 23:21:02 <psykose> which is just more to the point that keeping anything except the latest is a waste of space
2023-03-29 23:23:20 <minimal> I guess in theory you might want to use 3.17.2 (for example) if there's an issue with 3.17.3 (e.g. problem with updated kernel), but once that image (AMI/whatever) is used to create a VM I assume that VM will/should have "apk update/apk upgrade" run on it on a regular basis at which point the originating AMI becomes irrelevant
2023-03-30 00:04:32 <mcrute> psykose: if we publish an image we never want to retract it because image IDs get baked into cloud automation and then if the image disappears the automation breaks and causes sadness for people (and broken infrastructure)
2023-03-30 00:05:00 <mcrute> the real safe way would be for people to import the image to their account then we can't retract it accidentally but if they choose to use the easy path, which is our AMIs directly then we have this problem
2023-03-30 00:06:42 <mcrute> also images are immutable so we can't keep stable image IDs but swap the backing image
2023-03-30 00:08:27 <mcrute> a worst case example is if someone uses CloudFormation to declare an autoscaling group and their app mostly just chugs along, then we delete an AMI that they are using and all of a sudden they get a burst of traffic, the ASG tries to scale, can't because the image is gone and somebody takes an outage and it's effectively our fault
2023-03-30 00:08:51 <mcrute> it's a bad experience at best and makes people feel crappy about their distro choice at worst
2023-03-30 00:10:26 <mcrute> at least with AWS there is no such thing as a "global" image... they bill for storage and have regional isolation (also cross-region image copy would really suck for latency in many cases)
2023-03-30 00:13:39 <psykose> that sounds like there are no solutions then if you have to keep around every image for a million years
2023-03-30 00:13:57 <mcrute> that's kinda why this was designed the way it was :-)
2023-03-30 00:14:00 <psykose> fundamentally this is impossible unless you want to pay a million dollars for storage :p
2023-03-30 00:14:41 <mcrute> well the real answer IMO is to publish a clear support timeline for our AMIs and document how to import your own if you need more stability than we guarantee 
2023-03-30 00:14:55 <psykose> hmm
2023-03-30 00:14:57 <mcrute> we just never made that deal with people using the AMIs and we'd need to walk it back then have some sunset period
2023-03-30 00:15:11 <psykose> i have a more cynical answer that is more obvious i think, but that's not very productive
2023-03-30 00:16:21 <mcrute> I mean you're not wrong :-P
2023-03-30 00:16:38 <psykose> i don't think there's a good way even with "support periods" if the model at its core doesn't support the use
2023-03-30 00:17:05 <psykose> even if you put some arbitrary number in there ("all images published will be around for 3 months") it's a bit fundamentally annoying to have someone go around updating ids all the time
2023-03-30 00:18:10 <psykose> that's why most people never use this sha: functionality in `FROM` specs in dockerfiles, and instead pick something like 3.17 and always just get the latest 3.17 image, on top of usually upgrading on run
2023-03-30 00:18:11 <mcrute> well if our documented support period is say, 6 months, then if you know you can change images in 6 months you can easily use our images
2023-03-30 00:18:29 <psykose> stable release yes, and then you also just get the latest one without doing anything or needing to roll to new 'ami id' all the time
2023-03-30 00:18:34 <mcrute> if you have a longer support horizon you should import the image from our golden versions to your account directly where you control the lifecycle
2023-03-30 00:19:17 <mcrute> sadly, that is not how AMIs work
2023-03-30 00:20:00 <mcrute> they're meant to be a stable and immutable image that you can forever and always depend on... AWS has other high-level constructs to abstract AMI IDs away from you but they're pretty use-case specific so AMI ID is what lots of tools use
2023-03-30 00:20:51 <mcrute> but you're still right... we're just kinda stuck dealing with whatever design decisions the cloud providers make... I would have designed it different, for sure
2023-03-30 00:26:45 <psykose> it's not a fundamentally bad idea in itself, just hard to make work with lots of other things built around it
2023-03-30 00:26:53 <psykose> without*
2023-03-30 00:32:26 <minimal> "always depend on" - until for example the company who's AMIs you rely upon goes bankrupt and then the AMIs disappear ;-)
2023-03-30 00:34:59 <mcrute> minimal: but we can not be like them!
2023-03-30 00:35:26 <mcrute> minimal: or in this case "when that guy in Seattle gets sick of paying the storage bill"
2023-03-30 00:36:11 <psykose> idk why you're paying it at all :D
2023-03-30 00:36:42 <mcrute> it cost way less when I started this project and was more directly beneficial to me than it is today
2023-03-30 00:37:01 <mcrute> now it's mostly just because I like the Alpine community I guess :-)
2023-03-30 00:37:26 <mcrute> there was some talk about a formal legal organization for Alpine that would eventually adopt this but I got super busy and forgot to keep bugging ncopa about it
2023-03-30 00:40:00 <psykose> legal organisations don't sound very fun
2023-03-30 00:40:08 <psykose> sounds too business :D
2023-03-30 00:40:26 <psykose> as for paying for it it would make more sense if amazon had just sponsored all of the image storage for free
2023-03-30 00:40:31 <psykose> or otherwise just not get any images
2023-03-30 00:40:58 <psykose> it's how everything else works i guess
2023-03-30 00:58:31 <mcrute> yeah it's not... they deal with money
2023-03-30 00:59:29 <psykose> well, money is pretty common, if "some account owned by someone with credits in it" is money
2023-03-30 00:59:39 <psykose> that's how azure/linode is sponsored for alpine i guess
2023-03-30 00:59:57 <psykose> and we don't even use the former
2023-03-30 01:15:38 <psykose> clandmeter: i'm probably not that interested in it, but i think minimal should be in the meeting
2023-03-30 01:25:37 <mcrute> psykose: been meaning to get Azure/Google up and running I've just had precious little free time to do it, credits are nice
2023-03-30 01:26:15 <mcrute> I wonder if anyone around has good contacts at AWS... a lot of mine left recently... I still have a few I could poke but they're in weird unrelated places
2023-03-30 01:28:38 <psykose> i thought half the people here worked for aws :p
2023-03-30 01:47:00 <mcrute> I thought I was the only one
2023-03-30 01:47:16 <mcrute> emphasis on was... haven't been there in at least 2.5 years
2023-03-30 01:49:30 <psykose> wonder where i got that feeling
2023-03-30 02:24:17 <tomalok> i did 10 years at amazon/aws
2023-03-30 02:26:02 <tomalok> we have the base images for Azure, GCP, and OCI now (at least for edge) -- but they probably need some testing/tweaking.
2023-03-30 07:47:11 <ncopa> hum, aws images is a problem if it all depends on one guy in Seattle willing to pay the increasing storage bill and we can never delete anything
2023-03-30 07:48:40 <ncopa> i suppose we could let people pay for using it?
2023-03-30 13:40:48 <minimal> ncopa: the other approach would be to provide images for AWS that user's have to import into their account themselves (i.e. from the Alpine website)
2023-03-30 13:41:29 <minimal> but as some users can be lazy that might put them off using Alpine on AWS
2023-03-30 14:25:31 <mcrute> ncopa: well not in the immediate term, but certainly something a guy in Seattle would like to have a better plan for :-)
2023-03-30 14:27:20 <mcrute> I'm guessing other major distros have their storage donated to them... I guess I'll try to shake the AWS tree again
2023-03-30 14:40:31 <tomalok> i hear the guy in seattle does have a plan to not be the guy that always builds the cloud images
2023-03-30 14:42:19 <tomalok> or were we refering to the other guy in seattle who's been footing the AWS bill?
2023-03-30 14:43:55 <tomalok> anyways...  if the cloud providers officially provided our Alpine images, would they take on the responsibiility of paying for hosting them in all the regions?
2023-03-30 14:47:51 <minimal> normally many cloud providers themselves provide images for multiple distros
2023-03-30 14:48:55 <mcrute> tomalok: I mean there's the guy in Seattle who does most of the work and the other guy who's distracted but still pays the bill
2023-03-30 14:49:40 <mcrute> minimal: that's very much not the case in my experience on the inside, the big distros provide their own images
2023-03-30 14:50:06 <mcrute> the cloud providers do tend to have distro teams that help make sure things work well on their hardware though
2023-03-30 14:52:47 <minimal> mcrute: I'm referring to the likes of:
2023-03-30 14:52:48 <minimal> https://www.vultr.com/features/operating-systems/
2023-03-30 14:52:53 <minimal> https://www.linode.com/distributions/
2023-03-30 14:53:14 <minimal> https://www.digitalocean.com/products/droplets  https://www.digitalocean.com/products/droplets
2023-03-30 14:53:28 <mcrute> ahh I see, I wouldn't call those "clouds" no matter what they call themselves :-)
2023-03-30 14:53:33 <minimal> "Deploy your Droplet to the distribution you prefer"
2023-03-30 14:53:49 <mcrute> but yeah VPS providers have tended to be much more independent in their image creation
2023-03-30 14:54:17 <minimal> https://docs.hetzner.com/robot/dedicated-server/operating-systems/standard-images/
2023-03-30 14:54:20 <minimal> etc
2023-03-30 14:54:23 <mcrute> which is totally fine, the actual cloud providers mostly depend on distros providing their own images and if you're big enough they'll give you time with their distro team
2023-03-30 14:54:43 <minimal> well any distinction between "cloud" and "vps" is arbitrary
2023-03-30 14:55:55 <mcrute> I think we'll have to disagree on that one :-)
2023-03-30 14:56:33 <minimal> what makes a provider an "actual" cloud provider? APIs for provisioning etc? what's the criteria to distinguish?
2023-03-30 14:57:04 <mcrute> nah, level of infrastructure investment and who owns the infrastructure, as well as the primary market they address
2023-03-30 14:57:30 <minimal> so it's basically about money?
2023-03-30 14:57:43 <mcrute> IMO, cloud providers have massive infrastructure investment, they own most or all of that infrastructure themselves, and they tend to address a large corporate market
2023-03-30 14:57:44 <minimal> rather than technology
2023-03-30 14:58:28 <mcrute> I guess one bar I use to determine if a provider is a cloud or a VPS is this... would a large bank or medical company move their highly regulated workloads to the provider?
2023-03-30 14:58:37 <mcrute> if the answer is no, then you're probably a VPS
2023-03-30 14:58:40 <minimal> so who then are all the (few) cloud providers? Apart from AWS, Azure, GCE, and Oracle?
2023-03-30 14:58:41 <tomalok> random google https://www.scalahosting.com/blog/what-is-the-difference-between-vps-and-cloud-hosting/
2023-03-30 14:58:53 <minimal> mcrute: that's a subjective decision, not on "real" merits
2023-03-30 14:59:02 <mcrute> AWS, Azure, GCE, OCI, IBM, and Alibaba are the ones that come to mind
2023-03-30 14:59:07 <minimal> is IBMCloud a cloud provider?
2023-03-30 14:59:13 <mcrute> minimal: totally agreed, this is just my opinion
2023-03-30 14:59:38 <mcrute> and TBF I think VPSes are great and fill an important role in the ecosystem (I use them myself)
2023-03-30 15:00:04 <minimal> so then is there a plan to have an Alpine 2nd-class-VPS team? I'd assuming the Cloud Team who cover more than just the few big providers
2023-03-30 15:00:13 <mcrute> but it's in-vogue now to call any API-driven compute leasing service a cloud and that's not really correct
2023-03-30 15:01:08 <mcrute> IMO, yes we should address the VPS providers too... but... as you've pointed out they tend to make their own images so I think the amount of value we add there immediately is pretty low
2023-03-30 15:01:39 <mcrute> the big cloud providers don't have an easy path for people to use Alpine and that was the first thing I was hoping to address way back when I made the first AWS images
2023-03-30 15:02:03 <mcrute> tomalok has done a lot more work than I have at this point though and I think he has a bigger take on this than I do
2023-03-30 15:04:18 <minimal> the likes of Vultr aren't technically V~PS though - the term VPS goes bac k quite a few years to the days of chroot-based/jail-based or equivalent "servers" wheres Vultr, Digitalocean etc are VM based, just like AWS etc
2023-03-30 15:05:56 <psykose> i like this distinction between cloud and vps actually
2023-03-30 15:06:08 <psykose> because it makes it clear which one to use and which one to avoid with a 50 foot pole :-)
2023-03-30 15:06:30 <minimal> psykose: you mean avoid AWS, Azure, etc? ;-)
2023-03-30 15:06:44 <mcrute> psykose: trick question! the real answer is lease your own rack in a colo
2023-03-30 15:06:52 <psykose> more seriously i don't really get the distinction
2023-03-30 15:07:03 <psykose> most vps's have various integrated stuff and multiple services
2023-03-30 15:07:14 <psykose> is a cloud only the ones with a 10B market cap :p
2023-03-30 15:07:18 <minimal> indeed, any distinction is basically a "marketing" or similar driven definition
2023-03-30 15:08:19 <minimal> so until now I didn't realise the Alpine Cloud team was only for targetting AWS, Azure, GCE, OCI, IBM, and Alibaba
2023-03-30 15:08:22 <mcrute> psykose: well targeting enterprise adoption I think is more the key IMO, the rest is just an indicator... if you really look at the platforms they have a lot of interesting low level infrastructure that hasn't historically existed in the VPS market (overlay networks, key management systems, non-local storage, etc)
2023-03-30 15:08:49 <psykose> 'targeting enterprise adoption' is the exact kind of terminology that makes the distinction very meaningless to be frank
2023-03-30 15:09:01 <psykose> that just means 'a cloud' is 'the company with a big marketing department and 10B cap'
2023-03-30 15:09:03 <mcrute> minimal: don't consider my opinions to be official team policy, I'm just telling you what my vision for the project was when I started it... things can and should change as the community goals change
2023-03-30 15:09:33 <mcrute> psykose: I still hold it's more of a distinction of what kind of features they build
2023-03-30 15:09:36 <minimal> ok, I wish that vision had been stated as it appears I've had completely the wrong impression of what the team was intending to do since it started
2023-03-30 15:09:43 <psykose> i get what you're going for by feeling
2023-03-30 15:09:53 <psykose> i just don't think any of us have actual terms for what the difference is
2023-03-30 15:10:00 <psykose> and nothing here really qualifies so far
2023-03-30 15:10:11 <psykose> or well, it does
2023-03-30 15:10:21 <psykose> just if that's the qualification...
2023-03-30 15:11:28 <mcrute> minimal: despite the debate of semantics here and the original goals of the team, I think there's a lot of good reasons to support VPSes and provide good first-class images for them directly from the Alpine project
2023-03-30 15:12:18 <mcrute> psykose: I don't know that it really matters either, I'm just a bit pedantic
2023-03-30 15:12:26 <psykose> aren't we all :)
2023-03-30 15:21:33 <minimal> so is the target of Alpine Cloud "enterprise" cloud providers or "enterprise" customers of those cloud providers? As I'd expect enterprises to stick to prehistoric versions of RHEL with its corporate support rather than use Alpine
2023-03-30 15:30:01 <mcrute> neither I guess. most entreprises you ancient RHEL and have a support contract. the goal, broadly, is to make it easy for people to adopt official Alpine images on cloud providers
2023-03-30 15:30:28 <mcrute> the initial target was just the “big clouds”
2023-03-30 15:30:49 <mcrute> *use
2023-03-30 15:33:30 <minimal> ok, but from what you've been saying the target (initial  or o
2023-03-30 15:33:56 <minimal> oops, or otherwise) is "clouds" which you've defined as a small number of specific big providers
2023-03-30 15:34:34 <minimal> so if you're not targetting enterprise end users then why are you targetting enterprise "cloud" providers only?
2023-03-31 23:26:38 <tomalok> i managed to find enough space to get all the new AWS AMIs published to all the subscribed regions (looks like we're missing melbourne, mcrute?)  there were some old edge images that somehow snuck through and were published -- edge images have an EOL of 1 day, btw.  we MIGHT have enough clearance for the next release, but would probably be good to
2023-03-31 23:26:38 <tomalok> get the limits across all regions bumped up to 500 public AMIs 
2023-03-31 23:26:45 <tomalok> ikke: https://gitlab.alpinelinux.org/alpine/infra/alpine-mksite/-/merge_requests/62