How you upgrade a fleet
You do not run the upgrade. You change a number in Git, push, and the cluster replaces its own nodes one at a time. The interesting questions are what the unit of replacement is, and what happens when the rollout goes wrong.
Both answers depend on which provider you picked, and assuming the unit is the same everywhere is the fastest way to get burned.
The unit is provider dependent
Cluster API's abstraction is consistent. You change a version in a manifest, push to Git, and the provider reconciles. The unit of replacement is not.
With kubeadm, the version lives on the KubeadmControlPlane for the
control plane and on the MachineDeployment for workers. An upgrade is
immutable replacement by default. Bump the version and a new MachineSet appears,
which provisions new Machines, new instances, kubeadm joins them to the cluster, and
the old ones get drained and deleted one at a time. You don't run an in place upgrade
on a live node in the normal flow. The old node isn't upgraded, it's replaced. (Worth
knowing CAPI has an experimental in place update path behind the
InPlaceUpdates feature gate, so the absolute is immutable by default,
in place is an experiment.)
With EKS managed through CAPA, the control plane is AWS's. You set the version on
AWSManagedControlPlane and AWS upgrades it for you, no Machines, no
KCP, nothing you replace. Worker nodes roll through the managed node group
semantics. There is no kubeadm join involved because there's no kubeadm.
You can also run hybrid, an EKS control plane with self managed MachineDeployments for nodes, where the control plane is AWS's and the nodes are CAPI Machines. The point is the unit of replacement is provider defined, and the answer to "how do you upgrade" starts with "which provider."
Why it's a Git commit, not a sequence
The order of operations is not a runbook. It's the rollout strategy on the MachineDeployment. With kubeadm, the safe sequence is control plane first, then workers, but the "what order" answer is whatever the strategy says. The audit trail is the commit, and rollback is a revert. Nothing is ever upgraded by hand twice, because the state is in Git and the reconciler owns the how.
The rollout is designed to halt
This is the part that saves you. The MachineDeployment has a rolling strategy with
maxSurge, how many new Machines above desired, and
maxUnavailable, how many old Machines may be down while rolling.
Cluster API's default is maxUnavailable: 0, which means it will not
delete an old Ready Machine until the new one is Ready.
So when the new Machine fails to join, it never becomes Ready, and the rollout
halts in place. It doesn't churn, it doesn't delete the old nodes, the old nodes
keep serving. You get a stuck rollout instead of an outage. The Machine never gets a
NodeRef. It stays in the Provisioned phase and its NodeHealthy
condition goes False with a reason like NodeNotFound or
MachineWithoutNode. On the control plane side, watch the KCP's
Resized and MachinesSpecUpToDate conditions. The kubelet
and kubeadm logs on the instance are the breadcrumbs.
Worth being precise about the states, because the words confuse people. Provisioned means the infrastructure provider created the instance. It does not mean the kubelet is up, joined the cluster, and reported Ready. Joined is a separate gate, and a Machine can be provisioned for a long time while failing to join. That gap is where this failure lives.
The backstop that can bite you
MachineHealthCheck watches Machine conditions and remediates. A Machine with an unhealthy kubelet past the threshold gets deleted, and the MachineSet provisions a replacement. That's an auto healer, bad Machine in, new Machine out.
It's also the thing that can turn a fleet wide outage into a fleet wide rebuild
if you misconfigure it. Two knobs matter. maxUnhealthy caps remediation
when many Machines are unhealthy at once, so the controller doesn't start deleting
its way through a real incident. nodeStartupTimeout stops it from
killing Machines that are just slow to join. Self healing without those guardrails
is self destruction.
What keeps the other 49 clusters safe
Three layers, and they stack.
Environment promotion first. The version bump lands in the dev overlay, and only
moves to staging, then a canary, then the fleet, if health checks pass. One honest
nuance. Flux's healthChecks gate a Kustomization's Ready condition, but
plain Flux doesn't move a version between environments by itself. The chain is a
repo and PR process, or tooling like Weave GitOps Pipelines and Flux's promotion
support. The principle holds either way. The bad version stays gated in dev until
something proves it's safe to promote.
maxUnavailable: 0 second. Even the cluster that got the bad version
keeps running, because the rollout halts before any healthy node is replaced.
Pause third. If something is mid rollout and needs to freeze, pause the MachineDeployment or the Cluster and diagnose while everything stays up. Then the rollback is a revert commit and Flux reconciles back. One nuance for the control plane. KCP versions must move sequentially, so a mid rollout revert triggers another rolling upgrade rather than undoing cleanly, but a revert still gets you back where you were.
The boring conclusion
An upgrade is a desired state change, the unit of replacement is provider defined, and the rollout is built to halt before it hurts you. The discipline is the boring part. Promote through environments, respect the strategy, and know that a revert is the rollback. The fleet upgrades itself. You just have to stop it from upgrading badly.
What's the upgrade that looked fine in dev and fell over in prod?