Rebuilding GKE without reissuing certificates

Preserve stateInventory TLS and reserved IPs

Rebuild & restoreRestore TLS before Certificates

Prove recoveryCheck certificates and GitOps

The rehearsal adopted 49 of 49 certificates without reissuing. It is a recovery test, not a production recovery-time SLA.

Contribution at a glance

My responsibility

Rebuild rehearsal, bootstrap/teardown repairs and certificate and GitOps recovery checks.

Scope

One full rehearsal followed by reuse for three environment rebuilds during network renumbering.

Collaboration & boundaries

I owned the infrastructure procedure and validation; application recovery was checked through GitOps state.

Outcome

The rehearsal adopted 49/49 TLS certificates with zero reissues and restored 35 ArgoCD applications.

The problem

Overlapping network ranges prevented private connectivity between a newer platform and legacy services. Some GKE address ranges are immutable, so the change required rebuilding clusters. A rebuild that silently reissued every TLS certificate could exhaust a shared weekly issuance quota before the platform was healthy.

My contribution

I ran a destructive rebuild rehearsal in an isolated environment, corrected failures in the bootstrap and teardown scripts, and verified certificate adoption, reserved load-balancer addresses and GitOps recovery. I then used that process for three environment rebuilds as part of network renumbering.

The order of recovery matters

I restored TLS Secrets before recreating Certificate resources. That let cert-manager adopt existing certificates instead of requesting replacements. Reserved external addresses were preserved independently of the old subnet, so replacing the cluster did not require changing the associated DNS records.

Running the process exposed defects that a static review had missed: a guard rejected state created by the preceding step, missing resource kinds broke otherwise idempotent deletion, and removing External Secrets Operator before its finalizers completed could deadlock teardown. I fixed the lifecycle ordering and checked the resulting cluster state.

Results and measurement

The rehearsal reported 49 certificates found, 49 adopted, zero reissued and none left unready. They became Ready in about 20 seconds; that is certificate readiness time, not the duration of the complete rebuild. The certificate issuance quota remained at the same value before and after. The node identity changed, confirming this was a new cluster rather than a restart of the old one.

The method subsequently supported three cluster rebuilds during network renumbering without consuming additional issuance quota. Private peering was then activated and checked with a before/after connectivity test. This does not establish a production RTO or a company-wide disaster recovery guarantee.

What I would repeat

A recovery plan becomes credible when its destructive steps have actually run. Preserve the resources that carry identity, exercise restore order, and validate the recovered applications before calling the platform rebuilt.

Related work: the Terraform and Shared VPC foundation and staged workload migration.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

Reissuing certificates is simpler than restoring them, but risks issuance quotas. Restoring TLS state first lets the controller adopt existing identity.

Trade-off

A rehearsal proves the tested recovery path; it does not establish a production recovery-time SLA.

What I would improve next

Next improvement: schedule repeat rehearsals, record recovery times and test recovery when one preserved dependency is missing.