Decision room · DevOps
Release a deliberately broken container version, observe safe rollout signals and recover without guessing.
Start with the business or technical outcome
A rollback command is easy to memorise; knowing when and why to use it requires observable evidence. A confident choice requires more than a feature list: it requires constraints, evidence and a clear trade-off. A web Deployment has healthy version one, but version two fails its readiness condition and should not receive normal traffic.
Use a local or disposable training cluster. Keep the service stateless and the failure harmless. Define success before updating: ready replicas, request behavior and a bounded observation period.
RollingUpdate settings, probe behavior and application startup time interact. A probe that is too strict can make healthy software look broken; a shallow probe can route traffic to a nonfunctional pod.
What to understand before opening the tool
Understand Deployment desired state, ReplicaSets, rollout strategy, image tags or digests, readiness versus liveness versus startup probes, events and logs. A rollout may pause or fail even though some old replicas still serve traffic.
Rollback restores an earlier pod template revision; it does not reverse database migrations or external side effects automatically. State what the lab excludes.
kubectl rollout
Use it for: observe status, history, pause or undo a Deployment Keep as evidence: command transcript
Deployment Manifest
Use it for: define image, strategy, resources and probes as code Keep as evidence: versioned YAML
Events and Logs
Use it for: separate scheduling, image, probe and application problems Keep as evidence: timestamped evidence
Service Test
Use it for: confirm whether ready endpoints serve expected responses Keep as evidence: request samples before and after
Use a versioned manifest, controlled failed rollout, events and probe evidence, successful rollback and short incident note as the decision test. Define the constraints before comparing options, then explain which factor carried the most weight. A sound recommendation can be conditional: one option may fit a small learning environment while another suits a regulated or high-volume workload.
Work through the decision sequence
Establish a healthy baseline, introduce one deliberate defect and recover based on signals.
- Deploy the baselineApply version one, wait for availability and record ReplicaSet, pod and response state.Checkpoint: Healthy baseline transcript.
- Add meaningful probesConfigure paths and timings suited to the simple application, then verify endpoints.Checkpoint: Probe specification and success events.
- Release the faulty imageChange one version reference that causes readiness failure without destructive side effects.Checkpoint: Manifest diff and rollout start time.
- Observe before actingCheck rollout status, pods, ReplicaSets, endpoints, events and logs in a consistent order.Checkpoint: Evidence identifying the failing condition.
- Undo the revisionRollback to the known-good pod template and wait for the Deployment to stabilise.Checkpoint: Revision history and restored response.
- Write the learning noteRecord impact, signal, cause, recovery, limits and a prevention change.Checkpoint: Concise incident-style report.
Avoid scoring every criterion equally. Security, correctness and recoverability may be non-negotiable; convenience and speed can then be evaluated inside those boundaries.
Options worth comparing before you act
Use observable signals to avoid random restarts and premature conclusions.
| Decision or signal | Action to take | Evidence to retain |
|---|---|---|
| Pending pod | Describe pod; inspect scheduling and resource events | Scheduler event and node capacity context |
| Image pull failure | Inspect image reference and registry access | Exact event without credentials |
| Readiness failure | Test readiness endpoint and probe timing | Probe event plus container response |
| CrashLoopBackOff | Review previous container logs and exit reason | Exit code and last logs |
| Rollout stalls | Compare desired, updated, available and unavailable replicas | Rollout status and ReplicaSet state |
Trade-offs hidden by a quick answer
Random commands can erase evidence or make a transient symptom look fixed.
- Using latest image tags: A mutable tag weakens repeatability and makes revision identity unclear.
- Restarting pods before observing: Capture events and logs while the failure is present.
- Making liveness too aggressive: Repeated restarts can prevent a slow but valid startup; use probe types deliberately.
- Assuming rollback reverses data changes: Coordinate schema and external side effects separately.
- Ignoring capacity during surge: Rolling strategy can require temporary resources; test within cluster limits.
Turn the exercise into credible portfolio evidence
Store the small manifests, application version marker and a redacted command transcript in a repository. Add a timeline from deployment to detection and recovery.
Include a diagram of Deployment, ReplicaSets, pods, Service and probes. The key story is not “I ran undo”; it is “these signals justified undo and confirmed recovery.”
Explain it clearly in an interview
Explain the distinction between readiness and liveness, how a RollingUpdate changes ReplicaSets, which evidence you inspect first and why rollback is not a universal recovery for stateful changes.
Peer review before calling the work complete
Ask another learner to inspect the result without watching you build it. Give them the original scenario—a web Deployment has healthy version one, but version two fails its readiness condition and should not receive normal traffic.—and the evidence pack, but not your intended conclusion. They should be able to trace the input, identify the main decision and locate the proof of the output. If they cannot, improve the labels, timestamps or explanation instead of adding decorative screenshots.
Use this acceptance condition during the review: The baseline and failed revision are identifiable, evidence supports the diagnosis, traffic returns after rollback and the report names what rollback did not restore. Record one question the reviewer raised and the change you made in response. That small feedback loop makes the Kubernetes deployment rollback lab exercise more credible, easier to maintain and easier to explain under interview questioning.
Questions learners ask
What does kubectl rollout undo restore?
It restores a previous Deployment pod-template revision; external data or manually changed dependencies need separate recovery.
Why can old pods keep serving during a failed rollout?
RollingUpdate may retain available old replicas while new replicas fail to become ready, depending on strategy and state.
Should liveness and readiness use the same check?
Not automatically. They answer different questions and should reflect application behavior.
What should be saved for troubleshooting?
Manifest diff, rollout status, pod state, relevant events, logs and request evidence with timestamps.
Use current product guidance
Menus, fields, permissions and service behavior can change between product versions or tenant configurations. Check the Kubernetes Deployment documentation before applying version-sensitive steps in a live environment.
Build the complete skill path
Practise version control, CI/CD, containers, Kubernetes, infrastructure automation, observability and incident learning as one delivery system.
Final perspective
The real value of Kubernetes deployment rollback lab is the ability to complete a controlled task and defend the result with evidence. A learner who can show the input, explain the decision, verify the output and describe one realistic exception demonstrates far more than someone who has only memorised a menu path or definition.