Ephemeral Container Terminated Before Evidence Capture

The compromised container was terminated, evicted, or replaced (pod restart, autoscaling event, deployment rollout) before forensic evidence could be collected. The container filesystem and local state are gone; response must pivot to control-plane audit, node-level artifacts, and image-registry provenance.

Signals

  • •Kubernetes `kubectl get events` shows pod termination or eviction during the compromise window
  • •Container runtime logs reference the compromised container ID but the container is no longer listed in `docker ps` / `crictl ps`
  • •Deployment controller restarted the workload with a new pod identity between detection and response

Pivot Actions

  1. 1.Pull Kubernetes API audit logs (`--audit-log-path` or managed equivalent) covering the compromise window for the affected namespace/workload
  2. 2.Pull container runtime logs from the node (containerd events, Docker daemon logs, CRI-O journal) that survived the container termination
  3. 3.Pull the deployed image digest and correlate with the registry audit log to identify who pushed the image and when
  4. 4.For CRI-O/containerd nodes, check `/var/lib/containers/storage/overlay/` for residual layer data not yet garbage-collected
  5. 5.Query the CNI provider (Calico, Cilium, AWS VPC CNI) for flow records attributable to the container IP during its lifetime

Alternate Evidence Sources

  • •Kubernetes API server audit logs (kube-apiserver --audit-log-*)
  • •Container runtime logs on the node (journalctl -u containerd, docker daemon logs)
  • •Image registry audit logs (ECR, GCR, ACR, Harbor) for push/pull history
  • •CNI flow records from Calico, Cilium, or cloud VPC flow logs tagged to the container IP
  • •eBPF-based runtime security telemetry (Falco, Tracee, Cilium Tetragon) that logged container activity