geohashim

admission-replay

Replays a Kubernetes cluster's audit log through a proposed admission policy before it goes live. It reports which legitimate writes the policy would have blocked, and which of the writes you know were bad it would have let through.

Turning on a ValidatingAdmissionPolicy is a bet on traffic nobody has looked at. The usual advice is to run it in audit mode for a few weeks, which only tells you about the weeks to come. The API server's audit log already holds the past month of writes. admission-replay sends every one of them through the policy, using the API server's own admission code, and scores the result.

View on GitHub Is the replay faithful?

Plays the six stages below in order, about 55 seconds. Scroll or press Esc to stop.

Language
Go 1.24 or newer
Tests
38 passing (30 tests plus 8 subtests)
Admission code
k8s.io/apiserver v0.34.1, as in Kubernetes 1.34
Reads
audit.k8s.io/v1 events, one per line or an EventList, gzip or not
Checked against
A real kube-apiserver on kind, write by write
Binary
Static, about 69 MB
License
MIT
admission-replay compare examples/audit.log examples/policies/*.yaml --bad examples/bad.txt examples/audit.log: 198 writes from 2026-09-01 to 2026-09-30 (29.2 days), 3 labelled bad policy bad caught legit blocked by controllers warned skipped 1 targeted 3/3 0 0 21 0 2 registry-allowlist 1/3 0 0 0 0 3 strict-baseline 1/3 26 0 0 0

Real runs of admission-replay at commit 46bfbfb, output verbatim. The month is the repository's example: the company, its people and the three bad writes are invented, but the audit log was written by a real kube-apiserver on kind while those requests were sent to it.

1Read the audit log, write by write

The input is what the API server's --audit-log-path already writes: one audit.k8s.io/v1 event per line. An EventList and gzip work too. Only completed creates, updates, patches and deletes are kept.

  • The user is the impersonated one when there is one. Here every request came through one admin credential acting as a named person, and the person is who a policy's matchConditions should see.
  • Writes are put back in the order they arrived. The log backend writes an event when a request completes, so concurrent requests can appear out of order.
  • A write the cluster refused at the time is still replayed, but it never counts as blocked: it changed nothing.
  • Subresources are skipped, and a log below Request level has no object to replay.

One event from the example log, and the write it becomes

    The month, as read

      The event is line 103 of examples/audit.log, abridged to the fields that matter: the request and response bodies are cut to the one container. The counts are the whole log's, from admission-replay's own --json output. The one refused write is a controller's, made while the search-experiment namespace was being deleted.

      2Rebuild each write as admission saw it

      Validating admission runs after mutation, so the object to judge is the one the server stored, the event's responseObject. An update or delete also needs the object as it was, and that is the previous write to the same object in the log.

      • Namespaces change as the log changes them, so a namespaceSelector sees each namespace's labels as they were at that moment.
      • An object created before the log starts has no old object. Its updates are skipped and counted, unless --initial supplies a dump of the cluster from when the log begins.
      • A namespace never created in the log is replayed with only the name label the API server gives every namespace, and the report says which ones.

      The log-shipper DaemonSet, then a log cut in half

        oldObject: the write of Sep 1

        
                
        object: bob's write of Sep 19
        
              
        
              
        The same 15 days, three ways Illustrative input
        
              
        
                
        Replaystrict-baseline blockstargeted blockstargeted warnsSkipped
        
              
        
              
        
            
        
            
        The three writes are lines 49, 142 and 143 of the example log, cut to the shipper container and its volumes. For the second part the log was cut to start on Sep 15, and initial-sep-15.json was built from the log's own stored objects as of that moment, standing in for a kubectl get -o yaml dump. The first row counts the full month's replay over the same 87 writes. Every number is real output.
        
          
        
        
        
      
        
    
        

    
        
      
        
        
        
          
        3Run the API server's own admission code
        
          
        The evaluation isn't a reimplementation. Each write goes through the ValidatingAdmissionPolicy plugin from k8s.io/apiserver v0.34.1, running against an in-memory fake API that holds the policies, the bindings and the namespaces seen so far. Matching, matchConditions, variables, messageExpression, failurePolicy and CEL's cost limits are all the server's code.
        
          
          
            
        • Every binding is loaded as [Audit], so the plugin reports every failed validation instead of stopping at the first denial. The binding's real actions, Deny, Warn or Audit, are applied afterwards.
          • 
            
        • check parses a policy file the way kubectl apply --validate=strict would, and compiles every CEL expression with the server's environment.
          • 
            
        • CEL's authorizer always answers yes: there is no RBAC in the replay.
          • 
          
        
        
        
        
        
          
        
            
        
              
        strict-baseline's five validations on five writes
        
              
            
        
            
        
              
        
              
        FailedPassedNot matched: the policy covers Deployments only
        
              
        Policies that are wrong before they run Illustrative input
        
              
        
              
        
            
        
            
        The grid is strict-baseline's real verdicts, from replay --json; a validation counts as passed when it isn't among the write's failures. The three broken policies are strict-baseline with one change each, made for this page: validations spelt validatons, startsWith spelt startswith, and c.image spelt c.imag. The last one compiles, so check passes it; only the replay shows what it would do under failurePolicy: Fail.
        
          
        
        
        
      
        
    
        

    
        
      
        
        
        
          
        4Score the month, policy against policy
        
          
        compare replays the log once per candidate and ranks them: bad writes caught first, then legitimate writes blocked, then warnings. replay takes one candidate and lists its verdicts write by write, grouped by user and by the validation that failed.
        
          
          
            
        • A write that fails two validations counts once under each in the "by failed validation" totals, and once as blocked.
          • 
            
        • Writes made by the cluster's own controllers are counted apart, as "by controllers": a policy that blocks them breaks the cluster, not a deploy.
          • 
            
        • --json gives every verdict, for anything further.
          • 
          
        
          
        strict-baseline reads as the safest of the three: trusted registry, pinned tags, limits, non-root, nothing privileged. Over the month it blocks 26 legitimate writes and catches one bad one. It matches Deployments, so the DaemonSet never reaches it, and it checks containers, so the init container doesn't either.
        
        
        
        
        
          
        
            
        
              
        198 writes, three candidates, September
        
              
            
        
            
        
              
        
              
        Blocked a legitimate writeWarnedCaught a bad writeLet a bad write throughAdmitted
        
              
        
            
        
            
        Each mark is one write, placed at the time it was made; hover a mark to see the write. The verdicts are admission-replay's --json output for each candidate, and the table is compare's, verbatim.
        
          
        
        
        
      
        
    
        

    
        
      
        Decided by peopleAbove this line admission-replay reads, rebuilds, evaluates and scores. Below it, a person says which writes should never have got in, and the team decides what fails the build.
        
    
        
    
  
        

  
        
    
        
      
        
        
        
          
        5A person labels the bad writes
        
          
        --bad takes a file of audit IDs, each with an optional note: the writes an incident review says should never have been admitted. The score is only as good as these labels.
        
          
          
            
        • An ID that isn't a write in the log is an error, exit 2, so a typo can't quietly count as missed.
          • 
            
        • An ID listed twice is an error too.
          • 
            
        • Without labels nothing is caught or missed. A policy only gets credit for what someone labelled, and nothing marks an unlabelled bad write as missed.
          • 
          
        
        
        
        
        
          
        
            
        
              
        examples/bad.txt, found in the log
        
              
            
        
            
        
              
          
              
          
            
          
            
          The three labels are the repository's. bad-typo.txt is the same file with two digits of one ID swapped, made for this page Illustrative input. Both runs are real.
          
          
          
        
          
      
          
    
          

    
          
      
          
        
          
          
          6Gate the policy change in CI
          
          
          On a pull request that changes a policy, run replay against a recent log. --fail-on-block fails when a legitimate write in the window would have been blocked, and --fail-on-miss when a labelled one would have got through.
          
          
            
            
          • Exit 0 when nothing tripped, 1 when a gate tripped, 2 for bad input.
            • 
            
          • targeted holds: it catches all three and blocks nothing. It denies privilege and writable host mounts in every workload kind and every container list, trusts known registries, and only warns about missing limits.
            • 
            
          • The repository's own CI checks the README against the CLI, and the replay against a real API server, on every push.
            • 
          
          
        
          
        
          
          
          
            
          
              
          Two candidates through the gate, and CI on the repository
          
              
            
          
            
          
            
          
              
          CI run 38611892810 on 46bfbfb4 jobs, all passed
          
              
            
            
            
            
            Both gate runs are real, with --show missed and --show caught to keep them short; the terminals show their first lines and exit codes. Job names, results and durations come from GitHub's API for that run, rounded to whole seconds.
            
          
            
        
            
      
            
    
            

    
            
      
            
        
            
          
            Measured, not assumed
            
          
            The claim the tool rests on, re-run here against a real API server, and what the replay costs. One Linux machine, shared with other work while these ran.
            
        
            
        
            
          
            
            
            A real API server agrees
            
            
            make crosscheck starts a kind cluster on Kubernetes 1.34 per candidate, binds it in audit mode, resends the month's writes under their original audit IDs, and compares the server's own audit log with the replay.
            
            
            
              
            strict-baseline27
            
              
            targeted24
            
              
            registry-allowlist1
            
            
            
            
            Writes that failed a validation on both sides, of 185 compared each. Disagreements: 0, 0 and 0. The other 13 writes are controllers' and can't be resent. The whole run took 6 min 47 s.
            
          
            
          
            
            
            A month in about two seconds
            
            
            Wall time for compare of all three candidates over the 198 writes, three runs:
            
            
            2.08 s2.01 s1.48 s
            
            
            Each candidate starts its own copy of the admission plugin and compiles its policies; the binary is static and about 69 MB, because it carries the API server's admission and CEL stack.
            
          
            
          
            
            
            Build once, then seconds
            
            
            From a fresh copy, go vet and go test ./... took 3 min 10 s, mostly fetching and compiling the Kubernetes libraries. Run again:
            
            
            3 min 10 s cold10 s warm
            
            
            The slowest package is the CLI's, 5.7 s: its tests replay the example month and compare every docs/*.txt file the README quotes with the CLI's real output.
            
          
            
        
            
      
            
    
            

    
            
      
            
        
            
          
            38 tests, and the README checked by one
            
          
            Counted with go test -count=1 -json ./...: 30 tests plus 8 subtests, all passing, none skipped, with Go 1.27.2. go vet and gofmt are clean, and go test -race ./... passes, as CI runs it.
            
        
            
        
            
          
            
            
            
              
          
              
          
              
            
              Aimed at what a replay could get wrong
              
            
                
              
              • Engine: deny and admit, every failure reported, Warn and Audit not denying, a compile error denying under failurePolicy: Fail, a runtime error following the failure policy, matchConditions seeing the user, and a namespaceSelector following label changes.
                • 
              
              • Replay: updates see the old object, deletes need state, failed writes don't change state, a terminating namespace stays known, and an unknown one is assumed.
                • 
              
              • Audit log: events and EventLists, gzip, Metadata-level events skipped, and malformed input refused.
                • 
              
              • Policy files: strict parsing with eight ways to be wrong, error lines that point at the right document, and every example compiling.
                • 
              
              • CLI: exit codes, --json, compile errors in check, and the README's quoted outputs regenerated and compared.
                • 
            
              
            
              CI runs vet, gofmt and the race-enabled tests on Go 1.24 and stable, checks that docs/ is current, and runs the crosscheck against kind on every push.
              
          
              
        
              
      
              
    
              

    
              
      
              
        
              
          
              
            
              Deliberately missing
              
            
              
              
              Webhooks and other admission
              Only ValidatingAdmissionPolicy is replayed. Gatekeeper, Kyverno, MutatingAdmissionPolicy and Pod Security Admission already shaped the stored objects; their own decisions aren't replayed.
              
              
              Parameters
              A policy with paramKind or paramRef is refused with that message, not evaluated without them.
              
              
              RBAC
              CEL's authorizer answers as it would for a cluster-admin, so a policy that branches on it is replayed as if everyone may do everything.
              
              
              Byte-exact objects
              The object is rebuilt from responseObject, so a policy that reads the new resourceVersion, or status a controller wrote in between, can differ from the cluster.
              
              
              A verdict on the policy
              It shows how a policy would have treated the past, scored against what someone labelled. It can't say a policy is good.
              
            
              
          
              
          
              
            
              Run it
              
            
              git clone https://github.com/ahmed-hashim-pro/admission-replay
cd admission-replay
make build          # a static binary, minutes cold
make test           # vet, then 38 tests

./admission-replay compare examples/audit.log \
  examples/policies/*.yaml --bad examples/bad.txt
./admission-replay check examples/policies/targeted.yaml

# Your cluster: a log at RequestResponse level
./admission-replay replay audit.log policy.yaml \
  --initial initial.yaml --bad bad.txt \
  --fail-on-block --fail-on-miss
              
            
              Go 1.24 or newer, or only Docker: the README shows the one-line build in golang:1.25. make crosscheck also needs kind and kubectl.