Case study · ditto← Back to the portfolio

ditto

Mutation testing cheap enough to run while you are still writing the code.

Stage a change and ditto breaks it on purpose, one mutant at a time, then tells you which breaks your tests let through. It pays only for the lines you touched.

Get startedSee how to read this verdict ↓
~/projects/shopTrimmed
$ ditto staged --threshold 0.8┃ price/price.go — 4 mutants… the mutant list, the baseline and the survivors summary, elided …┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅┃ 🧬 Mutant survived: price/price.go:10:16 → Comparison┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┃  func IsFree(price int) bool {┃ -    return price <= 0┃ +    return price < 0┃  }┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅… the diff header and two more survivors, elided …┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓┃ • Total:        4                    ┃┃ • Killed:       1                    ┃┃ • Survived:     3                    ┃┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┨┃ ⨯ Score:     0.25 (minimum: 0.80)    ┃┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ditto: the mutation score is below the configured minimum of 0.80
Real output from ditto v0.10.0: IsFree returns price <= 0 and its test checks -1 and 5, so three boundary mutants survive and the run exits 1. Marked blocks were elided.
The verdict

A score you can act on, or a refusal that says why

If the suite were already red, every mutant would look killed and the score would be perfect. This is what stands between that number and you.

Reported kills, by reason50 of 78 mutants reported killed
Credits your suite
Assertion
Your suite gets no credit22%
Build failureDeadlineUnknown

  • The suite is proved green before anything is scored

    A baseline run of your command on unmutated code opens every release. If it is red, the run stops there and shows you the command's own output.

  • Every kill arrives with the reason it died

    Assertion, build failure, deadline or unknown. Only an assertion credits your suite, as the bar above shows.

  • The score is taken over the mutants that compiled

    A mutant that never became a program leaves the score entirely, so what you read is the share of runnable mutants your tests caught.

  • Every survivor arrives with an address

    path:line:column → Operator (what it replaced), listed before any diff, so you can jump straight to it.

What a run costs

Fast enough to run on every change

Every mutant reruns your test command, so ditto cuts how many times it runs. The same twelve mutants come back in 0.4 to 1.3 seconds instead of 10 or 11.

  • ditto staged

    Pay for the lines the change actually touched

    484test runs
    8% of the original bill

    It reads the changed ranges from the git index, so a function you changed costs only the operators that fire on its lines.

  • --gated

    One compilation carries every mutant in the file

    13538command invocations
    28% of the original bill

    Each file compiles once and all of its mutants run from that build. Where the two paths disagree, the gated one is right.

The operators

Fifteen ways to break your code on purpose

Each one edits the syntax tree the way a real mistake would: an operator flipped, a constant nudged, a loop cut short. Fourteen are on before you configure anything, from the command or from `Release`; the fifteenth is added by naming it.

  • Arithmetic
    • +↔-
    • *↔/
    • %↔*
  • Arithmetic Assignment
    • +=-=*=/=%=&=|=^=<<=>>=&^=→=
  • Arithmetic Assignment Invert
    • +=↔-=
    • *=↔/=
    • %=↔*=
  • Bitwise
    • &↔|
    • ^→&
    • &^→&
    • <<↔>>
  • Comparison
    • <↔<=
    • >↔>=
  • Comparison Invert
    • >↔<=
    • <↔>=
    • ==↔!=
  • Comparison Replace
    • &&operand→true
    • ||operand→false
  • Float Decrement
    • x→x-1.0
  • Float Increment
    • x→x+1.0
  • Integer Decrement
    • n→n-1
  • Integer Increment
    • n→n+1
  • Loop Break
    • break↔continue
  • Loop Condition
    • condition→false
  • Range Break
    • range→earlybreak
  • Cancel NilOpt in
    • context.CancelCauseFunc(err)→(nil)

Write one for your own domain

A virus is any struct satisfying the `viruses.Virus` interface, so a mutation that only means something in your codebase is a type and a call to `WithViruses`. The `dittotesting` package carries the helpers the shipped fifteen are tested with, so yours gets the same treatment.

Add it

From install to a score you trust, in four runs

Install the command and walk through a real run, from a dry check to a passing score. Wiring it into a test binary you already run takes the library path below instead.

  1. 01

    Install the binary

    go install github.com/Disble/ditto/cmd/ditto@latest

    One binary on your PATH, ready for ditto staged from a repository root.

  2. 02

    See what it would mutate

    ditto staged --dry

    It lists the staged ranges it would judge and runs nothing, so you know the cost first.

    ~/projects/shopTrimmed
    $ ditto staged --dryditto: 1 staged file(s) under ~/projects/shop  price/price.go: 150-251
  3. 03

    Mutate it with a minimum

    ditto staged --threshold 0.8

    Every survivor comes back with its line and the break it made, the same three shown at the top of the page, and a score under the minimum exits 1.

  4. 04

    Cover the boundary and run it again

    ditto staged --threshold 0.8

    Two assertions, for 0 and 1, kill the three survivors, and the run exits 0.

    ~/projects/shopTrimmed
    $ ditto staged --threshold 0.8… progress, elided …┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓┃ • Total:        4                    ┃┃ • Killed:       4                    ┃┃ • Survived:     0                    ┃┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┨┃ ✓ Score:     1.00 (minimum: 0.80)    ┃┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Every subcommand, as -h prints it
ditto -hCaptured output
ditto — mutation testing for Go   ditto run [flags]       mutate a repository and report what survived  ditto staged [flags]    mutate only what a staged change justifies  ditto changed [flags]   mutate only what a committed change justifies  ditto version           the module version this binary was built from                          (also -v, --version) Run `ditto run -h` for its flags.
From a test binary

One import and a build tag put the same engine inside the suite you already run, with `WithTestCommand` to name a different test command.

go get github.com/Disble/ditto
Then run
go test -v -tags=mutation
mutation_test.go
//go:build mutation package main_test import (	"testing" 	"github.com/Disble/ditto") func TestMutation(t *testing.T) {	ditto.Release(t)}
Open the repositorypkg.go.dev/Disble/ditto

ditto is a fork of gtramontina/ooze by Guilherme J. Tramontina. All the good ideas here are his, and the licence and copyright stay with him; it is MIT, as the original is.

You may also like