Testing code paths

“The only way to know that a test actually works is when it fails when you make a code change. — Simon de Lang

There have been some really thoughtful questions that were raised, both while writing tests, as well as reading up on the best practises and what to make of test metrics like code coverage.

step 1 : Code Coverage

which parts of your code are TOTALLY UNTESTED ?

Because we are currently in the era of React, therefore the test runner + code coverage tool I constantly interact with are Jest and Istanbul (which Jest uses)

Jest - test runner, with helpers to describe your tests Istanbul - code coverage calculator

How does Istanbul work?

it might be a good idea to try writing my own

step 2 : Mutation Testing

HOW GOOD are your tests that you have written at catching bugs ?

This cool idea (probably quite old, because it makes so much sense to do) of mutation testing (e.g. Jumble, or any of the others) is about mutating your tests to see if they still pass. If they do pass, it means your tests have been kinda shitty (because a test that passes ALL THE TIME doesn't catch anything)

there's a paper on mutation testing with javascript

step 2.5 : help the developer find out which code path is more important to test than others

  1. place trackers within each function to be tested
  2. keep track of what params are called, and how many times
  3. aggregate them together and generate a chart showing which:
    • functions are called the most and
    • which functions have the highest volatility (params change often)
  4. the one which are both called the most and have the highest number of changing parameters are most critical to cover with good tests
  5. use mutation testing to improve your tests (i guess)

step 3 : and the seed of an idea that I can't seem to get out of my head

therefore it is in a blogpost, with hopes that it will germinate and turn into something bigger down the line, if not by me, then by a reader of this blogpost.

what this idea wants to solve

how it might work

  1. run actual code (not tests)
  2. introspect all parameters
  3. create params (either manually, assisted by specifying ranges, or a normal distribution depending on param type)
  4. generate basic tests for all codepaths
  5. run mutation testing on those tests
  6. provide mutation result as feedback to the generator
  7. ??? somehow generate better tests that create less mutants
    • perhaps by tweaking conditionals?
    • or other means yet unconsidered?

in essence, think about framing the question of testing code as a generator-discriminator pair, like a Generative Adversarial Network, where tests are generated and pitted against the mutator, with the KPI being generating tests that the mutation tester cannot create mutants for.

ref

mutation testing

code coverage tools