Gary: Friday release failure (0.7 stuff) five whys

Problem was that we were unable to make a fast release from trunk because of multiple qa issues in non-feature-flagged behavior.

  • Problems were not discovered sooner
  • Insufficient continuous QA procedure
  • The qa procedure in our review process is not sufficient
  • We don’t have a well defined or sufficient exploratory/aggressive QA step for branch reviews

Possible solutions:

  • Weekly exploratory testing must happen
  • We require better in-depth exploratory testing of branches

Another five-whys branch

  • Problems were not discovered sooner
  • Insufficient continuous QA procedure
  • landing things incrementally makes it difficult to identify when exploratory testing is appropriate

Somewhat related issues

  • We released on Friday

Proposal for solution:

  • weekly half hour to hour exploratory testing using both sandbox and improv (real environment nice to have).
  • bugs that block release are marked critical.
  • we release every other week.
  • Gary will not request another Friday release 😛