Robust Automated Functional and Regression Testing
Having an automated set of tests that exercise system functionality is a critical safety net that helps to raise the bar for quality. Plenty of testing tools exist, but the key is that such tests run automatically in the build pipeline, in a manner similar to that of unit tests and integration tests.
For large systems, one difference is that, even when automated, only a small subset of these tests should be in the test suite that runs every time a developer checks in code. Even when automated, functional tests (a broad category that tends to cover regression tests as well as tests for currently developed functionality) run far more slowly than unit tests do. If every build requires 30 minutes of tests, eventually regressions will not be run on every build. But such tests should be run at least once per day, if not multiple times per daywith developers noticing and reprioritizing to any failing tests.