They say we shouldn't judge a book by its cover. I think the same goes for builds. You shouldn't judge a build by its error output. Sorry, that might have been a bad joke, but let me explain. When we run a build, one of the primary things we want to know from a build process is if it passed or failed. This is the feedback loop process that we need to understand if we need to go back and fix something in our code, or, if it's okay to proceed with the next step in our changes. However, as our code matures and our builds get increasingly more complicated, for example, more requirements, more complex build steps, larger amounts of parallel builds, you get the idea. We start to make it more likely that our builds are going to fail for reasons we can't immediately explain, and not all failures are going to be bad. That keeps us from moving our software assets forward. In this lesson, we'll cover some techniques for classifying build failures and describe some of the different types of build failures that might occur in our continuous integration environment. We'll also discuss some techniques for identifying and deciding on next steps for taking action on those build failures as well. Understanding the reasons for build failures is an important factor of knowing what our continuous integration system is trying to tell us in terms of the feedback loops. The feedback loops of a continuous integration system can help us make decisions about fixing our software, improving the reliability of our software, or understanding the overall system's reliability. Techniques for classifying failures can help us reduce overall friction and understanding next steps on what actions we should or shouldn't take to move the software forward to production environments. Well-run continuous integration systems that work in unity with continuous delivery methods will mean that every submission to our software configuration management environment is a potential shippable release. In order to take advantage of this capability for DevOps enabled teams, we must have quick methods to determine if the feedback loops for our continuous integration system are signaling, if potential shippable releases are actually shippable. Failure classifications can help us narrow down the actions that we take to resolve the feedback from the continuous integration system. We can start by creating three basic types of classifications. The first is consistent failures that would normally occur to code defects or test failures. The second is inconsistent failures that don't appear to be occurring due to software changes being made. The third is recognizable failures that are occurring due to external requirements or dependencies. Each of these classifications can help a team organize an action plan to get the right help or automate the right work that quickly resolves a failures and helps the team make decisions on the shippability of a software change. Let's go over how it might be possible to elevate and move forward decisions on each of these classifications. Most failures that will occur in our continuous integration environment will be classified as consistent failures. These are the types of failures that occur due to software defects such as a syntax issue, or issues with logic that go against our test cases that have been built for our software. These types of failures will be easy to reproduce in terms of reruns of the build and have some type of information that leads us to the location where the software is defective. In our continuous integration environment, we want to use exit status codes from the tools we're using to identify the failures such as compilers, linters, or test scripts. For example, an exit status other than zero from a compile command can be captured in a step from our continuous integration system to take some further action such as failing the build. The failed build will then provide the primary method to signal to the developer or owners of the software asset that the change made is not ready to be shipped. Let's look at our second classification. Inconsistent failures are what happens when the failures can't be reproduced. These types of failures become the basis for inconsistent failures in our software assets. These could occur for many reasons including when creating deterministic builds as we discussed in earlier lessons. For example, the software assets may have created some dependency or state for test cases that require a given execution order. However, the continuous integration environment is testing in a random order or might have introduced new parallel execution mechanics that introduce a new order. To capture these types of failures, we will need to record existing failures for a given run of the build in the current job. Then, when the failures are detected in the current job, a second step to rerun the tests can be performed in an automated way. If we are able to compare the test case results from the prior run and determine the differences and failures from the second run, then we will have a list of inconsistent failures. On large projects, dealing with inconsistent failures is important, since it may signal a need for maintenance of the test cases so that they don't suffer from code decay or lack of deterministic build capability. Having methods for dealing with inconsistent failures can mean the difference between understanding when a part of your software is still shippable or not. Future decisions on how to handle such failures such as ignoring or fixing the failure can also be taken. Not all projects will need to decide to deal with inconsistent failures right away however. If your project has large volumes and is key as a key bottleneck to shipping your assets, it may be important to take actions on inconsistent failures sooner than later. Our final type of failure classification is based on the types of failures that have been recorded in the past, which have received proper triage. We call these recognizable failures. Recognizable failures are typically known issues or known requirements in our software configuration management environment. Making pass or fail decisions based on recognizable failure types can help remove the need for our developers to repeat analysis which takes up valuable time to understand why the failure occurred a second time. Having a data source for identifying these types of failures during final build analysis can help us determine a list of actions that can be taken to resolve these types of failures for future software builds. We may take an action such as asking for an updated version of a vendor library, or escalating an issue for the continuous integration infrastructure maintainers. Or perhaps, simply ignoring and retrying the operation. Building a consistent way to capture, analyze, and decide on actions with recognizable failure types can greatly improve the efficiency in managing a software asset. Okay. You should now be familiar with the three basic types of build failure classifications, we can sort our failures into. Consistent failures will be the most common type of failure and will be the cause for most mistakes in our software assets. Inconsistent failures will signal to us that we might have non-deterministic causes for the failures in our builds. As software assets get more complex, we will need to develop techniques to identify in triage these types of failures with our development team. Finally, we learned about recognizable failures and the importance of recording actions we took to triage those types of failures in the past, so that we could easily take actions on remediating those types of failures on future builds. Thinking carefully about how you capture and report on failures for your continuous integration environment can have a large impact on managing your software asset. The methods you use to triage and manage issues that come up with your software assets can determine the speed and throughput for which you can ship your changes. Taking steps towards creating failure classification capabilities in your continuous integration environment can greatly improve throughput and productivity that your continuous integration environment is the luring for your team and product.
