Considerations to the importance of proper testing prior to a production deployment.
Today I will talk about the importance of proper and thorough testing prior to deploying an application, patch, or even a configuration change to a production system.
It is very easy for the IT department to be caught up in a false sense of urgency when it comes to the production release of a piece of software, an update, or a configuration change. The user base, management, or both, may be applying pressure to have whatever it may be, delivered for use. Proper and thorough testing takes time, and as everyone knows, Time is Money. Budgetary limitations and scheduling deadlines may also factor into the pressure to “Hurry up and get it tested so we can go live!” This pressure may result in rushed and less than adequate testing, or in some cases, no testing at all! This is the epitome of “False Economy”, as a lack of testing almost guarantees problems like lost time, bad data, system outages, and ultimately unnecessary and avoidable expense.
Application developers are amazingly intelligent people who are able to think through complex processes and procedures, and wrap their minds around mathematical calculations that could fill a black board. However, even the most thorough and careful coder makes mistakes now and then that may not show up when they run their applications interactively on their development systems. The developer’s systems may have APIs, references, or other components installed that are required for their applications, but if any one of those components are missed when it comes to preparing the deployment package, problems are very likely to occur. This is where the need for testing outside of the developer’s main working environment becomes imperative.
A test environment should mimic its production environment as closely as possible in both hardware and software. Production level server hardware can be pricey, so it may be unrealistic for the test system hardware to mirror the production hardware exactly, but in most cases a reasonable simulation of a production load can be run on a test server that has even significantly less resources available. Care should be given to keep the environments in sync with regards to 3rd party component updates, patches, and OS updates as even the slightest discrepancy could make the difference between an operational system with a successful deployment, and production outages with unwanted negative results.
One of the most important aspects of software testing is the person or team doing the testing. If they are not intimately familiar with the use and functionality – and the expected results of what they are testing, they may not be able to identify or recognize problems. Careful and organized testing processes should be planned and identified, as well as documented, in order to ensure that consistent and repeatable testing is possible. This will allow for easy replication of particular scenarios that may have caused or resulted from bugs in previous versions.
Testing complex software, applications, or configuration can be very time consuming, tedious, and repetitious. Hours of testing could simply need to be redone should one minor problem arise that causes the developer to make code changes. Reporting bugs and problems to developers in a way they can resolve can be a challenge as well, especially in situations where a developer is unable to replicate the issue. This is where organized and documented testing procedures must be adhered to, as repeatability and replicating is how a development team identifies and resolves issue.
A thorough testing process can also be a great way to further the understanding of the software or process, improve documentation, and even lead to improvements for the product or process being tested. Proper testing allows for a technical person or team to interact with the software, use different combinations and options, test production scenarios, and most importantly – figure out how to break it!