This has become my favorite topic of late, and it is becoming more and more evident to me that less people know about this set of disciplines than should. It seems that a large number of IT people claim to be experts in some area of IT after just 1 or 2 years of experience. Let me just state that I do not claim to be an expert in any area of IT, even after being performance testing for 23 years. I learn something new each day and I often learn from other people the most. There are no experts, just people with varying degrees of knowledge, who decide to specialize in a particular area or discipline (Specialists).
I like what Scott Barber (Perfplus) says about IT opinions, which is that no one person is really correct and that we should use other peoples opinions constructively. Even when someone is obviously wrong you can learn from this experience as well, like thinking to yourself “why does this person think this way?”. Anyhow with all that out of the way let me start giving you my opinions on Software Performance Management, and this set of opinions is based on my many years of learning. I would suggest if you agree somewhat with what I say that you join the linkedin group Software Performance Management Specialists as I am growing a field of people in the industry that I respect and can bounce ideas and opinions off. Please leave comments if you have some.
Three Pillars of Software Performance Management
Everyone talks a lot about performance testing but very few talk about software performance management. Performance testing is just one of the 3 pillars, that are vital in the very complex world of deploying software solutions. Some companies are using this diagram and I think this is very encouraging. It highlights in a diagram what would take many words to explain.
Software performance management must have a complete solution of al 3 disciplines, Performance Testing, Predictive monitoring and Capacity planning. At the core and heart of all these 3 disciplines are some very skillful analysis techniques. Absolute none of these disciplines are easy IF they are done correctly.
I could talk for ages on each of these topics but I will try and be as brief as possible as you can find out more on the Internet about each of these topics yourselves. My goal here really is to highlight that you MUST employ all 3 of these disciplines to achieve an ongoing robust software solution.
Lets begin with Performance Testing. How many performance testing exercises are being conducted correctly? I would have to say without a doubt very few. I see and hear everyday people writing load simulation scripts that are not reflective of realistic business or workload usage of the application or system under test. The test environments are not reflective of target production environment or the differences in environments are not understood and factored in. Metrics collected or not collected throughout the test are not analyzed or interpreted correctly. Tests are not conducted often enough and results not summarized accurately and explained to the business.
The positive side to this though is at least some performance testing is better than none. Yes, and we have all found serious performance problems and saved the embarrassment and monetary disasters that could have happened, had we done none. The point is that we can all improve in this area. Let me just state my view on this and then encourage you to research more in each and every area associated with this.
Performance testing firstly must be planned accurately with as much business input as possible. Before any performance test is executed. you must meet with the business and talk to them. After all, it is their interests that you are looking after and in most cases they are paying your bill. Meet with them and ask them many questions about what they are expecting you to achieve. From these meetings you need 2 core outcomes. A workflow usage pattern and some performance test objectives, which may come in the form of Non functional requirements or service level agreements. Documents may exist for these however you still need to meet with the business. Once you have the workflow usage pattern or an application simulation model, you can then commence building scripts.
Once you build your load simulation scripts (lets not talk about tools here, I am anticipating writing a blog on a complete set of tools that will support the software performance management practice), keep the test objectives and SLAs in the forefront of your mind. This will ensure that you not only build scripts with the corect transaction timers, counters and error checking, but your tests meet the performance and business objectives.
Now you can execute your scripts correctly. The next part of this equation is ensuring that you collect server and application metrics storing these in a database of some form. In fact you should continue to collect these metrics constantly before and after tests. I will not go through all the metrics you need as you need many, let me just say the more metrics you collect the better your analysis will be. You will need to ensure though if you are collecting many metrics that you have an automated process of picking up those metrics that have significant enough variation to cause concern. Again this is a big topic that you need to research constantly.
The best place to performance test is in production in a controlled manner but if you cant do this for whatever reason, then make sure you understand your test environment and the differences that will impact the results and analysis of your tests.
The last thing I would like to say abut performance testing is that you must continue to performance test every subsequent software release, where a technical impact analysis shows there is a risk to performance of the application. Remember only one line of code or one configuration parameter can destroy the performance of your application.
Now, lets talk about predictive monitoring. Firstly when we talk about predictive monitoring we are talking about pro-active motoring. Monitoring that will tell us our application is nearing a performance issue. Predictive analysis can be achieved by collecting application and server metrics continuously. Thresholds should then be assigned for each metric, which will alert us that we are approaching a problem. This in itself is no easy task IF done correctly. Do your homework here and get an appropriate tool and by the way you don’t need to spend thousands of dollars on a tool to do this. This predictive monitoring obviously needs to be implemented in your production environment as well as test environments. The results need to be examined by your capacity planning, infrastructure and performance specialists. Read about what metrics you should collect, aside from the obvious. This is also dependent on your application and supporting application infrastructure.
Lets move to capacity planning. Capacity planning is the discipline of collecting and analyzing application, server and business metrics, to assist with ensuring your application will scale without issue. There are modeling algorithms and predictive analysis techniques that can be performed with sufficient historical data. Capacity planning is a skill that should not be under estimated. A good capacity planner has an excellent understanding of business projections and impeccable analysis techniques. Please read and research this area a bit more if required.
The heart of these 3 pillars as mentioned is analysis. Performance test metrics, predictive analysis metrics, capacity-planning models all need various analysis techniques. Just make sure that depending on the area you are specializing in that you also understand what analysis you should perform.
This is all I want to say about software performance management, and encourage everyone to learn more. To reiterate I just want you to know of the 3 pillars and the heart, to then study and learn: –
1. Performance testing
2. Predictive Monitoring
3. Capacity Planning
The heart is analysis.