Performance Tuning and The Tedious Repeated Test Run

One of the things I do from time to time is tune the performance of applications (ranging from the very large and complex to the very small). Sometimes I work solo on these types of projects, and other times I work with others. There are 3 aspects to this exercise the bear close examination, so that you minimize the time wasted chasing down deadends and false paths, and being able to know with a high degree of confidence what does and does not work. The main elements that I focus on are:

  1. Only change one variable at a time
  2. Always reset the system to a known, good state, and to the same state, on each test run
  3. Have the same results in multiple test runs with the same settings and starting state

Change One Variable At A Time

One critical aspect in performance tuning is adhering to the good, scientific principle of only changing a single variable at a time. I have watched others try “tuning”, fiddling with multiple parameters and configuration settings and then trying to figure out why it got faster, why it didn’t matter, or why things got worse. I, too, have been guilty of changing multiple variables, usually in a bid to hurry things up, and then confusing myself as to what worked and what didn’t. Trying to go faster usually ended with the process actually taking longer while dead-ends are explored to futility because of unexpected variable interaction.

Only changing one variable at a time is tedious, and depending on how long it takes to do some kind of test run, it can be time-consuming. But without taking this methodical approach to tuning, you may never really know which setting actually helped (or hurt) and which had no effect. You also won’t know with any degree of confidence if it took both variables to get the desired result. By changing one thing at a time, and leaving the rest of the system and configuration untouched, you can have a greater degree of confidence that the setting has the desired effect (or you know what the negative result will likely be).

Reset To Known Good State Each Time

It is very important that you can get your system or application back to a known good state prior to any sort of performance test run. Hopefully restoring to a known, good state doesn’t take too long, but it is essential. The initial state of the system is a variable in the equation: if you have your application or system in a different state on each test run, again, you don’t really know if a variable or parameter setting really has the effect you want. This state means right down to the exact same database records, database table state, same state for log files, state for running processes, etc. For systems I’ve worked on in the past, the easiest way to get to a zero state was to truncate database tables that get populated during the test run, and remove all log and output files. I always made sure to stop all the processes associated with the application under review and start the application from scratch for each run. Different systems will obviously have different issues with being in the exact same initial configuration.

Repeatable Results

To know with greater certainty that a particular system configuration works as you expect, it is important that you be able to repeat the results. This means getting the system back to the exact same state it was in prior to the first successful run, applying the change and running the test again. You could repeat this ad infinitum, and the more test runs you can reliably repeat, the greater the degree of confidence you can have in those particular settings. There may be a point of diminishing returns, so you need to decide how many runs are enough. For me, I want at least 2 runs, back to back, that show the same results. If I have the time and resources, I’ll do 2 runs on one day, then come back another day, reset and configure the system or application to where it was the first 2 times, and do the run again. I’ll even try to do it at a different time of day, if only to eliminate the possibility that something that happened to occur on the day of the original tests that didn’t happen on the second day.

Discipline and Structure Are The Underlying Concepts

Performance tuning requires a combination of imagination, analytic ability, the ability to see patterns in the data, and being willing to put up with the tedium of “change one thing, run a test, examine the results, reset, repeat”. Using checklists and pre-designed forms to capture the data and what you did help keep track of everything. But without discipline, performance tuning by “just trying some stuff” can end up as an exercise in futility.