Knuth is a pretty smart person
A long time ago, in a galaxy right here, Donald Knuth wrote “premature optimization is the root of all evil”. This was in his 1974 paper “Structured Programming With Go To Statements”, yet this issue is still with us in various forms.
Performance optimization is a technique for improving a system’s effectiveness pertaining to running time or memory usage. Optimization should be done after there is a working system which needs its effectiveness improved. If optimization is performed during the initial implementation, it is by definition premature as there is nothing yet to improve.
We encounter this problem regularly because Knuth’s quote is taken out of context. The full quote starts off “Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered.”
In other words, this is not an inherent problem with programming; we actually cause this problem ourselves due to how we think about programming.
We’re spending precious time needed for optimization and debugging (time that will be needed once the project is completed) at a point in time when we do not have any assurances that the work being done is useful. Furthermore, we can easily make mistakes while optimizing. An optimized system is also more complex to debug and maintain than the same non-optimized system.
The impetus for present rambling
Our team recently encountered a quite bothersome performance issue with a client’s site occurring in a system that, given the amount of performance related modules installed, made little sense.
This system was designed and implemented before we ever got to look at it and had been running for some time. These modules were installed for “performance reasons”, according to the documentation: boost, advagg, entitycache, memcache, and CDN. While these modules do all have their place and usefulness, they were mostly not a good fit for this project.
These modules make the system more complex compared to a plain Drupal setup, especially regarding stack tracing and profiling. These modules each implement optimization techniques and all come with side effects which make debugging and maintenance more complex.
It was only after removing almost all of these elements from the system (and still experiencing the performance issue) that a colleague was able to determine that the bottleneck we were looking for was in an area completely unaffected by any of these modules and fix the problem.
In other words, the optimization performed by all of these modules, which were ostensibly installed to deal with performance, was a myth. They were all instances of premature optimization. This site functions well without them, and leaving them in place made discovering the actual performance problem exceedingly difficult.
How to win
In closing, finding the underlying problem was a big improvement for the project, but it should have been a less painful process.
To get to a path to happiness, we all need to recognize our own thoughts that lead to premature optimization while they are happening and avoid acting on them. Once a working system is in place it can be examined using tools designed for the task and then optimization techniques can be applied where they are needed most to ensure the effective operation of the system.
Don't spend time on premature optimization, and actual optimization will cost less. It's a win-win for clients and developers.
Bonus feature: “Stop doing that.” --Douglas Crockford
Premature optimization can also take another slightly less obvious form.
Not performance related at all, and still common, is the desire to write “clever” code or “save keystrokes” by writing expressions very tersely. This is most definitely not what we should be spending time thinking about and yet it does happen (and has spurred endless debate, more wasted time). It is for this reason that I beleive it is imperative to work with a coding standard for your team and stick to it rigourously through peer code reviews (and internalizing the standard as much as possible). Doing so can systematically eradicate this particular form of the problem.