Benchmarking ColdFusion Code

Today's developer has an incredible number of resources available to them. Tools, utilities, programming languages, and community all contribute to the process of crafting the elegant solutions to complex problems we are tasked with everyday. However, with this broad spectrum of resources comes a greater need to focus on what should be factored into the decision making process. While ColdFusion developers already have good tools and support to aid in this process, there is a void. This article introduces the cf_benchmark custom tag and benchmarking of ColdFusion code to the developer.

How Do We Decide on a Solution?

Although this article is about benchmarking ColdFusion code, there's no question that this is one component of a process used to decide which solutions we apply to our programming problems. Every developer is different. One may decide on a solution because it's easier, or perhaps the most maintainable. Others may choose an obscure, spaghettied solution because they don't want anyone else touching their code. Some may even just use a solution because it's what the last developer did or it was the first hit on a Google search. Nonetheless the act of benchmarking code, no matter what environment you're working in, should and now will become a regular chore in your quest for the perfect solution.

Why Benchmark?

Whether you're just starting out or are a seasoned developer, the ability to effectively benchmark your code will prove as an invaluable asset. When faced with multiple answers to programming questions, it's good to have the best answer, at least in relation to performance, in your back pocket. Whether you work on a small project where performance is of relatively little consequence or you work on a large enterprise application where every bit counts, benchmarking is still more about the developer. It's about getting close to your code and understanding the impact of the algorithms you choose. Speaking of which, let's do that.

The cf_benchmark/cf_benchmarkblock Custom Tag

The cf_benchmark custom tag is a tool which enables the dynamic evaluation of blocks of CFML for n iterations, while recording and reporting execution times. There are no required attributes for the cf_benchmark and cf_benchmarkblock tags but you'll probably want to use some if not most of them to tweak results and gain a better understanding of what is being reported. Perhaps we can best see this through example. Save and execute the code in Figure 1.1 below.

Figure 1.1 - String Building Example

This test revolves around testing the average execution times of three string building techniques. Each cf_benchmarkblock tagset will be executed 1000 times by including the code in the master template. Once the job is complete, a comparison report will be embedded in the master template which describes which technique was on average faster than the other. Also note that all code inside the cf_benchmarkblock tags must be escaped to avoid evaluation before the benchmark job. To make the job of developing code for benchmarks easier, a Dreamweaver extension has been developed that will take care of this for you on every save operation. For complete information on the cf_benchmark and cf_benchmarkblock custom tags as well as the cf_benchmark Dreamweaver extension, please see the README file in the download.

On our first run, the code in Figure 1.1 yielded the comparison report in Figure 1.2 below.

Figure 1.2 - String Building Comparison Report

This report demonstrates the following:

• Traditional was 12.83% faster than ListAppend and 163.64% faster than cfsavecontent.
• ListAppend was 133.66% faster than cfsavecontent.
• Traditional performed 1073 evaluations per second.

Since we are dealing with an array of outside factors including server startup costs, it is recommended to run the job a couple of times to get a better feel for the results. You might also want to increase the number of iterations if the job completes too quickly. After a second run, we are given the report found in Figure 1.3 below.

Figure 1.3 - String Building Comparison Report, Second Run

This report demonstrates the following:

• ListAppend was 27.99% faster than Traditional and 196.56% faster than cfsavecontent.
• Traditional was 131.70% faster than cfsavecontent.
• ListAppend performed 1637 evaluations per second.

Just from these two runs, we can see there are clearly two very good prospects to work with. With some tweaking, we will be able to confirm which option is the best for the solution we seek.

Best Practices

1. Only benchmark on development systems.
2. Identify the common techniques you use and benchmark them against competing techniques.
3. Continually re-evaluate your benchmarks. As code changes, so will your benchmarks.
4. Don't just rely on the numbers. Factor in readability, maintainability, and reuse.
5. Use benchmarking as an opportunity to explore unique and creative solutions. You might be surprised at what you come up with.

While there are many factors considered when writing code, we hope that benchmarking makes your list of to-do's each and every time.