Stages of Tool selection

In the stage of tool selection industry experts have suggested following four major criteria for selection of testing tools.

1) Meeting requirements.

2) Technology expectations.

3) Training / skills.

4) Management aspects.

Let us elaborate these criteria one by one.

1) Meeting Requirements:

a) There are many tools available in the market today

but rarely do they meet all the requirements of given product or a given organization. Evaluating different tools for different requirements involves lot of effort, money and time. Huge delay is involved in selecting and implanting test tools.

b) Test tools may not provide backward or forward compatibility with the product-under-test (PUT).

c) Test tools may not go through the same amount of evaluation for new requirements. For example some tools had Y2K-problem.

d) A number of test tools cannot distinguish between a product failure and a test failure. This increases analysis time and manual testing. The test tools may not provide the required amount of trouble-shooting/debug/error messages to help in analysis. For example, in case of GUI testing, the test tools may determine the results based on messages and screen coordinates at run-time. Hence, if the screen elements of the product are changed, it requires the test suite to be changed. The test tool must have some intelligence to proactively find out the changes that happened in the product and accordingly analyze the results.

2) Technology Expectations:

a) In general, test tools may not allow test developers to extend / modify the functionality of the framework. So, it involves going back to the tool vendor with additional cost and effort. Very few tools available in market provide source code for extending functionality or fixing some problems. Extensibility and customization are important expectations of a test tool.

b) A good number of test tools require their libraries to be linked with product binaries. When these libraries are linked with the source code of the product, it is called as the “instrumented code”. This causes portion of testing be repeated after those libraries are removed, as the results of certain types of testing will be different and better when those libraries are removed. For example, the instrumented code has a major impact on the performance testing since the test tools introduce an additional code and there could be a delay in executing the additional code.

c) Finally, test tools are not 100% cross-platform. They are supported only on some O.S. platforms and the scripts generated from these tools may not be compatible on other platforms. Moreover, many of the test tools are capable of testing only the product, not the impact of the product/test tool to the system or network. When there is an impact analysis of the product on the network or system, the first suspect is the test tool and it is uninstalled when such analysis starts.

3) Training Skills:

Test tools require plenty of training, but very few vendors provide the training to the required level. Organization-level training is needed to deploy the test tools, as the users of the test suite are not only the test team but also the development team and other areas like SCM (Software Configuration Management). Test tools expect the users to learn new language/scripts and may not use standard languages/scripts. This increases skill requirements for automation and increases the need for a learning curve inside the organization.

4) Management Aspects:

A test tool increases the system requirement and requires the hardware and software to be upgraded. This increases the cost of the already-expensive test tool. When selecting the test tool, it is important to note the system requirements and the cost involved in upgrading the software and hardware needs to be included with the cost of the tool. Migrating from one test tool to another may be difficult and requires a lot of effort. Not only is this difficult, as the test suite that is written cannot be used with other test tools but also because of the cost involved. As the tools are expensive and unless the management feels that the returns on investment (ROI) are justified, changing tools are generally not permitted.

Deploying a test tool requires as much effort as deploying a product in a company. However, due to project pressures, test tools effort at deploying gets diluted, not spent. Thus, later it becomes one of the reasons for delay or for automation not meeting expectations. The support available on the tool is another important point to be considered while selecting and deploying the test tool.

Finally how do we proceed with Tool Selection Process?

Following are the seven steps to select and deploy a test tool in an organization.

Step – 1: Identify your test suite requirements among the generic requirements discussed. Add other requirements, if any.

Step – 2: Make sure experiences discussed in previous sections are taken care of.

Step – 3: Collect the experiences of other organizations, which used similar test tools.

Step – 4: Keep a checklist of questions to be asked to the vendors on cost / effort / support.

Step – 5: Identify list of tools that meet the above requirements and give priority for the tool, which is available with the source code.

Step – 6: Evaluate and shortlist one / set of tools and train all test developers on the tool.

Step – 7: Deploy the tool across test teams after training all potential users of the too