Out of the box tester: Automation testing: how tools are important?

Automation testing is a buzzword in today’s testing discussion. Whenever anyone met with buddy from other testing organization there will be very common chat “how do you automate your testing task; which tools you are using and so on”.

Today I am going to write on this discussion and how fruitful this automation; machine base testing.

There are two types of testing tools available to implement; one from different vendors and other from open source. I guess I have left one more type that is tools using different language and for this type there is no barrier. As per the testing needs we can build the tools/script.

Testing is a sequence of actions and cannot be accurately executing by testing tools. There needs to be intervention of human action.

Vendor tools: Scripts are created using record and even can write manually and these scripts are always vendor defined structured e.g. C like, VB like and so on. There are so many vendor’s namely Mercury now HP Mercury, Segue Inc., IBM. These tools are easy for the non-programmers since scripts are creating by record and play back system.

These vendor’s tools are very expensive and all organization cannot afford of these specially the load test tools.

I have read the article from Bret Pettichord “Hey Vendors, Give Us Real Scripting Languages” and I was commenting: Vendors can easily make standard language base scripting and that will be benefited both for vendor and tester. Only drawback is; they will not be able to earn money by certification :)

Now we can use few tools for free: WATIR, WATIN, Selenium and these are really good for scripting.

Still I will say these tools are useful and if we cannot afford of these we have choices :)

Open source tools: Watir, WatiN, Selenium, iMacro, JMeter and so on. These tools are very popular in the automation testing world since it’s FREE. We have our choice for functional regression test as well performance test. We are happy to use these tools and now we are not depending on vendor’s tools.

There is drawback for having knowledge only on open source tools that is many big organization wish to have test engineer with vendor’s certification and they will go for those vendor base testing tools but fact is it’s not always. Personally I am happy with these open source tools but sometime got weakness for vendor tools since I have started with those tools in my CSTP courses. Another drawback of open source tools is having bug or limitation since its open and there is no dedicated team to support it. We need to grow the community to help the open source tools more mature.

Programming language: different programming languages used for specific test; now days this is common practice to build different tools for testing purpose and other time creating scripts for resolving the harness of vendor and open source tools.

How these tools are important: One of the most important point for automation tool is it can be very useful for repetitive tasks. Data entry type job could be manageable with least effort and easily. For this type of task we can use iMacro; this is very easy to generate the scripts and change it as needed and run for the next time. It saves time while regression test since the automation tools take care of the maximum scenario coverage as we have prepared earlier.

But if we like to do some complex functional job with automation test tools; we need an expert in any tools. Because logics need to be implemented and writing the scripts from one to another tool vary. So, first of all we need to select the proper tools for the targeted job done. Changes in the code are another vital point for doing automated tests; how frequently the codes and UI are changing. If there are frequent major changes then the test scripts need to be work out for working with the changes.

I have used Watir for one of our big project; its fun. But problem comes e.g. I have used some XPath since there was no ID or NAME to capture the steps. Later what happened, the HTML & CSS are changed and now my XPath is not working. It’s simply horrible, my XPath is getting the pages it’s supposed to not.

One thing I have noticed; if we go for automation testing then development time itself there should be test suitable code so that it will be easier to do write test script. Otherwise we have to face those problems and it’s easy to minimize if we have taken care earlier.

Things to consider for automation test:

Test plan & early start: If we plan to go for automation testing; first thing to considered is test plan. We must include development team and overview of the importance of the automation test and how it will be effective. The plan and the meeting should be as early as possible before the projects start so that it could come up with the resolution as I have mentioned earlier. Late start always makes problem since it’s always better something to change earlier stage of the project. In the test plan itself we could come with tools selection, talents assignment, time estimation and targeted plan for the automation task.

100% automation test: I am doing testing job last 6 years and 90% of my execution was manual and planning, test case writing, project/task assigning. Still I believe that 100% coverage of automation test is not possible better to say will not be a wise decision.

“Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results. Since most products require tests to be run many times, automated testing generally leads to significant labor cost savings over time. Typically a company will pass the break-even point for labor costs after just two or three runs of an automated test.”

Manual testing, on the other hand, is a process that adapts easily to change and can cope with complexity. Humans are able to detect hundreds of problem patterns, in a glance, an instantly distinguish them from harmless anomalies. Humans may not even be aware of all the evaluation that they are doing, but in a mere "sequence of actions" every evaluation must be explicitly planned. Testing may seem like just a set of actions, but good testing is an interactive cognitive process. That's why automation is best applied only to a narrow spectrum of testing, not to the majority of the test process.

If you set out to automate all the necessary test execution, you'll probably spend a lot of money and time creating relatively weak tests that ignore many interesting bugs, and find many "problems" that turn out to be merely unanticipated correct behavior.

My opinion for automation test; it saves many things e.g. time, cost, effort and faster but it cannot save MANUAL test.

References: