Software Test Project Estimation: An use cases and functions based approach.

Dear All,

After a long time finally I have published my Software Test Project Estimation Tool.
I have started working on it back to 2008 and now 2016.

Tool link: http://www.4beats.net/test-project-estimation-tool/

IEEE DOI:
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7058650

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6962440

Please use the tool for software test project estimation and provide your feedback.

Share to others for estimation.

New story of chicken and cow.

The fable of The Chicken and the Cow is about commitment to a project or cause. When producing a dish made of thigh meat and eggs, the cow provides the thigh meat which requires his sacrifice and the chicken provides the eggs which are not difficult to produce. Thus the cow is really committed in that dish while the chicken is only involved, yet both are needed to produce the dish.

The fable of the Chicken and the Cow is used to illustrate the differing levels of project stakeholders involved in a project. The basic fable runs:

        A Cow and a Chicken are walking down the road.

        The Chicken says: "Hey Cow, I was thinking we should open a restaurant!"

        Cow replies: "Hm, maybe, what would we call it?"

        The Chicken responds: "How about 'meat-n-eggs'?"

        The Cow thinks for a moment and says: "No thanks. I'd be committed, but you'd only be involved!"

Manual Testing and more.

5 years back I got assignment writing on Manual testing. We know about it but when I have started searching on Manual testing it was really hard to find info. I was really surprised for not finding enough resources and because all the way we are using the term and working. Then why it’s too hard to find it?

I have studied on it and made documents for Manual testing. Even today I cannot recall the reference paper those I have followed. I have added few things from wiki and it could be useful for people not to memorizing but having an idea.

 

Manual Testing

                a. Paper based test cases

                b. Tools for version management of test cases

                c. Usability Test

*Definition of Manual Testing: Developing and executing tests that rely primarily on direct and continuous human interaction, especially evaluating correctness and test status (pass, fail, warn, etc.)

Manual testing is when testers:

·         Enter data

·         Observe behaviors

·         Actively run tests

Note: Manual testing may involve the use of tools

·         To create specific test conditions like load or errors

·         To capture performance statistics or internal states

**Stages: There are several stages of Manual testing. They are:

• Unit Testing: This initial stage in testing normally carried out by the developer who wrote the code and sometimes by a peer using the white box testing technique.

• Integration Testing: This stage is carried out in two modes viz. as a complete package or as an increment to the earlier package. Most of the time black box testing technique is used. However, sometimes a combination of Black and White box testing is also used in this stage.

• System Testing: In this stage the software is tested from all possible dimensions for all intended purposes and platforms. In this stage Black box testing technique is normally used.

• User Acceptance Testing: This testing stage carried out in order to get customer sign-off of finished product. A 'pass' in this stage also ensures that the customer has accepted that the software is ready for their use.

Process of Manual Testing:

• Preparing Test Cases

                                                SRS (Software Requirement Specification)

                                                UserStorySpec

A/V (Audio/video) files

ESSS (Electronic Static Screenshots)

RAD template

• Reviewing Test Cases
• Test Executes

*Challenges of Manual Testing:

• How do you staff the manual test team?

• What do test neophytes need to know?

• Can you maximize use of scarce equipment with manual testing?

• Can you sell management on the plan?

• What technical and managerial pitfalls exist for manual testing?

*Sizing the Manual Test Effort:

Given a set of tests to run in a period of time, plan number of techs needed

                a. Test considerations

·         For each test case, know…

Person-hours effort

Wall-clock duration

Dependencies and prerequisites

b. Use project-planning software to create schedule, keeping in mind...

c. Project considerations

·         For process and team, know overhead of…

Reporting defects

Documenting test status

Communication, e-mail and management guidance

Breaks

Blocking issues/debugging

d. Rules of thumb

·         6 test hrs/8-10 hr day

·         75% downtime (bad SW), 25% (good SW)

*Technical Caveats:

a. Tests Well-Suited for Manual Testing

·         Functional

·         Use cases (user scenarios)

·         Error handling and recovery

·         Localization

·         User interface

·         Configurations and compatibility

·         Operations and maintenance

·         Date and time handling

·         Installation, conversion, and setup testing

·         Documentation and help screens

b. Tests Poorly Suited for Manual Testing

·         Monkey (or random)

·         Load, volume, and capacity

·         Reliability and stability (MTBF)

·         Code coverage

·         Performance

·         Standards compliance

Caution: Using manual testing inappropriately can mislead people about the extent of test coverage

Weaknesses of Manual Testing:

• Incomplete testing
• Skipped input event sequences
• Incorrect verification of output
• Personal attention required
• Unattended testing ruled out
• Aborting a test implies manual retesting
• Increased cost
• Regression testing implies re -executing test cases
• Overall testing cost and effort multiplies
• No unattended testing
• Same test

Different inputs

Different versions

Different platforms

**Comparison to Automated Testing:

Test automation may be able to reduce or eliminate the cost of actual testing. A computer can follow a root sequence of steps more quickly than a person, and it can run the tests overnight to present the results in the morning. However, the labor that is saved in actual testing must be spent instead authoring the test program. Depending on the type of application to be tested, and the automation tools that are chosen, this may require more labor than a manual approach. In addition, some testing tools present a very large amount of data, potentially creating a time consuming task of interpreting the results. From a cost-benefit perspective, test automation becomes more cost effective when the same tests can be reused many times over, such as for regression testing and test-driven development, and when the results can be interpreted quickly. If future reuse of the test software is unlikely, then a manual approach is preferred.

Things such as device drivers and software libraries must be tested using test programs. In addition, testing of large numbers of users (performance testing and load testing) is typically simulated in software rather than performed in practice.

Conversely, graphical user interfaces whose layout changes frequently are very difficult to test automatically. There are test frameworks that can be used for regression testing of user interfaces. They rely on recording of sequences of keystrokes and mouse gestures, then playing them back and observing that the user interface responds in the same way every time. Unfortunately, these recordings may not work properly when a button is moved or relabeled in a subsequent release. An automatic regression test may also be fooled if the program output varies significantly (e.g. the display includes the current system time). In cases such as these, manual testing may be more effective

*All about Manual Testing:

·         Full test automation is not always possible

·         Manual testing is an effective and economical alternative

·         Manual testing involves facing some unique but tractable challenges

·         Proper planning of the manual effort and training of the technicians are critical

·         Manual testing can solve problems for you and for management

References:

1. http://www.rbcs-us.com/images/documents/ShoestringManualTesting%28Slides%29.pdf (*)
2. http://en.wikipedia.org/wiki/Manual_testing (**)
3. Asterisks are for reference indication

Automation testing: how tools are important?

Automation testing is a buzzword in today’s testing discussion. Whenever anyone met with buddy from other testing organization there will be very common chat “how do you automate your testing task; which tools you are using and so on”.

Today I am going to write on this discussion and how fruitful this automation; machine base testing.

There are two types of testing tools available to implement; one from different vendors and other from open source. I guess I have left one more type that is tools using different language and for this type there is no barrier. As per the testing needs we can build the tools/script.

Testing is a sequence of actions and cannot be accurately executing by testing tools. There needs to be intervention of human action.

Vendor tools: Scripts are created using record and even can write manually and these scripts are always vendor defined structured e.g. C like, VB like and so on. There are so many vendor’s namely Mercury now HP Mercury, Segue Inc., IBM. These tools are easy for the non-programmers since scripts are creating by record and play back system.

These vendor’s tools are very expensive and all organization cannot afford of these specially the load test tools.

I have read the article from Bret Pettichord “Hey Vendors, Give Us Real Scripting Languages” and I was commenting: Vendors can easily make standard language base scripting and that will be benefited both for vendor and tester. Only drawback is; they will not be able to earn money by certification :)

Now we can use few tools for free: WATIR, WATIN, Selenium and these are really good for scripting.

Still I will say these tools are useful and if we cannot afford of these we have choices :)

Open source tools: Watir, WatiN, Selenium, iMacro, JMeter and so on. These tools are very popular in the automation testing world since it’s FREE. We have our choice for functional regression test as well performance test. We are happy to use these tools and now we are not depending on vendor’s tools.

There is drawback for having knowledge only on open source tools that is many big organization wish to have test engineer with vendor’s certification and they will go for those vendor base testing tools but fact is it’s not always. Personally I am happy with these open source tools but sometime got weakness for vendor tools since I have started with those tools in my CSTP courses. Another drawback of open source tools is having bug or limitation since its open and there is no dedicated team to support it. We need to grow the community to help the open source tools more mature.

Programming language: different programming languages used for specific test; now days this is common practice to build different tools for testing purpose and other time creating scripts for resolving the harness of vendor and open source tools.

How these tools are important: One of the most important point for automation tool is it can be very useful for repetitive tasks. Data entry type job could be manageable with least effort and easily. For this type of task we can use iMacro; this is very easy to generate the scripts and change it as needed and run for the next time. It saves time while regression test since the automation tools take care of the maximum scenario coverage as we have prepared earlier.

But if we like to do some complex functional job with automation test tools; we need an expert in any tools. Because logics need to be implemented and writing the scripts from one to another tool vary. So, first of all we need to select the proper tools for the targeted job done. Changes in the code are another vital point for doing automated tests; how frequently the codes and UI are changing. If there are frequent major changes then the test scripts need to be work out for working with the changes.

I have used Watir for one of our big project; its fun. But problem comes e.g. I have used some XPath since there was no ID or NAME to capture the steps. Later what happened, the HTML & CSS are changed and now my XPath is not working. It’s simply horrible, my XPath is getting the pages it’s supposed to not.

One thing I have noticed; if we go for automation testing then development time itself there should be test suitable code so that it will be easier to do write test script. Otherwise we have to face those problems and it’s easy to minimize if we have taken care earlier.

Things to consider for automation test:

Test plan & early start: If we plan to go for automation testing; first thing to considered is test plan. We must include development team and overview of the importance of the automation test and how it will be effective. The plan and the meeting should be as early as possible before the projects start so that it could come up with the resolution as I have mentioned earlier. Late start always makes problem since it’s always better something to change earlier stage of the project. In the test plan itself we could come with tools selection, talents assignment, time estimation and targeted plan for the automation task.

100% automation test: I am doing testing job last 6 years and 90% of my execution was manual and planning, test case writing, project/task assigning. Still I believe that 100% coverage of automation test is not possible better to say will not be a wise decision.

“Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results. Since most products require tests to be run many times, automated testing generally leads to significant labor cost savings over time. Typically a company will pass the break-even point for labor costs after just two or three runs of an automated test.”

Manual testing, on the other hand, is a process that adapts easily to change and can cope with complexity. Humans are able to detect hundreds of problem patterns, in a glance, an instantly distinguish them from harmless anomalies. Humans may not even be aware of all the evaluation that they are doing, but in a mere "sequence of actions" every evaluation must be explicitly planned. Testing may seem like just a set of actions, but good testing is an interactive cognitive process. That's why automation is best applied only to a narrow spectrum of testing, not to the majority of the test process.

If you set out to automate all the necessary test execution, you'll probably spend a lot of money and time creating relatively weak tests that ignore many interesting bugs, and find many "problems" that turn out to be merely unanticipated correct behavior.

My opinion for automation test; it saves many things e.g. time, cost, effort and faster but it cannot save MANUAL test.

References:

1. http://www.stickyminds.com/sitewide.asp?ObjectId=2326&Function=edetail&ObjectType=
2. http://www.stickyminds.com/sitewide.asp?ObjectId=2084&ObjectType=COL&Function=edetail
3. http://www.io.com/~wazmo/papers/seven_steps.html
4. Test Automation Snake Oil V2.1 6/13/99: http://www.satisfice.com/articles/test_automation_snake_oil.pdf
5. http://www.io.com/~wazmo/papers/seven_steps.html
6. Software Test Automation: A Real-World Problem, Cem Kaner, Ph.D., J.D.

Startup testing part 1

I am starting here my new sharing “Startup testing part 1” for those who likes to start career of software testing. This will give a basic idea about testing and glimpse of it.

Definition:

• Testing is a process of evaluating a system by manual or automation means and verifies that it satisfies specified requirements or identifies difference between expected and actual result.
• Quality provides customer satisfaction for the first time and every time. It is the factor affecting an organizations long term performance and improves productivity and competitiveness.

Why Testing?

• Software testing is important as it may cause mission failure, impact on operational performance and reliability if not done properly.

• Deliver quality software products; satisfy user requirements, needs and expectation.
• Uncover defects before the products install in production, it can save a huge loss.

Participants in Testing:

• Software Customer
• Software User
• Software Developer
• Tester
• Information Service Management
• Senior Organization Management

Some Recent Major Computer System Failures Caused by Software Bugs:

• According to news reports in April’04 software bug was determined to be a major contribution to the 2003 Northeast blackout, the worst power system failure in North American history. The failure involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and economic losses estimated at $6 billion. The bug was reportedly in one utility company’s vendor supplied power monitoring and management systems, which was unable to correctly handle and report on an unusual confluence of initially localized events. The error was found and corrected after examining million of lines of code.

• In India September ’04 Aircel Cell Company got a defect in their prepaid subscriber billing system. Result nearly one month we got all free outgoing. They found the defects earlier but correcting the defects itself it took long time. I don’t have the exact official estimation of loss. But I made all ISD call nearly 100 hours free of cost.

• A software bugs in a Soviet early warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999. The software was supposed to filter out false missile detections caused by Soviet Satellites picking up sunlight reflections off cloud tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a’...funny feeling in my gut’, decided the apparent missile attack was a false alarm. The filtering software was rewritten.  

Software Development Life Cycle:

• Requirement- SRS (Software Requirement Specification)

                             SRAS (Software Requirement & Analysis Specification)

                             FS (Functional Specification)

• Design- HLD (High Level Design)

                    LLD (Low Level Design)

• Coding- According to code format
• Testing
• Implementation
• Maintenance

Testing Economic & Cost:

Traditional Test			Continuous Test
Accumulated Test Cost	Accumulated Error Remaining	Development Cycle	Accumulated Test Cost	Accumulated Error Remaining
0	20	Requirement	10	$10
0	40	Design	15	$25
0	60	Code	18	$42
$480	12	Testing	4	$182
$1690	0	Production	0	$582

Testing:

Static (Review)

Dynamic (Execution)

Static:

       Only review not execution of the program

Dynamic:

Structural (logic, white box testing, developer)

Functional (no logic, black box testing, tester)

What is Test Plan?

·        Road map for the entire testing activity

What are Test Cases?

·        Set of procedures which we execute in our system to find defects

Primary Role of Software Testing:

·        Determine whether the system meets specification (Producer View)

·        Determine whether the system meets business and user needs (Customer View)

Role of Tester:

• Find defect not correcting the defects

What is Defects?

• A defect is a variance from a desired product attributes
• Variance from customer/user expectation

Classification of Defects:

• Wrong (ER! = AR)
• Missing (Missing some point)
• Extra (Extra point)

Regression Test:

Tester-> 1000-test cases-> 100 defects-> developer-> tester

Functional Testing:

• Structure of the program is not considered
• Test cases are decided base on the requirements or specification of the program or module
• Hence it is called “Black Box” testing

Structural Testing:

• Concerned with testing the implementation of the program
• Focus on the internal structure of the program
• The intention of structural testing is not to be exercise all the different I/P or O/P condition but to exercise the different programming structure and the data structure of the program

Testing Levels:

• Unit Testing
• Integration Testing
• System Testing &
• Application Under Test (AUT) or

User Acceptance Test (UAT)

Unit Testing:

• LLD
• Module Testing
• Individually Testing
• White Box Testing
• Developer job

1. Test each module individually
2. Follow White Box Testing (logic of the program)

Integration Testing:

• LLD+ HLD (Developer+ Tester)
• Communication+ Data Flow
• WB+ BB= Gray Box
• Integrate two or more module i.e. Communicate between modules
• Follow a White Box Testing (testing the codes)

System Testing:

• Confirms that the system as a whole delivers the functionality originally required.
• Follow Black Box Testing
• Functionality Testing, Tester job

User Acceptance Testing:

• Building the confidence of the client and users is the role of the acceptance testing phase
• It is depend on the business scenario
• Red Box Testing (crucial)