Balancing ATDD, GUI Automation & Exploratory Testing

Conclusion

AUTOMATION

TDD/ATDD

EXPLORATORY TESTING

Exploration happens at all of these levels, and is an approach and a mindset, not a methodology.

Automation happens at many levels.

Use automation to extend your reach and senses.

Let automation take you as many interesting places as possible.

TDD focuses on writing correct and clean code.

ATDD focuses on making sure the functionality being delivered makes good on the goals and promises made to our customers.

Working together, the odds of quality software are much more heavily in our favor.

GUI AUTOMATION

ATDD

WHAT IS IT?

Places the focus on User Stories from the expectations of the customers.

Is geared towards the members of the team that are not necessarily programmers.

Uses tools that allow a more natural language to present the requirements.

Acceptance Test Driven Development (ATDD) builds atop of Test Driven Development (TDD)

Proliferation of tools that help simulate the actions of users.

Follow standard workflows

(logging in, navigating to pages, clicking on links, filling in forms, etc.).

TESTING

GUI AUTOMATION DEFICIENCIES

“The art and the act of asking questions of a product, then developing & devising new and more inventive questions based on the answers we receive.”

TDD AND ATDD ARE NOT "TESTING"!

GUI tends to be the most fragile layer of an application.

Tremendous overhead of script maintenance.

Not good at checking context.

WHERE DOES GUI AUTOMATION EXCEL?

`

TDD and ATDD are design processes.

Help develop clean, well functioning code.

Help ensure software being developed meets needs of the project

Tremendous blessing when dealing with repetitious steps (set up and take down).

Important part of Acceptance Test verification.

Semi proxy for human interaction.

TDD and ATDD work well with Continuous Integration.

IN PROCESS AUTOMATION: CRUMBS

PREPARING FOR AUTOMATION: TERMS

(C)onfirmation, Coverage Criteria and Complexity

(R)isk, Robustness and Reliability

(U)sefulness & Usability

(M)aintainability and Manual Effort

(B)asis and Bias

(S)pan, Separation and Security

WHERE DO TDD AND ATDD EXCEL?

WHERE DO TDD AND ATDD FALL SHORT?

(T)ools & Technology

(E)xecution

(R)equirements and Risk

(M)aintenance

(S)ecurity

• Disciplined approach to developing cleaner software.
• Act as a brake on "cowboy code".
• New programmers can see code in context with the tests that have been written.
• Keeps issues based on dependencies to a minimum.
• Programmers focus on "just enough design" and "just enough development".
• Entire product team does design and development work.

Often unrealistic expectations.

Only be as good as the overall skill of the developer(s) using it.

GUI AUTOMATION vs. TDD/ATDD

Socialtext Automation (Wikitests)

Helps with constructing acceptance test cases

Helps to check and demonstrate the acceptance criteria has been met.

GETTING THE MOST FROM YOUR TAXI TRIPS

USE PERSONAS TO VISUALIZE GOALS

ZEN MIND, BEGINNER MIND

Put yourself into the shoes of as many potential customers of our product as I can.

That means understanding the interests, desires and ways of interaction of many different groups of people.

The ways that they interact with the devices they use to discover content/interact is very different.

Domain Expert Level Exploratory Testing

Novice Level Exploratory Testing

They are different.

Source of mental friction is knowing the eccentricities of your application and when they can get in the way of effective testing and exploration.

SESSION BASED TECHNIQUES

• Developing small charters and executing on them in set periods of time.
• Focus testing areas that are most important.
• Focus on a small sub system.
• Keep our testing fresh, our eyes and minds focused, work on what's most important.

FREIGHT TRAINS vs. TAXI CABS

Freight and passenger trains limited to the rails the trains can ride on.

Taxi cab can be called to appear anywhere and take us anywhere.

TAXI CAB SCRIPT

EXAMPLE TAXI CAB SCRIPT

Jumping off allows the tester to look in a varity of places that might not be obvious

Script uses Fitnesse style table structure

Comments show points of Interest to "Hop Off" look for SEE

TAKE ME TO...

Can We Create a Balance?

• TDD, ATDD, and front end GUI automation are processes.
• Exploratory Testing is a process and a mindset.
• Developing and defining acceptance is exploratory.
• Front End GUI Automation starts from capturing discoveries made while examining requirements.

EXPLORATION IS A MINDSET

TDD/ATDD ARE EXPLORATORY

Exploratory Testing is a way of thinking about questions and answers.

Exploratory Testing does not live outside of automated testing.

Automation and Computer Aided Testing can be central to exploration.

• Testing is part of the initial development effort.
• Testing comes first.
• Putting the emphasis on TDD/ATDD makes sure that it meets the criteria it is being designed for.

Testers can also help guide and provide consideration for testability in areas the programmers might not consider.

Asking "what if" questions vital at this stage

DELIVER BUSINESS VALUE

THE ROLE OF TESTERS

Question 1: "is what I am doing helping to deliver real value to my customers"?

If we put the business value first, that will guide us in our testing efforts and the approaches that we use [Crispin, 2008].

”Testing is dead" == testing needed to be a part of an entire process of quality and improvement, starting at the earliest stages of design and development.

Exploration, automation, and testing must come earlier in the process, and involve not just testers, but everyone on the product team.

EXPLORATORY TESTING

“Scientific thinking in real-time” [Bach, 2012]

Puts test design and test execution together.

The tests that we did before inform the tests that we will do now. Those tests will inform the tests we will perform later.

EXPLORATORY TESTING’S DEFICIENCIES

EXPLORATORY TESTING VS. AUTOMATED TESTING

Less likely to find new revelations by going over the same ground (pesticide paradox)

Not enough time to go through all possibilities.

Sentient humans do it. Sentient humans get bored.

Instead of saying "automated testing", I prefer to use the term "computer aided testing".

Exploratory Testing can, and often does, use elements of computer aided testing to help accomplish its goals.

AUTOMATION CAN’S AND CANNOT’S

WHERE DOES EXPLORATORY TESTING EXCEL?

Sequence of steps that can provide answers about the state or condition of a product.

Imagination and creativity open up possibilities.

When there is little in the way of formalized documentation.

Setting the Stage

Different groups would have us believe that each is the magic bullet.

Three approaches that complement each other

DO WE STILL NEED TESTERS?

WHAT’S HAPPENED SINCE 2001?

• Twelve years since the Agile Manifesto
• Test Driven Development (TDD) has emerged as a standard to guide code development driven by tests.
• Acceptance Test Driven Development (ATDD) extends model.
• Proliferation of automated testing tools for a variety of platforms and needs.

NO: Testing is dead. With the development of these tools, developers now do the testing earlier and up front [Savioa, 2011].

YES: Apply reason and dynamic thought to problems. Thinking, actively engaged testers are needed now more than ever [Tomlinson, 2011].

Balancing ATDD, GUI Automation & Exploratory Testing

Michael Larsen - http://mkltesthead.com/ - Twitter: @mkltesthead

Based on Jim Harvey's speech structures