With testing efforts and approaches transforming Agile adoption, testers in the teams have struggled to keep the pace and deliver high quality. While the intent is to bring more inclusiveness to testing efforts, the focus for testers is now more on understanding business needs, end-user experience, and product exploration. Agile teams need to do experience-based testing more often to reflect upon key business and end-user perspectives and build a strong feedback loop for product quality. Session-based testing is a great way to bring along all team members and have focused and dedicated time spent on exploring the application under test. This article intends to cover how to practice Exploratory Session-Based Testing within your teams and projects.
Why SBTs?
Exploratory testing is a very strong discipline and a highly undervalued practice. Often, team members indulge in exploratory testing only if they have time. The idea of SBTs is to bring in a highly intuitive way of exploring and experimenting with the underlying SUT.
How can we benefit?
- Bring in a natural way of testing without any preparation, pre-requisites, or strict guidelines.
- Versatile approach to testing
- Finds issues much faster as compared to traditional script-based testing
- Triggers creativity and critical thinking
Pre-requisites:
A team of 2 – 3 members can group up and arrange for an hour-long dedicated uninterrupted Exploratory Testing (ET) session. The session should define a Mission or Charter. The team chooses to spend this dedicated slot to work as a team and go bug hunting. A typical approach is to be more ad-hoc and try odd use cases while trying to break the system. One test idea leads to the other with a group of people interacting.
Guidelines:
- Time box the exercise to typically 1 – 2 hours. Although, having extended SBTs with smaller breaks is fine.
- Restrict external interferences – turn off email notifications, instant messaging, etc.
- Learn more about the product and uncover the unknowns (go beyond your regular regression testing)
- Choose your mission around areas that are end-user critical from a business standpoint (UI / API / Backend / no restrictions on the app layers to test)
- Think out of the box to test and break areas in the scope of the mission
- Record the entire session for future references (this also helps in record keeping)
- Assign one person as a record keeper – this person would be responsible for capturing and documenting the findings from the ET session
- If possible, bring along a mix of Engineering / QA / PO knowledge to make the session more effective
- Cap every session to max of 2 hours. If longer sessions are needed, break them into n * 2 hours.
- Avoid too much paperwork and keep it simple
Measuring Outcomes and Metrics:
- Allocate a Session ID to every SBT (maybe use Jira to capture the same)
- Include a charter summary, date, and time along with testers/participant details
- Every SBT should follow with a Debriefing meet with leads where we summarize the PROOF:
- Past – What happened during the session?
- Results – What was achieved during the session?
- Obstacles – What got in the way of good testing?
- Outlook – What still needs to be done?
- Feelings – How does the tester feel about all this, and how can we improve next time onwards?
- [Owner: Record Keeper] Derive a high-level view, maybe in percentages, of the time spent (Avoid exact numbers) in a particular session as per the following categories:
- Time spent in Setup (an upward trend in setup time indicates poor Test Environment setup. Team should identify such challenges and work towards getting a more testable and ideal test environment in place)
- Time spent in actual Bug Hunting and Test Execution (this is the core intent of SBTs. A healthy time spent here means sessions are going well. Although, make note of the time spent here is mainly around the original mission statement, or did the team diverge into other threads of testing)
- Time spent in Investigation, Deep Analysis, and Reporting (this can be part of the SBT, but at times, the team would look outside for gathering more information specific to the observations found)
- [Owner: Record Keeper] Derive a summary of total issues found and ensure their closure (either as false alarms or effective defect tracking tickets. Make sure all observations are brought to logical closures)
- Any SBT can also have certain opportunities as an outcome. Opportunities are anything found outside the mission or charter. It is perfectly fine to divert from the mission. It is encouraged that we know and categorize missions and opportunities separately
- Few opportunities could be vast to plan a completely dedicated SBT
- You can also reflect upon the SBT sessions with project status reporting. Example – We conducted 6 SBTs this cycle and found 15 blockers, two critical and four medium issues, along with three new opportunities.
Takeaways:
SBTs are a great way to collaborate. In the Agile world, with testers finding little time away from their regular deliverables, story testing, and automation activities, SBTs are a great way to break monotonous testing and come closer to the business as well as end-user pain areas. The dedicated and focused “no interruption” slots bring in tremendous business and functional learnings to the team and encourage critical thinking. As the team matures with running SBTs, it paves the way for bringing a fun element to testing and at the same time brings up dramatic improvements in overall quality.
(The author is Mr. Bangur Kshitij Kamalkishor, Principal Engineer – Quality Engineering, Altimetrik and the views expressed in this article are his own)
