Over the last month I’ve been looking at Zephyr, a test management system that touts itself as “Next Generation”. What exactly is Zephyr and what does it have to offer to the testing community?
“Taking a completely realistic approach to how Test Teams work, collaborate and interact with each other in their department and the rest of their world, Zephyr brings together a comprehensive set of features, a really slick UI and Web 2.0 features at a price point that makes it very affordable for all team sizes.
Zephyr is based around the concept of Desktops & Dashboards. Every role in a Test Department has a customized Testing Desktop with relevant applications that allow them to do their jobs faster and better, as they all share data from a centralized repository and communicate via a collaborative backbone. Dashboards are automated and live, keeping the whole company updated on every aspect of testing and product quality.”
At a high level Zephyr offers:
- Testing Desktop
- Metrics & Reporting
- Test Case Repository
- Resource Management
- Project Management
- Release Management
- Test Case Creation
- Test Execution Planning
- Test Execution
- Document Management
- Defect Tracking
- Import and Export
- User Interface
The 20,00 foot view of Zephyr is this (see the related “How Zephyr works” video here):
Now, I’ve been managing test cases in a Excel spreadsheet for years, a fairly advanced one at that. It utilizes Visual Basic for Applications (VBA) and gets the job done very well for me, several other QA Engineers, and a few managers. So, having to consider a test management system that costs $$ it a hard to swallow…If it’s not broke, don’t fix it right? With that eating at me, when looking at Zephyr I decided to compare it to what I have and currently use. Let me tell you what my KISS test management system consists of:
- Quick and easy test case writing
- Consistent test plan and case format
- Reusable test case library that contains commonly used test cases
- Brief but technical test case writing format
- Test case state statistics by section and total
- Testing summary for all testing sections/worksheets
- Test case trends via charts
- Automated coloring of test case status for quick visual reference
- Automated test case to build mapping
- Test case to defect mapping
- Simultaneously sharing between multiple testers
- Tester assignment by section/worksheet
Granted, it’s not perfect, but again it works very well and people really like it. With my testing world as the level set, let’s jump into the good, the bad, and the things to think about if your considering Zephyr:
You can specify users to run specific test cases or whole sections. Very nice!
I can drag and drop individual cases from one folder to another, but I can’t figure out for the life of me how to drag sub-folders of test cases into another folder (they move not copy). I also was unable to successfully import a previous export. If indeed this is possible it’s not easy or intuitive (drag and drop or export/import). This sucks, compared to me simply selecting one or several rows in Excel by hitting CTRL+C and then CTRL+V. Simple copying of test cases is extremely important to me.
- Zephyr uses the states of Pass, Failure, Unexecuted, WiP, and Blocked. These are good test states, but it lacks the two states that are geared a bit for the Test Lead: Duplicate (DUP), and Not-Applicable (NA). These are important states to me, primarily because a test case written by a test lead should never be deleted but sometimes they can be redundant across sections of test cases (needing DUP) or not-applicable because the requirements have changed, or the requirement needs to be assessed no matter what (needing NA). I don’t see a good way to manage these scenarios in Zephyr with the states provided.
- I’m a big fan of usability, and colors help with that a lot. I don’t like the fact that Zephyr doesn’t color their test case state (e.g. pass=green, fail=red).
Test Case Library and Templating
The nearest thing I could find to be “Test Case Library like” is their import/export from file feature. I found that using it was truly cumbersome since I’m used to copying and pasting sets of test cases from one place to another within seconds. If you have or want the ability to hold a library of test cases, or better yet a library of templated test cases then you’re going to have to get really crafty with their app infrastructure (e.g. create a project and consider that your library).
In Zephyr, there is a lot of data in a lot of different screens and for the most part that is a very good thing. This was confusing at first, but the more I used and learned the app the more it made sense. The client is the browser with a Flash app running inside it. Working in a Flash app didn’t make usability or intuitiveness any easier though. The learning curve for me was a bit steep due to fumbling around with right click context menus in some places and not others to find features (if it didn’t exist I’d get the “Flash Settings” context). This is a huge pet peeve of mine and reminded me of working in a poorly written Java GUI.
A++. Love them. They’re informational and visually appealing!
Zephyr requirements traceability is lacking. You can attach a requirement document to a test case but can’t point to a specific requirement within that document. I suppose a guy could hijack an existing text input field to create a requirement number or reference in the attached doc (unless you’re okay with embedding it within the test case description). This is discouraging if you’re looking to tie a test case to a a specific requirement number.
You can assign and schedule test resources to your projects. This is really nice! Currently people/resources can be input into and then managed through Zephyr. I didn’t find an evidence of integration Active Directory or LDAP though. This could be a pain if you have many people on the QA team.
Zephyr integrates with Bugzilla. Good choice Zephyr! However, if you’ve customized your Bugzilla interface this feature won’t work for you (yet) since the default Bugzilla interface is duplicated inside of Zephyr.
Zephyr allows sharing, It even manages test roles: Manager, lead, tester. Roles would be nice in large QA departments. Again this might be a bit more convenient to manage with Active Directory or LDAP integration.
I’m a little discouraged by the price, mostly because they touted it as inexpensive during beta. After release the license model and cost is: “a simple yet flexible licensing model based on monthly subscriptions. Each user license is a low $65/user/month”. Let me help you with a few prices for a ONE year subscription: 10 users = $7,800, 100 users = $78,000. That’s not quite “almost giving it away!” as they state on their front page.
In a nutshell, I think Zephyr has done a great job with sharing the tester and test lead world with each other and management. However from a test lead perspective I’m a bit disappointed: The way test cases are written, managed, and copied needs to be improved vastly. Writing test cases in Zephyr is not any easier or better than Mercury Quality Center (which frankly sucks in my opinion). That’s a serious problem when 1/4 of a test lead’s life is spent writing test cases. Zephyr is new though, give it some time and I think it will shine. In all fairness, Zephyr does a WHOLE lot more than what my spreadsheet is capable of, but I don’t think I need that whole lot more, especially when I lose my test case writing convenience.
Everybody’s needs, situation, and environment are different though, so go “kick the tires” yourself at: http://demo.yourzephyr.com