Archive for April, 2008

Why We Should Drop the "Unit" in "Unit Test Frameworks"



I hate it.

Unit test this. Unit test that.

Let’s put “unit” in our testing framework so we can test “units” cuz that’s all ANYBODY is EVER going to test.

Testing outside of the “unit” is forsaken. Thou shalt never test more than a unit. Test more than a unit and ye shall be pelted with stones and crucified.

Let’s see… What do we have here for unit test frameworks:

brentUnit (even Brent Strange can have a unit testing framework)
JUnit (notice the word unit)
nUnit (look at me, I test units)
mbUnit (I’m a follower, I can’t think outside of the box. Uhm…let’s test units.)
csUnit (CHEESE & RICE, this is INSANE)

OH HELL…Just go here for the gi-hugey list:

You back? See what I mean? There’s like one-cajillion testing frameworks that are for “Unit Testing”.

Unit, unit, unit. It’s freakin’ beyond dumb.

Tell me…What happens once this developer phase of “Oh, I gotta go write some unit tests, with my unit test framework so my code is better” evolves to tests beyond “unit”? You know, like writing some tests that are “integration” or “functional? It’ll happen people.


I, and others have been using “unit testing” frameworks as an automation test harness for years now. All kinds of tests.. Tests that aren’t “unit”.

Yeah, insane. Unheard of I know. What? You going to crucify me now?

Drop the unit. Get over it. These are “Testing frameworks”.

Microsoft, nice namespace: Microsoft.VisualStudio.TestTools.UnitTesting

Don’t give me any crap about how “unit testing frameworks” are for developers. Good one rocket scientist…Let’s create separate frameworks to basically do the same thing. One for Dev, one for QA. Brilliant…

Six Different Ways To Execute Tests in Visual Studio 2008


I remember the 1st time I tried to kick-off a “unit test” from Visual Studio 2005. I had to defer to the Web, I couldn’t figure it out. A year later and through everyday use of VS 2005 and VS 2008 I’ve slowly discovered various ways to get those tests rolling. Here are 6 different ways to start a test in Visual Studio 2008:

“Test Tools” Toolbar


Context Menu


Keyboard Shortcuts

Ctrl+R, Ctrl+T: Execute tests in current context
Ctrl+R, Ctrl+C: Execute tests in current test class
Ctrl+R, Ctrl+N: Execute tests in current namespace
Ctrl+R, Ctrl+A: Execute all tests in solution
Ctrl+R, Ctrl+D: Execute the tests in the last test run
Ctrl+R, Ctrl+F: Execute the failed tests of the last test run

Test List Editor


Test Results Pane


“Test” Menu


Interview with Brent Strange: The Best Software Tester In the World


The votes are in and it’s official. Brent Strange has been crowned “The Best Software Tester In the World”. Over the course of  the 27 day voting period the software testing world was quite a buzz. Articles, blog posts, forums, and clubs have been consumed with facts, opinions and skeletons for the 7 candidates over the last 4 weeks. QA hubs such as StickyMinds, TestReflections, SQAForums, and SoftwareTestingClub were overtaken with this “best in the world” shakedown as QA professionals and interested developers voiced their opinions. Oddly enough, the recent and parallel presidential campaign even took notice.

An interview in the to be published June edition edition of Better Software Magazine  between Vimh and Brent bring to light why the community’s decision on this heralding QA Engineer was not just Internet viral:

Vimh: So Brent, how does it feel being declared the “Best Software Tester In the World”?

Brent: I’m speechless Vimh. I don’t deserve this. I test. I do my job. There is no Best Tester In the World. We all bring Quality Assurance to the table in our own unique ways. Honestly, this award should go to the entire Software Quality Assurance community.

Vimh: The people have spoken Brent. You are the “Best Software Tester In the World”. How do you think the community came to that decision?

Brent: I’m not sure. I’m guessing that those that I’ve worked with past and present voted for me, others I assume were followers of my small footprint in the QA community with my blog If not acquaintances or followers I suppose it was just viral. 

Vimh: Is it true that you once found and reported 70 defects in one day?

Brent: [Laughing] No… it was 59 defects, but technically 14 of those were enhancement requests.

Vimh: That’s a sign of somebody thinking outside of the box I suppose.

Brent: Inside and outside. You’ve got to be both places at once in this business.

Vimh: Where do you see yourself in 5 years?

Brent: Testing. Writing software to making testing easier and faster.

Vimh: How about 10 years?

Brent: White sand beach with Corona in hand.

Vimh: [Laughing] Sounds like a great goal!

Brent: It’s not a goal Vimh. It’s destiny. The Internet is gold mine my friend.

Vimh: Agreed. If all goes well I’ll see you on that beach with my trophy wife.

Brent: See you there Vimh. I’ll see you there…

Vimh: So now that you are empowered with the title “Best Software Tester In the World” do you picture yourself strutting into a developer’s cube and saying “listen to be me beeeaaaatch, it’s not “function as designed”, it’s a freakin’ defect and you’re gonna fix it”.

Brent: Uhmm…No.

Vimh: How about: “Hey dumb-ass, you ever heard of unit test?”

Brent: Dude, not funny. We’re all on the same team. [Pretty pissed off sounding] Listen, turn this interview around or I’m going to have hang-up.

Vimh: Okay, I apologize….Brent, you’ve had a few “skeletons in the closet” exposed during the last month.

Brent: Yes, 2 of them to be exact. I have the “never went to college” and “has no testing credentials” monkeys on my back.

Vimh: Do you think that hurt or helped your campaign? 

Brent: [Laughing] Call me a “Rocket Scientist”, my “un-educated” guess says “help”.

Vimh: Why do you think that is?

Brent: The reality of it is that a QA Engineer’s greatest skill is the openness and ability to lean quickly and then apply it towards quality initiatives. You don’t need a degree or certificate to do that. Matter of fact, I don’t even know of an entity that teaches that skill. I think the SQA community recognizes that. James Bach has brought that to light for us.

Vimh: Interesting. So you’re saying that anybody can be a good tester?

Brent: Not “anybody” but “anybody that is open to learn anything, has the ability to lean quickly, and then apply it toward a given task”. Software is “anything”, it changes daily. SQA Engineers have to stay on their toes and be able to use anything to help prove something.

Vimh: I never thought about it like that. Wow, “anything” is a lot to learn! That’s respectful. It’s known that you don’t care to manage QA. Why?

Brent: I’m a tester AND teacher at heart. Historically a QA manager relentlessly fights the quality assurance battle all day and often times compromises due to budget, schedule and company ignorance. His/her persistence typically pays off in the long run (several years sometimes) though. But…Have you ever taught a fellow QA Engineer something they didn’t know and then see that twinkle in their eye when they envision using that lesson in their testing tasks? It’s awesome! To me, that’s finding more defects by teaching other people to find defects. We BOTH just made software better…together. [Loudly] AT THE END OF THE DAY!

Vimh: Sounds like a pyramid scheme!

Brent: You said it! I’m a walking “QA Amway” and “QA Mary Kay“!

Vimh: [Laughing] Is it true you once said you knew the LDAP protocol to gain a Development team’s confidence and respect for testing, but you really didn’t know anything about it?

Brent: ONCE? [Chuckling] QA is often looked down upon because they are the 2nd tier information receivers and are expected to magically know everything right off the bat, which is near impossible. A developer can sit, stew, learn, and write code for a new technology for days, weeks, or months and then simply hand it off to QA expecting them to know the same AND know how to test it. To ensure quality and gain respect of that developer I need to  be “open to learn anything, have the ability to lean quickly, and then apply it toward the given task”. The Internet is my best friend. So, yes. Quite often “I am clueless, I make sure I project confidence, I go learn the subject after the fact, and then complete the task”. There is no harm in that is there?

Vimh: No, I guess not. Especially when you hammer a project with 70 defects in a day.

Brent: [Laughing] Yes, especially when you hammer a project with 59 defects…

Thoughts on Zephyr the ‘Next Generation Test Management System’



Over the last month I’ve been looking at Zephyr, a test management system that touts itself as “Next Generation”. What exactly is Zephyr and what does it have to offer to the testing community?

“Taking a completely realistic approach to how Test Teams work, collaborate and interact with each other in their department and the rest of their world, Zephyr brings together a comprehensive set of features, a really slick UI and Web 2.0 features at a price point that makes it very affordable for all team sizes.

Zephyr is based around the concept of Desktops & Dashboards. Every role in a Test Department has a customized Testing Desktop with relevant applications that allow them to do their jobs faster and better, as they all share data from a centralized repository and communicate via a collaborative backbone. Dashboards are automated and live, keeping the whole company updated on every aspect of testing and product quality.”

At a high level Zephyr offers:

  • Testing Desktop
  • Dashboards
  • Metrics & Reporting
  • Test Case Repository
  • Resource Management
  • Project Management
  • Release Management
  • Test Case Creation
  • Test Execution Planning
  • Test Execution
  • Document Management
  • Defect Tracking
  • Collaboration
  • Import and Export
  • User Interface
  • Integration

The 20,00 foot view of Zephyr is this (see the related “How Zephyr works” video here):


Now, I’ve been managing test cases in a Excel spreadsheet for years, a fairly advanced one at that. It utilizes Visual Basic for Applications (VBA) and gets the job done very well for me, several other QA Engineers, and a few managers. So, having to consider a test management system that costs $$ it a hard to swallow…If it’s not broke, don’t fix it right? With that eating at me, when looking at Zephyr I decided to compare it to what I have and currently use. Let me tell you what my KISS test management system consists of:

  • Quick and easy test case writing
  • Consistent test plan and case format
  • Reusable test case library that contains commonly used test cases
  • Brief but technical test case writing format
  • Test case state statistics by section and total
  • Testing summary for all testing sections/worksheets
  • Test case trends via charts
  • Automated coloring of test case status for quick visual reference
  • Automated test case to build mapping
  • Test case to defect mapping
  • Simultaneously sharing between multiple testers
  • Tester assignment by section/worksheet

Granted, it’s not perfect, but again it works very well and people really like it. With my testing world as the level set, let’s jump into the good, the bad, and the things to think about if your considering Zephyr:

Tester Assignment
You can specify users to run specific test cases or whole sections. Very nice!

I can drag and drop individual cases from one folder to another, but I can’t figure out for the life of me how to drag sub-folders of test cases into another folder (they move not copy). I also was unable to successfully import a previous export. If indeed this is possible it’s not easy or intuitive (drag and drop or export/import). This sucks, compared to me simply selecting one or several rows in Excel by hitting CTRL+C and then CTRL+V. Simple copying of test cases is extremely important to me.

Test States

  1. Zephyr uses the states of Pass, Failure, Unexecuted, WiP, and Blocked. These are good test states, but it lacks the two states that are geared a bit for the Test Lead: Duplicate (DUP), and Not-Applicable (NA). These are important states to me, primarily because a test case written by a test lead should never be deleted but sometimes they can be redundant across sections of test cases (needing DUP) or not-applicable because the requirements have changed, or the requirement needs to be assessed no matter what (needing NA). I don’t see a good way to manage these scenarios in Zephyr with the states provided.
  2. I’m a big fan of usability, and colors help with that a lot. I don’t like the fact that Zephyr doesn’t color their test case state (e.g. pass=green, fail=red).

Test Case Library and Templating
The nearest thing I could find to be “Test Case Library like”  is their import/export from file feature. I found that using it was truly cumbersome since I’m used to copying and pasting sets of test cases from one place to another within seconds. If you have or want the ability to hold a library of test cases, or better yet a library of templated test cases then you’re going to have to get really crafty with their app infrastructure (e.g. create a project and consider that your library).

Desktop client
In Zephyr, there is a lot of data in a lot of different screens and for the most part that is a very good thing. This was confusing at first, but the more I used and learned the app the more it made sense. The client is the browser with a Flash app running inside it. Working in a Flash app didn’t make usability or intuitiveness any easier though. The learning curve for me was a bit steep due to fumbling around with right click context menus in some places and not others to find features (if it didn’t exist I’d get the “Flash Settings” context). This is a huge pet peeve of mine and reminded me of working in a poorly written Java GUI.

A++. Love them. They’re informational and visually appealing!

Zephyr requirements traceability is lacking. You can attach a requirement document to a test case but can’t point to a specific requirement within that document. I suppose a guy could hijack an existing text input field to create a requirement number or reference in the attached doc (unless you’re okay with embedding it within the test case description). This is discouraging if you’re looking to tie a test case to a a specific requirement number.

Resource Management
You can assign and schedule test resources to your projects. This is really nice! Currently people/resources can be input into and then managed through Zephyr. I didn’t find an evidence of integration Active Directory or LDAP though. This could be a pain if you have many people on the QA team.

Defect Tracking
Zephyr integrates with Bugzilla. Good choice Zephyr! However, if you’ve customized your Bugzilla interface this feature won’t work for you (yet) since the default Bugzilla interface is duplicated inside of Zephyr.

Zephyr allows sharing, It even manages test roles: Manager, lead, tester. Roles would be nice in large QA departments. Again this might be a bit more convenient to manage with Active Directory or LDAP integration.

I’m a little discouraged by the price, mostly because they touted it as inexpensive during beta. After release the license model and cost is: “a simple yet flexible licensing model based on monthly subscriptions. Each user license is a low $65/user/month”. Let me help you with a few prices for a ONE year subscription: 10 users = $7,800, 100 users = $78,000. That’s not quite “almost giving it away!” as they state on their front page.

In a nutshell, I think Zephyr has done a great job with sharing the tester and test lead world with each other and management. However from a test lead perspective I’m a bit disappointed: The way test cases are written, managed, and copied needs to be improved vastly.  Writing test cases in Zephyr is not any easier or better than Mercury Quality Center (which frankly sucks in my opinion). That’s a serious problem when 1/4 of a test lead’s life is spent writing test cases. Zephyr is new though, give it some time and I think it will shine. In all fairness, Zephyr does a WHOLE lot more than what my spreadsheet is capable of, but I don’t think I need that whole lot more, especially when I lose my test case writing convenience.

Everybody’s needs, situation, and environment are different though, so go “kick the tires” yourself at:

ArtOfTest Now Has a Record/Playback Tool



ArtOfTest, creators of the browser automation framework WebAii, released a record/playback tool that works from within Visual Studio 2008! The beta release of “Design Canvas 1.0” fills a huge gap in the ArtofTest automation solution. Not only has ArtOfTest provided this great functionality, but this functionality looks really, REALLY cool. See for yourself with their 30 second QuickStart videos. I haven’t had time to dig into it myself yet but I’m really excited to do so. From an automation engineer that spends much of his day writing tests in Visual Studio, watching these videos made my mouth water. I’m thinking that ArtOfTest is going to take a lion’s share in the automation space in the next few years. They’ve chosen a great platform (.NET), a great IDE (VS 2008), and are working toward providing test and automation solutions for new technologies such as SilverLight.

James Bach the QA Hero


James Bach. Man, that guy…

He’s my QA hero.


It’s not a man-crush though, so don’t get all freaked out.

That guy…is able to eloquently match words to the things we struggle with in and as QA. He writes and speaks things that I’ve struggled to put to words for years.

Recently the TestCrew interviewed James and he spoke of things that I tried to describe recently:

TestCrew: We understand that you’re working on a new book about self-education and you will be sharing some ideas on this topic during the June CAST conference. What is the most important thing you’ve learned about self-education during your career?
James: The single most important thing about that I’ve learned about self-education is that it isn’t an activity I sit down to, as in “now I will educate myself.” Rather, it’s a way of looking at the world as a puzzle that I must solve, and most of my waking life is an opportunity to discover clues that help me solve it. Walking along a beach can be self-education, or noticing how my son plays a video game. My education comes largely by paying attention while life is going by, and noticing the patterns. For the tester, rapid self-education is essential. We get a half-baked product and must make sense of it. That’s fundamentally a learning task.

If you are in QA and you don’t follow James Bach then you are just making it hard on yourself. He paves the way. When he speaks…LISTEN. I guarantee you’ll learn something.

I mean, really… Look at him, how can you pass up a hero like this?


Silverlight 2 API and UI Automation


Microsoft SilverlightJeff Wilcox at Microsoft hat put together a great article entitled “Silverlight Unit Testing” that shows us how to use the newly available unit testing framework that comes with Silverlight 2 (currently in Beta 1). The framework has the ability to test the API as well as run the tests in all the browser that support Silverlight 2!

Check out Jeff’s video-cast entitled “walkthrough of the control unit tests” for a quick peek at how this stuff works.

This is really exciting from an automation standpoint. As an automation engineer I always fear new technologies and their hurdles that might make software not so automatable….much like that darn Flash. Here we are almost 10 years later and the best thing we have to “help” automate Flash is iMacros.

Post navigation