Did you ever wish that you didn’t sell your Sega 16, NES or ColecoVision at that garage sale for $5.00? Do you miss those classic games like ExiteBike, Super Marion Bros., Goonies, Mrs. Pacman, and Donkey Kong? The other day Alex pointed me to http://www.game-oldies.com to help relive those days when games were games and not simulators to help make you a precise killing machine. Game-Oldies has 1137 games, runs on JRE 5.0, and is addicting. The Java emulator does a decent job but the game-play can get slow and a bit annoying when your system is doing other things in the background. Is it legal? Beats me. Time will tell.
Hey, can you beat 1:01 on the first race in ExciteBike? BRING IT, DON’T SING IT…
Here at Corillian I am responsible for testing of the product Intelligent Authentication (IA). IA is a .NET Web Service written in C# and it is the first Corillian product released that is a Web Service. IA is also the first Web Service that I’ve ever tested. When I accepted the QA position on the Corporate Security team I knew I had some new, unique challenges ahead me with finding testing tools and techniques to successfully test a Web Service. We’ve released 1.0 and are now working on “1.1”. I spent a good portion of the 1.0 development time trying to find the perfect application and thus, here is my story:
First off, what was I looking for? My test tool needs weren’t too complex:
Create and combine test cases to run sequentially
Provide a test case status and overview
Allow data driven testing (static and dynamic)
Compare regression history
Allow the ability to send SOAP calls that aren’t compliant with the WSDL
Ability to validate valid SOAP responses and SOAP faults
I started by asking around which revealed little, and then perusing the Web. I found and looked through half a dozen home grown tools but nothing went beyond consuming a WSDL and sending requests to it. The tools I’ve listed below are worthy of a mention and the last tool listed is highly recommended (ParaSoft SOATest):
MindReef SOAPScope (achieves goals 5 & 6) Everybody on the IA team has SOAPScope and recommended that I start there. That I did. Playing around with the tool revealed that it is great for creating SOAP traffic but it didn’t take long to realize that this tool was designed for developers and not testers who wanted to store test cases. Since SOAPScope runs in Internet Explorer I toyed with the thought of driving the interface with Segue SilkTest and using SilkTest to store my test cases. This seemed like a lot of work for testing something as simple as a Web Service. There has to be a better solution for testing out there…
Segue .NET Explorer (achieves goals 1, 2, 1/2 of 3, 6) Corillian has used Segue products for quite a few years now; SilkTest and Silk Performer. A recent release of Silk Performer also provided an application with it called .NET Explorer. This app provides a quick and easy way to create Web Service test scripts without any .NET programming knowledge. I was able to quickly point to the WSDL, see all the methods, and then start creating test cases within the GUI. First impression: Wow! This is easy AND I can store & track my test cases! Second and last impression: Wow! This app is full of defects (I submitted 6 to Segue, 2 critical/showstoppers). After about 2 weeks I felt like I was testing 2 apps; .NET Explorer and Intelligent Authentication. It had a lot of little quirks that I found myself working around every day. The final straw for me was when the WSDL revealed a major change and the app didn’t know how to react; the result was that I lost all of my test cases/scripts that I had built over the last two weeks. Ouch… Fortunately I had them documented so that I could rebuild in whatever new test app I could find. NEXT!
MindReef SOAPScope and Sputnik Beta (2, 4, 5, don’t remember if it did 6) Now I was worried. I was already half way through the project and I didn’t have a reliable tool. I felt good about my testing progress but my automated regression suite was gone. Not good when I’m the only tester. I quickly went back into searching mode, looking high and low on the Web for any tool that dealt with SOAP. Around the same time, my manager Greg Hughes pinged Scott Hanselman to see if he had seen or heard about tools for SOAP. Scott had just received an email about Beta testing a product name Sputnik. Ahh, a ray of light from the stormy clouds looming over my head. I installed the Beta and found that Sputnik wasn’t really much different than SOAPScope. It used SOAPScope but it had a bit more to it. The addition allowed me to save a SOAP request and response and treat it as a test case that could be run again. This was a step forward from SOAPScope but it didn’t offer much more than that. The application was slow and clunky and didn’t give me much hope. It just wasn’t enough, I actually felt better about using .NET Explorer. After already losing once I wasn’t going to bet the bank on a beta product. Time to move on. Since then, Sputnik has become “MindReef Coral” and I haven’t had a chance to review whether or not it gained more features since the beta.
ParaSoft SOATest (1, 2, 3, 4, 5, 6) More and deeper searches on the Web, with creative key words, eventually revealed a product by Parasoft called SOAPTest (recently renamed SOATest). Their Website revealed to me that it did everything I needed it to do. It seemed too good to be true after all the research and trials that I did but there was only way to find out. I installed SOATest and waited for the call from the representative who had my trial key. My expected call ended up being a game of phone tag (I hate getting keys this way) and I ended up with the key a couple days later. Once I had my key I spent a few hours over the weekend putting the app through the test; making sure it could do everything all the other apps could do, and attempting to do all the things I needed the tool to do. The learning curve was quite a bit steeper than the other apps but for a good reason, this application is by far more robust. It does everything I need and more. There are too many things to spell out myself so here is the blurb from Parasoft’s site:
Ensure the reliability, quality, security and interoperability of your Web service.
Penetration testing integrated with functional testing for complete coverage.
Uniform test suites can be rolled over from unit testing to functional testing to load testing to security testing.
Prevent errors, pinpoint weaknesses, and stress test long before deployment.
Verify data integrity and server/client functionality.
Identify server capabilities under stress and load.
Accelerate time to market.
Scriptless Web Services testing.
WS-Security, SAML, Username Token, X.509, XML Encryption, and XML Signature support.
UDDI support: query verification, validation, and load testing.
Automatic test creation from WSDL, WSIL, UDDI and HTTP Traffic.
Complete coverage testing workflow through complex scenarios and multiple service endpoints.
WSDL schema and semantic verification and compliance to WS-I Basic Profile 1.1.
Data-driven testing through data sources (Excel, CSV, Database Queries, etc).
MIME Attachment support.
Windows Perfmon, SNMP, and JMX Monitors.
Detailed Report generation in HTML, XML and Text formats.
Real-Time graphs and charts.
HTTP 1.1 w/Keep-Alive Connection
See what I’m saying? A ton of stuff! So here I am months and 1000+ automated test cases later. I love it. It works. It’s easy. It’s robust. It makes build regression a snap for me. SOATest is the bomb! It only has a few small downfalls:
The price is a little large for small companies (around $4000 a seat).
Occasionally the window frames don’t refresh correctly (it’s a Java application).
There aren’t a ton of robust Web Service testing tools out there, so hopefully this post saves you some time in your search and decision!
Hey there browser compatibility tester! Why do you look so down? What? Your customer made the absurd request to test their new Web site with Netscape 4.03 on Unix? Wow, that’s odd. I bet you’re down because you don’t know where to find that download… No worries my friend, Netscape archives their products and has them available for download to the public (well most of them, there are the products that they pull off the Web the day after release because of BIG defects). The root of their public FTP is at: ftp://ftp.netscape.com./pub/ or http://ftp.netscape.com./pub/ (both accessible from your browser). You can find Netscape browsers as old as 3.04 and new as 188.8.131.52.
Netscape 3.04 – 4.8 can be found in the /communicator directory: http://ftp.netscape.com./pub/communicator/ (“Communicator” is not a browser name it is a suite of applications, much like MS Office. “Navigator” is the browser that comes within the suite).
In 2010 your pizza gal/guy is sure to steal your identity. Adcritic.com has an ad that gives you a preview of how our personal information will be used for/against us in 2010. The moral of the story is… do something about it by joining the American Civil Liberties Union at ACLU.com. Is fighting the government the fix? I doubt it. I think your best bet is to move deep into Montana (under a grove of trees so that satellites can’t photograph your house), don’t go to the doctor, don’t use the internet, pay for everything with cash, and hide your money under your mattress.
Consider the following text field in a balance transfer form:
<input name=”amount” type=”text” maxlength=”3″>
Let’s say this form has a $999 transfer limit controlled by the HTML maxlength property set to 3 (yes, it’s cheesy & foolish validation).
Happy testing! I’ll talk about bypassing validation with a proxy another day…
A spammer from Florida has been slapped with an 11 BILLION dollar fine for sending out 280 million spam e-mails. The number equates to $10 an e-mail the economic equivalent of the death penalty:
However, Kramer said he intends to take the spammers for every penny they have. “I will seek the economic equivalent of the death penalty,”
I’m the webmaster for ReedvilleBaseball.com and I just sent out 400 emails for Spring baseball registrations. A couple of people replied to me asking me why they were on the list and to take them off (The list came from last years registrations). What if one of those people decided to sue Reedville Baseball? That equates to $4000… That doesn’t work very well for our non-profit organization.
This should make us all think twice about our email lists. What is the legal definition of spam? How do I help our company avoid a lawsuit?
Is your defect database full of screenshot/images? Do you or your team ALWAYS attach screenshots? Screenshots in defects have a good and a bad side. Here’s my take on it:
Images can make a monster sized database. A majority of testers aren’t equipped with an image capture tool that provides lower quality and smaller sized images so they default to the good ol’ screen capture and paste that into Microsoft Paint. Using MS Paint, a 800×600 screenshot saves a BMP at 1.35 MB and a JPG at 123 KB (A JPG at 25% quality in Photoshop = 92 KB). MS Paint defaults it’s save extension to BMP and a lot of people don’t know much about image compression so they’ll leave it at BMP. Let’s crunch the numbers… (1 project with 250 defects containing screenshots) x (1.35 MB) = 337.5 MB of images. That’s a lot of MB for ONE project. Multiply that by 30 projects a year (Corillian probably does double that EASY) equates to over 1 Gig of images! Get my point?
Defect screenshots can promote laziness. You know what I’m talking about, that defect that you open that says “See attached screenshot” and little to nothing else. See what in the attached screenshot? How good the GUI looks when saved as a BMP? 🙂 This can be a serious problem with new testers and will cause a lot of back and forth between tester and developer as the real details are ironed out. Obviously this is very inefficient when the tester could have prevented the confusion by wrapping the image with descriptive text about the defect.
Images can be easier to comprehend (if the defect reader knows exactly what they are looking for in the image).
Images provide evidence. When the defect can’t be reproduced, you have that screenshot to go back on and then you don’t feel like you were crazy on crack at the time because the screenshot says so.
Screenshot advice for testers and team leads
Only use screenshots when the error is so technically confusing to describe that a screenshot will SUPPORT the description. Take the time to describe the problem, document the error text and provide a good description of steps to reproduce. By providing these things along with the screenshot you now have a defect that is highly searchable (preventing duplication) and has a supplemental image in case the text description didn’t cut it.
Testers: If you’re going to take a screenshot save it as a JGP or GIF not BMP. Leads: If possible, empower your testers with an image capturing tool that creates compressed images.
Over the holidays I took the opportunity to install the Windows Vista CTP Build 5270. I wasn’t about to take the risk of destroying any of my current Window installations so I thought I’d give it a whirl with Microsoft Virtual PC. I knew this would probably be a challenge, Virtual PC on an IBM T42 laptop, but I had the time. All in all my experience was… typical.
Installation Attempt #1: Enter the key, accept the terms, select installation type (upgrade or custom) and then select the install location. This is where my first hang-up occurred. The disk was showing as unallocated and also wanted 16 GBs to install. Hmmm… 16 Gigs, okay, if you say so. Problem was that the disk was marked as unallocated and the wizard wouldn’t let me do anything about it. So I exited out of the install and created a partition in hope that the installer would see it.
Attempt #2: Enter the key, accept the terms, select installation type (upgrade or custom) and then select the install location. The disk still showed as unallocated but this time I was able to work around it by selecting the “New” link. At this point I’d thought I’d try to cheat the system and create an 8 Gig partition. No go. I could select 8 Gigs but would get an error when proceeding on. After I allocated the 8 Gigs, I was given the format option. A quick format and I was prompted with “Windows will now finish installing automatically”. SWEET! I watched the progress bar slowly work its way up to the half way point where it decided to stay for a LOOONG time. I’m talking hours. I still hadn’t lost hope until I started Googling around for other Vista install experiences and found that one person had installed it within 20 minutes (not on a VPC). Twenty minutes? I’ve been stuck on the half way point for nearly 3 hours now. At this point I started to question whether or not the drive could read the CD correctly. The light was blinking every once in a while; it just didn’t seem aggressive enough. My patience had run thin and the braggart with the 20 minute install got me thinking something was wrong. So, I popped out the CD and checked it for smudges. Nope, looks good. I put the CD back in and was rudely rejected by the Vista installer. The installer had decided it was going to quit without the CD and it didn’t give me the option to try to point back to the CD. They should change that one button alert box text to something like “Damned if you do, damned if don’t. Click here stupid”. Ouch…
Attempt #3: Enter the key, accept the terms, blah, blah, and there I was again waiting patiently at the half-way indicator. Still not discouraged from my experience I looked at the progress indicator as half full and not half empty. I wasn’t going to wait around this time though. So I set my laptop on my chair and did some things around the house while it tried to break the half-way mark. When checking in about an hour later I found my laptop turned off. What the F… Picking up my laptop revealed that it was hot. A little too hot. Looks as though it overheated and shut itself down (no, the side vents weren’t blocked. Only the bottom). The fact that it shut down was good for my laptop and bad for Vista . Ok, note to myself…Don’t put laptop down micro-fiber chair while installing Vista .
Attempt #4: PROP LAPTOP UP IN PRECARIOUS POSITION ON MICRO-FIBER CHAIR. Enter the key, accept the terms, blah, blah, watch the progress meter go to half way, and go to bed. When I woke in the morning and checked in I was happy to see a prompt for Country selection. Woohoo! I worked my way through 5 more install screens and the Vista desktop appeared before my eyes (Albeit ugly at 640×480 with 16 colors). I fixed this by running the “Install or Update Virtual Machine Additions” from within the VPC. Whew…I did it!
In a nutshell, Vista took me one whole day to install with Microsoft Virtual PC. Granted, it could have been quicker if I hadn’t popped the CD out on try #2 but c’mon…. the progress bar was stuck at half-way for 3 hours.
Desktop Experience First visual impression…. pretty and Mac like. I was surprised to see that when Vista came up for the first time I had a network connection even though the setup doesn’t ask for the configuration. Impressive? Scary! My wireless network is encrypted and requires a key and I didn’t put it in…. So either I was connected to the neighbor’s unprotected network or the VPC was magically porting my connection through the laptop’s wireless connection. It seemed to be the later. Looking through Vista’s ” Network Center ” revealed little to nothing about where my network connection was coming from (wireless vs. local area connection). The Control Panel’s Network List revealed a bit more info. I did have a wireless connection but it wasn’t connected. My IP address revealed an IP delivered from router, so I assume it had to be running through the current laptop’s connection but had acquired my own IP for the VPC. Maybe this network tunnel was the reason for the stuck at half-way progress bar? Only Billy Wonka knows.
Under VPC….the experience is SLOW. My laptop has a Pentium M 1.8 GHz with 2 GB RAM and I allocated 1 GB of RAM to the virtual machine. It seems like it would help out a bit but it didn’t. The slowness is practically unbearable. In my enthusiasm to check out the cool new features I suffered through waiting 5-10 seconds for each mouse over, click and window load. Upping the priority to High or Real-time for VirtualPC.exe within Task Manager helps a bit but can cause VPC lockups when running some things that are graphic intensive (like Media Center or the new Game ” Purble Palace “).
The new features are pretty cool though. Lots of new stuff to sort through: Windows Mail, Media Center , Media Player 11, Internet Explorer 7, Windows Photo Gallery, Windows Defender and Parental Controls (I’m really excited to lock the kids out of their PC on timed intervals). I like the new look and feel (the circular start button is outside of the box). I’m sold. Why? I can see the usability is better. No longer will I have to spend so much time maintaining the wife and kids computers (ages 33, 15, and 11. Each has their own PC). I think things are so self explanatory that THEY can do a lot of the things that I spend so much time setting up and doing for them (Virus scanning, SpyWare, OS updates, network configuration). Also, the DEFAULT security is going to help the misinformed, unknowing and absent minded users. Vista is a great leap in helping users regain control of their computer (if I had a nickel for every friend and family member that wanted to buy a new computer because theirs was too slow. Later, after installing a few recommended basic tools, I reveal to them that it was riddled with SpyWare and viruses).
I like it! I don’t like it in a VPC though…Too SLOW and the install is misleading.
Check out TheElderGeekVista.com. He took the time to take screenshots of his similar experience.