A General Guide to Testing New Software Products
So you’ve been asked to test that new version of your software product. Your colleagues on the development team have been working away for months and at long last the much-anticipated release has landed on your desk for testing.
Everyone is excited about the new features and anxious to get the latest-and-greatest in front of your customers. But don’t be fooled; you are an extremely important part of the development process and your work is essential to ensuring a smooth rollout.
So today we are providing some guidelines to help you make sure your product is good to go.
Keeping the user in mind
You are the last line of defense for the user
The development team is well intentioned and will catch many software defects before sending the software to you. But they have deadlines, budgets and a host of other pressures to get the software out the door.
Furthermore, they have been focusing on getting the code working for so long that they can’t be expected to anticipate all the ways the users are going to break it.
Your job is to protect the user from bugs and usability issues by catching them before they are shipped.
You can think of the development team as Congress, perpetually locked in complex negotiations with the code. And you are the President, who is responsible for ensuring that what comes out of Congress is up to snuff. As President, you have veto power…and as a tester, you also have veto power over the software deployment.
It’s your job to determine whether the code released by the team is amazing or terrible, and if it is terrible, it’s your job to ensure that the team knows what to fix.
Don’t just report the error messages
An error message is a good indication that something is wrong, but it isn’t the only kind of problem you should watch out for. Unintuitive layout, confusing language, and clunky workflows all confuse the user and will generate support calls and general unhappiness.
- If you open the software and it confuses you, file a bug report for it.
- If you try an operation and accidentally do the wrong thing the first time, report it.
- If you clicked a button expecting it to do one thing only to find that it does something else, and the button you were looking for is elsewhere, chances are good that many of your users will do the same thing and part of your job is to stop that from happening.
What you bring to the table is a fresh set of eyes to the user interface and your feedback is invaluable to the software development process.
So don’t be afraid to file a bug report for that stuff too; the worst that can happen is that management decides not to fix it.
Don’t be afraid to be creative in your test methods
You may be given a list of test steps to follow but don’t limit yourself to what is on the list. Try to get an understanding of the problem the software is expected to solve and use that as a basis for testing.
Sometimes the design needs as much testing as the implementation, so put yourself in the user’s shoes and ask, “Does this software help me do my job as I was promised it would?” Then make up your own tests with that in mind.
Cultivating the “Testing Mindset”
Don’t take instructions from the development team
They wrote the software and so we can agree that they are the experts on what the software does.
But not necessarily what it is supposed to do.
They may have purposefully or inadvertently cut a few corners or even overlooked entire features. So if you are getting your testing instructions from them, then your testing may have the same deficiencies as the software itself.
Get instructions on how the software should be used from requirements/design documentation, proposals, slides or meeting minutes from early-stage discussions, or just have a quick chat with management to see what they expect the software to be and do.
Attempt to break the software
This is central to the “testing mindset”. Your development team built the software so that when you take the exact correct steps it does the right thing.
But they may not have put protections in place against the user taking unexpected steps. So it is up to you to use the software in a way that the development team may not have foreseen.
- If there is an email field, try entering an email address with two “@” symbols.
- If there is a phone number field, try entering weird characters like “~”.
- If there is a comments field, try copying and pasting the entire text of the US Constitution in there. The system should either cancel improper user input (e.g. attempting to type “~” into a phone number field does nothing) or it should display a user-friendly error message that conveys why the system can’t accept that input and what the user needs to change to proceed.
Figure out the parts of your product that are notoriously inconsistent or unreliable in general (or especially prone to breakage when the code is changed) and hit them hard.
Every software product has an Achilles Heel; if you aren’t sure where it is, your tech support team will be full of ideas.
Make sure your testing is broad as well as deep. If your product provides a search screen that includes multiple fields to search on, try every field instead of just the most common fields, and check the results to ensure they are filtered appropriately.
If your settings screen provides an “Apply” button as well as a “Save” button, make sure both work even though the vast majority of users will use “Save” rather than “Apply”.
Report everything that goes wrong
Don’t worry about seeming too nitpicky; that’s your job. Management can decide what to fix and what not to fix, but they can’t make that call if they don’t know what went wrong.
Remember that there are a lot more users than there are testers, so if you found an issue you can be certain that they will find it in a fraction of the time.
Also, don’t worry about time or budget constraints when deciding whether to file a bug. If your project is in a time crunch then it is more likely that the fix for your bug will be postponed until the next release, but management needs to make that call.
Your job is to provide as complete a picture as possible as to the quality of this release. So file bug reports as if there were an infinite amount of time and an infinite budget at your team’s disposal and let management worry about what gets fixed when.
Avoiding common testing pitfalls
Do some testing as a user with limited privileges
A common mistake when developing software is to always run as a system administrator or another user with a high level of permissions.
The problem with that is most users don’t have a lot of permissions and the vast majority will only have a few; so make sure that you modify your permissions to the level of the average user before testing. Just dropping privileges alone can cause bugs to surface.
You should also spend time making sure users with limited permissions are not able to see anything they shouldn’t see or perform any operation they shouldn’t be able to perform.
Run a few searches and see if you can get access to records that limited users shouldn’t see. Attempt to access the Administrator screens and make changes.
Don’t forget to test for non-functional requirements
Not all requirements involve creating records or viewing information stored in the database. Your software may also have requirements as to how it is presented, how it is secured, or what resources are available. Some examples:
Accessibility requirements (all government software is governed by Section 508). The bare minimum for a web application includes:
- Providing an “alt” attribute for images
- Ensuring that text fields have an associated label with the text field’s ID in the “for” attribute
- Ensuring that fields can navigated using the tab key
- Ensuring that no information is conveyed on the basis of color alone (e.g. if a list of records has certain rows highlighted red, there should be a separate column containing the text version of whatever the red highlighting signifies)
A requirement that all features are described in a help doc – if this is the case, you should open the help docs to ensure that any new features are described there.
Your software may have client-specific configuration – such as features that should only be available to one client and hidden for everyone else. If so, make sure you apply the configuration for each client and verify that they are able to see everything they are supposed to see and nothing else.
Web applications may require support from multiple browsers or multiple versions of the same browser. If that applies to your product make sure you test the application in those browsers.
Providing great feedback
So you’ve followed the advice above and found a bug! You typed something and clicked a button and got a big, ugly error message. You are ready to fire off a bug report and get back to the hunt.
Hold on there, hot shot! A half-baked bug report is not going to convince your development team that their software needs more time in the oven. Here are a few things you should do before creating that bug report:
Try it again – You need to make sure your bug can be reproduced reliably.
Try again with different input – Whenever possible you should try to narrow down the circumstances under which the bug occurs. If you got your error after submitting a form with a few fields left blank, try filling in all of the fields.
If there are multiple ways to do the same thing and one of them is broken, see if those other methods are also broken. If your system allows the user to update their email address in the User Profile screen as well as the Address Book, and you get an error when using the User Profile screen, go into the Address Book and see if that method works.
Remember that your development team thinks that their software is perfect, so they may be reluctant to believe you when you say it isn’t.
You can save everyone time and frustration by including all of the details necessary to fix the bug right in the bug report so that you will not need to be asked for clarification. Each bug report should contain at least the following:
Which task or operation are you trying to perform? Examples: placing an order, view a layer on a map, view a user’s contact information
What steps did you take? It is impossible to be too specific here. Example: ‘From the main page I clicked Contacts, then scrolled down to John Doe and double-clicked the record. On the details form I attempted to change the email address from firstname.lastname@example.org to email@example.com and received the error” URLs are helpful and take almost no time to include. If there is a certain file you tried to upload when you got the error, attach the file to the bug report.
What happened when you took those steps? The exact text of error messages is crucial, and screenshots or videos are very helpful in this area. If there is no error, and instead you get a spinning “loading” wheel or a progress bar that never fills up, include that information. It is critical that you say what actually happens. Even if nothing happens at all you should explicitly say so.
Other pieces of information you should consider adding to the bug report:
- How many times you tried it.
- Anything you did to try to work around the error (and whether each workaround attempt was successful)
- Offer to show the developer the error happening on your computer.
We just discussed some great ways to help ensure a timely yet thorough testing phase, but it doesn’t end there. Things will inevitably go wrong. Your users will find bugs you didn’t find.
Don’t get bogged down in the failures you may experience. Instead look for ways to tweak your process to avoid those failures in the future. Your testing process and methods will always be a work-in-progress so be patient with yourself and keep in mind all the bugs you have rescued your users from.
Patrick Scanlon is a Senior Developer for the Federal division of GISi and has extensive experience developing and deploying web applications to government systems.