Software is not Jewellery

I have been testing an app for almost a month now. The app has got about 161 features and runs into about 300,000 lines of code. Plus it interacts with 2 other components written in a different languages altogether. So obviously it’s a big task testing this app. I have been running through the (incomplete) test speck, and writing results, communicating the results to the developers, reproducing a given problems so developers can resolve it. Any testing activity will involve the following major parts:

1. Identifying all usage scenarios (you could also call this writing a complete test spec)

2. Verifying that no usage path results in unstable behavior

3. If a instability is found, it should be reproduced and reported

4. Verifying that the software’s output is what the user wants

The real trouble in testing an app is the fact that all these activities have to be repeated for every build that comes around. This could be as often as twice a week. Unfortunately, it is not possible to repeat all test cases in this much time. So we cut corners here and there. The result is, we don’t have control over quality. The question arises: How can you reduce testing time? So lets go point by point here about all testing activities.

Identifying all usage scenarios

This is always THE MOST DIFFICULT TASK in testing. In a moment I will explain why. Let us say we are testing Microsoft Word. I have Office Word SP3 build ver 11.8215.8202 installed on my machine here. So here is what I did:

Steps to Reproduce:

1. Open Word. Word opens with a new document

2. Enter some random text like “The quick brown fox jumps over the lazy dog”

3. Open Find dialog (you could use Ctrl-F) -> Click over the More button at the bottom of the find dialog to open the complete functionality Find Dialog

4. Enter some text in the “Find What:” text box

Here is how it would look

5. Close the opened file through File->Close. The file closes but the search dialog is open

6. Without doing anything else clcik on Format in the bottom. User menu opens

7. Click on Font.. Option and you have your bug

You can read the message box that says that you got to save your file and there is really very less RAM on your drive. All this when you DON’T have any file open. And there is more

7. Try to click on the menu items without closing the insufficient memory dialog. Now navigate to any other application and the menu still appears as on the screen

And more..

8. Navigate back to word and try to close it by hitting the red close button. The app never closes. You can of course close it by calling task manager and killing Word.

Microsoft released three service packs for Word 2003 and I could find a bug in less than half an hour. I am not saying that Microsoft is dumb. I am just saying that there are so many many usage scenarios for Word that unknown bugs are inevitable. There is a big chance that Microsoft already knows about this bug. And they did not fix it because far too less user’s will encounter a problem like this. And fixing this bug would probably need a change in architecture so huge that it will be very very expensive. And this is reality.

If Microsoft let this bug go, even after thousands of hours of testing, there is a big chance that this could happen to many others. Any application of decent size will have bugs. Because there will be some usage scenarios that went untested and unreported before the app was delivered. Identifying all user scenarios is not only close to impossible but also impractical for the project budget. Software developers have an unspoken understating of software quality. I will try and explain some precautions we could take to make sure our software caters well to all usage scenarios.

So the question is how do we make sure we have handled all usage scenarios well. Hopefully I will elaborate on that in the coming articles.