Skip to content

Defect baselining – An approach to better defect management

April 20, 2010

What do you do when faced with a lot of bugs… especially on the eve of a release? How do you handle this with minimum risk factor?

The Context

The team: 21 members including 9 QA/testers working on development/enhancement and maintenance of a suite of 18 Add-on products for Microsoft Dynamics CRM versions 1.2, 3.0, and 4.0.

Releases: Worldwide releases, supporting all previous released versions (via upgrades) of the products in multiple languages [English, French, German, and Dutch], deployed across multiple environments.

Initially, the releases were very eventful. The testing team went about its job of finding and reporting the defects in a release candidate. The defects were prioritized by the product management and fixes done accordingly by the dev team. The number of bugs used to be overwhelming, so much threatened to cause burn out. This, inspite of all the quality assurance activities, like following coding standards, code reviews, unit testing, and formal testing. There was a growing concern that something is amiss.

On careful analysis of the problem at hand, we figured out that a lot of defects had been in the system since quite some time and were not a direct result of recent development and enhancements. Moreover those bugs really didn’t bother the users. This could be most probably because the bugs didn’t affect or hinder the way they used the system. Or maybe they were able to find a way around.

That is when we got around to the idea of baselining the defects that are present in the system. During the next regression cycle we documented the defects that were found, in an excel sheet and cross checked these with the defects that were already logged in the project management tool. Well! it was one huge exercise.. but worth it.

At the end of this exercise, facilitated by the information collected, we were able to classify the defects

* Duplicates – a number of duplicate issues were logged in the tool So we removed the duplicates. It served in cleaning the system as well as in reducing the bug count!!

* Reopened issues – defects that had been closed in an earlier version but were opened again

* Legacy issues – defects that were already found in earlier version and still in open status (the ones with lowest priority)

* New issues – defects seen only in the current regression

* Redundant issues – those that were ‘open’ but had become irrelevant in the current system due to changes in functionality/expected behavior

We deleted the duplicate and the redundant defects. This resulted in a cleaner tool and a considerable decrease in the number of defects! The legacy issues, we listed in what we called the Baseline defects sheet. It had information like defect number, defect title and the version number in which it was reported. So we were left with only the reopened and the new defects to deal with 🙂

When this information became available, it was a huge relief to everyone concerned. The numbers now looked more manageable. It helped..

* the product management in better prioritization

* the dev team to focus on defects that would be prioritized for the release

* the testing team to focus on better testing of the system

Going forward, the Baseline defects sheet became a very important QA artifact; a ready reference that saved us a lot of effort and time. Defects found in regression were cross checked with the Baseline defects sheet. We did not log or report the bugs that were already in there. This considerably reduced the number of defects, i.e, the defects that were actually opened or introduced during the current sprint. For every release, the lower priority known issues along with the version number, were added to the baseline sheet. Overall, this helped in optimum utilization of available time and work force and decreased the stress levels in the team.

Eventually, the releases became very stabilized.. in fact they became non events! We sometimes managed to release a couple of days prior to the published release date. Seems too good to be true? This needs to be seen to be believed!! Of course this was not entirely due to base lining of defects. There were other factors involved too. But this was a huge contributing factor and in the bigger scheme of things it led to what is called as ‘sustainable productivity’ 🙂

2 Comments leave one →
  1. Jini permalink
    April 20, 2010 12:57 pm

    wow…that must have been a huge effort….something that I can relate to! defect baselining also helps when:
    1) we are doing integration testing
    2) multiple teams are involved in testing the same systems
    In this situation, team 1 and team 2 were testing both system 1 and system 2…ending up logging same issues increasing the count of defects…and stress levels of the dev team! And to add to it, the UAT was happening parallely…and the same defects were logged by UAT….the management could only see the huge number of defects and the situation was quite scary… then we got down to ‘defect baselining’….removed all duplicates….and prioritized the issues. helped a lot!
    nice post!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: