MSDN Blog Postings

Before I came to Microsoft, we always joked about the quality of the software products – “Don’t use V1.0” – everyone has heard that.  But now that I’m at Microsoft, I see what goes on behind the scenes to try to make sure the products are release quality before customers get their hands on them. It’s pretty impressive.

Like any company, Microsoft balances shipping products that people want (and often demand!), and ensuring there very few bugs. Everyone knows the longer you test a product, ideally the fewer bugs are in it when it ships. On the other hand, the longer you test, the less chance you get to make the window of opportunity!

First of all, products are built every day. Microsoft builds “from-source” using timestamps in the source control system to ensure they can re-build a product, in a known way. Products are not only compiled and linked, they are built several ways: release-quality, debug-quality, code-coverage, and up to 6 different ways to inform developers and testers that the code being shipped is of reasonable quality, and meets the legal requirements set by Microsoft. Additionally, each day localized versions of the product have to be produced, and frequently each build is run through a set of Build Verification Tests (automated tests to inform the team that the build made the “first cut” or not).

Once the product is “built”, it is then tested using more automation than I’ve ever seen outside Microsoft. The tests cover the GUI and the API parts of the product. Every day, someone on the product team looks over those results and ensures they mean something, and are not just a set of percentages. The automated testing is designed to make sure that the maximum amount of code-paths are executed to ensure the highest ship quality possible. Not many companies invest the time it takes for developers to write test cases, but Microsoft does - it’s a serious investment in product quality!

Manual testing is happening all the time. Because testers are familiar with the product objectives, they catch lots of things that the automated tests don’t. Also, they sit in the same area as the developers, so there is rapid closure of many issues. Testers are often engaged before the product has been formally built the first time – I know that sounds crazy, but by starting early, reviewing what the developers are going to do, and planning the tests cases ahead of time, the testers stay in-step with developers as the product development cycle progresses.

Every product team has a third set of people to ensure the overall product metrics are met. The Test Leads, Development Leads, and Program Managers regularly meet to review all bugs, and ensure progress is being made. This helps us balance business objectives with product quality. Several times quality issues will force a reevaluation of ship schedule, which tells me Microsoft is serious about quality.

Many products go through a “dogfood” phase prior to release – which means that the people inside Microsoft use the product just like customers and provide critical feedback to the development team. Before formal release to customers, Microsoft has a variety of programs in place to allow previews of upcoming releases so customers get a “front row seat” with the product – again, to ensure higher quality in the shipping product!

And finally, Microsoft ships the product! But the quality doesn’t stop there – there is a massive amount of personnel supporting all these products that have reached the customer!

When you think about all the processes used to ensure one product gets out the door to meet the market demand and quality metrics, it becomes truly impressive when you look at the list of products that have shipped to know that all these metrics are applied to every product.

Thanks for letting me give you an inside tour of Microsoft release quality!

- Bruce