Testing an operating system for a modern smart phone is nothing short of a monumental task. Testers working on a phone platform are challenged with ensuring the critical functionality customers expect from a smart phone and the 1st party applications that expose rich integrated features. Also, the platform must enable and support 3rd party applications produced by many developers around the world desired by our many customers. This is an inside look at how 4000 engineers at Microsoft work together to design, develop and test the Windows Phone platform. We will discuss the product lifecycle, and explore the continuous integration development approach used in Windows Phone Division. Of course, we will primarily focus the discussion on testing strategies including high volume automation, “health” monitoring and assessments, self-host testing, and scenario-based testing approaches used to help deliver a unique smart phone experience designed to delight our customers.
There are many challenges in testing today’s complex software systems. One of the most challenging problems is testing various combinations of multiple inputs that determine the expected output condition or state. Imagine you are testing a feature with 20 interdependent parameters and each parameter has 5 possible input conditions. The total number of possible combinations is greater than a half trillion; at one test per millisecond it would take more than 3000 years to test all possible combinations. Which combinations do we test?
Combinatorial testing is a systematic way to effectively to produce a set of tests to evaluate input combinations. Coupled with data-driven automated tests combinatorial testing can significantly increase test coverage and help reduce overall risk.
This interactive hands-on workshop provides detailed examples of how to use a powerful, highly configurable combinatorial analysis tool that is freely available to systematically test complex interdependent parameters. Attendees will learn:
- How to analyze and model variables for input parameters or configuration settings.
- How to customize a model file for pairwise analysis.
- How to use advanced features of the PICT tool such as weighting, conditionally constraining interaction of specific variables, negative testing, seeding specific combinations, and output randomization.
- How to design an automated data-driven test to increase test coverage.