How to run a successful beta test on Windows Phone
This article provides tips and guidance for how to run a successful beta test on Windows Phone.
Consumers are extremely effective testers, and increasingly unforgiving of poorly behaving apps. As a negative rating in the early stages can have a lasting effect on the success of an app (even if defects are later fixed), effective beta testing is essential. Beta testing provides the same "real world" testing, but by a friendly audience who are excited to help improve the app.
Windows Phone allows you to beta test an app by distributing a customer-ready release build to a limited number of users for a limited time. The reasons and mechanisms for beta testing are well documented in Beta testing your app and in-app products (Dev Center): in essence you supply a list of Microsoft Account IDs and their associated email addresses, DevCenter gives the app a basic test and sends you a link which your beta test group can use to download the app.
What is not so well documented is how to run a successful beta test on Windows Phone. This article provides a number of tips and tools that can help.
How to get people interested
Any user with a Microsoft Account and a Windows Phone device is a potential beta tester. Ask for help in any community you're part of, including developer communities, special interest groups, workshops. Any beta testing is better than nothing, so don't forget your friends and family.
There are a number of "media" organisations who have great social media channels for Windows Phone app users. It is worth reaching out to them and seeing if they will cover the launch of a beta:
There are also some local communities that can help find beta testers - for example PocketPC.ch can help find beta testers for German language users.
If you're working through an app incubator program (like AppCampus) they may be able to help promote your beta test.
How to deal with signups
For very small-scale beta test developers can use any number of tools, ranging from having to email the developers directly, Google Docs, tweeting or posting in a forum.
These approaches do not scale well because of the need to maintain the list of people interested in the app separately of the email solution (resulting in need for copy-pasting etc). For larger scale testing, consider using a CRM or other system to manage large numbers of users and emails.
At time of write, we only cover one: MailChimp.
MailChimp is a emailing marketing service. It provides a sign-up form with which you can collect information needed from potential beta testers including name, device type, Microsoft Account ID etc. It gives you a list of your signups which you can email to, and also send to Microsoft Dev Center for adding to the beta.
MailChimp also allows you to segment the list of testers. Segmenting by signup date or "batch" allows you to manage adding them to the list in DevCenter, and to just send the signup link to new testers (instead of everyone).
Tip: Getting users from MailChimp into the Windows Phone developer centre
The Windows Phone developer centre requires that beta users be added to a semi-colon delimited list. This can be a tiresome if there is a significant number of users. To make this process easier I made a little Windows Forms app that takes in a CSV form (which you can export from MailChimp or Google Docs), verifies all of the emails and then pastes them to the clipboard separated by semi-colons. Download and find out more about it here.
Make it easy for testers to provide feedback
Making it easy for testers to provide feedback will help in getting the best bug reports and enhancement suggestions. One approach is to add a "feedback" link to all app menus so that there is never more than two clicks to ask for help. If "contextual" information can be provided by the app (for example the current page, stack, or crash information), this should be pre-seeded in the email.
The tips/techniques described in this article were collated from experience beta testing Foundbite for Windows Phone 8 with around 800 users. Please feel free to enhance this article with other tips, tools and suggestions.
Note: This article was originally posted as Running a Beta on Windows Phone (blog). The author has kindly posted it here and we've modified it for wiki format.