Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries over the next few weeks. Thanks for all your past and future contributions.
Using Mailchimp to manage a Windows Phone Beta test
In Windows Phone 8 we can distribute our apps as "beta". This allows us to publish our app without passing certification to a number of users identified by their LiveIDs, so they can test the app and sent feedback to us.
Windows Phone 8
This article discusses the importance of using the beta distribution options before publicly launching an app in the store. It explains how to create a beta distribution, and how to manage it and distribute it to get the most value while reducing the time needed to manage all of the needed data.
Lets start by taking a look at what Windows Phone beta distribution offers to us.
Windows Phone Beta distribution
At first sight, the beta distribution option allows us to distribute our app to a group of people who wants to test it and sent feedback and bugs to us. We could do the same sending to the users the generated XAP file, but for that each person to have the Windows Phone SDK installed and a developer unlocked phone to test the app. Even the persons in the beta doesn't need to be developers, it could be great to have some final users testing our app, so they doesn't are "contamined" by the technology itself. We need to simplify the beta access. Distributing it using the Windows Phone store allow each tester to download it from the phone, only clicking on a link.
To publish a beta version of our app, we only need to select "beta" option in "Distribution channels" section of the first step. This section is located in the collapsed option "More options" at the end of the page. After selecting this option, the textbox below is enabled to add the LiveIDs of the beta participants. (With a max. of 10.000 liveIDs). Also is good to know beta versions doesn't expire. Some months ago Microsoft removed the 90 days beta limitation from the store. Now if you want to discontinue your beta, you need to unpublish it.
Here we can get a first problem: As the beta distribution allows up to 10.000 LiveIDs, How can we manage all this IDs? There are a lot of ways to do that. From a text file to an excel document. But out there you can find systems that add more useful features to the IDs management. In the vast majority of cases, LiveIDs are also people personal email account, and that is going to be our primary communication way with our testers. For each beta version we publish, we need to send details about what features it have, what bugs are fixed, and the identified but not fixed bugs or the features identified but not included. This way the testers can try the app and don't invest time in things we have discovered already. Personally, i like one tool over others. Is free, allows you to manage suscriptors, sent communications by email and have reports about who read the mails or who click on mail's links. It is MailChimp.
Managing our beta with MailChimp
MailChimp is a tool mainly for email campaings. It's widely used in web development for sending notifications to users. It even has APIs so we can connect our apps with it and get data, send emails by code in an easy way. It gave us a lot of interesting features:
- Create lists of users with custom information, like LiveIDs.
- Create and personalize subscription forms, and expose them automatically in a public URL.
- Create mailing campaigns.
- Programming the date and time to send a campaign.
- Get statistics about open/click rate in one campaign.
- Export the subscriptors list to csv.
- It's FREE!!.
So, we have a free tool to manage our users, do automatic communications without exposing our private mail to be blacklisted, get useful statistics and get a csv file with all our subscribers information. This enable us to expand our beta program to more and more people, because the management time needed is low enought to include more people in it. And the more, the better when talking about beta testing.
After registering a new account in MailChimp, the first thing we need to do is to create a new List in which we can include our testers:
One of the lists options is to manage "sign up forms", to create and customize the form people can use to subscribe to our beta. The forn designer is very powerful and allows us to totally customize the look & feel of our page:
- Keep the form simple enought, only ask for the needed data. The most complex the form is, the most probable a user skip it.
- Customize it a bit. You don't need to invest 5 days of design work on it, but at least invest some hours and give it a look familiar with your app.
- Translate the form to each language you think you want your users to speak. This way you make your testers comfortable with the subscription proccess.
When you had your form finished, in the upper left of the page you are going to find the public URL of that form. Launch it to the world. Use every channel you has access to: twitter, facebook, forums... spread the word about your app and the beta phase. As we said already, in this stage the more the better for us.
Now we are ready to start our beta. We have some people registered in our Mailchimp list, one beta version ready to be launched but... wait a minute! how are we going to receive the feedback about our app? Are we going to use our personal email, the work one? The best option i come to is to have a different email for this kind of things. Maybe an email for all our apps feedback GreatApps@outlook.com, or maybe an email address just for each app, so we don't lost emails. In my case, i have only some apps published, i have an email for each app, for example for PHOTOLab i have firstname.lastname@example.org. This way all the feedback and bugs are easy to track and manage.
Sending communications to our testers
We are at it:
- Beta version published: check!
- LiveIDs added: check!
- Email account for app: check!
- Mailchimp subscriptors: check!
So, is time to communicate with our subscriptors. As we said before, MailChimp allows us to send mail campaings. A mail campaing is basicly an email sent to all our subscribers. MailChimp allows us to make HTML emails, multimedia emails, or text emails. i prefer text emails because they aren't frequently blocked and more easy to write in different languages. What to include in the email? This is up to you, but at least, be grateful with the users, and clearly state:
- What bugs are fixed in this version.
- What features are included in this version.
- What known bugs aren't fixed in this version.
- What features aren't included in this version.
This way, users know where to take a look, and what to overcome. This is an example email from PHOTOLab:
And Voila! now, you only need to repeat the proccess over and over again. Once you feel the important bugs are fixed and the app is feature complete, at least for a first version. What number of versions are needed for that? It depends on every app, in the case of PHOTOLab i published 5 beta versions. Even after publish your first version to the public store, continue doing betas if you feel you need it. They are free of charge, so use them!!.