×
Namespaces

Variants
Actions

Archived:Symbian Signed Test Criteria

From Nokia Developer Wiki
Jump to: navigation, search

Archived.pngArchived: This article is archived because it is not considered relevant for third-party developers creating commercial solutions today. If you think this article is still relevant, let us know by adding the template {{ReviewForRemovalFromArchive|user=~~~~|write your reason here}}.

This article becomes obsolete as the Symbian Signed service was closed.

Article Metadata
Compatibility
Platform(s):
Symbian
Article
Created: aaumala (02 Nov 2011)
Last edited: aaumala (14 Jan 2014)

This document combines the previously separately held test criteria for basic 'Symbian Signed tests' and the more extensive 'Nokia tests'.
A print-ready PDF version of this document is available for download here.


Contents

Test criteria for 'Express Signed' and 'Certified Signed' test channels

Introduction

Background

These tests will be used by the Symbian Signed Test House to conduct the Symbian Signed testing for both Certified Signed and for the audit of an Express Signed submission. They must also be used by developers to ensure that their application satisfies the criteria for Symbian Signed prior to submission for either Express Signed or Certified Signed.

In case your application ends up in Express Signed Audit, if your application fails to satisfy this criteria, you will be excluded from the Express Signed service until you have successfully completed one round via Certified Signed with the same application.

This version of the test criteria is in effect from 5th January 2010.

What is included in the testing — and what is not

The tests contained in this document have been mandated by Nokia, in agreement with key partners in the community, as the minimum set of criteria for an application to be Symbian Signed.

We explicitly exclude any evaluation of the following from our test criteria:

  • Universal device coverage — submissions are tested on representative devices and not every device
  • Application functionality — we make no judgement on the functionality of the application as long as the test criteria are adhered to
  • Adherence to any published style guides

Finally, Symbian Signed does not evaluate the content of the application in any way and no judgement is made as part of the Symbian Signed process on the appropriateness of the content of any submission.

For some select partners or for applications specifically granted to access very sensitive capabilities/privileges such tests are needed, and, in those cases the test houses are asked to conduct an extra set of Nokia tests on top of the test cases described in this section.

Providing Guidance to the Test House

Due to the nature of some of the tests and the diversity of mobile applications, some guidance from the developer may be necessary to explain how certain test cases apply to their application. The notes accompanying each test case explain where guidance may be necessary. Any such guidance should be given by the developer in the submission statement. And, in some cases, it is necessary to fill in a waiver document at the Symbian Signed portal.

Testing VoIP applications

If you are testing a VoIP application, then there are some special tests which must be performed due to the nature of the application. Details can be found in the section "Additional Tests for VoIP applications".

Test Criteria

Submission Checks

General checks that are performed on all submissions, similar to the actual test cases in that the application submission can fail on any of these checks.

CHECK 1 — Correct UIDs

Application UIDs

The application UIDs must be from the protected UID range, and the UIDs must be allocated to the Symbian Signed account being used to submit the application.

Vendor ID

If a VID is specified, it must come from the correct UID range (0x70000000-0x7fffffff) and must correspond to the company submitting the application.

If no VID is specified, then a value of 0 should be included.

Package File UID

The package UIDs specified must be owned by the developer making the submission. For Symbian v9.x and later the package UIDs must come from the protected UID range and the UIDs must be allocated to the Symbian Signed account being used to submit the application.

If the package file UID is 0x10202BE9 (update to central repository) or 0x101F7989 (update to c32) then you must work with a device manufacturer or Symbian to get your submission signed.


CHECK 2 — Completed Submission Statement

The submission statement must be completed with all required information.


CHECK 3 — Malware check

The submission may not include any viruses, worms, malware, Trojan horses, time bombs or any other malicious code.

Nokia or appointed Nokia contractor will scan every submission coming through Symbian Signed -service. Nokia may share the application and information submitted with the application, including but not limited to the developer information to assure the submission did not include any malware.

If malware is found Nokia will suspend the user's account.

Please consider the golden rule: the User must always be in control. If the application is invoking any cost inducing actions, gathering user data, logging user actions or physical location - without the user's consent - the application might be labeled as spyware and rejected by this Check 4. If you application by nature must perform any of these actions it would be best to apply for a waiver for such behaviour, before submitting your application for final testing.


Symbian Signed Tests

TEST 1 — Installation

TEST STEPS

Before starting the test round, use a file manager to note the free user space available on the phone. You will need this information in test 8.

1

Install the application being tested.

The application must install without error, and without notes or issues that may confuse the end user.

2

During installation note the version number presented to the user.

The version number must match that specified during submission.

3

Verify that the application has successfully installed on the device by navigating to the area on the phone where new applications are installed.

The application should present one or more icon(s) on the phone.


Notes


The User must always be in control.

For any submissions which do not appear obviously once installed, the submitter must include details in the submission statement of how successful installation can be verified.

If the content does not appear obviously on the device once installed, and specific instructions are lacking in the submission statement, then this test will be failed.



TEST 2 — Application start/stop behaviour

TEST STEPS

1
Start the application by selecting the icon or following the steps outlined in the submission statement

Navigate to the Task Manager and check that the application appears there.

2
Close the application from the Task Manager.

Exit the Task Manager, and re-launch the Task Manager.

The application must no longer appear in the Task Manager.

3
Start the application as in Step 1.

Go to the Task Manager to verify that the application is running.

The application must appear in the task manager.

4
Close the application from within the application UI and then return to the Task Manager.

The application must no longer be running and must no longer appear in the task manager.

5
Restart the application as in Step 1.

Navigate to the Task Manager.

The application must once again appear in the Task Manager.


Notes The User must always be in control.

An application which must run in the background does not need to appear in the Task Manager or present a UI so long as the developer justifies this behaviour during submission.

All applications must have some way of verifying that they are running on the device, though, and the developer should provide this information.



TEST 3 — Application credentials

TEST STEPS

1

With the application running, check the name of the application displayed on the phone.

The application must display the same name on the phone as stated during submission.

2

Note the functionality of the application as it runs on the device.

The basic functionality of the application must match that declared during submission.

Notes

Step 1 does not apply to applications which do not have a UI

VoIP applications must present a UI in order to pass this test.



TEST 4 — No disruption to voice calls

TEST STEPS

1
With the application installed and running use a second phone to call the test device.

The incoming call must be indicated to the user on the test device.

2
Answer the call on the test device.

You must be able to conduct a conversation with the other party without interference from the application being tested.

3
End the call in the normal way on the test device.

The voice call must be ended.

4
From the test device, make a call to a second phone. Answer the call from the other device.

The call must be indicated on both devices, and you must be able to conduct a conversation with the other party without interference from the application being tested.

5
End the voice call from the second device.

The call must be ended on both devices.

6
Place a test call to the emergency 112 number from the device.

*Please check in your territory for the approved way to make test calls to the emergency services.


Notes


The User must always be in control.

If the application being tested has the MultimediaDD capability, and has audio functionality, then that functionality must be in use whilst this test is performed. Particularly, it should be checked that the audio from the application is faded down to allow the user to hear the telephone call.

VoIP applications will need to run this test both by having the handset held to the user's ear, and, by using a headset. The test should be run with a VoIP call in progress, and the incoming GSM call should be announced with call waiting tones.

TEST 5 — No disruption to text messages

TEST STEPS

1

With the application installed and running, send a text message to the test device.

The incoming text message must be notified to the user as per their alert settings.

2

Read the text message on the test device and choose to reply. Send the reply.

The reply must be received at the second device.

3

From the standby screen on the test device, navigate to the "new text message" option and create a new message. Send the message to the second device.

The message must be received at the second device.




TEST 6 — Auto-start behaviour

TEST STEPS

1

With the application running, find the settings for the application — either within the application itself or from the settings option on the device.

There must be an option which allows the user to enable/disable auto-start functionality.

2

Ensure that the setting for auto-start behaviour is disabled, and restart the device.

The application must not start on device boot.

3

Now change the setting so that auto-start behaviour is enabled for the application and restart the phone.

The application must start when the phone boots.


Notes


The User must always be in control.

If the application does not have auto-start functionality, then this test does not need to be run.





TEST 7 — No disruption to key device applications

TEST STEPS

1

Ensure that the contacts, messaging and calendar applications are populated with data and start the application as in Test 2.

After the application has been installed and used, the data entered into those applications must not be altered in any way without the user being aware.

2

With the application running, navigate to the messages application and create a new message.

Save that message to the drafts folder and then open and edit it.

Finally, delete the message from the drafts folder and delete a message from the inbox.

All of the above actions should be possible without interference from the installed application.

3

Navigate to the contacts application.

Create a contact, then edit that contact and then delete it.

The application should not interfere with any of the actions above without notifying the user and giving them option to avoid the change.

4

Navigate to the calendar application.

Create an appointment in the calendar. Edit the appointment and then delete it.

The application should not interfere with any of the actions above without notifying the user and giving them option to avoid the change.

5

Use the web browser on the device to go to a web page which is known to work on the network being used.

It must be possible to create a data connection and to access the web page selected.


Notes

The User must always be in control. The application must not access or make logs of User Data without the user's consent.

If the application, as part of stated functionality, makes changes to user data then an exception can be claimed here. The functionality must be described in the documentation with the application and all data other than that mentioned in the user guide must remain untouched as described in the test case.

The data used in this test case is also needed for Test 8, so leave the data on the device when proceeding straight into Test 8.



TEST 8 — Un-install

TEST STEPS

1

Stop the application as described in Test 2 and uninstall the application using the system installer.

The application must be uninstalled without error.

2

Following the same steps as in Test 1, navigate to where you would expect to see the application icon.

The application icon must not longer be present on the device.

If you used another method to verify successful installation in Test 1 then use this method to ensure that the application has been uninstalled.

3

Check the contacts, messages and calendar applications to ensure that that the data present in Test 7 is still present in those applications.

4

Using the same file manager as at the start of Test 1 check that the amount of user space available on the device is either the same as that found in step 1 or that any difference between the space available before and after fulfils the following criteria.

a) Excluding user-generated and downloaded content, the application leaves no more than 100Kb of data on the phone after uninstall

b) Any data left on the device after install matches the explanation given during the submission process


Notes

The User must always be in control.

The content left mentioned in step may only consist of inert data; no functional components may be left behind after un-install.

You should start this test with the application data from Test 7 still in place on the device.



TEST 9 — Device adaptation

TEST STEPS

Note: The following test steps should be run on the list of devices corresponding to the UIDs specified in the .pkg file. The lead device list can be found at Symbian Signed Device Table.

1
Install the application onto the device

The application should install on the device or present an error message to explain that it cannot install onto that device.

2
Launch the application.

The application should run on the device or present an error message to explain that it cannot run on that device.

3
Briefly examine the application whilst running.

UI elements should be functional and text should be readable in the main screen of the application.

4
If the device on which the application is currently being tested supports portrait and landscape screen modes, start the application and then switch between the screen modes.

The application should continue to be functional, and usable, in both screen orientations of the device, whether or not the application rotates in response to the screen mode change.

5
Close the application from the application UI

The application should stop running.

6
Uninstall the application from the phone.

The un-installation should happen without error and the application must be un-installed.


Notes

Applications which do not present a UI to the user in normal usage do not need to run this test.

On the primary device — on which all of the other test cases have been run — only step 4 of this test should be performed as all of the other steps of this test case are covered elsewhere.

Additional Tests for VoIP applications

Note that Test 3 and Test 4 both contain additional notes which apply to the testing of VoIP applications. Please read and apply these notes when running those tests on VoIP applications.

Test 10 — Additional emergency call testing for VoIP apps

TEST STEPS

Note: These test steps should be performed twice — once with a SIM card in the device and once without.

1

With the VoIP application running in the background, but with no VoIP call in progress, initiate an emergency call in the usual way.

The emergency call must be placed over the GSM/CDMA network successfully.

2

With the VoIP application running in the background with a VoIP call in progress, initiate an emergency call in the usual way.

The emergency call must be placed over the GSM/CDMA network successfully and the VoIP call should be terminated or placed on hold.

3

With the VoIP application in the background, and an emergency call active make a VoIP call to the device.

The incoming VoIP must be rejected, and the emergency call must not be interrupted.


Appendix A: Waivers

What is a waiver?

If your application doesn't meet one (or more) of the tests, then you must apply for a waiver. A waiver is a statement from Nokia that it is acceptable that your application does not meet a certain test case and that your application can be signed even with that omission.

Do I need a waiver?

If your application doesn't fulfill a particular test case, then you should first check the notes underneath the test as it the test may not apply to your particular application.

If your application is not covered by the notes, then you must submit a waiver request and have it approved before your application will be signed.

If you are using Express Signed, then you must submit your waiver request before making your submission as waivers will not be retrospectively approved for failed audits.

How do I apply for a waiver?

Details of how to apply for a waiver can be found at Waiver (Symbian Signed).

Decisions on waivers will be made as quickly as possible, and you will be informed as soon as a decision has been made.

Appendix B: Device Manufacturer Capabilities

What are device manufacturer capabilities?

The three most sensitive capabilities — AllFiles, DRM and TCB — are available to developers only when working in cooperation with Nokia. They are not available through the normal Symbian Signed channels.

How do I get permission to use those capabilities?

You submit a request for a permission to use these capabilities via Symbian Signed website; the 'AllFiles, DRM, TCB' tab. If the request is accepted at the portal you will get access to use the capability with a Development Certificate. To use the capability in a commercially/publicly distributed application you must also reach a legal agreement on the use of the capability with the device manufacturer - Nokia will be in contact with you when the initial approval has been given to use the capability, and you will be given access to the 'Invitation only' testing channel (please read more on that subject below).






Test criteria for 'Invitation Only' test channel

1 Introduction

This document describes the test cases that are applied to Symbian OS native applications in addition to Symbian Signed tests. All applications that will be embedded to Nokia devices must be tested according to these test cases. Meeting Nokia test criteria is a requirement for third-party applications that are delivered via Nokia sales channels, including:

  • Applications preinstalled to ROM, memory card, or hard-disk drive
  • CD-ROMs in devices' sales packages
  • Nokia Web/mobile Web page downloads

These test cases have been written for and focus on Nokia Symbian^3, and later, devices.

Some of the test cases are only applied to a limited group of applications that make use of Symbian Platform Security capabilities that require manufacturer approval (AllFiles, DRM, and TCB). Test cases that all applications with manufacturer-approved (=sensitive) capabilities must pass are defined in Section 3.1, ‘Test cases for applications requiring manufacturer-approved capabilities’. It is also strongly recommended that the part of the application having manufacturer capabilities be separated to its own SIS package if architecturally feasible.

These tests are done simultaneously with Symbian Signed tests, as described above. In addition to meeting the comprehensive requirements in these criteria, the tested application must comply with all applicable laws and regulations when developing, marketing, and selling the application to the end users.

This version of the test criteria is in effect from October 9, 2009.

2 Application UI and usability

Applications delivered via Nokia sales channels must use the same UI style and conventions as the system applications. It is not possible to list all the UI guidelines and test cases in this document, and therefore it is reasonable to refer to the UI style guides that Nokia has created and check that the application is in line with the UI style. The guidelines presented in the UI style guides must be followed. All significant deviations must be explained in detail and will be investigated case by case.
All UI style guides are available at http://www.developer.nokia.com.

3 Test cases

3.1 Test cases for applications requiring manufacturer-approved capabilities

The test cases (NOK-xx) defined in this section apply to all S60 applications from 3rd Edition onwards that require manufacturer-approved capabilities (AllFiles, DRM, TCB), even though the applications are not part of any Nokia total product offering (TPO) project.

NOK-01: Nokia values


Test description:

Applications must be in line with Nokia values and therefore must not have any references to the issues listed below.


Steps to conduct the test:

1. Start the application.
2. While using the application, check that there are no references to:

  • Violence against humans, excessive violence against animals, abuse of humans/animals.However, cartoon-style violence is permitted, provided there is no blood and no graphic, gratuitous, or humiliating depictions of death or violence.
  • Gratuitous depiction of violence against animals/nonhuman characters.
  • Depictions of sex, pornography, or pedophilic imagery.
  • Alcohol drinking.
  • Substance abuse including drugs and drug taking.
  • Use of knives/axes/swords/guns in human vs. human combat.
  • Racism.
  • Overtly political messages, contentious political/global events, for example, World War II, terrorism.
  • Religious imagery, unless in real-life context.
  • Gambling using money.
  • Stealing or robbing, for example, a bank robbery where the player is perpetrating the crime.
  • Promotions of criminality and criminal actions (depiction of criminal activity where the player is the perpetrator of the crime).
  • Material targeted and marketed solely at young children, for example, Teletubbies.
  • Nokia, for example, name or logo (except if explicitly agreed upon with Nokia).
  • Application must not intrude on the individual’s privacy.



Expected result:
The application starts correctly and does not have any references to the listed items.


Notes:


Exceptions:


NOK-02: Reserving MIME types on installation or startup


Test description:
If the application supports changing known MIME-type associations, the default value should be No. The default values in the device must be maintained.


Steps to conduct the test:

1. Start the application.
2. Check from the application settings that the default value for common content type associations is ‘No’.
Expected result:
The application starts correctly and the default value is ‘No’ (no content types are automatically associated with the application).
3. Check that the user is able to define which common content types are opened with the application.
Expected result:
The user is able to define which content types are opened with the application.
4. Remove application from device
5. Check that default content type associations are restored



Expected result:
The application is removed correctly and the default content type associations are restored.


Notes:


Exceptions:
If the application defines an own MIME type this case does not apply.


NOK-03: Memory card removal/insertion while auto-starting an application


Test description:
The removal or insertion of the memory card while application is auto-starting shall not cause problems.


Steps to conduct the test:

Precondition: Application is installed to the memory card.

Test case A:
1. Reboot the device after the application has been installed.
2. Insert memory card to the device before typing in the SIM PIN code.

Test case B:
1. Reboot the device after the application has been installed.
2. Insert memory card to the device right after typing in and accepting the SIM PIN code.

Test case C:
1. Reboot the device after the application has been installed.
2. Remove memory card from the device before typing in the SIM PIN code.

Test case D:
1. Reboot the device after the application has been installed.
2. Remove memory card from the device right after typing in and accepting the SIM PIN code.

Test case E:
1. Reboot the device after the application has been installed.
2. Remove memory card.
3. Reboot the device again.

Repeat the same test cases when the application is installed to C: (if supported by the application).



Expected result:
In all test cases check that the device switches on without problems and that removing/inserting the memory card doesn’t cause anything unexpected (device to reboot itself, jam, etc.).


Notes:
This test case applies only if the application supports auto-start.
This test case applies only if the application can be installed to or uses files on the memory card.


Exceptions:


NOK-04: Installation of an application with sensitive capabilities


Test description:
Applications with sensitive capabilities can be installed only to Nokia devices


Steps to conduct the test:

  1. Check if the application requires some of the following sensitive capabilities: AllFiles, DRM, or TCB.
  2. Open the .pkg file and check that the installation is limited to only Nokia devices:
    IF manufacturer = 2 ; (2 is Nokia)
    ;This part will then contain the installation information
    ;about the files of the application
    ELSE
    "badmanufacturer.txt"-"", FILETEXT, TEXTEXIT
    ENDIF

  3. Install the application to target device/s.

Expected result:</b>
Application installs correctly.
Applications that require TCB and/or DRM and/or AllFiles cannot be installed to any other manufacturer device than Nokia.
Installation must be limited to one or several devices as long as the rules mentioned above are valid.


Notes:
This test case applies only to applications that are targeted at Nokia devices with Symbian Platform Security and require manufacturer capabilities.


Exceptions:



NOK-05: Themes


Test description:
The application works correctly with different themes.


Steps to conduct the test:

1. Start the application.
2. Change the current theme to another one.
3. Use the application for five minutes.

Expected result:
The application works correctly without any usability problems (information is readable). If the theme supports a screen saver, make sure the screen saver runs correctly.

4. Change the theme to another device-specific theme.
5. Use the application for five minutes.



Expected result:
The application works correctly without any usability problems (information is readable). If the theme supports a screen saver, make sure the screen saver runs correctly.


Notes:
Two different themes which can be found from the device by default are to be used in the testing.


Exceptions:



NOK-06: Simultaneous connections


Test description:
The application can use simultaneous connections properly (if supported in the device and/or network and connections are used by the application).


Steps to conduct the test:

1. Start the application.
2. Start the data connection from the application.
3. Check the information displayed in the Connection manager.

Expected result:
The application starts correctly and the connection manager shows the application’s data connections correctly.

4. Start the Web browser and go to www.nokia.mobi.

Expected result:
The Web browser starts correctly and the www.nokia.mobi page is displayed.

5. Go to the Connection manager and disconnect the connection.

Expected result:
The connection can be disconnected from the Connection manager.

6. Go to the Web browser and go to www.nokia.mobi.

Expected result:
The Web browser starts correctly and the www.nokia.mobi page is displayed.

7. Start the data connection from the application.
8. Check the information displayed in the Connection manager.
9. Close the application.

Expected result:
The connection is not terminated if there are other applications using the same access point.

10. Go to the Connection manager and disconnect the connection.
11. Close the Web browser.



Expected result:
The Web browser closes correctly.


Notes:
This test case applies only if simultaneous connections are supported by the device and/or network. S60 3rd Edition devices and most networks do not support this feature. If there are two applications using the same access point it is shown as one connection in the Application manager.
WLAN connections do not apply in this test case.


Exceptions:



NOK-07: Profiles


Test description:
The application does not play sounds if the silent profile is active.


Steps to conduct the test:

1. Go to Profiles and activate the ‘General’ profile.
2. Start the application.
3. Play sounds in the application.

Expected result:
The sounds are heard.

4. Go to Profiles and activate the ‘Silent’ profile (use the default device settings).
5. Play sounds in the application.



Expected result:
Sounds are not played.


Notes:
S60 3rd Edition devices: When sounds are played and a notification is received (for example, incoming SMS/MMS/Bluetooth/IrDA), only a beep sound is heard even though the General profile is activated and some ring tone is defined for notification. This is a designed feature in S60 3rd Edition devices.


Exceptions:
Sounds are played in Silent mode if the event is triggered by the user. For example, when the user selects a tone in the Player application, a sound is played.


NOK-08: Offline profile


Test description:
The application follows the Offline profile correctly when making connections. Applicable only if the application can make connections or send messages.


Steps to conduct the test:

1. Go to Profiles and activate the ‘Offline’ profile.
2. Start the application.
3. Try to establish a connection (via CSD, GPRS, Bluetooth, or WLAN).

Expected result:
The application does not make CSD/GPRS connections.
The application is not able to make a WLAN or Bluetooth connection without asking for permission from the user.
A clear note to the user is displayed about the connection not being available.

4. Send an SMS/MMS using the application.



Expected result:
All the messages are saved to the Messaging application Outbox to be sent later.


Notes:


Exceptions:



NOK-09: WLAN connectivity


Test description:
The application can utilise WLAN correctly.


Steps to conduct the test:

1. Start the application.
2. Make a connection with the application to a WLAN network.

Expected result:
The application starts correctly and finds and connects to an available WLAN network.

3. Take the device outside WLAN coverage.



Expected result:
The application and the device do not crash.
The application and the device do not jam.
The application can change the connection automatically if the user has defined it. Available in S60 3rd Edition, Feature Pack 2 and newer devices.


Notes:
The test is applicable only for applications that utilise WLAN.


Exceptions:



NOK-10: Application interaction with WLAN browsing


Test description:
The application does not affect the device’s ability to browse over WLAN.


Steps to conduct the test:

1. Start the application.
2. Start the Web browser and go to www.nokia.mobi.

Expected result:
The application starts and runs correctly. The connection is made and the www.nokia.mobi page is displayed.

3. Switch back to the application.

Expected result:
The application runs correctly.

4. Switch back to the Web browser.
5. End browsing by disconnecting the connection from the browser application.
6. Switch back to the application.

Expected result:
Browsing is ended correctly and the application runs correctly after browsing over WLAN is ended.

7. Close the application.



Expected result:
The application closes correctly.


Notes:


Exceptions:



NOK-11: Multilanguage applications


Test description:
The application selects the device language during startup.


Steps to conduct the test:

1. Start the application.
2. Check that the application uses the device language if available. If the application does not have such a language, it will use the default language, which usually is English.
3. Change the device language.
4. When the device has restarted, start the application.
5. Check that the application uses the device language or the default language.
6. Repeat the test beginning from step 3 until all languages have been tested. Usually the tests need to be done with two to three devices with different language packages.



Expected result:
The application selects the device language during the startup.


Notes:
The localisation also includes Help.


Exceptions:
The selected device language is not supported by the application the default language will be used.


NOK-12: Backlight usage


Test description:
The application must not leave lights on.


Steps to conduct the test:

1. Start the application.
2. Leave the application on the foreground and check that the backlight is turned off after a period of no activity.

Expected result:
The application starts correctly and lights are turned off after a period of time when the application is on the foreground. The period of time is the same as defined in the device’s settings for light timeout.

3. Move the application to the background and check that the backlight is turned off after a period of no activity.

Expected result:
Lights are switched on when the application is put to the background (key press activity), but lights are turned off after a period of time when the application is in the background.

4. Switch to the application with the Application key.
5. Close the application.
6. Check that the backlight is turned off after a period of no activity.



Expected result:
The application closes correctly and lights are turned off after a period of time.


Notes:


Exceptions:
The first expected result can be ignored for applications that by nature require having lights on, such as:

  • Maps
  • Route instructions
  • Video
  • TV
  • Flash


NOK-13: Camera handling


Test description:
The application must free the camera resource. This applies only if the application uses the camera.


Steps to conduct the test:

1. Start the application.
2. Activate the camera functionality in the application.
3. Switch the application to the background.
4. Open the System Camera application.

Expected result:
The System Camera application can be used.

5. Close the System Camera application.
6. Bring the application being tested to the foreground.

Expected result:
The System Camera is closed correctly.
The application can use the camera.

7. Close the application.
8. Open the System Camera.



Expected result:
The application closes correctly
The System Camera can be used.


Notes:


Exceptions:




NOK-14: Music player handling


Test description:
The application does not affect the device’s ability to play music. This applies only to applications that have sound in them.


Steps to conduct the test:

1. Start the application.
2. Start the system music player and play some music.
3. Switch back to the application.

Expected result:
The application starts correctly.
The music player opens correctly and starts playing the music.
The application runs correctly and music continues to play in the background.

4. Switch to the music player.
5. Press the forward button in the music player.
6. Press the backward button in the music player.
7. Switch back to the application.
8. Press the volume button.

Expected result:
The music player functions work as expected.
The volume button also works when the music player is in the background.

9. Close the music player.
10. Close the application.



Expected result:
The music player closes without any notification or noise. The application closes normally.


Notes:
Dedicated music buttons in the device must be tested as well.


Exceptions:




NOK-15: DRM-protected content – Content sending


Test description:
DRM-protected content cannot be sent in decrypted form from an S60 system application or via a third-party application.


Steps to conduct the test:

1. Download DRM-protected content.
2. Send the content via the following applications: Media, Messages, File Manager, Music Player, and RealPlayer.
3. Send the content via the third-party application.



Expected result:
Content cannot be sent, or content was sent but cannot be used in the receiving device/computer.


Notes:
This test must be passed by an application that provides content-sending facilities either as a separate application or as a plug-in to S60 applications.


Exceptions:




NOK-16: DRM-protected content – File handling


Test description:
Content files handled by S60 system applications via a third-party plug-in or directly via a third-party application cannot be transferred in unprotected form.
Content files handled by the third-party application cannot be transferred in unprotected form when accessed from a desktop PC.


Steps to conduct the test:

1. Download DRM-protected content to the device’s internal memory.
2. Copy the downloaded file from the internal memory to a memory card/device hard disk via the following applications: Media, File Manager.
3. Copy the downloaded file from the internal memory to a memory card/device hard disk via the third-party application.
4. Verify that the copied content file is in encrypted form, for example, by trying to use it on a desktop PC.
5. Access the downloaded content via the third-party application from the desktop PC.
6. Verify that the accessed content file is in encrypted form, for example, by trying to use it on a desktop PC.



Expected result:
Content cannot be copied, or if it can be copied, the original DRM encryption is still in place and was not affected by any operation done by a third-party application.


Notes:
This test must be passed by an application that provides file-handling facilities (copying, backup/restore, access from a desktop PC) either as a separate application or as a plug-in to S60 applications.


Exceptions:




NOK-17: DRM-protected content – Backup/restore


Test description:
A third-party application does not remove protection from content files in backup/restore operations.


Steps to conduct the test:

1. Download DRM-protected content to the device’s internal memory.
2. Take a backup using the third-party application.
3. Restore the backup to another device.
4. Verify that the copied content file is in encrypted form, for example, by trying to use it on a desktop PC.



Expected result:
The original DRM encryption is still in place and was not affected by any operation done by a third-party application.


Notes:
This test must be passed by an application that provides file-handling facilities (copying, backup/restore, access from a desktop PC) either as a separate application or as a plug-in to S60 applications.


Exceptions:




NOK-18: DRM-protected content – Multimedia playback


Test description:
DRM-protected content cannot be played in S60 system applications via a third-party application or directly via a third-party application.


Steps to conduct the test:

1. Download DRM-protected audio and video content.
2. Initiate playback in the following applications: Music Player, RealPlayer.
3. Initiate playback in the third-party application.



Expected result:
The content cannot be played by the third-party application.


Notes:
This test must be passed by an application that provides multimedia facilities either directly or as a plug-in to S60 applications.


Exceptions:




NOK-19: Application key


Test description:
Application key functionality is in line with the S60 UI Style Guide.


Steps to conduct the test:

1. Start the application.
2. Put the application to the background and do a short key press to the Application key to open the Application menu.

Expected result:
The application stays in the background.

3. Do a short key press to the Application key again and check that the Idle screen is displayed.

Expected result:
The application stays in the background.

4. Repeat steps 2 and 3 a few times.



Expected result:
The application stays in the background.


Notes:
For more UI and user-experience guidelines, seeUI test criteria.
Exceptions:
This could be an exception to third-party task managers where the press of the Application key is to launch itself.



NOK-20 Application power consumption (PowerTest tool)


Test description:
Check that the application’s power consumption is on a reasonable level. (The PowerTest tool required for this test case is not yet publicly available. You can run similar tests using the Nokia Energy Profiler tool available at store.nokia.com.)


Steps to conduct the test:

1. Ensure that you are using a clean device (if you are not sure, format the device before you continue).
2. Install the PowerTest package (and the needed additional components).
3. Install the application to be tested.
4. If necessary, perform the setup steps of the application to be tested.
5. Start the PowerTest application.
6. Start a new test from the Options menu and select the application that should be tested.
7. Wait for the automatic test to finish (35 min). You will hear a beep when the test has finished.
8. Bring the PowerTest tool to the foreground and click OK to start the active test phase.
9. Use the test application as you would use it normally. Note that this is not a stress test.
10. Send the test results to your PC and attach the .png image to your final test report.

For more detailed instructions, refer to the user guide supplied with the PowerTest tool.



Expected result:
The application should not drop the battery lifetime below 4 days (the result should not be “fail”).
The power consumption should not be excessive for this type of application (< 40 mAh/h in standby mode). If the tested application uses WLAN, Bluetooth, or another network connection, the power consumption is higher than usually. The power consumption is also higher when playing MP3s or listening to the radio.


Notes:
Failing this test is not fatal for now. The result should be reported as an observation only.
The battery level does not need to be full when running this test.
The device must not be connected to a charger when running this test.


Exceptions:
The tool works on devices compatible with S60 3rd Edition, FP1 and onwards. This test cannot be run on older devices. This test will not be run for ‘always on’ type of applications, such as email or antivirus applications.


3 Test cases

3.2 Test cases for Nokia TPO applications

The test cases (TPO-xx) defined in this section apply to applications that are part of a Nokia TPO project. The applications included in the TPO must also pass the NOK-xx test cases defined in Section 3.1, ‘Test cases for applications requiring manufacturer-approved capabilities’.


TPO-01: UI, softkey labels, selection key, and Options menu


Test description:
The application is in line with the relevant UI style guide.
Softkey labels are in line with the S60 UI Style Guide. Selection key functionality is in line with the S60 UI Style Guide. The options menu is in line with the S60 UI Style Guide.


Steps to conduct the test:

1. Start the application.
2. Check that the softkey labels in every screen and menu item are in line with the UI style guide.


Expected result:

  • The left softkey (LSK) is Options, Select, Yes, or OK (positive effect).
  • The right softkey (RSK) is Back, Exit, Cancel, or No (negative effect). Even if LSK has nothing.
  • RSK is labeled Exit in the idle state of the application and closes the application. RSK could also be Hide in some applications as per the nature of the application.
  • RSK is Done in editable forms when LSK is Options. If there is no Options menu, LSK is Done and RSK is Cancel.
  • The selection key (middle softkey, MSK) opens the focused item and selects one option. It results in the same function as the first item in the Options menu. With dialogs, the MSK is the same as the LSK.
  • If MSK defines a label (this is possible in S60 3rd Edition, Feature Pack 2), the MSK must do the function specified in the label of the key.
  • The selection key must not directly activate any function that the user would not expect in the given situation. The first item in the Options menu is the functionality that is triggered by the selection key or joystick press. This is usually ‘Open’, ‘Select’, ‘Change’, etc.
  • ‘Exit’ must be the last item in the Options menu.
  • The Options menu must not contain items that cannot be triggered from the current application state.


Notes:
For more UI and user-experience guidelines see http://www.developer.nokia.com/usability.

Exceptions:

TPO-02: Consistent UI


Test description:
The application UI is consistent throughout the application.


Steps to conduct the test:

1. Start the application.
2. Use the application and check the consistency of

  • Terms
  • Colours
  • Layout
  • Sounds
  • Action sequences

In:
i. Intro/splash screen
ii. Main menu and submenus
iii. Help
iv. About
v. Settings menu


Expected result:
The UI is consistent throughout the entire application.

  • The same terms are used for the same things throughout the application.
  • The terminology in the Settings menu is consistent with the system applications.
  • Setting titles are clear and easy to understand and correspond to the actual item the user is about to modify.
  • Colours and layout are used consistently throughout the application. Use of a different colour can help indicate a change or deviation.
  • The same sound is not used for different purposes.
  • The actions are sequenced in the same way throughout the application.


Notes:
For more UI and user-experience guidelines see http://www.developer.nokia.com/usability.


Exceptions:


TPO-03: International diversity


Test description:
The application supports international diversity.


Steps to conduct the test:

1. Start the application.
2. When using the application, check the following areas:

  • Different characters, numerals, special characters, and diacritic characters
  • Date and time formats
  • Weight and measure formats
  • Telephone number and address formats
  • Names and title formats
  • Numeric and currency formats
  • Capitalisation
  • Icons, colours
  • Etiquette, policies, tone, formality, metaphors
  • Terminology



Expected result:

  • Different characters, numerals, special characters, and diacritic characters are appropriate for the target audience of the application.
  • Date and time formats are appropriate for the target audience of the application.
  • Weight and measure formats are appropriate for the target audience of the application.
  • Telephone number and address formats are appropriate for the target audience of the application.
  • Names and title formats are appropriate for the target audience of the application.
  • Numeric and currency formats are appropriate for the target audience of the application.
  • Capitalisation is appropriate for the target audience of the application.
  • Icons and colours are appropriate for the target audience of the application.
  • Etiquette, policies, tone, formality, and metaphors are appropriate for the target audience of the application.
  • Terminology is appropriate for the target audience of the application. In particular, pay attention to the use of terminology that could disturb, for example, ‘Abort’.



Notes:

Exceptions:

TPO-04: User’s guide or functionality specification


Test description:
A user’s guide, functionality specification, or other document describing the application’s features and functions must be provided.


Steps to conduct the test:

1. Check that a user’s guide, functionality specification, or feature list is provided.
2. Familiarise yourself with the documentation.
3. While using the application, check that the application behaves as described in the documentation.



Expected result:
A user’s guide, functionality specification, or other document is provided. The application matches the provided documentation. The application has all the documented features and it does not have any undocumented features.


Notes:


Exceptions:
If the application is very simple and easy to use and includes Help, other documentation is not mandatory.



TPO-05: Help, About, and Settings.


Test description:
The application must have Help and About, and they can be accessed from the main menu. The settings must function as expected.


Steps to conduct the test:

1. Start the application.
2. Check that Help exists and can be accessed from the main menu.

Expected result:
Help exists and offers help on using the different application features.

3. Check that About exists and can be accessed from the main menu.
4. Check that the appropriate information is available in About.

Expected result:
About exists and includes the following information:

  • Application name and version.
  • Developer’s name.
  • Contact information for end-user support.


5. Change some of the application settings (if applicable) and return to use the application.



Expected result:
New settings are available right after they have been set and the user closes the Settings menu.


Notes:


Exceptions:




TPO-06: Usability testing — Wait note


Test description:
The application must never leave the user in a position where the state of the application is unknown or the application appears to be unresponsive (that is, may have locked up).


Steps to conduct the test:

1. Start the application.
2. Use the application.
3. Observe the following parts of the application:

  • Main menu and its submenu screens.
  • Application usage.




Expected result:

  • The user must be notified if he or she has to wait for more than three seconds for an activity to complete. In addition, the notification must appear one second after launching the activity.
  • An example: A note containing a progress or wait graphic. Whenever possible, the user should be able to stop the operation.



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.


Exceptions:




TPO-07: Usability testing — Text and selection


Test description:
The application follows usability guidelines and provides a good user experience.


Steps to conduct the test:

1. Start the application.
2. While using the application, check the following areas of the application:

  • Intro/splash screen
  • Main menu and submenus
  • Help
  • About
  • Settings menu
  • Main functionality of the application



Expected result:

  • Error notes are valid, informative, and easy to understand.
  • Terminology is not too technical and is familiar to target users
  • Text is not ALL CAPS; only the first word begins with a capital letter.
  • There are no truncated texts or abbreviations.
  • There are no grammatical mistakes.
  • There are no terms and sentences that may cause misunderstandings.
  • Images do not substitute for essential text.
  • Editable components are in the correct format when the user starts to edit them (numeric, alphabetic).
  • Editable components do not accept invalid input and give an understandable error note if invalid input is given.
  • Radio buttons and multiselection lists are used correctly. If the user can select only one option, a radio-button list is used.



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.


Exceptions:
Common abbreviations (for example, ‘i.e’, ‘etc.’) can be used if acceptable for the application type.


TPO-08: Usability testing — Layout and navigation


Test description:
The application follows usability guidelines and provides a good user experience.


Steps to conduct the test:

1. Start the application.
2. Check these areas of the application:

  • Intro/splash screen
  • Main menu and submenus
  • Help
  • About
  • Settings menu
  • Main functionality of the application




Expected result:

  • Every view is easily readable.
  • The screen is fully used when applicable.
  • Text has good contrast with the background.
  • When possible, the application uses prefilled forms to enable a more pleasant form-filling experience.
  • By default, lists loop and have a scroll bar on the right side (from S60 2nd Edition, FP2 onwards).



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.


Exceptions:
Common abbreviations (for example, ‘i.e’, ‘etc.’) can be used if acceptable for the application type.



TPO-09: Usability testing — End key and Cancel/Back


Test description:
The application follows usability guidelines and provides a good user experience.


Steps to conduct the test:

1. Start the application.
2. While using the application, check:

  • End key functionality
  • Availability of ‘Cancel’ and ‘Back’




Expected result:

  • Pressing the End key (red key) closes the application (from S60 2nd Edition, FP2 onwards).
  • ‘Back’ returns the application to the previous state.
  • ‘Cancel’ stops an action that the user triggered, and returns the application to the state where it was before the user triggered the action.



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.


Exceptions:
Applications that run constantly may go to the background when pressing the End key if the RSK is Hide.



TPO-10: Installation into the correct applications menu folder


Test description:
The application creates and installs into the defined destination in the application menu.


Steps to conduct the test:

1. Install the application to a device.
2. Check that the application is installed into the correct folder, as defined in the application .aif file (prior to S60 3rd Edition) or app_reg.rss file

(S60 3rd Edition).



Expected result:
The application installs into the folder defined in the application’s .aif or app_reg.rss file. If the defined folder does not exist, the application creates this folder and installs into it.


Notes:
If the installation folder is defined in the .aif or .app_reg.rss file, the developer should provide the file when submitting the application to tests and let the test house know the installation folder.
The app_reg.rss file has a ‘group name’ attribute that has the value of the folder where the application will be installed to. Please note that the uninstaller cannot delete this folder.
This test case applies only to Nokia TPO projects where the product program has defined some other target folder than the default. If the developer and test house don’t have this information available, then the application should be installed to the default folder and this test case can be ignored.


Exceptions:
If the subfolder information does not exist then this case is ignored.



TPO-11: SIS file naming


Test description:
The SIS file name must include the application name and correct version.


Steps to conduct the test:

1. Check the name of the application’s .sis file and compare it to the available information in About and the .pkg file.
2. The application’s .sis file name must include the application name and correct version, and they are the same as in About.



Expected result:
Application’s .sis file naming follows this guideline:

<application name>_<target device/platform version>_<version>.sis
The version number is the same as the one defined in the .pkg file and About for

the application.



Notes:
Sis file naming example:

testapp_nokiaN71_v_1_3_0.sis

or

testapp_S60_3_0_v_1_3_0.sis


Exceptions:




3 Test cases

3.2.1 Test cases for Nokia TPO applications with touch UI

The test cases (TPOT-x) defined in this section apply to applications that have a touch UI and are part of a Nokia TPO project. These applications must also pass the NOK-xx test cases defined in Section 3.1, ‘Test cases for applications requiring manufacturer-approved capabilities’, and the TPO-xx test cases defined in Section 3.2, ‘Test cases for Nokia TPO applications’.


TPOT-01: UI element size (touch UI only)


Test description:
The application UI elements must be large enough for a pleasant user experience.


Steps to conduct the test:

1. Start the application.
2. While using the application, check UI element sizes and visible areas vs. active areas in the following application features:

  • Main menu and submenus
  • Settings menu
  • Main functionality of the application




Expected result:
The target minimum size of a UI element is 7 x 7 mm. In the Nokia 5800 XpressMusic device, the 7 x 7 mm is equivalent to 70 x 70 pixels.
The active area of the component must not be smaller than the visible area of the component



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.
The test house will randomly choose one item per feature for measuring. The developer is expected to make sure that all the items are in the correct size range.



Exceptions:
When components are located near the edge of the display, the active and touchable area can be fully extended to the edge of the display (that is, beyond the component’s visible graphics).
For example in games where the game board is essential to fit the screen (for example, Sudoku) or in applications that are meant to be used with Stylus, the size of the UI element can be smaller than 7 x 7 mm (equivalent to 70 x 70 pixels).


TPOT-02: Touch interaction — Select and activate (touch UI only)


Test description:
Touch interaction must follow the S60 UI Style Guide.


Steps to conduct the test:

1. Start the application.
2. While using the application, check the interaction styles in the following areas of the application:

  • Main menu and submenus
  • Settings menu
  • Main functionality of the application




Expected result:
The following will operate with one tap:

  • Buttons
  • Icons
  • Items within Options menus

The first tap selects the item and the second tap opens the item unless another item is selected.



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.
The test house will randomly choose one item per feature for measuring. The developer is expected to make sure that all items behave correctly.


Exceptions:
It is possible to open or activate an item from a focusable view directly with a single tap if the focus is already on the given item.



TPOT-03: Fixed toolbar (for touch UI only)


Test description:
Only available buttons are enabled in the fixed toolbar.


Steps to conduct the test:

1. Start the application.
2. Check the functioning of the fixed toolbar in the main use of the application.



Expected result:
The fixed toolbar buttons that do not do anything in the situation/view are dimmed, that is, are in a temporarily disabled state.



Notes:
For more UI and user-experience guidelines and a usability checklist, visit www.developer.nokia.com/usability.
The test house will randomly choose one item per feature for measuring. The developer is expected to make sure that all items behave correctly.


Exceptions:
The application does not use the fixed toolbar.



4 Terms and abbreviations

TermDefinition
ApplicationAn application is installed physically into a device and executed locally. Applications are developed with Java™ (includes

applets/MIDlets), C++, etc. A chess game is an example of an application.

BS7925-1 Standard for application testing terminology.
DRM Digital Rights Management.
CSDCircuit-switched data.
Error A human action that produces an incorrect result. (BS7925-1)
FailureA deviation of the software from its expected delivery or service. (BS7925-1)
FaultA manifestation of an error in software. If encountered, a fault may cause a failure. (BS7925-1)
Functional SpecificationA document that describes the expected application usage. It may include Release notes, Help, User’s Guide, or

Requirement specification.

Java terminologyJava™ Platform, Micro Edition (Java™ ME), is a Java edition that consists of different consumer

electronics’ Java specifications and technologies.
Mobile Information Device Profile (MIDP) is a Java profile for Java ME; it provides an open application execution environment for the S60 platform.
A MIDlet is an application as specified under the MIDP specification.

MMSMultimedia Messaging Service.
Nokia Testing Nokia Testing describes the testing process, which consists of Symbian Signed and additional Nokia tests performed for the

application. A third-party software application that has passed this process will be signed with Symbian Signed.

OSOperating system.
S60An S60 user interface is a smartphone platform designed for Symbian OS by Nokia. It supports mobile browsing, multimedia messaging (MMS),

and content downloading, as well as a host of personal information management (PIM) and telephony applications.

S60 platformS60 is a device software platform for smartphones that Nokia licenses to other mobile device manufacturers as a source-code

product.

SISSymbian Installation Script (.sis). The makesis tool available in the SDK reads the .pkg file and includes all files listed in it. The

resulting file, the Symbian Installation Script, is an executable file for the application installer, which provides a standardised user interface for

installing any Symbian application onto the Symbian device.
SMSShort Message Service.
Third-party software applicationAn application that has been created by a third-party company. In this document, all references are to

applications created for Nokia mobile device platforms.

TPOTotal product offering. This refers to the entirety of a particular build of a given device — that is, all its hardware, software, and

features.


5 References and related documentation

Symbian Signed process description and test criteria available at www.symbiansigned.com.

This page was last modified on 14 January 2014, at 11:08.
343 page views in the last 30 days.
×