×
Namespaces

Variants
Actions

Building a communication system for visually impaired users on the Asha software platform

From Nokia Developer Wiki
Jump to: navigation, search
Featured Article
29 Dec
2013

This article explains how to build a simple but effective communication system for visually impaired people, using the features offered by the Asha software platform.

Note.pngNote: This is an entry in the Nokia Asha Wiki Competition 2013H2.

Article Metadata
Code ExampleTested with
SDK: Asha SDK 1.1
Devices(s): Nokia Asha 501
CompatibilityArticle
Created: jappit (12 Dec 2013)
Last edited: kiran10182 (29 Dec 2013)

Contents

Introduction

This article shows the building blocks of a communication system designed for visually impaired users. The system includes various features that helps users to easily read received SMS messages, including:

  • RSVP reader
  • Text-to-Speech
  • audio/vibrating notifications


Messaging app for visual impaired people

The app illustrated in this article can be used in various, different scenarios:

  • as a one-way communication system, allowing visually impaired users to receive important notifications from a single source (for instance, a doctor)
  • as a two-way communication system, allowing visually impaired users to chat via SMS messages

The RSVP reader

Rapid Serial Visual Presentation presents to the user words one by one in a rapid succession, allowing the user to focus on one word at a time. The following sections show how a RSVP reader can be implemented.

Text splitting and drawing

In order to be presented word by word, text needs to be split into words. Since Java ME has no inbuilt split feature, the following code is used to break the text to be visualized:

String[] splitText(String text)
{
Vector words = new Vector();
 
int wordStartIndex = 0;
int spaceIndex = 0;
 
int counter = 0;
 
while((spaceIndex = text.indexOf(' ', wordStartIndex)) > 0)
{
if(spaceIndex > wordStartIndex)
{
String word = text.substring(wordStartIndex, spaceIndex);
words.addElement(word);
}
wordStartIndex = spaceIndex + 1;
}
if(wordStartIndex < text.length() - 1)
words.addElement(text.substring(wordStartIndex));
 
String[] wordsArray = new String[words.size()];
words.copyInto(wordsArray);
 
return wordsArray;
}

Note.pngNote: For better results, text could be also split in correspondence with punctuation characters.

Once split, text has to be presented a word at a time. For better results and more control on the UI the app uses low level graphics, and specifically the Canvas object, whose paint method is defined as follows:

protected void paint(Graphics g)
{
int canvasWidth = getWidth();
int canvasHeight = getHeight();
 
g.setColor(bgColor);
g.fillRect(0, 0, canvasWidth, canvasHeight);
 
if(words != null)
{
String word = words[currentWordIndex];
 
Font font = DirectUtils.getFont(Font.FACE_SYSTEM, Font.STYLE_PLAIN, fontSize);
g.setFont(font);
 
g.setColor(textColor);
g.drawString(word, canvasWidth / 2, canvasHeight / 2, Graphics.HCENTER|Graphics.BASELINE);
}
}

In the above code, the words array contains the text already split into words with the method described before, and currentWordIndex contains the index of the word to be displayed on the screen.

Since it is possible to have words that are wider than the available screen width, the code should take those into account. The method used below is to wrap the word in order to fit the maximum possible number of characters onto each line. Other approaches (for instance, reducing the text size) are possible.

if(font.stringWidth(word) <= canvasWidth)
g.drawString(word, canvasWidth / 2, canvasHeight / 2, Graphics.HCENTER|Graphics.BASELINE);
else
{
Vector wordPieces = new Vector();
 
int pieceStart = 0;
String piece;
for(int i = 0; i < word.length(); i++)
{
piece = word.substring(pieceStart, i);
 
if(font.stringWidth(piece) > canvasWidth)
{
wordPieces.addElement(piece);
pieceStart = i;
}
}
if(pieceStart < word.length() - 1)
wordPieces.addElement(word.substring(pieceStart));
 
int yStart = (canvasHeight - fontSize * wordPieces.size()) / 2;
 
for(int i = 0; i < wordPieces.size(); i++)
{
g.drawString((String)wordPieces.elementAt(i), canvasWidth / 2, yStart + fontSize * i, Graphics.HCENTER|Graphics.TOP);
}
}

Animating the visualization

Now that the main logic is ready, words need to be automatically presented one by one in a rapid succession. The following code uses wordInterval for the default time to display each word, and uses a Java Timer to perform the animation:

presentationTimer = new Timer();
presentationTimer.schedule(new TimerTask()
{
public void run()
{
if(currentWordIndex < words.length - 1)
{
currentWordIndex++;
repaint();
}
else
{
// text is ended
}
}
}, wordInterval, wordInterval);

Navigating with gestures

In order to give the user more control on the presented text, the app also manages flick gestures to move forward/backward within the word sequence. Gesture recognition is initialized as follows:

GestureInteractiveZone gizCanvas = new GestureInteractiveZone( GestureInteractiveZone.GESTURE_FLICK|GestureInteractiveZone.GESTURE_TAP );
GestureRegistrationManager.register(this, gizCanvas);
GestureRegistrationManager.setListener(this, this);

The gestureAction method of the GestureListener interface can then be defined as follows:

public void gestureAction(Object container, GestureInteractiveZone gestureInteractiveZone, GestureEvent gestureEvent)
{
if(gestureEvent.getType() == GestureInteractiveZone.GESTURE_TAP)
{
// handle the tap gesture to start/stop text presentation
}
else if(gestureEvent.getType() == GestureInteractiveZone.GESTURE_FLICK)
{
float flickAngle = Math.abs(gestureEvent.getFlickDirection());
 
if(flickAngle < Math.PI / 4)
// move to previous word
else if(flickAngle > 3 * Math.PI / 4)
// move to next word
}
}

In the above code, once a flick gesture has been detected, the angle of the gesture is analysed and, depending on the gesture direction, the text is moved forward or backward.


Text to Speech

Text to speech (TTS) enables users to listen to received messages, which can be especially useful for visually impaired users.

Since Java has no inbuilt TTS support, the Twilio service is used. Inside our app we make a special call to the service, passing the SMS text. Shortly afterwards Twilio calls our phone back and reads out our message.

Upon signup, Twilio assigns a phone number that will be used to perform calls and send SMS messages, and asks you to verify your phone number by sending an SMS message. Those two phone numbers can be used with a trial Twilio account to perform testing.

Among many features, Twilio offers the possibility to make a phone call to a specific number and say the desired message. This feature can be accessed through a standard REST call to the URL https://api.twilio.com/2010-04-01/Accounts/<ACCOUNT_NUMBER>/Calls.json, by passing the following POST parameters:

  • the calling number
  • the number to be called
  • the Twilio URL to be called when the call connects


The last parameter actually contains the URL of another Twilio call, that instructs Twilio on how to behave and what to say once the call is answered. In order to let Twilio say something, the http://twimlets.com/echo?Twiml= URL has to be used, passing as argument an XML containing the message that must be reproduced:

<Response>
<Say>This is the message that will be reproduced</Say>
</Response>

The HTTP call must be authenticated using Basic access authentication, so base64-encoding the account credentials as shown in this article: HTTP basic access authentication in Java ME.

You can find your Twilio account by navigating at: https://www.twilio.com/user/account

Twilio assigns a number to your account

The following code defines all the necessary parameters needed to perform the Twilio call:

// The Twilio number generated at signup
String fromNumber = "123456789";
 
// The user phone number (must be verified with Twilio)
String toNumber = "987654321";
 
// The Twilio account number
String twilioAccount = "<YOUR_TWILIO_ACCOUNT>";
// The Twilio password
String twilioPassword = "<YOUR_TWILIO_PASSWORD>";
 
// the XML containing the Twilio commands to say the message text
String callCommands = "<Response><Say>" + text + "</Say></Response>";
// the URL to be called once the Twilio phone call is answered
String callURL = "http://twimlets.com/echo?Twiml=" + urlEncode(callCommands);

The above parameters are then user to perform the actual call to the Twilio URL "https://api.twilio.com/2010-04-01/Accounts/" + twilioAccount + "/Calls.json".

The Twilio response must then be checked to see for possible errors. If the call performed without errors, Twilio replies with a 201 HTTP status code, and with a JSON containing, among various informations, the following attributes:

  • date_created: the date when the call was created
  • account_sid: the Twilio account used to perform the call
  • to and from containing the phone numbers specified in the Twilio call

Following a successful Twilio call, the user receives a phone call within a few seconds: once the call is answered, the user can listen to the received SMS message.

The user touches the TTS button
The app sends the request to Twilio
The request completes successfully
The user receives the Twilio phone call and listen to their SMS message


Notifications

Notifications are fundamental in every messaging app, since allow the user to be alerted of a new incoming message.

A short notification sound can be played by using the MMAPI as follows:

void notificationSound()
{
Player player = null;
try
{
player = Manager.createPlayer(getClass().getResourceAsStream("/message.mid"), "audio/midi");
player.setLoopCount(1);
player.start();
}
catch(Exception e)
{
e.printStackTrace();
}
}

Similarly, a short vibration can be performed when the message is received by using the DeviceControl class from the Nokia UI API:

void notificationVibra()
{
DeviceControl.startVibra(100, 1000);
}

Handling the incoming SMS message

Java apps can listen to incoming messages by using the WMAPI, that allow to be notified when a new message is received on a specific port, and to retrieve its content.

In order to listen for incoming messages, the MessageListener interface can be used, implementing its notifyIncomingMessage as follows:

public void notifyIncomingMessage(final MessageConnection conn)
{
new Thread()
{
public void run()
{
try
{
Message message = conn.receive();
 
if(message != null)
{
TextMessage textMessage = (TextMessage)message;
 
String messageText = textMessage.getPayloadText();
 
// handle the new message
}
}
catch (IOException e)
{
// manage the exception
}
}
}.start();
}

Once implemented the MessageListener interface, the Java app can start listening for messages as shown below.

void startListening()
{
try
{
MessageConnection mc = (MessageConnection) Connector.open("sms://:6553");
 
mc.setMessageListener(this);
}
catch(Exception e)
{
e.printStackTrace();
}
}

Launching the app on received SMS

A messaging app must be able to handle messages even if the app is not open when a message is received. In Java, this can be performed by using PushRegistry.

A Java app can be registered to be launched upon the arrival of a new message on a specified port by adding the following static declaration to the JAD:

MIDlet-Push-1: sms://:6553,com.jappit.rsvpclient.RSVPClient,*

Where:

  • sms://:6553 defines the protocol (sms) and the port (6553) the app wants to connect to
  • com.jappit.rsvpclient.RSVPClient defines the full MIDlet class name
  • * defines the allowed senders (any in this case)

Sending a message to a specific port

In order to be handled by the Java app, new messages must be sent to the port specified by the app itself. This can be performed by using the WMAPI as follows.

MessageConnection connection = null;
 
String address = "sms://" + receiverPhoneNumber + ":6553";
 
connection = (MessageConnection)Connector.open(address);
 
TextMessage message = (TextMessage)connection.newMessage(MessageConnection.TEXT_MESSAGE);
 
message.setPayloadText(messageText);
 
connection.send(message);
Sample app sending an SMS message to a specific port number

Further development

Many improvements over the illustrated app are possible, and the following sections illustrate some of those.

Share destination

The new Share API allows Java app to register themselves as share destinations. This way, when another app declares that it wants to share something, the specified Java app appears among the possible destinations.

Using this feature for this app would allow users to use the implemented features (RSVP, TTS) to easily read text on their devices, by just sharing those with the app.

Registering the Java app as a share destination can be done statically by adding the following declaration to the JAD file:

MicroEdition-Handler-1: com.jappit.rsvpclient.RSVPClient, text/plain, , share open-shared,

The above declaration specifies the full MIDlet class name com.jappit.rsvpclient.RSVPClient), together with the content that the app can be considered as a share destination (text/plain) and the relevant actions (share and open-shared).

After registering, the app should start listening for invocations. This is done by implementing the RequestListener interface, that defines an invocationRequestNotify method. A possible implementation, that retrieves the text of the current invocation, is the following:

public void invocationRequestNotify(ContentHandlerServer handler)
{
Invocation req = handler.getRequest(false);
 
String[] args = req.getArgs();
 
for(int i = 0; i < args.length; i++)
{
if(args[i].indexOf("text=") == 0)
{
String sharedText = args[i].substring(5);
 
// display the shared text
}
}
}

Once the RequestListener interface is implemented, it can be used to listen for incoming request as follows.

ContentHandlerServer handler = Registry.getServer(RSVPClient.class.getName());
 
handler.setListener(requestListener);

By performing the above steps, the Java app is a fully functional share destination. Implementing a second Java app that shares a sample text, like shown by the following code snippet, causes the share dialog to be presented, and the Java app to be presented among the possible destinations.

String[] args = new String[]{"text=A sample text"};
Invocation invocation = new Invocation(null, "text/plain", "com.nokia.share");
invocation.setResponseRequired(false);
invocation.setAction("share");
invocation.setArgs(args);
 
Registry registry = Registry.getRegistry(midletClassName);
 
registry.invoke(invocation);
An external app wants to share some text
The Asha share dialog is presented to the user
The app is selected and start to present the shared text

Identifying the SMS sender

When a new message is received, the sender's number can be retrieved via its the Message.getAddress() method. It would be anyway ideal, especially in a two-way messaging application, to have the sender's name, if registered as a contact on the device, shown together with the message text.

This feature can be implemented by using the PIM API as follows:

ContactList contacts = (ContactList) PIM.getInstance(). openPIMList(PIM.CONTACT_LIST, PIM.READ_ONLY);
 
Contact searchItem = contacts.createContact();
 
searchItem.addString(Contact.TEL, PIMItem.ATTR_NONE, number);
 
Enumeration result = contacts.items(searchItem);
 
Contact item = null;
 
String contactName = null;
 
if(result.hasMoreElements())
{
item = (Contact) result.nextElement();
contactName = item.getStringArray(Contact.NAME, Contact.ATTR_NONE)[Contact.NAME_GIVEN];
break;
}

Local reminders

By using PushRegistry, the app could be used to let users create local reminders, that will automatically notify users.

Landscape mode

To allow for more readability, the app could be configured to automatically launch in landscape mode by using the Nokia-MIDlet-App-Orientation JAD attribute, or can give the user an option to be manually adjust the display orientation depending on his/her needs by using the Orientation class.

Settings screen

Different users have different needs, and this can be especially true for visually impaired users. In the app we can provide a dedicated Settings screen to allow them to use different colours, font sizes and text speed.

Message list

Messages received by a Java app do not end up in the device default message inbox. For this reason, it is vital that the app stores the received messages (for instance, using RMS), and to let the user view those messages.

Downloads

Full source code of the presented app is available at the following download links.

Note.pngNote: In order to use the Text-to-Speech features of the app, it's necessary to create a Twilio account and configure the proper parameters within the app.

Summary

This article illustrates how the features offered by the Asha platform allow to build new usage experiences for visually impaired users. The many available APIs allow the presented Java app to fully integrate with the device, so creating a seamless experience for the user.

This page was last modified on 29 December 2013, at 19:34.
141 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×