Following code snippet illustrate how to extract tokens from the available data.
CDesC16Array *array= new (ELeave) CDesC16ArrayFlat(10);
_LIT(KData,"This is the token example." \
"You will get different tokens in array.");
TPtrC token = lex.NextToken();
if(token.Length() == 0)
Note: This is just an example which extract tokens from string defined using LIT macro. However you can use this code in general terms. You can extract tokens from descriptors, file data etc. Here Array granularity is taken as 10. You can use granularity on your application's discretion.