×

Discussion Board

Page 1 of 2 12 LastLast
Results 1 to 15 of 18

Thread: Parsing Error

  1. #1
    Registered User
    Join Date
    May 2013
    Posts
    270

    Parsing Error

    I am making an app for s40. I am not able to parse complete data from following link http://tandonindia.com/story_new_xml?user=1
    I m getting out of memory error . I checked the page info of the link and size is 1,142.46 KB (1,169,877 bytes).

    How can i parse all my data at once and store it in rms?

  2. #2
    Registered User
    Join Date
    May 2013
    Posts
    270

    Re: Parsing Error

    anyone..........?

  3. #3
    Registered User
    Join Date
    May 2013
    Posts
    10

    Re: Parsing Error

    This is due to low heap size(2 Mb). When you try to read large data ,heap size of your application gets full and application throws out of memory error. Break your xml data in parts at server side, so that you can read it easily. This is the way you can handle the error.

  4. #4
    Nokia Developer Champion
    Join Date
    Feb 2008
    Location
    Ahmedabad, Gujarat, India
    Posts
    3,852

    Re: Parsing Error

    also the diagnostics tool which is present with nokia sdk 2.0 gives you a rough idea or heap memory consumption you can see that is memory is the only reason.

    Apart from that how you parse is also very important, what parser you have used and how you are making your models give more details on that !!

  5. #5
    Nokia Developer Moderator
    Join Date
    Feb 2006
    Location
    Budapest, Hungary
    Posts
    28,570

    Re: Parsing Error

    It also matters what parser you are using. Honestly said I do not know the available parsers, but for this document size with the typical ~2 MB heap limit, you may want to parse directly from the InputStream (so try not downloading the data into a String), and preferably parse in a SAX-like way and immediately write the results into the RMS.

  6. #6
    Registered User
    Join Date
    May 2013
    Posts
    10

    Re: Parsing Error

    Storage capacity of RMS very limited and it won't be able to store large data as in this case(1 Mb). It would be better to write the data in files.

  7. #7
    Registered User
    Join Date
    Dec 2012
    Location
    India
    Posts
    264

    Re: Parsing Error

    The content sure looks a lot.If you are not able to fix it ,then the compromise would be to write the content off into storage as you get more content.Save the title alone in memory.When you click on the title, you can retrieve the content from storage.

    I am not sure about how XML parsers are implemented,but maybe for this particular case you may have to implement a custom parser that only parses the data progressively,stores it in storage and discards old data stored in memory as it goes.

    This is just my suggestion.A better solution maybe available.

  8. #8
    Nokia Developer Champion
    Join Date
    Feb 2009
    Location
    Noida, India
    Posts
    3,073

    Re: Parsing Error

    Use any SAX parser, directly parse the incoming stream of XML in chunks with chunk size of 512kb (without putting it in any String variable) - store in RMS / Consume the results as soon as you get the required values., in a loop till you reach the end of stream.
    thanks,
    ~Amitabh
    (Champion of the Month -Aug'13)
    Follow me on my blog for Innovative Mobile Apps

  9. #9
    Registered User
    Join Date
    May 2008
    Location
    Surat Thani
    Posts
    258

    Re: Parsing Error

    In the other thread about this subject, I already suggested (as over megabyte of data may take long to download, even if the parsing works) that you change the server side API if possible, and fetch only the story titles, then fetch each story when the user wants to read them. I guess you don't have access to the server side then?

    It appears there's a story posted every week. So I guess you have a way to get only stores created after a certain date? In that case (in case it doesn't cause copyright issues), why not include the stories up to now in the JAR package in a file format of your choosing, and only get the updated stories in the first (and every consequent) start up. This would not only be more convenient for the user (no long wait during the first start up), but also reduce total amount of data transferred - JAR files are compressed.

    If you want to go with parsing all stories from the big XML, you could go with a SAX parser (as Amit recommends) or a pull parser, which can parse data directly from the input stream. Or, as the structure of the XML is known, you could even implement your own "parsing", and have basically zero overhead:

    Loop:
    1. Read (and discard) bytes from the stream until you come across consequent bytes "<item>"
    2. Store bytes from the stream into a byte array until you come across "</item>"
    3. Create a string from the byte array, specifying encoding as UTF-8. You now have a string with all the data for one story.
    4. Extract data from the tags with normal String operations (indexOf, substring). You might also want to do a XML decode for the contents. Write the data to RecordStore or do whatever.

  10. #10
    Registered User
    Join Date
    May 2013
    Posts
    270

    Re: Parsing Error

    Yes i dont have access at the server side. Maximum access given to me is that link through which i can parse all the data. oK I will try with suggestions you all gave. Thanks all. Well i have used kxml parsing and here is my code.
    If i put condition to parse till 208 value i.e after line pe = parser.read(); if i put if(v1.size()<208) and in else i write break. It take till 208 records but it keeps on moving in while (!xmlParsingDone) loop for 2 min. Is there a way it goes out directly from whole code at once after break?

    XmlParser parser = null;
    ParseEvent event = null;
    String story_week = "";
    String story_year = "";
    String story_subject = "";
    String story_storypath = "";


    Vector v1=new Vector();
    Vector v2=new Vector();
    Vector v3=new Vector();
    Vector v4=new Vector();
    try{
    System.out.println(Runtime.getRuntime().totalMemory());
    System.out.println(Runtime.getRuntime().freeMemory());


    String url="http://tandonindia.com/story_new_xml?user=1";

    //String url="http://tandonindia.com/check.php?user="+total;
    HttpConnection hc = (HttpConnection)Connector.open(url);
    InputStream is = hc.openInputStream();
    Reader reader = new InputStreamReader(is);



    parser = new XmlParser(reader);

    boolean xmlParsingDone = false;

    while (!xmlParsingDone)
    {
    event = parser.read();
    ParseEvent pe;
    switch (event.getType())
    {
    case Xml.START_TAG:

    // Pick up Title for display
    String tagName = event.getName();



    if ("Week".equals(tagName)) //................................for week
    {


    pe = parser.read();


    if(pe.getText()!=null)
    {
    story_week=pe.getText();


    }
    else
    {
    story_week="null";
    }
    v1.addElement(story_week);

    }

    if ("Year".equals(tagName)) //........................................for year
    {
    pe = parser.read();

    if(pe.getText()!=null)
    {

    story_year = pe.getText();
    }
    else
    {

    story_year="null";

    }
    v2.addElement(story_year);

    }


    if ("subject".equals(tagName)) // ,,,,,,,,,,,,,,,,,,,,for subject
    {


    pe = parser.read();


    if(pe.getText()!=null)
    {

    story_subject = pe.getText();
    }
    else
    {

    story_subject="null";

    }
    v3.addElement(story_subject);


    }

    if ("storypath".equals(tagName)) //...............................for story
    {
    pe = parser.read();

    if(pe.getText()!=null)
    {

    story_storypath = pe.getText();
    }
    else
    {

    story_storypath="null";

    }
    v4.addElement(story_storypath);

    }

    break;
    case Xml.END_TAG:
    break;
    case Xml.END_DOCUMENT:
    xmlParsingDone = true;
    break;
    case Xml.TEXT:
    break;
    case Xml.WHITESPACE:
    break;
    default:

    }

    }
    System.out.println("sizeeeeeeeeeeeeeeeeeeeeee of week--list---------------"+v1.size());
    System.out.println("sizeeeeeeeeeeeeeeeeeeeeee of year--list---------------"+v2.size());
    System.out.println("sizeeeeeeeeeeeeeeeeeeeeee of caption--list---------------"+v3.size());
    System.out.println("sizeeeeeeeeeeeeeeeeeeeeee of storypath--list---------------"+v4.size());
    }
    catch (IOException ioe){

    System.out.println("XML Parsing Error: " + ioe.toString());
    ioe.printStackTrace();
    }finally
    {
    parser = null;
    }

    //TODO Auto-generated method stub

    }
    Last edited by Manpreet_Singh; 2013-07-23 at 05:50.

  11. #11
    Nokia Developer Champion
    Join Date
    Feb 2008
    Location
    Ahmedabad, Gujarat, India
    Posts
    3,852

    Re: Parsing Error

    I think all of them have provided some really good solutions so if i should be at your position i should have started with:

    1. what parser i am using.
    2. am i using the parser properly, like not holding huge references for huge amount of time.
    3. am i directly parsing from the InputStream.

    If you are perfect on all above then TongMuan has give some nice suggestions !!

  12. #12
    Registered User
    Join Date
    May 2013
    Posts
    270

    Re: Parsing Error

    @im2amit ,,,can i have some good tutorials on sax parsing?
    Last edited by Manpreet_Singh; 2013-07-23 at 06:12.

  13. #13
    Registered User
    Join Date
    May 2008
    Location
    Surat Thani
    Posts
    258

    Re: Parsing Error

    Quote Originally Posted by Manpreet_Singh View Post
    Yes i dont have access at the server side. Maximum access given to me is that link through which i can parse all the data. oK I will try with suggestions you all gave. Thanks all. Well i have used kxml parsing and here is my code.
    Just to make sure, there's no API to get only stories updated since the last time? So, if I run the app for the first time this week, it will download all the stories, and if I run it again next week, it will download all the same stories again (+next week's story)?

    Regarding the out of memory error, I think the issue is not the kxml parser, but the fact that you keep all the parsed data in the memory at once - you add to the vectors everything you get from the parser. Without adding the stuff to the vectors, does the parsing complete successfully?

    If the parsing goes ok in that case, something like this would probably work:

    Code:
    String subject; String story; String week; String year;
    
    LOOP:
    - Get the next parseEvent
     - If it's item start tag, set subject, story, week and year Strings to null
     - If it's week/year/story/subject tag, store its contents to to corresponding String
     - If it's item end tag
        - Check that we have all the info (subject, story, week, year not null)
        - Store the info somehow, without keeping everything in the memory (you were talking about RecordStore before).

    Quote Originally Posted by Manpreet_Singh View Post
    If i put condition to parse till 208 value i.e after line pe = parser.read(); if i put if(v1.size()<208) and in else i write break. It take till 208 records but it keeps on moving in while (!xmlParsingDone) loop for 2 min. Is there a way it goes out directly from whole code at once after break?
    How about simply setting xmlParsingDone to true? You might also need to somehow tell the parser that you're not using it anymore, to make sure that the inputstream/reader will be closed.

  14. #14
    Nokia Developer Champion
    Join Date
    Feb 2008
    Location
    Ahmedabad, Gujarat, India
    Posts
    3,852

    Re: Parsing Error

    also along with what TongMuan has mentioned i would recommend using a separate thread for the network connection , i think with your present implementation your UI is blocked until you get response from the server !!

  15. #15
    Registered User
    Join Date
    May 2013
    Posts
    270

    Re: Parsing Error

    TongMuan.....actually i put a logic that all my stories will be added to rms once user runs app for first time and on background i will show the loading progress bar. Since the story is coming every sunday so only the latest story will be added to rms. I am putting a check for that(not every time all database will be added).
    String subject; String story; String week; String year;

    LOOP:
    - Get the next parseEvent
    - If it's item start tag, set subject, story, week and year Strings to null
    - If it's week/year/story/subject tag, store its contents to to corresponding String
    - If it's item end tag
    - Check that we have all the info (subject, story, week, year not null)
    - Store the info somehow, without keeping everything in the memory (you were talking about RecordStore before).
    I am not getting this logic because some of the steps i am using in my code, what changes i have to make in my parsing code? and abt rms ...earlier i thought that i will add my data in rms once it is collected in vectors but i am getting out of memory error there
    Last edited by Manpreet_Singh; 2013-07-23 at 09:48.

Similar Threads

  1. Parsing of GCC 4.4.1 error message in Carbide.c++ 2.6
    By mgroeber9110 in forum Carbide.c++ IDE and plug-ins (Closed)
    Replies: 12
    Last Post: 2011-09-15, 10:45
  2. In XML parsing I get -996 error.
    By SymbianTH in forum Symbian
    Replies: 3
    Last Post: 2010-04-05, 14:49
  3. -998 error while parsing on 5th Edition
    By sunitha.m13 in forum Symbian
    Replies: 1
    Last Post: 2009-10-22, 14:23
  4. Error while parsing XML in j2me app
    By knights123 in forum Mobile Java General
    Replies: 2
    Last Post: 2009-07-06, 17:58
  5. e32std.h parsing error
    By Tatavarthi in forum Symbian
    Replies: 2
    Last Post: 2008-03-25, 07:08

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •