×

Discussion Board

Results 1 to 7 of 7
  1. #1
    Registered User
    Join Date
    Mar 2009
    Posts
    8

    problems with string interpolation speed

    i am currently coding a rest server for the n95, the server let's you access sensor data (gps, wlan, bluetooth etc) stored in a e32db, which is all running smooth so far, well except for the serializing. the service returns xml by default, kml, json and rss on demand, i have templates for each of them, they consist of up to 3 files for each format, i load them in variables and then build the outgoing file by concatenating them after inserting the values, so the files may look something like this:
    Code:
    template1:
    <?xml>
    <root sensor="%(sensor)s">
    %(entries)s
    </root>
    
    template2:
     <entry name="%(name)s" %(otherattributes)s>
      %(sometext)s
     </entry>
    and they are processed as follows:
    Code:
    tmpl1 = load template1
    tmpl2 = load template2
    
    entries = ""
    for entry in dbrows:
     otherattributes = "date=\"somedate\""
     name = entry["name"]
     sometext = entry["description"]
     
     entries += tmpl2 % {"name": name, "otherattributes": otherattributes, "sometext": sometext}
    
    output = tmpl1 % {"sensor": sensor, "entries": entries}
    return output
    this seems to be the straight forward thing to do for me, but the problem is for a 30kb file this is taking up to 25 seconds, which is insanely long, so there must be something that is slowing this down terribly, i have no prior experience with python and i didn't find anything related to my problem.
    i'd appreciate any help here. thanks in advance.

  2. #2
    Super Contributor
    Join Date
    Nov 2007
    Location
    Sertaozinho/Brazil
    Posts
    768

    Re: problems with string interpolation speed

    Hum, nearly 1 package per second ... so, could you show your transmission routine ?

    Marcelo

  3. #3
    Registered User
    Join Date
    Mar 2009
    Posts
    8

    Re: problems with string interpolation speed

    the transmission is handled by the mobile web server and i am not talking about how long the whole request takes to come back, the 25 seconds come from taking the time before and after serializing, it's sent as an attribute of the root element of the xml file.

  4. #4
    Super Contributor
    Join Date
    Mar 2003
    Location
    Espoo, Finland
    Posts
    976

    Re: problems with string interpolation speed

    Your system is very complicated, so "naturally" it requires a lot of time. Templates with 3 files each: file system access is slow. GPS requires several seconds to activate even under the most optimal circumstances, several minutes usually, even 15 mins without Assistance (A-GPS). Even importing socket requires seconds.

    If you're not happy with performance, maybe you should measure where all the time is spend? Then you can consider whether you want to simplify your system in some places or use lazy import.

    Here's a lazy profiler:

    Code:
    # Lazy import, after user looking at screen:
    #import time
    #t0 = time.clock()
    import socket
    import urllib
    #print "Time: %f" % (time.clock() - t0)
    # 5 seconds in emulator DEBUG mode
    # 3.8 seconds in emulator NON_DEBUG mode
    Cheers,

    --jouni

  5. #5
    Registered User
    Join Date
    Mar 2009
    Posts
    8

    Re: problems with string interpolation speed

    i have to correct myself a little, it seems to take 25 seconds for a 200kb file, someone else checked the size and obviously it wasn't right, so this doesn't feel that horrible anymore, but still i feel like it's taking too long, on the other hand i don't really have any comparisons, but still.

    i have to reiterate though, the time i measured is only dictionary access, string interpolation and string concatenation, imports are done at the beginning, this has no effect, file loading takes 0.02 seconds, sensors (like gps) have no impact here, i read them with a separate program that writes them to a db.

    i can give some measurements of the other parts of this program, for a 500kb output, it takes 6s to read the database and write all rows in a list, 2s to convert all of those rows into dictionaries and then 70s to write these rows into xml. and that jump just seems out of whack to me.

  6. #6
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: problems with string interpolation speed

    Quote Originally Posted by zarzu View Post
    Code:
    entries = ""
    for entry in dbrows:
      ...
      entries += ...
    This is always a bad idea in Python or any other language with immutable strings (Java, .NET). What is happening here is that for each dbrows entry, a new entries string is created and the old one destroyed. The new entries string and the old one are bigger and bigger each round, slowing down the execution due to unnecessary pressure on memory allocator and garbage collector of Python.

    Here's how you usually do it in Python:

    Code:
    entries = []
    for entry in dbrows:
      ...
      entries.append(...)
    
    entries_joined = "".join(entries)

  7. #7
    Registered User
    Join Date
    Mar 2009
    Posts
    8

    Re: problems with string interpolation speed

    i really hoped that was it, tried it, but it actually increased the time needed for processing by 10%

    correction: i was a bit over enthusiastic with using the join it seems, i only used it on the really large concatenations now and it's running 10% faster than the original version.
    thanks for the tipp!
    Last edited by zarzu; 2009-03-13 at 22:12.

Similar Threads

  1. illegal unicode escape in j2me when i splite string...
    By chaoraksa in forum Mobile Java General
    Replies: 36
    Last Post: 2009-02-04, 08:56
  2. Replies: 0
    Last Post: 2009-01-02, 09:50
  3. TextEditor with two errors
    By ahashim in forum Personal Profile
    Replies: 0
    Last Post: 2006-08-29, 23:43
  4. Convert bytes in String
    By decisor in forum Mobile Java General
    Replies: 8
    Last Post: 2006-08-03, 18:56
  5. global root overflow(please help)
    By sushant_125 in forum Mobile Java Networking & Messaging & Security
    Replies: 3
    Last Post: 2004-05-14, 08:24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •