×

Discussion Board

Results 1 to 8 of 8
  1. #1
    Registered User
    Join Date
    Dec 2004
    Posts
    14

    urllib out of memory

    This simple test returns and error at around 100 iterations, is there any way around this?
    I've seen a few posts about memory leaks but no solutions

    Code:
    import urllib
    x=1
    while 1:
    	google = urllib.urlopen("http://www.google.co.uk").read()
    	print "got"
    	print x
    	x = x + 1
    Thanks

    Mikie

  2. #2
    Registered User
    Join Date
    Dec 2009
    Posts
    9

    Re: urllib out of memory

    I have a similar error, try to change the URL to a IP address.

  3. #3
    Super Contributor
    Join Date
    Nov 2007
    Location
    Sertaozinho/Brazil
    Posts
    768

    Re: urllib out of memory

    Quote Originally Posted by dfr View Post
    I have a similar error, try to change the URL to a IP address.
    Could you say me if this action really works ? If yes, we may have a leak in urllib and it should be reported at Maemo.

    Python uses reference count, not garbage collector. Have you tried an explicit remotion, like:

    Code:
    import urllib
    x=1
    while 1:
    	google = urllib.urlopen("http://www.google.co.uk").read()
            google = None
    	print "got"
    	print x
    	x = x + 1

  4. #4
    Registered User
    Join Date
    Dec 2004
    Posts
    14

    Re: urllib out of memory

    Quote Originally Posted by marcelobarrosalmeida View Post
    Could you say me if this action really works ? If yes, we may have a leak in urllib and it should be reported at Maemo.

    Python uses reference count, not garbage collector. Have you tried an explicit remotion, like:

    Code:
    import urllib
    x=1
    while 1:
    	google = urllib.urlopen("http://www.google.co.uk").read()
            google = None
    	print "got"
    	print x
    	x = x + 1

    This fails also, this is 1.9.7

    DFR is correct, using an IP seems to fix it my phone is on 1000 iterations and counting using googles ip address, so it maybe be something to do with the DNS code in urllib?

  5. #5
    Registered User
    Join Date
    Dec 2004
    Posts
    14

    Re: urllib out of memory

    You could get around it like this
    Code:
    import urllib
    from socket import gethostbyname_ex
    
    def dnshack(url):
        url = url.split("/")
        length=len(url)
        ip=gethostbyname_ex(url[2])
        ip = ip[2][0]
        result = url[0]+"//"+ip
        for x in range(length-3):
            result=result+"/"+url[x+3]
        return result
    	
    urlx = dnshack("http://www.google.com/")
    	
    x=1
    while 1:
    	google = urllib.urlopen(urlx).read()
    	print "got x"
    	print x
    	x = x + 1
    my code is probably quite convoluted and ugly, but it seems the socket code for getting the dns has a leak.
    If I use gethostbyname_ex in the loop it causes the same memory error.

    So I just move the dns outside the loop by doing the dns once at the start of the script before the loop


    actually that code doesnt work 100% not sure I always get the right ip from the list from the dns server
    or I have some problem with virtual servers im trying to get to....
    Last edited by mikiex; 2009-12-07 at 23:08.

  6. #6
    Super Contributor
    Join Date
    Nov 2007
    Location
    Sertaozinho/Brazil
    Posts
    768

    Re: urllib out of memory

    Quote Originally Posted by mikiex View Post
    This fails also, this is 1.9.7

    DFR is correct, using an IP seems to fix it my phone is on 1000 iterations and counting using googles ip address, so it maybe be something to do with the DNS code in urllib?
    So, it seems a leak. I will test tonight and report it at Maemo. Your workaround is great and I just suggest the following change:

    Instead
    Code:
        for x in range(length-3):
            result=result+"/"+url[x+3]
    You may use
    Code:
        result = url[0] + "//" + ip + "/" + "/".join(url[3:])
    Marcelo

  7. #7
    Registered User
    Join Date
    Dec 2004
    Posts
    14

    Re: urllib out of memory

    Thank you Marcelo for the suggested change and for testing it

  8. #8
    Super Contributor
    Join Date
    Nov 2007
    Location
    Sertaozinho/Brazil
    Posts
    768

    Re: urllib out of memory

    Quote Originally Posted by mikiex View Post
    Thank you Marcelo for the suggested change and for testing it
    You are welcome ! I reported the problem:

    Out of memory when using several DNS resolutions in urllib

Similar Threads

  1. memory jump while running game
    By TacB0sS in forum Mobile Java Games
    Replies: 21
    Last Post: 2009-06-20, 11:56
  2. Excessive midlet memory consumption
    By paulip in forum Mobile Java General
    Replies: 1
    Last Post: 2007-10-09, 19:24
  3. Replies: 0
    Last Post: 2006-02-20, 11:51
  4. can not successfully link any sample using .NET
    By lobotomat in forum Symbian Tools & SDKs
    Replies: 2
    Last Post: 2002-08-20, 00:29
  5. 9210 and 9210i java memory
    By allams in forum Mobile Java General
    Replies: 0
    Last Post: 1970-01-01, 02:00

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
×