×

Discussion Board

Results 1 to 12 of 12
  1. #1
    Regular Contributor
    Join Date
    Nov 2007
    Posts
    147

    How to unzip large zip file

    Hi, does anybody know of a lib or extension that can handle large zip files without running out of memory?

    I tried ZipFile and I tried to backport ZipFile from 2.6 as it has improvements in handling large files but run into issue of missing basestring.

    Thanks

    Raf

  2. #2
    Registered User
    Join Date
    Sep 2008
    Posts
    36

    Re: How to unzip large zip file

    An optimised zip file handler could be created in Symbian C++ as an extension. This would have the size limit imposed by your phone, not by Python.

  3. #3
    Regular Contributor
    Join Date
    Nov 2007
    Posts
    147

    Re: How to unzip large zip file

    I looked at the zipfile module and think that the issue is that it attempts to read each individual file contained in the zip file into memory. Thus, when it encounters a file that is larger then avail. memory, it throws out of memory error.

    In my opinion, the only correct way to unzip a file would be by using a stream and unzip the file in chunks. But I havent found a module hat would do this yet.

    Raf

  4. #4
    Regular Contributor
    Join Date
    Nov 2007
    Posts
    147

    Re: How to unzip large zip file

    A "backported" ZipFile module from 2.6 worked after a minor edit. In case anybody runs into the same issue (it unzipped a 30MB file in ~30 seconds on N95-3).

    Module is too large to post in this forum so email or msg me if you want it.

    Raf

  5. #5
    Registered User
    Join Date
    Dec 2008
    Posts
    1

    Re: How to unzip large zip file

    Hi Raf,

    I'm interested in your backported-to-2.5 2.6 zipfile implementation. How might I get it from you? -- Eric

  6. #6
    Regular Contributor
    Join Date
    Nov 2007
    Posts
    147

    Re: How to unzip large zip file

    Send me a private message with your email I will email it to you.

  7. #7
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: How to unzip large zip file

    Can you create a diff against the unmodified source and post that here?

  8. #8
    Regular Contributor
    Join Date
    Nov 2007
    Posts
    147

    Re: How to unzip large zip file

    Here is a UNIX diff of original 2.6 zipfile.py and one that works with pys60:

    Code:
    655c655
    <     def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=False):
    ---
    >     def __init__(self, mfile, mode="r", compression=ZIP_STORED, allowZip64=False):
    680c680
    <         if isinstance(file, basestring):
    ---
    >         if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):
    682c682
    <             self.filename = file
    ---
    >             self.filename = mfile
    685c685
    <                 self.fp = open(file, modeDict[mode])
    ---
    >                 self.fp = open(mfile, modeDict[mode])
    689c689
    <                     self.fp = open(file, modeDict[mode])
    ---
    >                     self.fp = open(mfile, modeDict[mode])
    695c695
    <             self.filename = getattr(file, 'name', None)
    ---
    >             self.filename = getattr(mfile, 'name', None)
    Since I hate UNIX diff, here is the modified __init__ method of the ZipFile class that needs to be replaced in the 2.6 module:

    Code:
        def __init__(self, mfile, mode="r", compression=ZIP_STORED, allowZip64=False):
            """Open the ZIP file with mode read "r", write "w" or append "a"."""
            if mode not in ("r", "w", "a"):
                raise RuntimeError('ZipFile() requires mode "r", "w", or "a"')
    
            if compression == ZIP_STORED:
                pass
            elif compression == ZIP_DEFLATED:
                if not zlib:
                    raise RuntimeError,\
                          "Compression requires the (missing) zlib module"
            else:
                raise RuntimeError, "That compression method is not supported"
    
            self._allowZip64 = allowZip64
            self._didModify = False
            self.debug = 0  # Level of printing: 0 through 3
            self.NameToInfo = {}    # Find file info given name
            self.filelist = []      # List of ZipInfo instances for archive
            self.compression = compression  # Method of compression
            self.mode = key = mode.replace('b', '')[0]
            self.pwd = None
            self.comment = ''
    
            # Check if we were passed a file-like object
            if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):
                self._filePassed = 0
                self.filename = mfile
                modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'}
                try:
                    self.fp = open(mfile, modeDict[mode])
                except IOError:
                    if mode == 'a':
                        mode = key = 'w'
                        self.fp = open(mfile, modeDict[mode])
                    else:
                        raise
            else:
                self._filePassed = 1
                self.fp = file
                self.filename = getattr(mfile, 'name', None)
    
            if key == 'r':
                self._GetContents()
            elif key == 'w':
                pass
            elif key == 'a':
                try:                        # See if file is a zip file
                    self._RealGetContents()
                    # seek to start of directory and overwrite
                    self.fp.seek(self.start_dir, 0)
                except BadZipfile:          # file is not a zip file, just append
                    self.fp.seek(0, 2)
            else:
                if not self._filePassed:
                    self.fp.close()
                    self.fp = None
                raise RuntimeError, 'Mode must be "r", "w" or "a"'
    Hope it helps.

    Raf

  9. #9
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: How to unzip large zip file

    Thanks!

    One observation:

    Quote Originally Posted by raf1hh View Post
    Code:
    if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):
    "" and '' are both non-Unicode strings, since Python allows either kind of quotes and makes no distinction between them. So this works equally well:

    Code:
    if type(mfile) == type("") or type(mfile) == type(u""):

  10. #10
    Regular Contributor
    Join Date
    Oct 2008
    Posts
    135

    compression using zlib in symbian

    hi,

    i am using compression and decompression on client side using zlib

    on server side reverse process is doing in java

    i am unable decompress data on client side

    error code: -11502

    KEZlibErrStream -11501
    KEZlibErrData -11502

    i am not getting where i am wrong

  11. #11
    Nokia Developer Moderator
    Join Date
    May 2007
    Location
    21.46 N 72.11 E
    Posts
    3,796

    Smile Re: compression using zlib in symbian

    Quote Originally Posted by rahul.shalik.makode View Post
    hi,

    on server side reverse process is doing in java

    i am unable decompress data on client side
    Check your server implementation. We can only help to create or extract a zip archive using zipfile in Python
    Pankaj Nathani
    www.croozeus.com

  12. #12
    Super Contributor
    Join Date
    Mar 2003
    Location
    Espoo, Finland
    Posts
    976

    Re: compression using zlib in symbian

    Quote Originally Posted by rahul.shalik.makode View Post
    i am using compression and decompression on client side using zlib
    on server side reverse process is doing in java
    i am unable decompress data on client side

    error code: -11502

    KEZlibErrStream -11501
    KEZlibErrData -11502
    1) Try to decompress on server side, where you did compression.
    2) Try to decompress with 7-Zip

    ...in other words, verify that your compression is working correctly before debugging why client side cannot decompress

    Cheers,

    --jouni

Similar Threads

  1. Problem with GCCE build in Carbide.vs
    By raffaelbechara in forum Symbian
    Replies: 6
    Last Post: 2008-10-20, 16:06
  2. [announce] PyUIQ. Python for UIQ 2.1 and 3.x
    By OscarBernabeu in forum Symbian
    Replies: 35
    Last Post: 2008-01-17, 09:12
  3. Contacts fields order??
    By timatima in forum Symbian
    Replies: 1
    Last Post: 2007-06-08, 13:51
  4. S60 theme questions
    By Nokian_Spb in forum Symbian Media (Closed)
    Replies: 3
    Last Post: 2006-08-17, 06:11

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
×