×

Discussion Board

Page 1 of 2 12 LastLast
Results 1 to 15 of 20
  1. #1
    Regular Contributor
    Join Date
    Feb 2004
    Location
    Budapest, Hungary, Europe
    Posts
    168

    Question Ability to change the default heap size

    Hi all,

    Is there any way to have influence on the heap size of the host (Python) application whose context my script is running in? I guess there is no way of doing that in case of the Python Shell, unless you re-build it. But that would be an undesirable option, since you would need to ship the re-compiled component along with your own, right? I checked it out if Ensymble could help me, but found that although it does support generating stand-alone applications, it does not allow us changing the heap size.

    Background: I'd like to take photos with my N95's built-in 5MP camera. I've noticed, however, that doing that might require up to ~800k (with maximum JPEG quality) from free RAM, i.e. the host application's own heap. That along with the fact that Python Shell uses the default heap size (1MB) suggests me that that might be the reason why I'm getting (-4) KErrNoMemory errors every now and then even after taking a single shot.

    Any ideas, suggestions, as to how I could solve this problem in Python?

    Thanks in advance,

    Tote

  2. #2
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: Ability to change the default heap size

    Quote Originally Posted by tote_b5 View Post
    Is there any way to have influence on the heap size of the host (Python) application whose context my script is running in?
    ...

    Any ideas, suggestions, as to how I could solve this problem in Python?
    Heap size is set during compilation and stored in the EXE header. Ensymble includes an EXE stub with each PyS60 application to start the Python interpreter. By recompiling the EXE stub the heap size can be selected freely. Ensymble will then need to be patched with the new EXE stub. The PyS60 interpreter SIS can remain unchanged.

    Ensymble could provide a command line option to set the heap size. After all, the EXE stub header is already modified to set the application UID and capability bit mask. If there's enough interest I can release a new version of Ensymble with this feature.

  3. #3
    Regular Contributor
    Join Date
    Feb 2004
    Location
    Budapest, Hungary, Europe
    Posts
    168

    Question Re: Ability to change the default heap size

    Quote Originally Posted by jethro.fn View Post
    Heap size is set during compilation and stored in the EXE header. Ensymble includes an EXE stub with each PyS60 application to start the Python interpreter. By recompiling the EXE stub the heap size can be selected freely. Ensymble will then need to be patched with the new EXE stub. The PyS60 interpreter SIS can remain unchanged.

    Ensymble could provide a command line option to set the heap size. After all, the EXE stub header is already modified to set the application UID and capability bit mask. If there's enough interest I can release a new version of Ensymble with this feature.
    Okay, that sounds like a viable option. Could you give me some pointers as to how I could recompile the exe stub? Please note that I'm a bit uncertain regarding what an exe stub really is. I have checked out Ensymble source carefully and seen that you use the stub in question. However, how that stub is made is still a big question mark to me as well as how I could have influence on the content of it.

    And, of course, you can count on me in needing the new version of Ensymble already supporting heap size changing. But since I expect that to take some time, I expect that I'll need to hack the stub by myself first.

    Tote

  4. #4
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: Ability to change the default heap size

    Quote Originally Posted by tote_b5 View Post
    Could you give me some pointers as to how I could recompile the exe stub? Please note that I'm a bit uncertain regarding what an exe stub really is.
    You need to download PyS60 source (pys60-1.4.2_src.zip) from Sourceforge. Inside it there is a src\py2sis\templates_eka2 directory. There are several files with names ending in .template. Py2sis from Nokia uses these files to compile an EXE stub for the application. Ensymble does not require a working Symbian C++ toolchain but uses a precompiled EXE stub instead and patches it when required.

    You need to apply the patch pytest.cpp.template.diff from Ensymble source package to the appropriate file using the Patch command line tool. Then you also need to replace every ${{...}} portion in every file ending in .template with an actual value. You'll need to invent some values, for example 12345678 for the UID. Finally, remove the .template suffix from all filenames and compile. You can take a look at the py2sis program source to figure out how it modifies the templates and compiles the final binary.

    Ensymble stores the EXE stub as a base-64-encoded binary inside cmd_py2sis.py (variable exestubdata). Replace it with the compiled binary and run install.sh (or look what it does and do it by hand if you're not running Linux/Mac OS X/other Unix-like system) to generate a new ensymble.py.

  5. #5
    Regular Contributor
    Join Date
    Feb 2004
    Location
    Budapest, Hungary, Europe
    Posts
    168

    Thumbs up Re: Ability to change the default heap size

    Thanks for the hints, I finally managed to change the heap size for stand-alone PyS60 applications. Now I can take large photos (biggest 1.4MB in size) without any memory problems.

    Please note that I noticed another problem with Ensymble: it was not able to cope with my base64 encoded private key in which it was NOT explicitly mentioned whether it was an RSA- or DSA-encoded key. In other words, it either expected
    Code:
    -----BEGIN DSA PRIVATE KEY-----
    or
    Code:
    -----BEGIN RSA PRIVATE KEY-----
    boundary, whereas mine was simply
    Code:
    -----BEGIN PRIVATE KEY-----
    .

    I'm not sure if the last format should always be defaulted to RSA, however, since mine was RSA-encoded (got from TrustCenter, btw) I adjusted cryptutil.py script so that it works the same way in the last two cases.

    Thanks again,

    Tote

  6. #6
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: Ability to change the default heap size

    Quote Originally Posted by tote_b5 View Post
    ... whereas mine was simply
    Code:
    -----BEGIN PRIVATE KEY-----
    .

    I'm not sure if the last format should always be defaulted to RSA, however, since mine was RSA-encoded (got from TrustCenter, btw)
    Thanks for the tip. I've never seen that kind of key myself, but I mostly use recent versions of OpenSSL anyway. This feature will be added to the upcoming release of Ensymble. Heap size adjustment works already, but I've yet to update the documentation.

  7. #7
    Registered User
    Join Date
    Mar 2008
    Posts
    25

    Re: Ability to change the default heap size

    Hello guys,

    I've run into the same problem. I need to make use of large strings streamed over a socket, but after I stream 1 or 2 of such strings I get "Memory Error". I get the error either when trying to decode the strings, either when reading from the socket (the error doesn't appear in the same spot every time) so I'm guessing to increase the heap size.

    Can you please post a more detailed tutorial on how to do this?

    Thanks

  8. #8
    Regular Contributor
    Join Date
    Feb 2004
    Location
    Budapest, Hungary, Europe
    Posts
    168

    Re: Ability to change the default heap size

    Hi,

    Well, I always got KErrNoMemory (aka out-of-memory) errors and that was a clear indication that I needed more. The max heap size in PyS60 applications is 1MB by default - is it really not enough for you? On a resource-constrained device it's always recommended to keep as few data in memory as possible - do you persist (or do whatever else) your data once it's not needed anymore?

    A generic "Memory error" seems suspicious to me, it might have something to do with small heap or might not.

    Finally, as to your problem of not understanding fully what Jethro wrote: what is your problem actually? What do you not understand?

    Cheers,

    Tote

  9. #9
    Registered User
    Join Date
    Mar 2008
    Posts
    25

    Re: Ability to change the default heap size

    Quote Originally Posted by tote_b5 View Post
    Hi,

    Well, I always got KErrNoMemory (aka out-of-memory) errors and that was a clear indication that I needed more. The max heap size in PyS60 applications is 1MB by default - is it really not enough for you? On a resource-constrained device it's always recommended to keep as few data in memory as possible - do you persist (or do whatever else) your data once it's not needed anymore?

    A generic "Memory error" seems suspicious to me, it might have something to do with small heap or might not.

    Finally, as to your problem of not understanding fully what Jethro wrote: what is your problem actually? What do you not understand?

    Cheers,

    Tote
    I am trying to draw a big 3d scene using openGL, so my vertices and indices list that I am using are quite big. I must stream them over a socket from the pc, so when I serialize the two lists (for a medium-sized object) I get strings of 750k length.
    So, since I can't import the gc module to call the gc manualy,
    once I recieve the strings an decode them, I make sure to delete all the unnecesary data (del variable). Even so, I get the generic error: "Memory Error" which I can't figure out. So I am guessing to increase the heapsize...
    I should mention that so far I have done all my testing only on the 3rd MR Emulator!

    Quote Originally Posted by jethro.fn View Post
    You need to download PyS60 source (pys60-1.4.2_src.zip) from Sourceforge. Inside it there is a src\py2sis\templates_eka2 directory. There are several files with names ending in .template. Py2sis from Nokia uses these files to compile an EXE stub for the application. Ensymble does not require a working Symbian C++ toolchain but uses a precompiled EXE stub instead and patches it when required.
    i've done this.

    Quote Originally Posted by jethro.fn View Post
    You need to apply the patch pytest.cpp.template.diff from Ensymble source package to the appropriate file using the Patch command line tool.
    i am not sure how to do this.

    Quote Originally Posted by jethro.fn View Post
    Then you also need to replace every ${{...}} portion in every file ending in .template with an actual value. You'll need to invent some values, for example 12345678 for the UID.
    i've done this. replaced all
    ${{PY2SIS_APPNAME}} with ${{str3D}}
    and
    ${{PY2SIS_UID}} with ${{0x01234567}}
    and
    ${{PY2SIS_CAPS}} with ${{LocalServices Location NetworkServices PowerMgmt ProtServ ReadDeviceData ReadUserData SurroundingsDD SwEvent TrustedUI UserEnvironment WriteDeviceData WriteUserData}}

    not sure if it's correct and how to replace this block:
    ${{if PY2SIS_AUTOSTART>0
    START RESOURCE ${{PY2SIS_UID[2:]}}.rss
    TARGETPATH \Resource\Apps
    END
    }}



    Quote Originally Posted by jethro.fn View Post
    Finally, remove the .template suffix from all filenames and compile. You can take a look at the py2sis program source to figure out how it modifies the templates and compiles the final binary.
    not sure what to compile and what :| i'm sorry if am too much of a noob...

    Quote Originally Posted by jethro.fn View Post
    Ensymble stores the EXE stub as a base-64-encoded binary inside cmd_py2sis.py (variable exestubdata).
    i can't find this variable insite cmd_py2sis.py
    Quote Originally Posted by jethro.fn View Post
    Replace it with the compiled binary and run install.sh (or look what it does and do it by hand if you're not running Linux/Mac OS X/other Unix-like system) to generate a new ensymble.py.
    how do i do this on windows?


    thanks a LOT for all your help

  10. #10
    Regular Contributor
    Join Date
    Mar 2008
    Posts
    151

    Re: Ability to change the default heap size

    Completely out of my depth here, but just wanted to check you'd seen the miso module which apparently let's you do stuff with heap sizes etc. Sorry if this is of no use!

    http://pdis.hiit.fi/pdis/download/miso/

    http://pdis.hiit.fi/pdis/download/mi...-1.92-api.html

  11. #11
    Registered User
    Join Date
    Mar 2008
    Posts
    25

    Re: Ability to change the default heap size

    Quote Originally Posted by jtullis View Post
    Completely out of my depth here, but just wanted to check you'd seen the miso module which apparently let's you do stuff with heap sizes etc. Sorry if this is of no use!

    http://pdis.hiit.fi/pdis/download/miso/

    http://pdis.hiit.fi/pdis/download/mi...-1.92-api.html
    Hey,
    Thanks for the answer. I have seen this module, don't know how to install it for the emulator but anyways from what I see you can't increase the heapsize of the app. This can only be done by recompiling pys60 or when you build your sis file out of the .py files.
    I will try to use it to see at least how the memory is used and if the limited size of the heap is the problem for the generic error "Memory Error".

  12. #12
    Regular Contributor
    Join Date
    Feb 2004
    Location
    Budapest, Hungary, Europe
    Posts
    168

    Re: Ability to change the default heap size

    Patching the .cpp file should be straightforward if you're familiar with any source control management tool, such as Subversion. If not, even then it's not too difficult: check out what's the difference between \epoc32\tools\py2sis\template2_eka2\pytest.cpp and ensymble\exestub\pytest.cpp. Latter contains information for the merger tool (such as Patch, I assume, that I've never used): e.g. - means that you should remove that line, + means that you should add it, and so on.

    When replacing "variables" that are enclosed between {{ and }}, please keep in mind that you should remove these signs, too. That is, a line specifying e.g. capabilities must look like this:
    CAPABILITY NetworkServices Location

    What to compile? Well, if you're familiar with the build process in Symbian, then it should be okay. Otherwise, you should get familiar with it.

    Base-64 encoding: yeah, there's a variable in the python file in question, named execstubdata. It contains characters only from a pre-defined 6-bit dictionary (hence 2^6=64). You should take the entire content of generated .exe, which is in binary. Convert that to Base-64 form - I used Total Commander for that purpose, for example. Copy the generated Base-64 output in place of this variable and that's it.

    Finally, obviously you don't need to run an .sh file on Windows (you could do that with some tricks, but why to bother?). The point is to invoke squeeze.py script in the right way, i.e. feeding it the right parameters in the right form. It should not be difficult if you're a bit familiar with Unix script shell.

    Tote

  13. #13
    Registered User
    Join Date
    Mar 2008
    Posts
    25

    Re: Ability to change the default heap size

    Thank you for the pointers. I will try it right away and give feedback.
    Thanks again !

  14. #14
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: Ability to change the default heap size

    Quote Originally Posted by sstoica View Post
    Thank you for the pointers. I will try it right away and give feedback.
    Thanks again !
    What version of Python are you running on a PC? I can create a pre-release version of Ensymble with the heap size adjustment feature. I've been busy with other things lately and developing Ensymble has been a lower priority activity for me... The heap size adjustment feature is implemented in Ensymble already but testing and documentation is still not done, and there are other features I would like to implement in the next release.

  15. #15
    Super Contributor
    Join Date
    May 2004
    Location
    Tampere, Finland
    Posts
    524

    Re: Ability to change the default heap size

    Here is a pre-release of Ensymble v0.27, making it possible to adjust the heap size of a PyS60 application:



    New features:

    Both the py2sis and altere32 commands feature a new option --heapsize. It takes a comma-separated list of two integer values; the minimum and maximum heap size. Values can have a k or M suffix to denote kilobytes and megabytes, respectively. If only one value is given, both values are set to that value.

    Minimum heap size determines if an application can start at all. If there is less memory available than the minimum heap size of an application, an error may be displayed (nothing was displayed when tested on a Nokia N95) and the application is not started. Default minimum heap size is 4 kB.

    Maximum heap size limits the amount of memory available to an application. The application cannot allocate more memory than this limit. The default maximum heap size is 1 MB.

    When setting the minimum heap size to a larger value than the maximum heap size, a phone may not even install the application (tested on a Nokia N95).

    For example, when an application needs to be able to use 5 MB of memory:

    Code:
    ensymble.py py2sis --heapsize=4k,5M myapp.py

Similar Threads

  1. heap size problem
    By dinesh547 in forum Symbian
    Replies: 22
    Last Post: 2008-04-04, 13:29
  2. Replies: 1
    Last Post: 2007-10-30, 08:51
  3. Increasing Heap size
    By earamsey in forum Mobile Java General
    Replies: 3
    Last Post: 2007-10-18, 17:58
  4. How to change a control size when it becomes focused??
    By aladin3006 in forum Symbian User Interface
    Replies: 2
    Last Post: 2007-05-03, 09:28
  5. nokia 6230 max. heap size
    By giovanni_m in forum Mobile Java General
    Replies: 3
    Last Post: 2004-08-29, 08:46

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
×