×

Discussion Board

Results 1 to 9 of 9
  1. #1
    Regular Contributor
    Join Date
    Feb 2006
    Posts
    112

    http & outputstream memory problem

    I am trying to send a video file through Http-connection. In order to achieve large file uploads, I have divided the sending into multiple parts and the server sides is putting them back together. It is working, but the problem is that it will keep in reserve the amount of memory that the video is. I have run some tests and just reading the file from memory card (through FileConnection) doesn't reserve the memory. But when I open HttpConnection and write the file to the OutputStream, then it will take the memory. I have tried closing OutputStream and HttpConnection, setting them to null, running System.gc(); but nothing seems to free up that memory!

    My personal feeling is that the OutputStream will not get closed and it will have all the data written to it in a buffer or something. This feeling I get when I tried this:

    os.close();
    os.write(something..);

    and it did not give any errors.

    Here is the relevant part of my code:


    Code:
    // opening the file connection and inputstream
    fc = (FileConnection)Connector.open("file:///somefile", Connector.READ);
    is = fc.openInputStream();
    
    // opening http connection and outputstream
    HttpConnection http = (HttpConnection)Connector.open(url, Connector.WRITE);
    http.setRequestMethod(HttpConnection.POST);
    http.setRequestProperty("User-Agent", "Profile/MIDP-2.0 Configuration/CLDC-1.1");
    http.setRequestProperty("Content-Type", type);
    http.setRequestProperty("Connection", "close");
    
    OutputStream os = http.openOutputStream();
    
    // algorithm that will read 1024 bytes at a time in 200k packages
    for (int i = startBytes; i < endBytes; i += packetSize) {      
    byte b[] = new byte[1024];
         if ((i + packetSize) < fileSize) {
        int length = is.read(b, 0, packetSize);
         } else {
        int left = (int)fileSize - i;
        int length = is.read(b, 0, left);
         }
    
    // writing into the output stream - these lines will cause the "memory leak", without these, it will not happen
         os.write(b);
    }
    os.flush();
    
    // closing stuff that I have tried
    int rc = http.getResponseCode();
    os.close();
    //os.write(Integer.toString(rc).getBytes());
    http.close();
    System.gc();


    Anyone know how I can free up that memory somehow reserved by OutputStream?

  2. #2
    Super Contributor
    Join Date
    Dec 2005
    Location
    Europe/Poland/Warsaw
    Posts
    1,699

    Re: http & outputstream memory problem

    hi,

    could you reuse b[] declaring it prior to looping and then:
    Code:
     os.write(b, 0, length);
    where "length" varies according to your algorithm?

    regards,
    Peter

  3. #3
    Regular Contributor
    Join Date
    Feb 2006
    Posts
    112

    Re: http & outputstream memory problem

    I could do that. Would it solve my memory problem though?

  4. #4
    Super Contributor
    Join Date
    Dec 2005
    Location
    Europe/Poland/Warsaw
    Posts
    1,699

    Re: http & outputstream memory problem

    hi,
    not sure, I'm newbie still,

    someone knows how to correctly write such test case code for submitted issue (OutOfMemoryError):
    Code:
        void runTest() throws IOException{
            ByteArrayOutputStream os = new ByteArrayOutputStream();
            // 700kb resource file for test
            InputStream is = getClass().getResourceAsStream("/resources/phantom.mpg");
            byte[] b = new byte[512];
            int read = 0;
            while((read = is.read(b, 0, 512))!= -1){
                os.write(b);   
            }
            is.close();
            os.close();
        }
    if run via emulator it should through most probably OutOfMemoryError uncough exception (while writing bytes into output stream),
    bytes should be written by chunks,
    thanks in advance,
    regards,
    peter

  5. #5
    Regular Contributor
    Join Date
    Feb 2006
    Posts
    150

    Talking Re: http & outputstream memory problem

    Hi pillar and peter!

    I'm working with HTTP connections and I've almost send 4KB packets and without any problems of memory!

    The best way, as I know, is to use a "ByteArrayOutputStream" object, like peter shows in his POST, and when all the information is stored in it you will only have to "write" and "flush" it in the corresponding "HttpConnection"!
    I am absolutely sure about the reliability of this code:

    Your problem might be due to the written data is greater than the chunked HTTP packet's length and so you are writing directly into the "HttpConnection" this issue is manifested by means of an OutOfMemoryException! The HTTP packet is sliced into "chunked" data packets (pieces) when the total length is greater than the HTTP PAYLOAD maximum length!

    Code example:


    // opening the file connection and inputstream
    fc = (FileConnection)Connector.open("file:///somefile", Connector.READ);
    is = fc.openInputStream();

    // opening http connection and outputstream
    HttpConnection httpc = (HttpConnection)Connector.open(url, Connector.WRITE);
    httpc.setRequestMethod(HttpConnection.POST);
    httpc.setRequestProperty("User-Agent", "Profile/MIDP-2.0 Configuration/CLDC-1.1");
    httpc.setRequestProperty("Content-Type", type);
    httpc.setRequestProperty("Connection", "close");

    //Open the OS object pointed to the HTTP connection
    OutputStream os = httpc.openOutputStream();

    //I think you won't have problems with Memory if you use this BAOS
    //Create the BAOS object
    ByteArrayOutputStream baos = new ByteArrayOutputStream();

    //Write the FILE in the BAOS object
    writeFileInBaos(baos);//After this, baos will content the target file!

    //CHEKS that the BAOS object's size()>0 before writing into the OS object!
    //Only an unique writting in the HTTP connection, it works PERFECTLY for me!
    os.write(baos.toByteArray());//I used this one!

    //OR this equivalent
    //byte[] b=baos.toByteArray();
    //os.write(b,0,b.length());

    //Flush the data (optional if we read the HTTP_ResposeCode later)
    os.flush();

    //Get the HTTPResponse (it "flushes" the OUTPUT data automatically)
    int rc=http.getResponseCode();

    //Close the HTTP connection
    if (os!=null){
    os.close();
    }
    if (httpc!=null){
    httpc.close();
    }

    //Close FILE conneciton
    if (is!=null){
    is.close();
    }
    if (fc!=null){
    fc.close();
    }

    //Call Garbage Collector if you want to do it!
    System.gc();
    }

    If this example doesn't work for you, then try to use a DataOutputStream instead of an OutputStream pointing to your HTTP Connection!

    Sincerely, I finally used the DataOutputStream object but I tested my code with an OutputStream object previously!

    Best regards and good luck!

    Summerman. :-)---
    Last edited by Summerman; 2006-10-24 at 12:49.

  6. #6
    Regular Contributor
    Join Date
    Feb 2006
    Posts
    112

    Re: http & outputstream memory problem

    Summerman, thank you for extensive example. Unfortunately I have to report back that I was not able to make it work without memory problems. I tried using ByteArrayOutputStream and writing that to OutputStream or DataOutputStream, but there was no difference. Could the problem be related to the fact that since I send multiple packages, it will somehow keep open the connection?

    How much is the http payload maximum length? Right now I'm trying to transfer files of 1,8mb and 7mb. I have tried writing 4k, 50k, 100k, 200k and 400k in one http request. Is there a better amount?

    This is really a tricky problem - my problem is at halt before this is figured out :/ Has anyone been able to transfer big files through http without running into these memory problems?

  7. #7
    Registered User
    Join Date
    Sep 2006
    Posts
    14

    Re: http & outputstream memory problem

    Hello friends,
    I am also trying to send Multiple photos in a single connection,for this I am coverting the images to byte array, then appending the each array(image) in a single Big bytearray then writing this array to the stream.And my code is working fine for one or two photos. But when i am sending more then two or three photos it gives the IO problem"http 400,bad Request". I am not able to find out the actual reason for this problem .Is this error due to to memory shortage?

    Please help me regarding this problem..

    I AM DESPERATELY WAITING FOR REPLIES ON THIS TOPIC.

    Ragards
    RAJESH

  8. #8
    Regular Contributor
    Join Date
    Oct 2006
    Posts
    157

    Re: http & outputstream memory problem

    I've succeeded writing a small amout of data each time to HttpConnection's output stream. Even if it's a large file it works fine (without filling up memory) on S60-devices (at least N73) and some Sony Ericsson devices (k800i, w880i).

    However on S40 (or at least 6280/6288) I get an out of memory exception. I beleive this is because the java engine works differently (and by some reason puts everything in the output stream in memory before transfer).

    Does anyone have any idea how to get around this? I been trying by flushing the output stream each time I write new data to it, but it doesn't make any diffence.

    The only workaround I could figure out is to do multiple connections, and then put together all data server-side. But this is a little messy.

  9. #9
    Registered User
    Join Date
    Aug 2007
    Posts
    1

    Re: http & outputstream memory problem

    I'm working with Nokia E90 and N9500 and I'm facing the very same problem. For some reason the device first gathers all the request data to some internal buffer and the request is not sent before I call HttpConnection.getResponseCode(). Sending larger than 8 MB at a time seems inpossible on the devices I mentioned. I never load the data in to the device's memory before sending: I always open the data to be sended as InputStream and read&write directly from it writing directly to my OutputStream using write().

    The only solution I have been able to find out is to send the file in small pieces.

Similar Threads

  1. Memory Problem in Nokia 6630
    By hiteshkkk in forum General Development Questions
    Replies: 0
    Last Post: 2006-05-24, 16:06
  2. Memory leakage problem??
    By symnewbie in forum Symbian C++
    Replies: 1
    Last Post: 2005-06-10, 08:03
  3. Strange memory problem
    By VMarinov in forum Symbian C++
    Replies: 1
    Last Post: 2005-01-22, 09:10
  4. HTTP Connection problem with real series 40 device
    By pagadi in forum Mobile Java Networking & Messaging & Security
    Replies: 2
    Last Post: 2004-06-17, 09:28
  5. please help~ Memory Problem With Nokia 3650
    By esinu in forum Mobile Java General
    Replies: 2
    Last Post: 2003-07-30, 03:28

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •