Yes, it is a tricky problem.
Decompressed, an image is likely to take (depending on device) between two and four bytes per pixel. At four bytes per pixel, a 5Mpixel image needs 20Mb of memory... bad news for almost anything running Java ME.
In 2Mb of heap, with a need to keep two copies of the image in memory, even 640x480 could push you over the edge.
Avoid calling System.gc(). It won't have any benefit on any device with a compacting garbage collector (all but the very earliest of S40 devices), and causes a big performance drop on some devices (especially Nokia Series 60 devices).
Pretty much your only option would be to maintain images in compressed form, and decompress the image only in small parts. For example, decompress one scanline, process it, then compress the processed version before decompressing the next scanline. The built in decoders are not likely to help you, so you'd need to implement your own, not a trivial job unless you've done this before.
Forcing the camera resolution:
This is likely to throw an exception if the resolution you ask for is not supported by the camera.
byte snapshot = control.getSnapshot("encoding=image/jpeg&width=" + WIDTH + "&height=" + HEIGHT);