Limit Space and Memory Used by Imagemagick

ImageMagick memory usage

ImageMagick needs 8 bytes per pixel if you are using a Q16 build. A Q8 build only needs 4 bytes per pixel.

82500 * 40900 * 8 = about 27Gbytes
82500 * 40900 * 4 = about 13.5 Gbytes

The size of the PNG is irrelevant; ImageMagick stores them uncompressed.

Possibly ImageMagick is trying to hold two copies -- your 100 small images plus the large result. It may be that you'll have enough memory plus disk to run your conversion with ImageMagick-Q8.

How can I prevent OOMs when resizing JPGs with ImageMagick (without increasing memory)?

I think you have the arguments in the wrong order.

If you're on linux, /usr/bin/time is very convenient for measuring peak memory use. You are running this command:

$ /usr/bin/time -f %M:%e \
    convert \
        5500kb-image.jpg \
        -limit memory 5MiB -limit map 10MiB \
        -resize 400 5500kb-image-resized.jpg
346060:2.37

ie. 350mb of memory and 2.37s of CPU time.

If you put the limit first, you see:

$ /usr/bin/time -f %M:%e \
    convert \
        -limit memory 5MiB -limit map 10MiB \
        5500kb-image.jpg -resize 400 5500kb-image-resized.jpg
105468:2.64

Now it's just 105mb peak, and about the same runtime. You need to set the limits before you load the image.

As Mark says, vipsthumbnail is quite a bit faster and less memory hungry. I see:

$ /usr/bin/time -f %M:%e \
    vipsthumbnail 5500kb-image.jpg --size 400 -o 5500kb-image-resized.jpg
133676:0.28

130mb and 0.28s.

That's still rather high. Your image is a progressive JPG --- these must be loaded entirely into memory before thumbnailing can start, and it's not possible to exploit things like jpeg shrink-on-load.

If I convert your image to a regular jpeg, I see:

$ /usr/bin/time -f %M:%e \
    vipsthumbnail 5500kb-image-regular.jpg --size 400 -o 5500kb-image-resized.jpg
40200:0.09

40mb and 0.1s of CPU.

vipsthumbnail can resize very large images with only a small amount of memory, for example:

$ vipsheader st-francis.jpg 
st-francis.jpg: 30000x26319 uchar, 3 bands, srgb, jpegload
$ /usr/bin/time -f %M:%e \
    vipsthumbnail st-francis.jpg --size 400 -o 5500kb-image-resized.jpg
49692:2.52

50mb and 2.5s.

Memory limit reached creating a thumbnail of large image with Imagick

If you want to limit the memory used by ImageMagick you should set one of the environment variables (e.g. MAGICK_MEMORY_LIMIT) or modify the policy.xml file. For more info visit the following page: http://www.imagemagick.org/script/resources.php

ImageMagick: how to achieve low memory usage while resizing a large number of image files?

Without having direct access to your system it's really hard to help you debugging this.

But you can do three things to help yourself narrowing down this problem:

Add -monitor as the first commandline argument to see more details about what's going on.
(Optionally) add -debug all -log "domain: %d +++ event: %e +++ function: %f +++ line: %l +++ module: %m +++ processID: %p +++ realCPUtime: %r +++ wallclocktime: %t +++ userCPUtime: %u \n\r"
Temporarily, don't use '*.ppm[1280x1280]' as an argument, but use 'a*.ppm[1280x1280]' instead. The purpose is to limit your wildcard expansion (or some other suitable way to achieve the same) to only a few matches, instead of all possible matches.

If you do '2.' you'll need to do '3.' as well otherwise you'll be overwhelmed by the mass of output. (Also your system does seem to not be able to process the full wildcard anyway without having to kill the process...)

If you do not find a solution, then...

...register a username at the official ImageMagick bug report forum.
...report your problem there to see if they can help you (these guys are rather friendly and responsive if you ask politely).

PHP ImageMagick use 10gb~ memory when resizing a big image

If your attempting to configure ImageMagick's memory polices to use more RAM, before caching to disk, define the AREA limit by modifying policy.xml or set an environment variable.

Setting Memory limit in Environment

This is probably the quickest way for one-off process.

MAGICK_AREA_LIMIT=8GB convert -resize 90% source.jpg destination.jpg

Also attempt adjusting other environment vars

MAGICK_MAP_LIMIT - Maximum amount of memory map in megabytes to allocate for the pixel cache.
MAGICK_MEMORY_LIMIT - Maximum amount of memory in megabytes to allocate for the pixel cache from the heap.

Setting Memory limit with Policy.xml

Edit the policy.xml under your {$PREFIX}/lib/ImageMagic-X.X.X/config directory.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
<!ELEMENT policymap (policy)+>
<!ELEMENT policy (#PCDATA)>
<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
<!ATTLIST policy name CDATA #IMPLIED>
<!ATTLIST policy rights CDATA #IMPLIED>
<!ATTLIST policy pattern CDATA #IMPLIED>
<!ATTLIST policy value CDATA #IMPLIED>
]>
<policymap>
  <policy domain="resource" name="memory" value="8GB"/>
  <policy domain="resource" name="map" value="8GB"/>
  <policy domain="resource" name="area" value="8GB"/>
</policymap>

Verification

Whatever method you choose, you can verify by running the identify utility.

$ identify -list resource

  File       Area     Memory        Map       Disk   Thread  Throttle       Time
--------------------------------------------------------------------------------
  1920        8GB        8GB        8GB  unlimited        1         0  unlimited