Node.Js Heap Out of Memory

Node.js heap out of memory

If I remember correctly, there is a strict standard limit for the memory usage in V8 of around 1.7 GB, if you do not increase it manually.

In one of our products we followed this solution in our deploy script:

 node --max-old-space-size=4096 yourFile.js

There would also be a new space command but as I read here: a-tour-of-v8-garbage-collection the new space only collects the newly created short-term data and the old space contains all referenced data structures which should be in your case the best option.

Javascript heap out of memory while running a js script to fetch data from an api every minute- javascript/node.js

From the data you've provided, it's impossible to tell why you're running out of memory.

Maybe the working set (i.e. the amount of stuff that you need to keep around at the same time) just happens to be larger than your current heap limit; in that case increasing the limit would help. It's easy to find out by trying it, e.g. with --max-old-space-size=8000 (megabytes).

Maybe there's a memory leak somewhere, either in your own code, or in one of your third-party modules. In other words, maybe you're accidentally keeping objects reachable that you don't really need any more.

If you provide a repro case, then people can investigate and tell you more.

Side notes:

according to your output, heap memory consumption is growing to ~4 GB; not sure why you think it tops out at 1.2 GB.
it is never necessary to invoke global.gc() manually; the garbage collector will kick in automatically when memory pressure is high. That said, if something is keeping old objects reachable, then the garbage collector can't do anything.

A very strange JavaScript heap out of memory issue

After a good few hours of debugging with trial and error, I fixed with the following changes to config.yml:


  build_and_push_image:
    machine:
      image: ubuntu-2004:202101-01
    resource_class: arm.xlarge
    steps:
      - checkout
      - docker/pull:
          images: 'node:16.14.2'
      - aws-ecr/build-and-push-image:
          aws-access-key-id: AWS_ACCESS_KEY_ID
          aws-secret-access-key: AWS_SECRET_ACCESS_KEY
          aws-cli-version: latest
          create-repo: true
          skip-when-tags-exist: true  
          dockerfile: Dockerfile
          extra-build-args: >
            --build-arg GPG_ENCRYPT_PASSPHRASE
            --build-arg BUILD_STAGE
            --build-arg CI_BUILD=true

-         platform: linux/arm64/v7
+         platform: linux/arm64

          push-image: true
          registry-id: AWS_ACCOUNT_ID
          region: $AWS_REGION
          repo: $PROJECT_NAME
          tag: $version_tag

I'm not sure why this works, but here's my best thoughts for others who are blocked by this in the future:

Some miconfiguration of the docker buildx command from the CircleCI orb. I'm also noticing that a lot of people are having problems with Docker image caching using the orb, and maybe there's some issue with the command itself (see https://github.com/CircleCI-Public/aws-ecr-orb/issues/202). It looks something like this:

docker context create builder

# This here is interesting - it is using qemu to emulate for builds 
# See https://github.com/tonistiigi/binfmt

docker run --privileged --rm tonistiigi/binfmt --install all
docker --context builder buildx create --use
docker --context builder buildx build \
  -f "${PARAM_PATH}"/"${PARAM_DOCKERFILE}" \
  ${docker_tag_args} \
  --platform "${PARAM_PLATFORM}" \
  --progress plain \
  "$@" \
  "${PARAM_PATH}"

https://github.com/docker/buildx/issues/1036 missing qemu dependencies
https://github.com/moby/buildkit/issues/1946#issuecomment-767374876 libseccomp blocking unknown (new) syscalls

npm install - javascript heap out of memory

You can launch NPM using :

node --max-old-space-size=8000 $(which npm) install -g ionic

As described here, the default is 4000 (4Gb).

Node.Js Heap Out of Memory