Hadoop Mapper Is Failing Because of "Container Killed by the Applicationmaster"

Hadoop Mapper is failing because of Container killed by the ApplicationMaster

Include below properties in yarn-site.xml and restart VM,

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>

<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>4</value>
   <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

Why the identity mapper can get out of memory?

After some hours of researching and trial and error I realized that the machines I provisioned for the TASK group were small instances with not much memory and, more interestingly, that the point in which I was running out of memory was during shuffling instead of mapping.

Related Topics