Difference Between "On-Heap" and "Off-Heap"

Difference between on-heap and off-heap

The on-heap store refers to objects that will be present in the Java heap (and also subject to GC). On the other hand, the off-heap store refers to (serialized) objects that are managed by EHCache, but stored outside the heap (and also not subject to GC). As the off-heap store continues to be managed in memory, it is slightly slower than the on-heap store, but still faster than the disk store.

The internal details involved in management and usage of the off-heap store aren't very evident in the link posted in the question, so it would be wise to check out the details of Terracotta BigMemory, which is used to manage the off-disk store. BigMemory (the off-heap store) is to be used to avoid the overhead of GC on a heap that is several Megabytes or Gigabytes large. BigMemory uses the memory address space of the JVM process, via direct ByteBuffers that are not subject to GC unlike other native Java objects.

Difference between On-heap and Off-heap memory

All memory is native memory, however the JVM manages and record memory in it's JVM heap (not the same as the native heap)

Offheap is a Java term for memory not managed directly. However, it can be managed indirectly using direct ByteBuffer(s) as proxy objects for raw native memory.

What is the difference between off-heap, native heap, direct memory and native memory?

1) Heap memory: memory within the JVM process that is used to hold Java Objects and is maintained by the JVMs Garbage Collector.

2) Native memory/Off-heap: is memory allocated within the processes address space that is not within the heap and thus is not freed up by the Java Garbage Collector.

3) Direct memory: is similar to native, but also implies that an underlying buffer within the hardware is being shared. For example, a buffer within the network adapter or graphics display. The goal here is to reduce the number of times the same bytes is being copied about in memory.

Finally, depending upon the OS then extra native allocations (assigning of the memory address space) can be carried out via Unsafe alloc and/or by memory mapping a file. Memory mapping a file is especially interesting as it can easily allocate more memory than the machine currently has as physical ram. Also note, that the total address space limit is restricted by the size of a pointer being used, a 32bit pointer cannot go outside of 4GB. Period.

How to set OFF-HEAP or ON-HEAP Memory in Apache Ignite?

In the current memory architecture (Apache Ignite 2.x, see this link) you can't choose on-heap memory only.

The data is always stored on the off-heap. It will be fetched (although not completely) onto the heap to perform all kinds of processing. For example, while JOIN and WHERE of an SQL query can mostly be done right in the off-heap, the final result set must be fetched onto the heap.

In your case, you're committing more memory to Ignire than you have. You commit 12GB off-heap + 15GB heap = 27GB, which is obviously more than your 16 GB RAM.

In general, you should commit just enough memory to store the data (+ indexes and overhead!) you need on the off-heap, and you can give the rest to the heap. You should also leave some RAM free for the system needs. But with persistence enabled you can have even more data than off-heap memory - but keep in mind that if your data doesn't fit in the off-heap data region, performance will suffer.

By the way, remove your pageEvictionMode - it has no meaning when persistence is enabled (yes, documentation fails to highlight that).

Finally, if you get an OutOfMemory for your heap space, it probably means that your SQL result set is too large. To workaround that you can either

make the result set smaller by splitting your SQL query into several ones returning less data
increase heap size
use SqlFieldsQuery.setLazy(true) - with this flag Ignite will try to split the result set into chunks, if possible, and load them onto heap one by one

apache ignite on-heap and off-heap memory

An on-heap cache is in addition to the off-heap storage.

So:

No
Anything you save into the cache/table
There are two kinds of eviction. Eviction on data regions is at the page level rather than the record level, so a number of records can be evicted together. On-heap cache works on a row level. Evicting a record from the on-heap cache does not evict it from the off-heap cache. Evicting from off-heap also evicts from the on-heap cache.