3 ms·
Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in a JVM with a 200 MB heap within which they allocate buffers for in-memory da
by marshray 14y ago
Yes, section 3.4: "Heap size - By default each Hadoop map and reduce task is run in
a JVM with a 200 MB heap within which they allocate
buffers for in-memory data. When the buffers are full,
data is spilled to storage, adding overheads. We note
that 200 MB per task leaves substantial amounts of memory unused on modern servers. By increasing the heap
size for each JVM (and hence the working memory for
each task), we improve performance. However too large
a heap size causes garbage collection overheads, and
wastes memory that could be used for other purposes
(such as a RAMdisk). For the scale-out configurations,
we found the optimal heap size for each job through trial
and error. For the scale-up configuration we set a heap
size of 4 GB per mapper/reducer task (where the maximum number of tasks is set to the number of processors)
for all jobs."