JVM Diagnostics: Separate CPU, Waiting, and Memory Before Tuning
Build an evidence chain from service symptoms to threads, JFR, garbage collection, and native memory in Java 17.
Read in Chinese →Engineering notes with synthetic examples. Version scope and references are included in each article.
Scope: HotSpot JDK 17, with G1 as the garbage-collector example. Replace the example PID
12345before running commands. In containers, also verify PID namespaces and diagnostic-tool permissions.
“The endpoint is slow, so increase the heap” is an expensive guess. Latency may come from queueing, lock contention, networking, CPU throttling, or garbage collection. The first task is to distinguish those explanations.
Capture evidence from the same time window
Record incident onset, deployed version, request volume, error rate, latency percentiles, and dependency latency. Inspect process CPU, memory, container limits, and disk at the same time. Comparing a GC chart with request and host metrics from a different period makes false causal explanations easy to construct.
Then establish the runtime environment:
jcmd 12345 VM.version
jcmd 12345 VM.flags
jcmd 12345 GC.heap_info
jcmd 12345 Thread.print -l
Use jcmd 12345 help to check supported commands and review their impact before running them. The commands are documented in JDK 17 jcmd
. A command working on a development machine does not establish that it is available inside a production container.
When CPU is high, locate the work
High CPU may reflect more business computation, serialization, regular expressions, busy waiting, or GC. A thread stack offers a clue, but one snapshot can land in incidental code. Repeated occurrences of the same stack, supported by CPU sampling, are stronger evidence.
During an appropriate diagnostic window, a short JFR recording can help:
jcmd 12345 JFR.start name=incident settings=profile duration=60s filename=/tmp/incident.jfr
Inspect hot stacks, locks, allocations, and I/O events, then connect them back to the request path. The profile configuration adds overhead and should first be evaluated in a comparable environment. Store recordings somewhere with enough capacity and controlled access. See the JDK diagnostic tools guide
for JFR capabilities and operation.
An application may also lack CPU time even when process CPU looks modest, because the container is being throttled. Adding threads can intensify that competition. Read process metrics alongside host conditions and container quotas.
When CPU is low but latency is high, inspect waits
Determine whether threads are waiting for database connections, HTTP connections, locks, queues, or futures that never complete. Then inspect utilization and waiting time for the corresponding resource.
If workers are waiting for a connection pool whose connections are occupied by slow SQL, increasing workers from 32 to 128 will not make the database finish more queries. It adds waiters. The fix must address the actual bottleneck: query performance, connection hold time, or dependency capacity.
Also inspect dependencies between tasks in the same executor. A task that submits a child task to a saturated pool and then waits synchronously for it can cause thread starvation. A diagram of who is waiting for whom is more useful than a count of threads in WAITING state.
Rising memory is not automatically a heap leak
Heap occupancy normally rises within a GC cycle. The more useful questions are whether post-GC live data keeps growing under comparable traffic and which objects remain reachable. Unbounded caches, queues, and unregistered listeners can all produce growth that follows the code’s logic but violates its capacity assumptions.
Process RSS also includes direct memory, thread stacks, metaspace, and other native allocations. If the heap is stable while the process grows, extend the investigation to those areas. Native Memory Tracking can compare a baseline with later measurements when enabled at startup. It does not account for every third-party native allocation, so it is not a complete RSS ledger.
A heap dump can expose retention paths, but collection may cause pauses, consume substantial disk space, and capture business data. Collect one when it is needed to answer a specific question and capacity is available. Lower-cost observations should narrow the hypothesis first.
Tune GC for a confirmed problem
For G1, record pause causes and frequency, allocation rate, post-GC live data, and large-object behavior. The pause target is a tuning objective, not a hard latency guarantee. The official guidance recommends starting with defaults and an appropriate heap limit, then tuning for observed behavior: G1 Tuning .
Change one major variable at a time and compare throughput, tail latency, CPU, and memory under similar load. Changing heap size, GC settings, executors, and connection pools together makes an improvement difficult to explain and the next incident harder to diagnose.
A useful incident record connects the symptom and its timing, the evidence that ruled out alternatives, the mechanism changed by the fix, and the regression results that support the conclusion.