Repository navigation
fix(server): keep Server crash diagnostics after a container restart - #3258
Conversation
When a hugegraph/server container exited during startup, kubectl logs showed only "Starting HugeGraphServer failed". The cause stayed in logs/ inside the container, and Kubernetes dropped it on restart. - Set STDOUT_MODE=true in Dockerfile-hstore. apache#2980 did this for the standalone Dockerfile and missed this one. - In conf/log4j2.xml, also send WARN and above from the hadoop, zookeeper, sofa, netty and commons loggers to console. They were file-only. - In hugegraph-server.sh, always set -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath and -XX:ErrorFile under $LOGS. The heap-dump flags used to be dropped whenever JAVA_OPTIONS was set, and hs_err files went to the install directory. The file names carry the launch time because HotSpot will not overwrite an existing crash log or heap dump, and a restarted container often reuses the PID. The flags go before JAVA_OPTIONS and -j, so values given there still win; a flag already set in JAVA_TOOL_OPTIONS or JDK_JAVA_OPTIONS, which the JVM reads before the command line, is left out. - Say "container logs" rather than "docker logs" in the start failure message, since the same image runs on Kubernetes. - Cover the new flags and their overrides in test-java-security-properties.sh, and document the log directory and the Kubernetes mount in docker/README.md. This applies to every Server install, bare metal included: an OOM now writes a heap dump to logs/ even when JAVA_OPTIONS is set. Fixes apache#3256
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3258 +/- ##
=========================================
Coverage 41.71% 41.71%
- Complexity 7036 7037 +1
=========================================
Files 762 762
Lines 67056 67060 +4
Branches 9020 9021 +1
=========================================
+ Hits 27970 27975 +5
- Misses 35876 35877 +1
+ Partials 3210 3208 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: When telemetry is enabled, crash options supplied through JAVA_TOOL_OPTIONS can be discarded together with the launcher defaults. Evidence: static trace of the crash-option scan and the telemetry environment assignment.
With telemetry on, the launcher replaced JAVA_TOOL_OPTIONS with the OpenTelemetry -javaagent option. The crash-file defaults leave out any flag already set in JAVA_TOOL_OPTIONS, so an operator's -XX:ErrorFile there was dropped twice and HotSpot wrote hs_err to the working directory. Append the agent to the existing value instead, and cover the telemetry path in test-java-security-properties.sh with a stubbed agent checksum.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Holding approval until the two findings below are addressed in the published branch. I prepared fixes and regression checks locally, but cannot push to the PR's fork, so those changes are not part of this review's commit.
The local fix lets the JVM handle diagnostic-option precedence by prepending defaults to JAVA_TOOL_OPTIONS, and removes the temporary telemetry JAR through the test's EXIT cleanup. Focused checks reproduced the quoted heap-dump opt-out failure on this PR head and passed with the local fixes, covering quoted environment options, JDK argument files, telemetry preservation, and cleanup on failure. Shell syntax, whitespace, and editorconfig checks passed. A full Maven compile was attempted but failed in hugegraph-common under Maven's locally selected JDK 26; full-build validation remains outstanding.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Heap-dump filenames can collide when a container restarts within one second and reuses its PID, so consecutive failures cannot retain distinct dumps. Evidence: hugegraph-server.sh:138,154; OpenJDK 11 defaults the heap dump writer to overwrite=false.
Review follow-ups on the crash-file defaults: - Put the defaults at the front of JAVA_TOOL_OPTIONS instead of on the command line, and drop the shell scan of JAVA_TOOL_OPTIONS and JDK_JAVA_OPTIONS. The JVM reads the operator's JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, the command line and _JAVA_OPTIONS after the defaults, so any of them overrides a default. Word splitting in the shell missed quoted options and @argfiles, and could treat flag-like text inside a property value as a flag. - Add a counter to the launch time in the file names when a heap dump or crash log with that name already exists in $LOGS. A restart in the same second with the same PID no longer targets an existing file. - Test the values a real JVM resolves, covering a quoted opt-out, an @argfile in JDK_JAVA_OPTIONS, flag text in a property value, operator paths in JAVA_OPTIONS, name collisions and telemetry. The EXIT cleanup now removes the telemetry jar and log fixtures, so a failing step does not leave them in the distribution.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The latest-head checks pass, but one test-fixture cleanup issue remains. Evidence: the new crash-file fixture uses fixed paths under the supplied distribution's logs/ and removes them unconditionally.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 10/10. Summary: At ac9d98b the heap dump and crash log defaults go first in JAVA_TOOL_OPTIONS, so the JVM's own precedence lets JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, JAVA_OPTIONS, -j and _JAVA_OPTIONS override them, file names get a counter when a name is already taken in logs/, and the telemetry agent is appended instead of replacing JAVA_TOOL_OPTIONS. The four earlier inline threads are addressed in this head and I found no new issues. Evidence: static review of the full diff and of hugegraph-server.sh, start-hugegraph.sh, docker-entrypoint.sh, Dockerfile-hstore and log4j2.xml at this head. CI at this head: 22 of 22 checks pass, and the build-server (rocksdb, 11) job ran the 'Run Java security properties tests' step, which starts a real JVM with the launcher's captured options, with conclusion success.
The collision check created two fixed crash-file names under the supplied distribution's logs/ and removed them afterwards, which would delete real files that already had those names. Record and remove only the files this run creates.
…iles Review follow-ups: - Docker cluster guide: no 1.7.0 image sets STDOUT_MODE, the standalone hugegraph/hugegraph:1.7.0 used on the page included. Describe stdout logging as current master behaviour for all images, and widen the version note to every 1.7.0 and older image, linking #2980 and #3258. - Server quickstart: the defaults now go first in JAVA_TOOL_OPTIONS (apache/hugegraph#3258), so list every source that overrides them, _JAVA_OPTIONS included, describe the counter added to a taken file name, and mention the "Picked up JAVA_TOOL_OPTIONS" startup line. - Chinese pages: wrap only at existing spaces, so a soft break no longer renders as a space between two Chinese characters.
|
The two red checks look unrelated to this change: |
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 9/10. Summary: At e7c16d2 the HStore Server image logs to stdout, third-party WARN reaches the console, and the heap dump and crash log defaults go first in JAVA_TOOL_OPTIONS so every operator source still overrides them. The earlier inline threads are addressed, including the fixture cleanup fix in this head, and I found no new issues. Evidence: static review of the full diff plus docker-entrypoint.sh, both Server Dockerfiles, log4j2.xml and server-ci.yml at this head; on JDK 11.0.32, quoted HeapDumpPath and ErrorFile values with a space in the path resolve correctly, and an exec'd JVM wrote its OOM dump to logs/java_pid_.hprof with the launcher's PID. CI at this head: all 22 checks pass, including the rocksdb lane that runs test-java-security-properties.sh.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Unique crash-dump names preserve each OOM dump across restarts, but a persistent log volume can grow by a heap-sized file on every repeated OOM. Please document recurring-dump capacity and cleanup, or add configurable retention. Evidence: hugegraph-server.sh:148-155 generates a per-launch HeapDumpPath; docker/README.md:237 recommends a PersistentVolumeClaim and sizes it for a dump.
Each launch writes its heap dump to a new file, so a Server that keeps running out of memory adds a heap-sized dump per restart until a persistent /hugegraph-server/logs volume is full. Say that nothing removes old dumps, and how to size the volume, clean up, or turn heap dumps off.
Every launch writes its heap dump to a new file (apache/hugegraph#3258), so a server that keeps running out of memory adds a heap-sized dump per restart until the volume is full. Say that nothing removes old dumps, and how to size the volume, clean up, or turn dumps off, in the Docker guide and the Server quickstart, English and Chinese.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 9/10. Summary: Since e7c16d2 this head adds only the docker/README.md section on repeated OOM dumps, which answers the last open thread, plus a master merge that changes none of the PR's files. I found no new issues. Evidence: I read the full diff against merge base 5004a35 and confirmed the java -version probe runs before the JAVA_TOOL_OPTIONS export and that the launcher execs java. On a real JDK 17, a command-line -XX:-HeapDumpOnOutOfMemoryError overrides the launcher's quoted defaults in JAVA_TOOL_OPTIONS, as the README says. After the CI refactor, test-java-security-properties.sh still runs in build-server (rocksdb, 11) and passed at this head. All 28 checks pass.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Two issues remain: crash-file naming can race when pods share a log PVC, and some pre-JVM startup errors still appear only in the file log. Evidence: exact-head source paths in the inline comments; all 28 checks passed.
- Crash-file names now carry the host name, the pod name on Kubernetes, before the launch time. Pods sharing one log volume each run as PID 1 and could start in the same second; with only the PID and time they could pick the same name. - Launcher errors that stop the Server before Java starts (unsupported JDK, too little free memory, unknown GC option, missing security properties, telemetry agent download or checksum failure) now go to stderr as well as hugegraph-server.log, through one report_error helper. start-hugegraph.sh passes that stderr to the terminal or the container log, so kubectl logs --previous shows these causes too. - Tests cover the host part of the names, another host in the same second, unsafe host characters, and a preflight error on stderr and in the server log. docker/README.md is updated to match.
apache/hugegraph#3258 now puts the host name, the pod name on Kubernetes, into heap dump and crash log names so servers sharing one log directory do not pick the same name. Update the Server quickstart in English and Chinese to match.
Resolve the hugegraph-server.sh conflict with the Java 17 baseline (apache#3261): keep the crash-file defaults in JAVA_TOOL_OPTIONS, drop the removed JAVA_VERSION > 9 --add-exports block (now in jvm-module.options), and report the new missing module options error through report_error.
apache/hugegraph#3258 dropped the startup cleanup of heapdump_* directories, since a computer-job JVM can keep using one after its Server exits. Say that each start leaves a directory, empty unless something ran out of memory, and that old ones can be deleted once no JVM from that launch is running. English and Chinese.
Launcher (hugegraph-server.sh): - Create the heap dump directory after every preflight check, just before exec, so a launch that fails before Java starts leaves nothing. - Claim the name with a plain mkdir, so concurrent launches with the same host name and second get distinct names; treat a dangling symlink as taken. - When the directory cannot be created (a full volume, for example), warn and dump into logs/ instead of refusing to start. - Cap the host part at 64 characters and sanitize it with LC_ALL=C. - Skip the defaults with a warning when the logs path contains a double quote or %, which JAVA_TOOL_OPTIONS cannot carry. - Pass -Dhugegraph.bootstrap.error.log in every mode. With STDOUT_MODE the HStore image had stopped copying fatal bootstrap errors into hugegraph-server.log. - report_error writes the log file before stderr and uses printf. - The name check tolerates failglob. - Correct the comments: HotSpot truncates an existing crash log on JDK 17+, HeapDumpPath expands %p from JDK 25, and @argfiles only work in JDK_JAVA_OPTIONS or on the command line. CI: list heap dumps instead of printing them in ci-service-utils.sh, leave them out of the HStore diagnostics artifact, and keep them out of the riscv64 log grep. Tests: check JDK_JAVA_OPTIONS and _JAVA_OPTIONS as the launcher passes them, -j overriding a default, a dangling symlink, that a failed preflight leaves no directory, the mkdir-failure fallback and an unwritable logs/. Clean up only this run's directories, using a test-only host name, and remove OOM dumps on any exit. Keep the OOM runs' output for diagnosis, read flag values that contain spaces, and ignore the runner's JVM option variables. Docs: docker/README.md gains a version scope, the child-JVM-wide opt-out, a warning not to remove a running Server's directory, one log volume per pod, hostNetwork, ephemeral-storage and medium: Memory notes, and the Helm chart's missing logs volume; the Helm README points to it.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The crash-diagnostics changes are largely in place, but two Docker documentation details need clarification. Evidence: exact-head launcher and Compose configuration.
…OOL_OPTIONS - configure_riscv64_libatomic takes an optional error handler, like ensure_path_writable, and the launcher defines report_error before calling it, so a missing libatomic.so.1 also reaches hugegraph-server.log. Other callers keep the plain stderr message. - docker/README.md says JAVA_TOOL_OPTIONS has to be set in the container's environment (docker run -e, a Compose environment: entry, Kubernetes env); the shipped Compose files do not pass it through. - The test checks the riscv64 error reaches stderr and the server log, with uname and ldconfig mocked.
- Say "all taken" when the 1000 counter values for a stamp are used, instead of naming the 1000th directory as uncreatable. - In the test, feed PrintFlagsFinal output to awk through a here-string and filter bootstrap stderr with one awk call; with a pipe, an early awk exit could kill the writer with SIGPIPE under pipefail on large output. - docker/README.md: both images declare VOLUME /hugegraph-server, so a Compose recreate keeps the files and docker compose down leaves an unattached anonymous volume until down -v; recommend a named volume or bind mount. Note that the full-volume fallback writes plain java_pid<pid>.hprof files into logs/, where a reused PID cannot write over an earlier dump.
Update the Server quickstart and the Docker guide, English and Chinese, to match apache/hugegraph#3258: - -j or JAVA_OPTIONS turns heap dumps off for the server JVM only; JAVA_TOOL_OPTIONS reaches the JVMs it starts. - Never move the newest heapdump_* directory of a running server. - The directory is created even when dumps are off, and the server starts anyway when it cannot be created. - One log volume per pod, ephemeral-storage and sizeLimit, no medium: Memory, and the Helm chart's missing logs volume. - The 1.7.0 note says dumps used to go straight to logs/java_pid<pid>.hprof. - Name the script, give the launch time format, fix the garbled sentence about why directories are kept, and polish the Chinese.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The review found three documentation corrections around Kubernetes hostname naming, per-Pod PVC subpaths, and heap-dump fallback when the volume is full. Evidence: Exact-head launcher and Helm template review, plus Kubernetes documentation.
- On Kubernetes the host name is the pod name, or spec.hostname when set; drop the hostNetwork exception and note that the counter also separates pods sharing a host name. - Call the fallback to logs/ best effort: a full volume may have no room for the dump either. - subPathExpr: $(POD_NAME) only isolates pods when POD_NAME comes in through the Downward API; show the env and volumeMounts snippet and say the Helm chart sets neither.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The crash-diagnostics changes are covered, but the telemetry test fixture can leave an unintended file through a dangling symlink. Evidence: static trace of test-java-security-properties.sh:922-925 and cleanup at 934-936.
The telemetry case created plugins/opentelemetry-javaagent.jar in the supplied distribution when `[[ ! -e ]]` said it was missing. A dangling symlink there passes that check, so `: >` followed the link and created its target, possibly outside the distribution, while cleanup removed only the link. Copy bin/ and conf/ into TEMP_DIR and put the empty agent jar there, so the test never writes into the supplied plugins/. TEMP_DIR cleanup now covers the fixture, and a failing launcher run reports its stderr.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Four non-blocking edge cases remain in crash-diagnostic paths, the test harness, and its operator documentation. Evidence: exact-head trace of the HStore entrypoint, launcher options, and test path handling.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The change keeps Server crash diagnostics across container restarts as described: the HStore image now logs to stdout, launcher errors reach stderr and hugegraph-server.log, and the crash-file defaults sit first in JAVA_TOOL_OPTIONS so every operator source still overrides them. Three non-blocking points remain: child heap dumps in one launch directory are keyed only by PID, the new test assumes a non-symlinked distribution path, and one README sentence overstates the empty-directory behaviour. Evidence: read the full exact-head diff at a2d9772 plus HugeGraphServerBootstrap.reportFatal, AbstractComputer's ProcessBuilder path, docker-entrypoint.sh and start-hugegraph.sh; all latest-head checks pass (gh pr checks 3258). No local test run: the launcher test needs a built distribution.
- HotSpot names each dump java_pid<pid>.hprof, so a JVM that reuses an earlier PID within the same launch cannot write its dump. Say so in the launcher comment and docker/README.md instead of promising one dump per JVM. - docker/README.md: only errors that hugegraph-server.sh reports reach hugegraph-server.log (when logs/ is writable); earlier entrypoint steps such as init-store.sh in the HStore image report to stderr only. The empty heapdump_* directory is created unless that failed and dumps fell back to logs/. - test-java-security-properties.sh: resolve SERVER_ROOT with pwd -P, the way the launcher resolves its own location, so a distribution passed through a symlink no longer fails the path assertions.
- HotSpot names each dump after the JVM's PID, so a JVM that reuses an earlier PID from the same start cannot write its dump. - Each start creates its empty heapdump_* directory unless creating it failed and dumps fell back to logs/. English and Chinese quickstart, matching apache/hugegraph#3258.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The exact-head review found two non-blocking issues in test cleanup and startup documentation. Evidence: static review of the exact-head diff; latest-head CI has one failure in VertexCoreTest.testAddVertexWithCollectionIndex with ConcurrentModificationException.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The crash-diagnostics change works as described for normal install paths, but when the logs path contains % or a double quote the launcher now turns heap dumps off entirely, where the base script still enabled them through the JAVA_OPTIONS default it no longer sets. Evidence: read the full exact-head diff against merge base e62c961 plus the launcher, start-hugegraph.sh, docker-entrypoint.sh, log4j2.xml and AbstractComputer; ran the head's crash-option block (hugegraph-server.sh:290-339) with LOGS=
The collision, other-host and preflight cases used fixed host names
(test-pod-a, test/pod-b, test-preflight) and registered the directories
the launcher would create for them for removal. An empty directory with
one of those names left by an interrupted run was then deleted by the
next run even though that run did not create it.
Derive those host names from TEST_HOST (hgtest-<pid>-a and so on), so
no earlier run can own them, drop the fixed-path list, and let the EXIT
cleanup match only TEST_HOST and "${TEST_HOST}-*" names.
docker/README.md: the "Picked up JAVA_TOOL_OPTIONS" line appears when
the JVM starts; a start that fails its preflight checks exits first.
The defaults cannot go into JAVA_TOOL_OPTIONS when the logs path holds
a double quote or %, and that branch only warned, so such installs lost
the -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} flags the
base always passed: a regression.
In that case keep the base behaviour for the Server: put both flags at
the front of JAVA_OPTIONS, so an operator value there still wins, and
skip only the ErrorFile default. The warning says so, the test covers a
% path and a " path in isolated copies, and docker/README.md describes
the case and qualifies the per-start directory sentence.
apache/hugegraph#3258 keeps the 1.7.0 behaviour for the server when the logs path contains a double quote or %: heap dumps go to logs/ itself, with no per-launch directory and no crash log default, and the launcher warns. Say so in the English and Chinese quickstart.
|
Two macOS core-test failures on recent heads are not from this change, which only touches the launcher scripts, their test, CI log handling and docs:
On the same commits the same core suites pass on |
Resolve the hugegraph-server.sh conflict with apache#3305: keep reporting an unknown GC option through report_error, with apache#3305's new message (supported options: g1, ZGC).
|
|
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The special-path fallback can override the documented heap-dump opt-out, and child JVM PID reuse can truncate an earlier fatal-error log within one Server launch. Evidence: static trace of exact-head JAVA_TOOL_OPTIONS/command-line ordering and AbstractComputer's inherited child-process environment.
The fallback for a logs path with a double quote or % put the heap dump flags on the command line, which the JVM reads after JAVA_TOOL_OPTIONS, so the documented JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError opt-out no longer worked for the Server there. Put the flags at the front of JAVA_TOOL_OPTIONS instead, quoting the path with the quote character it does not contain (HotSpot accepts single quotes there), so every operator source overrides them again. A path with both quote characters gets only a warning. The ErrorFile default stays off for these paths. HotSpot also names crash logs only by %p, so a child JVM that reuses an earlier child's PID within one launch truncates its crash log. The launcher cannot rename it; say so next to the heap dump caveat in the comment and docker/README.md. The test checks the resolved values on a real JVM for %, ", ' with % and both-quote paths, with and without the opt-out.
- HotSpot names crash logs by PID too, so a JVM that reuses an earlier PID from the same start replaces that JVM's crash log. - For a logs path with a double quote or %, the JAVA_TOOL_OPTIONS opt-out still applies, and a path with both quote characters gets no heap dump default. English and Chinese quickstart, matching apache/hugegraph#3258.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: At d3db8e7 the HStore Server image logs to stdout like the standalone image, WARN and above from the Hadoop, ZooKeeper, SOFA, Netty and Commons loggers also goes to the console appender, and the heap dump and crash log defaults sit at the front of JAVA_TOOL_OPTIONS with one heapdump_/ directory per start, so every operator option source still overrides them. All 30 review threads are resolved, including my blocking one from adafb74: a logs path containing % or a double quote now keeps -XX:+HeapDumpOnOutOfMemoryError, set from the front of JAVA_TOOL_OPTIONS (e9dec9c, d3db8e7). I found no new issues. Evidence: read the full diff against merge base fd64f74 plus start-hugegraph.sh, docker-entrypoint.sh, both Server Dockerfiles, log4j2.xml, AbstractComputer.java and server-tests.yml at this head. With a copy of bin/ and conf/ and a stub java under bash 3.2: 25 launches in the same second with one host name claimed 25 distinct directories; logs paths containing %, a double quote, or both quote characters gave the documented fallbacks; an unwritable logs/ exited 1 with the message on stderr. On JDK 17.0.20 a JVM started with the same quoted option form, with a space in the path, wrote its out-of-memory dump to heapdump_x/java_pid.hprof, and a later -XX:-HeapDumpOnOutOfMemoryError in JAVA_TOOL_OPTIONS overrode the single-quoted fallback form. The log4j2 Delete rules (/.log and /.gz) match neither logs/hs_err*.log nor the .hprof dumps. bash -n on the six changed scripts and git diff --check are clean. CI at this head: 33 checks pass, 2 are skipped, none fail, and the server_rocksdb (Java 17) job ran the 'Run Java security properties tests' step with success. I did not build the Docker images or run test-java-security-properties.sh locally.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Thanks, we could simplify the docker-doc later
3 participants
Purpose of the PR
hugegraph/server) keeps crash diagnostics only inside the container, so a restart on Kubernetes loses them #3256When a
hugegraph/server(HStore) container exits during startup,kubectl logs --previousshows onlyStarting HugeGraphServer failedand a pointer tologs/hugegraph-server.log. That file, anyhs_err_pid*.logand any heap dump live inside the container, and Kubernetes discards them when it restarts the pod. In #3203 the stack trace was only recovered by mounting a volume at/hugegraph-server/logsbefore the first boot.Main Changes
Dockerfile-hstoresetsSTDOUT_MODE=true. fix(docker): enable docker logs for pd/store/server containers #2980 added it to the standalone, PD and Store Dockerfiles and missed this one.conf/log4j2.xml: theorg.apache.hadoop,org.apache.zookeeper,com.alipay.sofa,io.nettyandorg.apache.commonsloggers areadditivity="false"with only thefileappender. Each gets<appender-ref ref="console" level="WARN"/>, so WARN and above reaches stdout and INFO stays in the file. Root andorg.apache.hugegraphalready log to console. Audit and slow-query logs are unchanged.hugegraph-server.sh:-XX:+HeapDumpOnOutOfMemoryErrorand-XX:HeapDumpPathused to sit inside theJAVA_OPTIONSdefault block, so settingJAVA_OPTIONSdropped them, and nothing set-XX:ErrorFile, sohs_errfiles went to the install root. Now the defaults go at the front ofJAVA_TOOL_OPTIONSon every start. The JVM reads the operator's ownJAVA_TOOL_OPTIONS,JDK_JAVA_OPTIONS, the command line (JAVA_OPTIONS,-j) and_JAVA_OPTIONSafter that, so any of them overrides a default.logs/heapdump_<host>_<launch time>/, where each JVM writesjava_pid<pid>.hprof; crash logs arelogs/hs_err_pid%p_<host>_<launch time>.log. Child JVMs (computer jobs) inherit the environment, andHeapDumpPathonly expands%pfrom JDK 25, so the directory is what gives every JVM its own dump. The host name separates pods on a shared volume, and a counter claimed with a plainmkdirseparates restarts and concurrent launches. HotSpot truncates an existing crash log (JDK 17+) and will not write a heap dump over an existing file, which is why the names must be unique.exec, so a start that fails earlier leaves nothing. If it cannot be created (a full volume, say), the launcher warns and dumps intologs/instead of refusing to start. It never deletes dump directories, because a computer-job JVM can keep using one after its Server exits.-y true), the OpenTelemetry-javaagentoption is appended toJAVA_TOOL_OPTIONSinstead of replacing it.logs/orplugins/, missing riscv64libatomic.so.1, unsupported JDK, too little memory, unknown GC option, missing security properties, telemetry download or checksum failure) go throughreport_errortohugegraph-server.logand stderr, so they reachkubectl logstoo.-Dhugegraph.bootstrap.error.logis passed in every mode; withSTDOUT_MODEthe HStore image would otherwise stop copying fatal bootstrap errors intohugegraph-server.log.$LOGS, so they followLOGS_OVERRIDEif feat(dist): Make conf/logs/plugins/pid paths overridable and fix log4j2 config resolution #3253 lands. The two PRs touch nearby lines in this script.util.sh:ensure_path_writable(imbajin's commit) andconfigure_riscv64_libatomictake an optional error handler; other callers are unchanged.start-hugegraph.sh: inSTDOUT_MODEthe failure message points at "container logs ('docker logs' or 'kubectl logs')".ci-service-utils.shlists heap dumps instead of printing them, the HStore diagnostics artifact leaves them out, and the riscv64 smoke test's log grep skips them.docker/README.md: a "Server logs and crash files" section (version scope, what reaches stdout, file names, volumes per pod, sizing, cleanup, the opt-out throughJAVA_TOOL_OPTIONSin the container environment).helm/hugegraph/README.mdnotes that the chart mounts no logs volume yet.Behaviour changes to be aware of:
JAVA_OPTIONSis set, and they move fromlogs/java_pid<pid>.hprofintologs/heapdump_*/.heapdump_*directory, empty unless something ran out of memory, and the JVMs a Server starts now get the dump and crash-log defaults too.Picked up JAVA_TOOL_OPTIONS: ...on stderr at startup.Not in this PR:
immediateFlush="false"on thefileappender (mentioned in the issue). WARN and above now also reach the console, which flushes each line.-XX:ErrorFile, which is left for a separate PR.Verifying these changes
test-java-security-properties.sh(run by server-ci) starts a real JVM with each launcher run's capturedJAVA_TOOL_OPTIONS,JDK_JAVA_OPTIONS,_JAVA_OPTIONSand command line, and checks the values it resolves: the defaults, overrides from each source (quoted,@argfile,-j), name collisions with a fixed clock and a dangling symlink, another host, two real JVMs running out of memory and each leaving a dump, telemetry, a failed preflight leaving no directory, themkdir-failure fallback, an unwritablelogs/and the riscv64 libatomic error. It passes on JDK 11 locally (with the minimum lowered) under bash 3.2 and 5, leaves nothing in the distribution, and its new checks fail when the matching launcher behaviour is broken. I did not build or run the Docker images.Does this PR potentially affect the following parts?
hugegraph-server.sh, and thehugegraph/serverimage now logs to stdoutDocumentation Status
Doc - TODO: required documentation is pending; complete it before merging.Doc - Done: documentation is included here or linked below.Doc - No Need: no user-visible documentation is affected.Documentation files in this PR or paired hugegraph-doc PR:
docker/README.md(in this PR)