Godot Game Server

The game server is a Godot 4.7 C# headless project, built and run as a container. See Deployment for hosting and measurement.

Runtime behavior

• Creates an ENetMultiplayerPeer on GAMESERVER_PORT (default 7777)
• Accepts RPC-based authentication
• Maintains PlayerSession state
• simulates movement
• broadcasts snapshots
• polls admin commands
• writes database heartbeats

Environment variables

• MOONDOG_DB or ConnectionStrings__Default
• JWT_SECRET
• JWT_ISSUER (default Moondog)
• JWT_AUDIENCE (default MoondogClient)
• GAMESERVER_PORT (default 7777)
• GAMESERVER_MAX_CLIENTS (default 100)
• GAMESERVER_NAME (default main)

Logging

Never use GD.Print in Moondog.GameServer/. Use GD.PrintErr.

Godot buffers stdout inside a container, so a GD.Print never reaches docker logs. This is not a style preference — it hid two failures in a row. An auth rejection was invisible, so a secret mismatch was chased in the wrong place; and Recovered X from fall at y=-50 was invisible, which was the only evidence that the world had no floor. Neither produced an error. The server simply looked quiet, which is the worst failure mode for a diagnostic channel.

GD.PrintErr writes to stderr, which Docker captures unbuffered. The rule is absolute because there is no case in this service where buffered-and-lost beats unbuffered-and-visible.

scripts/check_logging.py enforces it and runs as the final step of scripts/check_rpc.py, so it also runs inside the Docker image build. The client's probe output is the one exception — scripts/divergence_probe.py parses PROBE_START and DIVERGENCE from stdout, so that stays on print deliberately.

RPC contract

The authoritative list is rpc_contract.json, enforced by scripts/check_rpc.py. Both sides are validated against it, so the docs below are a convenience and the JSON is the truth — run the guard after any signature change:

python3 scripts/check_rpc.py --root .

That also runs scripts/check_logging.py — see Logging.

Client calls (on Main):

• Auth(string token)
• ClientInput(float moveX, float moveZ, float yaw, bool jump, bool sprint, int seq)
• ClientChat(string text)

Server calls (on Main):

• AuthResult(bool ok, string message, string characterId, float x, float y, float z, float rotY)
• ReceiveState(string json)
• ReceiveBroadcast(string message)
• ReceiveKicked(string reason)

Server calls on Main/Chat:

• ReceiveChat(string from, string text)
• ReceiveHistory(string json)

Client calls on Main/Chat:

• RequestHistory(int count)

The trailing int seq on ClientInput is the client's input sequence number. The server echoes the last one it consumed in every state broadcast, which is what lets the client measure how stale a snapshot is instead of assuming. Without it the client has to hardcode a guess at the link's round trip.

Database access

All periodic database work runs on a background thread over one persistent connection, never inline on the main loop.

This is not an optimisation. _Process and _PhysicsProcess share a thread, so a synchronous query between them stops physics ticks and state broadcasts for its duration. The server used to open a fresh connection per query — a TCP connect plus an auth handshake — and run it inline, once per second. Measured on this box, idle, over a five-second pacing window:

	median stall	worst
inline (before)	137 ms	1868 ms
background thread	109 ms	361 ms

The worst case is the one that matters: a two-second hitch, roughly once a minute, on a machine whose CPU was only 24% busy. It was I/O wait, not compute, which is why it was invisible in every CPU-shaped metric.

What moved

job	every	main thread now
PollCommands	1 s	drains applied commands
WriteHeartbeat	5 s	passes a player count it already holds
SaveAll	15 s	passes a snapshot taken under _lock

SaveAll also no longer holds _lock across its writes. It used to hold it for a network round trip per player, on the thread everything else needs.

Rules for anything added later

• Never open a connection or run a query on the main thread.
• Anything that touches _sessions runs on the main thread via the worker's completion callback. Anything that only touches the database runs on the worker. StandablePosition is the trap: it queries terrain through the engine, which is legal only on the main thread, so clamping happens before submitting rather than on the worker.
• Ordering is safe because the worker is a single thread with a FIFO queue and only the main thread submits. A later save cannot land before an earlier one. That guarantee does not survive switching to a connection pool.
• dbq on the pacing line is the queue depth. It should sit at 0; a persistent backlog means writes are landing late.

Not yet moved

Chat history still queries inline. It is rare and does not sit on a hot path.

The login path used to. IsBanned and TryLoadCharacter ran inside the Auth RPC, so every login stopped the simulation for two connection handshakes plus two queries. Both now run on the worker, and the handshake resumes in CompleteLogin on the main thread when the result arrives:

Auth RPC ──validate──▶ worker: ban check + character load ──▶ CompleteLogin (main thread)

CompleteLogin re-checks the session identity, not just its presence. A peer that disconnected mid-query leaves the work in flight, and a recycled peer id would otherwise have one player's login applied to whoever holds that id by the time the result comes back.

Known issue

A rejected token never reaches the client. The failure path sends AuthResult(false, ...) and calls DisconnectPeer immediately, so ENet tears the session down before the reply is flushed. The client sees a bare disconnect, retries three times, and finally shows a connection failure — not the reason. The server logs the real cause, and it is a good one ("token was not signed with this server's secret"), but the player is told nothing.

Worth fixing by deferring the disconnect a frame or two so the reply flushes first. Left alone here because it predates the threading work and changing auth timing belongs in its own commit with its own test.

Determinism

The server's movement simulation can be recorded and replayed to answer one question: given the same inputs, does the server reproduce the same past? That is what gates a replay-based rewrite, and it is a different question from the client-side true_err telemetry, which compares the client's prediction against the server. A rewrite would discard client prediction and simulate on the server, so client prediction fidelity is beside the point.

Both env vars default to empty and change nothing:

# Record a live session to replay/session.csv
GAMESERVER_RECORD_INPUTS=/replay/session.csv

# Re-run it and print a verdict; opens no network port
GAMESERVER_REPLAY=/replay/session.csv

./replay is bind-mounted, because the container is recreated between the recording and replay arms and an in-container path would not survive that — the replay would fail to find its own input, which looks exactly like a pass.

The replay drives the shipping ProcessMovement rather than a copy. A harness that reimplemented the movement would be testing a different program, and a verdict about it would be about the wrong thing.

What it records, and why each field is load-bearing

Per tick: input as the server consumed it, position before and after, velocity before, and the tick delta. All four turned out to be necessary, and each was added because the harness produced a confident wrong answer without it:

field	missing it produces
position	replay compares two different worlds across a session boundary
velocity	momentum carries over from the previous chain; MoveAndSlide steers velocity rather than setting it, so position alone does not determine the tick
delta	gravity and the acceleration ramp are scaled by it; substituting the replay's own delta yields a one-ULP divergence that compounds
input-as-consumed	the client's last-sent input and the server's mailbox differ on any tick that saw no packet
rotation	the collider is a capsule and contact generation depends on its orientation. Added on the reasoning that it must matter; it turned out to be zero throughout every session recorded here, and padding it in did not fix the failing case. Kept because it is genuinely state, and kept honest here because the hypothesis was wrong.

Floats are written with G9 and read back as float, so the log round-trips System.Single exactly. A format that rounded to millimetres would hide the class of difference being looked for.

Replay may also record itself (GAMESERVER_RECORD_INPUTS set alongside GAMESERVER_REPLAY). That makes replay-of-replay a fixed-point test, which separates "the log is incomplete" from "the world differs" — two observations that look identical from the outside and call for completely different fixes.

Replay resyncs only where the recording itself jumps — a reconnect — and reports how many independent segments it ran. Position is deliberately not forced every tick; the body is driven forward by ProcessMovement and its output is what gets compared, so feeding the recorded position back in each tick would replace the test with a tautology that always passes.

Result

Each recorded tick stores input, position, velocity, rotation and the tick delta. All four are load-bearing — see below — and the harness rejects a log missing any of them rather than parsing it partially, because silently shifting the columns produces a confident, entirely wrong verdict.

Measured across six independent 45–50 s sessions covering walking, sprinting, jumping, turning and slope edges:

sessions replayed bit-identically	5 of 6
sessions that diverged	1
first divergence in that session	tick 1484 of 2928
maximum position error	0.000122 m

The simulation is deterministic in the strong sense — same inputs, same outputs, to the bit — for the great majority of sessions. One session in six drifts by a single float ULP partway through and compounds to about a tenth of a millimetre.

What was ruled out

Warm collision state does not matter. The same log replayed three times in one process — first cold, then with Terrain3D's region data already built from the previous pass — produced identical divergence points and identical max errors to the last digit:

run 1/3  first_divergence=1484  max_error=0.000122076m
run 2/3  first_divergence=1484  max_error=0.000122076m
run 3/3  first_divergence=1484  max_error=0.000122076m

So the drift is not a cold-cache artifact, and no amount of pre-warming fixes it. That eliminated the most attractive explanation and left only engine-internal state outside the body's kinematics.

Replay of a replay is exact. Recording the replay's own output and replaying that reproduces it perfectly. So the engine is a pure function of the state the log carries — position, velocity, rotation, input, delta. When a live session fails to replay, the missing state is in the world, not in the body.

Rotation is not the missing field, despite being the obvious candidate. It was added on that reasoning, and the failing log was re-tested by padding in zero rotation columns. It still diverged at tick 1484 with the same error, so the addition was correct hygiene and not the fix. Recorded here because the hypothesis was plausible, testable, and wrong — and because the padded re-test is what actually settled it.

Consequences for a replay design

The simulation is deterministic enough to replay. This was the open question and the answer is a qualified yes: five of six sessions reproduce exactly, and the sixth by a tenth of a millimetre after ~25 seconds.

Periodic checkpoints are still the right design, but for a different reason than first recorded here. An earlier version of this document concluded that bit-exactness ended after ~40 seconds and that checkpoints were therefore mandatory. That figure came from a harness whose log was missing velocity, delta and rotation, so it was measuring the harness rather than the simulation. The corrected picture is that exactness usually holds for the whole session and occasionally does not, which is a weaker and more manageable claim.

Whatever the residual is, it is not schedulable, so a replay design should checkpoint and verify rather than assume, and should not treat sub-millimetre drift as a correctness failure.

Confirming the harness can fail

A test that always passes is worthless, so the harness is checked against a deliberately broken input. Perturbing one moveX value by 0.05 on a single tick of an otherwise clean log:

REPLAY: first divergence at tick 500 (seq 518)...
REPLAY: NON-DETERMINISTIC: 967/1467 ticks differed. Max position error 0.00417m

It localises the fault to the exact tick. Without this check the passing runs above would be unfalsifiable.

Off-terrain spawns

A character whose stored position has no terrain under it is relocated rather than dropped into the void. This was found by the determinism harness, not by inspection: the test character had been saved at z=-4000.0, the world edge, which StandablePosition clamps to exactly that — and GetGroundHeight answers GroundPlaneHeight wherever get_height() is NAN, so the clamp produced a perfectly reasonable standing height over a void. The body fell, RecoverFromFall put it back through the same function, and the character looped between y=1 and y=-50.5 indefinitely, one fall recovery per cycle.

HasGroundAt already existed, was already called, and its result was already logged as on_terrain=False on the spawn line — then nothing acted on it.

RelocateToGround now searches outward in rings for the nearest real ground and falls back to GAMESERVER_DEFAULT_SPAWN (default -900,720, the documented flat control point). A character saved somewhere unusable reappears where everyone starts rather than being permanently stuck. If neither the stored position nor the default has ground, the server says so and the existing terrain-less guard drops the player with an explanation rather than inventing a position.

This also invalidated earlier movement measurements: a probe run against this character was largely measuring a fall into the void, which is the most likely source of the hard snaps and metre-scale outliers seen in earlier telemetry.

Auditing stored positions

A character saved off the terrain grid is invisible until it is a problem, and GAMESERVER_AUDIT_SPAWNS=1 reports it at startup rather than leaving it to be discovered by a movement probe:

AUDIT: Taval    (  -871.8,  1.00, -4000.0)  ground=NO  out_of_bounds=False
                would_spawn=( -900.0, 1.00, 720.0)  relocate=4720.1m
                <-- NO TERRAIN AT STORED POSITION
AUDIT: 5 character(s), 1 off terrain, 0 out of bounds.

It reports the resolved spawn position and the relocation distance rather than only the stored coordinates, because that difference is what login will actually move someone by and how far is the useful part.

Advisory only. It does not relocate anyone: relocation happens at login, and an audit that silently rewrote positions would hide a persistence bug behind a convenient log line.

The audit also reports embed — how far the capsule centre will sit below where a resting body belongs — and flags anything past the tolerance:

AUDIT: FreshOne (0.0, 0.00, 0.0)  ground=yes  would_spawn=(0.0, 1.00, 0.0)  embed=-0.10
AUDIT: FreshOne (0.0, 0.00, 0.0)  ground=yes  would_spawn=(0.0, 0.00, 0.0)  embed=+0.90  <-- SPAWN WOULD BE BURIED

That column exists because the audit originally could not see the most serious spawn bug there was. It computed would_spawn by calling the same function the login path calls, so it faithfully reproduced whatever that path decided instead of checking whether the decision was right — and reported a character spawning 0.9 m inside the terrain as perfectly healthy. "No ground here" and "ground here, but the body is buried in it" have to be two different findings.

Spawn placement composes, and must keep composing

Two functions, two jobs, and the split matters:

• RelocateToGround finds a horizontal position with terrain under it.
• StandablePosition turns a stored Y into a height the capsule can stand at.

Every exit from the first goes through the second.

They briefly stopped composing when RelocateToGround was given a fast path returning the stored position untouched whenever ground already existed. It looked like a harmless shortcut: there is terrain here, so there is nothing to do. But the portal creates characters with no position at all, so a new character's stored Y is 0, and the origin does have ground — so the shortcut returned Y=0 and put the centre of a 1.8 m capsule at the ground surface, leaving 0.9 m of it inside the terrain. Such a character cannot resolve its own floor and cannot move.

It presented as a brand-new-character bug because everyone else's position had been saved by a working spawn, and those Y values are already correct and pass through unchanged. Nothing failed, no log line was wrong, and the character audit blessed it.

check_rpc.py now asserts the composition, as a structural check on the call graph rather than on the arithmetic — numeric assertions would pass or fail on whatever values happen to be in the file today, and what actually broke was a function not being called.

This exists because the determinism harness tripped over an off-terrain character by accident — the wrong way to find it. Verified to actually catch one by moving a character to a known void position and confirming it was flagged and counted, rather than trusting a tool that only ever prints "fine".

State broadcast

State goes out every other physics tick — 30 Hz against a 60 Hz simulation. The simulation itself is unchanged; only how often authority is published.

Halving the broadcast rate halves the packets, and the payload carries every authenticated player, so this is the term that scales quadratically with population. It is also close to free: the client's deadzone is sized from measured staleness rather than an assumed interval, so it widens to suit on its own and nothing needs retuning.

Measured like-for-like, six runs per scenario:

scenario	p50 @ 30 Hz	p50 @ 60 Hz	fires @ 30 Hz	fires @ 60 Hz
straight	0.082	0.098	6	6
turn	0.159	0.166	3	10

No divergence penalty. The difference is within run-to-run noise in both directions, and correction actually fires less often on turns.

Two things had to move with it, and both would otherwise have been silent:

• The client's STATE_INTERVAL_SEC, its fallback for when staleness cannot be measured. It equals BroadcastEveryTicks / PhysicsTicksPerSecond; leaving it at one tick would under-size the allowance by 2× after a reconnect. check_rpc.py asserts the relationship, which is why it stays a literal rather than being computed from the two constants above it.
• The pacing stall threshold, previously a fixed 50 ms. At 30 Hz the nominal gap is 33 ms, so 50 ms is only 1.5× nominal and ordinary jitter would be reported as a stall. It is now three nominal intervals, and the log labels the value (over_100ms) instead of hardcoding the number.

broadcast_every_ticks is now pinned in rpc_contract.json against both sides. A client assuming 60 Hz against a server publishing 30 Hz would mis-size its fallback allowance and nothing else would notice.

ReceiveState carries a JSON payload with one entry per authenticated session: id, name, x, y, z, ry, vx, vy, vz, floor, seq.

floor is whether the authoritative body was on the floor when the snapshot was taken. The client needs it to tell "the server has already walked off this ledge and is falling" apart from "we disagree about where the floor is" — a distinction it cannot make from position alone, and one that matters on steep terrain.

Movement

This host cannot time movement correctly

The development VPS suspends the game-server process from outside for 100–300 ms at irregular intervals. The evidence is conclusive: on an idle server with no players connected, the physics tick reports step_max=0ms, proc_max=0ms and gc_pause=0ms — the process executed nothing — while gap_max reaches 252 ms. There is no code path that idles for a quarter of a second. The same software on the LAN host shows no stalls at all.

Consequences, and they are not the same in both directions:

• Movement behaviour measured here is sound. This host is what caught the client and server loading different terrain — a real bug, invisible everywhere else.
• Movement timing measured here is not. Any divergence figure taken on this box includes stall artefacts, and the stall is 100–300 ms of a stopped clock against errors measured in tenths of a metre.

Every baseline in this section was measured before that was known and should be treated as suspect. They are recorded rather than deleted because the relative comparisons are still meaningful; the absolute numbers are not a reference.

PACING prints a STALL line when a window crosses the threshold, so the log says so without anyone having to read gap_max:

STALL gap=107ms step=0ms -- process idle while wall clock passed; this host is
  suspending it. Movement figures taken now are not a reference.

The step=0ms is what makes that sentence true rather than a guess. Where the main loop is genuinely blocked in our own code, step is large and the line names a phase instead of blaming the platform.

See Moving to a host without the problem for what to do about it, Already ruled out for the hypotheses that have already been disproved, and Starting toolkit for the scripts to start from.

Moving to a host without the problem

The first three steps re-establish a reference. Step 4 is the actual work — the re-measurement is mechanical and takes minutes, but nothing has been decided until the numbers have been read against the suspect table above.

1. Check STALL never appears for a few minutes of idle. That is the whole acceptance test — on the LAN host the line never fires.
2. Re-measure the baseline table above. Every absolute figure in it was taken on the stalling host.
3. Re-run the WAN comparison in Measuring at a realistic link. The localhost 0.291 figure in particular is not a reference.
4. Then do a proper movement pass. Read the fresh numbers against the table above and decide what, if anything, is worth changing — deliberately rather than assuming something is wrong because the previous numbers were. Take a capture from a tester on the new host as the cross-check: analyze_capture.py plus what they saw on screen is the pairing that has actually settled questions here.
5. Drop this notice, and drop the "suspect numbers" caveat with it.
6. Keep the gap/step/gc/proc columns on PACING regardless. They are the difference between "the server stalls" and "the server is not running", and they are the canary that says a new host has the same problem.

Already ruled out, with the measurement that ruled it out

This is the expensive part. Each of these was a confident answer that survived being wrong until something was measured, and re-deriving them costs hours.

hypothesis	what killed it
Link latency causes the stutter	real and present, but the spikes reproduce at 1 ms RTT, where there is none. r(true, st) = +0.007
The client is stalling	dt reaches 343 ms, which looks like a hang, but r(true, dt) = −0.104 and the spike itself arrives at dt = 21 ms
The body is falling	fl is is_on_floor(), not an airborne flag. 80% was grounded, not falling
Terrain3D collision stalls the loop	150 s each way, idle: 25 clean windows with collision off vs 23 with it on. Indistinguishable
.NET GC	gc0/1/2 = 0, gc_pause = 0 ms
cgroup throttling	cpu.max = max, nr_throttled = 0
The database	dbq = 0, proc_max = 0 ms
Scheduler preemption	runqueue wait grew ~40 ms per 10 s
Region filenames duplicate	Terrain3DUtil::string_to_location takes the first 3 and last 3 chars and strips underscores, so terrain3d_00_01 is (0, +1) and terrain3d_00-01 is (0, −1). A hyphen is a minus sign. All 223 files are distinct
Duplicate region files need cleaning	consequence of the row above. There was never anything to clean
Dev box and server have different terrain	true, and fixed. /contract was never mounted, and the sync ran with >/dev/null 2>&1, so 96 modified regions reached the client and not the server
Engine.MaxFps pacing the loop with no headroom	true, and fixed. Clean-run mean 0.291 → 0.079
This process blocks the main loop	true, but not the stall: step_max is 0 ms and proc_max is 0 ms. Nothing in the process is executing while wall clock passes

The one open movement item

RecoverFromFall guards on _terrainReady — a global flag — where the hazard is ground missing at a position. A blank region therefore passes the guard that was written for exactly this case, and the method's own comment describes the result as "the player jitters, velocity resets, and the correction loop fights a world that cannot be played."

HasGroundAt(x, z) already exists and is used for spawn placement; it is not used here. Low urgency now the map is filled, but an off-region edge still gives the silent respawn-into-the-void loop rather than an explanation.

Starting toolkit

None of this needs rebuilding:

• scripts/wan_probe.sh — shapes loopback to a given RTT/jitter/loss, pins the character to a fixed start, verifies the RTT before measuring and refuses to report if the link is not the one it claims, and removes the shaping on EXIT/INT/TERM. --delay 0 for a control.
• scripts/analyze_capture.py --db — reads the newest capture straight out of the collector, decompresses it, splits moving from still, and groups divergence by 25 m map cell. --fail <metres> makes it a gate.
• godot_client/scripts/terrain_grid.gd — walks has_ground across the world and prints a walkability map in metres, resolving the region→world mapping by measurement rather than assumption.
• scripts/probe_at.py — places a character at a known position and probes there, which is the only way to reproduce a specific piece of ground.
• F3 in the client — geotags every capture at no effort to the tester, on the same 25 m grid as the analyzer. The position readout on F1 shows live divergence so the tester can see an excursion as it happens.

Grounded sprint is the regime nobody had measured

The probe scenarios are idle, straight, turn, figure8, sprint, sprint_jump, stop, edge. The baseline divergences recorded against them are:

scenario	speed	on-floor	mean divergence
idle	—	yes	0.000
straight	walk	yes	0.272
figure8	walk	yes	0.271
turn	walk	yes	0.350
sprint_jump	8 m/s	no	0.436
stop	8 m/s	release	0.969

sprint was added after the first tester capture reported "lots of rubber banding" at 1.71 m mean divergence, and nothing above could explain it.

sprint_jump looks like the sustained-sprint scenario and is not one. It jumps every 30 ticks — half a second — against an arc of about 0.7 s, so the body is airborne essentially the whole run: measured at 7% on-floor. Ground friction, ground snap and slope response are all disengaged for the entire measurement, which makes it a poor proxy for a player holding sprint on the ground.

Suspect numbers. All of the above were measured on the host described at the head of this section, before its stalls were known. Treat the absolute values as indicative only.

sprint is sprint on the ground, no jumping, in the --probe-yaw direction.

Where the divergence actually is

scripts/analyze_capture.py groups a capture by 25 m map cell. Running sprint for 20 s from a control spawn originally produced this:

(150, -375)   n=93   mean=1.441   max=4.071   100% on-floor
(150, -350)   n=95   mean=0.717   max=2.058   100% on-floor
(150, -325)   n=94   mean=0.567   max=1.658   100% on-floor
(150, -400)   n=53   mean=0.555   max=1.369   100% on-floor

2.3× worse than the cells either side of it, on flat ground — vertical gap 0.000 and terrain contribution 0.000 throughout, with the body pinned at y=0.90. Purely horizontal, and spike-and-recover rather than accumulation:

     z    true       h
-351.3  0.618   0.049
-356.2  2.418   2.018   ← spike
-364.1  1.236   0.302
-371.4  0.569   0.169
-378.3  0.302   0.035

Locating the stall

PACING now reports the tick gap and the tick step separately, plus everything outside the physics tick:

gap_max=247ms step_max=0ms phase=simulate fps=39
  gc0=0 gc1=0 gc2=0 gc_pause=0ms proc_max=0ms phase=db_drain

Both matter because they mean different things. Step is time spent inside _PhysicsProcess; gap is time between one tick ending and the next beginning. A large gap with a small step means the thread was not running at all. A large step means this process is blocking on something of its own. Those have opposite fixes, and "the tick took 346 ms" does not distinguish them.

That split immediately ruled out most of the field:

candidate	measurement	verdict
our own code	step_max 0–2 ms, proc_max 0 ms	innocent
.NET GC	gc0/1/2 = 0, gc_pause = 0 ms	innocent
hypervisor CPU theft	steal = 0 across all uptime	innocent
cgroup throttling	cpu.max = max, nr_throttled = 0	innocent
database	dbq = 0, proc_max = 0 ms	innocent
client contention	reproduces with no client connected	innocent
scheduler preemption	runqueue wait grew ~40 ms per 10 s	innocent
engine frame pacing	Engine.MaxFps	found

Engine.MaxFps was set to 60, equal to PhysicsTicksPerSecond. Godot paces the main loop with a deliberate sleep, and leaving zero headroom against the tick it has to keep up with meant the loop slept through physics intervals and woke late.

Unbounded (MaxFps = 0, Godot's "no limit") removed the pacing. Measured on the probe at localhost:

	before	after
fps	39–60	109–145
gap_max	up to 328 ms	28–108 ms
clean-run mean	0.291	0.079
clean-run max	1.292	0.537

It is an improvement, not a cure. Gaps of 100 ms+ still occur, and the client-error/server-stall correlation is still 4 for 4.

Terrain3D is not the cause

collision_mode 3 (full game) was set to 0 — collision disabled — and the server watched for 150 seconds each way, idle, no players connected:

	windows	clean	stalls	worst gap
collision off	31	25	10	214 ms
collision on	31	23	12	252 ms

Indistinguishable. Terrain3D's collision work is not what stalls the loop. (The change was reverted; it was a diagnostic, and players fall through the world without it.)

What is left is not in the process

Put the measurements together. An idle server with no players connected shows:

gap_max=252ms  step_max=0ms  proc_max=0ms  gc_pause=0ms  fps=44

step_max and proc_max are zero, so this process executed nothing. GC never ran. No cgroup quota, no throttling, no runqueue wait, no database work, no Terrain3D collision. And still ~200 ms of wall-clock passes between two physics ticks.

There is no code path that idles for 200 ms. Time is passing while the process is not running, which means it is being suspended from outside — the host suspending the vCPU in a way this guest's accounting does not attribute to steal. That is consistent with the tester suspecting the VPS, and it is the first candidate in this entire investigation that survived every measurement.

The practical consequence is that this may be a property of this machine rather than of the software. The LAN host at 192.168.0.149 is the way to find out: if it does not stall, the movement behaviour the testers see is a dev-environment artefact, and the fix is a host, not a line of code.

check_rpc.py had a bug that would have pushed people straight back to the value this fixes: it read MaxFps = 0 as zero frames per second rather than Godot's "unlimited" sentinel, and failed the server for being configured correctly.

The rubber banding is a server stall, not the network

Reproduced at localhost RTT, so latency is not required to produce it. Four identical probe runs — same start, same ground, same code — against the server's own pacing log:

run	client max error	server over_100ms	server max frame
1	1.698	2	125 ms
2	0.316	0	63 ms
3	0.315	0	54 ms
4	4.170	7	346 ms

Every run where the client diverged had server stalls. Every clean run had none. No CPU contention (6% client, ~1% server), and the database queue depth was zero throughout, so it is the main loop blocking rather than something waiting on a resource.

The signature in the capture is exact. Through a stall:

true: 1.871 → 2.138 → 2.405 → 2.671 → 2.938 → 3.072 → 3.338 → 3.605

The error grows by 0.267 m per packet, which at 8 m/s is 33.4 ms — one broadcast interval. The client keeps moving while the server does not, and the gap grows by exactly one packet's worth of travel each time. dt over the same window is 0,1,0,0,0,1: packets arriving back-to-back, a backlog draining after the freeze. The error then collapses as the server catches up — which is the yank the player feels.

This is why the movement looked like a networking problem and was not one. Two plausible wrong answers came first, and both were killed by measurement:

• Link latency. The tester's link is ~158 ms and the threshold scales with it, so lag is real and present — but the spikes reproduce at 1 ms RTT, where there is no latency to explain anything.
• A stalled client. dt reaches 343 ms, which looks like the client hanging. But r(true, dt) = −0.104, and the spike itself arrives at dt = 21 ms. The client is fine; it is waiting.

Consequence for lag compensation. Extrapolating the server forward by the measured one-way delay would make this worse: during a 346 ms freeze the server is not merely stale, it is not advancing, so any velocity-based extrapolation overshoots by the full stall. Compensation has to wait until the stall is gone.

Measuring at a realistic link

Every movement number in this project used to be taken at localhost RTT, which is not where anyone plays. The real figures only arrived as tester captures, and could only be interpreted after the fact — a slow loop that hid the cause for hours. scripts/wan_probe.sh reproduces the tester's measured link locally:

scripts/wan_probe.sh                                  # the measured tester link
scripts/wan_probe.sh --scenario stop --seconds 18
scripts/wan_probe.sh --delay 0                        # control

netem applies its delay per direction, so RTT comes out near 2× — 2.14× in practice, because a ping's mean includes the jitter high side. 74 ms of shaping measures as ~147 ms RTT, against 158 ms measured on the tester's link.

It verifies the RTT before measuring and refuses to report if the link is not the one it claims: a probe run against the wrong link produces numbers that look fine and mean nothing. It also removes the shaping on EXIT, INT and TERM — leaving the dev box at 160 ms would slow every later command and look like something else entirely.

Baselines, sprint, 8 m/s:

	mean	p50	p90	max
localhost, 1 ms RTT	0.291	0.318	0.581	1.292
shaped, 147 ms RTT	1.481	1.431	1.831	3.965
tester capture, 158 ms RTT	1.329	0.683	3.082	3.349

The harness lands within 11% of the tester, which is close enough to A/B a change against. stop for comparison, since it is the case most at risk from anything that compensates for lag:

	moving	still
localhost	0.154	0.118
shaped, 154 ms	1.298	0.150

What the threshold is actually doing

The horizontal threshold grows with measured staleness:

var lag_gap := maxf(speed_xz, server_speed_xz) * staleness_sec / SOFT_CORRECT_RATE

SOFT_CORRECT_RATE is 0.25, so the allowance is four times the distance travelled in half an RTT. On the tester's 158 ms link at 8 m/s that is a measured ht mean of 2.624 m, and correction fired 3 times in 2848 packets.

That is why the game mostly feels fine on a slow link and occasionally does not. The threshold absorbs ordinary lag, so there is no constant rubber banding — but when genuine divergence exceeds it, the correction is a multi-metre yank. The tester's report was "seems alright so far" alongside a capture that was 15% over the 3.0 m hard-snap threshold for forty seconds. Both are true.

The remaining divergence is bounded rather than accumulating:

window     mean_err    p90     >3.0m
0-416         0.157   0.282      0%
416-832       2.600   3.215     75%   ← spike
832-1248      2.929   3.042     29%
1248-1664     1.762   2.774      1%
1664-3328     0.430   0.719      0%

and it is not the machine: fps was a flat 60.0 and staleness 79 ms both inside and outside the spike.

It was the client and the server disagreeing about the map

That hotspot was not a movement bug. The dev stack kept two copies of the terrain, and dev-server.sh synced one of them from a path that was never mounted:

python3 /sync_terrain.py --src-root /contract/godot_client --dest-root "$SRC" >/dev/null 2>&1

/contract was absent, so the sync found nothing, printed nothing, and the container started normally on whatever terrain was already in /src. The client loaded godot_client/map_data — 223 regions — while the server served Moondog.GameServer/map_data — 113. A map change that modified 96 existing region files therefore reached the client and not the server, so the two simulated different surfaces. Terrain3D reports regions=113 ground_samples=5/6 before the fix and regions=223 ground_samples=6/6 after.

The region containing (150, −375) is terrain3d_00-02.res, and it changed between those revisions (86,351 → 86,270 bytes). Same route, same start, same scenario, before and after the mount:

	before	after
mean	0.603	0.277
worst cell	1.441 @ (150, −375)	0.383 @ (150, −425)
cell (150, −375)	1.441	0.236

The hotspot cell fell 6×, the route average halved, and the profile flattened — no cell stands out any more. 0.277 m is now below the documented sprint_jump baseline of 0.436 and in line with straight (0.272) and figure8 (0.271).

The lesson is the shape of the failure, not the terrain. Nothing in any log said the two sides disagreed; the symptom was a discrete divergence spike on flat ground that correlated with nothing — speed r=0.19, yaw r=0.11, staleness r=0.01. Two hypotheses were tried against it and both failed their own test before the real cause was found: link latency (predicted 0.74 m, observed 1.71 m, r=0.007) and a falling character (wrong: fl is is_on_floor(), so 80% was grounded).

dev-server.sh now exits 1 with a FATAL line naming the missing mount rather than starting on terrain it cannot verify. Starting on stale terrain is worse than not starting: it produces a world where two processes disagree about where the ground is, and it presents as a movement bug.

That is also why the correction log carries position. It did not when the first tester capture arrived, and that capture was undiagnosable — 1.71 m of mean divergence correlating with nothing measurable.

Character bodies

The server uses CharacterBody3D with a capsule collider (radius 0.4, height 1.8). FloorMaxAngle is 60°, not Godot's default 45°, and that is a measured decision rather than a preference.

Terrain3D's collision is faceted: the face a capsule actually meets is a triangle between samples, not the sampled gradient. Sampling a 6561-point grid at the documented spawns:

spawn	documented gradient	max face	faces over 45°
flat	—	1.3°	0.00%
slope38	38°	52.5°	9.43%
slope41	41°	58.5°	16.90%

At 45° the engine rejected one face in six at slope41, so is_on_floor() returned false on ground the player was visibly standing on. Gravity then applied to a character that had not moved, the floor flag oscillated, and correction fought it — every measurement of divergence, deadzone and gravity looked normal, because the defect was upstream of all of them.

Reproduce the table with:

docker compose --profile game run --rm -e GAMESERVER_DUMP_SLOPES=1 game-server

Re-run it after any terrain change. It is the only way to answer "is this ground steeper than the floor limit" without guessing, and guessing is how a 45° limit survived next to a 58.5° face.

Each player body is added to the world at spawn. Inputs are transformed by yaw before simulation.

Simulation constants are shared with the client and guarded by the movement section of rpc_contract.json, because a one-sided change does not drop an RPC — it produces rubber-banding, unreachable speeds or jumps of the wrong height, which is much harder to notice:

constant	value
GRAVITY	9.8
WALK_SPEED	5.0
RUN_SPEED	8.0
JUMP_VELOCITY	4.5
ACCELERATION	8.0
INPUT_NORMALIZE_ABOVE	1.0
FLOOR_MAX_ANGLE_DEG	45.0
FLOOR_SNAP_LENGTH	0.1
FLOOR_STOP_ON_SLOPE	true

Terrain

The ground is Terrain3D, with map_data loaded from the repository. The server sets collision_mode = FULL_GAME in Main.tscn, which is mandatory: the default DYNAMIC_GAME builds collision around a Camera3D and a headless server has none, so players would fall through the world.

TerrainProbe.gd is a thin GDScript wrapper exposing ground_height and ground_normal to the C# side. It exists because GDExtension C# bindings are not reliably generated by headless or CI builds — naming the type from C# breaks dotnet build — while GDScript talks to the extension natively.

Both sides load byte-identical map_data. That is a hard requirement for comparing positions across hosts.

A server with no terrain

The failure this section exists to prevent: a server whose map_data is empty starts normally, logs nothing, has no collision geometry, and drops every player through the world. It is not a crash and not an error — login works, the heartbeat is alive, the port is open. The only symptom is that everyone falls, which reads as a movement bug some distance from its cause.

Three layers guard it:

1. scripts/sync_terrain.py refuses to succeed with zero files. It used to check only that the source directory existed, so a checkout with no region files reported success. It now exits non-zero and names the empty mapping.
2. TerrainProbe.is_ready() requires real ground. terrain.data != null is not a usable test — Terrain3D constructs an empty data object when the region files are absent, so that check passes on a server with no floor at all. It now samples six points spread across the world and requires at least one finite height. The startup log reports how many answered, so a partial map reads as ground_samples=5/6 rather than a bare yes.
3. The server says so repeatedly. Once every 10 seconds while the fault persists, and a player who falls while terrain is missing is dropped with an explanation rather than respawned into the void — which would just fall again forever. A healthy server falls off a cliff normally.

To check a running server:

docker compose logs game-server | grep -i "terrain"

ground_samples=0/6, or a repeating warning, means the build had no terrain.

Loaded terrain is not the same as ground under the player

Terrain can be perfectly loaded and a character can still have nothing to stand on. get_height() returns NaN outside the defined region grid, and GetGroundHeight falls back to a flat constant there — so a character in the void reports the same ground height as one on a hill, StandablePosition finds nothing to clamp, and the spawn looks entirely normal.

What happens next is a silent loop: the character falls, RecoverFromFall respawns them at the same fallback height plus clearance, and they fall again, forever. No warning, no error, no disconnect. To a player it is jitter — the character fights the correction loop and never moves.

Every spawn is therefore logged with whether there is real ground beneath it:

Spawn Yarrbeard at (4000.0, 3.0, 4000.0) ground=0.00 on_terrain=False

on_terrain=False means that character cannot stand there. ground=0.00 alongside it is the fallback constant, not a measured height. Check with:

docker compose logs game-server | grep "Spawn "

A Recovered X from fall at y=-50 repeating every few seconds for the same character is the same condition observed from the other side.

A character can be on terrain and still be buried in it

This is the quieter version, and it is what a character created at the map origin gets. Terrain is loaded, the point has real ground under it, the spawn clamp finds nothing to correct — and the character still cannot move, because a capsule resting on a surface has its centre half its height above it, and the stored Y can be below that.

A capsule buried in a static collider cannot resolve its own floor: is_on_floor() returns false, depenetration fights the terrain, and the character jitters in place. Nothing errors and nothing warns.

embed reports the gap directly — how far the capsule centre sits below where a resting body should be. Negative is clear of the ground:

Spawn clamped for Yarrbeard: (0.0, 0.0, 0.0) -> (0.0, 1.0, 0.0)
Spawn Yarrbeard at (0.0, 1.0, 0.0) ground=0.00 on_terrain=True embed=-0.10

Before the lower bound was corrected, that same character logged embed=0.90 and covered 0.23 m in six seconds of straight-line input, against 28.9 m from a valid spawn. Identical client, identical server, identical terrain — only the stored Y differed, by exactly one capsule half-height.

StandablePosition therefore rejects any Y more than EmbeddedTolerance below ground + StandingHeight, and re-derives it. The tolerance covers the sink floor snapping legitimately produces. The upper bound is deliberately left loose: tightening it would risk relocating characters who saved mid-jump.

note

A new character is created with PosX/PosY/PosZ left at their defaults of zero, which is the origin. The spawn clamp now repairs that on first login, but a deliberate default spawn is worth adding separately.

Spawn safety

A stored position can be mid-air: a 15-second autosave that lands during a jump, or a row written before the clamps existed. Feeding one back as a spawn drops the player into a fall before they have input, which reads as a movement bug.

StandablePosition() is the single check, applied on both load and save. The vertical band is deliberately generous — standing puts the capsule centre ~0.9 m up and a jump peaks just over 1 m above that, so the upper bound sits at ground + 4 m. An autosave that catches a jump passes through untouched, so a real position is never silently relocated on login.

Auth validation

JWTs are validated with HmacSha256, issuer, audience, and lifetime. The character_id claim must match a character owned by the user.

Diagnosing a rejected token

The portal signs tokens and the game server verifies them, so both must hold the same JWT_SECRET. When they disagree, .NET reports it as:

IDX10517: Signature validation failed. The token's kid is missing. Keys tried: '...'

This message is misleading and worth reading past. There is no JWKS endpoint and no key rotation here — both sides use one shared symmetric secret and set no key id, so "the token's kid is missing" does not mean a key was looked up and missed. The key was found and tried; the HMAC simply did not match. The underlying exception is SecurityTokenSignatureKeyNotFoundException, whose name is equally unhelpful. Treat it as a secret mismatch.

To confirm which deployment the token came from, compare fingerprints. The game server prints one at startup:

JWT_SECRET fingerprint 5e436cee (must match the portal's Jwt__Secret)

If a token is rejected, the log repeats the fingerprint alongside the reason. Two servers printing different fingerprints cannot accept each other's tokens — check docker compose logs game-server for the peer id from the error to confirm which server actually rejected it.

The most common causes, in order:

1. Wrong portal. A token from a different deployment — usually logging in through the public portal while pointing the client at a local game server. Set MOONDOG_HOST and MOONDOG_PORTAL together; setting only the first is enough to produce this. The client prints both on startup, which settles it without a log dive.

2. Stale container. JWT_SECRET changed in .env after the first run. Compose reads .env on recreate, not on restart, so docker compose restart leaves the old value in place. Use docker compose up -d --force-recreate portal game-server.

3. Portal advertising an address the client cannot reach. This one does not fail authentication — it routes the client somewhere wrong first, and the signature error is the second problem. The token response carries host and port from the portal's own GameServer:Host, and the client obeys. The shipped default is 127.0.0.1, which is correct only when client and server share a machine. On a separate host it means "the client machine's own loopback", so every client dials itself. Ask the portal what it actually returns:

   curl -s -X POST http://<portal>:8082/api/gameauth/token \
     -H 'Content-Type: application/json' \
     -d '{"email":"you@example.com","password":"...","characterId":null}' \
     | grep -o '"host":"[^"]*"'

   Then set GAMESERVER_HOST in .env to an address clients can reach and recreate the portal — only the portal reads it.

Note that the shipped default in docker-compose.yml and the fallback inside ServerMain.cs are the same string. They were briefly different, which made a game server run without JWT_SECRET set — a bare dotnet run, or a compose override dropping the variable — silently disagree with the portal.

The client surfaces the server's reason on the login screen. A rejected token is not retried: reconnecting re-presents the same token to the same server, so it can only produce the same rejection.