Watching and Tearing Down Function Nodes

A function node is a machine you do not log into. There is no shell, no tail -f, no dashboard on the box itself. The gateway is the only window you have, and it gives you three of them: what the node is doing right now, what is loaded on it, and what has been done to it in the past.

That turns out to be enough, provided you know which window answers which question. Most confusion about a misbehaving node comes from checking the wrong one.

Liveness

function_node_health calls the node’s one unauthenticated endpoint and returns two fields:

{"status": "ok", "uptime_ms": 7909948651}

status answers whether the machine is up and serving. uptime_ms is the BEAM’s wall clock since it started, and it is more informative than it looks: it resets on every machine restart. A node you deployed to last week reporting an uptime of ninety seconds has restarted, which means anything you thought was loaded in memory came back only if the gateway replayed it.

If health does not answer at all, the machine is either stopped or still waking. Nodes scale to zero when idle and start again on the next request, so the first call after a quiet period can arrive before the machine is serving.

Inventory: memory against durable

Two tools look superficially similar and mean different things. The gap between them is where most surprises live.

function_node_modules lists what is compiled into the running BEAM right now, with exports and a compile timestamp:

{"module": "Elixir.Mathy", "name": "Mathy", "version": 1,
 "exports": ["add/2", "divide/2", "multiply/2", "subtract/2"],
 "compiled_at": 1777018853264}

function_node_sources lists what the gateway has persisted for that instance, which is the bundle the node replays into memory when it boots. function_node_source returns one module’s full text.

Memory is what you can call. Sources are what survives a restart. Every successful compile writes both, so under normal use they agree. When they disagree, the durable side is the one that decides the node’s future: a module present in modules but absent from sources is running on borrowed time and will be gone after the next idle stop.

Reading the two side by side is the fastest way to answer “will this still work tomorrow?”

function_node_list covers the instances themselves rather than their contents: name, status, URL, and volume fields for every node you own. The status values are provisioning, ready, failed, destroying and destroyed, and a failed row carries the error that stopped it. Destroyed instances drop out of the listing, because the registry soft-deletes rather than removing the row.

Persisted state

Values your code wrote through the key/value store are inspectable from outside the node. function_node_state_list shows what keys exist for an instance, function_node_state_get returns one value, and function_node_state_delete removes it.

One caveat on reading: the gateway stores these as opaque bytes. It never deserialises them, which is deliberate, so state_get hands back base64 along with the byte size and last-updated timestamp rather than a readable value. You get the shape and the size for free; decoding the contents requires the node.

The audit log

Four operations write an audit row: provision, compile, call, and destroy. Health checks, listings and state reads do not, which keeps the log a record of things that changed something rather than a request log.

Each row carries the instance name and owner, the caller (recorded separately from the owner, so a future grant model has somewhere to put the distinction), the action, the module and function where relevant, a truncated preview of the arguments, a result status of ok, error, not_allowed or timeout, a short result summary, and the wall-clock duration.

Two properties are worth knowing. Rows are denormalised with the instance name and owner written in full, so they remain readable after the instance itself is gone – the history outlives the thing it describes. And an audit write that fails logs loudly but never aborts the operation that triggered it, on the reasoning that a lost row is a smaller problem than a user action that fails because logging did.

The log deliberately does not record internal retries, Fly API probes or HTTP redirects. One user-visible operation, one row.

Read it with function_node_audit, or on the instance’s audit page.

Reading a failure

Because every call writes its result, a failing node usually explains itself. Three signatures come up often enough to recognise on sight.

instance not found, surfacing to the caller as :not_found. Instances are resolved by the pair of instance name and owner, using the username of whoever is calling. This is not a permission error and it will not appear as one: the caller can hold every function-node permission there is and still get this, because the node simply is not theirs. Nodes are not shared between accounts. If this appears after a workflow that used to pass, check which user the run was created by before you check anything else.

http 502 with an empty body. The machine was asleep and the request arrived before it was serving. This clears itself. A retry a few minutes later against an unchanged node succeeding is the confirmation.

http 400 carrying {"module_not_loaded": true}. The node is up and answering, but the module named in the call is not in its memory. Either it was never compiled to this instance, or the machine restarted and the replay did not put it back. This is the case where reading modules against sources pays for itself: if the source is persisted, recompiling restores it and it will survive the next restart; if it is not, you are looking at the reason the module disappeared.

Duration is a useful secondary signal on all three. A first attempt that takes several seconds followed by fast failures is a machine waking up and then answering; three uniformly fast failures mean it was awake the whole time and is refusing on the merits.

Recovering a failed provision

A provision that times out is not always a provision that failed. The gateway waits for the node’s own health endpoint, and DNS for a brand-new Fly app occasionally propagates slower than that patience allows, leaving a failed row in front of a machine that is genuinely running.

The retry path checks Fly directly rather than going through DNS, and flips the row to ready if the machine is started and its health check is passing. It is the right first move on any provision that failed near the end of its timeline rather than at the start. A provision that failed early left no machine to find, and the recorded error will say so.

Tearing down

function_node_destroy runs in a fixed order, and the order is the point:

1. Mark the registry row destroying.
2. Revoke the sub-token.
3. Destroy the Fly app, which cascades to any attached volumes.
4. Delete the persisted sources and key/value state for the instance.
5. Mark the row destroyed with a timestamp.

The token dies before the machine does. Between those two steps there is a window where user code could still be executing, and in that window it holds a credential that no longer authorises anything – a node being torn down cannot get one last call through the gateway. Doing this in the other order would leave exactly that gap open.

What survives teardown is the audit log. The instance row is soft-deleted rather than removed, and audit rows keep the instance name and owner inline, so the history of what that node did remains readable after the node and its code and its state are all gone.

In short

Health tells you if it is up and whether it restarted. Modules tells you what you can call now; sources tells you what will still be there tomorrow. The audit log tells you what happened and, through its result summaries, usually why. Destroy revokes before it deletes, and leaves the record behind.