Skip to main content
In high-load production systems, redundancy is what keeps the service up: when one node stops serving, another is already there to take over. A pool brings that redundancy to your RPC. A pool puts several nodes behind one domain and one TLS certificate. Callers reach the domain; each request is answered by a node its executor last saw running, probing OK, and reporting itself synced. A node that stops meeting that bar is dropped from rotation, and so is one you take out for maintenance — the domain your clients call stays the same. Each executor in a pool runs a gateway for the pool’s traffic. A request lands on one executor — by DNS round-robin, or the client’s own choice — and that executor answers from its own node whenever that node qualifies; if it doesn’t, the request is forwarded to another executor in the pool. So nodes buy redundancy and executors buy capacity: a second node on the same host stands by rather than sharing the load.
Failover covers a member that refuses the request — a closed port, an unreachable host, a certificate the caller can’t verify. A member that accepts a request and then goes quiet ends that one call with a 502 and is not retried elsewhere: the request may already have executed, and re-sending it to another node would run it twice. A pool masks a dead node, not a wedged one.
Open Pools in the sidebar. A pool is fixed to one chain/network (set by its first node) and one domain (set when you create it) — to serve a different pair or domain, create another pool.
A pool needs at least one member reachable at a public IPv4 address, and every pool executor must accept inbound TCP 443 — that’s where the pool’s endpoint terminates TLS. Each member serves at its node’s verified public IP, which Novacula detects and verifies for you; where it can’t, set it by hand with Set IP (see Public IP verification). It has to be a public IPv4 literal — a DNS name doesn’t work. A node with no verified public IP is withheld — the pool keeps serving through its other nodes.

​
Create a pool

Create pool opens a four-step wizard:
1

Domain

Choose the pool domain — the exact hostname clients will call (for example rpc.example.org), hostname only, no scheme, path, or port. It can’t be changed after the pool is created. Creating the pool immediately generates a private key and CSR for this exact domain; the private key stays in the Hub. To serve a different domain, create another pool.
2

Nodes

Select 1–100 nodes. The first node fixes the pool’s chain and network — every later node must match. Only compatible, unpooled nodes are offered.
3

Review

Confirm the domain, the pool’s chain/network, and the initial members before anything is saved.
4

Setup

Publish the DNS records shown, then send the CSR to your certificate authority. Uploading the signed certificate here is optional — you can install it later from the Certificate tab. Until it’s installed, each executor serves only its own node: a call to another executor has no certificate to verify, so there is no failover between hosts yet.
Creating the pool locks in its domain and generates its private key before any node is added. If adding the nodes fails, the pool itself is kept — correct the selection and retry just that step. Once created, the pool has six tabs: Overview, Members, Access, Certificate, DNS, and Activity. Activity needs audit access, so Members of the organization see the first five.

​
Overview

The Overview tab is the pool at a glance — its client endpoint (the domain callers use), its chain/network, and a setup status with a readiness checklist over Domain, TLS certificate, DNS records, and Members. The status describes the setup, not live traffic: The Overview tab also shows Request distribution — for each member, a graph of the pool RPC requests it handled over the last 5 minutes, as a 1-minute average in req/s sampled every 5 seconds, with the latest rate beside the member’s name — so you can see whether traffic is spread across the pool or leaning on one member. A member with no sample reads No data; when nothing was served in the window, it says so.

​
Members

The Members tab lists the pool’s nodes — each row shows the node, its executor (name and public IP), the member’s status, and whether it’s in rotation:
  • In rotation — nothing in the pool’s setup keeps this node out. Whether it serves a given request is decided by its executor on every check-in, from the node’s live state against the rest of the pool, so a node that is stopped or still syncing still reads in rotation here.
  • Withheld — the node has no verified public IPv4 address, so callers have no way to reach it; the pool serves through its other nodes. Fix it with Set IP — see Public IP verification.
  • Paused — the pool skips this node while its traffic is paused, and the node keeps running. Pause or resume a member from its row on this tab — see Pause member traffic.
The Status column is a separate reading — the executor’s latest judgement of the member itself, with when it was last observed:
  • Healthy — synced and close enough to the frontier to serve.
  • Lagging — running, but its block height has fallen behind the frontier.
  • Unhealthy — not serving: not synced, unreachable, failing its checks, or its executor has stopped checking in.
  • Paused — deliberately out of rotation; the node itself keeps running.
  • Unknown — not observed yet, or the reading is older than the executor’s check-in window.
Status is what the executor sees; in rotation is whether the pool’s setup lets the node serve — so a healthy node can still be Withheld, and a node reading in rotation can be Lagging at the moment. A node qualifies on three things: it reports itself synced, it isn’t paused, and its block height is close enough to the pool’s own frontier. The frontier is the reference tip the pool measures against — the second-highest height among its synced members, or the highest where fewer than three of them report, so one member claiming a bogus height can’t drag it up. The allowance is roughly 20 seconds of chain time plus one check-in interval, with a floor of two blocks: about half a minute on a fast chain, and two blocks — some twenty minutes — on Bitcoin. A member further behind than that drops out of rotation until it catches up. Each executor measures its own members from what it observed at its last check-in, and asks its peers for theirs, so every host in the pool judges the same frontier. A member nobody can measure is kept in rotation rather than dropped — it serves, but it doesn’t vote on the frontier. Requests go to the local node first when it qualifies, then to whichever member sits closest to the frontier. If no member qualifies, that executor has nothing to serve the pool with: calls it receives on the domain come back 421, and its /nvcl/health turns 503 so a load balancer in front can route around it. Add nodes by dragging one from Compatible nodes onto the list, or with Add nodes (added all-at-once, up to 100). The two lists differ in what they show you: Compatible nodes lists only nodes on the pool’s chain/network that aren’t already pooled, while Add nodes lists every node in the organization and disables the ones it can’t take, each with its reason — Already in <domain>, a chain/network mismatch, or its own RPC endpoint still configured. Selecting a node in Compatible nodes opens its node page, whatever state it’s in — use the browser back to return here. Create node deploys a brand-new node straight into the pool. It reuses the pool’s deployment template — the chain, network, node type, clients, and versions of the existing members — so you enter only a Node ID and pick a compatible executor; the node is created and added in one step. The template has to be unambiguous, so Create node is offered only when the pool already has members and they agree: a pool whose members differ in node type, clients, versions, or inherited settings has no single template to copy, and the dialog sends you to the full Deploy Node flow instead. It also needs an active executor that supports that exact chain, network, node type, clients, and versions — compatible executors that are disconnected stay listed but can’t be picked until they’re back. A pool member needs no public port of its own — the pool’s gateway reaches it over loopback on an Agent, and through the in-cluster Service on an Operator. Set a member’s RPC exposure to Local, and the pool domain becomes the only way in. Set that exposure before the node joins. A node that still publishes its own RPC endpoint is listed among the candidates but blocked — in Add nodes, in the drag-and-drop list, and in the create wizard’s Nodes step alike — with Disable this node’s RPC endpoint settings before adding it to a pool. And once a node is pooled its RPC settings are managed by the pool: the node’s own RPC tab turns read-only and links back here. To change a member’s exposure, use Edit RPC on its row in this tab — it opens the same exposure dialog as an unpooled node, and the pool is now the only place a pooled node’s RPC is set. A member can still read also public in this list — it joined before this guard, or its endpoint was set through the API rather than the console. Joining a pool never closes a node’s port for you, and that flag is how you find the ones that are still open. On Kubernetes that’s more than hygiene. A member in Direct mode is published as a NodePort or LoadBalancer instead of through the Service the pool’s gateway dials, so another pod in the same cluster can’t fail over to it — the same goes for Public domain with RPC-key auth, since the gateway carries no key. Local is the mode that keeps failover working inside a cluster.

​
Public IP verification

A pool member serves callers at its node’s public IPv4 address, and Novacula verifies that the address actually reaches the node before the member enters rotation. It detects and verifies the address on its own where it can; a member whose address isn’t verified reads Public IP required and stays Withheld. To set it by hand, use Set IP on the member’s row (it’s also offered in Add nodes, on the DNS tab, and when you create a node straight into the pool). Enter the node’s public IPv4 literal and choose Check — the Hub dials the address itself and confirms it reaches this node, moving from Checking… to Verified (or an error such as This IP points to another executor or This IP does not serve this node). Save stores it on the node once verified. You no longer set this by hand in a config file — the address belongs to the node and is verified from the console, where the old advertised_host executor setting needed a file edit and a restart. That setting still exists on an Agent, and automatic detection reads it first, ahead of the executor’s other known addresses and the address the Hub saw it connect from; but nothing is adopted until the Hub has dialled it and confirmed it reaches this node. The address must be a public IPv4 literal; a DNS name or a private address is refused. Setting a member’s IP needs rights to manage nodes.

​
Pause member traffic

Pause traffic takes a node out of rotation by hand — for maintenance, or to steer traffic off a host — without stopping it. Use the Pause traffic action (the ⏸ icon) on a member’s row (it reads Resume once the node is paused); the row dims and shows a Paused badge. A paused node keeps running and stays a pool member, it just stops receiving pool traffic. Resume makes it eligible again — live health checks still decide whether it actually re-enters rotation. Pause and resume are deliberately symmetric, and neither takes effect the instant you click: both reach a member host on its next check-in, every 10 seconds by default.
  • On pause, the member leaves rotation on that check-in with none of the caution a failing member gets — it isn’t held in by the drain delay, and it doesn’t spend the cap that limits how many members may leave rotation in one tick. Requests already in flight are not cut off: the pool chooses a member per request, so calls already dispatched finish where they were sent, and only the ones after that go elsewhere.
  • On resume, the member is let straight back in on the next check-in, without the re-entry probation a recovering member serves — an explicit instruction shouldn’t have to earn its way back. It still has to qualify at that moment on sync state and lag, so a node resumed while it is still catching up stays out until it has.
You can’t pause the pool’s last active member: the console refuses it (Resume a member or add another node, then try again.), so a pool is never left with nothing to serve. Pausing takes the same rights as managing the pool — Owners and Admins.

​
Access

The pool endpoint is not open: you decide who may call it and how fast. The Access tab has three panels. Access policy sets who is allowed at all — key holders and guests, key holders only, guests only, or closed, which refuses every call with 401. Set up at least one key or the guest allowance before you point clients at the domain. You can also set a pool ceiling — a cap on the sustained rate you may grant in total, so a single key or a generous guest allowance can’t sign the pool up for more than you intend. Like every limit here it is a per-host figure: the ceiling bounds the sum of the live keys’ sustained rates plus the guest total, measured on one member host. Burst allowances are shown but never counted against it. Client keys are the credentials you hand to callers:
  • Create a key with a label and its own rate limit — a sustained rate (requests per second) plus a short burst, which must be at least the sustained rate. The limit is enforced per executor host, not across the pool as a whole.
  • The secret is shown once, at creation, in a dialog — copy it then, because it’s stored only as a hash and can’t be shown again. A key secret looks like nvcl_pk_ followed by a random string.
  • A caller presents the key one of three ways: an Authorization: Bearer <key> header, the reserved URL path /nvcl/k/<key> (stripped before the request reaches a member), or a ?key=<key> query parameter.
  • Revoke a key to cut it off — it stops working on the member hosts in about twenty seconds. Revoking is never blocked, even for a pool’s last key; but if that leaves the pool with no key and guests off, the console warns you first, because callers will then be refused.
Guest access lets keyless callers in. Turn it on and set a total rate (sustained plus burst) for all guests together, and optionally a per-client-IP limit so no single address can use the whole guest budget. Whether that per-IP limit binds is the host’s call, not the pool’s — only the host knows whether a caller’s address reaches it trustworthily. An Agent enforces it by default: it owns its own edge, so it can stamp an address the caller can’t forge. An Operator does not, because a Kubernetes ingress sits in front of the gateway and Novacula can’t know what that ingress does with the client address. When you save a limit the pool can’t bind, it names the hosts it won’t bind on — and the guest total still holds there, so what you lose is fairness between guests, not the limit itself. Saving any Access change — a key created or revoked, guest access set or cleared, the ceiling changed — is confirmed with a dialog you dismiss with Got it: Changes saved. Member hosts may take 1–2 minutes to apply this change. It waits for your acknowledgement rather than flashing a notification, because the Hub doesn’t report which member hosts have picked the change up yet — so don’t assume it’s enforced everywhere the instant you save.
Per-IP guest limiting on an Operator is opt-in, has no chart value, and is a claim you are making about your own cluster. Add it to the Operator chart’s extraConfig:
Turning it on asserts that your ingress overwrites X-Nvcl-Client-Ip on every request to a pool domain, and that nothing else in the cluster can reach the gateway’s port. Neither is checked, and a forged header can’t be detected — so a client that can set that header picks its own per-IP bucket.
Access control needs support on both sides, and the two behave differently:
  • The connected Hub is too old — the tab reads that pool access isn’t supported, and the endpoint behaves as it did before: reachable without a key.
  • A member’s executor is too old — the pool refuses to take keys or a guest allowance at all, naming that executor. And a pool that already carries them is withheld from such an executor: that host stops serving the pool rather than serve it unprotected. Upgrade the executor before gating the pool.

​
Throttling refused callers

A caller that keeps getting refused — a missing or wrong key, or one over its limit — is itself rate-limited, so a client hammering the domain with rejected requests can’t tie a host up. Each host counts refusals per client address, and once an address runs over its allowance the host stops answering it until the allowance refills. This rides on the same address-trust switch as the per-IP guest cap: an Agent applies it by default; an Operator only where you’ve asserted the ingress stamps a trustworthy client address (see the warning above). The defaults fit most pools — about 1 refusal per second with a burst of 20, across up to 10,000 tracked addresses per host. To change them, set [gateway.auth_throttle] in the executor config (/etc/nvcl/agent.toml on an Agent, or the Operator chart’s extraConfig):

​
Certificate

The pool’s private key and CSR are generated for its domain when you create it; the private key stays in the Hub. On the Certificate tab (also the create wizard’s Setup step):
  1. Get the CSR signed — Copy CSR or Download CSR and send it to your certificate authority.
  2. Install the signed chain — paste the signed chain (starting with -----BEGIN CERTIFICATE-----) or Choose File, then Save certificate. Paste the full chain — the leaf followed by its intermediates (fullchain.pem, not cert.pem): a certificate submitted without its issuer is refused, and the message names the missing issuer.
The tab shows the certificate state — Not issued yet, Installed, Expires in N days once it’s within 30 days, or Expired — with the exact expiry date beside it. (The Overview checklist words the healthy case as Valid until <date>.) Renewing is the same step: install a fresh chain before the current one expires.

​
Automatic certificates (ACME HTTP-01)

Instead of the manual CSR round-trip above, the Hub can obtain the certificate itself over ACME HTTP-01 — the executors answer the challenge on the pool domain. The Certificate tab’s Obtain a certificate switch picks the method, Automatic issuance or Upload certificate, and the tab names the mode currently configured for the pool. Before you start, point the pool domain at its member hosts and make them reachable on port 80 — that’s where the CA validates the challenge. Accepting the CA’s agreement. The Automatic issuance view names the certificate authority and links its subscriber agreement, both read from the CA’s own metadata. If the CA publishes an agreement this Hub hasn’t accepted yet, tick I accept this certificate authority’s subscriber agreement before Start issuance; the acceptance is recorded as the agreement’s exact URL, so a changed agreement asks again while one already on record stands. While an order runs, the view shows its status — Scheduled, Waiting for member hosts, Being validated by the certificate authority — and, if it fails, the reason and the earliest time of the next check. Cancel issuance ends the order; Refresh status re-reads it. The certificate reads Installed only once the CA has issued it and the chain is in place. The Upload certificate view keeps the manual steps above; uploading a chain replaces the certificate and ends any running order. It also carries the older HTTP-01 challenge panel for an external ACME client — the challenge token, a countdown to its expiry, and one row per executor (Serving, Waiting, or Update stalled) — whose Cancel challenge only clears the published token. Only owners and admins can start or cancel issuance and read the CA metadata; members can watch the status. On a Hub without automatic issuance the tab says so and offers the upload only.

​
DNS

The DNS tab lists the records to create at your DNS provider — an A record per executor public IPv4 that answers for the pool, so the domain round-robins across your nodes’ hosts. Each row shows the type, name, value (the executor’s public IPv4), and which executor and nodes it covers. While the pool page is open, the console checks these records for you: it resolves the pool domain through Google Public DNS and compares the public A and AAAA answers with the addresses the pool expects. The lookup runs in your browser, which means the pool’s domain name is sent to Google Public DNS — a third party — to be resolved. Nothing else about the pool leaves Novacula, and the check only reads: it creates and changes nothing. Novacula still can’t see your DNS zone, so treat this panel as a convenience, and for a domain you would rather not resolve through a public resolver, verify the records at your provider instead. A mismatch doesn’t claim the pool isn’t serving: while the domain still resolves to some of the expected addresses the pool reads At risk, and only when it resolves to none of them does it read Needs attention. Either way, the addresses it flags are the ones missing from, or extra to, what your current members expect — not proof that no member is being reached. Remove the records it lists under Remove and add the ones under Missing at your DNS provider, then choose Check again.

​
Activity

The Activity tab logs what’s happened to the pool — membership changes, certificate installs, and the like. A brand-new pool reads Nothing has happened to this pool yet.

​
The pool endpoint

Clients call https://<pool-domain>. Who may call it, and how fast, is set on the Access tab: a caller without a valid key (and with guests turned off) is refused, and one that exceeds its key’s or the guest rate limit gets 429 with a Retry-After telling it how long to wait. On top of that the endpoint refuses the chain’s node-control methods (Bitcoin’s stop and setban, and their equivalents elsewhere), and applies the flat concurrency cap in the table below. The reserved path /nvcl/health is answered by the executor you reached itself, keyless and never forwarded to a member. It answers 200 (ok) while that executor has a member of its own qualified to serve the pool, and 503 (no eligible member) when it doesn’t — a peer it could still forward to does not count. A third answer, 200 (degraded), means the host is still serving but from routing information that’s gone stale — for example while a member is draining; treat it as serving, with reduced confidence. GET and HEAD both work, and the answer is never cached. That degraded state also rides on ordinary RPC responses: while a host is serving on stale routing, every response carries an X-Nvcl-Pool-Degraded: stale header (exposed to browsers via CORS), so a client can notice it without polling the health path. Point an uptime monitor or load-balancer health check at it, and read 503 as “stop sending this host traffic”, not “the pool is down”: the other hosts may well still be serving, which is exactly the case where you want traffic moved off this one. What it serves is narrower than a node’s own endpoint: Send heavy calls — a wide eth_getLogs, a debug_trace* — to a node directly: through a pool they can come back as a 502 while the same call succeeds against the node itself.

​
Test the endpoint

Test through proxy sends a sample request to the pool domain from your browser and shows you the response, so you can confirm the endpoint answers before pointing clients at it. When guest access is enabled the check runs without a key; the Pool key field appears only when the pool requires one. On a multi-client pool such as Ethereum the sample calls the pool’s primary RPC role — the execution JSON-RPC — rather than a consensus endpoint.

​
Permissions

  • View pools — any organization member.
  • Create and delete pools, add or remove members, install certificates, and manage Access (client keys and the guest allowance) — Owners and Admins.
  • The Activity tab rides on audit access, so it too is Owners and Admins; a Member doesn’t see the tab.
A pool’s domain, chain, and network can’t be changed after creation, by any role. See Roles and permissions.