FAQ / Troubleshooting

Networking

Nodes don't discover each other

1. Verify the seed node is running and reachable: ping <seed-host>
2. Check firewall rules: inter-node port (default 10000) must be open
3. Confirm seed_.enabled_ is true on non-seed nodes
4. Verify seed_.node_port_ matches the seed's ports_.node_port_
5. Check logs for [svc] Connected to peer messages

Mesh is unstable / nodes keep dropping

• Increase gossip_.suspect_timeout_s_ (default: 10s) in high-latency networks
• Ensure all nodes have synchronized clocks (NTP)
• Reduce gossip_.cycle_interval_ms_ for large clusters (50+ nodes)

---

JWT / Authentication

JWT token is rejected

1. Verify the signature key matches between generator and configuration
2. Check the token hasn't expired (exp claim)
3. Ensure requires_auth matches on the route — routes with requires_auth = true need a valid JWT in the Authorization header

Token generation fails

• Run keys_generator to create new signing and encryption keys
• Confirm the base64 keys are correctly copied into config.jwt_.keys_
• Tokens generated with --grants must use valid JSON

---

Cache

Cache values not syncing between nodes

1. Verify mesh networking is enabled (app.run_mesh())
2. Use state->cache_set() instead of cache.set() — the state-level method propagates to other nodes
3. Check that nodes are connected (see networking FAQ above)
4. Anti-entropy sync runs periodically — changes are eventually consistent, not immediate

Cache key not found

• Check TTL: keys with ttl_ms > 0 expire automatically
• Verify the key was set on the same node (read-your-writes is guaranteed only on the writing node)
• Use cache.get_metadata(key) to check expiration and creation times

---

Database

Connection pool exhausted

• Increase max_size in create_pool() (default: 32)
• Check for unclosed queries — each pool->run() callback must complete before the connection returns to the pool
• Use pool->resize(new_max) to dynamically increase capacity

Query timeout

• Verify the MySQL server is reachable on the configured host:port
• Increase read_timeout_seconds_ in config.node_ (default: 10s)
• For long-running queries, break them into smaller chunks

---

General

App fails to start ("Address already in use")

The port is already occupied by another process. Either:

• Change the port in configuration
• Kill the process using the port: lsof -ti :8080 | xargs kill

App fails to start (other errors)

Wrap app.run() in try-catch:

try {
    app.run();
} catch (const std::exception& e) {
    fmt::println(stderr, "Startup failed: {}", e.what());
    return 1;
}

Common causes:

• Invalid configuration file path
• TLS certificate files not found
• WAL directory not writable
• Boost/OpenSSL version mismatch

Logs show nothing

1. Check that ENABLE_SILENCE is not defined at compile time
2. Verify config.logging_.enabled_ is true
3. Ensure the log scope includes the messages you want to see (config.logging_.scope_bitset_)
4. By default, LOG_INFO_SVC(config, ...) requires the "service" scope to be enabled

---

Next: CLI Tools.