Operations
import { Aside } from ‘@astrojs/starlight/components’;
Health model
Section titled “Health model”GET /ready returns 200 only when required ClickHouse objects are reachable and reader state is ready. A dependency outage should remove the process from traffic without restarting it.
Use a TCP liveness check on the PulsusDB port and an HTTP readiness check on /ready.
Self-observability
Section titled “Self-observability”Scrape GET /metrics for ingest volume, queue pressure, protocol errors, query latency, ClickHouse behavior, and process health. Pair it with:
/buildinfofor version correlation;/configfor the effective redacted configuration;- ClickHouse query logs for slow or expensive generated SQL;
X-Pulsus-Explain: 1for a specific query plan.
Capacity controls
Section titled “Capacity controls”| Control | Protects |
|---|---|
PULSUS_BATCH_BYTES |
Memory used by one ingest push |
PULSUS_INGEST_QUEUE_BYTES |
Total buffered ingest memory |
PULSUS_CH_POOL_SIZE |
Concurrent ClickHouse work per process |
PULSUS_QUERY_TIMEOUT |
Wall time consumed by one query |
| Query-specific point and byte budgets | Rust-side evaluation and result cardinality |
Scale writers when ingest saturation or queue pressure is sustained. Scale readers for query concurrency and dashboard fan-out. ClickHouse capacity, table ordering, and merge health remain the dominant storage considerations.
Connection spreading
Section titled “Connection spreading”CLICKHOUSE_SERVERS configures several ClickHouse HTTP endpoints. PulsusDB uses zone-preferring round robin and temporarily demotes endpoints after retryable transport failures. Background probes return recovered endpoints to rotation without consuming capacity needed by live requests.
Set PULSUS_AVAILABILITY_ZONE explicitly or enable supported cloud metadata detection with PULSUS_AZ_DETECT.
Security baseline
Section titled “Security baseline”- terminate TLS at PulsusDB or at a trusted ingress;
- set both API Basic auth values when the listener is not otherwise protected;
- change the admin UI’s default
adminpassword; - store ClickHouse and PulsusDB credentials in a secret manager;
- keep compatibility endpoints disabled unless required;
- restrict
/config,/metrics,/buildinfo, and explain output to trusted networks; - use a dedicated ClickHouse user with the minimum required database privileges.
Failure behavior
Section titled “Failure behavior”ClickHouse unavailable
Section titled “ClickHouse unavailable”Readiness fails. Queries fail rather than returning silently incomplete data, unless PULSUS_SKIP_UNAVAILABLE_SHARDS=true explicitly permits partial shard results. Writers apply bounded queuing and optional spooling according to configuration.
Ingest pressure
Section titled “Ingest pressure”PulsusDB returns 429 when the bounded queue is full. Upstream Collectors should retry with backoff. A push larger than the configured per-request ceiling receives 413 and must be batched smaller at the sender.
Query too broad
Section titled “Query too broad”The request fails with a named limit error. Narrow the time range or add selective matchers before increasing limits; otherwise one client can consume the shared query budget.
Backups and schema changes
Section titled “Backups and schema changes”Back up ClickHouse using the tooling appropriate to your deployment. PulsusDB’s current schema creation script is destructive: it drops and rebuilds the database. Do not treat it as an online migration mechanism.