API Reference
SPYDER exposes a REST API on the metrics port (default :9090) for runtime control, monitoring, and querying discovered data.
Interactive Reference
REST API for controlling and inspecting a running SPYDER DNS reconnaissance probe.
The API runs on the same port as the Prometheus metrics endpoint (default :9090).
All endpoints require a Bearer token in the Authorization header. Tokens are
scoped to one of three permission levels: read, write, or admin.
Admin scope is a superset of write, which is a superset of read.
Servers
Submit a single domain for crawling
Enqueues a single domain hostname into the crawl task channel. The domain will be picked up by the next available worker. Returns 503 if the task channel is full.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Domain accepted
Submit multiple domains for crawling
Enqueues multiple domains at once. Accepts JSON array or newline-delimited text/plain. Domains that cannot be enqueued (channel full) are counted as rejected.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Bulk submission result
Pause all crawl workers
Pauses all workers in the pool. Workers finish their current task but do not pick up new ones until resumed.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Workers paused
Resume paused crawl workers
Resumes all paused workers so they begin pulling tasks from the queue again.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Workers resumed
Gracefully stop the probe process
Initiates a graceful shutdown of the entire SPYDER process by cancelling the root context. The response is sent before shutdown begins.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Stop initiated
Get the current runtime configuration snapshot
Returns the full current runtime configuration as a JSON object, including all nested sections (crawling, batch, output, etc.).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Current configuration
Hot-patch runtime configuration
Applies a partial configuration update. Tier 1 fields are applied immediately, Tier 2 fields produce warnings, and Tier 3 fields (e.g. Redis, mTLS) are rejected as they require a restart.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated configuration and any warnings
Get aggregate crawl statistics
Returns aggregate hub statistics including total batches processed, edges, domains, IPs, and certificates seen.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Current stats snapshot from the hub
Get batch event history
Returns the recent batch history ring buffer from the hub, ordered newest first.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Array of recent batch events
Get high-level probe status
Returns a combined status snapshot including uptime, worker pool state, pending emitter counts, and discovery progress.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Probe status overview
Get worker pool statistics
Returns the current worker pool statistics: target count, active count, and paused state.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Worker pool stats
Scale the worker pool to a target count
Dynamically scales the worker pool to the specified count. Workers are added or drained gracefully.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Scaling result
List all circuit breaker states
Returns circuit breaker state for all known hosts, including current state (closed/open/half-open) and failure counts.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Map of host to circuit breaker state
Reset all circuit breakers
Resets all circuit breakers to the closed state, allowing requests to resume to previously-failing hosts.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
All breakers reset
Reset a single circuit breaker by host
Resets the circuit breaker for a specific host back to the closed state.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
Breaker reset
Get per-host rate limit configuration and active hosts
Returns the global per-host rate limit configuration and per-host limiter statistics.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Current rate limit settings and per-host stats
Update global per-host rate limit settings
Updates the global per-host rate limit and burst values. Also patches the runtime config to persist the change.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated rate limit settings
Notify that a per-host rate limiter will expire via TTL
Per-host rate limiter entries expire automatically via TTL and cannot be individually reset.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
Informational response
Check whether a dedup key has been seen
Checks whether a given deduplication key has been seen. The call itself marks the key as seen (side effect of the Seen() method).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"domain|example.com"Responses
Dedup check result
Delete a specific dedup key
Removes a single key from the dedup store. Only supported by backends that implement the Delete interface.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
Responses
Key deleted
Clear all dedup entries (admin only)
Clears the entire dedup store. Only supported by backends that implement the Clear interface. Admin scope required.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Dedup store cleared
Get robots.txt cache status
Returns the status of the robots.txt LRU cache. Individual entries cannot be listed.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Cache status message
Attempt to clear the robots.txt cache (TTL-based; informational)
Robots cache entries expire via TTL; explicit clear is not currently supported.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Informational response
List spooled batch files
Lists all files in the spool directory (failed batch deliveries awaiting retry), with file names, sizes, and timestamps.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Spool directory listing
Delete all spooled batch files
Deletes all files in the spool directory.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Number of files deleted
Retry delivery of spooled batches
Triggers a retry of all spooled batches by calling the emitter's Drain method.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Retry initiated
Get emitter status and pending batch counts
Returns the emitter's current state including pending edge and node counts, ingest URL, and spool directory path.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Emitter status
Force-flush the current in-memory batch
Forces the emitter to immediately flush any buffered edges and nodes to the ingest endpoint.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Flush triggered
Get the current TLD exclusion list
Returns the list of TLDs currently excluded from crawling.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Current exclusion list
Replace the entire TLD exclusion list
Replaces the entire TLD exclusion list with the provided array.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated exclusion list
Add a single TLD to the exclusion list
Adds a single TLD to the exclusion list if not already present.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated exclusion list
Remove a TLD from the exclusion list
Removes a single TLD from the exclusion list.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"edu"Responses
Updated exclusion list
Get discovery limits and counters
Returns the current discovery state: number of domains discovered, max domain limit, and whether continuous mode is enabled.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Discovery settings
Update discovery limits
Updates discovery settings. Can change max_domains (also updates the sink's atomic limit) and continuous mode.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated discovery settings
Inspect the work queue (not implemented)
Queue introspection is not yet implemented.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Endpoint not yet implemented
Clear the work queue (not implemented)
Queue clear is not yet implemented.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Endpoint not yet implemented
Remove a host from the queue (not implemented)
Queue item removal is not yet implemented.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
Endpoint not yet implemented
Stream log output (not yet implemented)
Server-Sent Events log stream. Not yet implemented.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Endpoint not yet implemented
Change the log verbosity level at runtime
Changes the runtime log level by patching the logging config.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Level applied
Get metrics endpoint reference (use /metrics for Prometheus format)
Returns a pointer to the Prometheus /metrics endpoint. Use that endpoint directly for scraping.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Metrics endpoint info
Reset Prometheus counters (not supported; requires restart)
Prometheus counters cannot be reset at runtime; a process restart is required.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Informational response
Get current telemetry configuration
Returns the current OpenTelemetry configuration (endpoint, service name, insecure flag).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Telemetry config
Update telemetry settings (informational; requires restart to take effect)
Telemetry configuration changes require a restart to take effect. Changes are stored but not applied.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Informational response with current telemetry config
Get health checker registration status
Health checkers are registered programmatically. Use GET /health for current status.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Checker info
Register a health checker (not supported)
Dynamic checker registration via API is not supported.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Endpoint not yet implemented
Remove a health checker (not supported)
Dynamic checker removal via API is not supported.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
Responses
Endpoint not yet implemented
Get current output configuration
Returns the current output configuration (format, ingest URL, spool directory).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
Output config
Update output configuration (Tier 2 — applied with warnings)
Updates output configuration fields. Subject to tier-based validation.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Updated output config and warnings
Perform a live DNS lookup for a host
Performs a live DNS resolution for the given host, returning A/AAAA, NS, CNAME, MX, and TXT records.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
DNS resolution results
Fetch the TLS certificate for a host
Fetches the TLS certificate from the given host on port 443 and returns parsed certificate metadata.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
TLS certificate details
Get HTTP client configuration
Returns the current HTTP client configuration (timeouts, connection pool settings).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
HTTP client config
Update HTTP client configuration (requires restart)
HTTP client configuration changes require a restart.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Endpoint not yet implemented
List all API keys (admin only)
Lists all configured API keys with their names and scopes. Admin scope required.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Responses
All registered API keys
Create a new API key (admin only)
Creates a new API key with a randomly generated token. Returns the key value (only shown once). Admin scope required.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Request Body
Responses
Created API key (key value shown only on creation)
Delete an API key (admin only)
Revokes and deletes an API key. Admin scope required.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
The API key token to delete
Responses
Key deleted
List discovered domain nodes
Lists discovered domains with optional filtering by apex domain or search string. Supports cursor-based pagination.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Query Parameters
Opaque pagination cursor from next_cursor in the previous response
Maximum items to return per page
1100050Filter by apex domain
"example.com"Substring search on hostname
Responses
Paginated domain list
Get a single domain with its edges
Returns details for a specific domain including all related edges (inbound and outbound).
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Path Parameters
"example.com"Responses
Domain detail with edges
List discovered edges
Lists discovered edges with optional filtering by type, source, or target. Supports cursor-based pagination.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Query Parameters
Opaque pagination cursor from next_cursor in the previous response
Maximum items to return per page
1100050Filter by edge type
"RESOLVES_TO""USES_NS""ALIAS_OF""USES_MX""LINKS_TO""USES_CERT"Filter by source node
Filter by target node
Responses
Paginated edge list
List discovered IP nodes
Lists discovered IP addresses with optional search filtering. Supports cursor-based pagination.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Query Parameters
Opaque pagination cursor from next_cursor in the previous response
Maximum items to return per page
1100050Substring or CIDR search on IP address
Responses
Paginated IP list
List discovered TLS certificate nodes
Lists discovered TLS certificates with optional search filtering by subject or issuer CN. Supports cursor-based pagination.
Authorizations
API key issued by POST /api/v1/keys. Pass as Authorization: Bearer <key>.
Keys prefixed with spyder_ followed by 64 hex characters.
Parameters
Query Parameters
Opaque pagination cursor from next_cursor in the previous response
Maximum items to return per page
1100050Substring search on subject CN or issuer CN
Responses
Paginated certificate list
Authentication
All endpoints require an API key passed via the Authorization header:
Authorization: Bearer <api-key>Keys have one of three scopes: read, write, or admin. See Authentication for details.
Base URL
http://localhost:9090/api/v1/