Observability

Documentation · API reference

Metrics require an explicitly configured observer or metrics adapter. Creating a cache does not connect a metrics backend or register collectors by itself.

DialCache provides first-party adapters for Prometheus and Datadog. Both use caller-created, caller-owned clients and preserve one backend-neutral set of bounded labels.

Choose Prometheus or Datadog for setup. The metric catalog defines shared semantics; the outcome tables below it explain individual feature signals.

Reading the signals

Start with source load and caller latency, then explain changes with the cache metrics. An earlier-layer hit stops traversal; a coalesced follower does not repeat the leader's full read/miss trail.

Signal	Interpretation
Requests and misses by layer	Which layer actually serves or falls through
Miss reason	Absence, logical expiry, invalidation fencing, or an unclassified miss
Disabled reason	Intentional policy/ramp skips versus invalid configuration
Errors and fallback duration	Dependency failures and source cost, including recovered source failures
Shadow outcomes and value ages	Comparison verdicts, fill activity, drops, and detached failures
Recovery outcomes and value ages	How often an older snapshot serves during eligible source failures
Compression size, ratio, and duration	Prepared payload savings versus CPU and scheduling cost
Future timestamp offset	Observed frames ahead of the reader clock; an incomplete clock-health signal

Durations and ages use seconds; sizes use bytes. Namespace, use case, and key type should remain bounded application-defined labels. No metric includes cache ids, arguments, payloads, or raw error text.

Miss reasons

Miss observations carry one required reason. Bundled backends in every port emit the same bounded values:

reason	Meaning
value_absent	No retrievable value: never populated, physically expired, evicted, Redis nil, or tracked MGET wrong-type-as-nil. All local misses use this reason.
expired	A supported valid non-future Redis frame reached its logical fresh age, including retained stale candidates and frames beyond the recovery maximum.
watermark_fenced	A supported positive-timestamp tracked frame was rejected at or below a valid observed watermark, before deserialization.
unclassified	Other real misses, including unknown adapter results, malformed frames/metadata, invalid or future timestamps, and deserialization failures.

Read errors and timeouts are errors, not ordinary misses. The observed watermark used for refill suppression is separate from the reason; a missing value can carry a valid fence.

Prometheus

Install prom-client separately:

npm install prom-client@^15.1.3

This API excerpt assumes an application HTTP router named app. It creates an application-owned registry and passes an explicit adapter to DialCache:

import { Registry } from "prom-client";
import { DialCache } from "dialcache";
import { createPrometheusDialCacheMetrics } from "dialcache/prometheus";

const registry = new Registry();

const dialcache = new DialCache({
  namespace: "users-api",
  metrics: createPrometheusDialCacheMetrics({
    registry,
    prefix: "myapp_",
  }),
});

app.get("/metrics", async (_req, res) => {
  res.type(registry.contentType).send(await registry.metrics());
});

The adapter requires a caller-owned Registry. It never uses the global default registry, and it does not clear or otherwise own the registry lifecycle. prefix defaults to "" and is concatenated literally with each metric name; include any desired separator yourself.

Multiple adapters with the same registry and prefix reuse existing collectors when their type, help, labels, histogram buckets, and exemplar mode match. Adapter construction fails before registering anything if a same-name collector has an incompatible schema. Use a unique prefix or separate registry to resolve a collision. DialCache's collectors do not enable exemplars, so an exemplar-enabled collector with the same name is incompatible.

Create an application-owned prometheus.Registry, then call dialcache.NewPrometheusMetrics(registry, prefix) and supply the result with WithMetrics. Existing compatible collectors created by the DialCache adapter are reused. For externally registered collectors, use NewPrometheusMetricsWithBindings with their actual instances and schemas; native Go registries cannot expose every empty histogram detail for validation.

Shadow admission also needs WithShadowOutcomes. When WithMetrics already forwards every event to the exporter, that hook should only enable admission; forwarding the outcome again would double-count it. See the Go API.

Enable the prometheus feature and create a PrometheusObserver using an application-owned prometheus::Registry. Attach it through the cache builder's observer method. Clone one observer for caches sharing a registry/prefix: the native registry cannot return an existing collector, so creating another observer with the same names returns PrometheusError::Conflict.

The bundled observer opts into shadow outcomes. See the Rust API.

Python exposes the backend-neutral metrics callback contract. Connect events to your application-owned Prometheus registry; this port does not currently ship a Prometheus exporter. Preserve the shared labels, names and units below.

Histogram buckets

Bucket boundaries are fixed; the adapter has no bucket customization option:

Metric family	Unit	Finite bucket boundaries
All timers	Seconds	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10
Serialized and stored sizes	Bytes	100, 1000, 10000, 100000, 1000000, 10000000
Compression ratio	Ratio	0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.9, 1
Shadow and recovery value ages	Seconds	1, 5, 15, 60, 300, 900, 3600, 10800, 43200, 86400, 259200, 604800
Future timestamp offsets	Seconds	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 5, 15, 60, 300, 900, 3600, 10800, 43200

Prometheus metrics

See the metric catalog for names, types, labels, and meanings.

Datadog

Install hot-shots separately:

npm install hot-shots@^17.0.0

This API excerpt assumes an application logger. It creates a caller-owned DogStatsD client and passes it to the Datadog adapter:

import StatsD from "hot-shots";
import { DialCache } from "dialcache";
import { createDatadogDialCacheMetrics } from "dialcache/datadog";

const dogStatsD = new StatsD({
  host: process.env.DD_AGENT_HOST ?? "127.0.0.1",
  globalTags: {
    service: "users-api",
    env: process.env.DD_ENV ?? "development",
  },
  errorHandler: (error) =>
    logger.warn("DogStatsD error", { error }),
});

const dialcache = new DialCache({
  namespace: "users-api",
  metrics: createDatadogDialCacheMetrics({
    client: dogStatsD,
    observationMetricType: "distribution",
    namespace: "dialcache",
  }),
});

function shutdown(): void {
  // Close the client yourself once outstanding DialCache calls have settled.
  // DialCache never flushes or closes it.
  dogStatsD.close();
}

hot-shots is the supported and tested client, but the adapter depends only on the exported DatadogDogStatsDClient structural interface. Construction requires all three methods, increment, histogram, and distribution, to be functions, regardless of the selected observation mode.

DialCache does not:

• import or install hot-shots;
• create a client;
• flush buffers;
• close sockets; or
• otherwise own the client lifecycle.

Supply an application-owned DogStatsDClient to NewDatadogMetrics through DatadogMetricsOptions, then connect it using WithMetrics. The client must implement Increment, Histogram and Distribution; choose an observation mode explicitly. The application owns buffering, transport errors, flushing and shutdown. Shadow admission separately requires WithShadowOutcomes, as in the Prometheus setup. See the Go API.

Construct a DatadogObserver with an application-owned DogStatsdClient and DatadogOptions. Choose histogram or distribution and a metric namespace. Attach it through the cache builder's observer method. The bundled observer opts into shadow outcomes. The application owns delivery, flushing and transport shutdown. See the Rust API.

Connect the Python metrics observer to your application-owned DogStatsD client. This port does not currently ship a Datadog exporter. The application owns metric delivery, flushing and transport shutdown.

Distribution or histogram

The observation mode is required.

Choose "distribution" when latency and size percentiles must aggregate across hosts. Enable the desired distribution percentiles and aggregations in Datadog.

Choose "histogram" when host-level histogram aggregation matches the existing Datadog setup. The choice applies uniformly to every duration, size, ratio, age, and offset observation emitted by the adapter. Both modes produce Datadog custom metrics.

Metric volume depends on tag combinations and selected aggregations. Review Datadog's custom-metrics billing guidance before rollout.

Do not send both observation types under the same metric namespace. When changing types, use a new namespace during migration so one metric identity never mixes histogram and distribution points.

Datadog namespaces

The metric-name namespace defaults to dialcache. It is separate from the logical cache namespace emitted as the cache_namespace tag.

The Datadog metric namespace must:

• start with a letter;
• contain only letters, numbers, underscores, and dot-separated non-empty segments; and
• produce final metric names no longer than 200 characters.

The adapter rejects invalid namespaces and overlong final names instead of relying on client-side normalization. The longest name it builds is <namespace>.shadow_mismatch_logging.count, so the namespace can be at most 170 characters.

A hot-shots prefix is applied after the adapter constructs the name. Include that prefix when checking final length, and avoid accidentally combining it with the adapter namespace. Client-level globalTags are appended by hot-shots; the table below lists only tags added by DialCache.

Any client-added prefixes or global tags are application configuration. Account for them when checking final name length and series cardinality.

Any client-added prefixes or global tags are application configuration. Account for them when checking final name length and series cardinality.

Any metric-name prefix or global tags come from the application's observer. Account for them when checking final name length and series cardinality.

Datadog metrics

The adapter emits exact increments of 1 for counters and preserves seconds and bytes without unit conversion:

Datadog metric	Type	Prometheus equivalent
dialcache.request.count	Count	dialcache_request_counter
dialcache.miss.count	Count	dialcache_miss_counter
dialcache.disabled.count	Count	dialcache_disabled_counter
dialcache.error.count	Count	dialcache_error_counter
dialcache.invalidation.count	Count	dialcache_invalidation_counter
dialcache.coalesced.count	Count	dialcache_coalesced_counter
dialcache.shadow.count	Count	dialcache_shadow_validation_counter
dialcache.shadow.value_age	Distribution or histogram	dialcache_shadow_value_age_histogram
dialcache.shadow_mismatch_logging.count	Count	dialcache_shadow_mismatch_logging_counter
dialcache.future_timestamp_offset	Distribution or histogram	dialcache_future_timestamp_offset_histogram
dialcache.stale_recovery.count	Count	dialcache_stale_recovery_counter
dialcache.stale_recovery.value_age	Distribution or histogram	dialcache_stale_recovery_value_age_histogram
dialcache.compression.count	Count	dialcache_compression_counter
dialcache.get.duration	Distribution or histogram	dialcache_get_timer
dialcache.fallback.duration	Distribution or histogram	dialcache_fallback_timer
dialcache.serialization.duration	Distribution or histogram	dialcache_serialization_timer
dialcache.serialization.size	Distribution or histogram	dialcache_size_histogram
dialcache.stored.size	Distribution or histogram	dialcache_stored_size_histogram
dialcache.compression.ratio	Distribution or histogram	dialcache_compression_ratio_histogram
dialcache.compression.duration	Distribution or histogram	dialcache_compression_timer

Labels and meanings are shared with the metric catalog.

Synchronous client throws are isolated when DialCache invokes the adapter. DialCache also consumes thenables returned by adapter hooks, but this adapter does not forward every client return value: only shadowValidation, shadowMismatchLogging, and staleRecovery return the counter call's result. A custom DogStatsD client must handle its own asynchronous delivery failures, including rejected promises. Direct adapter calls do not have DialCache's observer guard. Configure client error handling and shutdown as part of application ownership.

Errors and panics in metrics callbacks are isolated when DialCache invokes its observer. A custom client still owns asynchronous delivery errors and shutdown after a callback returns.

Observer failures are isolated from cache and source results. A custom client still owns delivery errors and shutdown after a callback returns.

Observer and logger failures are isolated from cache and source outcomes. The application owns delivery errors and shutdown after a callback returns.

Metric catalog

This table uses Prometheus names and types, without the optional prefix. Datadog metrics map to the same observations and labels.

Metric	Type	Labels	Description
dialcache_request_counter	Counter	cache_namespace, use_case, key_type, layer	Cache-layer requests that reached an enabled layer
dialcache_miss_counter	Counter	cache_namespace, use_case, key_type, layer, reason	Cache misses, classified by one required bounded reason
dialcache_disabled_counter	Counter	cache_namespace, use_case, key_type, layer, reason	Cache skips (context, policy_disabled, invalid_ttl, invalid_ramp, ramped_down, config_error)
dialcache_error_counter	Counter	cache_namespace, use_case, key_type, layer, error, in_fallback	Cache/fallback errors and the bounded tracked_ttl_clamped configuration signal
dialcache_invalidation_counter	Counter	cache_namespace, key_type, layer	Invalidation calls for the layers touched
dialcache_coalesced_counter	Counter	cache_namespace, use_case, key_type, scope	Coalesced requests split by request_local or process scope
dialcache_shadow_validation_counter	Counter	cache_namespace, use_case, key_type, outcome	Sampled Redis shadow-job outcomes
dialcache_shadow_value_age_histogram	Histogram	cache_namespace, use_case, key_type, outcome	Age in seconds of the validated Redis value at shadow verdict time, recorded for match and mismatch
dialcache_shadow_mismatch_logging_counter	Counter	cache_namespace, use_case, key_type, outcome	Confirmed-mismatch warning attempts: logged, hook_error, render_error, or logger_error
dialcache_future_timestamp_offset_histogram	Histogram	cache_namespace, use_case, key_type, layer	Positive offset in seconds for a valid frame dated after the observing process clock
dialcache_stale_recovery_counter	Counter	cache_namespace, use_case, key_type, outcome	Classifier-authorized stale-recovery checks: served, miss, or deserialization_error
dialcache_stale_recovery_value_age_histogram	Histogram	cache_namespace, use_case, key_type, outcome	Actual return-time age in seconds of a retained value, recorded only for served
dialcache_compression_counter	Counter	cache_namespace, use_case, key_type, layer, outcome	Payload compression outcomes: writes record compressed, below_threshold, not_smaller, or write_over_limit; reads record decompressed, fallback_raw, or read_over_limit
dialcache_get_timer	Histogram	cache_namespace, use_case, key_type, layer	Cache get latency in seconds
dialcache_fallback_timer	Histogram	cache_namespace, use_case, key_type, layer	Elapsed time until the underlying function settles or timeout rejection is delivered
dialcache_serialization_timer	Histogram	cache_namespace, use_case, key_type, layer, operation	Redis serializer dump/load latency
dialcache_size_histogram	Histogram	cache_namespace, use_case, key_type, layer	Serialized Redis payload size in bytes, before compression
dialcache_stored_size_histogram	Histogram	cache_namespace, use_case, key_type, layer	Prepared Redis payload size in bytes, after compression and escaping; before dispatch
dialcache_compression_ratio_histogram	Histogram	cache_namespace, use_case, key_type, layer	Compressed-to-original payload size ratio for compressed writes
dialcache_compression_timer	Histogram	cache_namespace, use_case, key_type, layer, operation	Payload compression and decompression latency in seconds

policy_disabled means that a process-local or remote layer has no effective TTL after runtime overlays. This is an intentional policy result, including the default when operation policy is omitted, rather than a configuration-loading failure.

Every metric includes cache_namespace, even disabled-context, key-construction, coalescing, and invalidation paths that do not have a constructed key. Its value is the instance namespace, which defaults to urn.

The layer label is:

• request_local;
• local, meaning process-local;
• remote;
• remote_shadow for Redis reads, fills, serialization, compression, and payload sizes performed by detached shadow jobs; or
• noop for disabled-context, key-construction, and config-provider failures where no cache layer was reached.

The bounded scope label on dialcache_coalesced_counter distinguishes request_local from process. scope="process" coordinates calls only within one DialCache instance; separate instances in the same process do not share in-flight state. A use case with coalesce: false emits no coalesced counter; each caller instead emits its own request, miss, duration, and error metrics.

Shadow outcomes

dialcache_shadow_validation_counter reports one terminal outcome for each admitted or explicitly dropped shadow job. Datadog exposes the same bounded outcomes through dialcache.shadow.count:

outcome	Meaning
match	The cached and source values matched.
mismatch	They differed, and a confirmation read found the original Redis payload unchanged.
superseded	They differed, but the Redis payload changed or disappeared before confirmation.
filled	A clean shadow miss was populated successfully.
fill_fenced	A timestamp check skipped tracked fill dispatch against the watermark observed in the initial read.
fill_error	Preparing the payload (serialization or compression) or writing a clean-miss fill failed.
redis_error	The initial detached Redis read failed.
source_error	The source-of-truth read failed.
deserialization_error	The retained Redis payload could not be deserialized for comparison.
comparison_error	The native comparator failed (including a TypeScript non-boolean result).
confirmation_error	The confirmation Redis read failed.
timeout	The shadow deadline expired.
dropped	Per-key deduplication or the instance flight cap rejected the job.

The outcome counter deliberately has no layer or cache-id label. Operational Redis metrics produced inside the same job use layer="remote_shadow", which keeps detached work separate from caller-serving layer="remote" telemetry. See Shadow validation for the read, confirmation, fill, and deadline semantics behind these outcomes.

Stale recovery outcomes

dialcache_stale_recovery_counter and dialcache.stale_recovery.count record one outcome for each classifier-authorized recovery check:

Outcome	Meaning
served	A retained candidate passed return-time age checks and supplied the result
miss	No candidate remained eligible
deserialization_error	Decoding the retained candidate failed

Only served emits the corresponding value-age observation. Recovery adds no second Redis request, miss, or read-duration sequence. The source failure and fallback duration remain visible even when a snapshot serves. Classifier denial emits no recovery outcome. See Stale-on-error.

Value ages and clock offsets

Shadow value age is reported only for match and confirmed mismatch, at verdict time. Recovery age is reported only for served, at return time. Both use the observing application's epoch clock minus the frame's writer timestamp. Shadow age uses the original C0 timestamp, even if confirmation finds identical payload bytes with a newer timestamp. It clamps to zero after clock rollback and skips nonfinite age observations.

The future-offset histogram records a positive offset for valid decoded frames ahead of the observer clock. Ordinary and initial-shadow reads then miss; confirmation can retain the frame only for comparison. Invalid timestamps never enter histogram sums. Repeated reads can observe the same future frame.

Normal cache calls supply positive finite clock offsets. Directly calling a backend adapter bypasses parts of the cache observer boundary; follow that adapter's native input contract.

Use external fleet clock monitoring as well: workload observations cannot detect every skew direction or determine which node is wrong. Its dedicated histogram buckets cover millisecond-scale through multi-hour faults.

Compression metrics

Compression telemetry is bounded and uses layer="remote" for caller-serving work or layer="remote_shadow" for detached shadow work.

Write-side outcomes are:

• compressed: zstd plus its envelope was smaller and selected for the prepared Redis payload;
• below_threshold: the serialized payload did not reach the configured threshold; this check runs before the size ceiling;
• not_smaller: compression ran, but the marked result was not smaller than the raw stored form; and
• write_over_limit: the serialized value reached the threshold but exceeded the 512 MiB decompression ceiling and was kept raw for the attempted write. This is a capacity signal, not an error.

Read-side outcomes are:

• decompressed: a marked zstd payload was restored;
• fallback_raw: native zstd rejected a marked payload for a reason other than the output limit, so it was passed unchanged to the serializer; and
• read_over_limit: decompression would exceed the 512 MiB ceiling, so the stored bytes were passed unchanged to the serializer. Treat this as a corruption or integrity signal.

Raw reads do not emit a compression outcome. With compression: false, new writes are still escaped when necessary but emit no compression outcome; reads continue to report marked values because disabling writes does not disable decoding.

dialcache_size_histogram measures serializer output before compression and is the distribution to use when selecting thresholdBytes. dialcache_stored_size_histogram measures the prepared bytes after compression or binary-envelope escaping. DialCache records it before final shadow-deadline/fence gates and before calling the Redis client, so it is not proof that a write was dispatched or succeeded. The ratio histogram is emitted when compression selects the smaller representation, at the same pre-write stage.

Compression duration is observed when zstd runs and produces either compressed or not_smaller; decompression duration is observed for each marked payload that produces a read-side outcome.

A zstd exception while preparing a write records error="compression" and the cache write fails open. Decompression rejects neither the cache call nor the observer path directly: rejected marked bytes reach the configured serializer. If load rejects, it records serialization_load. An ordinary fresh read becomes a refreshable miss; shadow comparison reports deserialization_error without repair, and retained recovery preserves the original source rejection. See Serialization.

Use duration, ratio and pre/post-size series together when changing threshold or level. A good compression ratio does not establish acceptable runtime cost. The compression guide explains native execution (including Rust's bounded CPU executor), the envelope and mixed-version rollout.

Confirmed mismatch warnings

Shadow metrics remain bounded and contain no cache ids or values. A use case can separately opt into one warning after a terminal mismatch is confirmed. Logging is default-off, does not replace the outcome metric, and does not activate shadow work without an enabled shadow-outcome hook.

The warning contains stable metadata and the logical cache key capped at 2 KiB, plus what the use case's logging policy selects, such as native-JSON strings for the cached and source comparator inputs capped at 8 KiB each. Value-bearing fields are not redacted by truncation.

The policy can also select the JSON Pointers of the fields that differ, alone or with the differing values. Each warning attempt, a confirmed mismatch while logging is on, emits one shadowMismatchLogging event with the use-case labels and an outcome:

outcome	Meaning
logged	Every selected field was built and the logger accepted the warning
hook_error	The use case's diff hook or the built-in diff failed
render_error	A value or a hook entry's side had no JSON rendering
logger_error	The logger failed

When several fail, hook_error ranks above render_error, which ranks above logger_error. When the outcome is not logged, the warning carries it as loggingOutcome, except for logger_error, which the logger never received. With logging off nothing is emitted, so the metric never counts mismatches; use Shadow outcomes for that. The bundled Prometheus and Datadog adapters export it as dialcache_shadow_mismatch_logging_counter and dialcache.shadow_mismatch_logging.count.

The optional shadowMismatchLogging adapter hook receives the event. hook_error means shadowMismatchDiff threw or returned something other than an array. logged means the logger's warn returned, or the promise it returned resolved; logger_error means warn threw, or that promise rejected. A hook_error or render_error is recorded at once, without waiting for the logger. Otherwise, for a logger that returns a promise, the event waits for that promise, and one that never settles emits nothing.

Observers and metrics adapters receive an Event whose Kind is shadowMismatchLogging, with the outcome in Event.Outcome; the WithShadowOutcomes hook does not receive it. hook_error means Operation.MismatchDiff returned an error or panicked, or the built-in computation could not parse back a map, slice or JSONObject nested more than 10,000 levels deep. A struct whose MarshalJSON writes a number outside double range, or whose JSON nests that deep, has no rendering and reports render_error instead. A Go logger returns nothing, so a Warn panic is its only logger_error.

Observers receive Event::ShadowMismatchLogging with the use-case labels and a ShadowMismatchLoggingOutcome, and MetricKind::ShadowMismatchLogging maps it to the names above. hook_error means the mismatch_diff hook panicked, or the built-in computation could not parse a rendering back: one nested 128 or more levels deep, or, with serde_json's raw_value feature on and arbitrary_precision off, a RawValue number outside double range. render_error also covers a selected field the operation has no renderer for and work the bounded CPU executor rejected. A Logger panic is the logger_error.

Observers receive an event dictionary whose event is shadowMismatchLogging, with cacheNamespace, useCase, keyType and outcome. hook_error means shadow_mismatch_diff raised or returned something other than a list. logged means the logger's warning returned, or the awaitable it returned completed; logger_error means warning raised, or that awaitable raised or was cancelled. DialCache awaits an awaitable the logger returns. A hook_error or render_error is recorded at once, without waiting for the logger; otherwise the event waits for the awaitable, so one that never settles emits nothing. Python ships no exporter; map the event to dialcache_shadow_mismatch_logging_counter or dialcache.shadow_mismatch_logging.count to match the bundled adapters.

See Confirmed mismatch logging for confirmation semantics, exact fields, JSON behavior, operational limits, and data-handling considerations.

Error categories

The error label reports the operation that failed instead of copying the native error type or message:

error	Meaning
key_construction	The cache-key selector or identity construction failed
config_resolution	Runtime or layer configuration validation or resolution failed
cache_read	A process-local read or non-timeout remote read failed
cache_read_timeout	A remote read exceeded its effective DialCache deadline
cache_write	A process-local or remote cache write failed
tracked_ttl_clamped	A dispatched tracked write requested retention above the one-hour physical cap
serialization_load	Deserializing a Redis payload failed
serialization_dump	Serializing a value for Redis failed
compression	zstd compression failed while preparing a Redis write
invalidation	Writing an invalidation watermark failed
fallback	The source loader failed or exceeded its DialCache deadline
unknown	Reserved for a future failure site that cannot be classified otherwise

DialCache defines these values itself, so they are identical for every adapter.

A valid explicit invalidation without a configured remote adapter is still an invalidation attempt: it emits the invalidation count and error="invalidation" then returns the native missing-remote error. Invalid buffer inputs fail before these observers run.

Caller-serving remote-read timeouts use layer="remote" and in_fallback="false". They are errors rather than misses, and the remote get-duration observation includes the wait. Coalesced followers do not multiply the timeout error. Deadline details remain out of labels and are available on the native read-timeout error.

Detached initial and confirmation reads attribute their operational metrics to remote_shadow. Read failures, including read timeouts, can report redis_error or confirmation_error without a matching error log. The overall shadow deadline instead reports timeout; see Shadow outcomes.

Raw thrown values, error names, messages, cache ids, arguments, and Redis keys are never included in labels. When DialCache logs a cache-plumbing failure, the raw details remain available through the configured logger; not every metric error or shadow outcome has a matching log entry.

The explicitly opted-in confirmed-mismatch warning is a separate value-bearing log. Its outcome metric carries labels only.

in_fallback remains the explicit distinction between cache plumbing and application fallback failures.

Custom adapters

Implement DialCacheMetricsAdapter and pass it through new DialCache({ metrics }) for another telemetry backend.

Hook	Required	Value
request(labels)	yes	One active cache-layer lookup.
miss(labels)	yes	One cache miss with required bounded reason (MissMetricLabels).
disabled(labels)	yes	One skipped layer or no-layer invocation with a bounded reason.
error(labels)	yes	One bounded failure site with inFallback.
invalidation(labels)	yes	One explicit remote invalidation call.
coalesced(labels)	no	One follower that joined request-local or process-scoped work.
shadowValidation(labels)	no	One terminal sampled-shadow outcome. This hook must be implemented for shadow jobs to execute.
observeShadowValueAge(labels, seconds)	no	Age for shadow match/mismatch verdicts; does not gate admission.
shadowMismatchLogging(labels)	no	One confirmed-mismatch warning attempt with a bounded outcome; emitted only while mismatch logging is on.
observeFutureTimestampOffset(labels, seconds)	no	Positive decoded-frame clock offset; does not change read decisions.
staleRecovery(labels)	no	One authorized recovery outcome; omission does not disable recovery.
observeStaleRecoveryValueAge(labels, seconds)	no	Return-time age for served recovery only.
compression(labels)	no	One bounded compression or decompression outcome.
observeGet(labels, seconds)	yes	Cache-read duration in seconds.
observeFallback(labels, seconds)	yes	Fallback duration in seconds.
observeSerialization(labels, seconds)	yes	Serializer dump/load duration in seconds.
observeSize(labels, bytes)	yes	Serialized remote payload size in bytes, before compression.
observeStoredSize(labels, bytes)	no	Prepared remote payload size in bytes, after compression and escaping; emitted before client dispatch.
observeCompressionRatio(labels, ratio)	no	Compressed-to-original size ratio when compression selects the prepared representation.
observeCompression(labels, seconds)	no	Compression or decompression duration with operation="compress" or operation="decompress".

The root package exports DialCacheMetricsAdapter and every associated label, reason, error-kind, layer, scope, and shadow-outcome type, including ShadowValidationMetricLabels, ShadowValidationOutcome, ShadowMismatchLoggingMetricLabels, ShadowMismatchLoggingOutcome, CompressionMetricLabels, CompressionOperationMetricLabels, and CompressionOutcome, CacheMissReason, MissMetricLabels, StaleRecoveryMetricLabels, and StaleRecoveryOutcome. shadowValidation remains optional so existing custom adapters keep compiling, but DialCache does not admit shadow work when the configured adapter omits it. The Prometheus and Datadog adapters implement the hook.

The compression hooks are also optional for source compatibility with existing custom adapters. They control observation only: omitting them does not disable compression or decompression. The Prometheus and Datadog adapters implement all four hooks.

Metrics and logger methods are typed void and invoked as fire-and-forget observers. DialCache also defensively consumes, but never awaits, a thenable returned at runtime.

Synchronous throws and rejections of those returned thenables are isolated so telemetry cannot change cache correctness, fallback results, or shadow outcomes. This guard applies when DialCache invokes the observer, not to direct calls to an adapter or to asynchronous work whose promise the hook does not return.

Implement MetricsAdapter.ObserveEvent(Event) error and register it with WithMetrics, or use WithObserver for a callback. The adapter receives every backend-neutral diagnostic event. Panics and adapter errors are isolated.

WithShadowOutcomes explicitly permits shadow jobs and receives terminal outcomes; WithRecoveryOutcomes is an optional recovery observer. Missing the recovery hook does not disable recovery. Avoid forwarding an event both through WithMetrics and an outcome hook to the same exporter.

A confirmed-mismatch warning attempt emits a shadowMismatchLogging event whose Outcome is logged, hook_error, render_error, or logger_error. Use the generated Go API for event and adapter types. Preserve the bounded metric names, labels and units in the catalogue above.

Implement Observer and attach it through the cache builder. It receives typed Event values; MetricKind maps them to the shared names, labels and units. Exhaustive matches over Event and MetricKind need a ShadowMismatchLogging arm. Opt into shadow outcomes through observes_shadow_outcomes to admit jobs. The bundled exporters do this; merely logging mismatch warnings does not.

A custom Logger receives structured LogEvents. Observer/logger failures do not change cache results. Use the generated Rust API for exact native traits and event variants.

Pass metrics=callback or an object with synchronous observe(event). Event dictionaries use the shared names and camelCase labels, such as cacheNamespace, useCase and keyType. Supplying an observer enables shadow-outcome reporting; an optional supports("shadowValidation") method can opt out.

The logger option accepts a standard Python-compatible warning logger. DialCache calls warning(message, details), and when any warning's call returns an awaitable, DialCache drives it in a task of its own on the event loop. Unless a hook_error or render_error was already recorded, a confirmed-mismatch warning's shadowMismatchLogging outcome waits for that awaitable. An awaitable that never settles keeps its task pending, holding the awaitable and, for a confirmed mismatch, its namespace, use-case and key-type labels, but neither compared value. Each such warning retains one task: the cost of counting a delivery that fails after warning returns as logger_error. See the generated Python API for signatures.

A custom adapter may buffer or transmit asynchronously, but it owns delivery, flushing, resources, and shutdown after the call returns. Keep application-owned namespace, use-case, and key-type labels stable and low-cardinality, and preserve the seconds and bytes units shown above.

Preserve the logical namespace as the backend's cache_namespace label or tag, including paths where key construction or cache traversal did not happen.

Without an observer that opts into terminal shadow outcomes, shadow work is disabled even if policy selects a shadow cohort or enables mismatch warnings. Ordinary cache behavior and stale recovery remain available without metrics.

See Metric migrations when upgrading collectors, miss queries, or exhaustive outcome mappings.