# OpenTelemetry troubleshooting

> Fix missing or rejected Elide telemetry: check that a destination is configured, read configuration errors, inspect pipeline counters with status(), and see exports in a local collector.

## Nothing arrives

Telemetry runs only when a destination exists. Check, in order:

| Check | How |
| --- | --- |
| A destination is configured | `OTEL_EXPORTER_OTLP_ENDPOINT`, `ELIDE_OTEL_PROVIDER`, `OTEL_TRACES_EXPORTER=otlp`, a `--otel-*` flag, or `destinations` in `elide.pkl` |
| Telemetry is not turned off | No `--no-otel`, `OTEL_SDK_DISABLED=true`, or `enabled = false` |
| The signal is not turned off | No `OTEL_<SIGNAL>_EXPORTER=none`, `traces { enabled = false }`, or `runtime { traces = false }` |
| The provider accepts the signal | See [Providers](https://elide.help/docs/opentelemetry/providers); Datadog sends traces and PostHog sends traces only when listed |
| A one-shot run sees the project | `elide run file.ts` reads `elide.pkl` only when the file contains `opentelemetry` |
| The configuration is accepted | `--verbose` prints why telemetry is not active; see [Configuration errors](#configuration-errors) |
| Your code is active | `active` in [`elide:telemetry`](https://elide.help/docs/opentelemetry/javascript#elidetelemetry) or `active()` in [`elide.telemetry`](https://elide.help/docs/opentelemetry/python#elidetelemetry) |

```ts
// check.ts
import { active, flush, status } from "elide:telemetry";

console.log("active:", active);
console.log("flushed:", await flush());
console.log(status());
```

With nothing listening on the endpoint:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4999 OTEL_EXPORTER_OTLP_TIMEOUT=1000 elide run check.ts
```

```text
active: true
flushed: false
{
  active: true,
  released: false,
  steppedAside: [],
  spans: {
    started: 0,
    unsampled: 0,
    ended: 0,
    live: 0,
    exported: 0,
    dropped: 0,
    tableFull: 0,
    reclaimed: 0,
    handedOff: 0,
    exitDropped: 0
  },
  logs: { emitted: 2, filtered: 0, exported: 0, dropped: 0, handedOff: 0, exitDropped: 0 },
  exportFailures: 1,
  metrics: { handedOff: 0, exitDropped: 0 },
  exportsAbandoned: 0
}
```

`active: false` means no destination was configured. `flushed: false` with `exportFailures` above zero means the
destination could not be reached or rejected the request.

## Configuration errors

A bad value in `elide.pkl` or a flag, a missing credential, or a signal a provider does not accept leaves telemetry
off for the run. The program still runs; `--verbose` prints the reason:

```bash
ELIDE_OTEL_PROVIDER=honeycomb elide run --verbose app.ts   # HONEYCOMB_API_KEY is not set
```

```text
[warn] elide.telemetry Telemetry is not active: telemetry destination 'honeycomb':
environment variable HONEYCOMB_API_KEY is not set
```

The reason names the bad value, missing credential, or unsupported signal. Check, for each
destination:

- its credential variable is set: `SENTRY_DSN`, `DD_API_KEY`, `POSTHOG_PROJECT_TOKEN`, `HONEYCOMB_API_KEY`,
  `GRAFANA_CLOUD_OTLP_CREDENTIAL`, or the variable named by `credential`, `dsn`, or `headerEnv`;
- every signal in `signals` (or `ELIDE_OTEL_SIGNALS`) and its protocol are accepted by the provider;
- `endpoint` starts with `http://` or `https://`, and Grafana has a `region` or `endpoint`;
- `--otel-protocol` is `grpc`, `http/protobuf`, or `http/json`.

A bad environment variable value, such as `OTEL_TRACES_SAMPLER=bogus`, is ignored, and the setting keeps its
`elide.pkl` or default value.

## A provider rejects data

- Check that the credential variable is set in the environment Elide runs in, and holds the full value: a Sentry DSN,
  a Grafana `<instance id>:<token>` pair.
- Check the site or region: a Datadog key only works on its own site, and Honeycomb and PostHog EU accounts need
  `region = "eu"`.
- Check the protocol: Sentry, PostHog, and Grafana take `http/protobuf` only.

Rejected or failed exports are retried with backoff on 429, 502, 503, and 504 responses and on connection errors, up
to the destination's `timeout` (10 seconds by default), then dropped and counted in `exportFailures`.

## Data stops under load

Recording never waits on export. When a destination queue is full, spans and log records are dropped and counted:
`status().spans.dropped`, `status().logs.dropped`. Reduce volume with a sampler
(`traces { sampler = "parentbased_traceidratio"; ratio = 0.1 }`) or a higher `logs.level`, or point Elide at a
nearby collector.

`spans.tableFull` counts spans started while more than 8,192 were open, and `spans.reclaimed` counts spans Elide ended
for you, marked `elide.span.reclaimed`; both usually mean spans that are never ended.

## Short scripts lose data

At exit Elide hands what is pending to a background exporter (see [At exit](https://elide.help/docs/opentelemetry/configuration#at-exit)),
which keeps retrying after the program has ended. Data arrives a moment after exit, not before it. If it never
arrives:

- `ELIDE_OTEL_SIDECAR_LOG=/tmp/elide-otel.log` makes the exporter write what it does, without headers or bodies.
- The exporter gives up on a request after two minutes. Point Elide at a collector it can reach.
- `elide run --otel-shutdown=inline` sends from the exiting process instead, for up to `shutdownTimeout`, and prints
  `Telemetry shutdown failed` when that runs out. Raise it for a slow destination, or call `flush()` before the
  script ends:

```pkl
opentelemetry {
  shutdownTimeout = 15.s
}
```

## An SDK took over

`An OpenTelemetry SDK registered its own tracer provider; Elide's JavaScript tracer provider steps aside` means your
application registered an OpenTelemetry SDK, which now owns that signal and its exporters. `status().steppedAside`
lists the signals it took. Remove the SDK to use Elide's export, or call `release()` before the SDK starts to silence
the warning. See [Using an OpenTelemetry SDK](https://elide.help/docs/opentelemetry/javascript#using-an-opentelemetry-sdk).

## See what Elide sends

Run an OpenTelemetry Collector that prints every request:

```bash
docker run --rm -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector:0.115.1
```

Or [otel-desktop-viewer](https://github.com/CtrlSpice/otel-desktop-viewer), which shows traces, metrics, and logs in a
browser. Point `OTEL_EXPORTER_OTLP_ENDPOINT` at either.