# OpenTelemetry configuration

> Configure Elide's OpenTelemetry export with the elide.pkl opentelemetry block, OTEL_* and ELIDE_OTEL_* environment variables, and --otel-* flags, and how they combine.

Telemetry is configured from three places. Higher rows win:

| Source | Scope | Example |
| --- | --- | --- |
| `--otel-*` flags | One run | `elide run --otel-endpoint=http://localhost:4318 app.ts` |
| `OTEL_*` and `ELIDE_OTEL_*` variables | One process and its environment | `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` |
| `opentelemetry` block in `elide.pkl` | A project | `destinations { ["collector"] { … } }` |
| Defaults | | Off until a destination exists |

Telemetry stays off until a destination exists in one of them. With no destination, Elide starts no telemetry
threads, opens no connections, and the OpenTelemetry APIs in your code do nothing.

## When each source is read

| Command | `elide.pkl` block | Variables and flags |
| --- | --- | --- |
| `elide run` (project entrypoint), `elide serve` | yes | yes |
| `elide run file.ts`, `elide file.py` inside a project | yes, when the project's `elide.pkl` contains `opentelemetry` | yes |
| `elide run file.ts` outside a project | no | yes |
| `elide build`, `elide test`, `elide install` | yes | yes |

A one-shot run looks for the word `opentelemetry` in the project's `elide.pkl` before evaluating it, so a block
inherited from another module is not seen by `elide run file.ts`; projects without telemetry pay nothing.

## The `elide.pkl` block

```pkl
amends "elide:project.pkl"

name = "checkout"

opentelemetry {
  service { name = "checkout"; version = "1.4.2"; environment = "production" }
  resource { ["cloud.region"] = "us-east-1" }

  destinations {
    ["collector"] { endpoint = "http://localhost:4317"; protocol = "grpc" }
    ["sentry"] { provider = "sentry" }
  }

  traces { sampler = "parentbased_traceidratio"; ratio = 0.25 }
  metrics { interval = 30.s }
  logs { level = "info" }
  runtime { logs = "warn" }
}
```

Credentials never go in `elide.pkl`. A destination names the environment variable that holds them (`credential`,
`dsn`, `headerEnv`), and Elide reads it at startup.

### Top level

| Property | Meaning | Default |
| --- | --- | --- |
| `enabled` | `false` turns telemetry off, whatever the environment says. | — |
| `service.name` | `service.name` resource attribute. | `unknown_service:elide` |
| `service.version`, `.namespace` | `service.version`, `service.namespace`. | — |
| `service.environment` | `deployment.environment.name`. | — |
| `resource` | Extra resource attributes: strings, booleans, integers, floats. | — |
| `destinations` | Named places to send telemetry; see below. | none |
| `shutdownTimeout` | Longest the final export may take at exit; see [At exit](#at-exit). | `5.s` |

### Signals

| Property | Meaning | Default |
| --- | --- | --- |
| `traces.enabled` | Record spans your code creates. | `true` |
| `traces.sampler` | `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`. | `parentbased_always_on` |
| `traces.ratio` | Ratio for the ratio samplers, 0 to 1. | `1.0` |
| `traces.propagators` | `tracecontext`, `baggage`, `none`: what the OpenTelemetry API in your code injects and extracts. | `tracecontext`, `baggage` |
| `metrics.enabled` | Record instruments your code creates. | `true` |
| `metrics.interval` | Export interval. | `60.s` |
| `metrics.temporality` | `cumulative` or `delta`. | `cumulative`; `delta` when Datadog receives metrics |
| `logs.enabled` | Export log records your code emits. | `true` |
| `logs.level` | Minimum severity: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `off`. | `info` |
| `logs.console` | Export `console.*` and Python `logging` output as log records. | `true` |
| `profiles.enabled` | Sample guest CPU profiles (experimental). | `false` |
| `profiles.frequency` | Samples per second, 1 to 10000. | `100` |
| `runtime.traces` | Elide's server and client HTTP spans. | `true` |
| `runtime.metrics` | Elide's HTTP, process, and error metrics. | `true` |
| `runtime.logs` | Minimum severity of Elide's own log records to export. | `warn` |
| `propagation.baggage` | W3C baggage on Elide's HTTP server and clients; see [Baggage](https://elide.help/docs/opentelemetry/correlation#baggage). | off |

### Destinations

Each entry in `destinations` is one place telemetry goes. The name is yours; it appears in error messages.

| Property | Meaning | Default |
| --- | --- | --- |
| `provider` | `otlp`, `sentry`, `datadog`, `posthog`, `honeycomb`, `grafana`. See [Providers](https://elide.help/docs/opentelemetry/providers). | `otlp` |
| `endpoint` | Base URL. HTTP protocols append `/v1/traces`, `/v1/metrics`, `/v1/logs`. For a provider, replaces its host. | `http://localhost:4318`, or `http://localhost:4317` for gRPC |
| `protocol` | `grpc`, `http/protobuf`, `http/json`. | `http/protobuf` |
| `headers` | Literal, non-secret request headers. | — |
| `headerEnv` | Headers read from environment variables: `["authorization"] = "COLLECTOR_TOKEN"`. | — |
| `credential` | Variable holding a provider's API key or token. | per provider |
| `dsn` | Variable holding a Sentry DSN. | `SENTRY_DSN` |
| `site` | Datadog site. | `DD_SITE`, then `datadoghq.com` |
| `region` | PostHog or Honeycomb region (`us`, `eu`), or a Grafana Cloud zone. | `us` |
| `signals` | `traces`, `metrics`, `logs`, `profiles`. | every signal the provider takes by default |
| `compression` | `gzip` or `none`. | `gzip` |
| `timeout` | Per-request deadline, retries included. | `10.s` |
| `perSignal` | Per-signal `endpoint` (used as given), `protocol`, `headers`, `headerEnv`, `compression`, `timeout`. | — |

Several destinations can be active at once. Each receives only the signals it lists, or that its provider takes by
default:

```pkl
opentelemetry {
  destinations {
    ["collector"] { endpoint = "http://otel-collector:4318" }
    ["errors"] { provider = "sentry" }
    ["metrics"] { provider = "datadog"; site = "us5"; signals { "metrics" } }
  }
}
```

A collector that needs a token:

```pkl
opentelemetry {
  destinations {
    ["collector"] {
      endpoint = "https://otel.example.com"
      headerEnv { ["authorization"] = "OTEL_COLLECTOR_AUTH" }
      perSignal { ["metrics"] { endpoint = "https://metrics.example.com/otlp/v1/metrics" } }
    }
  }
}
```

The complete schema, with every other `elide.pkl` block, is in the [elide.pkl reference](https://elide.help/docs/elide-pkl-reference#opentelemetry-block).

## Environment variables

Elide reads the standard OpenTelemetry SDK variables. An invalid value is ignored, and the setting keeps its manifest
or default value.

| Variable | Effect | Default |
| --- | --- | --- |
| `OTEL_SDK_DISABLED` | `true` turns telemetry off. Only a `--otel-*` flag turns it back on. | `false` |
| `OTEL_SERVICE_NAME` | `service.name`. | `unknown_service:elide` |
| `OTEL_RESOURCE_ATTRIBUTES` | Extra resource attributes, `key=value,key=value`, values percent-encoded. | — |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL for `otlp` destinations; creates one named `otlp` when there is none. | — |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc`, `http/protobuf`, `http/json`. | `http/protobuf` |
| `OTEL_EXPORTER_OTLP_HEADERS` | Request headers, `key=value,…`. Treated as secrets. | — |
| `OTEL_EXPORTER_OTLP_COMPRESSION` | `gzip` or `none`. | `gzip` |
| `OTEL_EXPORTER_OTLP_TIMEOUT` | Request deadline in milliseconds. | `10000` |
| `OTEL_EXPORTER_OTLP_CERTIFICATE` | PEM file of trusted CA certificates; replaces the system trust store. | system |
| `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_KEY` | PEM client certificate and key for mutual TLS. | — |
| `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT` | Full URL for one signal, used as given. | — |
| `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_{PROTOCOL,HEADERS,COMPRESSION,TIMEOUT,CERTIFICATE,CLIENT_CERTIFICATE,CLIENT_KEY}` | The same settings for one signal. | — |
| `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, `OTEL_LOGS_EXPORTER` | `none` turns that signal off, for your code and for Elide. `otlp` with no destination anywhere sends that signal to `http://localhost:4318`. | — |
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Sampler and ratio, as `traces.sampler` and `traces.ratio`. | `parentbased_always_on` |
| `OTEL_PROPAGATORS` | `tracecontext`, `baggage`, `none` for the API in your code. Does not turn on baggage for Elide's HTTP server and clients. | `tracecontext,baggage` |
| `OTEL_METRIC_EXPORT_INTERVAL` | Metric export interval in milliseconds. | `60000` |
| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `cumulative` or `delta`. | `cumulative` |
| `OTEL_METRICS_EXEMPLAR_FILTER` | `trace_based`, `always_on`, `always_off`. See [Exemplars](https://elide.help/docs/opentelemetry/correlation#metrics-and-exemplars). | `trace_based` |
| `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN` | `logs` records errors as log records only; `logs/dup` also adds `exception` span events. | `logs/dup` |

Elide adds shorthands for providers and for settings the standard variables do not cover:

| Variable | Effect | Default |
| --- | --- | --- |
| `ELIDE_OTEL_PROVIDER` | Adds a destination for a provider, named after it. | — |
| `ELIDE_OTEL_REGION` | That destination's site or region: `us5` (Datadog), `eu` (PostHog, Honeycomb), `prod-us-east-0` (Grafana). | provider default |
| `ELIDE_OTEL_SIGNALS` | That destination's signals, e.g. `traces,logs`. | provider default |
| `ELIDE_OTEL_PROFILES` | `true` samples guest CPU profiles at 100 per second; a number sets the frequency. | off |
| `ELIDE_OTEL_BAGGAGE_INBOUND` | `true` admits served requests' `baggage` headers. | `false` |
| `ELIDE_OTEL_BAGGAGE_OUTBOUND` | `true` sends the current baggage on outgoing requests. | `false` |
| `ELIDE_OTEL_BAGGAGE_ALLOW` | Baggage keys allowed in both directions, comma-separated. | all keys |
| `ELIDE_OTEL_SHUTDOWN` | How the final export leaves at exit: `sidecar`, `inline`, `drop`. See [At exit](#at-exit). | `sidecar` |

`OTEL_EXPORTER_OTLP_*` apply to every destination that uses the `otlp` provider, including ones named in `elide.pkl`.
Provider credentials come from the variables listed on [Providers](https://elide.help/docs/opentelemetry/providers).

## Flags

| Flag | Effect |
| --- | --- |
| `--otel-endpoint=URL` | Adds or updates a destination with this endpoint. |
| `--otel-provider=PROVIDER` | Names that destination's provider; `otlp` when omitted. |
| `--otel-protocol=PROTOCOL` | `grpc`, `http/protobuf`, or `http/json` for that destination. |
| `--otel-shutdown=MODE` | `sidecar`, `inline`, or `drop`; see [At exit](#at-exit). |
| `--no-otel` | Turns telemetry off for this run, whatever the manifest and environment say. |

The flags take `=` and apply to `run`, `serve`, `test`, `build`, and `install`:

```bash
elide run --otel-endpoint=http://localhost:4317 --otel-protocol=grpc app.ts
elide test --no-otel
```

## At exit

When a program ends, Elide still holds telemetry it has not sent: the last log records and spans, and a final
collection of metrics. The shutdown mode decides how that leaves the process:

| Mode | At exit | Exit waits |
| --- | --- | --- |
| `sidecar` | Hands the final export to a background exporter, which sends it. | under 250 ms, never on the network |
| `inline` | Sends the final export from the exiting process. | up to `shutdownTimeout` |
| `drop` | Discards the final export and counts it. | no |

`sidecar` is the default on Linux and macOS. The exporter is a small `elide` process, one per user, started by the
first exit that needs it. It accepts requests only from processes of the same user, keeps them in memory only, sends
each one once with the usual retries, and exits once no Elide process of that user is left and its queue is sent or
has expired. It never runs longer than 15 minutes. If the exporter cannot be started or reached in time, the final
export is dropped and counted, never sent from the exiting process.

On Windows, `inline` is the only mode; `sidecar` falls back to it with a warning. Choose `inline` to have the exiting
process send the data itself, for example when nothing may outlive the program:

```bash
elide run --otel-shutdown=inline app.ts
ELIDE_OTEL_SHUTDOWN=drop elide run app.ts
```

## How sources combine

- Destinations merge by name. A destination from `--otel-provider=honeycomb` and one named `honeycomb` in
  `elide.pkl` are the same destination; the flag's fields win.
- `OTEL_EXPORTER_OTLP_*` settings apply to every `otlp` destination. With no `otlp` destination, an endpoint
  variable creates one named `otlp`.
- `elide.pkl` and flag values are checked strictly: a bad value leaves telemetry off for the run, with a warning; see
  [Configuration errors](https://elide.help/docs/opentelemetry/troubleshooting#configuration-errors). Bad variable values are skipped.
- A destination left with no enabled signal is skipped. When no destination receives anything, telemetry stays off.
- Configuration is read once, at startup. Your code cannot change destinations, headers, or the resource.

## Samples

- [OTLP to a local collector](https://elide.help/docs/code-samples#otlp-to-a-local-collector)