Observability (OpenTelemetry)

The observability system uses OpenTelemetry (OTel) to produce traces and metrics from both the API server and the background worker. It is designed from the ground up to be platform-agnostic — no vendor-specific SDKs, no proprietary codecs. Switch backends by changing a single environment variable.

Design Principles

1. Vendor-Neutral: Only official @opentelemetry/* packages. Zero lock-in to Signoz, Grafana, Datadog, Sentry, or any other backend.
2. gRPC First: OTLP over gRPC (port 4317) for high-throughput, low-latency export. Suitable for financial-system traffic volumes from day one.
3. Automatic Instrumentation: Traces and metrics are produced by instrumentation wrappers — no manual span creation for standard operations.
4. Separate Service Identity: The API server and background worker each report as distinct services with their own resource attributes, making it easy to distinguish trace origin in any dashboard.
5. Graceful Lifecycle: Telemetry SDK starts before application code and shuts down after the server closes, ensuring no data loss during deployments.

Architecture

┌─────────────────────┐    ┌─────────────────────────┐
│   API Server        │    │   Background Worker      │
│   (remote-eaze-api) │    │   (remote-eaze-worker)  │
│                     │    │                         │
│  ┌───────────────┐  │    │  ┌───────────────────┐  │
│  │  NodeSDK      │  │    │  │  NodeSDK          │  │
│  │  - HTTP       │  │    │  │  - Prisma         │  │
│  │  - Fastify    │  │    │  │  - IORedis        │  │
│  │  - Prisma     │  │    │  │  - Pino           │  │
│  │  - IORedis    │  │    │  │  - BullMQ Otel    │  │
│  │  - Pino       │  │    │  └───────┬───────────┘  │
│  └───────┬───────┘  │    └──────────┼──────────────┘
└──────────┼──────────┘               │
           │                          │
           │      OTLP / gRPC         │
           │      (port 4317)         │
           └──────────┬───────────────┘
                      │
                      ▼
         ┌────────────────────────┐
         │  OTLP-Compatible      │
         │  Backend              │
         │  (Signoz / Grafana /  │
         │   Datadog / New Relic │
         │   / Jaeger / etc.)    │
         └────────────────────────┘

Platform Compatibility

The system exports standard OTLP over gRPC. Any backend that accepts OTLP gRPC can consume the data:

No code changes required — just set the environment variable.

Service Configuration

API Server (remote-eaze-api)

The API service instruments all layers that handle inbound traffic and external calls:

Health endpoint exclusions:

HTTP instrumentation:   ignoreIncomingRequestHook → req.url === "/health"
Fastify instrumentation: ignorePaths → "/health"

This prevents health-check spam from polluting trace data in load-balanced environments.

Background Worker (remote-eaze-worker)

The worker service has a focused instrumentation set — no HTTP or Fastify, since it processes jobs rather than serving requests:

The BullMQ integration is not an OpenTelemetry instrumentation package. It is a BullMQ-specific wrapper (BullMQOtel) passed to Queue and Worker constructors to create spans for each job lifecycle event.

gRPC vs HTTP Export

The implementation uses gRPC because financial systems produce high trace volumes and benefit from lower serialization and network overhead. Port 4317 is the OTel convention for gRPC receivers.

Sampling Strategy

Sampling controls what percentage of traces are collected, balancing observability with storage cost.

OTEL_SAMPLING_RATIO=1.0    → 100% of traces (development default)
OTEL_SAMPLING_RATIO=0.1    → 10% of traces (typical production)
OTEL_SAMPLING_RATIO=0.01   →  1% of traces (high-volume production)

The system uses ParentBasedSampler with TraceIdRatioBasedSampler as the root:

Incoming request (no parent)
       │
       ▼
  TraceIdRatioBasedSampler
  (decides based on OTEL_SAMPLING_RATIO)
       │
       ▼
  Sampled? ──Yes──→ All child spans are sampled
       │
       No
       │
       ▼
  Entire trace discarded

Parent-based sampling ensures complete distributed traces — if a root span is sampled, all of its children are sampled too. There are no partial traces with missing spans.

Resource Attributes

Each service reports identity and environment metadata:

These attributes appear in every exported span and metric, enabling filtering by service and environment in any dashboard.

Audit-Trace Correlation

The audit log system captures the active trace ID alongside every audit event:

traceId: trace.getActiveSpan()?.spanContext().traceId

This creates a bidirectional link:

Trace Dashboard ──→ Find trace by ID → See full request lifecycle
Audit Log Search ──→ Find traceId field → Jump to trace in dashboard

Every audit log entry includes traceId in its context fields, indexed for fast lookups.

Log Correlation

Pino instrumentation automatically injects trace and span IDs into log entries:

{
  "level": 30,
  "time": 1710000000000,
  "msg": "Transaction processed",
  "trace_id": "abc123def456",
  "span_id": "789ghi012"
}

This enables log-to-trace and trace-to-log navigation in any observability platform that supports OTel log correlation.

Startup and Shutdown

Startup: Pre-Load Pattern

Telemetry must initialize before application code so that early operations (database connections, plugin registration) are captured.

Production (Docker):

CMD ["node", "--enable-source-maps", "--import", "./dist/telemetry.js", "dist/main.js"]

The --import flag loads the telemetry module before the main entry point, ensuring the NodeSDK is fully initialized before Fastify starts.

Development:

A dev-bootstrap.ts file imports telemetry first, then the application. This guarantees consistent behavior across environments.

Shutdown: Graceful Flush

Both services flush pending telemetry on termination:

SIGINT / SIGTERM received
      │
      ▼
  Server closes (stops accepting requests)
      │
      ▼
  Database disconnects
      │
      ▼
  Telemetry SDK flushes remaining spans/metrics
      │
      ▼
  Process exits (code 0) or error exit (code 1)

A singleton guard prevents double-shutdown:

let shutdownPromise: Promise<void> | null = null;

export const shutdownTelemetry = async (): Promise<void> => {
  if (!shutdownPromise) {
    shutdownPromise = sdk.shutdown().catch((err: unknown) => {
      console.error("Telemetry shutdown error:", err);
    });
  }
  await shutdownPromise;
};

If the SDK has already been shut down, subsequent calls await the same promise rather than attempting a second shutdown.

Frontend Instrumentation

Configuration

The endpoint must point to a gRPC receiver. Port 4317 is the standard OTel gRPC port. Do not include a path suffix (e.g., /v1/traces) — the SDK appends the correct paths automatically.

Package Dependencies

All telemetry packages are scoped under @opentelemetry/* (official OpenTelemetry packages only):

Root (shared):

API Service additional:

Worker / Queue package: