-
Notifications
You must be signed in to change notification settings - Fork 620
Description
What version of OpenTelemetry are you using?
"@google-cloud/opentelemetry-cloud-monitoring-exporter": "0.21.0",
"@google-cloud/opentelemetry-cloud-trace-exporter": "3.0.0",
"@google-cloud/opentelemetry-cloud-trace-propagator": "0.21.0",
"@opentelemetry/api": "1.9.0",
"@opentelemetry/auto-instrumentations-node": "0.66.0",
"@opentelemetry/exporter-metrics-otlp-proto": "0.207.0",
"@opentelemetry/sdk-metrics": "2.2.0",
"@opentelemetry/sdk-node": "0.207.0",
What version of Node are you using?
20.18.1
What did you do?
When using Google Cloud Firestore with OpenTelemetry automatic instrumentation, Firestore's telemetry wrapper calls span.addEvent(...) in a stream 'end' handler after the gRPC instrumentation has already ended the RPC span (onReceiveStatus). This results in runtime errors from the OpenTelemetry SDK: "Operation attempted on ended Span {traceId:..., spanId:...}". The event name and attributes are valid; the problem is that the span has already been ended.
To reproduce:
-
Enable OpenTelemetry automatic instrumentation via a preload or by initializing NodeSDK with instrumentations including @opentelemetry/instrumentation-grpc.
-
Run a Firestore streaming operation (e.g., runQuery / streaming read) that triggers the stream 'end' flow in the Firestore client.
-
Observe errors like "Operation attempted on ended Span {traceId: ..., spanId: ...}" and the diagnostic logs correlating addEvent and span.end stacks.
What did you expect to see?
No errors.
Consider whether span end timing should be configurable for streaming RPCs or whether documentation should call out this interaction so SDK users and other instrumentations can adjust.
What did you see instead?
Observed behavior / diagnostics
- The gRPC instrumentation ends the span in its client onReceiveStatus handler; the stack for span.end originates in:
- @opentelemetry/instrumentation-grpc -> clientUtils endSpan (onReceiveStatus)
- The addEvent call originates in Firestore's index.js, inside the stream 'end' handler.
- Diagnostic logs captured during reproduction show:
- addEvent arguments:
- name: "Firestore.runQuery: Completed"
- attributes: { response_count: 1 }
- spanEndedFlag: true
- call stacks (abbreviated):
- addEvent call stack:
at SpanImpl.addEvent (...) // patched diagnostic
at Span.addEvent (node_modules/@google-cloud/firestore/..)
at PassThrough.<anonymous> (node_modules/@google-cloud/firestore/..:1446)
- span end stack:
at SpanImpl.end (...) // patched diagnostic
at endSpan (node_modules/@opentelemetry/instrumentation-grpc/build/src/clientUtils.js:84:18)
at ClientReadableStreamImpl.<anonymous> (..clientUtils.js:110:9)
at ClientReadableStreamImpl.emit (node:events:...)
at Object.onReceiveStatus (node_modules/@grpc/grpc-js/build/src/client.js:364:24)
Additional context
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.