Skip to content

Firestore telemetry calls span.addEvent after gRPC instrumentation ended the span (causes "Operation attempted on ended Span" errors) #3203

@putnap

Description

@putnap

What version of OpenTelemetry are you using?

    "@google-cloud/opentelemetry-cloud-monitoring-exporter": "0.21.0",
    "@google-cloud/opentelemetry-cloud-trace-exporter": "3.0.0",
    "@google-cloud/opentelemetry-cloud-trace-propagator": "0.21.0",
    "@opentelemetry/api": "1.9.0",
    "@opentelemetry/auto-instrumentations-node": "0.66.0",
    "@opentelemetry/exporter-metrics-otlp-proto": "0.207.0",
    "@opentelemetry/sdk-metrics": "2.2.0",
    "@opentelemetry/sdk-node": "0.207.0",

What version of Node are you using?

20.18.1

What did you do?

When using Google Cloud Firestore with OpenTelemetry automatic instrumentation, Firestore's telemetry wrapper calls span.addEvent(...) in a stream 'end' handler after the gRPC instrumentation has already ended the RPC span (onReceiveStatus). This results in runtime errors from the OpenTelemetry SDK: "Operation attempted on ended Span {traceId:..., spanId:...}". The event name and attributes are valid; the problem is that the span has already been ended.

To reproduce:

  • Enable OpenTelemetry automatic instrumentation via a preload or by initializing NodeSDK with instrumentations including @opentelemetry/instrumentation-grpc.

  • Run a Firestore streaming operation (e.g., runQuery / streaming read) that triggers the stream 'end' flow in the Firestore client.

  • Observe errors like "Operation attempted on ended Span {traceId: ..., spanId: ...}" and the diagnostic logs correlating addEvent and span.end stacks.

What did you expect to see?

No errors.

Consider whether span end timing should be configurable for streaming RPCs or whether documentation should call out this interaction so SDK users and other instrumentations can adjust.

What did you see instead?

Observed behavior / diagnostics

- The gRPC instrumentation ends the span in its client onReceiveStatus handler; the stack for span.end originates in:
	- @opentelemetry/instrumentation-grpc -> clientUtils endSpan (onReceiveStatus)
- The addEvent call originates in Firestore's index.js, inside the stream 'end' handler.
- Diagnostic logs captured during reproduction show:
	- addEvent arguments:
		- name: "Firestore.runQuery: Completed"
		- attributes: { response_count: 1 }
	- spanEndedFlag: true
	- call stacks (abbreviated):
		- addEvent call stack:

at SpanImpl.addEvent (...)                              // patched diagnostic
at Span.addEvent (node_modules/@google-cloud/firestore/..)
at PassThrough.<anonymous> (node_modules/@google-cloud/firestore/..:1446)

		- span end stack:
		
at SpanImpl.end (...)                                   // patched diagnostic
at endSpan (node_modules/@opentelemetry/instrumentation-grpc/build/src/clientUtils.js:84:18)
at ClientReadableStreamImpl.<anonymous> (..clientUtils.js:110:9)
at ClientReadableStreamImpl.emit (node:events:...)
at Object.onReceiveStatus (node_modules/@grpc/grpc-js/build/src/client.js:364:24)

Additional context

Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingpriority:p2Bugs and spec inconsistencies which cause telemetry to be incomplete or incorrect

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions