eBPF agent saturates internal queue (sending queue is full) on one Kubernetes node even with restricted instrumentation

[merouaneagar]

Description

Hello team,
I’m facing an issue where the eBPF OTel agent saturates its internal queue (sending queue is full) repeatedly on one Kubernetes node, even with:

• restricted namespaces in discovery.instrument
• additional exclude_services
• reduced instrumentation scope
• and the OpenTelemetry Collector in front of Instana (not sending directly)

Environment:

• Kubernetes cluster: OpenShift 4.x (IBM Cloud)
• eBPF instrumentation version: v0.2.0
• Image: custom registry (ghcr.io/open-telemetry/opentelemetry-ebpf-instrumentation/ebpf-instrument)
• Mode: DaemonSet (12 nodes)
• Only one node out of the 12 produces the issue

Observed behavior:

On one worker node, the agent repeatedly logs:

time=... level=ERROR msg="error sending trace to consumer" error="sending queue is full"
time=... level=ERROR msg="error sending trace to consumer" error="sending queue is full"
time=... level=ERROR msg="error sending trace to consumer" error="sending queue is full"
time=... level=ERROR msg="error sending trace to consumer" error="sending queue is full"

This happens continuously.
The logs also show very frequent process attachment:

instrumenting process cmd=/myapp/app/bin/tg_loader ...
instrumenting process cmd=/myapp/app/bin/tg_daemon ...
instrumenting process cmd=/usr/bin/coreutils ...
instrumenting process cmd=/usr/bin/bash ...

Even after tightening the configuration (namespace restrictions, exe_path exclusions, etc.), the issue persists on this node only.

Troubleshooting done:

• Restricted discovery.instrument to only the binaries of interest (/myapp/app/bin/*)
• Excluded noisy executables (bash, coreutils, conmon)
• Tried with and without OpenTelemetry Collector
• Added sampling on the collector side
• Enabled (or prepared to enable) k8s-cache component
• Verified node load (CPU/Memory OK)
• Verified that the OTLP receiver is not lagging

Questions:

1. Is the internal eBPF queue size configurable?
2. Is this behavior expected under high syscall churn?
3. Could the process attach/detach loop cause queue saturation?
4. Is this known / documented behavior in v0.2.0?
5. Should the sampling or filtering also apply before the queue?
6. Any recommended mitigations for a single-node hotspot?

Thanks!

[grcevski]

Hi @merouaneagar, we are working to silence that error, it's not an actual error, the queue can fill up if your collector cannot keep up with the number of traces sent and the exporter will retry. Picking a smaller batch will have you see that error less. The default queue size if 4K and you can make it smaller by setting OTEL_EBPF_OTLP_TRACES_MAX_QUEUE_SIZE=1024 or less.

[merouaneagar]

Hi @grcevski
Thank you for the support.
My issue has now been resolved, and everything is working as expected.
You can go ahead and close this ticket.

Really appreciate the help and the work you are doing on this project!

[marctc]

Let me reopen it because since it's not a real issue it's a valid complain that has to be addressed, and this is a good issue for tracking purpouses.

[nobbs]

Ran into the same problem after upgrading Beyla from v2.7.2 to v2.7.5 - I did not check the versions in between. I noticed not only a surge of the "sending queue is full" errors on Beyla, but also a significant drop in traces ingested to Tempo, see the screenshots for the last 5 hours.

Tried to fix it by setting OTEL_EBPF_OTLP_TRACES_MAX_QUEUE_SIZE to 1024, which didn't help. 512 solved it for now.

Not sure what caused the issue to start, as it was not present with v2.7.2 of Beyla and before.

[marctc]

@nobbs thanks for the info. This started to happen after merging this PR #692, which was fixing other stuff but was also enabling batching by default