.NET Env. Variable Conflict

[Miqueasher]

Bug Report

Symptom

Describe the bug
"OpenTelemetry .NET has complex environment variable precedence rules where OTEL_EXPORTER_OTLP_ENDPOINT overrides signal-specific endpoints, creating an either/or scenario for example:

Traces Work, Custom Metrics Fail:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4316 # Traces to port 4316
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:4318/v1/metrics # Ignored

Custom Metrics Work, Traces Fail:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # Metrics to port 4318
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4316/v1/traces # Ignored"

Expected behavior
OTEL_EXPORTER_OTLP_ENDPOINT is expected to be a baseline/default open port like "http://localhost:4316"
while specified ports take precedence like:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:4318/v1/metrics
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4316/v1/traces

Screenshots
If applicable, add screenshots to help explain your problem.
Image 1 shows the exporter attemption to export to port 4318/v1/traces when specified trace port shows 4316. The custom metrics were being exported properly here.

Image 2 shows the agent based metrics being refused due to meter not being subscribed to the provider. The Traces were being sent here

Runtime environment (please complete the following information):

• OpenTelemetry Automatic Instrumentation version: 1.7.0
• OS: Amazon Linux 2 (x86_64)
• .NET version: 8.0

Additional context
Please note that when the image of the logs were taken the only changes to anything regarding the program were the endpoint variables in the main.tf (terraform) file.
When OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4316 is set traces pass, custom metrics fail.
When OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 is set traces fail, custom metrics pass.
When using separate endpoints OTEL_EXPORTER_OTLP_TRACES_ENDPOINT|OTEL_EXPORTER_OTLP_METRICS_ENDPOINT Metrics endpoint takes precedence and fails traces. custom metrics pass.
when using separate endpoints with OTEL_EXPORTER_OTLP_ENDPOINT assigned as well OTEL_EXPORTER_OTLP_ENDPOINT will take precedence and pass whatever is specified according to it's endpoint.
for example:
OTEL_EXPORTER_OTLP_ENDPOINT= 4316 Traces pass; Metrics fail.
OTEL_EXPORTER_OTLP_ENDPOINT= 4318 Metrics pass; Traces fail.

Also it is important to note that the dual setup has been implemented and works properly in the other 3 languages it's used in (Python, Java, Node.js)

Reproduce

Steps to reproduce the behavior:

1. Use dotnet sample app found here: https://github.com/aws-observability/aws-application-signals-test-framework/tree/main/sample-apps . The terraform files can be found here: https://github.com/aws-observability/aws-application-signals-test-framework/tree/main/terraform/dotnet/ec2/default
2. Add custom metrics using global agent according to this doc: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AppSignals-CustomMetrics.html#AppSignals-CustomMetrics-OpenTelemetry
3. Set env variables as needed to export custom metrics to port 4318 alongside original app-signals telemetry.
   Only traces and custom metrics will be effected.

_{**Tip**: [React](https://github.blog/news-insights/product-news/add-reactions-to-pull-requests-issues-and-comments/) with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding `+1` or `me too`, to help us triage it. Learn more [here](https://opentelemetry.io/community/end-user/issue-participation/).}

[nrcventura]

I believe that this was addressed in version 1.13.0. #4593 and its associated PR resulted in a change to no longer delegate environment variable configuration to the SDK implementation for the OTLP exporter.

[Miqueasher]

If I'm understanding correctly the fix was for the grpc protocol that the SDK defaulted too. I don't believe that fix applies to my issue since I'm using http/protobuf as the only method. My issue only has to do with a bug inside of the endpoint variables themselves. I also tested it with the 1.13.0 version and was still running into the same issues.

[nrcventura]

The fix addresses the mixing and matching of the signal specific and general environment variables for the exporter. Previously auto instrumentation deferred most of the OTLP configuration the underlying SDK. Some of the environment variables were checked in auto instrumentation in order to enforce a different default OTLP protocol than the SDK, but the majority of the configuration was handled by the SDK code. The underlying SDK does not yet handle the mixing and matching of the signal specific and general environments. The PR for that fix changes the behavior so that auto instrumentation no longer relies on the SDK to handle the configuration logic.

I suggest testing your situation with version 1.13.0.