Add mTLS client authentication for OTLP/OTAP receivers #1539

lalitb · 2025-12-05T23:42:56Z

Adds mutual TLS (mTLS) client certificate authentication to OTLP/OTAP receivers with parallel handshake processing and timeout protection.

What's New

1. mTLS Client Certificate Verification

Client CA loading from file, PEM, or system pool
Standards-compliant X.509 validation via WebPkiClientVerifier
Mandatory authentication when CA is configured

2. Parallel Handshake Processing

.buffer_unordered(MAX_CONCURRENT_HANDSHAKES)  // 64 concurrent handshakes
Prevents head-of-line blocking from slow/malicious clients.

3. Handshake timeout

  tokio::time::timeout(handshake_timeout, acceptor.accept(conn))
  Default 10s, configurable. DoS protection against slowloris-style attacks.

Configuration

Enable mTLS

  receivers:
    otlp:
      tls:
        cert_file: /path/to/server.crt
        key_file: /path/to/server.key
        client_ca_file: /path/to/client-ca.crt  # NEW
        handshake_timeout: "10s"  # NEW (default: 10s)

System CA Pool

  receivers:
    otlp:
      tls:
        include_system_ca_certs_pool: true  # NEW

What's Deferred

Client CA certificate reload (separate PR)
Client-side TLS for exporters (separate PR)
TLS version/cipher configuration

This PR adds mutual TLS (mTLS) support to the OTLP and OTAP gRPC receivers, enabling client certificate authentication for enhanced security. Features: - Client CA certificate configuration (client_ca_file, client_ca_pem) - Optional system CA certificate pool inclusion - Configurable handshake timeout for DoS protection - Concurrent TLS handshake support (up to 64 parallel handshakes) - Certificate hot-reloading with configurable reload interval Configuration options: - tls.cert_file/cert_pem: Server certificate - tls.key_file/key_pem: Server private key - tls.client_ca_file/client_ca_pem: CA for client cert verification (enables mTLS) - tls.include_system_ca_certs_pool: Include system CAs for client verification - tls.handshake_timeout: Timeout for TLS handshake (default: 10s) - tls.reload_interval: Interval for certificate reload checks (default: 5m) Tests added: - mTLS client certificate verification tests - Handshake timeout enforcement test - Concurrent handshake non-blocking test - TLS reload integration test fixes

…ceivers-mtls

codecov · 2025-12-05T23:45:30Z

Codecov Report

❌ Patch coverage is 78.03030% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.47%. Comparing base (dfca971) to head (69437a9).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1539      +/-   ##
==========================================
- Coverage   83.50%   83.47%   -0.04%     
==========================================
  Files         428      428              
  Lines      119410   119479      +69     
==========================================
+ Hits        99713    99733      +20     
- Misses      19163    19212      +49     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`84.64% <78.03%> (-0.06%)`	⬇️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.26% <ø> (ø)`
syslog_cef_receivers	`∅ <ø> (∅)`
otel-arrow-go	`53.50% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lalitb · 2025-12-05T23:46:22Z

rust/otap-dataflow/crates/otap/Cargo.toml

 arc-swap = { workspace = true, optional = true }
 rustls = { workspace = true, optional = true }
-rustls-pemfile = { workspace = true, optional = true }
+rustls-pki-types = { workspace = true, optional = true }


rustls-pemfile is no longer maintained, so replaced by rustls-pki-types, which is maintained by rustls project.

https://github.com/open-telemetry/otel-arrow/pull/1536/files can be reverted with this.

yes, we can either revert that as part of this PR, or I can do it in a followup after this is merged. Thanks for updating the unmaintained dependency @lalitb!

lalitb · 2025-12-05T23:52:39Z

rust/otap-dataflow/crates/otap/src/otap_receiver.rs

                match maybe_tls_acceptor {
                    Some(tls_acceptor) => {
-                        let tls_stream = create_tls_stream(listener_stream, tls_acceptor);
+                        let tls_stream = create_tls_stream(listener_stream, tls_acceptor, handshake_timeout);


handshake_timeout is added to protect against slow/malicious clients that could hold connection slots indefinitely by not completing the TLS handshake.

Without a timeout, a malicious client could:

Open a TCP connection

Never send TLS ClientHello (or send it very slowly)

Hold the connection slot forever, preventing legitimate clients from connecting

Repeat this to exhaust server resources (DoS attack)

lalitb · 2025-12-05T23:54:55Z

rust/otap-dataflow/crates/otap/src/tls_utils.rs

+/// - Allows concurrent handshakes to prevent slow clients from blocking others
+/// - Limits memory overhead for pending handshake state
+/// - May need adjustment based on actual workload characteristics
+const MAX_CONCURRENT_HANDSHAKES: usize = 64;


Not mTLS-specific - this is a general TLS hardening to prevent DoS via connection flooding. Limits concurrent handshakes to 64, applying backpressure when exceeded. Without this limit, an attacker could overwhelm the server by initiating thousands of handshakes simultaneously.

albertlockett · 2025-12-08T17:11:00Z

rust/otap-dataflow/crates/otap/src/tls_utils.rs

+        // If system CAs were requested but no certs were loaded (and no user CA provided),
+        // warn the user that mTLS is effectively disabled or might not work as expected.
+        // Note: client_ca_pem is empty here.
+        if config.include_system_ca_certs_pool == Some(true) {
+            log::warn!(
+                "include_system_ca_certs_pool is true, but no CA certificates were loaded. mTLS will be disabled."
+            );
+        }
+        builder.with_no_client_auth()


Would it make sense to return an error here? It seems like it would be better from a security perspective not to fail open like this

albertlockett · 2025-12-08T17:13:18Z

rust/otap-dataflow/crates/otap/src/tls_utils.rs

+                                // TLS handshake failed - log and continue
+                                log::debug!("TLS handshake failed: {}", e);
+                                None


Do you think this is important enough to log at warn level? Or would that just create too much noise in the event that there were many failed handshakes?

albertlockett · 2025-12-08T17:23:51Z

rust/otap-dataflow/crates/otap/tests/mtls_tests.rs

+    if skip_if_no_openssl() {
+        return;
+    }


These tests look great!

Could we avoid skipping them if we used something like rcgen instead of openssl? https://docs.rs/rcgen/latest/rcgen/

I'm thinking it'd nice to ensure that these tests could always run. That way we can rely on them to catch regressions w/out worrying about if openssl accidentally got removed from some test runner.

If this is feasible, we can make the change a future PR (I don't want to block this PR from getting merged for this)

jmacd · 2025-12-08T21:28:06Z

rust/otap-dataflow/crates/otap/src/tls_utils.rs

+        // Allow concurrent handshakes to prevent slow/malicious clients from blocking others
+        .buffer_unordered(MAX_CONCURRENT_HANDSHAKES)


Cool, how this works.

lalitb added 3 commits December 5, 2025 15:14

Merge branch 'receivers-mtls' of github.com:lalitb/otel-arrow into re…

2265518

…ceivers-mtls

lalitb requested a review from a team as a code owner December 5, 2025 23:42

github-project-automation bot added this to OTel-Arrow Dec 5, 2025

github-actions bot added the rust Pull requests that update Rust code label Dec 5, 2025

Merge branch 'main' into receivers-mtls

db38f89

lalitb commented Dec 5, 2025

View reviewed changes

Merge branch 'main' into receivers-mtls

69437a9

albertlockett reviewed Dec 8, 2025

View reviewed changes

jmacd approved these changes Dec 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add mTLS client authentication for OTLP/OTAP receivers #1539

Add mTLS client authentication for OTLP/OTAP receivers #1539

lalitb commented Dec 5, 2025

Uh oh!

codecov bot commented Dec 5, 2025 •

edited

Loading

Uh oh!

lalitb Dec 5, 2025

Uh oh!

cijothomas Dec 6, 2025

Uh oh!

albertlockett Dec 8, 2025

Uh oh!

lalitb Dec 5, 2025

Uh oh!

lalitb Dec 5, 2025

Uh oh!

albertlockett Dec 8, 2025

Uh oh!

albertlockett Dec 8, 2025

Uh oh!

albertlockett Dec 8, 2025

Uh oh!

jmacd Dec 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Allow concurrent handshakes to prevent slow/malicious clients from blocking others
		.buffer_unordered(MAX_CONCURRENT_HANDSHAKES)

Add mTLS client authentication for OTLP/OTAP receivers #1539

Are you sure you want to change the base?

Add mTLS client authentication for OTLP/OTAP receivers #1539

Conversation

lalitb commented Dec 5, 2025

What's New

1. mTLS Client Certificate Verification

2. Parallel Handshake Processing

3. Handshake timeout

Configuration

Enable mTLS

System CA Pool

What's Deferred

Uh oh!

codecov bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Dec 5, 2025 •

edited

Loading