INTRODUCTION: THE MILLISECOND RACE

In Lessons 7.1–7.3, we built the API Gateway, the rate limiter, the circuit breakers, and the retry policies. The total gateway latency is 4.38ms (Kong) or 19.38ms (AWS). However, we can do better. The Open Banking SLA (850ms p95 in the UK) is generous, but the TPP’s user experience depends on low latency. Reducing the API Gateway latency from 4.38ms to 2.5ms is a 43% improvement, which translates to a faster, more responsive TPP application.

This lesson focuses on latency optimisation at the connection layer. We configure connection pooling to the upstream microservices, reusing TCP/TLS connections to avoid the overhead of establishing new connections for each request. We enable keep-alive for upstream connections, reducing the TLS handshake overhead from 5ms to 0ms (for subsequent requests). We enable HTTP/2 on the API Gateway, which multiplexes multiple requests over a single connection, eliminating head-of-line blocking and reducing latency. We also tune the TCP parameters (Nagle’s algorithm, TCP_NODELAY) to reduce latency.

We derive the optimal connection pool size using Little’s Law: Concurrency = RPS × Latency, and prove that a pool size of 50 connections is sufficient for 1,000 RPS. We also derive the optimal keep-alive timeout (60 seconds) using the exponential distribution of request inter-arrival times, balancing the overhead of keeping connections alive against the cost of establishing new connections.

LEARNING OBJECTIVES

  1. Design Connection Pooling to Upstream Services—implementing connection pooling for downstream microservices, using max_connections = 50 (as derived from Little’s Law), and proving that the pool reduces the average request latency by 2ms.

  2. Enable Keep-Alive on Upstream Connections—configuring the keep-alive timeout (keepalive_timeout = 60s) and the keep-alive requests (keepalive_requests = 1000), and proving that keep-alive reduces the TLS handshake overhead from 5ms to 0ms for subsequent requests.

  3. Enable HTTP/2 on the API Gateway—configuring HTTP/2 for the API Gateway (multiplexing, server push), and proving that HTTP/2 reduces the p95 latency by 5-10% for typical Open Banking payloads.

  4. Tune TCP Parameters—disabling Nagle’s algorithm (tcp_nodelay = on), and tuning the TCP buffer sizes (tcp_rmem, tcp_wmem) to reduce latency and improve throughput.

  5. Quantify the Optimised Gateway Latency—summing the optimised latencies (TLS termination: 2ms, keep-alive: 0ms, routing: 0.05ms, plugins: 2.33ms), and proving that the total p95 latency is under 4.38ms (pre-optimisation) and 2.5ms (post-optimisation).

  6. Derive the Optimal Keep-Alive Timeout—using the exponential distribution of request inter-arrival times to set the keep-alive timeout, balancing the overhead of keeping connections alive against the cost of establishing new connections.

PART 1: CONNECTION POOLING — Little’s Law and the Optimal Pool Size

The API Gateway connects to downstream microservices (Payment Service, Consent Service, Balance Service). Establishing a TCP/TLS connection for each request adds 5ms of latency (TLS handshake). Connection pooling reuses connections, eliminating the handshake for subsequent requests.

Little’s Law:

Concurrency = RPS × Latency

For a downstream service:

  • RPS = 1000 requests per second (peak).

  • Latency = 50ms (p95).

  • Concurrency = 1000 × 0.05 = 50 connections.

The Optimal Pool Size:

max_connections = ceil(Concurrency) = 50 per downstream service.

Latency Reduction:

  • First request (cold start) : 5ms (TLS handshake + TCP setup).

  • Subsequent requests (warm) : 0ms (connection reused).

  • Average reduction: 2ms (assuming 60% of requests are subsequent).

Kong Configuration:

text
proxy_http_version 1.1
proxy_set_header Connection ""

NGINX Configuration:

text
upstream payment_backend {
    server payment-service:8080;
    keepalive 50;
}

PART 2: KEEP-ALIVE ON UPSTREAM CONNECTIONS

Keep-alive allows the API Gateway to reuse the TCP/TLS connection for multiple requests. The connection remains open for a specified timeout.

The Keep-Alive Timeout:

We set the keep-alive timeout to 60 seconds. If no request is sent on the connection for 60 seconds, the connection is closed.

The Keep-Alive Requests:

We set keepalive_requests = 1000. After 1,000 requests, the connection is closed and a new one is established.

Latency Reduction:

 
 
Scenario Latency Explanation
Without Keep-Alive 5ms New TLS handshake for each request.
With Keep-Alive (first request) 5ms Handshake still required.
With Keep-Alive (subsequent) 0ms Connection reused.
Average (60% reuse) 2ms 5ms × 0.4 + 0ms × 0.6 = 2ms.

NGINX Configuration:

text
keepalive_timeout 60s;
keepalive_requests 1000;

Kong Configuration:

text
proxy_http_version 1.1
proxy_set_header Connection ""

PART 3: HTTP/2 — Multiplexing and Server Push

HTTP/2 multiplexes multiple requests over a single connection, eliminating head-of-line blocking (where a slow request blocks subsequent requests on the same connection).

HTTP/2 Features:

 
 
Feature Benefit Latency Reduction
Multiplexing Multiple requests on one connection. 5-10%
Server Push ASPSP pushes resources (e.g., OpenAPI doc) before requested. N/A
Header Compression HPACK compression reduces header size. N/A

Latency Reduction:

For typical Open Banking payloads (JSON, < 10 KB), HTTP/2 reduces the p95 latency by 5-10%.

NGINX Configuration:

text
listen 443 ssl http2;

Kong Configuration:

text
http2_ssl_port 443;

PART 4: TCP PARAMETER TUNING — Nagle’s Algorithm and Buffer Sizes

TCP parameters can significantly impact latency.

Nagle’s Algorithm:

Nagle’s algorithm delays sending small packets to improve network efficiency. This adds 40ms of latency for small payloads (e.g., JSON requests). We disable it:

text
tcp_nodelay on;

TCP Buffer Sizes:

We increase the TCP buffer sizes to improve throughput:

text
tcp_rmem 4096 87380 6291456;
tcp_wmem 4096 16384 6291456;

PART 5: THE OPTIMISED LATENCY BUDGET

 
 
Component Pre-Optimisation (ms) Post-Optimisation (ms) Explanation
TLS Termination 2.0 2.0 Unchanged
Routing (Trie) 0.05 0.05 Unchanged
Plugin Pipeline 2.33 2.33 Unchanged
Upstream TLS Handshake 2.0 0.0 (reused) Keep-alive eliminates handshake.
HTTP/2 Multiplexing 0.0 -0.5 5-10% latency reduction.
TCP Overhead 0.0 -0.5 Nagle disabled, buffers tuned.
Total (p95) 4.38ms 2.38ms  

Conclusion: The optimised API Gateway delivers a p95 latency of 2.38ms, a 46% improvement over the baseline.

PART 6: THE OPTIMAL KEEP-ALIVE TIMEOUT — The Exponential Distribution

We set the keep-alive timeout to 60 seconds. If the inter-arrival time of requests is exponentially distributed with mean 1 / λ, the probability that a connection remains idle for more than T seconds is e^(-λT).

Example:

  • λ = 1 / 10 (one request every 10 seconds).

  • T = 60 seconds.

  • P(idle > 60) = e^(-60/10) = e^(-6) = 0.00248 (0.25% of connections are idle for > 60 seconds).

  • The overhead of keeping a connection alive for 60 seconds is negligible compared to the cost of establishing a new connection.

CLOSING — THE MILLISECOND MASTERY

Latency optimisation is a continuous process. By tuning NGINX workers, enabling HTTP/2, implementing connection pooling, and disabling Nagle’s algorithm, the certified practitioner can reduce the API Gateway latency to under 2.5ms. This frees up valuable time for the business logic (e.g., consent validation, payment processing), ensuring the end-to-end SLA is met.

Key Takeaways:

  • Connection Pooling: max_connections = 50 (derived from Little’s Law).

  • Keep-Alive Timeout: 60 seconds.

  • HTTP/2: 5-10% latency reduction.

  • TCP_NODELAY: Disable Nagle’s algorithm.

  • Optimised Latency: 2.38ms (p95).

Transition to Lesson 7.5: With the gateway optimised, we now turn to DDoS Protection, WAF (Web Application Firewall), and Edge Security—how to protect the Open Banking API from volumetric attacks (DDoS) and application-layer attacks (SQL injection, XSS), using the AWS WAF, CloudFlare, and rate-limiting at the edge. We will formalise the WAF rule algebra and quantify the false positive rate.

 
 
PreviousNext