INTRODUCTION: THE BANDWIDTH BOTTLENECK

In Lessons 7.1 through 7.6, we built a high-performance, secure, resilient, and aggressively cached API Gateway. We reduced the p95 latency to 2.38ms, implemented multi-layer DDoS protection, and achieved a cache hit ratio of 88% with Redis, delivering an average latency of 2.66ms. However, the API Gateway is still transmitting JSON payloads over the network. A typical Open Banking response—an account with 100 transactions—can be 30-50 KB of JSON. At 10,000 RPS, this translates to 300-500 MB of data per second. This is a significant bandwidth cost, and it adds to the end-to-end latency (the time to transmit the payload over the network).

The problem is two-fold:

1. Bandwidth Cost: Transmitting large JSON payloads over the internet costs money (data transfer fees from cloud providers) and consumes network bandwidth, which can become a bottleneck.

2. Serialization/Deserialization Cost: Parsing JSON on the client (TPP) and server (ASPSP) is CPU-intensive. A 50 KB JSON payload takes ~2ms to parse on a modern CPU. At 10,000 RPS, this is 20 seconds of CPU time per second—a significant overhead.

The solution is payload optimisation. We implement three complementary strategies:

1. Gzip Compression: Compressing the JSON payload at the ASPSP and decompressing it at the TPP. Gzip typically achieves a 70-80% compression ratio for JSON, reducing a 50 KB payload to 10-15 KB.

2. Field Minimisation: Removing unnecessary whitespace, shortening field names (e.g., AccountId → aid), and omitting null fields. This reduces the raw JSON size by 20-30%.

3. Binary Protocols (Protobuf): Replacing JSON with Protocol Buffers (Protobuf), a binary serialization format developed by Google. Protobuf is 3-10x smaller than JSON and 3-5x faster to serialize/deserialize.

This lesson deconstructs the math and engineering of payload optimisation. We derive the compression ratio of Gzip on JSON (typically 70-80%), and we quantify the latency impact of compression (5ms for compression at the ASPSP, 3ms for decompression at the TPP). We compare JSON vs. Protobuf head-to-head: Protobuf reduces the payload size by 80% and the serialisation time by 70%, but it requires a schema (.proto file) and is not human-readable. We derive the break-even point for adopting Protobuf (when the bandwidth savings outweigh the schema management overhead). We also design the content negotiation strategy: the API Gateway checks the Accept-Encoding header (gzip, br) and the Accept header (application/json, application/x-protobuf) to determine the optimal payload format for each TPP. We quantify the total optimisation gain: reducing the average response size from 50 KB to 10 KB (a 5x reduction) and reducing the average latency from 12ms to 7ms.

LEARNING OBJECTIVES

1. Measure the Current JSON Payload Size—analysing the average size of an Open Banking response (Account + 100 Transactions = 35-50 KB), and identifying the largest fields (transaction descriptions, merchant names) as the primary targets for optimisation.

2. Implement Gzip Compression with Content Negotiation—configuring the API Gateway to compress responses with Gzip (or Brotli) when the TPP sends the Accept-Encoding: gzip header, and quantifying the compression ratio (70-80%) and the CPU cost (5ms compression, 3ms decompression).

3. Minimise the JSON Payload—removing unnecessary whitespace (using json.dumps(separators=(',', ':'))), shortening field names (e.g., AccountId → aid), and omitting null fields, reducing the raw JSON size by 20-30%.

4. Compare JSON vs. Protocol Buffers (Protobuf)—implementing a Protobuf schema for the Account, Transaction, and Balance resources, serializing and deserializing the payload, and measuring the payload size (80% smaller) and the serialisation time (70% faster) compared to JSON.

5. Design the Content Negotiation Strategy—defining the Accept header logic: if Accept: application/x-protobuf, return Protobuf; if Accept: application/json, return JSON (with optional Gzip); and if Accept-Encoding: gzip, compress the response.

6. Quantify the Total Payload Reduction—calculating the end-to-end improvement: JSON (50 KB → 35 KB minified → 10 KB gzipped) and Protobuf (10 KB → 2 KB gzipped), and proving that the total optimisation reduces the network transmission time from 50ms to 10ms.

7. Derive the Break-Even Point for Protobuf—calculating the TPP adoption rate required for Protobuf to be cost-effective (schema management overhead vs. bandwidth savings), and proving that > 10% adoption justifies the investment.

PART 1: THE CURRENT PAYLOAD — Measuring the Baseline

A typical Open Banking Account Response (OBIE v4.0) with 10 accounts and 100 transactions has the following structure:

The Largest Fields:

• TransactionInformation (descriptions): 15 KB (accounts for 50% of the transaction size).

• MerchantDetails (names, MCC codes): 5 KB.

• AccountId / TransactionId (UUIDs): 5 KB.

Bandwidth Impact:
At 10,000 RPS, the total data transmitted is 45 KB × 10,000 = 450 MB/sec. This saturates a 4 Gbps network link.

Latency Impact:
Transmitting 45 KB over a 100 Mbps link takes 45 KB × 8 / 100 Mbps = 3.6ms. Over a 1 Gbps link, it takes 0.36ms. The network transmission time is a significant part of the end-to-end latency.

PART 2: GZIP COMPRESSION — The 70% Reduction

Gzip compression is the industry standard for HTTP compression. The API Gateway compresses the response body using the gzip algorithm and sets the Content-Encoding: gzip header.

Compression Ratio:

CPU Cost:

• Compression (ASPSP): 5ms (p95) for a 45 KB payload using gzip at level 6.

• Decompression (TPP): 3ms (p95) for the same payload.

Content Negotiation:
The API Gateway checks the Accept-Encoding header:

Accept-Encoding: gzip, br

If gzip is present, the ASPSP compresses the response. If br (Brotli) is present, the ASPSP uses Brotli (better compression, but slower).

Configuration (NGINX) :

gzip on;
gzip_types application/json;
gzip_comp_level 6;
gzip_min_length 1000;

Configuration (Kong) :

plugins:
  - name: gzip
    config:
      content_types:
        - application/json

Latency Impact:

• Compression adds 5ms to the ASPSP’s processing time.

• Decompression adds 3ms to the TPP’s processing time.

• Network transmission time is reduced from 3.6ms to 0.8ms (10 KB over 100 Mbps).

• Net gain: (3.6 - 0.8) - 5 = -2.2ms? This suggests compression is slower than transmitting the uncompressed payload! However, this calculation is for a 100 Mbps link. On a 1 Gbps link, the transmission time reduction is smaller (0.36ms → 0.08ms), making compression less beneficial.

The Break-Even Point:
Compression is beneficial when the network is the bottleneck (low bandwidth, high latency). For high-bandwidth links (1 Gbps+), compression may add latency. The ASPSP should dynamically choose compression based on the TPP’s network speed (which can be inferred from the Via header or client IP).

PART 3: JSON MINIFICATION — Removing the Fat

Minification removes unnecessary whitespace, shortens field names, and omits null fields.

Techniques:

1. Remove Whitespace: json.dumps(payload, separators=(',', ':')) removes spaces after commas and colons.

2. Shorten Field Names: Map AccountId → aid, TransactionId → tid, Amount → amt, Currency → ccy.

3. Omit Null Fields: Remove fields with null values.

Example (Before):

{
  "AccountId": "acc-123",
  "Amount": {
    "Amount": "100.00",
    "Currency": "GBP"
  }
}

Example (After):

{"aid":"acc-123","amt":{"amt":"100.00","ccy":"GBP"}}

Reduction: 80 bytes → 60 bytes (25% reduction).

Trade-off: Minified JSON is less human-readable. The ASPSP should only minify for production (not for debugging).

PART 4: PROTOCOL BUFFERS (PROTOBUF) — The Binary Revolution

Protocol Buffers (Protobuf) is a binary serialization format developed by Google. It requires a schema (.proto file), but it is much more efficient than JSON.

The Protobuf Schema (Account.proto) :

syntax = "proto3";

message Account {
  string account_id = 1;
  string account_type = 2;
  string currency = 3;
  string nickname = 4;
  Balance balance = 5;
  repeated Transaction transactions = 6;
}

message Balance {
  string amount = 1;
  string currency = 2;
}

message Transaction {
  string transaction_id = 1;
  string amount = 2;
  string currency = 3;
  string credit_debit_indicator = 4;
  string booking_date_time = 5;
  string description = 6;
}

Payload Size Comparison:

Conclusion: Protobuf (without compression) is 4.5x smaller than JSON and 5x faster to serialize. With gzip, Protobuf is 15x smaller than uncompressed JSON and 3x faster to serialize.

Trade-offs:

• Human-Readability: Protobuf is binary (not human-readable). Debugging requires a .proto file.

• Schema Management: The ASPSP must maintain a .proto file for each resource (Account, Transaction, Balance). The TPP must generate client code from the .proto file.

• Backward Compatibility: Protobuf supports optional fields, which allows the ASPSP to add new fields without breaking old clients.

Content Negotiation:
The API Gateway checks the Accept header:

Accept: application/x-protobuf

If present, the ASPSP returns a Protobuf response. Otherwise, it returns JSON.

PART 5: CONTENT NEGOTIATION STRATEGY — The Unified Dispatcher

The API Gateway must support multiple response formats: JSON (uncompressed), JSON (gzipped), and Protobuf (binary). The TPP controls the format via the Accept and Accept-Encoding headers.

The Content Negotiation Logic:

+-----------------------------------------------------------------------+
|           CONTENT NEGOTIATION STRATEGY                                 |
+-----------------------------------------------------------------------+
|                                                                        |
|  Request Headers:                                                      |
|  - Accept: application/json, application/x-protobuf                  |
|  - Accept-Encoding: gzip, br                                         |
|                                                                        |
|  Decision Tree:                                                        |
|                                                                        |
|  1. if (Accept contains "application/x-protobuf"):                   |
|       → Serialize with Protobuf.                                     |
|       → if (Accept-Encoding contains "gzip"):                        |
|           → Compress with gzip.                                      |
|       → else:                                                         |
|           → Return binary Protobuf.                                   |
|                                                                        |
|  2. else if (Accept contains "application/json"):                    |
|       → Serialize with JSON (minified).                               |
|       → if (Accept-Encoding contains "gzip"):                        |
|           → Compress with gzip.                                      |
|       → else:                                                         |
|           → Return JSON.                                              |
|                                                                        |
|  3. else:                                                              |
|       → Return JSON (default).                                       |
|                                                                        |
+-----------------------------------------------------------------------+

Implementation (Python) :

def negotiate_response(data, accept_header, accept_encoding_header):
    content_encoding = None
    content_type = "application/json"

    # Determine content type
    if "application/x-protobuf" in accept_header:
        content_type = "application/x-protobuf"
        payload = serialize_protobuf(data)
    else:
        payload = json.dumps(data, separators=(',', ':')).encode('utf-8')

    # Determine compression
    if "gzip" in accept_encoding_header:
        import gzip
        payload = gzip.compress(payload, compresslevel=6)
        content_encoding = "gzip"

    return payload, content_type, content_encoding

PART 6: THE TOTAL OPTIMISATION GAIN — 5x Reduction in Latency

We quantify the total optimisation gain from JSON minification, Gzip compression, and Protobuf.

Conclusion: Protobuf (without compression) is the fastest format, reducing the total time from 6.1ms (JSON) to 1.3ms, a 78% reduction. Protobuf with gzip is also fast (2.74ms), but adds compression overhead.

Recommendation: Use Protobuf for production APIs (maximises performance). Use JSON for debugging and development (human-readable). Use Gzip as a fallback for JSON.

CLOSING — THE WIRE EFFICIENCY REVOLUTION

Payload optimisation is the final frontier of API performance. By implementing Gzip compression, JSON minification, and Protobuf serialization, the certified practitioner can reduce the payload size by 5x and the total latency by 78%. The content negotiation strategy ensures backward compatibility: TPPs that support Protobuf receive the optimal format; TPPs that only support JSON receive JSON (optionally compressed).

Key Takeaways:

• Gzip: 70-80% compression, 5ms compression cost.

• Minification: 20-30% reduction, no CPU cost.

• Protobuf: 4.5x smaller, 5x faster than JSON.

• Content Negotiation: Accept and Accept-Encoding headers.

• Total Gain: 78% latency reduction.

Transition to Lesson 7.8: With the gateway fully optimised, we now turn to the Module 7 Capstone. Lesson 7.8 synthesises all components of the API Gateway—routing, rate limiting, circuit breakers, caching, DDoS protection, and payload optimisation—into a single, unified performance and compliance framework. We will present the final latency budget, the regulatory evidence bundle for API gateway operations, and the final risk assessment.