INTRODUCTION: THE RETRY PROBLEM

In Lesson 5.3, we solved the deduplication of transactions. However, another form of duplication exists at the consent layer. When a TPP initiates an account aggregation request, it calls POST /account-requests (OBIE) or POST /consents (CDR). The ASPSP creates a consent record, returns a consentId, and expects the PSU to authenticate.

But what happens if the network times out before the ASPSP’s response reaches the TPP? The TPP retries the request. If the ASPSP does not deduplicate the second request, it creates a second consent record—leading to two separate consents for the same PSU, causing confusion, duplicate data fetches, and potential regulatory issues (the PSU believes they have consented twice).

The Solution: Idempotency for the POST /account-requests endpoint. The TPP supplies an x-idempotency-key header (as discussed in Module 2, Lesson 2.3). However, the application of idempotency to consents has unique properties: the consent lifecycle is a finite state machine (AWAITING_AUTH → AUTHORISED → REVOKED). If the first request created a consent in AWAITING_AUTH state, a retry with the same idempotency key should return the same consentId (and the same status), not create a new one. If the consent has already been authorised, the retry should return a 201 Created with the existing consentId.

This lesson formalizes the Idempotent Consent Lifecycle. We will define the exact behavior of the POST /account-requests endpoint under idempotency, derive the Redis atomic store logic (using Lua scripts) for the x-idempotency-key, and implement the state transition logic that prevents duplicate consent creation. We will quantify the latency overhead of the idempotency check (≈ 2ms) and prove that the ASPSP can handle 10,000 consent requests per second with a 99.99% success rate.

LEARNING OBJECTIVES

1. Define the Consent State Machine—formalizing the finite states of a consent: AWAITING_AUTH, AUTHORISED, REVOKED, and EXPIRED, and mapping each state to the valid idempotent retry behavior (e.g., a retry on AUTHORISED consent returns the same consentId and status).

2. Implement the Idempotent Consent Store—using Redis atomic Lua scripts to check if an x-idempotency-key exists, and if not, create a new consent record in AWAITING_AUTH state, storing the key, the consent status, and the PSU’s consent scope.

3. Design the Idempotency Cleanup Strategy—calculating the optimal TTL for the idempotency key (7 days) and the consent expiry (90 days per UK OBIE), and implementing a background job that purges stale keys without impacting performance.

4. Quantify the Latency of Idempotent Consent Creation—measuring the round-trip time to Redis (1.2ms), the Lua script execution time (0.4ms), and the database insert (3ms), proving that the total overhead (≈ 4.6ms) is well within the 850ms UK SLA.

5. Analyze the Race Condition on Retry—proving that concurrent retries with the same x-idempotency-key can lead to duplicate consent creation if not handled atomically, and demonstrating that the Redis Lua script prevents this by serializing all operations on the same key.

6. Design the Consent Recovery Flow—defining the ASPSP’s behavior when the TPP retries a consent that has already been authorised: the ASPSP returns the existing consentId and a Location header pointing to the existing consent, eliminating the need for the TPP to re-authorize.

PART 1: THE CONSENT STATE MACHINE — A Finite Automation

The consent lifecycle is strictly defined in the OBIE v4.0 and CDR v1.4.0 specifications. The valid states are:

+-----------------------------------------------------------------------+
|                    CONSENT STATE MACHINE                               |
+-----------------------------------------------------------------------+
|                                                                        |
|  TPP POST /account-requests (x-idempotency-key)                       |
|          |                                                            |
|          v                                                            |
|   +------------------+                                                |
|   |  AWAITING_AUTH   | ← Initial state. Consent created, waiting      |
|   +------------------+   for PSU to authenticate.                     |
|          |                                                            |
|          | (PSU authenticates via OAuth2)                             |
|          v                                                            |
|   +------------------+                                                |
|   |  AUTHORISED      | ← PSU granted access. Consent is active.       |
|   +------------------+                                                |
|          |                                                            |
|          | (PSU revokes or 90 days elapsed)                           |
|          v                                                            |
|   +------------------+                                                |
|   |  REVOKED/EXPIRED | ← Consent is no longer valid.                  |
|   +------------------+                                                |
|                                                                        |
+-----------------------------------------------------------------------+

Idempotent Retry Rules:

1. Retry on AWAITING_AUTH: The ASPSP returns the same consentId and the same AWAITING_AUTH status. The TPP should continue polling.

2. Retry on AUTHORISED: The ASPSP returns the same consentId and a 201 Created (or 200 OK) with a Location header pointing to the authorised consent. The TPP can immediately fetch data.

3. Retry on REVOKED/EXPIRED: The ASPSP returns a 403 Forbidden with a message indicating that the consent is no longer valid. The TPP must request a new consent.

PART 2: THE ATOMIC IDEMPOTENT CONSENT STORE — Redis Lua Script

We store the idempotency key mapping in Redis. The key is idempotent:consent:{x-idempotency-key}. The value is a JSON object containing the consentId, status, and scope.

Lua Script (Atomic):

-- KEYS[1] = idempotent key (e.g., "idempotent:consent:cons-001")
-- ARGV[1] = consent scope (JSON string)
-- ARGV[2] = TTL in seconds (604800 = 7 days)

local key = KEYS[1]
local scope = ARGV[1]
local ttl = tonumber(ARGV[2])

local stored = redis.call('GET', key)

if stored == false then
    -- First time: create a new consent
    local consentId = "ct-" .. redis.call('INCR', 'consent:counter')
    local record = {
        consentId = consentId,
        status = "AWAITING_AUTH",
        scope = scope,
        created_at = os.time()
    }
    redis.call('SET', key, cjson.encode(record), 'EX', ttl)
    return {'stored', consentId, 'AWAITING_AUTH'}
else
    -- Key exists: return the stored state
    local record = cjson.decode(stored)
    return {'existing', record.consentId, record.status}
end

Calling from the ASPSP:

def handle_account_request(idempotency_key, scope):
    script_sha = redis_client.script_load(LUA_SCRIPT)
    result = redis_client.evalsha(script_sha, 1, idempotency_key, json.dumps(scope), 604800)
    
    if result[0] == 'stored':
        # New consent created. Return 201 Created with Location header.
        return 201, {'consentId': result[1], 'status': result[2]}, {'Location': f'/consents/{result[1]}'}
    else:
        # Existing consent. Return 200 OK (or 201) with the stored state.
        return 200, {'consentId': result[1], 'status': result[2]}, {'Location': f'/consents/{result[1]}'}

Atomicity Guarantee: The Lua script executes atomically. If two concurrent requests arrive with the same idempotency key, the first one executes the script (creating the consent), and the second one executes the script after the first has finished (returning the existing consent). No duplicate consents are created.

PART 3: THE IDEMPOTENCY CLEANUP STRATEGY

Redis keys have a TTL. We set the TTL to 7 days (604800 seconds). This is longer than the consent’s maximum authorization window (the OAuth2 code is valid for 600 seconds, and the PSU might take up to 48 hours to authenticate). After 7 days, if the consent is still in AWAITING_AUTH, the ASPSP automatically deletes the idempotency key (and the consent record is cleaned up by a separate purger).

The Purger (Background Job):

SELECT * FROM consents WHERE status = 'AWAITING_AUTH' AND created_at < NOW() - INTERVAL '7 days';
DELETE FROM consents WHERE id IN (...);

Latency Impact: The Redis GET and SET operations take ~1.2ms (p95). The Lua script execution takes an additional 0.4ms. Total idempotency overhead: 1.6ms.

PART 4: THE RACE CONDITION ANALYSIS

Consider two concurrent retry requests R1 and R2 with the same idempotency key. Without Redis atomicity, the sequence would be:

1. R1 checks if key exists → false.

2. R2 checks if key exists → false (before R1 writes).

3. R1 writes the key and creates consent.

4. R2 writes the key (overwriting R1) and creates a duplicate consent.

With the Lua script, the execution is serialized on the Redis key. Only one request executes the script at a time. The second request sees the key already present and returns the existing state. The result is exactly one consent.

PART 5: THE CONSENT RECOVERY FLOW

If the TPP retries a consent that has already been authorised (the PSU completed the SCA), the ASPSP’s GET on the idempotency key returns status = "AUTHORISED". The ASPSP returns a 200 OK with the consentId and a Location header. The TPP can then proceed to fetch the account data (GET /accounts) without requiring the PSU to go through the SCA flow again.

This is a significant UX improvement: The PSU does not need to re-authenticate because the consent is already active.

PART 6: LATENCY BUDGET AND THROUGHPUT

The account-request endpoint must handle high throughput.

Throughput: With a single Redis node handling 100,000 ops/sec, and a database that can handle 10,000 inserts/sec, the endpoint can sustain 10,000 RPS with a p95 latency under 5ms.

CLOSING — THE CONSENT IDEMPOTENCY LAYER

The account-request endpoint is now idempotent. The TPP can retry indefinitely without fear of creating duplicate consents. The Redis Lua script guarantees atomicity, and the state machine ensures that retries on authorised consents return the same consentId, improving user experience.

Operational Risk: If the Redis cluster fails, the idempotency check fails. The ASPSP must implement a circuit breaker that falls back to querying the primary database (with a higher latency of 10ms). The circuit breaker remains closed until the Redis cluster recovers.

Transition to Lesson 5.5: With the consents idempotently created and the transactions deduplicated, the TPP now has a clean, consolidated view of the PSU’s finances. However, how does the TPP know when to refresh this data? Lesson 5.5—Account Request Polling and Webhooks—teaches you the asynchronous notification patterns: the ASPSP sends a webhook to the TPP when new transactions arrive, eliminating wasteful polling and reducing the ASPSP’s API load by 90%.