Learning Objectives:
-
Understand the data availability problem and its importance
-
Master sharding and its role in scaling
-
Learn about data availability sampling and other solutions
-
Analyze the trade-offs of different data availability approaches
8.5.1: The Data Availability Problem
What is Data Availability?
Data availability is the guarantee that transaction data is available to all network participants. Without data availability, participants cannot verify the state of the blockchain or challenge invalid transactions.
The data availability problem arises when data is stored off-chain or only partially available on-chain. If data is not available, participants cannot independently verify the state of the blockchain.
The data availability problem is particularly relevant for Layer-2 scaling solutions, where transaction data is often stored off-chain.
The Data Availability Problem: ┌─────────────────────────────────────────────────────────────────────┐ │ Data Availability Problem │ │ │ │ On-Chain Data: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Data is available to all nodes │ │ │ │ • Nodes can verify state │ │ │ │ • Nodes can challenge invalid transactions │ │ │ │ • High cost │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ Off-Chain Data: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Data is not available to all nodes │ │ │ │ • Nodes cannot verify state │ │ │ │ • Nodes cannot challenge invalid transactions │ │ │ │ • Low cost │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ The Problem: How to ensure data availability with low cost? │ └─────────────────────────────────────────────────────────────────────┘
Why Data Availability Matters:
Data availability is essential for blockchain security. Without data availability, participants cannot verify the state of the blockchain, and they cannot challenge invalid transactions.
The data availability problem is particularly relevant for Layer-2 scaling solutions. If a malicious sequencer submits an invalid state root but withholds the underlying data, participants cannot verify the state and cannot challenge the sequencer.
Data availability also affects user protection. If data is not available, users may not be able to withdraw their funds from a Layer-2 solution.
Solutions to the Data Availability Problem:
Several solutions have been developed to address the data availability problem. These include sharding, data availability sampling, and data availability committees.
Sharding splits the blockchain into multiple shards, each with its own data. This reduces the storage requirements for each node while maintaining data availability.
Data availability sampling enables light clients to verify that data is available without downloading all of it. This reduces the bandwidth requirements for light clients.
Data availability committees are sets of trusted entities that ensure data availability. The committee stores the data and provides it to validators and users.
8.5.2: Sharding – Complete Guide
What is Sharding?
Sharding is a scaling technique that splits the blockchain into multiple shards, each with its own set of validators and data. Each shard processes its own transactions, and the shards are coordinated by a main chain.
The key insight of sharding is that the work of processing transactions can be distributed across multiple shards. This increases the total throughput of the blockchain while maintaining decentralization.
Sharding is a fundamental part of Ethereum’s scaling roadmap. Ethereum 2.0 will introduce sharding to increase the network’s throughput.
Sharding Architecture: ┌─────────────────────────────────────────────────────────────────────┐ │ Sharding Architecture │ │ │ │ Main Chain: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Coordinates shards │ │ │ │ • Stores shard state roots │ │ │ │ • Manages validator set │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────┼─────────────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ │ │ │ │ │ │ │ │ │ │ • Validators│ │ • Validators│ │ • Validators│ │ │ │ • Data │ │ • Data │ │ • Data │ │ │ │ • State │ │ • State │ │ • State │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
How Sharding Works:
The process of sharding involves several steps. First, the blockchain is split into multiple shards, each with its own set of validators. Validators are assigned to shards through a random process.
Each shard processes its own transactions independently. The shard’s validators validate the transactions and create blocks for the shard.
The shard’s state root is periodically submitted to the main chain. The main chain stores the state roots and ensures the integrity of the shards.
Cross-shard communication is enabled through the main chain. The main chain coordinates messages between shards and ensures that assets can be transferred between shards.
Ethereum Sharding:
Ethereum’s sharding implementation is a key part of its scaling roadmap. The implementation includes multiple phases, each introducing new features.
The initial phase introduces data shards, which provide additional data capacity for rollups. This enables rollups to achieve higher throughput at lower cost.
Later phases introduce execution shards, which enable smart contracts to run on shards. This will significantly increase the throughput of the Ethereum network.
Advantages and Limitations:
Sharding offers several advantages. It can achieve very high throughput, as the work is distributed across multiple shards. It also maintains decentralization, as each shard has its own set of validators.
However, sharding has limitations. It adds complexity to the blockchain, and cross-shard communication can be challenging. It also introduces new security challenges, such as shard takeovers.
Sharding also requires careful design to ensure security. The validator assignment process must be secure, and the cross-shard communication must be robust.
8.5.3: Data Availability Sampling
What is Data Availability Sampling?
Data availability sampling is a technique that enables light clients to verify that data is available without downloading all of it. This reduces the bandwidth requirements for light clients.
The key insight of data availability sampling is that light clients can sample small portions of the data and use probabilistic reasoning to verify the availability of the entire data.
Data availability sampling is a key component of Ethereum’s sharding implementation. It enables light clients to verify the availability of shard data without downloading all of it.
Data Availability Sampling: ┌─────────────────────────────────────────────────────────────────────┐ │ Data Availability Sampling │ │ │ │ Data: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ [D1][D2][D3][D4][D5][D6][D7][D8][D9][D10] │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ Sampling: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Light client samples random portions │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ Sample 1: D3 │ │ │ │ │ │ Sample 2: D7 │ │ │ │ │ │ Sample 3: D2 │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ Probabilistic Verification: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ If all samples are available: │ │ │ │ • High probability that data is available │ │ │ │ • Example: 10 samples → 99.9% confidence │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
How Data Availability Sampling Works:
The process of data availability sampling involves several steps. The data is split into small chunks, and the chunks are erasure coded. This ensures that the data can be reconstructed even if some chunks are missing.
Light clients randomly sample chunks from the data. If the chunks are available, the client assumes that the data is available. The more samples the client takes, the higher the confidence that the data is available.
If a client detects that some chunks are missing, they can issue a data availability challenge. This prompts the network to verify the availability of the data.
Erasure Coding:
Erasure coding is a technique that enables the reconstruction of missing data from a subset of available data. It is used in data availability sampling to ensure that data can be reconstructed even if some chunks are missing.
The key insight of erasure coding is that redundant information can be added to the data, enabling reconstruction from partial data. This is similar to RAID systems in traditional storage.
Erasure coding ensures that data availability can be verified even if some chunks are missing. It also enables light clients to verify data availability without downloading all of the data.
8.5.4: Proto-Danksharding and EIP-4844
What is Proto-Danksharding?
Proto-Danksharding is a proposed upgrade to Ethereum that introduces a new type of transaction called a “blob transaction.” Blob transactions contain additional data that is not accessible to the Ethereum Virtual Machine (EVM) but is stored on-chain for a limited time.
The key insight of Proto-Danksharding is that rollups need a place to store transaction data. By providing a dedicated data storage layer, Proto-Danksharding enables rollups to achieve higher throughput at lower cost.
Proto-Danksharding is a stepping stone to full Danksharding, which will introduce full sharding to Ethereum.
EIP-4844:
EIP-4844 is the Ethereum Improvement Proposal that introduces Proto-Danksharding. It proposes a new transaction type that includes a “blob” of data.
The blob data is stored on-chain for a limited time (approximately 18 days). The data is not accessible to the EVM, but it is available for rollups to use.
EIP-4844 significantly reduces the cost of rollup transactions. By providing a dedicated data storage layer, it enables rollups to achieve higher throughput at lower cost.
EIP-4844: Proto-Danksharding ┌─────────────────────────────────────────────────────────────────────┐ │ EIP-4844: Proto-Danksharding │ │ │ │ Blob Transaction: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • New transaction type │ │ │ │ • Contains blob of data │ │ │ │ • Data stored for ~18 days │ │ │ │ • Not accessible to EVM │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ Benefits: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Lower cost for rollups │ │ │ │ • Higher throughput │ │ │ │ • Improved scalability │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
Benefits of Proto-Danksharding:
Proto-Danksharding offers several benefits. It significantly reduces the cost of rollup transactions, as the blob data is much cheaper than calldata. It also improves the scalability of rollups, as they can process more transactions per batch.
Proto-Danksharding also improves the security of rollups. By providing a dedicated data storage layer, it ensures that rollup data is available on-chain.