External Storage Provider Architecture

Event streaming platforms such as Kafka and Azure Event Hubs are primarily designed for real‑time data processing and have inherent limitations on message size. By default, these platforms impose message size limits of approximately 1MB. This limitation poses a significant challenge when dealing with scenarios where state changes involve large blob or slob data. Examples of large data are documents and various binary data content such as multimedia data.

These large data items cannot be efficiently represented within the constraints of the default message size limit. An External Storage Provider is a potential solution that allows storing large data externally while still maintaining a connection to the parent event.

For details about configuring your size thresholds by setting limits for when large binary (blob) and string (slob) objects should be stored externally instead of being included directly in the JSON output, see the "SizeThresholds" section under "Event Stream Producer Configuration", in Chapter 3.

Key Features

The external storage provider has the following key features.

• Enables flexible storage options (for example, Azure Blob or the local file system)

• Provides a standardized interface for integrating custom storage solutions

• Maintains JSON schema compliance for event representation

• Enhances scalability by offloading large data to external storage

• Uses a special prefix ($) for property names to avoid conflicts

Benefits

The external storage provider has the following benefits.

• Supports representing database state changes that exceed Kafka or Event Hubs message size limits

• Increased modularity and extensibility

• Clear association of URIs with their corresponding object properties