This chapter is from the book
This chapter is from the book
This chapter is from the book
Streams
Oracle Cloud Streams is a fully managed, cloud-based messaging service offered by Oracle Cloud. It enables organizations to build real-time, event-driven applications by providing a secure and scalable messaging platform. The service allows developers to publish and subscribe to messages, enabling them to build applications that can process and react to events in real time. The platform supports a wide range of messaging patterns and protocols, including publish-subscribe, point-to-point, and request-response.
A producer publishes messages to a stream, which is an append-only log. These messages are distributed among Oracle-managed partitions for scalability. Partitions allow you to distribute a stream by splitting messages across multiple nodes (or brokers). Each partition can be placed on a separate machine, allowing multiple consumers to read a stream in parallel. A consumer reads messages from one or more partitions. Consumers can read from any partition regardless of where the partition is hosted. Each message within a stream is marked with an offset value, so a consumer can pick up where it left off if it is interrupted. Messages from a partition are guaranteed to be delivered in the same order they were produced. Consumers can read messages explicitly by providing the partition and offset, or as a member of a consumer group, which coordinates the consumption of an entire stream by the members of the group.
Some examples of how Oracle Cloud Streams can be used include building microservices architectures, real-time data processing pipelines, and event-driven applications. The platform can be integrated with other Oracle Cloud services, such as Oracle Functions and Oracle Autonomous Transaction Processing, to build and deploy cloud-native applications.
When setting up Streams, you should be aware of the following concepts.
Consumer: An entity that reads messages from one or more streams.
Consumer Group: A group of instances that work together to consume messages from all partitions in a stream. At any given time, messages from a specific partition can be consumed by only a single consumer in the group.
Cursor: A reference to a specific position in a stream, such as an offset or time in a partition, or a group’s current location.
Instance: A consumer group member is defined when a group cursor is created and group membership is maintained through interaction. Lack of interaction leads to a timeout and removes the instance from the consumer group.
Key: An identifier used to group related messages.
Message: A message encoded in Base64 is published to a schema-agnostic stream. The stream accepts various message formats, such as XML, JSON, CSV, and gzip. Producers and consumers must agree on the format.
Offset: The offset is the position of a message within a partition. Each message is identified by its offset, and consumers can choose to read messages starting from any offset. If the reading process is interrupted, the offset can be used to restart reading from where it left off in the stream.
Partition: A section of a stream. Partitions make it possible to distribute a stream by splitting messages across multiple nodes. This also enables multiple consumers to read from a stream simultaneously.
Producer: An entity that publishes messages to a stream.
Stream: A partitioned, append-only log of messages.
Stream Pool: A grouping that you can use to organize and manage streams, including any shared Apache Kafka or security settings.
Overall, Oracle Cloud Streams is designed to make it easier for organizations to build real-time, event-driven applications by providing a scalable and secure messaging platform in the cloud.
