Exploring Google Cloud Pub/Sub: Publish/Subscribe Messaging Explained

Blogs

How to Connect AWS Athena with Power BI: A Step-by-Step Guide
December 31, 2024
Hosting an Application
December 31, 2024

Exploring Google Cloud Pub/Sub: Publish/Subscribe Messaging Explained

Pub/Sub is a publish/subscribe (Pub/Subservice, a messaging service where the senders of messages are decoupled from the receivers of messages. 

The service is built on a core Google infrastructure component that many Google products have relied upon for over a decade. Google products including Ads, Search and Gmail use this infrastructure to send over 500 million messages per second, totaling over 1TB/s of data.  

We will briefly look at the publisher-subscriber (Pub/Sub) relationship below. 

  1. The publisher app connects to a topic in the Pub/Sub service. It then sends messages to this topic. 
  2. Messages are retained in message storage. 
  3. Messages are forwarded from the topic to all associated subscriptions individually. 
  4. The messages are transmitted by pushing to an endpoint or pulling from the service itself. 
  5. The subscriber application informs the Pub/Sub service that messages are received. 

 

The following are the components of a Pub/Sub service: 

  • Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic. 
  • Message: the data that moves through the service. 
  • Topic: a named entity that represents a feed of messages. 
  • Schema: a named entity that governs the data format of a Pub/Sub message. 
  • Subscription: a named entity that represents an interest in receiving messages on a particular topic. 
  • Subscriber (also called a consumer): receives messages on a specified subscription. 

 

Why Pub/Sub? 

Publish-subscribe is designed to decouple the producers of messages from the consumers of those messages. Instead of producers sending direct requests to the consumers with the data, they instead publish that data to a Pub/Sub service like Pub/Sub. The service asynchronously delivers those messages to interested consumers that have subscribed. 

The result is that the service absorbs all of the complications of finding consumers that are interested in the data. The service also manages the rate at which the consumers receive the data, based on their capacity. The decoupling allows data producers to write messages at high scale with low latency, independent of the behavior of the consumers. 

Pub/Sub offers highly scalable, reliable delivery of messages. While the service handles much of this automatically, we have control over different aspects of our publishers and subscribers that can affect availability and performance. The rest of this guide provides some details on these aspects 

 

Message Status in Pub/Sub 

In Google Cloud Pub/Sub, messages can exist in one of three states: acknowledged, unacknowledged, or negatively acknowledged. These states represent the lifecycle of a message as it is delivered to subscribers. 

  1. Acknowledged (Acked): Once a subscriber successfully processes a message, it sends an acknowledgment back to Pub/Sub. This confirms the message has been handled, and it is then removed from storage. If all subscribers to a topic acknowledge the message, it is deleted asynchronously from the system. 
  1. Unacknowledged (Unacked): If a subscriber does not acknowledge a message within the configurable time known as the ackDeadline, the message is considered unacknowledged. In such cases, the message may be redelivered to the subscriber or another one. This can happen if the acknowledgment is delayed or lost due to network issues. Unacknowledged messages remain available for delivery until their retention period expires. 
  1. Negatively Acknowledged (Nacked): When a subscriber signals that it cannot process a message (due to errors or invalid data), it sends a “negative acknowledgment” (nack).  

These states help ensure reliable message delivery and processing within Pub/Sub, allowing messages to be retried, acknowledged, or discarded based on the subscriber’s ability to handle them. 

  

Choosing a Pub/Sub publish and subscribe pattern 

When there are multiple Pub/Sub publisher and subscriber clients, we must also choose the kind of publish and subscribe architecture that we want to set up. 

Some of the supported Pub/Sub publish subscribe patterns include the following: 

  • Fan in (many-to-one). In this example, multiple publisher applications publish messages to a single topic. This single topic is attached to a single subscription. The subscription is, in turn, connected to a single subscriber application that gets all the published messages from the topic. 
  • Load balanced (many-to-many). In this example, a single or multiple publisher applications publish messages to a single topic. This single topic is attached to a single subscription that is, in turn, connected to multiple subscriber applications. Each of the subscriber applications gets a subset of the published messages, and no two subscriber applications get the same subset of messages. In this load balancing case, we use multiple subscribers to process messages at scale. If more messages need to be supported, we add more subscribers to receive messages from the same subscription. 
  • Fan out (one-to-many). In this example, a single or multiple publisher applications publish messages to a single topic. This single topic is attached to multiple subscriptions. Each subscription is connected to a single subscriber application. Each of the subscriber applications gets the same set of published messages from the topic. When a topic has multiple subscriptions, then every message has to be sent to a subscriber receiving messages on behalf of each subscription. If we need to perform different data operations on the same set of messages, fan out is a good option. We can also attach multiple subscribers to each subscription and get a load-balanced subset of messages for each subscriber. 

Conclusion 

We explored the key features and capabilities of Google Cloud Pub/Sub, a robust publish/subscribe messaging service designed to handle high-throughput, low-latency communication across distributed systems. By decoupling message producers (publishers) from consumers (subscribers), Pub/Sub enables scalable, asynchronous communication that is ideal for real-time analytics, event-driven architectures, and parallel processing workflows. 

Ultimately, Google Cloud Pub/Sub empowers us to build resilient, scalable systems that can handle large volumes of messages and distribute data to multiple services efficiently. Whether you’re ingesting real-time events, building an event-driven architecture, or processing large datasets, Pub/Sub offers a flexible and powerful solution that integrates seamlessly with other Google Cloud services. 


Nayan Sagar N K

Leave a Reply

Your email address will not be published. Required fields are marked *