As we begin this module, I would ask you to keep your mind open to new ways of doing things. Pub/Sub does streaming differently than probably anything you've used in the past; it may be a different model than what you've seen before. Pub/Sub provides a fully managed data distribution and delivery system, it can be used for many purposes, it is most commonly used to loosely coupled parts of a system. You can use Pub/Sub to connect applications within Google Cloud, and with applications on premise or in other Clouds to create hybrid data engineering solutions. With Pub/Sub, the applications do not need to be online and available all the time. And the parts do not need to know how to communicate to each other, but only to Pub/Sub, which can simplify system design. First Pub/Sub is not software; it is a service. So like all of the other serverless services we have looked at, you don't install anything to use Pub/Sub. Pub/Sub client libraries are available in C-Sharp, Go, Java, Node.js, Python, and Ruby. These RAP REST API calls which can be made in any language, it is also highly available. Pub/Sub offers durability of messages. By default, it will save your messages for seven days in the event your systems are down or not able to process them. Finally, Pub Sub is highly scalable as well. Google processes about 100 million messages per second across their entire infrastructure. This was actually one of the use cases for Pub/Sub early on at Google to be able to distribute the search engine and index around the world. Because we keep local copies of search around the world. As you might imagine, in order to be able to serve up results with minimal latency. If you think about how you would architect that, we are crawling the entire worldwide web. So we need to send the entire world wide web around the world. Either that or we would need to have multiple crawlers all over the world. But then you have data consistency problems, they would all be getting different indexes. Therefore, what we do is use Pub/Sub to distribute. As the crawler goes out, it grabs every page from the worldwide web. And we send every single page as a message on Pub/Sub, and it gets picked up by all local copies of the search index, so it can be indexed. Currently, Google indexes the web anywhere from every two weeks, which is the slowest to more than once an hour, for example, on really popular news sites. So on average, Google is indexing the web three times a day. Thus, what we are doing is sending the entire worldwide web over Pub/Sub three times a day. This should explain how Pub/Sub is able to scale. Pub/Sub is a HIPAA compliant service, offering fine grained access controls and end to end encryption. Messages are encrypted in transit and at rest. Messages are stored in multiple locations for durability and availability. You control the qualities of your Pub/Sub solution by the number of publishers, number of subscribers, and the size and number of messages. These factors provide tradeoffs between scale, low latency, and high throughput. How does Pub/Sub work? The model is very simple. The story of Pub/Sub is the story of two data structures, the topic and the subscription. Both the topic and the subscription are abstractions which exist in the Pub/Sub framework independently of any workers, subscribers, etcetera. The Pub/Sub client that creates the topic is called the publisher. And the Pub/Sub client that creates the subscription is called the subscriber. In this example, the subscription is subscribed to the topic. To receive messages published to a topic, you must create a subscription to that topic, only messages published to the topic after the subscription is created are available to subscriber applications. The subscription connects the topic to a subscriber application that receives and processes messages published to the topic. A topic can have multiple subscriptions, but a given subscription belongs to a single topic. In essence, it's an enterprise message bus. So how does it work? As you see here, there's a HR topic that relates to new hire events. For example, a new person joins your company and this notification should allow other applications that need to be notified about a new user joining to subscribe and get that message. What applications could tell you that a new person joined? One example is the company directory. This is a client of the subscription, also called a subscriber. However, Pub/Sub is not limited to one subscriber or one subscription. Here there are multiple subscriptions and multiple subscribers. Maybe the facility system needs to know about the new employee for badging and the accounting provisioning system needs to know for payroll, each subscription guarantees delivery of the message to the service. The subscriber clients are decoupled from one another and isolated from the publisher. In fact, we will see later that the HR system could go offline after it has sent its message to the HR topic and the message will still be delivered to the subscribers. These examples show one subscription and one subscriber, but you can actually have more subscribers for a single subscription. In this example, the badge activation system requires a human being to activate the badge. There are multiple workers, but not all of them are available all the time. Pub/Sub makes the message available to all of them, but only one person needs to fetch the message and handle it. This is called a pull subscription. The other examples are push subscriptions. Now a new contract arrives. Instead of entering through the HR system they go through the vendor offers, the same kinds of actions need to occur for this worker, they need to be listed in the company directory, then facilities need to assign them a desk. Account provisioning needs to set up their corporate identity and accounts, and the badge activation system needs to print an activate their contractor badge. A message can be published by the vendor office to the HR topic, the vendor office and the HR system are entirely decoupled from one another, but can make use of the same company services. You can see from this illustration how important the Pub/Sub is, therefore it gets the highest priority. When you receive messages from a subscription with a filter, you only receive the messages that match the filter. The Pub/Sub service automatically acknowledges the messages that don't match the filter. You can filter messages by their attributes. You don't incur egress fees for the messages that Pub/Sub automatically acknowledges you incur message delivery fees and seek related storage fees for these messages. You can create a subscription with a filter using the Cloud Console, the gcloud command-line tool, or the Pub/Sub API. You have learned generally what Pub/Sub does, next you will learn how it works and many of the advanced features it provides.
