With the theoretical background covered in the previous lesson, let's now shift our focus on actual implementation. How are things set up in Pub/Sub? Let's look at a little bit of code now. This example is using the client library for Pub/Sub. If we want to publish the message, we first create the topic. Then we can publish the topic on the command line. More commonly, it will be done in code. Here we get a publisher client, create a topic and publish the message. Notice the letter b in front of My first message. This is because Pub/Sub just sends raw bytes. This means that you aren't restricted to just text. You can send other data like images if you wanted to. The limit is ten megabytes. There are also extra attributes that you can include in messages. In this example, ou see, author = dylan. Pub/Sub will keep track of those attributes to allow your downstream systems to get metadata about your messages without having to put it all in the message and parse it out. So instead of serializing and de-serializing, it will just keep track of those key value pairs. Some of them have special meaning. We will see some of those shortly. To subscribe with Pub/Sub using the pull method, the code is similar. Select the topic, name the subscription. This is a pull subscription, so we will define a callback. When you were doing a pull subscription, it looks like this. You can pull the messages from the command line. You will see this in the lab. By default, it will just turn one message, the latest message. But there is a dash dash limit you can set. Maybe you want ten messages at a time. You can try that in the lab. You can also batch published messages. This just prevents the overhead of the call for individual messages on the publishing side. This allows the Pub/Sub publishing engine to wait and send 10 or 50 at a time. This increases efficiency. However, if you are waiting for 50 messages, this means the first one now has latency associated with it. So it is a trade off in your system. What do you want to optimize? But in any case, even if you batch publish, they still get delivered one at a time to your subscribers. We will practice this technique in the lab. Here is how you would set the batch in python code. Explore the documentation for other command options and settings. If messages have the same ordering key and are in the same region, you can enable message ordering and receive the messages in the order that the Pub/Sub service receives them. When the Pub/Sub service re-delivers a message with an ordering key, the Pub/Sub service also re-delivers every subsequent message with the same ordering key, including acknowledged messages. If both message ordering and a dead letter topic are enabled on a subscription, the ordering may not be true as Pub/Sub forwards messages to dead letter topics on a best effort basis. To receive the messages in order, set the message ordering property on the subscription you receive messages from. Please note that receiving messages in order might increase latency. You can set the message ordering property when you create a subscription using the cloud console, the gcloud command line tool or the Pub/Sub API. Pub/Sub is also going to help us with streaming resilience or buffering. What happens if your systems get overloaded with large volumes of transactions like Black Friday? What you really need is some sort of buffer or backlog, so that you can feed messages only as fast as the systems are able to process them. Pub/Sub has this as a built-in capability. Let's recap this then. When you look at the example on this slide in the example on the left, an overload of arriving data causes a traffic spike. This over drives the resources of the application as illustrated by the smoke. One solution to this problem is to size the application to handle the highest traffic spike, plus some additional capacity as a safety buffer. This is not only wasteful of resources, which must be retained at top capacity even when not being used. But it provides a recipe for a distributed denial of service attack by creating an upper limit at which the application will cease to behave normally and will exhibit non-deterministic behavior. The solution on the right uses Pub/Sub as an intermediary, receiving and holding data until the application has resources to handle it either through processing the backlog of work or by auto scaling to meet the demand. Erroneous records may cause your pipeline to get stuck or fail outright. I highly recommend implementing a dead letter queue and error logging to prevent these failure modes. These can help catch problems in user code and or a data shape. The idea behind exponential backoff is to add progressively longer delays between retry attempts. After the first delivery failure, Pub/Sub will wait for a minimum back off time before retrying. For each consecutive failure on that message, more time will be added to the delay up to a maximum delay. The maximum and minimum delay intervals are not fixed and should be configured based on local factors to your application. Cloud audit logs maintains three audit logs for each Google Cloud Project, folder, and organization, admin activity, data access, and system event. Admin activity audit logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. Data access audit logs contain API calls that read the configuration or metadata of resources, as well as user driven API calls that create, modify, or read user provided resource data. Admin activity audit logs are always written, you can't configure or disable them data access. Data access audit logs are disabled by default, because they can be quite large, they must be explicitly enabled to be written. Pub/Sub reports metrics to cloud logging, including Pub/Sub topic and Pub/Sub subscription, which you can monitor against service quota utilization in a dashboard and for setting notifications and alerts. Authentication is provided by service accounts. You can also directly authenticate users by their user accounts, and their identity is reported in audit logs. But user account authentication is not recommended. Access control is provided by IAM.
