The final layer of the Google Cloud infrastructure that is left to explore is big data and machine learning products. In this video, we'll examine the evolution of data processing frameworks through the lens of product development. Understanding the chronology of products can help address typical big data and machine learning challenges. Historically speaking, Google experienced challenges related to big data quite early, mostly with large datasets, fast changing data, and varied data. This was the result of needing to index the World Wide Web. As the internet grew, Google needed to invent new data processing methods. In 2002, Google released the Google File System or GFS. GFS was designed to handle data sharing and petabyte storage at scale. It served as the foundation for Cloud storage, and also what would become the managed storage functionality in BigQuery. A challenge that Google was facing around this time was also how to index the exploding volume of content on the web. To solve this, in 2004, Google wrote a report that introduce MapReduce. MapReduce was a new style of data processing designed to manage large scale data processing across big clusters of commodity servers. As Google continued to grow, new challenges arose. Specifically with recording and retrieving millions of streaming user actions with high throughput. The solution was the release in 2005 of Cloud Bigtable, a high performance, NoSQL database service for large analytical and operational workloads. With MapReduce available, some developers were restricted by the need to write code to manage their infrastructure, which prevented them from focusing on application logic. As a result from 2008-2010, Google started to move away from MapReduce as the solution to process and query large datasets. In 2008, Dremel was introduced. Dremels took a new approach to big data processing by breaking the data into smaller chunks called shards and then compressing them. Dremel then uses a quick optimizer to share tasks between the many shards of data in the Google data centers which process queries and delivered results. The big innovation was that Dremel autoscale to meet query demands. Dremel became the query engine behind BigQuery. Google continued innovating to solve big data and machine learning challenges. Some of the technology solutions released include Colosus in 2010, which is a cluster level file system and successor to the Google File System. Spanner in 2012, which is a globally consistent, scalable relational database. Pub/Sub in 2015, which is a service used for streaming analytics and data integration pipelines to ingest and distribute data. TensorFlow also in 2015, which is a free and open source software library for machine learning and artificial intelligence. 2018 breaths release of the Tensor Processing Unit or TPU, which you'll recall from earlier in this course. It's thanks to these technologies that the big data and machine learning product line is now robust. This includes Cloud Storage, Dataproc, Bigtable, BigQuery, Data flow, Firesto, Pub/Sub, Looker, Cloud Spanner, AutoML, and Vertex AI, the unified platform. These products and services are made available through Google Cloud. You'll get hands on practice with some of them as part of this course.
