Google cloud platform architects solutions 1 pdf

About the authorsVitthal Srinivasan is a Google Cloud Platform Authorized Trainer and certified Google Cloud Architect and Data Engineer.. Judy Raj is a Google Certified Professional Clo

Google Cloud Platform for Architects

Design and manage powerful cloud solutions

Vitthal Srinivasan

Janani Ravi

Judy Raj

BIRMINGHAM - MUMBAI

Google Cloud Platform for Architects

Copyright © 2018 Packt Publishing

All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the

information contained in this book is sold without warranty, either express or implied Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Vijin Boricha

Acquisition Editor: Rohit Rajkumar

Content Development Editor: Abhishek Jadhav

Technical Editor: Mohd Riyan Khan

Copy Editors: Safis Editing, Dipti Mankame

Project Coordinator: Judie Jose

Proofreader: Safis Editing

Indexer: Priyanka Dhadke

Graphics: Tom Scaria

Production Coordinator: Shantanu Zagade

First published: June 2018

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well asindustry leading tools to help you plan your personal development and advance your career For moreinformation, please visit our website

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub filesavailable? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, youare entitled to a discount on the eBook copy Get in touch with us at service@packtpub.com for more

details

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of freenewsletters, and receive exclusive discounts and offers on Packt books and eBooks

Contributors

About the authors

Vitthal Srinivasan is a Google Cloud Platform Authorized Trainer and certified Google Cloud

Architect and Data Engineer Vitthal holds master's degrees in math and electrical engineering fromStanford and an MBA from INSEAD He has worked at Google as well as at other large firms, such

as Credit Suisse and Flipkart He is currently in Loonycorn, a technical video content studio, of which

he is a cofounder

Janani Ravi is a certified Google Cloud Architect and Data Engineer She has earned her master's

degree in electrical engineering from Stanford She is currently in Loonycorn, a technical video

content studio, of which she is a cofounder Prior to co-founding Loonycorn, she worked at variousleading companies, such as Google and Microsoft, for several years as a software engineer

I would like to thank my family, dogs, colleagues at Loonycorn, and friends for making life so much fun!

Judy Raj is a Google Certified Professional Cloud Architect, and she has great experience with the

three leading cloud platforms, namely AWS, Azure, and the GCP She has also worked with a widerange of technologies in machine learning, data science, IoT, robotics, and mobile and web app

development She is currently a technical content engineer in Loonycorn She holds a degree in

computer science and engineering from Cochin University of Science and Technology Being a drivenengineer fascinated with technology, she is a passionate coder, an AI enthusiast, and a cloud

aficionado

I'd like to thank my coauthors and colleagues for all the support and encouragement I've received I'd also like to thank God and my parents for everything that I am and everything I aspire to be.

About the reviewer

Tim Berry is a systems architect and software engineer with over 20 years of experience in building

enterprise infrastructure and systems on the internet and mobile platforms He currently leads a team

of SREs building customer solutions on Google Cloud Platform for a managed services provider inthe UK Tim is a Google Certified Professional Cloud Architect and Data Engineer, a Red Hat

Certified Engineer, and systems administrator He holds Red Hat Certified Specialist status for

configuration management and containerized application development

I would like to thank my wife, Sophie, for her patience and inspiration; my amazing kids, Josh, Ellie, Tommy, Isaac, and Noah for making me laugh; and my parents for always supporting me I'd also like to say hi to my dogs, Ruby, Lilo, and Belle, and ask them to get down now please because I'm working.

Nisarg M Vasavada is a content engineer in Loonycorn He has pursued his master's in engineering

at GTU, and he has been an active member of technical education and research community with hispublications He loves writing and believes that simplifying complexities is the biggest responsibility

of an author

Being a part of this book's writing process was absolutely insightful I would like to thank and dedicate this book to my family, colleagues, and mentors for always looking after me Also, lots of love and warmth to my feline and canine siblings!

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today

We have worked with thousands of developers and tech professionals, just like you, to help themshare their insight with the global tech community You can make a general application, apply for aspecific hot topic that we are recruiting an author for, or submit your own idea

Table of Contents

Title Page

Copyright and Credits

Google Cloud Platform for Architects Packt Upsell

Why subscribe?

PacktPub.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you Preface

Who this book is for

What this book covers

To get the most out of this book

Conventions used Get in touch

Reviews

1 The Case for Cloud Computing

Genesis

Why Google Cloud Platform (GCP)?

Autoscaling and autohealing

Capital expenditure (CAPEX) versus operating expenses (OPEX)

Career implications

Summary

2 Introduction to Google Cloud Platform

Global, regional, and zonal resources

Accessing the Google Cloud Platform

Projects and billing

Setting up a GCP account

Using the Cloud Shell

Summary

3 Compute Choices – VMs and the Google Compute Engine

Google Compute Engine – GCE

Creating VMs Creating a VM instance using the web console

Creating a VM instance using the command line

VM customization options Operating system

Compute zone

Machine type Networks – aka VPCs

Storage options Persistent disks and local SSDs – block storage for GCE Understanding persistent disks and local SSDs

Creating and attaching a persistent disk

Linux procedure for formatting and mounting a persistent disk

Sharing a persistent disk between multiple instances

Resizing a persistent disk More on working with GCE VMs

Startup scripts Snapshots and images How to snapshot a disk

How to create an image of a disk

Cloud launcher

Deploying LAMP stack using GCE

Modifying GCE VMs

Summary

4 GKE, App Engine, and Cloud Functions

Creating a Kubernetes cluster and deploying a WordPress container

Using the features of GKE

Storage and persistent disks Load balancing

Auto scaling Scaling nodes with the cluster autoscaler

Scaling pods with the horizontal pod autoscaler

Hosted Docker containers with App Engine Flex

Running a simple Python application with App Engine Flex 

Cron Jobs with App Engine Flex

Advantages of GKE over Docker on VMs or App Engine Flex Google App Engine – standard

Hosted web apps with App Engine Standard Typical App Engine architecture

Deploying and running on App Engine Standard

Traffic splitting

Serverless compute with cloud functions

Cloud Functions triggered by HTTP

Cloud Functions triggered by Pub/Sub

Cloud functions triggered by GCS object notifications

Summary

5 Google Cloud Storage – Fishing in a Bucket

Knowing when (and when not) to use GCS

Serving Static Content with GCS Buckets

Storage classes–Regional, multi-regional, nearline, and coldline

Working with GCS buckets

Creating buckets

Creating buckets using the web console

Creating buckets using gsutil

Changing the storage class of bucket and objects Transferring data in and out of buckets

Uploading data to buckets using the web console

Uploading data to buckets using gsutil

Copying data between buckets using the web console

Copying data between buckets using the gsutil command line Using the Transfer Service (instead of gsutil or the web console) Transfer Service or gsutil?

Use case – Object Versioning

Object versioning in the Cloud Storage bucket Use case – object life cycle policies

Managing bucket life cycle using the web console

Manipulating object life-cycle via JSON file

Deleting objects permanently using the web console

Deleting objects permanently using gsutil Use case – restricting access with both ACLs and IAM

Managing permissions in bucket using the GCP console Use case – signed and timed URLs

Setting up signed URLs for cloud storage Use case – reacting to object changes

Setting up object change notifications with the gsutil notification watchbucket

Use case – using customer supplied encryption keys

Use case – auto-syncing folders

Use case – mounting GCS using gcsfuse

Mounting GCS buckets Use case – offline ingestion options

Summary

6 Relational Databases

Relational databases, SQL, and schemas

OLTP and the ACID properties

Scaling up versus scaling out GCP Cloud SQL

Creating a Cloud SQL instance

Creating a database in a Cloud SQL instance

Importing a database

Testing Cloud SQL instances Use case – managing replicas

Use case – managing certificates

Use case – operating Cloud SQL through VM instances

Automatic backup and restore

Cloud Spanner

Creating a Cloud Spanner instance

Creating a database in Cloud Spanner instances

Querying a database in a Cloud Spanner instance Interleaving tables in Cloud Spanner

Summary

Choosing storage for Bigtable

Solving performance issues

Ideal row key choices

Performing operations on Bigtable Creating and operating an HBase table using Cloud Bigtable Exporting/Importing a table from Cloud Bigtable

Scaling GCP Cloud BigTable

The Google Cloud Datastore

Comparison with traditional databases

Working with Datastore

When to use Datastore Full indexing and perfect index

Using Datastore

Summary

8 BigQuery

Underlying data representation of BigQuery

BigQuery public datasets

Legacy versus standard SQL

Working with the BigQuery console

Loading data into a table using BigQuery

9 Identity and Access Management

Resource hierarchy of GCP

Permissions and roles

Units of identity in GCP

Creating a Service Account

Working with cloud IAM – grant a role

Working with IAM – creating a custom role

Summary

10 Managing Hadoop with Dataproc

Hadoop and Spark

Hadoop on the cloud

Google Cloud Dataproc

Compute options for Dataproc

Working with Dataproc

Summary

11 Load Balancing

Why load balancers matter now

Taxonomy of GCP load balancers

HTTP(S) load balancing

Configuring HTTP(S) load balancing

Configuring Internal Load Balancing

Other load balancing

Summary

12 Networking in GCP

Why GCP's networking model is unique

VPC networks and subnets

The default VPC

Internal and external IP addresses

VPN and cloud router

Working with VPCs

Working with custom subnets

Working with firewall rules

Summary

13 Logging and Monitoring

Logging

Working with logs

More Stackdriver – creating log-based metrics

Monitoring

Summary

14 Infrastructure Automation

Managed Instance Groups

Cloud deployment manager

Summary

15 Security on the GCP

Security features at Google and on the GCP

Google-provided tools and options for security

Some security best practices

BeyondCorp – Identity-Aware Proxy

Summary

Video Intelligence API

Key Management Service – KMS

Vision API

Summary

17 Effective Use of the GCP

Eat the Kubernetes frog

Careful that you don't get nickel-and-dimed

Pay for what you allocate not what you use

Make friends with the gsuite admins

Try to find reasons to use network peering

Understand how sustained use discounts work

Read the fine print on GCS pricing

Use BigQuery unless you have a specific reason not to Use pre-emptible instances in your Dataproc clusters

Keep your Dataproc clusters stateless

Understand the unified architecture for batch and stream

Understand the main choices for ML applications

Understand the differences between snapshots and images

Don't be Milton!

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

originated at Google before being open-sourced, and the GCP is a natural choice of cloud on which torun them If you are a cloud professional today, time spent on mastering the GCP is likely to be anexcellent investment.

Using a public cloud platform was considered risky a decade ago and unconventional even just a fewyears ago Today, however, the use of the public cloud is completely mainstream—the norm, ratherthan the exception Several leading technology firms, including Google, have built sophisticated cloudplatforms, and they are locked in a fierce competition for market share

The main goal of this book is to enable you to get the best out of the GCP and to use it with

confidence and competence You will learn why cloud architectures take the forms that they do, andthis will help you to become a skilled, high-level cloud architect You will also learn how individualcloud services are configured and used so that you are never intimidated at having to build it yourself.You will also learn the right way and the right situation in which to use the important GCP services

By the end of this book, you will be able to make the most out of Google Cloud Platform design

Who this book is for

If you are a Cloud architect who is responsible for designing and managing robust cloud solutionswith Google Cloud Platform, then this book is for you System engineers and Enterprise architectswill also find this book useful A basic understanding of distributed applications would be helpful,although not strictly necessary Some working experience on other public cloud platforms would helptoo

What this book covers

Chapter 1, The Case for Cloud Computing, starts with the brief history of cloud computing.

Furthermore, the chapter delves into autohealing and autoscaling

Chapter 2, Introduction to Google Cloud Platform, gets you into the nitty-gritty of the Google Cloud

Platform, describing the diversity and versatility of the platform in terms of the resources available tous

Chapter 3, Compute Choices – VMs and the Google Compute Engine, explores GCE, which serves as

an IaaS provision of GCP You will learn to create GCE VMs, along with its various aspects such asdisk type and machine types

Chapter 4, GKE, AppEngine, and Cloud Functions, discusses the four compute options on the GCP,

ranging from IaaS through PaaS

Chapter 5 , Google Cloud Storage – Fishing in a Bucket, gets you familiar with GCS and gives an idea

of where it would fit within with your overall infrastructure

Chapter 6, Relational Databases, introduces you to RDMS and SQL We further dive deep into Cloud

SQL and Cloud Spanner that are available under GCP

Chapter 7, NoSQL Databases, takes you through Bigtable and Datastore This chapter explains how

Bigtable is used for large datasets, whereas on the other hand, Datastore is meant for far smaller data

Chapter 8, BigQuery, teaches you about the architecture of BigQuery and how it is Google’s fully

managed petabyte-scale serverless database

Chapter 9, Identity and Access Management, dives into how IAM lets you control access to all of the

GCP resources in terms of roles and permissions

Chapter 10, Managing Hadoop with Dataproc, helps you to understand Dataproc as a managed and

cost-effective solution for Apache Spark and Hadoop workloads

Chapter 11, Load Balancing, takes you through HTTP, TCP, and network load balancing with reference

to its concepts and implementation

Chapter 12, Networking in GCP, teaches you about Virtual Private Cloud Networks of GCP and their

infrastructure and how to create and manage our own VPC networks

Chapter 13, Logging and Monitoring, discusses how Stackdriver offers logging and monitoring

services of GCP resources for free up to a certain quota and then monitoring both GCP and AWSresources for premium account holders

Chapter 14, Infrastructure Automation, delves into the idea of how provisioning resources can be done

programmatically, using templates, commands, and even code

Chapter 15, Security on the GCP, mostly covers things such as how Google has planned for security on

the GCP

Chapter 16, Pricing Considerations, helps avoid sticker-shock and sudden unpleasant surprises

regarding the pricing of the services that you use

Chapter 17, Effective Use of the GCP, sharpens all of the GCP features and offerings that you learned

in the previous chapters to make sure that we conclude our journey on a satisfactory note

To get the most out of this book

1 First, go breadth-first Read each chapter rapidly, paying particular attention to the early bits and

to the rhymes They summarize the key points

2 Don’t forget to laugh while reading the rhymes! Seriously, pay attention to each line in the

rhymes as they are particularly packed with information

3 After you finish going through the entire book quickly, come back to the chapters that relate toyour specific use cases and go through them in detail

4 For the drills in the book, understand what step is trying to accomplish, then try it out on yourown In particular, also search for online updates for your most important use cases—the world

of cloud computing and the GCP is changing incredibly fast

Conventions used

There are a number of text conventions used throughout this book

CodeInText: Indicates code words in text, database table names, folder names, filenames, file

extensions, pathnames, dummy URLs, user input, and Twitter handles Here is an example: "A publicdataset named samples.natality is queried"

A block of code is set as follows:

ORDER BY weight_pounds DESC LIMIT 10;

When we wish to draw your attention to a particular part of a code block, the relevant lines or itemsare set in bold:

ORDER BY weight_pounds DESC LIMIT 10;

Any command-line input or output is written as follows:

curl -f -O import-1.1.2-shaded.jar

http://repo1.maven.org/maven2/com/google/cloud/bigtable/bigtable-beam-import/1.1.2/bigtable-beam-Bold: Indicates a new term, an important word, or words that you see onscreen For example, words

in menus or dialog boxes appear in the text like this Here is an example: "To upload the datafile,click on the Choose file button."

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome

General feedback: Email feedback@packtpub.com and mention the book title in the subject of your

message If you have questions about any aspect of this book, please email us at questions@packtpub.com

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do

happen If you have found a mistake in this book, we would be grateful if you would report this to us.Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Formlink, and entering the details

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be

grateful if you would provide us with the location address or website name Please contact us at

copyright@packtpub.com with a link to the material

If you are interested in becoming an author: If there is a topic that you have expertise in and you

are interested in either writing or contributing to a book, please visit authors.packtpub.com

Please leave a review Once you have read and used this book, why not leave a review on the sitethat you purchased it from? Potential readers can then see and use your unbiased opinion to makepurchase decisions, we at Packt can understand what you think about our products, and our authorscan see your feedback on their book Thank you!

For more information about Packt, please visit packtpub.com

The Case for Cloud Computing

Cloud computing is a pretty big deal in the world of technology, and in addition it is also a pretty bigdeal for those who are not quite in technology Some developments, for instance, the rise of Java andobject-oriented programming, were momentous changes for people who were completely into

technology at the time, but it was rare for a non-technical person to have to wake up in the morning,

read the newspaper and ask themselves, Wow, this Java thing is getting pretty big, will this affect

my career? Cloud computing, perhaps like machine learning or Artificial Intelligence (AI), is

different; there is a real chance that it, by itself, will affect the lives of people far beyond the world oftechnology Let's understand why

You will learn the following topics in this chapter:

A brief history of cloud computing

Autohealing and autoscaling—good technical reasons for moving to the cloud

Some good financial reasons for moving to the cloud

Possible implications of cloud computing on your career

In the beginning, Jeff Bezos created Amazon.com and took the company to a successful Initial Public

Offering (IPO) by 1997 Everyone knows Amazon.com, of course, and it has become a force of nature,dominating the online retail and diversifying into several other fields However, in the early 2000s,after the Dotcom bubble burst, the company's future was not quite as certain as now Even so, one ofthe many things that Amazon was doing right even then was architecting its internal computer systems

in a truly robust and scalable way

Amazon had a lot of users and a lot of traffic, and in order to service that traffic, the company reallyhad to think deeply about how to build scalable, cost-effective compute capacity Now you couldargue rightly that other companies had to think about the same issues too Google also had a lot ofusers and a lot of traffic, and it had to think really carefully about how to handle it Even so, mostobservers agree that a couple of important differences existed between the two giants For one,

Google's business was (and is) fundamentally a far more profitable one, which means that Googlecould always afford to overinvest in compute, secure in the knowledge that its money printing press inthe ad business would cover the costs For another, Google's primary technical challenges came inprocessing and making sense of vast quantities of data (it was basically indexing the entire internetfor Google Search) Amazon's primary technical challenges lay around making sure that the inherentlyspiky traffic of their hundreds of millions of users was serviced just right The spiky nature of

consumer traffic remains a huge consideration for any online retail firm Just consider Alibaba, which

did $25 billion in sales on Singles Day (11/11) in 2017.

Somewhere along the line, Amazon realized that it had created something really cool: a set of APIsand services, a platform in fact that external customers would be willing to pay for, and that wouldhelp Amazon monetize excess server capacity it had lying about Let's not underestimate the

magnitude of that achievement; plenty of companies have overinvested in servers and have extra

capacity lying around, but virtually none of them have built a platform that other external customersare willing and able to use and to pay top dollar for

So, in 2006, Amazon launched Elastic Compute Cloud (EC2), basically, cloud Virtual Machine (VM) instances, and Simple Storage Service (S3), basically, elastic object storage, which to this

day are the bedrock of the AWS cloud offerings Along the way, the other big firms with the moneyand technical know how to offer such services jumped in as well Microsoft launched Azure in 2010,

and Google had actually gotten into the act even earlier, in 2008, with the launch of App Engine

Notice how Amazon's first product offerings were basically Infrastructure as a service (IaaS), whereas Google's initial offering was a Platform as a service (PaaS) That is a significant fact and

with the benefit of hindsight, a significant mistake on Google's part If you are a large organization,circa 2010, and contemplating moving to the cloud, you are unlikely to bet the house on moving to anuntested cloud-specific platform such as App Engine The path of least resistance for big early

adopters is definitely the IaaS route The first-mover advantage and the smart early focus on IaaShelped Amazon open up a huge lead in the cloud market, one which they still hold on to