IT training extending kubernetes khotailieu

This report is not about getting started with Kubernetes; it won’tshow you how to use kubectl or how to install it via kubeadm.. A simplified Kubernetes architecture, with its more impor

Extending Kubernetes

Making Use of Work Queues,

Reconciliation Loops, Controllers, and Operators

Gianluca Arbezzano

Boston Farnham Sebastopol Tokyo

Beijing Boston Farnham Sebastopol Tokyo

Beijing

[LSI]

Extending Kubernetes

by Gianluca Arbezzano

Copyright © 2019 O’Reilly Media, Inc All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com) For more infor‐

mation, contact our corporate/institutional sales department: 800-998-9938 or cor‐

porate@oreilly.com.

Acquisitions Editor: John Devins

Developmental Editor: Virginia Wilson

Production Editor: Nan Barber

Copyeditor: Octal Publishing, LLC

Proofreader: Nan Barber

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Rebecca Demarest August 2019: First Edition

Revision History for the First Edition

2019-07-31: First Release

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Extending Kuber‐

netes, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, includ‐ ing without limitation responsibility for damages resulting from the use of or reli‐ ance on this work Use of the information and instructions contained in this work is

at your own risk If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of oth‐ ers, it is your responsibility to ensure that your use thereof complies with such licen‐ ses and/or rights.

Table of Contents

Preface v

1 Kubernetes Extensibility 1

Kubernetes Is a Framework 1

Unified User Interface for Operations 3

Conclusion 9

2 Client Side 11

kubectl Plug-Ins 11

API and SDK 13

Go Client Library 17

Conclusion 20

3 Server-Side Extensions and Primitives 21

Informers 21

Work Queues 26

Custom Resource Definitions 29

Pain Point 36

Conclusion 37

iii

“Kubernetes is too complex” is a phrase we have all heard at leastonce, and it is partly true Kubernetes has a solid architecturebecause it is the foundation for deploying and managing the dis‐tributed system life cycle The question I always raise is: “How doyou justify this complexity?”

For example, are you writing and applying a YAML specification viathe kubectl? Then you probably do not need a lot of the features thatmake Kubernetes so complex

Amazon Web Services (AWS) has the same problem A lot of devel‐opers keep saying that it is an expensive service, but this is not true.AWS hides a huge amount of complexity behind its bill and youneed to have the right people capable of justifying the cost When Isee how much control you have over your infrastructure with Ama‐zon Virtual Private Cloud (Amazon VPC), subnets, internet gate‐ways, and routers, I think the cost is justified This is even truerwhen I think about how I am building a private network via APIand not physically buying and cabling it The APIs are the way youintegrate the complexity of AWS into something that makes yourproduct better

Using the Kubernetes API to implement and automate flows or tohave more visibility into what is happening in your system is theway I justify the cost of Kubernetes, and in this report I explain how

I am doing it

This report is not about getting started with Kubernetes; it won’tshow you how to use kubectl or how to install it via kubeadm If youare looking for that kind of content, I recommend reading Kuber‐ netes: Up and Running (O’Reilly) as a great way to start your journey

v

with Kubernetes But if you are a software engineer with a goodknowledge of how Kubernetes works, I’d like to teach you how towrite small applications integrated with Kubernetes so that you can

CHAPTER 1

Kubernetes Extensibility

This first chapter highlights why I think extending Kubernetes via itsAPI is the right way to utilize all of its capabilities It gives you anidea of how you get to include your flow “for free” as part of theKubernetes architecture—things like authentication and authoriza‐tion, for example

This chapter demonstrates that there is much more that you can dobeside writing YAML specifications to apply via the command line

Kubernetes Is a Framework

When I think about Kubernetes, I do not see it as an end applica‐tion For example, it does not include MySQL, Jaeger, or other devel‐oper tools or databases To me, Kubernetes provides solid primitivesthat I can integrate into my application, such as the following:

kubectl run nginx image=nginx port=80

This is the first command everyone runs when they are learningKubernetes for the first time This action creates a new pod with anNGINX container image that exposes port 80

But what happens behind the scenes? Figure 1-1 illustrates this from

a very high perspective kubectl is a very powerful and flexible clientfor interacting with Kubernetes Because it is complex, it cannothave just one piece of code As you can see, there are many blocks.These are just the most important ones for this example, but even

so, there are many of them

1

Figure 1-1 A simplified Kubernetes architecture, with its more impor‐ tant services

The API Server works like a gateway that handles the API request.Based on the request, it reaches out to other components like etcd,kubelet, and the container runtime interface

Let’s take a look at them very quickly:

• etcd is a database developed by CoreOS and is not part of theCloud Native Computing Foundation (CNCF) It is used tostore all the stateful information required by Kubernetes

• kubelet is in the node scope It runs on every server or virtualmachine (VM), and it saves actions where it is running

• kubelet doesn’t “create a container”; that is not within its scope

To execute actions on a container, it communicates with what

we call the Container Runtime Interface (CRI) Docker is themost popular of them

If you are wondering how many components comprise Kubernetes,the answer is, a lot! And this is why it is complex The goal forKubernetes is to expose a layer of abstraction to deploy and managethe application life cycle All of the components you see on this page

are more or less easy to swap out to take advantage of a concrete ser‐vice provided by your infrastructure or software provider

Today, all cloud providers offer a Kubernetes-managed service, buteach of them runs it in a different way and with different integra‐tions

For example, Kubernetes 1.13 CoreDNS is the default DNS service,but if you are running on AWS, you can rely on Route53 Thismeans that you won’t manage the reliability of CoreDNS if you are

happy to use a managed version of it And I suspect that AmazonElastic Container Service for Kubernetes (Amazon EKS) usesRoute53.

This is why, in such a short amount of time, all of the cloud provid‐ers delivered a “stable enough” Kubernetes service They were able

to replace core Kubernetes components with those they developed

or ran

You can do the same In the beginning, Docker was almost theunique container runtime engine, but now there are alternatives,and as soon as they implement the CRI, you can use whichever youthink is better: CRI-O or containerd-CRI, for example

As I mentioned, this report focuses on the Kubernetes API—the oneprovided by the API Server—but there are APIs in all of the layers ofKubernetes

Unified User Interface for Operations

I am a software engineer, but I now work in the cloud and automa‐tion area and have done so for a couple of years Let me tell you why

I love what I do My goal is to make my colleagues happy and com‐fortable when interacting with their environments—building toolsthat make delivery, life cycle management, and observabilityfriendly

That’s why I am always against random Bash scripts in some reposi‐tory I like to provide a good feeling to my teammates, which I call

UX for Operations.

In practice, this means using a unified CLI or web UI across theboard to interact with the huge number of APIs provided by thethird-party services that, these days, we all use in operation

Kubernetes helps with that because, as a framework, it offers corre‐lated features and tools that you can include for free with a wiseintegration of your flows and resources Some of them are covered

in this chapter; others are covered in later chapters Here’s a list ofthem:

As you will see in Chapter 2, the authentication method is flexibleand powerful, you can authenticate applications running within oroutside of a Kubernetes cluster Combined with its authorizationmechanism, your integration will be far more secure just utilizingfeatures provided by Kubernetes.

Authorization

Authentication is but one part of the security paradigm When yourcompany begins to grow, you will need to group users according towhat each user’s capabilities are, based on roles For example, a sub‐set of users will need full access to create, delete, list, and monitorresources, whereas others will need read-only access to a subset ofresources

Do not make the mistake of thinking of authorization as somethingthat puts you or other colleagues behind somebody else Prevention

is important to make everyone comfortable interacting with a sys‐tem safely, and it increases confidence because you can be certain toavoid human mistakes like removing an entire namespace or agroup of nodes

Out of the box, Kubernetes addresses granular authorization byusing Role-Based Access Control (RBAC), a popular authorizationmechanism You can enable it by configuring your API Server usingthe flag authorization-mode=RBAC

In the Kubernetes RBAC API, a role contains rules that represent aset of permissions such as the following:

• The rule called pod-reader allows a user to grant get, watch,and list capabilities on a pod resource.

• The anonymous rule does not allow a user to do anything in thesystem

The permissions are additive, meaning that you cannot denyactions; you can only list what a role can do You can set a Role ifyou need a per-namespace granularity or a ClusterRole for anoverall cluster authorization

Following is an example copied from the kubernetes documenta‐tion:

verbs: ["get", "watch", "list"]

This Role is in the default namespace: it allows a user to list,

watch, or get pods You can assign to a user or an application only access to the pods in the default namespace

read-Let’s revisit the earlier autoscaling group example An autoscaler hasproperties such as Replicas that specify the number of nodesdeployed Removing an autoscaler is dangerous because it removesall of the Kubernetes nodes Integrating the autoscaler in Kuberneteswill help you to grant the authorization that your colleagues willhave on the resource You can give them the ability to only get oronly edit an autoscaler, avoiding accidental deletion or limitingaccess for those who don’t have the appropriate scope to understandthe implications for the entire system

Audit Logs

In operation, it is very important to understand who saved anaction This is not easy today because there are a lot of third-partyservices, APIs, and command-line interfaces (CLIs) to aggregate.Kubernetes provides detailed audit logs about what is going on andwho triggered that action

Unified User Interface for Operations | 5

The API-Server is responsible for generating and forwarding auditlogs to a backend based on custom audit policies.

Kubernetes generates an enormous number of events, but you canuse policies to filter what you need to forward and what you do notcare about You can catch these audit logs via log files or webhooks.All of the actions saved on your custom resource definition willbecome part of the audit logs In this way you will be able track who

is using them and how

You have the option to combine which authentication, authoriza‐tion, and audit logs that you would like to share with others Let’sassume that you’re on Amazon Web Services (AWS) and you need

to scale up an autoscaling group because you need to handle moretraffic You can log in to the AWS Console and change the valuemanually, but you will then lose track of the event, and nobody willknow who or why you did it if you do not use AWS CloudTrail, forexample Nowadays, with distributed companies, asynchronouscommunication is a requirement A lot of startups and companiesemploy the “run fast and break things” mindset for their site, butthis doesn’t mean that we should hide who performs an action

To avoid this issue and also to unify the user experience (UX) inoperation across your team, you can extend the set of objects sup‐ported by Kubernetes such as pod, PersistentVolume, Deployment,and StatefulSet with a new one called AutoScalingGroup This newobject is a bridge between the AWS Autoscaling Group and Kuber‐netes Using this, you can scale them up and down just as you wouldfor a pod via kubectl, for example, but Kubernetes requires a token

to accept this change, which means that the action has an owner,and it can be referenced with an AWS user, as well You can haveaudit logs from AWS, but the goal here is to establish a way to con‐nect all of the tools you use to do automation and to manage ourinfrastructure in order to have the simpler experience

As a bonus, potentially you can avoid people having access to theAWS Console if they don’t need it because the complexity will behidden by your integration

Events

Kubernetes is based on events Almost everything happens asyn‐chronously: when you ask for a pod, your request is stored and a

queue will pick it up, and then it will create the pod as youdescribed The process generates a lot of events that you can listen tovia kubectl:

$ kubectl run nginx image=nginx port=80

$ kubectl get event -n default

LAST SEEN TYPE REASON KIND MESSAGE

56s Normal Scheduled Pod Successfully assigned 56s Normal Pulling Pod pulling image "nginx" 54s Normal Pulled Pod Successfully pulled image 53s Normal Created Pod Created container

53s Normal Started Pod Started container

56s Normal SuccessfulCreate ReplicaSet Created pod: nginx-cdb6 56s Normal ScalingReplicaSet Deployment Scaled up replica set

kubectl is just one way to follow them If your Go application needs

to listen on a particular event, you can do that by using the kuber‐netes/client-go This is one of my best friends at work If you startwriting applications in Go that interact with Kubernetes, you will get

to know each other very well Many of the examples that you will see

in this report are in Go because it is the language with which I ammost familiar

I go into much more detail about how Go works in Chapter 2, butI’ll give you a taste here In the past we used to have a repeating andannoying issue with pods using PersistentVolumes for which thesolution was to restart the pod Sometimes, you do not have time tolook at the root cause, and you need to unblock your team Or evenworse, you know the solution but you need to patch Kubernetes byyourself In the meantime, I hacked a small Go application that lis‐tens to pod events and identifies the particular error: "MountVolume.WaitForAttach failed for volume" If this happens, theapplication is going to restart the pod by itself:

// You need to set the kubernetes client.

Unified User Interface for Operations | 7

1 A good friend of mine opened an issue about this particular error As of this writing, the solution is still forthcoming, with various workarounds.

// it is not part of this example

func newEvent(kbc *kubernetes.Clientset) func(obj interface{}) {

return func(obj interface{}) {

a new event is triggered.1

The event is associated with a particular resource It can be a specificpod or an instance of your custom resource definition You can seethis when you describe it in the Events section (the last one):

$ kubectl describe -n default pod <pod-name>

Normal Started 36m kubelet, node.internal Started container

When you include your resources as part of Kubernetes, you get thisevent framework available for you to use If you are the developerwriting the new resource, it helps you to implement it in a resilientand scalable way, using queues and triggering events If you are adeveloper who needs to integrate logic when a particular event hap‐pens in Kubernetes, you will know how to approach the problemeven if it is a Kubernetes-native resource such as a pod or a Deploy‐ment or something implemented by another teammate, such as theaforementioned autoscaler

Conclusion

As we close this first chapter, I hope that you can see how much eas‐ier Kubernetes makes useful features and why you should thinkabout integrating your flow behind its UX

You can make Kubernetes a commodity to achieve a simpler UX,stability, and security

Conclusion | 9

CHAPTER 2

Client Side

Now that you have a basic understanding of why Kubernetes’ exten‐sibility is so helpful in providing a solid user experience (UX) forOps, in this chapter we explore the Kubernetes API and the pro‐vided software development kits (SDKs)—in particular the GolangSDK We also look at how to write a kubectl plug-in

kubectl Plug-Ins

The first way to extend Kubernetes is via kubectl plug-ins Plug-insare a kubectl feature marked as stable in Kubernetes 1.14; you canverify your version by using kubectl version to run the appropri‐ate binary These are binaries in your $PATH that have the prefix

kubectl- I think Ops people deserve love and appreciation; here’s aplug-in that everyone should deliver to their laptop:

echo "Remember You are awesome!"

As you can see, this is nothing more than a Bash script Let’s call it

kubectl-awesome and make it executable:

$ chmod +x /kubectl-awesome

$ mv /kubectl-awesome /usr/local/bin

$ kubectl awesome

11

Remember You are awesome!

$ kubectl awesome version

stable-as-you

You cannot override kubectl commands using this trick A binarycalled kubectl-create-awesomeness won’t work because you cannotoverride the kubectl create command Usually the common pathhere is to prefix your command with the subject of the plug-in I amincreasing the coefficient of awesomeness! My plug-in is called

kubectl-awesomeness-create and it simply increments the currentavailable awesomeness:

$ kubectl awesomeness create

awesomeness = awesomeness + 1

It is again a Bash script in my $PATH called:

$ cat /usr/local/bin/kubectl-awesomeness-create

#!/bin/bash

echo "awesomeness = awesomeness + 1"

kubectl plug-ins are flexible and easy to write, but you do need tothink about how to distribute them I recommend forcing yourselfand your team to download kubectl (or only the plug-ins) from asource that you control This way, you will be able to distribute all ofthe plug-ins provided by you and your team You can use a privaterepository like Nexus or you can download them via an object storesuch as Amazon Simple Storage Service (Amazon S3) and a goodinstallation script These efforts will pay off in the long run becauseyour teammates will have a framework to reuse and develop checks

or shortcuts There is a good amount of operational experience lost

in everyone’s laptop; this is a strategy that will help to share it out inthe open

Here are some of the kubectl plug-ins that I like most and that Ihope will inspire you to write your own:

• iovisor/kubectl-trace helps you to debug containers loadingeBPF programs

• aylei/kubectl-debug is very popular and temporarily injects con‐tainers inside a pod to simplify the troubleshooting of a runningapplication

• eldadru/ksniff is a bridge between tcpdump and Wireshark to acontainer running within your cluster

API and SDK

Even if Kubernetes has a very high number of objects with a lot ofdifferent purposes, such as pods, Ingress, Service, StatufulSet, Job,and CronJob, all of them have fields in common, such as API ver‐sion, kind, labels, and annotations These common parts are usefulfor identification purposes, and also when you build automations,since you know these values will always be there Let’s take a practi‐cal example This is the YAML specification for the Service Kuber‐netes Resource In this section, we examine how a KubernetesObject is structured, and we highlight the most important fields usa‐ble to build automation:

All of the objects (also called resources) have a standard set of values:

• kind identifies the type of resources (pod, StatefulSet, Deploy‐ment)

• apiVersion describes resources provided by the same entity,and it is also used for versioning; for example:

— rbac.authorization.k8s.io/v1 can be read as “resourcespart of the rbac.authorization.k8s.io group, version 1.”

— batch/v1beta1 can be read as “resource part of the batchgroup, version v1beta1.”

As you can imagine, these two values are a requirement for everyresource that you will get from Kubernetes, and they also help theclients decode the JSON blob to the appropriate object

API and SDK | 13

There are other important metadata that are often used to take

advantage of automation: labels and annotations.

If you look at the structure, you will see that annotations and labelsare both key values, but annotations have the scope to track externalautomation The scope of labels is to categorize resources in sub‐groups

In the previous code example, you can see the label owner: sre.This categorizes everything that is owned by a specific team—in thiscase the sre team

You can filter contents by labels:

$ kubectl get service -l owner=sre

The annotation external-dns.alpha.kubernetes.io/hostname is

an example I took from a Kubernetes extension called external-dns.This extension watches Ingress and Services, and if it detects a par‐ticular annotation, it creates a DNS record to one of the supportedproviders such as AWS Route53, CoreDNS, CloudFlare, DigitalO‐cean It also manages its life cycle: when the Service is slated fordeletion, external-dns will remove the associated DNS record.You will find a lot of applications that use annotations to triggertheir automation, and you should remember to use them when youwrite your own code In Chapter 3, we look at informers, and I godeeper on how to monitor resources looking for those annotations

It is impossible to cover in detail how the Kubernetes API works inthis short report, but there are a couple of articles that you shouldread and that will help you to have a better understanding on thestructure of the API:

• API Conventions is about naming conventions, error codes, andvalidations This article will provide you with a better under‐standing of the overall structure of the API request andresponse

• The Kubernetes API is a short introduction directly from thedocumentation

There are a lot of client libraries, some of them officially maintained

by the Kubernetes organization, and others from the community, asdemonstrated in Figure 2-1

Figure 2-1 A sample list of Kubernetes client libraries (the full list is available at the Kubernetes website )

All of them have great documentation You can figure out whichversion to use based on your Kubernetes version by just looking atthe main README in the GitHub project

The APIs are described via OpenAPI (better known as Swagger) It

is a way to specify an API via YAML/JSON, and from there you cangenerate server and client code, documentation, test cases, and so

on It is very popular, and if you create a lot of APIs, I suggest thatyou to give it a try But let’s look at how Kubernetes uses OpenAPI tomanage documentation and to generate its clients in different lan‐guages

You can find the Kubernetes API OpenAPI documentation on theGitHub repository It is very long and difficult to read, but almost all

of the generated clients adhere to this definition For example, if youopen the Java Client library at the path java/kubernetes/src/main/

java/io/kubernetes/client/models/, you will find a list of directories

that represent the Object supported by the client split by ApiVer‐sion, as shown in Figure 2-2

API and SDK | 15

Figure 2-2 The kubernetes-client Java repository that shows generated models to interact with Kubernetes

If you open one of these files, you will see that the header includesthe annotation “NOTE: This class is auto generated by the swaggercode generator program.” You will find the same in the Python Cli‐ent Library under kubernetes/client/apis/; all of the files have the fol‐

lowing header:

# coding: utf-8

"""

Kubernetes

No description provided (generated by Swagger Codegen

OpenAPI spec version: v1.14.1

This is all generated code Choose one and see how it works Don’t

be concerned: it is an HTTP client just like all of the other ones youhave used in the past to interact with other services Chapter 3 is allabout the Go client library Here, however, I show you how to create

a small program that lists the running pods for a particular name‐space If you are not familiar with Go, the example is very easy andyou should be able to convert it to your language of choice