IT training oreilly edge resiliency khotailieu

5 Vulnerability When the Internet Is Your Network Backbone 5 Virtualization and Outsourcing of Services 6 Vulnerabilities Within Your Own Organization 8 Looming Security Threats 9 Unpred

Gary Sloper & Mark Wilkins

Managing Volatility Through DNS

Edge

Resiliency

Compliments of

Gary Sloper and Mark Wilkins

Edge Resiliency

Managing Volatility Through DNS

Boston Farnham Sebastopol Tokyo

Beijing Boston Farnham Sebastopol Tokyo

Beijing

[LSI]

Edge Resiliency

by Gary Sloper and Mark Wilkins

Copyright © 2018 O’Reilly Media All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or

corporate@oreilly.com.

Editor: Virginia Wilson

Production Editor: Melanie Yarbrough

Copyeditor: Octal Publishing Services,

Inc.

Proofreader: Christina Edwards

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Rebecca Demarest September 2018: First Edition

Revision History for the First Edition

2018-08-30: First Release

This work is part of a collaboration between O’Reilly and Oracle Dyn See our state‐ ment of editorial independence.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Edge Resiliency,

the cover image, and related trade dress are trademarks of O’Reilly Media, Inc The views expressed in this work are those of the authors, and do not represent the publisher’s views While the publisher and the authors have used good faith efforts

to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains

or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents

1 Edge Resiliency Is Critical to Your Business 1

What You Will Learn 3

Intended Book Audience 3

2 Exposing Buried Threats to Your Business Network 5

Vulnerability When the Internet Is Your Network Backbone 5

Virtualization and Outsourcing of Services 6

Vulnerabilities Within Your Own Organization 8

Looming Security Threats 9

Unpredictable, Uncontrollable Problem Sources 11

Conclusion 12

3 Strategies to Meet the Challenges 13

Strategy 1: Consider the End-to-End User Experience 13

Strategy 2: Embrace Processing at the Edge as Part of Your Total Design 14

Strategy 3: Engage with Your Cloud Provider to Arrive at the Optimal Topology 14

Strategy 4: Increase Redundancy and Reliability with Multicloud and Hybrid Cloud Strategies 15

Strategy 5: Involve DevOps Staff in All Aspects of Edge Services Planning and Implementation 16

Strategy 6: Inject Chaos to Find Weaknesses Before They Affect Customers in Production 16

Strategy 7: Use Managed DNS Functionality to Limit Endpoint Exposure and Network Volatility 18

Conclusion 19

iii

4 Managed DNS Services 21

Benefits of DNS 21

DNS Routing 23

When to Consider a Managed DNS 25

Conclusion 29

5 Choosing a Managed DNS Provider 31

Evaluation Period 31

Business-Critical Availability 31

A Focus on Security 32

Support 32

Easy-to-Use Tools 32

Conclusion 33

iv | Table of Contents

CHAPTER 1

Edge Resiliency Is Critical

to Your Business

In today’s 24/7 global business environments, resiliency is not only

an assumption by your customers, it’s a requirement for your suc‐cess Simply defined, IT resilience is an organization’s ability tomaintain acceptable service levels, no matter what challenges arise.From CTOs to networking IT staff, the threats and challenges toservices that live at the edge of the network—including both the useredge and the site edge—pose the potential for unplanned and cer‐tainly unwanted business disruptions

To understand the implications and solutions of resiliency at theedge, we first need to understand what the term “edge” really means

here In reality, there are multiple edges The user edge is where the end user sits and first interacts with the internet The network edge is

in front of the content or service that the user is trying to reach(think transit, content delivery network [CDN], domain name sys‐

tem [DNS], and so forth) The site edge is typically at the datacenter

or cloud infrastructure where the content or service resides Yourgoal is to get control as close to the user edge as possible

Sources that trigger instability for the myriad internet services thattoday’s enterprises depend on range from simple misconfigurations,

to large-scale natural disasters, to nefarious targeted attacks, as well

as business-driven internet routing decisions to meet traffic andsovereignty requirements The user edge is your customers’ first(and possibly only) interaction with the application or service that

1

they’re trying to access It’s where impressions are made—or fail.Traditionally, companies have focused on the user experience asthey interacted in expected, or unexpected, ways across the network.However, just as important, each edge location can also be a portalfor instability and threats These can come from unintentional sideeffects such as attempts to meet high-traffic requirements, physicalinfrastructure challenges (e.g., from a natural disaster), or deliberateattacks from bad actors.

The simple reality is that if your company relies on cloud-hostedapplications, which more and more are these days, internet volatilitynow has a greater impact on your business than at any given time inthe past The large numbers of medium-to-large enterprises thathave been moving into hybrid and multicloud implementations onlymagnifies the scope and likelihood of an impact

For the past few years, medium- to large-sized enterprises have beentransitioning away from doing everything in-house to using hostingproviders to support a sophisticated global presence This is a natu‐ral evolution as organizations scale, so this book will touch on thedue diligence they need to perform; the problems they mightencounter; and what they can do to optimize their performance,security posture, balance workloads, and steer traffic more effi‐ciently in a hybrid cloud or multicloud environment

The shift from hosting corporate applications on-premises to usingcloud-based service providers is an accepted practice for doing busi‐ness today And, like any key corporate resource, companies need tosafeguard and protect it Network resiliency (especially at the useredge) is your insurance policy against internet-based disruptions.Additionally, more organizations have begun to deploy multicloudenvironments using additional vendors or a private infrastructure tosupport their businesses This dynamic will continue to grow, takingadvantage of diversity and performance-based cloud services.Granted, when you depend on internet services that are a “blackbox,” some aspects will be out of your direct control In those areas,your business must rely somewhat on trust—trust in those who haveconstructed today’s complex internet, trust in the partners you workwith, and trust that the infrastructure you’ve invested in will mostlywork reliably and appropriately However, trust is not a strategy:24/7 global businesses face new exposures each day To combat thesechallenges, businesses must take responsibility for resiliency In this

2 | Chapter 1: Edge Resiliency Is Critical to Your Business

way, they can gain direct control to insure against the risks And itall starts by understanding the approaches that you can take toaccomplish this goal.

What You Will Learn

In the remaining chapters, we discuss these approaches and offerinsight and strategies for creating resiliency at the edge The goal is

to stabilize internet volatility, whatever the source The critical topics

we cover include the following:

• Recognizing volatility sources

• Optimizing performance and balancing workloads amid inter‐net volatility

• Steering traffic more efficiently

• Strengthening your security posture—not just in a traditionaldatacenter, but also in a hybrid and/or multicloud environment

• Working with DNS infrastructure, managed DNS, and edgeservices

We discuss common challenges and present clear examples todemonstrate the benefits of using managed DNS infrastructure tostrengthen edge resiliency And we offer assessment criteria forwhen you are deciding whether to incorporate a managed DNS pro‐vider into your resiliency strategy This, will, in turn, provideoptions and strengthen your ability to manage, challenge, and workaround any internet threats, disruption, or volatility

Intended Book Audience

We wrote this book for IT managers to help them proactively enable

a resiliency strategy in the face of planned and unplanned eventsfrom the user edge to the applications and services those users aretrying to reach Our goal is to help you prevent challenges that couldhave a negative impact on customer satisfaction and business out‐comes Business leaders must be aware and plan for these challengesbefore they happen, because today, our customers, our employees,and our reputations are all “living on the edge.”

What You Will Learn | 3

as high-availability and 99.999% uptime are tossed out as absolutes

in sales conversations and customer engagements Yet, the basis forsuch assertions is uncertain at best and completely unrealisticwithout a plan for dealing with services at the edge

In this chapter, we survey the classes of challenges to the networksand applications that your business and your users depend on Iden‐tifying these challenges can help you to see where you are exposed—and where you need to focus resources so that your customers aren’texposed to the effects of their disruption

We begin at the lowest level—the systems that underlie the datachannels you depend on

Vulnerability When the Internet Is Your

Network Backbone

The internet is based on a set of strategically connected “backbone”networks that are based on localized nodes The nodes are, in turn,based on other systems and many smaller networks connectingthose devices The key communication components that allow theinternet to function are managed and owned by a combination oftelecommunications (telco) companies, ISPs, and leased or pur‐

5

chased fiber implementations that provide connectivity from point

to point—all with their own vulnerabilities Though these core sys‐tems are hardened with monitoring and security measures, they arenot entirely insulated from internet volatility due to the multitude ofinterwoven and interconnected parts

Even in an environment in which you pay for dedicated cloud serv‐ices, the public transit network is rarely within your control beyondthe terms of service The immediate network resources your busi‐ness relies on might be totally owned and managed by the cloudprovider Or they might be dependent on or farmed out to a combi‐nation of multiple private companies that depend on other vendors.Each link in this critical chain must plan for and manage potentialimpacts, such as scheduled maintenances, aging equipment, turn‐over of support staff, and evolving technology

Even if the network components are managed and sound, that initself is not enough Today’s businesses don’t just run directly on aphysical box in a datacenter More and more applications and envi‐ronments are being virtualized, losing the distinction of how andwhere exactly they run In this kind of ephemeral environment, it ismore important than ever that we understand the virtual networklandscape This is the subject of our next section

Virtualization and Outsourcing of Services

Above the core of your network are the systems and applicationsthat run your business You might still have some dedicated hard‐ware within the territory that you own and operate, but these days it

is more common for the systems to be virtualized and running inthe hosted cloud As long as you have a basic “map” to guide you asyour applications are deployed, you might feel that you have fewerconcerns But, at the same time, you have less control because youcan’t always get to the actual systems themselves given that the cloudprovider manages them This paradoxical “less is more” implemen‐tation forms another point of interplay with your edge services thatmust be considered

Perceptions in these areas have had to evolve along with the tech‐nologies A few short decades ago, it was common for companies tohave large datacenters with on-premises hardware and staff to man‐age the targeted needs of the business Problems were usually local‐

6 | Chapter 2: Exposing Buried Threats to Your Business Network

ized and could be addressed within the infrastructure that thecompany managed.

Now, the physical datacenters of those days have more regularlybeen replaced by cloud-based systems running on hardware (oreven other virtual systems) in a remote datacenter that a third partymanages The “edge” is becoming the new “core.” The advantages ofthis change are evident:

• Businesses can redeploy former large datacenter staff to shift tocloud functions

• There is less need for specialized physical environments to sup‐port hardware

• There is less space required for a computing environment

• Businesses can focus on their core strengths

A corollary of this is outsourcing the key software applications abusiness uses This is commonly known as the “as-a-service” model.The three variants of this model in primary use today are software

as a service (SaaS), platform as a service (PaaS), and infrastructure

as a service (IaaS)

Such services can be provided for our use over the internet by any ofmultiple public cloud providers These examples illustrate that busi‐nesses often rely on cloud providers not only to host the applica‐tions they produce for customers, but—more and more—also toprovide the environments that they use to develop those applica‐tions In effect, businesses have not only exchanged the systems forcloud-based infrastructure, they have also created business-dependent partnerships with the service providers

Although we all like to think that we are unique and are a primaryfocus in these relationships, it is important to keep in mind thatselected partners are frequently managing requests, volatility, secu‐rity concerns, and other issues from hundreds, or even thousands,

of other customers Again, aside from the service-level agreement(SLA) terms you have agreed to, you have very little overall control

of the cloud provider

For all cloud scenarios, selecting partners that have the right level ofinternet experience is important The partner might offer tremen‐dous value in terms of services offered, but if it lacks experience

Virtualization and Outsourcing of Services | 7

dealing with the virtual and physical layers, your business could beleft at a significant disadvantage.

The key point to remember here is that no partner or service pro‐vider in the cloud will be as much of an expert on your business orcustomer needs as your own organization will be And, in the samevein, your organization must be the most committed to ensuringthat your customers can use your services even if the cloud servicesyou depend on fail This starts with approaches such as those that

we outline in Chapter 3 The plans you make now for this sort ofresiliency become the map that will safely guide your business acrossany “uncharted territory.”

In addition to constructing our “global view” of the cloud pieces that

we rely on, we must also have an “eyes wide open” view of our owninternal organization With the incredible number of moving pieceswithin companies today, it’s important to identify the potential risks

to stability where we can

Vulnerabilities Within Your Own Organization

Human errors occur in all organizations They are, for the mostpart, an accepted part of doing business, and it is likely that most donot rise to the level of affecting large sets of customers Enterpriseshave traditionally held the mindset that human errors affect mainlytheir internal systems And, as such, they are recoverable by havingbackups However, today’s systems offer more options and function‐ality than ever before Correspondingly, they can require more con‐figuration, understanding, and care than ever before Whereresources within a business need to oversee such complex environ‐ments, there is always a risk

For example, a simple typo in a server configuration could result indirecting incoming traffic to the wrong page Or incorrectly alteringthe schedule for a backup process could affect the availability of theresource during times of high demand

Enterprises must expect that, at some point, a human error willaffect the way customers will interact with them Waiting until thishappens and then reacting is a risky approach The user mightremain a customer if there is enough value beyond the inconven‐ience Or they might look elsewhere A better strategy is to arm yourservices living at the edge with the ability to detect and tolerate such

8 | Chapter 2: Exposing Buried Threats to Your Business Network

1 Jason Del Rey, March 2, 2017 “Amazon’s massive AWS outage was caused by human error”, recode.net.

2 Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region

disruptions and have intelligent responses in place to steer traffic.They can automatically work with the cloud services to recoverfunctionality with minimal downtime

With those internal to our organization, we are usually more con‐cerned about preventing accidental misuse rather than malevolentintent However, we must also guard against those intentionally tar‐geting today’s technology with negative intents What you don’t see(especially in your network) can hurt you

Small Error, Big Impact

On February 28, 2017, the Amazon Simple Storage Service (Ama‐zon S3) became unavailable in the Northern Virginia (US-EAST-1)Region The impact was substantial: “During AWS’ four-hour dis‐ruption, S&P 500 companies lost $150 million, according to analy‐sis by Cyence, a startup that models the economic impact of cyberrisk US financial services companies lost an estimate $160 mil‐lion ”1

This was not a problem with the systems, which have an extremelyhigh level of reliability In fact, they functioned exactly as intended.Rather, this was caused by simple human error

“At 9:37AM PST, an authorized S3 team member using an estab‐lished playbook executed a command which was intended toremove a small number of servers for one of the S3 subsystems that

is used by the S3 billing process Unfortunately, one of the inputs tothe command was entered incorrectly and a larger set of serverswas removed than intended.”2

Looming Security Threats

Today, more than ever, the internet offers a place for bad actors tohide as they try to manipulate others, illegally obtain data or goods,

or break or deny access to key parts of your business

Looming Security Threats | 9

The growing security threat from nefarious activities can wreakhavoc with networks and compromise your environment from theedge inward These attacks at the edge can come in many forms,such as the following:

• Distributed Denial of Service (DDoS) attacks These types ofattacks amplify the traffic directed to a website to the point atwhich the website systems are unable to keep up, and thus fail

• Malicious bots These are self-propagating malware programsthat infect systems to gather information, open backdoor access,

or launch attacks against systems They typically also connectback and report to a central server

• Attempts to hijack routes, IP addresses, or URLs to redirectusers to other websites or content

• Targeted inputs to circumvent checks or provoke error condi‐tions to gain internal access to the system and its data

• Ransomware attacks Bad actors trick users into installing soft‐ware that encrypts data, and then demand payment to unen‐crypt it

Increasingly, these security threats are automated and come fromgeographically diverse sources, disrupting the traffic highways of theinternet with no human involvement It is rare these days to go formore than a few weeks without hearing about a well-known organi‐zation having its data stolen or becoming the victim of a DDoS orother malware attack Entire businesses have been held hostage byransomware attacks; some have not survived

Guarding against these dangers demands more automated and pro‐active vigilance than at any other time in the history of the internet.Consideration and response planning for the different kinds ofattacks are essential before they happen Security and penetrationtesting (an authorized, simulated attack on a public website) are nolonger optional security checks This mitigation of an attack formsanother part of our edge resilience Web application firewalls, botmanagement solutions, network-based DDoS protection, and DNSall combine to help build a more resilient infrastructure Businessescan no longer afford to ignore who or what is trying to get into theirwebsite but should have a plan in place to address it should it occur.Just as a board of directors would want to stay in compliance with

10 | Chapter 2: Exposing Buried Threats to Your Business Network

IRS regulations for the organization, it should equally comply withbest practices from resiliency to security at the edge.

Unpredictable, Uncontrollable Problem

Sources

Beyond the quantifiable challenges, there are unexpected factorsthat affect internet service, and thus your business, if your applica‐tion design does not include resilient components designed for fail‐over at a moment’s notice

One such force that can strike businesses anywhere and at any time

is the weather Each year, we see many examples of how quicklyweather can bring about natural disasters with unpredicted impact.Many organizations gain experience in coping with these eventsthrough trial and error as they occur Take for instance when Hurri‐cane Sandy hit the New York City area in 2012 That storm and theensuing floods cost telco companies and their customers significantdowntime, not to mention the repair costs and revenue losses Astudy found that internet outages in the United States doubled dur‐ing Hurricane Sandy—up from a daily outage average of 0.3% to0.43% and taking four days to return to normal levels

Even telco providers themselves can be a source of internet stabilityissues Scheduled or unscheduled maintenance can disrupt internetservice if you have not planned for these occurrences All internet-based services need updates, and your application must be able todeal with these when they occur What’s more, because many telcoproviders are operating in a fiercely competitive environment, theyare continuously asked to do more with less as their profit marginscontinue to shrink On a practical level, this can mean that legacyequipment is not being replaced and updated Consider, for exam‐ple, the prevalence of copper-based telco cables In harsh climateswith heavy rain and snow, these in-ground cables break down,which has a negative effect on transmitting data via the internet andcan lead to outages or packet-loss issues that are especially insidiousand difficult to troubleshoot

These are just a few examples of the kind of events that your busi‐ness cannot control as you navigate today’s decentralized world Butyou can control the resiliency of your application; there are manytools available to ensure that your applications remain stable and

Unpredictable, Uncontrollable Problem Sources | 11

available regardless of the circumstances In Chapter 3, we offersome strategies.

Conclusion

Your presence on the web is key to the power and reach of yourbrand and the success of your business Disruptions that are nothandled gracefully can cause your customers to consider the compe‐tition and ultimately jeopardize your bottom line Regardless of thecause of the disruption, the site edge is where users will likely firstfeel any disruptive effects and first target their blame Cultivatingdeliberate awareness of the challenges facing your network andapplications puts you in a better position to take action and safe‐guard your users against them

12 | Chapter 2: Exposing Buried Threats to Your Business Network

CHAPTER 3

Strategies to Meet the Challenges

The scenarios outlined in Chapter 2 illustrate that although thecloud paradigm has become an always-available, easily accessibleendpoint for users, it can also represent a somewhat murky andmysterious platform inherent with unseen risk for many businesses

It is no longer sufficient to simply deploy applications into a cloudand assume that the end-to-end user experience will be what weexpect Where the cloud is insufficient to cover the risks, we mustmove more responsibility (and thus more reliability) toward resil‐iency at the user edge

Edge services were once the endpoints, gateways, interfaces, androuters located on our in-house networks Today’s edge services arehosted in the public cloud and must now be more intelligent, strate‐gic, and fault-tolerant than ever before They must not just allow ourusers to access applications when required; they must also allowthem to stay securely connected and able to complete their transac‐tions

In this chapter, we look at some strategies around edge services tomeet these challenges and provide users with the stable, reliableinteractions they expect

Strategy 1: Consider the End-to-End User

Experience

In any interaction with your website, application, or network, theuser’s experience establishes an impression of your business Take

13

some time to evaluate the scenarios that a user might encounter ifyour cloud provider encounters problems How able are you to pro‐vide a reasonable experience to the user until the situation can beresolved? Is fallback to a different provider or other endpoint feasi‐ble?

Thinking ahead to what kind of situations your users mightencounter in the event of a problem with your cloud environmentcan provide valuable foresight And, it can be the starting point toupdating your edge services to be able to compensate for issuescloser to your application

Strategy 2: Embrace Processing at the Edge as Part of Your Total Design

As edge services and edge devices become more advanced and pow‐erful, it is an oversight not to consider and take advantage of theirfunctionality as part of an overall edge-to-cloud deployment strat‐egy For example, in many cases, Internet of Things (IoT) edge devi‐ces can run analytics at the edge to produce useful, more compactdata rather than having to send it to the cloud for processing Pro‐cessing at the edge can reduce and/or complement processing thatwould normally be routed to the cloud This can also provide apathway for processing to continue even if the cloud functionality isdisrupted

Strategy 3: Engage with Your Cloud Provider

to Arrive at the Optimal Topology

Moving to the cloud can mean moving into a world where every ser‐vice is hosted and managed by the cloud provider Cloud servicesare usually designed in a security model with a shared level ofresponsibilities between the cloud provider and the customer Riskexposure can be best minimized by having open conversations at thebeginning of a relationship with any internet and managed DNSprovider with whom you partner

Conversations should not shy away from the questions that need to

be asked, such as these:

• How can we help address the challenges we face getting ourapplications closer to our users?

14 | Chapter 3: Strategies to Meet the Challenges