MySQL High Availability- P11

Rather than say “cloud computing is many things to many people,” the National stitute of Standards and Technology NIST defines cloud computing as follows:Cloud computing is a model for e

CHAPTER 14

Cloud Computing Solutions

Joel pushed his office door partially closed while he hung his jacket on the hook on theback of the door He jumped at the sound of knuckles rapping on his door “Come in,”

he said, as he pulled the door open and walked over to his desk When he turned toface his visitor, he knew who it was “Mr Summerson Good morning, sir.”

“Good morning, Joel Nice work on that report about high availability and scale-out

I especially like your recommendations on how we can improve throughput for some

of our products.”

“Thanks.” Joel held his breath, waiting for the tasking he knew was sure to come

“Last night the board and I signed a contract to customize one of our products for anew customer The ink isn’t dry on the contract, so I won’t go into any details, butsuffice it to say we will need a lot of new servers in a high availability setup Of course,they’ll be using MySQL for the database component.”

Joel tried to remember all the details he’d read about MySQL high availability, dering how much money he’d need to set up a host of servers He snapped out of histhoughts as his boss continued, “ then there’s that stuff about load balancing.”

won-After an uncomfortable pause, Joel said, “Yes, sir.”

“The problem is we don’t have resources to buy a bunch of servers, and the contract isfor a six-month period of service The board will never approve spending a bunch ofmoney on new hardware that we may not need after the contract expires Not to men-tion the dent to our profit margin.”

Joel didn’t know what to say, so he simply waited

“So, we’d like you to put together a cloud-based solution.” With that, Mr Summersonpatted Joel on the shoulder and left Joel stood for several moments before walkingbehind his desk and sitting down

Joel considered himself informed about emerging technologies, but he’d figured thecloud computing stuff was as fluffy as its namesake He picked up his dog-eared MySQLbook and flipped to the next chapter “Well, look at that,” he said, starting to read

The demands of today’s economy present new challenges as well as potentially newsolutions for information infrastructure planners No longer can organizations afford

to simply buy more hardware every time they need to add computing power Whilethe cost of computing hardware has gone down over the last decade, so too have or-ganizations’ profit margins, particularly of late

Thus, organizations must make decisions with a much stronger fiscal burden and lookfor the most affordable services and tools to increase their customer bases by expandingtheir product lines while reducing costs and increasing profits It is all about the money,after all

This need for affordable computational solutions has led technology providers to create

a new way to use computers using a pay-as-you-use philosophy that allows tions to purchase computational and data usage on an ad hoc basis This is the essence

organiza-of what computer scientists describe as cloud computing.

What Is Cloud Computing?

Cloud computing is one of those phrases that is often misunderstood and unfortunatelyhas several—and sometimes conflicting—definitions Some are quick to say it’s abuzzword that describes existing technologies, while others would rather debate thefiner academic and scientific (and sometimes social) aspects, and still others insist cloudcomputing is the future of information technology

Thus, some have flatly stated that cloud computing is nothing more than grid puting, while others suggest cloud computing could represent the entire Internet Bothviews are flawed Others will explain in detail all aspects of “as a service” concepts anduse those to define cloud computing Fortunately, these more nuanced views are muchcloser to the truth

com-Cloud computing is essentially a remix of a group of technologies that include gridcomputing and virtualization combined with application programming interfaces

(APIs) and utilities to supply access to the virtualized environments In his book Cloud Computing Architectures (O’Reilly, http://oreilly.com/catalog/9780596156374), GeorgeReese states, “There is nothing fundamentally new in any of the technologies that make

up cloud computing.” This is a sobering observation that some pundits and marketerswould rather not face However, the advancement comes in the packaging, throughwhich giants like Amazon have made it possible to use existing technologies in a wholenew way

In his treatise “15 Ways to Tell It’s Not Cloud Computing,” James Governor makessome strong statements concerning what is and what isn’t cloud computing To para-phrase his arguments, Governor makes it clear that cloud computing isn’t somethingthat takes a long time to explain, has a complex and steep learning curve, is isolated orrequires a dedicated connection to use, or requires you to purchase hardware Whetheryou agree with him or not, he forces patrons of companies trying to rebadge their

products with the word “cloud” in them to consider their purchases carefully becausethe cloud isn’t what most people think it is.

The term “cloud computing” is derived from the conceptual drawing that depicts sources hosted in a large network (a cloud) We use a cloud symbol because the resourceimplementation (e.g., the hardware, operating system, etc.) is hidden and has littlebearing on what the service is or what it provides—it is simply a service you can use.Thus, instead of seeing gateways, routers, and servers, you see the resources presented

re-as a service Consumers of the resources don’t care how the service is implemented; ofoverarching importance is for the service to solve the need and be available whenrequired

Rather than say “cloud computing is many things to many people,” the National stitute of Standards and Technology (NIST) defines cloud computing as follows:Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, ap- plications, and services) that can be rapidly provisioned and released with minimal man- agement effort or service provider interaction This cloud model promotes availability and is composed of five essential characteristics, three service models, and four deploy- ment models.

In-The characteristics that most cloud researchers define as essential to cloud computinginclude the following:

On-demand self-service

Customers can pick and choose what they need on an ad hoc basis without vention of the vendor or a third party

inter-Broad network access

The resources are available through existing networking capabilities

The three service models are:

Infrastructure as a Service (IaaS)

Resources are provided as virtual instances of complete hardware or operatingsystem platforms The client can add virtualized computational resources on de-mand (e.g., servers, load balancers) Thus, the components of an information in-frastructure are provided as components or middleware The consumer has access

to and control of the resources (e.g., the customer can manage an allocated server)

Platform as a Service (PaaS)

An API allows clients to create applications designed specifically to run on theprovider’s hardware (platforms) The vendor provides the hosting environmentand programming tools to permit customers to build solutions for the specificenvironment

Software as a Service (SaaS)

Software is provided as a resource in the form of applications that run on the vider’s hardware The consumer sees only the interface to the software, just likewith a desktop application The hardware, operating system, etc., are all hiddenand controlled exclusively by the vendor This is the oldest model currently inclu-

pro-ded in definitions of the cloud, and for many decades it was known as an cation Service Provider (ASP).

Appli-The deployment models refer to the availability or accessibility of the resulting solutionsand include:

There are many forms of virtualization If you have ever used Sun’s VirtualBox or crosoft’s VirtualPC, you have used a form of virtualization In essence, this technologycreates a pseudoplatform based on a conceptual computing hardware model For ex-ample, it is possible to run an instance of the Microsoft Windows operating system on

Mi-a Linux mMi-achine inside VirtuMi-alBox VirtuMi-alBox creMi-ates Mi-a softwMi-are-bMi-ased model of eMi-achcomponent in a PC These form a foundation upon which Windows can boot and run

as if it were on real hardware

This is just one form of virtualization There are several mechanisms for simulating thehardware, as well as optimizations for launching, execution, and management of in-stances The virtualization used in most IaaS solutions requires you to use prepackaged

machines (called images), where each virtual machine is called an instance of an image.

For example, the Amazon cloud uses Xen virtualization technology, an open sourceand common solution that permits scaling of the virtualized hardware (e.g., the number

of CPUs), fault tolerance, and other advantages

Furthermore, some vendors let you modify existing images to customize machines toyour needs, using either vendor-specific tools or a machine description in a specificformat This can be an issue if you decide you want to move from one vendor to another:your images may not be portable You should check the vendor’s documentation aboutimages and customizing images before you invest a lot of time and effort in this solution

Grid computing

In the days when computing power was limited and the need to solve complex analytical

or scientific problems was great, a technology that allowed programs to share extracomputing power among a community of connected machines was invented and

termed grid computing It works by breaking a problem down into smaller

computa-tional units that can be shipped to other machines for processing, then retrieving theresults and correlating them on one machine

The key technology that permits these machines to communicate is a sophisticatedqueuing mechanism This queuing mechanism is like a workflow and can be demon-

strated as follows A data manager machine delegates jobs and their data to slave

machines and then reads their results This involves setting up one or more queuingmachines that send the processing job to any connected computers and then returntheir results to the data manager When a user wants to participate in a grid computingprogram, he first connects his donor computer to a queuing machine and issues a pull

of the processing package (a job plus data) His machine executes the job and sendsthe results to the queuing machine A simplified example of this process is shown inFigure 14-1

Queuing systems are also present in cloud computing Thus, it is possible to migrateexisting grid computing solutions or build new grid computing solutions in cloudcomputing This capability is why some insist cloud computing is simply gridcomputing with virtualization But as you will see, there is far more to cloud computingthan just these two technologies.

of time, whereas grid solutions have very small execution times

The good news is that it is possible to build a transactional computing system in thecloud To do so, we must ensure the longevity of the computing resources and providemechanisms to allow the data to be segmented and processed in parallel If you’rethinking, “Hey, this sounds like server farms,” you are correct Most cloud computingvendors provide virtualized resources to support a transactional computing solutionincluding load balancers, persistent instances, and permanent assignment of network-ing resources

Figure 14-1 Grid computing workflow

In this case, the IP address becomes an elastic resource that you can assign to anyinstance you want It isn’t tied to a particular machine Similarly, disk resources can beelastic, so you can store your data on a disk resource and make it accessible to anyrunning instance in the cloud.

Elasticity solves the problem of running virtualized machines in a pooled hardwareconfiguration The machines become truly plug-and-play and you can easily create anddestroy them For example, you can swap a machine running one operating system foranother during development and (perhaps with a few minor changes) still access thesame data—no need to build a whole new database

Software libraries

You may be wondering how all of these technologies tie together and how it is possible

to work with resources in a dynamic environment The answer is that most cloud dors have a specific set of tools for creating and manipulating resources in the cloud.For example, Amazon has tools on its APIs for managing your resources, creating in-stances, creating volumes (disk objects), and much more These encompass the Ama-zon EC2 API Tools for working with cloud resources as well as Amazon EC2 AMI(Amazon Machine Image) Tools for creating and modifying machine images

ven-Similarly, Microsoft Azure has extensions to the NET development environment topermit you to build your cloud applications and run them in the Azure cloud Theresemblance ends here, though, because the Microsoft Azure environment also requiresyou to build your applications with these libraries, whereas Amazon does not

Thus, software libraries form the glue that allows all of these existing technologies towork together and become something greater than their parts—a cloud computing environment

Is Cloud Computing an Economical Choice?

There are analysts and pundits on both sides of this question The bottom line is: itdepends That is, it depends on which cloud provider you use, how many servers youneed (measured in computation time), how much space you need, and for how long.One comparative study shows that the cost of a cloud solution versus a traditionalsolution (buying your own hardware) for a typical e-commerce situation over a period

of five years comes out only slightly in favor of cloud computing

You could see this as evidence that cloud computing doesn’t offer much of a savings,and on the surface you’d be correct However, the details of the study show the initialinvestment for the traditional solution to be very high While it is true at the end of fiveyears an organization owns its own hardware (or amortizes it to the point of obsoles-cence), the hardware in the cloud is of no concern because it is not a factor in the cost.That is, there are no recurring costs for equipment upgrades when using a cloud-based

solution The study does not include this cost in the comparison and if it did, it is likelythat the difference in expenditures over five years would be considerably higher for thetraditional solution.

In order to answer the question of whether cloud computing is economical, you have

to do your homework The cost factor will be different for each organization and foreach project The best way to analyze this cost is to determine (or estimate) the number

of servers, how much data you will need to store, how much data will be moved around

in the cloud, and which additional features you need (VPN, load balancing, etc.) cifically, you need to examine the billable components of the cloud computing vendorand estimate the cost based on those parameters Once you have that, you can deter-mine the costs for a traditional solution, factoring in maintenance and upgrades, andthen make a fair comparison

Spe-Many clients use cloud computing not to save money, but for its flexibility We’ll cuss why it can be a valuable solution in the following section

dis-However, there are some organizations that view the use of cloud computing as hibited either by policy or fear For example, some organizations do not allow storage

pro-of their private data on systems that they do not own and that (conceivably) someadministrator outside of the organization (i.e., one of the cloud vendor’s employees)could use to gain access to the data If you find yourself facing these demons, you shouldcontact the cloud vendor and discuss your concerns and weigh the risks accordingly.One way to overcome this limitation is to segregate your data and use the cloud forpublic data only Also, in the case of Amazon Web Services (AWS), the cloud vendormay have features that permit you to isolate your cloud instances or connect them via

a VPN to your own IT infrastructure (see http://aws.amazon.com/vpc/ for more details

on Amazon’s VPN Cloud offering)

Cloud Computing Use Cases

Now that you have a good understanding of what cloud computing is, let’s examinewhat you can do with it All manner of organizations have found new and exciting usesfor cloud computing These include start-up companies looking for an inexpensiveentry into the market, researchers who need computational power for a limited time,and information technology managers who feel the budget crunch but must still meetthe needs of their users In this section, we discuss some common use cases for em-ploying cloud computing:

Traditional web services

Cloud resources supply content or applications to users on the Internet

Shared services

One or more applications run in the cloud and are shared by different users Anexample is an application that allows partners to collaborate and share data (e.g.,

a supply chain)

Scale-out from enterprise

This allows rapid expansion of an application using solutions that run in the cloudand connect to the enterprise

Cloud bursting

This allows users to deploy temporary resources quickly to solve immediate, term computational tasks

short-Research and development

This allows developers to explore multiple system and application configurationswithout the need for dedicated hardware

As you can see, there are many ways to use cloud computing, and more uses are beingdiscovered every day What we’ve seen in the field so far is just a start to the possibilities

Cloud Computing Benefits

The potential benefits of cloud computing include:

Reduced runtime and response time

By leveraging grid or scale-out techniques, it is possible to reduce the amount oftime a task takes to complete and even improve data access time dramatically Youcan use hardware-based solutions for similar effects, but at a severe investmentcost Cloud solutions allow you to generate as many machine instances as you needand pay only for what you use

Minimize infrastructure risk and maintenance

Hardware failure is no longer your responsibility The vendor owns and maintainsthe machines that run the cloud systems, so you do not require a large staff orinvestment in service providers

Lower cost of entry

With the ability to pay for only what you use, you no longer have to budget for alarge infrastructure that you may or may not use Best of all, you can grow yourinfrastructure on the fly and, better still, reduce it without having to amortize thehardware or declare it surplus

Increased pace of development

The lower cost of entry and the ability to pay only for what you need means youcan get a new application started with far less investment than in the past (on yourown or using a service provider’s hardware) This has the side effect of leveling thefield for start-up companies, allowing them to compete early in their development

Of course, for each advantage there is a corresponding drawback Some of the potentialrisks of cloud computing are:

Service failure

Service level agreements (SLAs) tend to be poorly defined (or nonexistent) in thecloud computing field, and if your underlying service is unreliable, you have littlerecourse

Potential runaway costs

If you experience an unusually heavy load and scale up to meet it, you may succeed

in meeting the need, but incur high usage costs in doing so

Cloud Computing Vendors

Whenever there is an emerging technology, there is inevitably an explosion of vendors,products, and services that all claim in one form or another to provide the new tech-nology Cloud computing is not immune to this phenomenon There are hundreds ofvendors that provide everything from specialized hardware, software services, andplatforms to turnkey pick-and-pay portfolios If you use the NIST definition presented

in “What Is Cloud Computing?” on page 478, you’ll quickly find that many vendorsaren’t meeting all of the tenets of the definition

However, there are a number of vendors that are striving to meet the full definition forcloud computing The following list names 10 of the top vendors, along with a briefdescription of the type of solutions each provides:

An IaaS provider specializing in virtual private data center solutions

IBM Blue Cloud

A cloud computing vendor offering virtualized SaaS, PaaS, and IaaS solutions

Joyent

An IaaS provider specializing in the needs of large enterprises

as a cloud computing solution, it will be a competitor to Google Docs and Microsoft’sonline Office suite plans Clearly, the concepts behind cloud computing are having aprofound effect on many of the big players in the information technology industry.Given the popularity, wide variety of services, maturity, and sophistication of the AWS,

we have chosen to focus on the Amazon cloud computing products in this chapter Weencourage you to consider and balance your specific needs when choosing a cloudcomputing vendor However, you will find the Amazon solutions are rapidly becomingcommonplace

The (Amazon) Cloud: Another Xerox?

Just as many of us refer to any office copier as a “Xerox machine,” and use the relatedverb “xeroxing,” many technology pundits describe cloud computing (or simply “thecloud”) in terms of what AWS provides Only time and industry adoption patterns willtell us whether Amazon’s solution becomes the standard by which all solutions arejudged However, we often hear cloud computing defined as having elasticity—a termmade (more) popular by Amazon’s choice to name its IaaS product “Elastic CloudComputing” (EC2)

AWS

Amazon offers a large portfolio of developer tools and solutions called AWS Whilecollectively they have become known as the “Amazon cloud,” the products and servicesprovided include computational services (cloud), content delivery, database systemsupport, e-commerce solutions, messaging, monitoring, networking, payment andbilling solutions for vendors, cloud storage solutions, support services for AWS prod-ucts, web traffic management, and workforce software solutions Amazon adds newproducts regularly, so by the time you read this, the list may have grown considerably.The Amazon products are fee-based, but are designed to allow users to consume andpay for services as they are needed

In the next section, we discuss the technologies available in AWS and provide a briefintroduction to using cloud computing the Amazon way.

A Brief Overview of Technologies

Since the technologies are numerous and diverse, we will focus only on areas that tain to building reliable data centers You will, of course, want to explore the details ofall of the AWS products, and you can do so by clicking on the Products tab at the top

per-of the page at http://aws.amazon.com.All of the following technologies are built on simple web services, making it easy tobuild applications that can communicate with every tool over a RESTful web interface:

Amazon Elastic Compute Cloud (EC2)

Along with Amazon Simple Storage Service (S3), makes up the heart of the Amazoncloud This is the main technology that makes the cloud possible It manages thevirtual computing resources

Amazon Elastic MapReduce

Uses the Hadoop framework to provide an environment for data-intensive tasks

Auto Scaling

Provides the ability to automatically scale your solution based on parameters youdefine This is a key feature in building highly available cloud solutions

Amazon CloudFront

A content management service that allows you to provide static and streaming

content (sometimes called active content) to users in widely different geographic

locations

Amazon SimpleDB

Provides very basic nonrelational database storage and retrieval

Amazon Relational Database Service (RDS)

Amazon’s offering of the MySQL database system You can use this service instead

of creating and managing your own MySQL installation

Amazon Fulfillment Web Service (FWS)

Provides a bundled set of e-commerce tools—the same tools that the now-famousAmazon merchant website uses

Amazon Simple Queue Service (SQS)

The queuing message service used in grid computing solutions

Amazon CloudWatch

Provides monitoring capabilities for all of your Amazon cloud resources

Amazon Virtual Private Cloud (VPC)

An exciting recent capability that permits enterprises to extend their infrastructuresprivately using a VPN into the Amazon cloud The resources you allocate in theVPC communicate with your internal infrastructure as if they were located on the

same internal network This could become a must-have for organizations that mustramp up their computational resources quickly.

Elastic Load Balancing

Another key component of the cloud services It provides the ability to balance theload of network traffic across your solution

Amazon Flexible Payments Service (FPS)

A development library that provides payment processing tools for building table foundations and e-commerce sites

Amazon Elastic Block Storage (EBS)

The key module that stores your data It is a block-level device that you can attach

to any instance for data storage and retrieval

AWS Import/Export

Services for importing and exporting large amounts of data into and out of yourcloud solutions

AWS Premium Support

Support services for all of the AWS products, providing one-on-one assistance forbuilding and running applications with the AWS products

Alexa Web Information Service

A service that gathers metadata about the traffic and structure of websites

Alexa Top Sites

A service that ranks websites based on traffic and frequency

Amazon Mechanical Turk

A collaborative solution that supports on-demand workforce solutions It is cifically designed to integrate human-based tasking into computational systems,such as photography, audio recording, and other human-centric tasks that arecommon among data providers and consumers

spe-Now that you have seen a list of the relevant products, we’ll focus on the essentialtechnologies you need to know to begin working with your first cloud solution Whilethis list is short and may seem to be a pittance compared to the long list we just laidout, these are by far the most frequently used technologies in building a cloud solution.Once you master these, you can begin to explore the more advanced services

Amazon EC2

The Amazon EC2 service was first released in 2006 to beta users and became publiclyavailable in 2008 EC2 is the power behind the dynamic computing capability in the

Amazon cloud It provides virtualized hardware for server instances that you can cate on the fly and that support a host of available operating systems and environments.EC2 is essentially what makes the cloud tick.

allo-EC2’s virtualization uses the open source Xen technology, allowing fine-grained ware virtualization and customization The Xen virtualization platform, which wascreated by XenSource (later purchased by Citrix) allows guest operating systems such

hard-as Linux, Windows, or Solaris to run hard-as virtual machines on the same hardware currently

con-A virtual machine in EC2 is known as an instance, and you can connect to, monitor,

and administer it as if it were an operating system running on dedicated hardware.One of the interesting capabilities of EC2 and Xen virtualization is support for virtualinstances of both 32-bit and 64-bit CPUs A CPU core or processing core is known as

a computational unit (CU), which, besides being the governor of processing power, isthe unit Amazon uses as a cost multiplier The more computational units you use, themore it costs to run the instance Hence, you should choose the smallest computational

unit that meets the needs of your task An instance can be one of several instance types, listed in Table 14-1.

Table 14-1 Instance types

Small 1 EC2 CU 1.7 GB 160 GB instance

stor-age (150 GB plus 10 GB root partition)

32-bit Moderate m1.small

Large 4 EC2 CU (2 virtual

cores with 2 EC2 CU each)

7.5 GB 850 GB instance

stor-age (2 × 420 GB plus

10 GB root partition)

64-bit High m1.large

Extra large 8 EC2 CU (4 virtual

cores with 2 EC2 CU each)

each)

1.7 GB 350 GB instance

stor-age (340 GB plus 10 GB root partition)

32-bit Moderate c1.medium

High-CPU extra large 20 EC2 CU (8 virtualcores with 2.5 EC2 CU

6.5 EC2 CU (2 virtual cores with 3.25 EC2 CU each)

17.1 GB 420 GB instance

stor-age (1 × 420 GB)

64-bit Moderate m2.xlarge

High-memory double extra large 13 EC2 CU (4 virtualcores with 3.25 EC2 CU

Type CPU Memory Local storage Platform I/O Name

High-memory quadruple extra large

26 EC2 CU (8 virtual cores with 3.25 EC2 CU each)

68.4 GB 1690 GB instance

stor-age (2 × 840 GB plus

10 GB root partition)

64-bit High m2.4xlarge

Amazon bills you based on the computational hours during which you’ve reserved yourvirtual machine, using the instance type as a cost multiplier The instances themselvescan run in one of several regions (locations around the world that host the hardware).For more information concerning instance pricing and configurations, see http://aws amazon.com/ec2/#pricing

EC2 instances use an AMI The AMI is composed of an operating system and anyadditional software you choose to preload Amazon has cataloged many prebuilt AMIs,which makes getting started with EC2 that much easier For instance, you can load up

a prebuilt AMI that runs a Linux, Apache, MySQL, PHP/Perl/Python (LAMP) stack

Amazon S3

Also in 2006, Amazon created S3 as its premier online storage web service S3 providesdevelopers with easy, secure, persistent online storage offering essentially unlimitedcapacity At a high level, S3 is similar (at least in concept) to a storage area network(SAN) in that the resources are available from any connected device Amazon chargesusage fees based on the data stored and the bandwidth consumed in storing and re-trieving data

Unlike a traditional filesystem that uses a directory structure, S3 uses an object store

mechanism called buckets, which you can define using names that are publicly readable.

That is, anyone can store something in your “mycompanyname” bucket, so beware ofusing common names like “database” or “public” or “documents.” Most users use theirdomain names in the name of their buckets, but even that doesn’t prevent others fromstoring something in the same bucket

Amazon refers to things stored in the buckets as objects, and each object can be as small

as a single byte or up to 5 GB Buckets and the objects they contain physically reside

in one of the two regions called availability zones (one encompassing a variety of data

centers in North America and the other encompassing data centers in Europe), but youcan access them from anywhere

Furthermore, Amazon provides web services that allow you to make use of S3 in almostany web-based application

The S3 mechanism is not meant to be a fast read/write mechanism and is really bestused for archival purposes like storing a customized AMI or a bulk data copy or backup.Thus, you would not want to use it to store your active databases

Amazon EBS

Amazon released EBS in 2008 It was a giant leap forward for cloud computing EBS is

a virtualized block storage device, like a disk drive Not only does it have the ance of a typical block device, offering fast read and write capability, but it is alsoindependent of a running instance This is really important because in the past, usershad to rely on bringing their data from S3 or outside the cloud and loading it on theinstance But the instance was volatile, so when it terminated (which could be unex-pected and occur without warning), you would lose any changes made on the instance

perform-So before EBS, you had to make your applications back up to S3 frequently or use toolssuch as a volume manager

With EBS, users can now create independent devices (called volumes or simply EBS stores) and attach them to any running instance, not unlike a USB hard drive EBS

volumes appear to developers as standard block devices that range from 1 GB to 1 TB

in size Much like their SAN counterparts, EBS volumes can be resized on the fly usingsnapshotting This is very handy, as it allows you to grow your disk usage as yourapplication and data grow

You can also use multiple EBS volumes for striping to improve throughput and I/Operformance Better still, EBS volumes are replicated among the Amazon EC2 availa-bility zones, which means that even if the zone you’re in suffers a disaster, your data isstill accessible EBS is therefore even more reliable than traditional disk storage systems.However, EBS volumes you attach to your EC2 instance must reside in the same avail-ability zone

EBS also supports backup using point-in-time snapshots to S3 Each backup is a ferential snapshot: only the blocks that have changed since the last snapshot are saved.Point-in-time snapshots provide an effective and efficient way to create durable back-ups, especially for your MySQL databases

dif-With all the capability of a normal disk and much more, EBS volumes are the perfectsolution for storing your MySQL database files You get snapshotting and similarbackup capabilities, and the sustainability is unparalleled

How Does It All Work?

Before we go into the specifics of setting up an AWS account and running instances of AMIs, it is important for you to understand conceptually how you interact with theAmazon cloud and how your virtualized servers are realized Figure 14-2 depicts aconceptual model of how the EC2 instances are realized and executed

Amazon Cloud Tools

There are two types of tools available for purchasing services, initiating resources, andmanaging them: a GUI and a command-line interface There are actually two types of

GUIs Amazon provides a web-based console to access its services, and there are anumber of plug-ins for web browsers that you can use as an alternative Amazon alsoprovides a set of utilities built on its EC2 API (called EC2 command-line tools).

Amazon console

Amazon provides a web-based interface for interacting with all of its products calledthe AWS Management Console You can find it at https://console.aws.amazon.com/ec2/ home

You will need an AWS account to launch it; we’ll show you how to do that in anupcoming section We want to introduce the console first to give you a better picture

of how to work with AWS and EC2

You can use the interface to create instances, connect to them, create EBS volumes andconnect them to instances, and much more As you will see, this is the default mecha-nism for working with EC2 and other AWS products Figure 14-3 shows the AWSManagement Console for a typical user

In Figure 14-3 and several others in this chapter, we have masked out specific portions that show user contextual information that is not per- tinent to the points discussed.

The left center of the screen shows a button labeled Launch Instance As you can guess,this is your starting point for creating EC2 instances On the left of the screen are links

Figure 14-2 How AMIs become server instances in EC2

for many of the AWS cloud resources, such as volumes (EBS), elastic IPs, load balancers,and more.

Across the top are the three tabs that you use to access the different groupings of cloudservices The default (shown) is the Amazon EC2 tab, followed by the Amazon ElasticMapReduce tab, which allows you to set up and execute grid computing solutions, andAmazon CloudFront, for control of your web content

We will look at more details and step through creating an EC2 instance in the nextsections

Browser plug-ins

If you want a bit more power on a single web page, you can use a browser plug-in forMozilla Firefox called Elasticfox You can download it at http://developer.amazonweb services.com/connect/entry.jspa?externalID=609

Elasticfox is a web-based GUI that executes the complete EC2 tools API, permittingyou to control every aspect of your EC2 instances, from creating instances to creatingvolumes and connecting them to instances and much more In many ways it is morepowerful than the AWS Management Console, in that it puts everything at your fin-gertips (or mouse pointer) Figure 14-4 shows Elasticfox in action

Be sure to read the Getting Started documentation located at http://developer.amazon webservices.com/connect/entry.jspa?externalID=1797 to set up and configure Elasticfoxfor EC2 access

Figure 14-3 AWS Management Console

Several other vendors and developers are building web-based alternatives to the AWSManagement Console One of the most noteworthy is the shareVM project You canfind more information about this solution at http://blog.sharevm.com/2009/01/09/web -based-ec2-console-alternative-to-elasticfox/.

EC2 command-line tools

You can also interact with EC2 using command-line tools There are two sets: the APIand AMI tools The API tools interact with EC2 and include tools for launching in-stances, creating and attaching volumes, managing security groups, and more The AMItools create and manipulate AMIs

You can download the EC2 API command-line tools from http://developer amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88 Followthe instructions for installing and configuring the tools for your host operating system.The documentation for the EC2 API tools is included in the Getting Started documen-tation, which you can find at http://docs.amazonwebservices.com/AWSEC2/latest/Com mandLineReference/ The EC2 user guide also contains documentation on the com-mand-line tools You can download the user guide from http://awsdocs.s3.amazonaws com/EC2/latest/ec2-ug.pdf

Take a moment to download and install the EC2 API tools They contain numerousutilities that allow you to perform a great many things in EC2 The following lists themore commonly used utilities:

ec2-add-key-pairCreates a new SSH key pair

EC2 API tools.

Figure 14-4 Elasticfox Firefox plug-in

ec2-run-instancesLaunches EC2 instances You must specify at least the name of the image and yourkey pair You can launch multiple instances at the same time.

ec2-describe-imagesLists available images Output includes the image ID, the location of the image inS3, and whether the image is available for launching There are a number of pa-rameters you can use to limit the search

ec2-stop-instancesStops or pauses instances You can stop multiple instances at the same time.ec2-start-instances

Starts or resumes instances You can start multiple instances at the same time.ec2-terminate-instances

Terminates instances You can terminate multiple instances at the same time.These are only a few of the commonly used commands There are also commands forworking with security groups, keys, images, volumes, and much more See the userguide for more details

You can download the EC2 AMI command-line tools from http://developer amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88 Followthe instructions for installing the tools for your host operating system

If you want to work with images and create your own customized images, you may alsowant to explore the following documentation links:

a popular fast food restaurant That’s way cheaper than buying a small server!

Amazon charges for many of its products, including computation time and storage Be sure to terminate your instances when they are not in use and purge temporary storage While the costs are minimal on a per- hour or per-data-size basis, you will still be billed, even if you are not actively using the resources Think of it like this: if you leave all the lights

on when you go on vacation for two weeks, you shouldn’t be surprised when you get the electric bill.

EC2 AMI tools.

In the next sections, we will show you how to get an AWS account, launch an instance,create a disk volume, and connect it to your running instance Later in this chapter wewill also show you how easy it is to use MySQL in the cloud.

Getting an account

The first thing you need is an Amazon AWS account To use the basic cloud services,you must create the account, sign up for EC2 access, and sign up for S3 and EBS services.Fortunately, the process is very simple:

1 Go to the AWS website

2 Click the Create an AWS Account link

3 Click the Sign Up Now button

4 Enter your desired sign-in ID (e.g., your email address) and choose the “I am a newuser” option Click the “Sign in using our secure server” button

If you already have an Amazon account for the merchant site, you can use this account instead of creating a new one.

5 Choose a password and verify it by entering it twice Also supply your email addresshere Click the “Create account” button

6 Enter the full details of your account, including your billing address and contactinformation You can optionally supply your own website name and URL Youmust accept the AWS customer agreement at the bottom of this page to continue.When you have read and agreed to the customer agreement, click the agreementcheckbox, enter the symbols displayed in the security check area, then click theContinue button

7 Check your email for a confirmation and follow the instructions provided there

8 Go to your account and supply your billing information on the Account page byclicking on the Payment Method link on the right side of the main page

You cannot use AWS EC2 until you have an active payment option.

To sign up for Amazon EC2 (or any service or product), go to the Products page andclick the link in the blue box that appears Then click the “Sign up for” button in thetop right of the screen and follow the on-screen instructions Repeat this process for S3and any other services or products you want to use

Getting your credentials

Access to EC2 and other AWS products requires one of several forms of security tocols These include your AWS sign-in name and password, access keys for accessingthe AWS API, X.509 certificates for SOAP protocol access to the AWS API, and keypairs used to access EC2 and CloudFront Be sure to check the sign-in requirements ofthe products you want to use to ensure you have the correct credentials

pro-Amazon recommends creating a folder to store your private keys Take a moment tocreate such a folder and protect it from casual discovery before you download yourprivate keys and other security credentials

Amazon login and password

To use the AWS Management Console and all account actions, you will need the loginname and password that you set up when you signed up for AWS

Access key ID and secret access key

To use the query APIs (e.g., to search for images) and many of the GUI tools (e.g.,Elasticfox), you will need an access key ID and a secret access key These are the mostfrequently requested credentials Use the following steps to see and create your accesskeys:

1 Go to the AWS main page

2 Click the Account tab

3 Click the Security Credentials link

You will be asked to sign in if you have not already done so.

4 Locate the Access Credentials section and click the Access Keys tab

5 If you created access keys when you signed up for AWS, they will be listed here

To see the secret access key, click the Show link next to the desired access key Youcan copy this key to a safe (access-protected) file on your system

6 If there are no access keys or you want to create a new access key, you can clickthe “Create a new access key” link

SOAP and EC2 command-line tools

Running the EC2 command-line tools or connecting using the SOAP protocols requires

an X.509 certificate and private key You should rotate your access keys periodically toreduce security risks Amazon will automatically rotate the keys every 90 days Use thefollowing steps to see and create your X.509 certificates:

1 Go to the AWS main page.

2 Click the Account tab

3 Click the Security Credentials link

You will be asked to sign in if you have not already done so.

4 Locate the Access Credentials section and click the X.509 Certificates tab This willdisplay all of the certificates you have and allow you to activate or deactivate them

5 To create a new certificate and private key, click the “Create a new certificate” link.This will create a new certificate and start a download of your private key

You are allowed to have two certificates If you already have two, you will need to delete one to create another.

6 When the dialog appears, click the Download Private Key File button and saveyour private key in the folder you created earlier

7 Click the Download X.509 Certificate button and save your certificate in the samefolder

8 Click the Close button when you are finished

You should now have one file that begins with pk- and another that begins with cert-.

These are your private key and certificate files

Credentials for working with CloudFront

CloudFront requires access via another specific key pair mechanism These are calledsimply key pairs and are often confused with the key pairs used with instances TheCloudFront key pairs work like the X.509 certificates in that you must access them onyour account’s security credentials page and you can have only two active at a time.Use the following steps to see your CloudFront key pairs or to create a new key pair:

1 Go to the AWS main page

2 Click the Account tab

3 Click the Security Credentials link

You will be asked to sign in if you have not already done so.

4 Locate the Access Credentials section and click the Key Pairs tab This will displayall of the certificates you have and allow you to activate or deactivate them.

5 To create a new key pair, click the “Create a new key pair” link This will create anew certificate and start a download of your private key

You are allowed to have two key pairs If you already have two, you will need to delete one to create another.

Credentials for working with instances

Instances require access via an SSH key pair These are created inside the AWS agement Console You can create as many as you need The best practice is to createone per instance to reduce risk of accidental or deliberate disclosure

Man-You will need to do this at least once to get an SSH key pair to launch your first instance.

These keys allow you to access your running instance without the need to rememberand manage passwords This works by embedding one of the keys in the instance andauthenticating you when you provide the other key pair on sign-in You can name thekey pairs to help you manage them You must specify a key pair by name when launch-ing an image

To create a new SSH key pair, follow the procedure shown here:

1 Go to the AWS Management Console

You will be asked to sign in if you have not already done so.

2 Click the Key Pairs link on the left side of the page

3 Click the Create Key Pair button and provide a unique name

4 Click the Create button This will create the key pair and begin a download of yourprivate key Save the key in the folder you created previously

5 Click the Close button

Other credentials

Located at the bottom of your accounts security credentials page are your sign-in dentials and your account identifiers If you want to change your email address or

cre-password, you can do so on this page If you want to collaborate with other AWSaccounts, you will need to provide those account holders with your account identifiersand they will need to supply you with theirs You can find your account identifiers onthe bottom of this page.

You can sign up for an optional service called AWS Multi-Factor Authentication, whichuses a key generator pad to further increase the security of your account If security is

a high concern when working in AWS, you should consider reading more about whatthis service can provide There are buttons and links on this page to assist you in de-termining whether you should use this service

Running an instance with the AMS Management Console

Running an instance in EC2 is very easy It may not seem that way at first, but Amazonhas done a great job of making the initial learning curve as gentle as possible

To run an instance in EC2, go to the http://aws.amazon.com page and click the “Sign

in to the AWS Management Console” link You will see a button labeled Launch stance in the left center of the screen (Figure 14-5)

In-Figure 14-5 Launching an instance in EC2

Next, you will need to choose the image to launch You will be presented with theRequest Instances Wizard, whose first page contains a list of available images provided

by Amazon You will see multiple tabs, allowing you to choose Quick Start (a list ofAmazon-provided images preconfigured for common tasks), a MyAMIs tab if you havecreated any AMIs, and finally the Community AMIs tab with a list of AMIs that de-velopers have created and provided for your use