Shell Scripting potx

Shell Scripting expert recipeS for linux, BaSh, and More introduction.. 353 part ii ⊲ i recipeS for SySteMS adMiniStration chapter 15 Shell Features.. Shell Scripting ExpErt rEcipEs for

Shell Scripting

expert recipeS for linux, BaSh, and More

introduction xxix

part ⊲ i aBout the ingredientS chapter 1 The History of Unix, GNU, and Linux 3

chapter 2 Getting Started 15

chapter 3 Variables 33

chapter 4 Wildcard Expansion 67

chapter 5 Conditional Execution 83

chapter 6 Flow Control Using Loops 111

chapter 7 Variables Continued 139

chapter 8 Functions and Libraries 161

chapter 9 Arrays 199

chapter 10 Processes 219

chapter 11 Choosing and Using Shells 255

part i ⊲ i recipeS for uSing and extending SySteM toolS chapter 12 File Manipulation 279

chapter 13 Text Manipulation 315

chapter 14 Tools for Systems Administration 353

part ii ⊲ i recipeS for SySteMS adMiniStration chapter 15 Shell Features 409

chapter 16 Systems Administration 427

chapter 17 Presentation 459

chapter 18 Data Storage and Retrieval .471

chapter 19 Numbers 483

chapter 20 Processes 501

chapter 21 Internationalization .517

Continues

index 539

Shell Scripting

ExpErt rEcipEs for Linux, Bash, and MorE

Shell Scripting

ExpErt rEcipEs for Linux, Bash, and MorE

Steve Parker

Indianapolis, IN 46256

www.wiley.com

Copyright © 2011 by Steve Parker, Manchester, England

Published by John Wiley & Sons, Inc., Indianapolis, Indiana

Published simultaneously in Canada

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with

respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose No warranty may be created or extended by sales or pro- motional materials The advice and strategies contained herein may not be suitable for every situation This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services

If professional assistance is required, the services of a competent professional person should be sought Neither the lisher nor the author shall be liable for damages arising herefrom The fact that an organization or Web site is referred to

pub-in this work as a citation and/or a potential source of further pub-information does not mean that the author or the publisher endorses the information the organization or Web site may provide or recommendations it may make Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available

in electronic books.

Library of Congress Control Number: 2011932268

Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Programmer to Programmer, and related trade dress are

trade-marks or registered tradetrade-marks of John Wiley & Sons, Inc and/or its affiliates, in the United States and other countries, and may not be used without written permission John Wiley & Sons, Inc., is not associated with any product or vendor mentioned in this book.

For my daughters, Bethany and Emily, and my wife, Jackie Putting up with a professional geek is never easy, particularly when it leads to a career which often means a lot of travel and time spent away from home Also to God, from whom comes all wisdom, intelligence, and learning The better we understand the Creation, the better chance we have of

understanding the Creator

For it is written:

“I will destroy the wisdom of the wise; the intelligence of the intelligent I will frustrate Where is the wise man? Where is the scholar? Where is the philosopher of this age? Has not God made foolish the wisdom of the world?…For the foolishness of God

is wiser than man’s wisdom, and the weakness of God

is stronger than man’s strength.”

1 Corinthians chapter 1, verses 19, 20, and 25

aBout the author

SteVe parker is a Unix and Linux consultant with 20 years’ experience with Unix, and 15 years’ experience with GNU/Linux He wrote and maintains the online shell scripting tutorial at

http://steve-parker.org/sh/sh.shtml

Steve provides IT consultancy services, and can also provide training courses in shell scripting

as well as Unix and Linux technologies He can be contacted via http://sgpit.com/

aBout the technical editor

John kennedy has worked with Linux (and Unix) as a system administrator since 1997 He has worked with Red Hat, SUSE, Debian, Ubuntu, Solaris, and HP-UX He started bash scripting in

2000 because he felt he was doing too much work and wanted something to do the tedious jobs for him

Before learning the joys of Linux and Unix, John was in the U.S Air Force for nine years working as

a communications systems operator and spent time in Germany, Texas, and England Since leaving the military he has lived in Nebraska and Pennsylvania, and is now back in England

John currently works as an Infrastructure Engineer for a media company based in London He lives near Oxford with his wife, Michele, and son, Kieran He has a daughter, Denise, who just finished her university degree in the U.S

When John is not on his computer, he enjoys watching football (soccer) with his son, spending time with his family, and relaxing

Mary Beth Wakefield

freelancer editorial Manager

thiS Book would not haVe happened without the help (and deadlines) that the people at Wiley gave me Every step of the process has been a new experience, and Christina Haviland has been a great mentor through each step John Kennedy has provided feedback and encouragement through-out, and Nancy Rapoport has shown a fantastic eye for detail

From a personal perspective, I would like to thank all of the people behind Acorn, Sinclair, and other companies in the early 1980s for making affordable computers for kids to learn real program-ming with Also the BBC for their foresight in the entire BBC Micro project, the TV programs that they put behind it, and the development work that they pursued The next generation needs something like the BBC Micro project; not using fancy IDEs to write apps for phones, but working

at the bare metal with real systems The Arduino project deserves credit for promoting this at the hardware level; it is an excellent project, making it easy to hack hardware without having to have a knowledgeable uncle on hand to translate resistor values The Free Software infrastructure, particu-larly with more recent injections like the Google Summer of Code, is another ideal breeding ground for this love of hacking to develop afresh for a new (GNU?) generation The idea of a generation

growing up knowing only how to use devices, not how to develop them, is a disturbing one The

projects mentioned above provide hope for the future

I also want to thank ICL, where I met Douglas and Capitan, Jit, and Ketan We tested DRS/NX, and had direct access to userspace and kernel developers That was a rare treat, and it was where I fell in love with Unix Also the guys who used to hang out on comp.unix.shell back in the days when Usenet was still readable; you taught us so much, and we must have seemed so nạve (which we were).What I gained at ICL by being employed by the same company as the kernel and userspace develop-ers became available to everyone with the GNU/Linux operating system In the course of writing this book, I have been able to quote e-mails written by people that I have never met (and probably will never meet) in the discussion of Unix, Linux, and shell features Similarly, in a professional context, I have had the honor of chatting online with the key developers of specific Linux kernel fea-tures to discuss how they are implemented in different versions of the Linux kernel, none of which would be possible with a different development model Similarly, Chet Ramey, the maintainer of the bash shell, has responded to emails about implementation details

From a professional and IT community perspective, I would like to thank Ken Thompson, Dennis Ritchie, Brian Kernighan, Doug McIlroy, David Korn, and Steve Bourne (to name but a few) for C, Unix, and the environment that is so easily taken for granted The concepts these visionaries came

up with have lasted for 40 years and more

I also thank Dr Richard M Stallman for giving the GNU project to the world, for the GPL and the Free Software Foundation, and for dedicating a lifetime to promoting software freedom The world needs idealists, and Stallman is one of these It is my belief that Stallman will be proved by history

to be right, that it is better to share developments than to hide them That is the scientific tradition, and it must also be applied to computer science if it is to be treated seriously as a scientific endeavor

Microsystems and the Solaris Operating Environment Also Jonathan Schwartz for making most of the company’s software Open Source (even buying StarDivision in order to release OpenOffice.org) and the contributions that JDS made to the GNOME project, at a time when a lot of the industry didn’t understand the model RIP Sun Microsystems.

contEnts

recipeS for uSing and extending SySte

contEnts

xxviii

The shell is actually a full programming language, with variables and functions, and also more advanced structures such as arrays (including associative arrays), and being so directly linked to the kernel, it has native file I/O primitives built into its very syntax, as well as process and job control All of the main features that Unix is best known for are available in the shell, and avail-able to shell scripts.

This book has been written to get the most out of the shell, and should have something to surprise any reader, regardless of background and experience This book is aimed at intermediate and expe-rienced Unix and Linux administrators, and it may be of interest to other advanced users, too The book assumes that you know your way around at least one flavor of Unix-like system, and have probably already written some shell scripts, but want to improve your craft

Experienced readers will probably want to skip the first two chapters; very experienced readers may want to skip the first four chapters, although there may well be details contained there that are worth revisiting

what thiS Book coVerS

This book addresses shell scripting, with a focus on Bourne shell and POSIX compatibility, but a wide coverage of more recent developments, particularly the Bash shell, which is almost universal in GNU/Linux operating systems, and is included with most commercial Unices too The KornShell is also widely available in most such operating systems, both closed and open source

how thiS Book iS Structured

This book is in four parts; the first part covers the fundamental features and syntax of the shell; the second part looks at the tools available that a shell script can make use of; and the third part has recipes covering a fairly broad range of topics Finally, the fourth part contains reference information

Part One is the longest of the four sections; it looks at variables, wildcards, conditional execution, loops, functions, arrays, and processes The theory is presented with lots of practical examples to demonstrate what is being covered A lot of these scripts are fairly simplistic because they are con-centrating on only one aspect of the shell

Part Two covers the tools external to the shell that make shell scripts more useful; these are broken down into three chapters on text, files, and general systems administration The examples in Part Two are a bit more real-world in their nature, and a bit longer and more complex than those in Part One.Part Three is a set of shell scripting recipes The hope is that you will find these recipes useful in their own right, but they have also been selected for their usefulness in demonstrating the topics covered in the first two parts of the book They also show numerous different approaches and techniques that can

be used in real-life shell scripts The real and practical issues are dealt with in this part of the book, without making concessions to explain one specific point These scripts do what is necessary to get init scripts written, write colorful real-time interactive games, parse HTML, control processes, translate scripts into multiple languages, write CGI scripts, create graphical reports, and more

Finally, Part Four lists some links for further reading, as well as a glossary of terms

what you need to uSe thiS Book

Chapter 2 addresses some of the options that are available for getting access to a shell ment of your own and getting it set up in a way that suits you Experimenting on live systems is one option, but not a good one Setting up a test account is better, and running a dedicated test machine, or a virtual machine, is even better Virtualization software, such as VirtualBox or VMWare Player, is available at no cost and provides a risk-free way of testing even the most risky of root-owned scripts

We present code in two different ways:

We use bold monofont type to emphasize text that is typed in by the user.

We use a monofont type with no bolding for code content and for system output.

Because many books have similar titles, you may fi nd it easiest to search by

ISBN; this book’s ISBN is 978-1-118-02448-5.

Once you download the code, just decompress it with your favorite compression tool Alternatively, you can go to the main Wrox code download page at www.wrox.com/dynamic/books/download aspx to see the code available for this book and all other Wrox books

errata

We make every effort to ensure that there are no errors in the text or in the code However, no one

is perfect, and mistakes do occur If you fi nd an error in one of our books, like a spelling mistake

or faulty piece of code, we would be very grateful for your feedback By sending in errata, you may save another reader hours of frustration, and at the same time, you will be helping us provide even higher quality information

To fi nd the errata page for this book, go to www.wrox.com and locate the title using the Search box or one of the title lists Then, on the book details page, click the Book Errata link On this page, you can view all errata that has been submitted for this book and posted by Wrox editors A complete book list including links to each book’s errata is also available at www.wrox.com/misc-pages/booklist.shtml

If you don’t spot “your” error on the Book Errata page, go to www.wrox.com/contact/techsupport shtml and complete the form there to send us the error you have found We’ll check the information and, if appropriate, post a message to the book’s errata page and fi x the problem in subsequent editions

of the book

p2p wrox coM

For author and peer discussion, join the P2P forums at p2p.wrox.com The forums are a Web-based system for you to post messages relating to Wrox books and related technologies and interact with other readers and technology users The forums offer a subscription feature to e-mail you topics

1 Go to p2p.wrox.com and click the Register link.

2 Read the terms of use and click Agree

3 Complete the required information to join as well as any optional information you wish to provide and click Submit

4 You will receive an e-mail with information describing how to verify your account and plete the joining process

com-You can read messages in the forums without joining P2P but in order to post

your own messages, you must join.

Once you join, you can post new messages and respond to messages other users post You can read messages at any time on the Web If you would like to have new messages from a particular forum e-mailed to you, click the Subscribe to this Forum icon by the forum name in the forum listing For more information about how to use the Wrox P2P, be sure to read the P2P FAQs for answers to questions about how the forum software works as well as many common questions specifi c to P2P and Wrox books To read the FAQs, click the FAQ link on any P2P page

Shell Scripting

ExpErt rEcipEs for Linux, Bash, and MorE

the history of unix, Gnu,

and Linux

The Unix tradition has a long history, and Linux comes from the Unix tradition, so to understand Linux one must understand Unix and to understand Unix one must understand its history Before Unix, a developer would submit a stack of punched cards, each card representing a command,

or part of a command These cards would be read and executed sequentially by the computer The developer would receive the generated output after the job had completed This would often

be a few days after the job had been submitted; if there was an error in the code, the output was just the error and the developer had to start again Later, teletype and various forms of timeshar-ing systems sped up the matter considerably, but the model was basically the same: a sequence

of characters (punch cards, or keys on keyboards — it’s still just a string of characters) ted as a batch job to be run (or fail to run), and for the result to come back accordingly This is signifi cant today in that it is still how data is transmitted on any computerized system — it’s all sequences of characters, transmitted in order Whether a text fi le, a web page, a movie, or music,

submit-it is all just strings of ones and zeroes, same as submit-it ever was Anything that looks even slightly ferent is simply putting an interface over the top of a string of ones and zeroes

dif-Unix and various other interactive and timesharing systems came along in the mid-1960s dif-Unix and its conventions continue to be central to computing practices today; its infl uences can be seen

in DOS, Linux, Mac OS X, and even Microsoft Windows

unix

In 1965, Bell Labs and GE joined a Massachusetts Institute of Technology (MIT) project known

as MULTICS, the Multiplexed Information and Computing System Multics was intended to be a stable, timesharing OS The “Multiplexed” aspect added unnecessary complexity, which eventu-ally led Bell Labs to abandon the project in 1969 Ken Thompson, Dennis Ritchie, Doug McIlroy, and Joe Ossanna retained some of the ideas behind it, took out a lot of the complexity, and came

up with Unix (a play on the word MULTICS, as this was a simplifi ed operating system inspired

by MULTICS)

1

An early feature of Unix was the introduction of pipes — something that Doug McIlroy had been

thinking about for a few years and was implemented in Unix by Ken Thompson Again, it took the same notion of streamed serial data, but pipes introduced the idea of having stdin and stdout, through which the data would fl ow Similar things had been done before, and the concept is fairly simple: One process creates output, which becomes input to another command The Unix pipes method introduced a concept that dramatically affected the design of the rest of the system

Most commands have a fi le argument as well, but existing commands were modifi ed to default

to read from their “Standard Input” (stdin) and “Standard Output” (stdout); the pipe can then

“stream” the data from one tool to another This was a novel concept, and one that strongly defi nes the Unix shell; it makes the whole system a set of generically useful tools, as opposed to monolithic, single-purpose applications This has been summarized as “do one thing and do it well.” The GNU toolchain was written to replace Unix while maintaining compatibility with Unix tools The devel-opers on the GNU project often took the opportunity presented by rewriting the tool to include additional functionality, while still sticking to the “do one thing and do it well” philosophy

The GNU project was started in 1983 by Richard Stallman, with the intention of

replacing proprietary commercial Unices with Free Software alternatives GNU

had all but completed the task of replacing all of the userspace tools by the time

the Linux kernel project started in 1991 In fact, the GNU tools generally

per-form the same task at least as well as their original Unix equivalents, often

pro-viding extra useful features borne of experience in the real world Independent

testing has shown that GNU tools can actually be more reliable than their

tradi-tional Unix equivalents (http://www.gnu.org/software/reliability.html).

For example, the who command lists who is logged in to the system, one line per logged-in session The wc command counts characters, words, and lines Therefore, the following code will tell you how many people are logged in:

who | wc -l

There is no need for the who tool to have an option to count the logged-in users because the generic

wc tool can do that already This saves some small effort in who, but when that is applied across the whole range of tools, including any new tools that might be written, a lot of effort and therefore complexity, which means a greater likelihood of the introduction of additional bugs, is avoided When this is applied to more complicated tools, such as grep or even more, the fl exibility of the sys-tem is increased with every added tool

In the case of more, this is actually more tricky than it seems; fi rst it has to fi nd

out how many columns and rows are available Again, there is a set of tools that

combine to provide this information In this way, every tool in the chain can be

used by the other tools.