Unix power tools

Đây là bộ sách tiếng anh cho dân công nghệ thông tin chuyên về bảo mật,lập trình.Thích hợp cho những ai đam mê về công nghệ thông tin,tìm hiểu về bảo mật và lập trình.

UNIX Power Tools, 3rd Edition

By Shelley Powers, Jerry Peek, Tim O'Reilly,

Mike Loukides, et al

Publisher : O'ReillyPub Date : October 2002ISBN : 0-596-00330-7Pages : 1136

The latest edition of this best-selling favorite is loaded with vital

information on Linux, Darwin, and BSD Unix Power Tools 3rd

Edition now offers more coverage of bash, zsh, and other new shells, along with discussions about modern utilities and

applications Several sections focus on security and Internet access There is a new chapter on access to Unix from Windows, and expanded coverage of software installation and packaging, as well as basic information on Perl and Python.

Like a News Magazine

Like a Hypertext Database

Programs on the Web

About Unix Versions

Cross-References

What's New in the Third Edition

Typefaces and Other Conventions

The Authors

The Fine Print

Request for Comments

Acknowledgments for the First Edition

Acknowledgments for the Second Edition

Acknowledgments for the Third Edition

Part I: Basic Unix Environment

Chapter 1 Introduction

Section 1.1 What's Special About Unix?

Section 1.2 Power Grows on You

Section 1.3 The Core of Unix

Section 1.4 Communication with Unix

Section 1.5 Programs Are Designed to Work Together

Section 1.6 There Are Many Shells

Section 1.7 Which Shell Am I Running?

Section 1.8 Anyone Can Program the Shell

Section 1.9 Internal and External Commands

Section 1.10 The Kernel and Daemons

Section 1.11 Filenames

Section 1.12 Filename Extensions

Section 1.13 Wildcards

Section 1.14 The Tree Structure of the Filesystem

Section 1.15 Your Home Directory

Section 1.16 Making Pathnames

Section 1.17 File Access Permissions

Section 1.18 The Superuser (Root)

Section 1.19 When Is a File Not a File?

Section 1.20 Scripting

Section 1.21 Unix Networking and Communications

Section 1.22 The X Window System

Chapter 2 Getting Help

Section 2.1 The man Command

Section 2.2 whatis: One-Line Command Summaries

Section 2.3 whereis: Finding Where a Command Is Located

Section 2.4 Searching Online Manual Pages

Section 2.5 How Unix Systems Remember Their Names

Section 2.6 Which Version Am I Using?

Section 2.7 What tty Am I On?

Section 2.8 Who's On?

Section 2.9 The info Command

Part II: Customizing Your Environment

Chapter 3 Setting Up Your Unix Shell

Section 3.1 What Happens When You Log In

Section 3.2 The Mac OS X Terminal Application

Section 3.3 Shell Setup Files — Which, Where, and Why

Section 3.4 Login Shells, Interactive Shells

Section 3.5 What Goes in Shell Setup Files?

Section 3.6 Tip for Changing Account Setup: Keep a Shell Ready Section 3.7 Use Absolute Pathnames in Shell Setup Files

Section 3.8 Setup Files Aren't Read When You Want?

Section 3.9 Gotchas in set prompt Test

Section 3.10 Automatic Setups for Different Terminals

Section 3.11 Terminal Setup: Testing TERM

Section 3.12 Terminal Setup: Testing Remote Hostname and X Display Section 3.13 Terminal Setup: Testing Port

Section 3.14 Terminal Setup: Testing Environment Variables

Section 3.15 Terminal Setup: Searching Terminal Table

Section 3.16 Terminal Setup: Testing Window Size

Section 3.17 Terminal Setup: Setting and Testing Window Name Section 3.18 A cshrc.$HOST File for Per Host Setup

Section 3.19 Making a "Login" Shell

Section 3.20 RC Files

Section 3.21 Make Your Own Manpages Without Learning troff

Section 3.22 Writing a Simple Manpage with the -man Macros

Chapter 4 Interacting with Your Environment

Section 4.1 Basics of Setting the Prompt

Section 4.2 Static Prompts

Section 4.3 Dynamic Prompts

Section 4.4 Simulating Dynamic Prompts

Section 4.5 C-Shell Prompt Causes Problems in vi, rsh, etc.

Section 4.6 Faster Prompt Setting with Built-ins

Section 4.7 Multiline Shell Prompts

Section 4.8 Session Info in Window Title or Status Line

Section 4.9 A "Menu Prompt" for Naive Users

Section 4.10 Highlighting and Color in Shell Prompts

Section 4.11 Right-Side Prompts

Section 4.12 Show Subshell Level with $SHLVL

Section 4.13 What Good Is a Blank Shell Prompt?

Section 4.14 dirs in Your Prompt: Better Than $cwd

Section 4.15 External Commands Send Signals to Set Variables Section 4.16 Preprompt, Pre-execution, and Periodic Commands Section 4.17 Running Commands When You Log Out

Section 4.18 Running Commands at Bourne/Korn Shell Logout

Section 4.19 Stop Accidental Bourne-Shell Logouts

Chapter 5 Getting the Most out of Terminals, xterm, and X Windows

Section 5.1 There's a Lot to Know About Terminals

Section 5.2 The Idea of a Terminal Database

Section 5.3 Setting the Terminal Type When You Log In

Section 5.4 Querying Your Terminal Type: qterm

Section 5.5 Querying Your xterm Size: resize

Section 5.6 Checklist: Terminal Hangs When I Log In

Section 5.7 Find Out Terminal Settings with stty

Section 5.8 Setting Your Erase, Kill, and Interrupt Characters

Section 5.9 Working with xterm and Friends

Section 5.10 Login xterms and rxvts

Section 5.11 Working with Scrollbars

Section 5.12 How Many Lines to Save?

Section 5.13 Simple Copy and Paste in xterm

Section 5.14 Defining What Makes Up a Word for Selection Purposes Section 5.15 Setting the Titlebar and Icon Text

Section 5.16 The Simple Way to Pick a Font

Section 5.17 The xterm Menus

Section 5.18 Changing Fonts Dynamically

Section 5.19 Working with xclipboard

Section 5.20 Problems with Large Selections

Section 5.21 Tips for Copy and Paste Between Windows

Section 5.22 Running a Single Command with xterm -e

Section 5.23 Don't Quote Arguments to xterm -e

Chapter 6 Your X Environment

Section 6.1 Defining Keys and Button Presses with xmodmap

Section 6.2 Using xev to Learn Keysym Mappings

Section 6.3 X Resource Syntax

Section 6.4 X Event Translations

Section 6.5 Setting X Resources: Overview

Section 6.6 Setting Resources with the -xrm Option

Section 6.7 How -name Affects Resources

Section 6.8 Setting Resources with xrdb

Section 6.9 Listing the Current Resources for a Client: appres Section 6.10 Starting Remote X Clients

Part III: Working with Files and Directories

Chapter 7 Directory Organization

Section 7.1 What? Me, Organized?

Section 7.2 Many Homes

Section 7.3 Access to Directories

Section 7.4 A bin Directory for Your Programs and Scripts Section 7.5 Private (Personal) Directories

Section 7.6 Naming Files

Section 7.7 Make More Directories!

Section 7.8 Making Directories Made Easier

Chapter 8 Directories and Files

Section 8.1 Everything but the find Command

Section 8.2 The Three Unix File Times

Section 8.3 Finding Oldest or Newest Files with ls -t and ls -u Section 8.4 List All Subdirectories with ls -R

Section 8.5 The ls -d Option

Section 8.6 Color ls

Section 8.7 Some GNU ls Features

Section 8.8 A csh Alias to List Recently Changed Files

Section 8.9 Showing Hidden Files with ls -A and -a

Section 8.10 Useful ls Aliases

Section 8.11 Can't Access a File? Look for Spaces in the Name Section 8.12 Showing Nonprintable Characters in Filenames Section 8.13 Counting Files by Types

Section 8.14 Listing Files by Age and Size

Section 8.15 newer: Print the Name of the Newest File

Section 8.16 oldlinks: Find Unconnected Symbolic Links

Section 8.17 Picking a Unique Filename Automatically

Chapter 9 Finding Files with find

Section 9.1 How to Use find

Section 9.2 Delving Through a Deep Directory Tree

Section 9.3 Don't Forget -print

Section 9.4 Looking for Files with Particular Names

Section 9.5 Searching for Old Files

Section 9.6 Be an Expert on find Search Operators

Section 9.7 The Times That find Finds

Section 9.8 Exact File-Time Comparisons

Section 9.9 Running Commands on What You Find

Section 9.10 Using -exec to Create Custom Tests

Section 9.11 Custom -exec Tests Applied

Section 9.12 Finding Many Things with One Command

Section 9.13 Searching for Files by Type

Section 9.14 Searching for Files by Size

Section 9.15 Searching for Files by Permission

Section 9.16 Searching by Owner and Group

Section 9.17 Duplicating a Directory Tree

Section 9.18 Using "Fast find" Databases

Section 9.19 Wildcards with "Fast find" Database

Section 9.20 Finding Files (Much) Faster with a find Database Section 9.21 grepping a Directory Tree

Section 9.22 lookfor: Which File Has That Word?

Section 9.23 Using Shell Arrays to Browse Directories

Section 9.24 Finding the (Hard) Links to a File

Section 9.25 Finding Files with -prune

Section 9.26 Quick finds in the Current Directory

Section 9.27 Skipping Parts of a Tree in find

Section 9.28 Keeping find from Searching Networked Filesystem

Chapter 10 Linking, Renaming, and Copying Files

Section 10.1 What's So Complicated About Copying Files

Section 10.2 What's Really in a Directory?

Section 10.3 Files with Two or More Names

Section 10.4 More About Links

Section 10.5 Creating and Removing Links

Section 10.6 Stale Symbolic Links

Section 10.7 Linking Directories

Section 10.8 Showing the Actual Filenames for Symbolic Links Section 10.9 Renaming, Copying, or Comparing a Set of Files Section 10.10 Renaming a List of Files Interactively

Section 10.11 One More Way to Do It

Section 10.12 Copying Directory Trees with cp -r

Section 10.13 Copying Directory Trees with tar and Pipes

Chapter 11 Comparing Files

Section 11.1 Checking Differences with diff

Section 11.2 Comparing Three Different Versions with diff3 Section 11.3 Context diffs

Section 11.4 Side-by-Side diffs: sdiff

Section 11.5 Choosing Sides with sdiff

Section 11.6 Problems with diff and Tabstops

Section 11.7 cmp and diff

Section 11.8 Comparing Two Files with comm

Section 11.9 More Friendly comm Output

Section 11.10 make Isn't Just for Programmers!

Section 11.11 Even More Uses for make

Chapter 12 Showing What's in a File

Section 12.1 Cracking the Nut

Section 12.2 What Good Is a cat?

Section 12.3 "less" is More

Section 12.4 Show Nonprinting Characters with cat -v or od -c Section 12.5 What's in That Whitespace?

Section 12.6 Finding File Types

Section 12.7 Squash Extra Blank Lines

Section 12.8 How to Look at the End of a File: tail

Section 12.9 Finer Control on tail

Section 12.10 How to Look at Files as They Grow

Section 12.11 GNU tail File Following

Section 12.12 Printing the Top of a File

Section 12.13 Numbering Lines

Chapter 13 Searching Through Files

Section 13.1 Different Versions of grep

Section 13.2 Searching for Text with grep

Section 13.3 Finding Text That Doesn't Match

Section 13.4 Extended Searching for Text with egrep

Section 13.5 grepping for a List of Patterns

Section 13.6 Approximate grep: agrep

Section 13.7 Search RCS Files with rcsgrep

Section 13.8 GNU Context greps

Section 13.9 A Multiline Context grep Using sed

Section 13.10 Compound Searches

Section 13.11 Narrowing a Search Quickly

Section 13.12 Faking Case-Insensitive Searches

Section 13.13 Finding a Character in a Column

Section 13.14 Fast Searches and Spelling Checks with "look" Section 13.15 Finding Words Inside Binary Files

Section 13.16 A Highlighting grep

Chapter 14 Removing Files

Section 14.1 The Cycle of Creation and Destruction

Section 14.2 How Unix Keeps Track of Files: Inodes

Section 14.3 rm and Its Dangers

Section 14.4 Tricks for Making rm Safer

Section 14.5 Answer "Yes" or "No" Forever with yes

Section 14.6 Remove Some, Leave Some

Section 14.7 A Faster Way to Remove Files Interactively

Section 14.8 Safer File Deletion in Some Directories

Section 14.9 Safe Delete: Pros and Cons

Section 14.10 Deletion with Prejudice: rm -f

Section 14.11 Deleting Files with Odd Names

Section 14.12 Using Wildcards to Delete Files with Strange Names Section 14.13 Handling a Filename Starting with a Dash (-)

Section 14.14 Using unlink to Remove a File with a Strange Name Section 14.15 Removing a Strange File by its i-number

Section 14.16 Problems Deleting Directories

Section 14.17 Deleting Stale Files

Section 14.18 Removing Every File but One

Section 14.19 Using find to Clear Out Unneeded Files

Chapter 15 Optimizing Disk Space

Section 15.1 Disk Space Is Cheap

Section 15.2 Instead of Removing a File, Empty It

Section 15.3 Save Space with "Bit Bucket" Log Files and Mailboxes Section 15.4 Save Space with a Link

Section 15.5 Limiting File Sizes

Section 15.6 Compressing Files to Save Space

Section 15.7 Save Space: tar and compress a Directory Tree Section 15.8 How Much Disk Space?

Section 15.9 Compressing a Directory Tree: Fine-Tuning

Section 15.10 Save Space in Executable Files with strip

Section 15.11 Disk Quotas

Part IV: Basic Editing

Chapter 16 Spell Checking, Word Counting, and Textual Analysis

Section 16.1 The Unix spell Command

Section 16.2 Check Spelling Interactively with ispell

Section 16.3 How Do I Spell That Word?

Section 16.4 Inside spell

Section 16.5 Adding Words to ispell's Dictionary

Section 16.6 Counting Lines, Words, and Characters: wc

Section 16.7 Find a a Doubled Word

Section 16.8 Looking for Closure

Section 16.9 Just the Words, Please

Chapter 17 vi Tips and Tricks

Section 17.1 The vi Editor: Why So Much Material?

Section 17.2 What We Cover

Section 17.3 Editing Multiple Files with vi

Section 17.4 Edits Between Files

Section 17.5 Local Settings for vi

Section 17.6 Using Buffers to Move or Copy Text

Section 17.7 Get Back What You Deleted with Numbered Buffers Section 17.8 Using Search Patterns and Global Commands

Section 17.9 Confirming Substitutions in vi

Section 17.10 Keep Your Original File, Write to a New File

Section 17.11 Saving Part of a File

Section 17.12 Appending to an Existing File

Section 17.13 Moving Blocks of Text by Patterns

Section 17.14 Useful Global Commands (with Pattern Matches) Section 17.15 Counting Occurrences; Stopping Search Wraps Section 17.16 Capitalizing Every Word on a Line

Section 17.17 Per-File Setups in Separate Files

Section 17.18 Filtering Text Through a Unix Command

Section 17.19 vi File Recovery Versus Networked Filesystems

Section 17.20 Be Careful with vi -r Recovered Buffers

Section 17.21 Shell Escapes: Running One UnixCommand While Using

Another Section 17.22 vi Compound Searches

Section 17.23 vi Word Abbreviation

Section 17.24 Using vi Abbreviations as Commands (Cut and Paste

Between vi's) Section 17.25 Fixing Typos with vi Abbreviations

Section 17.26 vi Line Commands Versus Character Commands

Section 17.27 Out of Temporary Space? Use Another Directory Section 17.28 Neatening Lines

Section 17.29 Finding Your Place with Undo

Section 17.30 Setting Up vi with the exrc File

Chapter 18 Creating Custom Commands in vi

Section 18.1 Why Type More Than You Have To?

Section 18.2 Save Time and Typing with the vi map Commands Section 18.3 What You Lose When You Use map!

Section 18.4 vi @-Functions

Section 18.5 Keymaps for Pasting into a Window Running vi

Section 18.6 Protecting Keys from Interpretation by ex

Section 18.7 Maps for Repeated Edits

Section 18.8 More Examples of Mapping Keys in vi

Section 18.9 Repeating a vi Keymap

Section 18.10 Typing in Uppercase Without CAPS LOCK

Section 18.11 Text-Input Mode Cursor Motion with No Arrow Keys Section 18.12 Don't Lose Important Functions with vi Maps: Use

noremap Section 18.13 vi Macro for Splitting Long Lines

Section 18.14 File-Backup Macros

Chapter 19 GNU Emacs

Section 19.1 Emacs: The Other Editor

Section 19.2 Emacs Features: A Laundry List

Section 19.3 Customizations and How to Avoid Them

Section 19.4 Backup and Auto-Save Files

Section 19.5 Putting Emacs in Overwrite Mode

Section 19.6 Command Completion

Section 19.7 Mike's Favorite Timesavers

Section 19.8 Rational Searches

Section 19.9 Unset PWD Before Using Emacs

Section 19.10 Inserting Binary Characters into Files

Section 19.11 Using Word-Abbreviation Mode

Section 19.12 Directories for Emacs Hacks

Section 19.13 An Absurd Amusement

Chapter 20 Batch Editing

Section 20.1 Why Line Editors Aren't Dinosaurs

Section 20.2 Writing Editing Scripts

Section 20.3 Line Addressing

Section 20.4 Useful ex Commands

Section 20.5 Running Editing Scripts Within vi

Section 20.6 Change Many Files by Editing Just One

Section 20.7 ed/ex Batch Edits: A Typical Example

Section 20.8 Batch Editing Gotcha: Editors Fail on Big Files Section 20.9 patch: Generalized Updating of Files That Differ Section 20.10 Quick Reference: awk

Section 20.11 Versions of awk

Chapter 21 You Can't Quite Call This Editing

Section 21.1 And Why Not?

Section 21.2 Neatening Text with fmt

Section 21.3 Alternatives to fmt

Section 21.4 Clean Up Program Comment Blocks

Section 21.5 Remove Mail/News Headers with behead

Section 21.6 Low-Level File Butchery with dd

Section 21.7 offset: Indent Text

Section 21.8 Centering Lines in a File

Section 21.9 Splitting Files at Fixed Points: split

Section 21.10 Splitting Files by Context: csplit

Section 21.11 Hacking on Characters with tr

Section 21.12 Encoding "Binary" Files into ASCII

Section 21.13 Text Conversion with dd

Section 21.14 Cutting Columns or Fields

Section 21.15 Making Text in Columns with pr

Section 21.16 Make Columns Automatically with column Section 21.17 Straightening Jagged Columns

Section 21.18 Pasting Things in Columns

Section 21.19 Joining Lines with join

Section 21.20 What Is (or Isn't) Unique?

Section 21.21 Rotating Text

Chapter 22 Sorting

Section 22.1 Putting Things in Order

Section 22.2 Sort Fields: How sort Sorts

Section 22.3 Changing the sort Field Delimiter

Section 22.4 Confusion with Whitespace Field Delimiters

Section 22.5 Alphabetic and Numeric Sorting

Section 22.6 Miscellaneous sort Hints

Section 22.7 lensort: Sort Lines by Length

Section 22.8 Sorting a List of People by Last Name

Part V: Processes and the Kernel

Chapter 23 Job Control

Section 23.1 Job Control in a Nutshell

Section 23.2 Job Control Basics

Section 23.3 Using jobs Effectively

Section 23.4 Some Gotchas with Job Control

Section 23.5 The "Current Job" Isn't Always What You Expect Section 23.6 Job Control and autowrite: Real Timesavers!

Section 23.7 System Overloaded? Try Stopping Some Jobs

Section 23.8 Notification When Jobs Change State

Section 23.9 Stop Background Output with stty tostop

Section 23.10 nohup

Section 23.11 Disowning Processes

Section 23.12 Linux Virtual Consoles

Section 23.13 Stopping Remote Login Sessions

Chapter 24 Starting, Stopping, and Killing Processes

Section 24.1 What's in This Chapter

Section 24.2 fork and exec

Section 24.3 Managing Processes: Overall Concepts

Section 24.4 Subshells

Section 24.5 The ps Command

Section 24.6 The Controlling Terminal

Section 24.7 Tracking Down Processes

Section 24.8 Why ps Prints Some Commands in Parentheses Section 24.9 The /proc Filesystem

Section 24.10 What Are Signals?

Section 24.11 Killing Foreground Jobs

Section 24.12 Destroying Processes with kill

Section 24.13 Printer Queue Watcher: A Restartable Daemon Shell

Script Section 24.14 Killing All Your Processes

Section 24.15 Killing Processes by Name?

Section 24.16 Kill Processes Interactively

Section 24.17 Processes Out of Control? Just STOP Them

Section 24.18 Cleaning Up an Unkillable Process

Section 24.19 Why You Can't Kill a Zombie

Section 24.20 The Process Chain to Your Window

Section 24.21 Terminal Windows Without Shells

Section 24.22 Close a Window by Killing Its Process(es)

Chapter 25 Delayed Execution

Section 25.1 Building Software Robots the Easy Way

Section 25.2 Periodic Program Execution: The cron Facility

Section 25.3 Adding crontab Entries

Section 25.4 Including Standard Input Within a cron Entry

Section 25.5 The at Command

Section 25.6 Making Your at Jobs Quiet

Section 25.7 Checking and Removing Jobs

Section 25.8 Avoiding Other at and cron Jobs

Section 25.9 Waiting a Little While: sleep

Chapter 26 System Performance and Profiling

Section 26.1 Timing Is Everything

Section 26.2 Timing Programs

Section 26.3 What Commands Are Running and How Long Do They

Take?

Section 26.4 Checking System Load: uptime

Section 26.5 Know When to Be "nice" to Other Users and When Not

To Section 26.6 A nice Gotcha

Section 26.7 Changing a Running Job's Niceness

Part VI: Scripting

Chapter 27 Shell Interpretation

Section 27.1 What the Shell Does

Section 27.2 How the Shell Executes Other Commands

Section 27.3 What's a Shell, Anyway?

Section 27.4 Command Evaluation and Accidentally Overwriting Files Section 27.5 Output Command-Line Arguments One by One

Section 27.6 Controlling Shell Command Searches

Section 27.7 Wildcards Inside Aliases

Section 27.8 eval: When You Need Another Chance

Section 27.9 Which One Will bash Use?

Section 27.10 Which One Will the C Shell Use?

Section 27.11 Is It "2>&1 file" or "> file 2>&1"? Why?

Section 27.12 Bourne Shell Quoting

Section 27.13 Differences Between Bourne and C Shell Quoting Section 27.14 Quoting Special Characters in Filenames

Section 27.15 Verbose and Echo Settings Show Quoting

Section 27.16 Here Documents

Section 27.17 "Special" Characters and Operators

Section 27.18 How Many Backslashes?

Chapter 28 Saving Time on the Command Line

Section 28.1 What's Special About the Unix Command Line

Section 28.2 Reprinting Your Command Line with CTRL-r

Section 28.3 Use Wildcards to Create Files?

Section 28.4 Build Strings with { }

Section 28.5 String Editing (Colon) Operators

Section 28.6 Automatic Completion

Section 28.7 Don't Match Useless Files in Filename Completion

Section 28.8 Repeating Commands

Section 28.9 Repeating and Varying Commands

Section 28.10 Repeating a Command with Copy-and-Paste

Section 28.11 Repeating a Time-Varying Command

Section 28.12 Multiline Commands, Secondary Prompts

Section 28.13 Here Document Example #1: Unformatted Form Letters Section 28.14 Command Substitution

Section 28.15 Handling Lots of Text with Temporary Files

Section 28.16 Separating Commands with Semicolons

Section 28.17 Dealing with Too Many Arguments

Section 28.18 Expect

Chapter 29 Custom Commands

Section 29.1 Creating Custom Commands

Section 29.2 Introduction to Shell Aliases

Section 29.3 C-Shell Aliases with Command-Line Arguments Section 29.4 Setting and Unsetting Bourne-Type Aliases

Section 29.5 Korn-Shell Aliases

Section 29.6 zsh Aliases

Section 29.7 Sourceable Scripts

Section 29.8 Avoiding C-Shell Alias Loops

Section 29.9 How to Put if-then-else in a C-Shell Alias

Section 29.10 Fix Quoting in csh Aliases with makealias and quote Section 29.11 Shell Function Basics

Section 29.12 Shell Function Specifics

Section 29.13 Propagating Shell Functions

Section 29.14 Simulated Bourne Shell Functions and Aliases

Chapter 30 The Use of History

Section 30.1 The Lessons of History

Section 30.2 History in a Nutshell

Section 30.3 My Favorite Is !$

Section 30.4 My Favorite Is !:n*

Section 30.5 My Favorite Is ^^

Section 30.6 Using !$ for Safety with Wildcards

Section 30.7 History by Number

Section 30.8 History Substitutions

Section 30.9 Repeating a Cycle of Commands

Section 30.10 Running a Series of Commands on a File

Section 30.11 Check Your History First with :p

Section 30.12 Picking Up Where You Left Off

Section 30.13 Pass History to Another Shell

Section 30.14 Shell Command-Line Editing

Section 30.15 Changing History Characters with histchars

Section 30.16 Instead of Changing History Characters

Chapter 31 Moving Around in a Hurry

Section 31.1 Getting Around the Filesystem

Section 31.2 Using Relative and Absolute Pathnames

Section 31.3 What Good Is a Current Directory?

Section 31.4 How Does Unix Find Your Current Directory?

Section 31.5 Saving Time When You Change Directories: cdpath Section 31.6 Loop Control: break and continue

Section 31.7 The Shells' pushd and popd Commands

Section 31.8 Nice Aliases for pushd

Section 31.9 Quick cds with Aliases

Section 31.10 cd by Directory Initials

Section 31.11 Finding (Anyone's) Home Directory, Quickly

Section 31.12 Marking Your Place with a Shell Variable

Section 31.13 Automatic Setup When You Enter/Exit a Directory

Chapter 32 Regular Expressions (Pattern Matching)

Section 32.1 That's an Expression

Section 32.2 Don't Confuse Regular Expressions with Wildcards

Section 32.3 Understanding Expressions

Section 32.4 Using Metacharacters in Regular Expressions

Section 32.5 Regular Expressions: The Anchor Characters ^ and $ Section 32.6 Regular Expressions: Matching a Character with a

Character Set Section 32.7 Regular Expressions: Match Any Character with (Dot) Section 32.8 Regular Expressions: Specifying a Range of Characters

with [ ]

Section 32.9 Regular Expressions: Exceptions in a Character Set

Section 32.10 Regular Expressions: Repeating Character Sets with * Section 32.11 Regular Expressions: Matching a Specific Number of Sets

with \ { and \ } Section 32.12 Regular Expressions: Matching Words with \ < and \ > Section 32.13 Regular Expressions: Remembering Patterns with \ (, \ ),

and \1 Section 32.14 Regular Expressions: Potential Problems

Section 32.15 Extended Regular Expressions

Section 32.16 Getting Regular Expressions Right

Section 32.17 Just What Does a Regular Expression Match?

Section 32.18 Limiting the Extent of a Match

Section 32.19 I Never Meta Character I Didn't Like

Section 32.20 Valid Metacharacters for Different Unix Programs

Section 32.21 Pattern Matching Quick Reference with Examples

Chapter 33 Wildcards

Section 33.1 File-Naming Wildcards

Section 33.2 Filename Wildcards in a Nutshell

Section 33.3 Who Handles Wildcards?

Section 33.4 What if a Wildcard Doesn't Match?

Section 33.5 Maybe You Shouldn't Use Wildcards in Pathnames

Section 33.6 Getting a List of Matching Files with grep -l

Section 33.7 Getting a List of Nonmatching Files

Section 33.8 nom: List Files That Don't Match a Wildcard

Chapter 34 The sed Stream Editor

Section 34.1 sed Sermon^H^H^H^H^H^HSummary

Section 34.2 Two Things You Must Know About sed

Section 34.3 Invoking sed

Section 34.4 Testing and Using a sed Script: checksed, runsed

Section 34.5 sed Addressing Basics

Section 34.6 Order of Commands in a Script

Section 34.7 One Thing at a Time

Section 34.8 Delimiting a Regular Expression

Section 34.9 Newlines in a sed Replacement

Section 34.10 Referencing the Search String in a Replacement

Section 34.11 Referencing Portions of a Search String

Section 34.12 Search and Replacement: One Match Among Many Section 34.13 Transformations on Text

Section 34.14 Hold Space: The Set-Aside Buffer

Section 34.15 Transforming Part of a Line

Section 34.16 Making Edits Across Line Boundaries

Section 34.17 The Deliberate Scrivener

Section 34.18 Searching for Patterns Split Across Lines

Section 34.19 Multiline Delete

Section 34.20 Making Edits Everywhere Except

Section 34.21 The sed Test Command

Section 34.22 Uses of the sed Quit Command

Section 34.23 Dangers of the sed Quit Command

Section 34.24 sed Newlines, Quoting, and Backslashes in a Shell Script

Chapter 35 Shell Programming for the Uninitiated

Section 35.1 Writing a Simple Shell Program

Section 35.2 Everyone Should Learn Some Shell Programming

Section 35.3 What Environment Variables Are Good For

Section 35.4 Parent-Child Relationships

Section 35.5 Predefined Environment Variables

Section 35.6 The PATH Environment Variable

Section 35.7 PATH and path

Section 35.8 The DISPLAY Environment Variable

Section 35.9 Shell Variables

Section 35.10 Test String Values with Bourne-Shell case

Section 35.11 Pattern Matching in case Statements

Section 35.12 Exit Status of Unix Processes

Section 35.13 Test Exit Status with the if Statement

Section 35.14 Testing Your Success

Section 35.15 Loops That Test Exit Status

Section 35.16 Set Exit Status of a Shell (Script)

Section 35.17 Trapping Exits Caused by Interrupts

Section 35.18 read: Reading from the Keyboard

Section 35.19 Shell Script "Wrappers" for awk, sed, etc.

Section 35.20 Handling Command-Line Arguments in Shell Scripts Section 35.21 Handling Command-Line Arguments with a for Loop Section 35.22 Handling Arguments with while and shift

Section 35.23 Loop Control: break and continue

Section 35.24 Standard Command-Line Parsing

Section 35.25 The Bourne Shell set Command

Section 35.26 test: Testing Files and Strings

Section 35.27 Picking a Name for a New Command

Section 35.28 Finding a Program Name and Giving Your Program Multiple

Names Section 35.29 Reading Files with the and source Commands

Section 35.30 Using Shell Functions in Shell Scripts

Chapter 36 Shell Programming for the Initiated

Section 36.1 Beyond the Basics

Section 36.2 The Story of : # #!

Section 36.3 Don't Need a Shell for Your Script? Don't Use One

Section 36.4 Making #! Search the PATH

Section 36.5 The exec Command

Section 36.6 The Unappreciated Bourne Shell ":" Operator

Section 36.7 Parameter Substitution

Section 36.8 Save Disk Space and Programming: Multiple Names for a

Program Section 36.9 Finding the Last Command-Line Argument

Section 36.10 How to Unset All Command-Line Parameters

Section 36.11 Standard Input to a for Loop

Section 36.12 Making a for Loop with Multiple Variables

Section 36.13 Using basename and dirname

Section 36.14 A while Loop with Several Loop Control Commands Section 36.15 Overview: Open Files and File Descriptors

Section 36.16 n>&m: Swap Standard Output and Standard Error

Section 36.17 A Shell Can Read a Script from Its Standard Input, but Section 36.18 Shell Scripts On-the-Fly from Standard Input

Section 36.19 Quoted hereis Document Terminators: sh Versus csh Section 36.20 Turn Off echo for "Secret" Answers

Section 36.21 Quick Reference: expr

Section 36.22 Testing Characters in a String with expr

Section 36.23 Grabbing Parts of a String

Section 36.24 Nested Command Substitution

Section 36.25 Testing Two Strings with One case Statement

Section 36.26 Outputting Text to an X Window

Section 36.27 Shell Lockfile

Chapter 37 Shell Script Debugging and Gotchas

Section 37.1 Tips for Debugging Shell Scripts

Section 37.2 Bourne Shell Debugger Shows a Shell Variable

Section 37.3 Stop Syntax Errors in Numeric Tests

Section 37.4 Stop Syntax Errors in String Tests

Section 37.5 Quoting and Command-Line Parameters

Section 37.6 How Unix Keeps Time

Section 37.7 Copy What You Do with script

Section 37.8 Cleaning script Files

Section 37.9 Making an Arbitrary-Size File for Testing

Part VII: Extending and Managing Your Environment

Chapter 38 Backing Up Files

Section 38.1 What Is This "Backup" Thing?

Section 38.2 tar in a Nutshell

Section 38.3 Make Your Own Backups

Section 38.4 More Ways to Back Up

Section 38.5 How to Make Backups to a Local Device

Section 38.6 Restoring Files from Tape with tar

Section 38.7 Using tar to a Remote Tape Drive

Section 38.8 Using GNU tar with a Remote Tape Drive

Section 38.9 On-Demand Incremental Backups of a Project

Section 38.10 Using Wildcards with tar

Section 38.11 Avoid Absolute Paths with tar

Section 38.12 Getting tar's Arguments in the Right Order

Section 38.13 The cpio Tape Archiver

Section 38.14 Industrial Strength Backups

Chapter 39 Creating and Reading Archives

Section 39.1 Packing Up and Moving

Section 39.2 Using tar to Create and Unpack Archives

Section 39.3 GNU tar Sampler

Section 39.4 Managing and Sharing Files with RCS and CVS

Section 39.5 RCS Basics

Section 39.6 List RCS Revision Numbers with rcsrevs

Section 39.7 CVS Basics

Section 39.8 More CVS

Chapter 40 Software Installation

Section 40.1 /usr/bin and Other Software Directories

Section 40.2 The Challenges of Software Installation on Unix Section 40.3 Which make?

Section 40.4 Simplifying the make Process

Section 40.5 Using Debian's dselect

Section 40.6 Installing Software with Debian's Apt-Get

Section 40.7 Interruptable gets with wget

Section 40.8 The curl Application and One-Step GNU-Darwin

Auto-Installer for OS X Section 40.9 Installation with FreeBSD Ports

Section 40.10 Installing with FreeBSD Packages

Section 40.11 Finding and Installing RPM Packaged Software

Chapter 41 Perl

Section 41.1 High-Octane Shell Scripting

Section 41.2 Checking your Perl Installation

Section 41.3 Compiling Perl from Scratch

Section 41.4 Perl Boot Camp, Part 1: Typical Script Anatomy Section 41.5 Perl Boot Camp, Part 2: Variables and Data Types Section 41.6 Perl Boot Camp, Part 3: Branching and Looping Section 41.7 Perl Boot Camp, Part 4: Pattern Matching

Section 41.8 Perl Boot Camp, Part 5: Perl Knows Unix

Section 41.9 Perl Boot Camp, Part 6: Modules

Section 41.10 Perl Boot Camp, Part 7: perldoc

Section 42.1 What Is Python?

Section 42.2 Installation and Distutils

Section 42.3 Python Basics

Section 42.4 Python and the Web

Part VIII: Communication and Connectivity

Chapter 43 Redirecting Input and Output

Section 43.1 Using Standard Input and Output

Section 43.2 One Argument with a cat Isn't Enough

Section 43.3 Send (Only) Standard Error Down a Pipe

Section 43.4 Problems Piping to a Pager

Section 43.5 Redirection in C Shell: Capture Errors, Too?

Section 43.6 Safe I/O Redirection with noclobber

Section 43.7 The ( ) Subshell Operators

Section 43.8 Send Output Two or More Places

Section 43.9 How to tee Several Commands into One Place

Section 43.10 Redirecting Output to More Than One Place

Section 43.11 Named Pipes: FIFOs

Section 43.12 What Can You Do with an Empty File?

Chapter 44 Devices

Section 44.1 Quick Introduction to Hardware

Section 44.2 Reading Kernel Boot Output

Section 44.3 Basic Kernel Configuration

Section 44.4 Disk Partitioning

Section 44.5 Filesystem Types and /etc/fstab

Section 44.6 Mounting and Unmounting Removable Filesystems

Section 44.7 Loopback Mounts

Section 44.8 Network Devices — ifconfig

Section 44.9 Mounting Network Filesystems — NFS, SMBFS

Section 44.10 Win Is a Modem Not a Modem?

Section 44.11 Setting Up a Dialup PPP Session

Section 44.12 USB Configuration

Section 44.13 Dealing with Sound Cards and Other Annoying Hardware Section 44.14 Decapitating Your Machine — Serial Consoles

Chapter 45 Printing

Section 45.1 Introduction to Printing

Section 45.2 Introduction to Printing on Unix

Section 45.3 Printer Control with lpc

Section 45.4 Using Different Printers

Section 45.5 Using Symbolic Links for Spooling

Section 45.6 Formatting Plain Text: pr

Section 45.7 Formatting Plain Text: enscript

Section 45.8 Printing Over a Network

Section 45.9 Printing Over Samba

Section 45.10 Introduction to Typesetting

Section 45.11 A Bit of Unix Typesetting History

Section 45.12 Typesetting Manpages: nroff

Section 45.13 Formatting Markup Languages — troff, LATEX, HTML, and

So On Section 45.14 Printing Languages — PostScript, PCL, DVI, PDF

Section 45.15 Converting Text Files into a Printing Language

Section 45.16 Converting Typeset Files into a Printing Language

Section 45.17 Converting Source Files Automagically Within the Spooler Section 45.18 The Common Unix Printing System (CUPS)

Section 45.19 The Portable Bitmap Package

Chapter 46 Connectivity

Section 46.1 TCP/IP — IP Addresses and Ports

Section 46.2 /etc/services Is Your Friend

Section 46.3 Status and Troubleshooting

Section 46.4 Where, Oh Where Did That Packet Go?

Section 46.5 The Director of Operations: inetd

Section 46.6 Secure Shell (SSH)

Section 46.7 Configuring an Anonymous FTP Server

Section 46.8 Mail — SMTP, POP, and IMAP

Section 46.9 Domain Name Service (DNS)

Section 46.10 Dynamic Host Configuration Protocol (DHCP)

Section 46.11 Gateways and NAT

Section 46.12 Firewalls

Section 46.13 Gatewaying from a Personal LAN over a Modem

Chapter 47 Connecting to MS Windows

Section 47.1 Building Bridges

Section 47.2 Installing and Configuring Samba

Section 47.3 Securing Samba

Section 47.4 SWAT and GUI SMB Browsers

Section 47.5 Printing with Samba

Section 47.6 Connecting to SMB Shares from Unix

Section 47.7 Sharing Desktops with VNC

Section 47.8 Of Emulators and APIs

Section 47.9 Citrix: Making Windows Multiuser

Part IX: Security

Chapter 48 Security Basics

Section 48.1 Understanding Points of Vulnerability

Section 48.2 CERT Security Checklists

Section 48.3 Keeping Up with Security Alerts

Section 48.4 What We Mean by Buffer Overflow

Section 48.5 What We Mean by DoS

Section 48.6 Beware of Sluggish Performance

Section 48.7 Intruder Detection

Section 48.8 Importance of MOTD

Section 48.9 The Linux proc Filesystem

Section 48.10 Disabling inetd

Section 48.11 Disallow rlogin and rsh

Section 48.12 TCP Wrappers

Chapter 49 Root, Group, and User Management

Section 49.1 Unix User/Group Infrastructure

Section 49.2 When Does a User Become a User

Section 49.3 Forgetting the root Password

Section 49.4 Setting an Exact umask

Section 49.5 Group Permissions in a Directory with the setgid Bit Section 49.6 Groups and Group Ownership

Section 49.7 Add Users to a Group to Deny Permissions

Section 49.8 Care and Feeding of SUID and SGID Scripts

Section 49.9 Substitute Identity with su

Section 49.10 Never Log In as root

Section 49.11 Providing Superpowers with sudo

Section 49.12 Enabling Root in Darwin

Section 49.13 Disable logins

Chapter 50 File Security, Ownership, and Sharing

Section 50.1 Introduction to File Ownership and Security

Section 50.2 Tutorial on File and Directory Permissions

Section 50.3 Who Will Own a New File?

Section 50.4 Protecting Files with the Sticky Bit

Section 50.5 Using chmod to Change File Permission

Section 50.6 The Handy chmod = Operator

Section 50.7 Protect Important Files: Make Them Unwritable Section 50.8 cx, cw, c-w: Quick File Permission Changes

Section 50.9 A Loophole: Modifying Files Without Write Access

Section 50.10 A Directory That People Can Access but Can't List Section 50.11 Juggling Permissions

Section 50.12 File Verification with md5sum

Section 50.13 Shell Scripts Must Be Readable and (Usually) Executable Section 50.14 Why Can't You Change File Ownership?

Section 50.15 How to Change File Ownership Without chown

Chapter 51 SSH

Section 51.1 Enabling Remote Access on Mac OS X

Section 51.2 Protecting Access Through SSH

Section 51.3 Free SSH with OpenSSH

Section 51.4 SSH Problems and Solutions

Section 51.5 General and Authentication Problems

Section 51.6 Key and Agent Problems

Section 51.7 Server and Client Problems

Glossary

Copyright © 2003, 1997, 1993 O'Reilly & Associates, Inc All rights reserved

Printed in the United States of America

Published by O'Reilly & Associates, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472

O'Reilly & Associates books may be purchased for educational, business, or sales promotional use

Online editions are also available for most titles (http://safari.oreilly.com) For more information contactour corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com

Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of

O'Reilly & Associates, Inc Many of the designations used by manufacturers and sellers to distinguishtheir products are claimed as trademarks Where those designations appear in this book, and O'Reilly &Associates, Inc was aware of a trademark claim, the designations have been printed in caps or initialcaps The association between the image of a hand drill and the topic of Unix is a trademark of O'Reilly

& Associates, Inc

While every precaution has been taken in the preparation of this book, the publisher and the author assume

no responsibility for errors or omissions, or for damages resulting from the use of the information

contained herein

How to Use This Book

This section refers to conventions used in the print book and explains how they were modified for theSafari version The numbers in the following images correspond to the list below

1 Summary Boxes You'll see gray shaded summary boxes all through the book (On Safari, the Summary Boxes are bordered sidebars with the title "Summary Box.") They summarize a topic and point you to articles with examples and further explanation.

Article/Section Number The first two digits indicate in which chapter the article resides; the last two

digits indicate the number of the article within that chapter The article number is used to refer to thisarticle in all cross-references throughout the book (On Safari, Article numbers correspond to Sectionnumbers.)

Cross-Reference in a Sentence To find out more about the topic displayed in gray type (On Safari,

this text is displayed in boldface.), see the article referenced by the number in parentheses immediatelyfollowing the term.

Cross-Reference in a Code Example When a reference occurs in an example, the

cross-referenced text and related article number appear in the left margin (On Safari, these cross-referencesappear above the code example.)

5 Globe If you don't want to type this script into a file yourself, or if we're talking about a C program that isn't shown, you can download it from the book's web site See the Preface for full details on the content available for download (Online version available at

http://examples.oreilly.com/upt3)

Screw Be careful with this feature, or you might get screwed.

7 Pushpin A note to keep in mind, or a helpful tip.

Bomb A bomb icon in the margin is a cross-reference to another article that explains the possible

trouble you might encounter using the tip or script in the current article (You can think of the bomb as across-referenced screw.) (On Safari, the Bomb appears above the paragraph it refers to.)

Author's Initials The author's full name is listed in the Preface.

A Book for Browsing

Like an Almanac

Like a News Magazine

Like a Hypertext Database

Programs on the Web

About Unix Versions

Cross-References

What's New in the Third Edition

Typefaces and Other Conventions

The Authors

The Fine Print

Request for Comments

Acknowledgments for the First Edition

Acknowledgments for the Second Edition

Acknowledgments for the Third Edition

A Book for Browsing

Technical books can be boring But this is not an ordinary technical book! This book is like an almanac, anews magazine, and a hypertext database all rolled into one Instead of trying to put the topics in perfectorder — and expecting you to start at the beginning, then read through to the end — we hope that you'llbrowse Start anywhere Read what you want (That's not quite true First, you should read this Preface

and the pages before it titled How to Use This Book They will help you get the most out of your time with

this book Next, you may want to skim through the Unix fundamentals in Chapter 1 Then read what you

want.)

Like an Almanac

The book is full of practical information The main purpose isn't to teach you concepts (though they're in

here) We've picked a lot of common problems, and we'll show you how to solve them.

Even though it's not designed to be read in strict order, the book is organized into chapters with relatedsubject matter If you want to find a specific subject, the table of contents is still a good place to start In

addition, several of the chapters contain shaded boxes These are like small tables of contents on a

particular subject, which might be even more limited than the scope of the chapter itself Use the Index

when you're trying to find a specific piece of information instead of a general group of articles about atopic

Like a News Magazine

This book has short articles Most show a problem and a solution — in one page or less The articles are

numbered within each chapter Not all articles are "how-to" tips Some articles have background

information and concepts

Like a Hypertext Database

Each article doesn't define all the concepts and words used Instead, it gives you "links" that let you get

more information if you need it It's easy to get more information when you need it, but just skip the link if you don't Unix Power Tools uses two kinds of links: those in a sentence and those in the margin For examples, see the pages before this Preface titled How to Use This Book.

Programs on the Web

The book describes scripts and freely available programs that are available on the web site An articleabout a program or file that's on the web site will have a globe icon next to it, like this To get one ofthese programs, use our visit the web site:

http://www.oreilly.com/catalog/upt3/

About Unix Versions

There are lots of similarities between different versions of Unix But it's almost impossible to write abook that covers every detail of every version correctly Where we know there might be big differences

or problems, we'll print a note in the text Other places, we're forced to use "weasel words" like "Some

versions of XXX will do ," without telling you exactly which versions When you see those weasel

words, what can you do?

If the command or feature won't destroy anything when it doesn't work, try it! For instance, don't

experiment with rm, the command that removes files But cat, a command that shows files, probably

won't hurt anything if some feature doesn't work with your version.

Look at the online manual or check your vendor's latest printed manuals However, even these can

be wrong For instance, your system administrator may have installed a local version of a commandthat works differently — but not updated the online documentation Be careful with "generic"

manuals, the kind you buy at a bookstore; there are lots of versions of Unix, and the manual may notmatch your version closely enough

Ask your system administrator or another "guru" for help before you use a command that might bedangerous

Cross-References

If a cross-reference is to a single word — for example, a command name like this: tar — the cross

reference is probably to an article that introduces that command Cross references to phrases — like this:

from a parent process to child process — are to an article that explains more about the concept or

problem printed in gray

Cross references don't necessarily give a complete list of all articles about a topic We've tried to pickone or a few articles that give the best information For a more complete list, use the Index

What's New in the Third Edition

There have been some big changes in Unix since we wrote the first edition in the early 1990s, and there'sbeen a surprising number of changes since the second edition, released in the late 1990s Well over half

of the articles have been revised, and we've expanded our coverage of the so-called small Unix flavors:Linux, FreeBSD, Mac OS X's Darwin, and so on

A major change to this edition was the addition of several new topics relevant to today's connected

world, including protecting your machine from attack and several articles related to Internet protocols.We've also added chapters with coverage of two of the more popular languages used in Unix: Perl andPython

Typefaces and Other Conventions

Italic

Is used for the names of all Unix utilities, switches, directories, and filenames and to emphasize newterms and concepts when they are first introduced It's also used in programs and examples to explainwhat's happening or what's been left out at the marks

Bold

Is used occasionally within text to make words easy to find — just like movie stars' names in the

People section of your local newspaper.

Constant width

Is used for sample code fragments and examples A reference in text to a word or item used in anexample or code fragment is also shown in constant width font

Constant width bold

Is used in examples to show commands or text that would be typed in literally by the user

Constant width italic, bold italic

Are used in code fragments and examples to show variables for which a context-specific substitutionshould be made (The variable filename, for example, would be replaced by some actual

filename.)

function(n)

Is a reference to a manual page in Section n of the Unix programmer's manual For example,

getopt(3) refers to a page called getopt in Section 3.

Ë?

Is used in some examples to represent a space charaË?cter

Is used in some examples to represent a space charaË?cter.

In addition, we also had several other authors contribute to all three editions — either people who

originally posted a good tip to Usenet, authors of Nutshell Handbooks who let us take material from theirbooks, or authors of software packages who let us take a few paragraphs from README files or otherdocumentation

Here's a list of authors and their initials:

DH Dave Hitz ML Mike Loukides

The Fine Print

Where we show an article from an author on Usenet, that person may not have thought of the ideaoriginally, but may just be passing on something he or she learned We attribute everything we can

Request for Comments

Please tell us about any errors you find in this book or ways you think it could be improved Our U.S.mail address, phone numbers, and electronic mail address are as follows:

O'Reilly & Associates, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

(800) 998-9938 (in the United States or Canada)

(707) 829-0515 (international/local)

(707) 829-0104 (fax)

bookquestions@oreilly.com (email)

Acknowledgments for the First Edition

This book wouldn't exist without Ron Petrusha As the technical book buyer at Golden-Lee, a major bookdistributor, he discovered us soon after we started publishing Nutshell Handbooks in the mid-1980s Hewas one of our early boosters, and we owed him one So when he became an editor at Bantam (whosecomputer-book publishing operations were later acquired by Random House), we took him seriouslywhen he started asking if there was anything we could do together

At first nothing seemed to fit, since by that time we were doing pretty well as a publisher We needed tofind something that we could do together that might sell better than something that either company might doalone Eventually, Ron suggested that we copublish a Unix book for Bantam's "Power Tools" series Thismade sense for both of us It gave Bantam access to our Unix expertise and reputation, and it gave us achance to learn from Bantam about the mass market bookstore trade, as well as build on their successful

"Power Tools" series

But what would the book contain? There were two features of Bantam's original DOS Power Tools that

we decided to emulate: its in-depth treatment of under-documented system features and its large

collection of freely available scripts and utilities However, we didn't want to write yet another book thatduplicated the format of many others on the market, in which chapters on each of the major Unix toolsfollow one another in predictable succession Our goal was certainly to provide essential technical

information on Unix utilities, but more importantly, to show how the utilities can be combined and used tosolve common (and uncommon) problems

Similarly, because we were weary of the multitude of endless tutorial books about Unix utilities, wewanted to keep the tone brisk and to the point The solution I came up with, a kind of "hypertext in print,"actually owes a lot to Dale Dougherty Dale has been working for several years on hypertext and onlineinformation delivery, and I was trying to get him to work with me on this project So I tried to imagine thekind of book that he might like to create (We have a kind of friendly rivalry, in which we try to leapfrogeach other with ideas for new and better books!) Dale's involvement never went far beyond the earlybrainstorming stage, but the book still bears his indirect stamp In some of the first books he wrote for me,

he introduced the idea that sidebars — asides that illuminate and expand on the topic under discussion —could be used effectively in a technical book Well, Dale, here's a book that's nothing but sidebars!

Dale, Mike Loukides, and I worked out the basic outline for the book in a week or two of brainstormingand mail exchanges We thought we could throw it together pretty quickly by mining many of our existingbooks for the tips and tricks buried in them Unfortunately, none of us was ever able to find enough time,and the book looked to be dying a slow death (Mike was the only one who got any writing done.) SteveTalbott rescued the project by insisting that it was just too good an idea to let go; he recruited Jerry Peek,who had just joined the company as a writer and Unix consultant/tools developer for our production

department

Production lost the resulting tug of war, and Jerry plunged in Jerry has forgotten more Unix tips and tricksthan Mike, Dale, or I ever knew; he fleshed out our outline and spent a solid year writing and collecting

the bulk of the book I sat back in amazement and delight as Jerry made my ideas take shape Finally,though, Jerry had had enough The book was just too big, and he'd never signed on to do it all alone! (Itwas about 1,000 pages at that point, and only half done.) Jerry, Mike, and I spent a week locked up in ourconference room, refining the outline, writing and cutting articles, and generally trying to make Jerry feel

a little less like Sisyphus

From that point on, Jerry continued to carry the ball, but not quite alone, with Mike and I playing "tagteam," writing and editing to fill in gaps I'm especially grateful to Mike for pitching in, since he had manyother books to edit and this was supposed to be "my" project I am continually amazed by the breadth ofMike's knowledge and his knack for putting important concepts in perspective

Toward the end of the project, Linda Mui finished up another book she was working on and joined theproject, documenting many of the freely available utilities that we'd planned to include but hadn't gottenaround to writing up Linda, you really saved us at the end!

Thanks also to all the other authors, who allowed us to use (and sometimes abuse!) their material Inparticular, we're grateful to Bruce Barnett, who let us use so much of what he's written, even though wehaven't yet published his book, and Chris Torek, who let us use many of the gems he's posted to the Netover the years (Chris didn't keep copies of most of these articles; they were saved and sent in by Usenetreaders, including Dan Duval, Kurt J Lidl, and Jarkko Hietaniemi.)

Jonathan Kamens and Tom Christiansen not only contributed articles but read parts of the book with

learned and critical eyes They saved us from many a "power goof." If we'd been able to give them

enough time to read the whole thing, we wouldn't have to issue the standard disclaimer that any errors thatremain are our own H Milton Peek provided technical review and proofreading Four sharp-eyed Usenetreaders helped with debugging: Casper Dik of the University of Amsterdam, Byron Ratzikis of NetworkAppliance Corporation, Dave Barr of the Population Research Institute, and Duncan Sinclair

In addition to all the acknowledged contributors, there are many unacknowledged ones — people whohave posted questions or answers to the Net over the years and who have helped to build the rich texture

of the Unix culture that we've tried to reflect in this book Jerry also singles out one major contributor tohis own mastery of Unix He says: "Daniel Romike of Tektronix, Inc (who wrote Section 28.5 and

Section 30.8 in the early 1980s, by the way) led the first Unix workshop I attended He took the time toanswer a ton of questions as I taught myself Unix in the early 1980s I'm sure some of the insights and neattricks that I thought I've figured out myself actually came from Dan instead."

James Revell and Bryan Buus scoured "the Net" for useful and interesting free software that we weren'taware of Bryan also compiled most of the software he collected so we could try it out and graduallywinnow down the list

Thanks also to all of the authors of the software packages we wrote about and included on the CD!

Without their efforts, we wouldn't have had anything to write about; without their generosity in makingtheir software free in the first place, we wouldn't be able to distribute hundreds of megabytes of softwarefor the price of a book

Jeff Moskow of Ready-to-Run Software solved the problem we had been putting off to the end: that of

packaging up all the software for the original disk, porting it to the major Unix platforms, and making iteasy to install This was a much bigger job than we'd anticipated, and we could never have done it

without Jeff and the RTR staff We might have been able to distribute source code and binaries for a fewplatforms, but without their porting expertise, we could never have ported all these programs to everysupported platform Eric Pearce worked with RTR to pre-master the software for CD-ROM duplication,wrote the installation instructions, and made sure that everything came together at the end! (Eric, thanksfor pitching in at the last minute You were right that there were a lot of details that might fall through thecracks.)

Edie Freedman worked with us to design the format of the book — quite an achievement consideringeverything we wanted the format to do! She met the challenge of presenting thousands of inline cross-references without distracting the reader or creating a visual monstrosity What she created is as attractive

as it is useful — a real breakthrough in technical book design, and one that we plan to use again and

again!

Lenny Muellner was given the frightful task of implementing all of our ideas in troff — no mean feat, and

one that added to his store of grey hair

Eileen Kramer was the copyeditor, proofreader, and critic who made sure that everything came together.For a thousand-plus page book with multiple authors, it's hard to imagine just how much work that was

Ellie Cutler wrote the index; Chris Reilley created the illustrations Additional administrative supportwas provided by Bonnie Hyland, Donna Woonteiler, and Jane Appleyard

—Tim O'Reilly

Acknowledgments for the Second Edition

After teaching myself about Unix for the past 15 years, I'm off to graduate school in Computer Science.Frank Willison, O'Reilly's Editor-in-Chief, fit this project into the summer between leaving my position atORA and starting school Frank didn't just give me something to do in the summer: the royalties shouldhelp to pay for my coursework (So, buy this book and support a student! ;-)) Gigi Estabrook editedthis edition and fielded my zillions of questions along the way Many thanks to Gigi, Frank, and ORA'sProduction staff Clairemarie Fisher O'Leary and Nancy Wolfe Kotary shared the jobs of productioneditor and project manager Madeleine Newell and Kismet McDonough-Chan provided production

support Sheryl Avruch, Nicole Gipson Arigo, and Danny Marcus provided quality control checks Lenny

Muellner provided extensive troff assistance and technical support Chris Reilley created the technical

were extensive enough to mention specially Ted Timar spotted problems that showed his deep

knowledge of Unix I'm glad he still found the book useful enough to read it — and to spot goofs in some

of our hairier tips Andrew T Young sent two long email messages: one a few years ago and another after

I contacted him He caught plenty of techno-goofs and also sent fixes for them Andy doesn't know justUnix: his background in English helped to sharpen a few rough spots in our folksy writing style Finally,Greg Ubben sent a 15-page (!) email message that took me most of a week to work through When I

tracked him down, three years after writing his message, he was even more helpful Greg wrote enough to

make into a small book — and, in fact, agreed to write a few new articles, too He's an expert in sed and

regular expressions (and Unix) who taught me a lot in our month of email messages back and forth I

deeply appreciate all that he's given to this book's readers

—Jerry Peek, jpeek@jpeek.com

Acknowledgments for the Third Edition

Though much of this book is new material or has been newly updated for changes in Unix, there is a corethat remains from previous editions The fact that this material has continued to be fresh, useful, and

relevant through the years is a testament to the abilities — technical and writing — of the original authors.This includes Tim O'Reilly and Jerry Peek, among others previously mentioned, who contributed to pastadditions We, the authors of this current edition, thank you We had a number of terrific reviewers

comment on this version of the text We appreciate the work of Dave Carrano, Chris DiBona, SchuylerErle, Jeff Kawski, Werner Klauser, Adam Langley, Arnold Robbins, Jaron Rubenstein, Kevin Schmidt,Jay Sekora, Joe Sloan, Nat Torkington, and Jay Ts

In addition, I would like to thank those who contribute their time and efforts on Unix systems, particularlythe open source versions of Unix such as FreeBSD, Linux, and now Darwin

—Shelley Powers

I'd just like to thank you all for inviting me to contribute to a book that helped me learn Unix a long timeago It's nice to be able to give something back, given how much the book helped me back in 1994 when Iwas just another Unix newbie

—Steven Champeon

Thank you, Amy and Joel, for the input and review and just for putting up with me through it, and Jasper,for being my strength when I needed it

—Deborah Hooker

Part I: Basic Unix Environment

Chapter 1

Chapter 2

CONTENTS

Chapter 1 Introduction

1.1 What's Special About Unix?

1.2 Power Grows on You

1.3 The Core of Unix

1.4 Communication with Unix

1.5 Programs Are Designed to Work Together

1.6 There Are Many Shells

1.7 Which Shell Am I Running?

1.8 Anyone Can Program the Shell

1.9 Internal and External Commands

1.10 The Kernel and Daemons

1.11 Filenames

1.12 Filename Extensions

1.13 Wildcards

1.14 The Tree Structure of the Filesystem

1.15 Your Home Directory

1.16 Making Pathnames

1.17 File Access Permissions

1.18 The Superuser (Root)

1.19 When Is a File Not a File?

1.20 Scripting

1.21 Unix Networking and Communications

1.22 The X Window System

1.1 What's Special About Unix?

If we were writing about any other operating system, "power tools" might mean "nifty add-on utilities toextend the power of your operating system." That sounds suspiciously like a definition of Unix: an

operating system loaded with decades' worth of nifty add-on utilities

Unix is unique in that it wasn't designed as a commercial operating system meant to run application

programs, but as a hacker's toolset, by and for programmers In fact, an early release of the operatingsystem went by the name PWB (Programmer's Work Bench)

When Ken Thompson and Dennis Ritchie first wrote Unix at AT&T Bell Labs, it was for their own useand for their friends and coworkers Utility programs were added by various people as they had problems

to solve Because Bell Labs wasn't in the computer business, source code was given out to universitiesfor a nominal fee Brilliant researchers wrote their own software and added it to Unix in a spree of

creative anarchy, which has been equaled only with Linux, in the introduction of the X Window System

(Section 1.22), and especially the blend of Mac and Unix with Darwin included in the Mac OS X

Unlike most other operating systems, where free software remains an unsupported add-on, Unix has taken

as its own the work of thousands of independent programmers During the commercialization of Unixwithin the past several years, this incorporation of outside software has slowed down for larger Unixinstallations, such as Sun's Solaris and HP's hp-ux, but not stopped entirely This is especially true withthe newer lighter versions of Unix, such as the various flavors of Linux and Darwin

Therefore, a book on Unix inevitably has to focus not just on add-on utilities (though we do include many

of those), but on how to use clever features of the many utilities that have been made part of Unix over theyears

Unix is also important to power users because it's one of the last popular operating systems that doesn'tforce you to work behind an interface of menus, windows, and mouse with a "one-size(-doesn't)-fit-all"programming interface Yes, you can use Unix interfaces with windows and menus — and they can begreat time savers in a lot of cases But Unix also gives you building blocks that, with some training andpractice, will give you many more choices than any software designer can cram onto a set of menus If youlearn to use Unix and its utilities from the command line, you don't have to be a programmer to do verypowerful things with a few keystrokes

So, it's also essential that this book teach you some of the underlying principles that make Unix such atinkerer's paradise

In the body of this book, we assume that you are already moderately familiar with Unix — a journeymanhacker wanting to become a master But at the same time, we don't want to leave beginners entirely at sea;

so in this chapter, we include some fundamental concepts We've tried to intersperse some simple tips andtricks to keep things interesting, but the ratio of concept articles to tips is much higher than in any otherpart of the book The concepts covered are also much more basic If you aren't a beginner, you can safelyskip this chapter, though we may bounce you back here if you don't understand something later in the book

Don't expect a complete introduction to Unix — if you need that, buy an introductory book What you'llfind here is a selection of key concepts that you'll need to understand to progress beyond the beginnerstage, as well as answers to frequently asked questions and problems In some ways, consider this

introduction a teaser If you are a beginner, we want to show you enough of Unix to whet your appetite formore

Also, don't expect everything to be in order Because we don't want you to get in the habit of readingthrough each chapter from beginning to end, as in most books, the articles in this chapter are in loose

order We've tried not to make you jump around too much, but we've also avoided a lot of the transitionalmaterial that makes reading most books a chore

—TOR, JP, and SP

1.2 Power Grows on You

It has been said that Unix is not an operating system as much as it is a way of thinking In The UNIX

Programming Environment, Kernighan and Pike write that at the heart of the Unix philosophy "is the idea

that the power of a system comes more from the relationships among programs than from the programsthemselves."

Most of the nongraphical utility programs that have run under Unix since the beginning, some 30 yearsago, share the same user interface It's a minimal interface, to be sure — but one that allows programs to

be strung together in pipelines to do jobs that no single program could do alone

Most operating systems — including modern Unix and Linux systems — have graphical interfaces that arepowerful and a pleasure to use But none of them are so powerful or exciting to use as classic Unix pipesand filters, and the programming power of the shell

A new user starts by stringing together simple pipelines and, when they get long enough, saving them for

later execution in a file (Section 1.8), alias (Section 29.2), or function (Section 29.11) Gradually, if theuser has the right temperament, he gets the idea that the computer can do more of the boring part of many

jobs Perhaps he starts out with a for loop (Section 28.9) to apply the same editing script to a series offiles Conditions and cases soon follow and before long, he finds himself programming

On most systems, you need to learn consciously how to program You must take up the study of one ormore programming languages and expend a fair amount of concentrated effort before you can do anythingproductive Unix, on the other hand, teaches programming imperceptibly — it is a slow but steady

extension of the work you do simply by interacting with the computer

Before long, you can step outside the bounds of the tools that have already been provided by the designers

of the system and solve problems that don't quite fit the mold This is sometimes called hacking; in othercontexts, it is called "engineering." In essence, it is the ability to build a tool when the right one is notalready on hand

No single program, however well thought out, will solve every problem There is always a special case,

a special need, a situation that runs counter to the expected But Unix is not a single program It is a

collection of hundreds of them, and with these basic tools, a clever or dedicated person can meet justabout any computing problem

Like the fruits of any advanced system, these capabilities don't fall unbidden into the hands of new users.But they are there for the reaching And over time, even those users who want a system they don't have tothink about will gradually reach out for these capabilities Faced with a choice between an hour spent on

a boring, repetitive task and an hour putting together a tool that will do the task in a flash, most of us willchoose the latter

— TOR

1.3 The Core of Unix

In recent times, more attention has been paid on the newer and more lightweight varieties of Unix:

FreeBSD, Linux, and now Darwin — the version of BSD Unix that Apple used as the platform for thenew Mac OS X If you've worked with the larger Unix versions, you might be curious to see how it differswithin these new environments

For the most part, basic Unix functionality differs very little between implementations For instance, I've

not worked with a Unix box that doesn't have vi (Section 21.7) installed Additionally, I've also not found

any Unix system that doesn't have basic functionality, such as traversing directories with cd (Section 1.16)

or getting additional help with man (Section 2.1)

However, what can differ between flavors of Unix is the behavior of some of the utilities and built-incommands, as well as the options Even within a specific Unix flavor, such as FreeBSD, installations can

differ because one installation uses the built-in version of a utility such as make (Section 40.3) and

another installation has a GNU version of the same application

An attempt was made to create some form of standardization with the POSIX effort POSIX, which stands

for Portable Operating System Interface, is an IEEE standard to work towards application

interoperability With this, C programs written on one flavor of Unix should work, with minimum

modification, on another flavor of Unix

Unfortunately, though the POSIX effort has had some impact on interoperability, there still are significantdifferences between Unix versions In particular, something such as System V Unix can differ

considerably from something such as Darwin

However, there is stability in this seeming chaos: for the most part, the basic Unix utilities and commandsbehave the same in all Unix flavors, and aside from some optional differences, how a command workswithin one environment is exactly the same as in another environment And if there are differences, usingthe facilities described in Chapter 2 should help you resolve these quickly

— SP

1.4 Communication with Unix

Probably the single most important concept for would-be power users to grasp is that you don't "talk"

directly to the Unix operating system Instead, you talk to a program — and that program either talks to Unix itself or it talks to another program that talks to Unix (When we say "talk" here, we mean

communication using a keyboard and a mouse.)

There are three general kinds of programs you'll probably "talk" to:

The program called the shell (Section 27.1) A shell is a command interpreter Its main job is to

interpret the commands you type and to run the programs you specify in your command lines By

default, the shell reads commands from your tty and arranges for other programs to write their results

there The shell protects Unix from the user (and the user from Unix) It's the main focus of this book(and the rest of this article)

An interactive command, running "inside" a tty, that reads what you type directly These take input

directly from the user, without intervention from the shell The shell's only job is to start them up Atext editor, a mail program, or almost any application program (such as word processing) includesits own command interpreter with its own rules This book covers a few interactive commands —

such as the vi editor — but its main focus is the shell and "noninteractive" utilities that the shell

coordinates to do what needs doing

A Graphical User Interface (GUI) with a desktop, windows, and so on On Unix, a GUI is

implemented with a set of running programs (all of which talk to Unix for you)

Unix was around long before GUIs were common, and there's no need to use a GUI to use Unix Infact, Unix started in the days of teletypes, those clattering printing devices used to send telegrams

Unix terminals are still referred to as teletypes or ttys (Section 2.7)

The core of the Unix operating system is referred to as the kernel (Section 1.10) Usually, only programstalk to the kernel (through system calls) Users talk to one of the three previous types of programs, whichinterprets their commands and either executes them directly or passes them on to other programs Theseprograms may, in turn, request lower-level services from the kernel

Let's look at a specific example of using the shell When you type a command to display files whose character filenames start with the letter "m":

four-???Section 1.13

% cat m???

it is the shell that finds the filenames, makes a complete list of them, and calls the cat (Section 12.2)

command to print the expanded list The cat command calls on the kernel to find each file on the disk and

print its contents as a stream of characters on the display

Why is this important? First of all, you can choose between several different shells (Section 1.6), each

of which may have different rules for interpreting command lines

Second, the shell has to interpret the command line you type and package it up for the command you arecalling Because the shell reads the command line first, it's important to understand just how the shellchanges what it reads

For example, one basic rule is that the shell uses "whitespace" (spaces or tabs) to separate each

"argument" of a command But sometimes, you want the shell to interpret its arguments differently For

example, if you are calling grep (Section 13.1), a program for searching through files for a matching line

of text, you might want to supply an entire phrase as a single argument The shell lets you do this by

quoting (Section 27.12) arguments For example:

% grep "Power Tools" articles/*

Understanding how the shell interprets the command line, and when to keep it from doing so, can be very

important in a lot of special cases, especially when dealing with wildcards (Section 1.13), like the *(asterisk) in the previous example

You can think of the relationship of the kernel, the shell, and various Unix utilities and applications as

looking like Figure 1-1.

Figure 1-1 Relationship of kernel, shell, utilities, and applications

Figure 1-1 shows that a user can interact with the shell, as well as directly with interactive commands

like cat and ls The shell transfers control to the commands it starts for you — then those commands may

write the output you see The shell also has some built-in commands (Section 1.9) that run directly withinthe shell itself All of the commands shown in Figure 1-1 interact directly with Unix itself

—TOR and JP

1.5 Programs Are Designed to Work Together

As pointed out by Kernighan and Pike in The UNIX Programming Environment, there are a number of

principles that distinguish the Unix environment One key concept is that programs are tools Like all goodtools, they should be specific in function, but usable for many different purposes

In order for programs to become general-purpose tools, they must be data independent This means threethings:

1 Within limits, the output of any program should be usable as the input to another.

All of the information needed by a program should be either contained in the data stream passed to it orspecified on the command line A program should not prompt for input or do unnecessary formatting ofoutput In most cases, this means that Unix programs work with plain text files that don't contain

"nonprintable" or "control" characters

If no arguments are given, a program should read the standard input (usually the terminal keyboard) andwrite the standard output (usually the terminal screen)

Programs that can be used in this way are often called filters.

One of the most important consequences of these guidelines is that programs can be strung together in

"pipelines" in which the output of one program is used as the input of another A vertical bar (|)