World Scientic www.worldscientific.com 7229 hc ,!7IJ8B4-cgbgfe! ISBN-13 978-981-4261-65-4 ISBN-10 981-4261-65-3 World Scientic Vol. 4 Computer and Network Security Firewall Design and Analysis Alex X. Liu This unique book represents the rst rigorous and comprehensive study of rewall policy design and analysis. Firewalls are the most critical and widely deployed intrusion prevention systems. Designing new rewall policies and analyzing existing rewall policies have been difcult and error-prone. This book presents scientically sound and practically useful methods for designing and analyzing rewall policies. This book is useful to a variety of readers. First, it can be used as a handbook for network/rewall administrators and network security professionals. Second, it can be used as an advanced textbook for graduate students and senior undergraduate students in computer science and engineering. Third, it is also suitable for non-experts in network security who wish to understand more about rewalls. The presentation of the book is detailed enough to capture the interest of curious readers, and complete enough to provide the necessary background material needed to delve further into the subject of rewalls and network security. Liu Vol. 4 Firewall Design and Analysis Firewall Design and Analysis 7229.04.10.Kwang Wei.ML.new.indd 1 10/7/10 10:39 AM Firewall Design and Analysis 7229tp.indd 1 10/1/10 1:56 PM N E W J E R S EY • L O N D ON • S I N G AP O R E • B E I J IN G • S H A N GH A I • H O N G K O N G • T A I P E I • C H E N NA I World Scientic Alex X. Liu Michigan State University, USA Vol. 4 Computer and Network Security Firewall Design and Analysis 7229tp.indd 2 10/1/10 1:56 PM SERIES IN COMPUTER AND NETWORK SECURITY Series Editors: Yi Pan (Georgia State Univ., USA) and Yang Xiao (Univ. of Alabama, USA) Published: Vol. 1: Security in Distributed and Networking Systems eds. Xiao Yang et al. Vol. 2: Trust and Security in Collaborative Computing by Xukai Zou, Yuan-Shun Dai and Yi Pan Vol. 3: Security in Ad Hoc and Sensor Networks by Raheem Beyah, Janise McNair and Cherita Corbett Vol. 4: Firewall Design and Analysis by Alex X. Liu KwangWei - Firewall Design.pmd 10/1/2010, 2:15 PM2 British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. Desk Editor: Tjan Kwang Wei ISBN-13 978-981-4261-65-4 All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. Copyright © 2011 by World Scientific Publishing Co. Pte. Ltd. Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE Printed in Singapore. FIREWALL DESIGN AND ANALYSIS Computer and Network Security — Vol. 4 KwangWei - Firewall Design.pmd 10/1/2010, 2:15 PM1 March 25, 2010 15:0 World S cientific Boo k - 9in x 6in Boo kFirewallDesignAnalysis Dedicated with love and respect to my pa rents Shuxiang Wang and Yuhai Liu (God rest his soul), to Huibo Heidi Ma to my twin sons Max Boyang and Louis Boyang, to whom I owe all that I am and all that I have accomplished. c ⃝[2010] IEEE. Reprinted, with permission, from Proceeding s of the 24th IEEE International Conference on Distributed Computing Systems 2004 (“Firewall Design: Consistency, Co mpleteness and Compactness”), Proceedings of the IEEE International Conference on Dependable Systems and Networks 2004 (“Diverse Firewall Design”), Proceedings of the IEEE International Conference on Dependable Systems and Networks 2005 (“A Model of Stateful Firewalls and its Prop erties”), IEEE Transactions on Parallel and Distributed Systems (“Diverse Firewall Design” and “Firewall Policy Queries”). This page is intentionally left blank January 13, 2010 14:41 World S cientific Boo k - 9in x 6in Boo kFirewallDesignAnalysis Preface Firewalls are the mo st critical and widely deployed intrusion prevention sys- tems. A firewall is a security guard placed at the point of entry between a private network and the outside Internet such that all incoming and outgo- ing packets have to pass through it. The function of a firewall is to exa mine every incoming or outgoing packet and decide whether to accept or discard it. This function is conventionally specified by a sequence of rules, where rules often conflict. To resolve conflicts, the decision for each packet is the decision of the first rule that the packet matches. Conseq ue ntly, the rules in a firewall are order sensitive. Because of the conflicts and order sensitivity of firewall rules, firewalls are difficult to design and analyze correctly. It has been observed that most firewalls on the Internet are poorly designed and have many errors in their rules. Towards the goal of correct firewalls, this book focuses on the following two fundamental pro ble ms : first, how to design a new firewall such that the number of errors introduced in the design phase is small; second, how to analyze an existing firewall such that we can detect errors that have been built in. For firewall design, we present two methods for designing stateless firewalls, namely the method of structured firewall design and the method of diverse firewall design, and a model for specifying stateful firewalls. For firewall analysis, we present two methods, namely firewall queries and firewall redundancy detection. The firewall design and analysis methods presented in this book are not limited to just firewalls. Rather, they are extensible to other rule-based systems such as general packet classification systems and IPsec. Alex X. Liu vii This page is intentionally left blank January 13, 2010 14:41 World S cientific Boo k - 9in x 6in Boo kFirewallDesignAnalysis Contents Prefac e vii 1. Prologue 1 1.1 Background and Motivation . . . . . . . . . . . . . . . . . 1 1.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Previous Work on Firewall Design . . . . . . . . . 3 1.2.2 Previous Work on Firewall Analysis . . . . . . . . 4 1.3 Contributions of the Book . . . . . . . . . . . . . . . . . . 5 1.3.1 Structured Firewall Design . . . . . . . . . . . . . 5 1.3.2 Diverse Firewall Design . . . . . . . . . . . . . . . 6 1.3.3 Stateful Firewall Model . . . . . . . . . . . . . . . 6 1.3.4 Firewall Queries . . . . . . . . . . . . . . . . . . . 7 1.3.5 Firewall Redundancy Detection . . . . . . . . . . 8 1.4 Overview of the Book . . . . . . . . . . . . . . . . . . . . 8 2. Structured Firewall Desig n 9 2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1.1 Consistency, Completeness and Compa ctness . . . 9 2.1.2 Structured Firewall Design . . . . . . . . . . . . . 12 2.2 Firewall Decision Diagrams . . . . . . . . . . . . . . . . . 13 2.3 FDD Reduction . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4 FDD Marking . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.5 Firewall Generation . . . . . . . . . . . . . . . . . . . . . 21 2.6 Firewall Compaction . . . . . . . . . . . . . . . . . . . . . 23 2.7 Firewall Simplification . . . . . . . . . . . . . . . . . . . . 26 2.8 Summary of Structured Firewall Design . . . . . . . . . . 28 ix [...]... as accept, accept -and- log, discard, and discard -and- log Our firewall design and analysis methods can be straightforwardly extended to support more than two decisions The firewall design and analysis methods presented in this book are not limited to just firewalls Rather, they are extensible to other rule- January 13, 2010 14:41 World Scientific Book - 9in x 6in Prologue BookFirewallDesignAnalysis 3 based... 9in x 6in BookFirewallDesignAnalysis Firewall Design and Analysis First, despite its simplicity, it can express a variety of state tracking functionalities Second, it allows us to inherit the rich results in stateless firewall design and analysis Third, it provides backward compatibility such that a stateless firewall can also be specified using our model 1.2.2 Previous Work on Firewall Analysis Previous... and ∣Σ∣ = ∣𝐷(𝐹1 )∣ × ⋅ ⋅ ⋅ × ∣𝐷(𝐹𝑑 )∣, where ∣Σ∣ denotes the number of elements in set Σ and each ∣𝐷(𝐹𝑖 )∣ (1 ≤ 𝑖 ≤ 𝑑) denotes the number of elements in set 𝐷(𝐹𝑖 ) Definition 2.2.1 (Firewall Decision Diagram) A Firewall Decision January 13, 2010 14 14:41 World Scientific Book - 9in x 6in BookFirewallDesignAnalysis Firewall Design and Analysis Diagram (FDD) 𝑓 over fields 𝐹1 , ⋅ ⋅ ⋅ , 𝐹𝑑 is an acyclic and. .. the designer to consider all types of traffic It also addresses the compactness problem because in the second step we first used two algorithms, a standard algorithm for decision diagram reduction and a new algorithm called firewall January 13, 2010 14:41 6 World Scientific Book - 9in x 6in BookFirewallDesignAnalysis Firewall Design and Analysis decision diagram marking, to combine rules together, and then... and effective SQL-like query language, called the Structured Firewall Query Language (SFQL), for describing firewall queries; a theorem, called the Firewall Query Theorem, as a foundation for developing firewall query processing algorithms; and an efficient firewall query processing algorithm January 13, 2010 14:41 8 1.3.5 World Scientific Book - 9in x 6in BookFirewallDesignAnalysis Firewall Design and Analysis. .. languages is still a sequence of rules and the rules may still conflict The three issues of consistency, completeness and compactness that are inherent in designing a firewall by a sequence of rules still remain In comparison, in this book, we present two new firewall design methods: Structured Firewall Design and Diverse Firewall Design The Structured Firewall Design method is the first method that addresses... Scientific Book - 9in x 6in BookFirewallDesignAnalysis Firewall Design and Analysis Based on Theorem 2.2.1 and 2.5.1, we now extend the equivalence relations on FDDs to incorporate the firewalls Given 𝑓 and 𝑓 ′ , where each is an FDD or a firewall, 𝑓 and 𝑓 ′ are equivalent iff they have identical accept sets and identical discard sets, i.e., 𝑓.accept = 𝑓 ′ accept and 𝑓.discard = 𝑓 ′ discard This equivalence... built in For firewall design, we present two methods for designing stateless firewalls, namely the method of structured firewall design and the method of diverse firewall design, and a model for specifying stateful firewalls For firewall analysis, we present two methods, namely firewall queries and firewall redundancy detection 1.3.1 Structured Firewall Design Designing a firewall directly by a sequence of rules... rules 1.3.2 Diverse Firewall Design Fundamentally, firewall errors result from human errors To reduce human errors, we present the method of Diverse Firewall Design in [Liu and Gouda (2004)] This method consists of two phases: a design phase and a comparison phase In the design phase, the same requirement specification of a firewall is given to multiple teams, who proceed independently to design the firewall... from Firewall State Firewall States 4.4.1 Truly Stateful and Truly Stateless Firewalls 4.4.2 Stateless Derivatives Firewall Properties 4.5.1 Conforming Firewalls 4.5.2 Proper Firewalls Epilogue 51 56 56 57 60 62 63 64 65 65 66 69 Structured Firewall Query Language 5.1.1 Firewalls . 4 Computer and Network Security Firewall Design and Analysis Alex X. Liu This unique book represents the rst rigorous and comprehensive study of rewall policy design and analysis. Firewalls. and network security. Liu Vol. 4 Firewall Design and Analysis Firewall Design and Analysis 7229.04.10.Kwang Wei.ML.new.indd 1 10/7/10 10:39 AM Firewall. Yuan-Shun Dai and Yi Pan Vol. 3: Security in Ad Hoc and Sensor Networks by Raheem Beyah, Janise McNair and Cherita Corbett Vol. 4: Firewall Design and Analysis by Alex X. Liu KwangWei - Firewall Design. pmd