Handbook of Data Quality

Transkript

1 Handbook of Data Quality

2

3 Shazia Sadiq Editor Handbook of Data Quality Research and Practice 123

4 Editor Shazia Sadiq University of Queensland Brisbane Australia ISBN ISBN (ebook) DOI / Springer Heidelberg New York Dordrecht London Library of Congress Control Number: ACM Computing Classification (1998): H.2, H.3, E.5, K.6 c Springer-Verlag Berlin Heidelberg 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (

5 Advisory Panel Carlo Batini Università degli Studi di Milano - Bicocca, Milano, Italy Yang Lee North Eastern University, Boston, MA, USA Chen Li University of California Irvine, Irvine, CA, USA Tamer Ozsu University of Waterloo, Waterloo, ON, Canada Felix Naumann Hasso Plattner Institute, Potsdam, Germany Barbara Pernici Politecnico di Milano, Milano, Italy Thomas Redman Navesink Consulting Group, Rumson, NJ, USA Divesh Srivastava AT&T Labs-Research, Florham Park, NJ, USA John Talburt University of Arkansas at Little Rock, Little Rock, AR, USA Xiaofang Zhou The University of Queensland, Brisbane, Australia v

6

7 To my late father, Aziz Beg historian, author and diplomat

8

9 Preface The impact of data quality on the information chain has been widely recognized since the onset of large-scale data processing. Furthermore, recent years have seen a remarkable change in the nature and usage of data itself due to the sheer volume of data, high accessibility leading to unprecedented distribution and sharing of data, and lack of match between the intention of data creation and its subsequent usage, to name a few. The importance of the understanding and management of data quality for individuals, groups, organizations, and government has thus increased multifold. The data (and information) quality domain is supported by several decades of high quality research contributions and commercial innovations. Research and practice in data and information quality is characterized by methodological as well as topical diversities. The cross-disciplinary nature of data quality problems as well as a strong focus on solutions based on the fitness for use principle has further diversified the related body of knowledge. Although research pluralism is highly warranted, there is evidence that substantial developments in the past have been isolationist. As data quality increases in importance and complexity, there is a need to motivate exploitation of synergies across diverse research communities. The above factors warrant a multipronged approach to the study of data quality management spanning: organizational aspects, i.e. strategies to establish people, processes, policies, and standards required to manage data quality objectives; architectural aspects, i.e. the technology landscape required to deploy developed processes, standards, and policies; and computational aspects which relate to effective and efficient tools and techniques for data quality. Despite a significant body of knowledge on data quality, the community is lacking a resource that provides a consolidated coverage of data quality over the three different aspects. This gap motivated me to assemble a point of reference that reflects the full scope of data quality research and practice. In the first chapter of this handbook, I provide a detailed analysis of the data quality body of knowledge and present the rationale and approach for this handbook, particularly highlighting the need for cross-fertilization within and across research and practitioner communities. This handbook is then accordingly structured into three parts representing contributions on organizational, architectural, ix

10 x Preface and computational aspects. There is also a fourth part, devoted to case studies of successful data quality initiatives that highlight the various aspects of data quality in action. This handbook concludes with a chapter that outlines the emerging data quality profession, which is particularly important in light of new developments such as big data, advanced analytics, and data science. The preparation of this handbook was undertaken in three steps. Firstly, a number of global thought leaders in the area of data quality research and practice were approached to join the initiative as part of the advisory panel. The panel members contributed significantly to the refinement of this handbook structure and identification of suitable chapter authors and also supported the review process that followed chapter submissions. The identified chapter authors were then invited to provide contributions on the relevant topics. Finally, all chapter contributions were reviewed by at least two experts. To ensure that the quality of the final chapters was not compromised in any way, some contributions were rejected or substantially revised over two or three review cycles. However, I am most grateful for the time devoted by all authors to produce high quality contributions and especially for the responsiveness of the authors towards making the required changes. I would like to take this opportunity to thank all the authors and advisors for their valuable contributions. A special thanks to Xiaofang Zhou, Divesh Srivastava, Felix Naumann, and Carlo Batini for their guidance and inspiration in the preparation of this handbook. Thanks to all the expert reviewers of the chapters, with a special thanks to Mohamed Sharaf for constant encouragement and advice. Last but not least, a big thanks to Kathleen Willamson, Yang Yang and Vimukthi Jayawardene for an enormous help in the editing and formatting work required for the preparation of this handbook. I hope that this Handbook of Data Quality will provide an appreciation of the full scope and diversity of the data quality body of knowledge and will continue to serve as a point of reference for students, researchers, practitioners, and professionals in this exciting and evolving area. Brisbane, Australia April 2013 Shazia Sadiq

11 Contents Prologue: Research and Practice in Data Quality Management... 1 Shazia Sadiq Part I Organizational Aspects of Data Quality Data Quality Management Past, Present, and Future: Towards a Management System for Data Thomas C. Redman Data Quality Projects and Programs Danette McGilvray Cost and Value Management for Data Quality Mouzhi Ge and Markus Helfert On the Evolution of Data Governance in Firms: The Case of Johnson & Johnson Consumer Products North America Boris Otto Part II Architectural Aspects of Data Quality Data Warehouse Quality: Summary and Outlook Lukasz Golab Using Semantic Web Technologies for Data Quality Management Christian Fürber and Martin Hepp Data Glitches: Monsters in Your Data Tamraparni Dasu xi

12 xii Contents Part III Computational Aspects of Data Quality Generic and Declarative Approaches to Data Quality Management Leopoldo Bertossi and Loreto Bravo Linking Records in Complex Context Pei Li and Andrea Maurino A Practical Guide to Entity Resolution with OYSTER John R. Talburt and Yinle Zhou Managing Quality of Probabilistic Databases Reynold Cheng Data Fusion: Resolving Conflicts from Multiple Sources Xin Luna Dong, Laure Berti-Equille, and Divesh Srivastava Part IV Data Quality in Action Ensuring the Quality of Health Information: The Canadian Experience Heather Richards and Nancy White Shell s Global Data Quality Journey Ken Self Creating an Information-Centric Organisation Culture at SBI General Insurance Ram Kumar and Robert Logie Epilogue: The Data Quality Profession Elizabeth Pierce, John Talburt, and C. Lwanga Yonke About the Authors Index