Analytics Press Logo Turning Numbers Into Knowledge: Mastering the Art of Problem Solving

By Jonathan G. Koomey, Ph.D.


Home
About
Contents
Reviews
Resources
Search
Buy
Contact Us
 
Preface

"Whatever failures I have known, whatever errors I have committed, whatever follies I have witnessed in private and public life have been the consequence of action without thought."

—BERNARD BARUCH


Quantitative problem solving is the process by which we take numbers and transform them into knowledge, using our instincts and experience to fill in when we don’t have all the answers. Although the technical aspects of this process are taught at many universities, the art of problem solving is rarely discussed and even more rarely written down. This book teaches the intricacies of that art and will help you become a first rate analyst in your chosen field.

After reading this book, you will be well equipped to make independent judgments about analysis used by others. You will know which key questions to ask, so you need never again be at the mercy of those who traffic in “proof by vigorous assertion.” You will also be more effective at conducting and presenting your own analyses, no matter what the topic.

Mastering the art of problem solving takes more than proficiency with basic calculations: it requires familiarity with how people use information and requires learning about things as diverse as exploring your ideology, telling good stories, and distinguishing facts from values. To give you a feeling for what to expect, I present an annotated chapter list below.

ANNOTATED CHAPTER LIST

This book contains five major sections, separated into 39 short chapters. Each chapter is compact and self-contained, and each summarizes key lessons I’ve learned over the years.

INTRODUCTION • THE INFORMATION EXPLOSION: This section briefly describes how analysis can help reduce the information overload that affects us all.

PART I • THINGS TO KNOW: These chapters summarize ideas to keep in mind as you read the rest of the book. More experienced analysts should delve into the ones they find most intriguing and skim the rest.

Chapter 1 • Beginner’s Mind: Start fresh and approach any problem like a beginner would, and you’ll surely see things that others will miss.

Chapter 2 • Don’t Be Intimidated: The difference between success and failure often depends on whether you are intimidated. By consciously refusing to be cowed, you can stack the odds in your favor.
 
Chapter 3 • Information, Intention, and Action: This chapter describes how humans respond to events, exploring the connections between what we measure, what we assume, and what we choose to do.
 
Chapter 4 • Peer Review and Scientific Discovery: Progress in science can be subject to human frailty, just as in any other human endeavor. The end result, however, is something you can count on, in large part because of the peer review process.

PART I I • BE PREPARED: A key determinant of your effectiveness is the quality of your preparation. Whether you’re building a house or chairing a meeting, preparation for the analysis tasks at hand can turn a potential disaster into a triumph.

Chapter 5 • Explore Your Ideology: Ideology provides a simplified model of the world that reflects your values and experiences and prevents paralysis in the face of the many choices you must make every day. Make sure you know your own belief system and those of others.
 
Chapter 6 • Get Organized: Working and living in chaos is like running a marathon with your feet tied together. Get your life in shape and keep it that way.
 
Chapter 7 • Establish a Filing System: Few mistakes are more maddening than knowing you have seen a relevant article and not being able to find it. By creating a good filing system, you can prevent this annoyance from ever happening again.
 
Chapter 8 • Build a Toolbox: My analytical toolbox is the set of tricks and techniques that I can pull out to solve particular problems. This chapter describes some key tools to consider for your own.
 
Chapter 9 • Put Facts at Your Fingertips: Every analysis requires data. Unless you’ve memorized the encyclopedia, you’ll want to keep some key reference sources within easy reach. This chapter describes the ones I find most useful.
 
Chapter 10 • Value Your Time: Remember that your time is your life, and that if someone is wasting your time, they are stealing your life. Identify your most productive times of day and protect yourself from interruptions during those periods. Unplug the phone. Go to the library. Take control of those times!
 

PART III • ASSESS THEIR ANALYSIS: When faced with the assertions of others, it’s good to know the right questions to ask. These chapters summarize hard-won knowledge about deciphering other people’s analyses.

Chapter 11 • The Power of Critical Thinking: Careful critical thinking is at the root of all good analysis. When the steps described in this chapter become second nature, you will have mastered its essence.
 
Chapter 12 • Numbers Aren’t Everything: Not everything that matters can be quantified, so make sure the unmeasurable doesn’t fall through the cracks.
 
Chapter 13 • All Numbers Are Not Created Equal: Numbers and calculations characterizing the physical world are almost always more certain than those describing human behavior. Many analysts wrongly imply that forecasts based on economic data are just as solid as science. They aren’t, so be forewarned.
 
Chapter 14 • Question Authority: This catch phrase of the 1960s is still applicable today. Authority figures can be wrong or biased, so investigate their assertions in the same way that you’d examine those of someone with whom you’re not familiar.
 
Chapter 15 • How Guesses Become Facts: Always remember that “official” statistics are based on calculations that are often poorly documented, incorrectly cited, or otherwise hazardous to your intellectual health.
 
Chapter 16 • Don’t Believe Everything You Read: Maintain a healthy skepticism, even of well-established sources. In this age of instant information transmission, rumor and error seem to propagate even more quickly than truth.
 
Chapter 17 • Go Back to the Questions: Any time you rely on survey data to make an important decision, refer back to the actual questionnaire upon which the survey data are based; otherwise, you risk misinterpreting the data.
 
Chapter 18 • Reading Tables and Graphs: First check for internal consistency, then see if the results contradict other facts you know to be true. Search for cognitive dissonance; any discrepancy between the author’s results and what you already know will help you investigate further.
 
Chapter 19 • Distinguish Facts From Values: Don’t be fooled by technical people who portray their advice as totally rational, completely objective, and value-free. If they have made a choice, they have also made a value judgment.
 
Chapter 20 • The Uncertainty Principle and the Mass Media: Just as the observer of a subatomic particle can disturb that particle by the act of observation, the observer of an institution can disturb that institution by observing and reporting on it. Members of the media (and the analysts who inform them) should take responsibility for the power they wield.

PART IV • CREATE YOUR ANALYSIS: Everyone develops his/her own techniques for creating cogent analyses, and in this section I summarize those I’ve learned. The importance of organization, clear thinking, careful definitions, systematic exposition, scrupulous documentation, and consistent comparisons cannot be overestimated, and you’ll learn about each of these here.

Chapter 21 • Reflect: Free yourself from interruptions and give yourself time to reflect. Without such time, you’ll never achieve your full problem-solving potential.
 
Chapter 22 • Getting Unstuck: Everyone gets stuck sometimes, but this pitfall need not hobble your efforts if you use the tricks in this chapter.
 
Chapter 23 • Inquire: When faced with a problem outside your expertise, don’t surrender! It’s an advantage to be unconstrained by the mental shackles most disciplines place on their practitioners. Some of the most important insights in modern thought came from people who could think “outside the box.”
 
Chapter 24 • Be a Detective: Detectives are real-world practitioners of the scientific method. The time-honored techniques of these seasoned problem solvers can help you in your efforts.
 
Chapter 25 • Create Consistent Comparisons: People often relate best to anecdotes. A consistent comparison is a well-chosen set of two anecdotes that illustrates your point in a compelling way. It is a powerful technique and one well worth learning.
 
Chapter 26 • Tell a Good Story: Scenario analysis is the art of structured storytelling, and it’s an essential tool for any good analyst. Most people don’t realize that this art is both highly developed and pertinent to many everyday situations.
 
Chapter 27 • Dig into the Numbers: Don’t be shy about delving into the actual numbers even if you’re a highly paid executive. You’ll learn things you’d never see if someone else crunches the numbers.
 
Chapter 28 • Make a Model: Models are “laboratories for the imagination,” and this chapter explores the subtleties of using them to explain the world around you.
 
Chapter 29 • Reuse Old Envelopes: You can calculate almost anything using only common knowledge–you just need to learn how to put this knowledge to use, and this chapter (which focuses on back-of-the-envelope calculations) is just the thing to help you do it.
 
Chapter 30 • Use Forecasts with Care: The future is uncertain, but people keep trying to forecast it anyway. Numerous pitfalls await, and without a keen eye for the tricks of this trade, you’ll be hard pressed to avoid them.
 
Chapter 31 • Hear All Sides: In any intellectual dispute, it pays to hear two well-prepared debaters argue their points before drawing any conclusions. Always make such debates fodder for your deliberations, and your decisions will benefit.

PART V • SHOW YOUR STUFF: Once you’ve done good work, you’ll want to present it effectively to readers or listeners. The chapters in this section give insights into making your results “grab” your audience, designing good tables and figures, and using those tables and figures to convey your key points. The section concludes by exploring effective use of the Internet in publishing your analysis.

Chapter 32 • Know Your Audience: The first rule of data presentation is to know your audience and present compelling information in a form your readers can easily grasp. Most analysts forget that other people don’t care nearly as much about their results as they themselves do.
 
Chapter 33 • Document, Document,Document: An astounding number of analysts routinely omit vital data and assumptions from their reports, but you should avoid this pernicious practice. The best analysts document everything, giving credit where credit is due, leaving a trail so they can remember, and creating a trail for others to follow. Documentation is also a key step in checking your work, because it forces you to think clearly about your analysis.
 
Chapter 34 • Let the Tables and Graphs Do the Work: When writing technical reports, create the analysis, tables, and graphs first; then, write around them. If the analysis is well thought out, the tables and graphs well designed, and the audience clearly defined, the report should practically write itself.
 
Chapter 35 • Create Compelling Graphs and Figures: Follow Edward Tufte’s rules for graphical excellence, and avoid the most common pitfalls in designing charts and graphs. Your goal should be to give to the reader “the greatest number of ideas in the shortest time with the least ink in the smallest space.”
 
Chapter 36 • Create Good Tables: A well-designed table is a work of art; a sloppy one is worse than useless. Make your tables a resource that your readers will keep as a reference for many years to come.
 
Chapter 37 • Use Numbers Effectively in Oral Presentations: Even veteran presenters show too many of the wrong numbers. Present only those numbers that support the story you are telling, and focus on that story, NOT on the numbers themselves.
 
Chapter 38 • Use the Internet: The old ways of publishing are fast being supplanted by web-based approaches. Learn about these new tools and put them to work for you.

Chapter 39 • Share and Share Alike: Some kinds of information are more valuable when shared. Although there's still work to be done, web technology is finally automating the data sharing process, allowing you to capture benefits in standardization, efficiency, and greater analytical insight.

CONCLUSIONS • CREATING THE FUTURE: This chapter gives perspective on why we use analysis in the first place. Understanding the world is a prerequisite for making it better!  

EPILOGUE • SOME PARTING THOUGHTS:After the first edition of Turning Numbers into Knowledge was published in 2001 I encountered some widely believed but erroneous statistics, and this epilogue recounts the lessons I learned in debunking them.

WHO SHOULD READ THIS BOOK

This book grew out of my experience in training analysts whom I’ve hired in the past decade. It is written for beginning problem solvers in business, government, consulting, and research professions, and for students of business and public policy. It is also intended for supervisors of such analysts, professors, and entrepreneurs (who may not consider themselves analysts but who need to create analyses to justify their business plans to potential investors). Finally, it covers many topics that journalists who focus on scientific or business topics will find useful.  

HOW TO USE THIS BOOK

There is no need to read the chapters in order. Go straight to those that interest you most, but skim the chapters you skip. You just might see something useful there that you did not expect.

Most chapters have “links” to other chapters, with graphical signposts indicating which chapter or major section to investigate for each link (the relevant chapter number appears inside). These signposts look like the link to Chapter 14 that appears in the right margin opposite this line in the printed version of the book.

All Uniform Resource Locators (URLs) discussed in the book are enclosed in triangular brackets, to set them off from the text. They appear as follows: <http://www.lbl.gov>. The brackets and any punctuation marks that precede or follow them are not part of the URL.

All URLs, as well as many key data files, are available in electronic form at <http://www.numbersintoknowledge.com>. If you have questions, comments, or suggestions, you can post them at this site. I’ll gladly evaluate them for inclusion in the next edition. I’m particularly interested in examples of large and public analytical blunders by people who should know better, examples of bad or good tables and graphs, and suggestions for how the book can be improved or expanded.

The endnotes contain references, attributions, and further information for the interested reader. The Further Reading at the end of the book does not attempt to be comprehensive. Rather, it contains selected sources for each chapter that I regard as most crucial for mastering the material. If I mention a book, I include it in the further reading section for the chapter in which I refer to it. At the beginning of the Further Reading section, I also include the list of my very favorite sources on this topic, which (in my opinion) are the “must read” items that all serious problem solvers should have on their shelves.  

WHAT'S NEW IN THE SECOND EDITION

In my revisions for this edition I've focused on tightening up the text, revising and improving the examples, updating data where appropriate, and updating and expanding the further reading section. I also added a new chapter on data sharing sites as well as an Epilogue, which describes some of what I've learned since the first edition of Turning Numbers into Knowledge was published in 2001. Finally, John Holdren graciously consented to write a new foreword. I hope you enjoy reading this book as much as I enjoyed writing it!


Like all other arts, the science of deduction and analysis is one that can only be acquired by long and patient study."

— SHERLOCK HOLMES


To download a copy of this preface in PDF form (512k), click here. To download the book jacket in PDF form (209k), click here. Feel free to distribute these PDFs to others who may be interested in Turning Numbers into Knowledge.

To download a copy of the latest free Adobe Acrobat reader (which reads PDF files), go here.