Research for Critical Thinking

The Keystone Approach                                                                                                           

The curved sides of an arch meet at the top where a wedge-shaped stone sits. This final piece -- called the “keystone” -- locks all the other stones into position. It is the central support for the entire structure.  Without it, everything would come tumbling down. When constructing an argument, researchers need to build strong support for their arguments by identifying the keystones for the most powerful pieces of evidence that reinforce the claim of the paper.

Logos: identifying a keystone document

Often, a single article, book, or website can be the keystone that helps organize the other important information about a topic. Sometimes these are obvious: Darwin’s Origin of the Species is the central source for evolutionary studies. The book established the key terms and concepts that produced modern biology. Darwin’s book was published in 1859, so the history of references to its discoveries is huge.

Other key texts are not as obvious. If you are interested in search engines and how they isolate us in a bubble of agreeable information, it is important to know about The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think. The “filter bubble” is a new term that describes how search engines work. The term is new; search engines are new; the problem is just becoming visible. A student who hears the book discussed in a Communications Studies class can use it to make connections to others who are thinking about the problem. You hear about these important sources in lectures, textbooks, and classroom discussion.

Good students use these central texts to find more information for the topic that they are researching. Of course, there are far fewer connections to this book than to The Origin of the Species, but part of what you are doing in both cases is re-drawing the map of a topic after finding out what is most important in the existing maps.


Using a keystone text is fairly simple. First, such works contain a bibliography. These items are the network that connects to the keystone. The expert who wrote the book names this network in order to strengthen its logos (logic, evidence). One way to make certain that the text really is a keystone document is to see how often others have used it. For example, if we go to a specialized search engine such as and search for The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think this is what appears:

[CITATION] The filter bubble: What the Internet is hiding from you

E Pariser - 2011 - Penguin

Cited by 258 Related articles All 14 versions Cite

The book has been used in 258 related articles (and many more since). This is strong evidence that the book is important and can be considered a keystone text for the topic.

But suppose that you are interested in stem cells, and a textbook mentions Hwang Woo Suk’s “Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst.” You turn to and find the following:

Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst

WS Hwang, YJ Ryu, JH Park, ES Park, EG Lee, JM Koo… - Science, 2004 -

Abstract Somatic cell nuclear transfer (SCNT) technology has recently been used to

generate animals with a common genetic composition. In this study, we report the derivation

of a pluripotent embryonic stem (ES) cell line (SCNT-hES-1) from a cloned human ...

Cited by 724 Related articles All 55 versions Cite

It seems as if you’ve discovered a keystone document: 724 citations seems impressive. On the other hand, your biology professor could tell you that the article is a famous example of scientific fraud. This requires that you not only take notes, but also that you read carefully, and that you speak with the experts in the field (your professor) who can explain that many of those citations are about the fact that the work is fraudulent. Research is a social, human activity, and seeing your professors as experts reflects that fact.

Professors help direct you to keystone sources, but there are other tools that help identify such works. Such work is usually published in a peer reviewed journal or a university press.

Techniques for Finding Expertise                                                                                  

Good searches require good questions. Good questions contain precise terms. Thus, the task seems simple: use good terms to search for reliable information. Of course, it is not that simple because you cannot know what terms are useful until you understand your topic. It is no surprise that students feel paralyzed early in the search process. Writers must recognize that the identification of key terms is a two-steps-forward, one-step-back process.

What are good places to build key terms for searches?

The list of key terms is incomplete at the beginning of your search. As writers read and take notes, they find new terms that can be used in additional searches. Good students keep a growing list of key terms drawn from the following:

Getting started

• Class notes: technical terms the professor has used or emphasized in class

• Textbooks: words emphasized with italics or bold print because they are shared by experts in the field.

• Titles of articles: the bibliographies provided by professors, textbooks, or others have terms that represent key concepts in how experts understand a topic

• Conversations with professors: professors have lifelong interests in the topic they teach. Asking about key terms is evidence of being a serious student.

Identifying expert terms and sources with online tools

• Use general search engines,, etc. to find overviews of the topic.

• Recognize documents that provide an expert overview of a topic; note that full text documents are not always necessary at this stage because writers need terms and concepts.

• Identify key scholars and follow them on social media such as Twitter, Facebook, and blogs to access important questions and current developments in the field.

Some of the most interesting topics are new. The book about “the filter bubble” is an example of a topic that is so new that there is not yet a huge amount of scholarship. However, many important scholars are writing about it in blogs, on Twitter, and in other digital publications. What they say is crucial to creating a good paper about the topic. Many scholars use Twitter to identify significant postings, new articles, and other material. The author of The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think is Eli Pariser. He “follows” 733 authorities on the topic. Anyone interested in Pariser’s work can log in to Twitter and see what he recommends. Some people may think that Twitter is only for announcing personal information. However, it has become a standard tool for pointing to important questions, terms, information, writers, and discoveries. You can set up your own Twitter account and collect information from the experts that experts read.

A professor whose class was writing about the “filter bubble” created a Twitter account for her students, and then emailed this message to the class:

For Twitter, think of it as a way to stay up to date on the kinds of things that were discussed in the Pariser talk and the essays we read. Too often, students are taught that research means looking up a few things on google and then they are done. This is not the way we should be doing things. We need to be aware of the on-going conversations about these ideas--in our case: privacy, filtering, algorithms....and be aware of how people are talking about them and thinking about them. We have talked about developing a "working knowledge" of a topic, and Twitter is one way to do that. If you take a look at what has been tweeted since class began, there is lots of stuff that pertains to what you are writing about and could serve as excellent resources for your paper. Lots of experts in various fields are on Twitter and they use it to communicate with people who are interested in the same things. So, if you find any relevant essays, you should tweet the link to me and I will send it out to the class.

Again, knowledge is social, and social media can strengthen that knowledge.

Tools for Finding Information                                                                                          

Search engines can find the sources needed to create a map of the existing information about a topic. While most students have “googled” information, few know how to select the right search engine, identify key terms, or do an advanced search. Knowing how to do these things saves time, and it also produces the useful results needed for a good paper.

What is a search engine?

Search engines are the librarians of the Internet. When you want to know about a subject, you ask the search engine where it is stored; then, it tells you how to get there. The search engine does not have the information. It tells where on the Internet that the material exists. Thus, no one ever “found it on Google.” Google only provides a list of links to other web sites. Like a librarian, the search engine does not know the subject, but it does know where to find information about it. Remember: a search engine does something; it finds things. Where does it find them? It finds them in an index of web sites, words and images. Sometimes the index includes materials that have been reviewed by experts. Sometimes the index comes from a wide range of sources, some of which are expert and others of which are not. It is important to know what the search engine has indexed and can provide to you.

What kinds of information do general search engines locate?

About 85% of all English language searches rely on Google, Bing, or Yahoo. These general search engines provide a map to other places with information related to the key words in the search. Note that different people will get different results even if they both use the same terms. Many websites and search engines record a user’s interests, past online choices, geographic location, and other factors. These factors are used to “personalize” the results of the search. In some ways, this is good. If a user searches for pizza, it helps that only pizza parlors in her/his area show up on the list. On the other hand, the results for politics, ideology, beliefs, and values are likely to be tilted toward those that agree with what the searcher already thinks. Personalization can turn a search engine into a kind of sneaky librarian who only sends you to information s/he wants you to know. Be suspicious of search results.

note: metasearch engines are search engines that combine the results of other search engines. Two important metasearch engines are and

Some general search engines, for example,, do not personalize results.

Do search engines search the entire web?

Even the most powerful search engine indexes only the surface of the web. The “deep web” escapes indexing, and it is probably 100 to 1,000 times the size of the surface web. Other search engines appear after arriving at a web site. They search only within the site. For example, many students turn to Purdue University’s splendid Online Writing Lab (OWL) at . This site has a search engine, but it only searches inside the Purdue database of materials. This assures users that the information has been created and approved by the Purdue writing program. Many businesses use a similar type of restriction so that customers stay within their products.

Social media such as Twitter and Facebook allow students to follow important scholars who provide links to new research, ideas, and sources. General search engines often miss such postings, but the following engines index social media:

     • [searches keywords on social media]

     • [enables searchers to look for individuals]

Identifying key scholars and following them is an important research task. Social media are vital tools, and students should use them.

How are the results of general searches organized?

The results of general search engines such as Bing, Google, and Yahoo appear on the page according to formulas (algorithms) that reflect your history of searches, your history of links followed, the number of links to a site, and many many many other factors. The results are not ranked in order of importance or quality. Be leery of using general search engines for academic purposes. Note that the results from specialized search engines are controlled by different factors (see below).

Are there specialized search engines?

Other search engines search specific types of publications. They give quick access to expert information. The following engines limit themselves to scholarly publications and are available without charge. Both engines listed below give links to related articles and to other works that cite an article.

For scholarly work in a variety of disciplines. Includes conference proceedings, books, and journals. Provides citation information. Helps focus searches, identity key journals, and key terms.

Presents links in an order influenced by the usual behavior of scholars: author, publisher, citation frequency, and language. Frequently cannot provide full texts of articles and other materials, but does make using the library databases easier.

For scientific information on both the surface web and much of the “invisible web” including pdfs, conferences papers, etc. It has indexed more than 400 million items.

Presents links according to their use of the search terms and according to the number of links that others make to the source.

By identifying scholarly sources, these sites become key tools in students’ research.

The Filter Bubble

Many search engines record what each user does online. Even if a searcher has not logged in, the search engine still knows many things that determine the results provided. This information includes the kind of computer system used, the browser, the location of the user, and probably 50-100 other facts. A searcher who signs in reveals much more. In either case, the result is both wonderful and dangerous. It is wonderful because the results are “personalized.” For example, if you are a poor student looking for a hotel in New York, it will not help you to be offered the rate for a room at The Plaza. Information about a Motel 8 in New Jersey might be much more useful. The search engine filters the results they send by keeping track of your other searches. Do you search inexpensive department stores for clothes, or do you search Bergdorf Goodman? Did you check prices on used cars or on Cadillacs? Your history tells about who you are and what you are likely to buy. This information makes you valuable because it lets the search engine make a promise to advertisers. It promises them that it will advertise their products to the most appropriate market. Personalization reveals something important: you are the product being sold. Your attention to web pages is valuable to advertisers, and the search engine is keeping track of what you are likely to want so that the ads can be posted in front of the people most likely to buy.  

Personalization also has risks. By keeping track of our beliefs, desires, and behavior, a search engine can become a sort of echo chamber where searchers only find material that agrees with what they already think. Liberals hear only liberal ideas; conservatives hear only conservative ideas. No one encounters ideas that challenge her or his beliefs. This is called “the filter bubble.” In the filter bubble, future information is based only on past choices. The future cannot escape the past. Many scholars worry the filter bubble will take away the benefits of surprise, contradiction, and argument. They fear it will harm both individuals and our culture’s need for genuine dialogue.

The Relevance Paradox

The filter bubble seems dangerous. However, critical readers understand that it is a mixed blessing. They recognize that writers have judged some evidence important. Those selections make some ideas more important than other ideas.

Good readers also try to recognize what has been ignored or not taken seriously. While good readers know that some things will be consciously “unknown” in the reading, they also know that there are “unknown unknowns” in any explanation.  Secretary of Defense Donald Rumsfeld said:

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.

Rumsfeld’s statement maps out the risks that face any critical reader. These highly skilled readers use their recognition of unknown unknowns to insist that even the best critical reading is unfinished.

What are library databases?                                                                                            

Library databases are collections of filtered information. They often contain information that is not available through search engines such as This information is organized to allow easy access to complete articles and other materials. Some organizations and institutions have built databases filled with carefully filtered information. For example, JSTOR contains all the articles from more than 1,000 journals. It also contains related materials. JSTOR relies on established experts to judge whether or not something should be included. Factors such as historical significance, citation patterns, subscribers, and recommendations from libraries and scholars determine if a journal or other source will be included. Most college libraries have at least several dozen databases, and most provide access to many more than that. Check your library’s web site to see the vast amount of material available to you.

Do some databases focus on specific disciplines?

Many databases cover several fields, but some are highly specific:

• Art: JSTOR, Project Muse, Art Full Text

• Business: Business Source Premier, Central Business File ASAP, Standard and Poor’s Net Advantage

• Computer Science: Computer Source, INSPEC

• Education: ERIC, Educator’s Reference Complete

• Engineering: Compendex, INSPEC

• History: Historical Abstracts, JSTOR

• Literature and Language: MLA International Bibliography, JSTOR, Voice of the Shuttle

• Mathematics: MathSciNet

• Medical Information: Pubmed

Are library databases the best sources of information?

Major databases act as specialist librarians who know what is recognized as most important about a subject. Not only do they filter out questionable content, but also they connect to the network of experts, institutions, publications, and ideas that define the field you are researching. Using library databases puts you inside the boundaries of a discourse community.

The information in library databases represents the careful judgment of experts. Such careful judgment takes time. Thus, there’s often a lag between research and publication. This lag reflects the nature of building knowledge. Nonetheless, some important issues erupt quickly and receive attention outside peer reviewed publications. Major scholars are likely to write about them in publications that are not part of library databases. These insights, discussions, and critiques appear in blogs, topical web sites, and “middle brow publications.” Middle brow publications present detailed information and background to audiences who are well-educated but not specialists. They do not use peer review, but the authors are highly qualified. Thus, you must recognize that good research reflects a wide range of information gathering. Good research uses Twitter, library databases, Facebook,, Bing, and other tools that keep you in touch with the dynamic nature of knowledge.

Other Databases and Online Resources                                                                 

Government databases cover many subjects. The sites end with .gov to signal the source of information and control. A few of the many valuable sites include the following:

• Bureau of Labor Statistics:

• Census Bureau:

• Centers for Disease Control and Prevention:


• Fedstats:

• National Archives:

• National Center for Education Statistics:

• National Institutes for Health: www.

• Many other government web sites provide valuable information.

Collections of complete books, articles, and other materials are available at the following sites:

• The Internet Archive:

• Online Books: online

• OpenLibrary:

• Project Gutenberg:

• Scholars’ Lab Digital Resources:

• ScribD:

• Google Books:

Advanced search techniques

No one lives long enough to read the millions of results that come from a poorly designed search. An effective search depends not only on using precise terminology, but also on knowing some simple rules for getting useful results. Three simple terms produce more useful results:

AND, OR, and NOT. Consider the following example: suppose you want to search for information about freshwater clams in Lake Erie:

Using quotation marks, AND, +, and blank spaces to narrow the result of searches

Entering freshwater clams produces about 1.6 million hits on The search finds websites with both words, but not necessarily in that order. The blank space between the words means “both words but not necessarily in that order or even next to each other.”

Enter ”freshwater clams” and the results drop to about 45,000 hits. The quotation marks narrow the search to a specific phrase. The search finds websites with only the phrase “freshwater clams” in exactly that order.

Entering ”freshwater clams” AND “Lake Erie” produces only 4,500 hits because the AND combines with two phrases in quotation marks. The search narrows to only sites with both phrases exactly as written in the search bar. Some search engines require a + sign or the word AND between words to narrow the results of a search. Google assumes that a blank space means “AND.”

Using OR to broaden the result of searches

When different terms refer to a similar concept (school, college, university), a good search may need to find sources for all three. The term OR enables a search engine to combine all three and provide more results. For example, a search to determine the average number of computers in all schools might need to use OR. Searching separately on Google gives the following results:

Entering school produces 700 million hits

Entering college produces 500 million hits

Entering university produces 400 million hits

The total hits for the individual searches is 1,600 million hits. These searches can be combined with “OR.”

A single search using school OR college OR university 1,000 million hits.

Note that the sum of the individual searches is larger than the single search using OR. “OR” works on two levels: first, it increases the number of hits by adding together the results for all the terms; second, it eliminates the duplicates of the separate searches.

Using NOT to narrow the result of searches

Some searches ask for information about a category, for example, fish. The search seeks information about the category but does NOT want information about trout. Using NOT can solve this problem:

Entering fish produces 656 million hits on

Entering trout produces 84 million hits on

Entering fish NOT trout produces 376 million hits on

Combining advanced search terms

AND, NOT, and OR can be combined in many ways. Experimenting with their combinations can produce highly specialized results. Information about such searches is available through a variety of sources including librarians, handbooks, and web sites.

Evaluating Information                                                                                                      

Teachers, professors, librarians, researchers, and scholars agree that specialized search engines, library databases, and specialized databases find expert information. Non-experts (students) can follow the advice of such people, but new questions emerge: “What is an expert?” What counts as expertise?” Such questions are symptoms of genuine critical thinking.

Most experts work in a network that includes specialized publications, other experts, professional organizations, and institutions such as universities. They have specific methods for studying their subjects. They generally agree that some discoveries are important to their field, and they have general ideas about what will emerge in the field. Recognize expert writing by doing the following:

Read the introduction, table of contents, works cited, and index. Ask the following:

• What questions guide the work?

• Do the works cited include scholarly journals and books?

• Is the source published by a university press, a professional society, or by a press that specializes in the topic?

• Is the source current? If not current, is it still important to the field?

• Use to find the most recent citation of the work.

Think about the author:

• Has the author published other material in the field?

• What is her/his education?

• With what institutions is she/he connected?

• If the author is anonymous, does the organization participate in the network of experts and institutions that form the discipline?

Clear evidence that connects to a claim. Ask the following:

• Does the source emphasize evidence or its claim?

• Are the methods for producing the evidence clearly stated?

• Are the claims precise?

• Is there enough evidence to support the claim?

Talk to your professor. Asking about the quality of a source shows that you are thinking like a professional. Professors are likely to offer advice, sources, key terms, important names, and other information that will direct and strengthen your work.

Keeping track of information

The TEQ Sheets enable you to build a useful history of your searches. Its three parts solve part of your note-taking. Save time by using its three parts:

Terms: listing the key terms used in the reading helps you understand its claims, and it gives you terms for future searches.

Anomalies: reading with the expectation that you will list what seems odd, interesting, and/or surprising makes you into a critical reader. You have to read actively.

Questions: reading with the expectation of asking questions about a source’s validity, reliability, and credibility helps you identify the “gaps in the map” that your paper will fill.

The TAQ sheet is both a reading tool and a writing tool. It creates an overview of the document and its potential value to your project. Help yourself by using it.

Annotated Bibliographies collect additional information about a reading by asking three basic questions:

• Summary: What is the most important information presented in the reading? This is a factual summary that does not include your responses to the information.

• Evaluation: What level of credibility does the source bring to discussions of the subject? Is it peer reviewed? What expertise can the author claim? Do the citations place the article on a map of clearly understood and accepted discoveries? For a detailed review of critical reading, see Chapter 2.

• Integration: How does the source sharpen your own map of the topic? Does it sharpen your idea by providing additional information? Does it sharpen your detailed additions to the map by challenging what you would like to say?

Paraphrase and Quotation

Good writers identify key sentences, facts, and phrases from important sources. Showing the source of such material strengthens a paper. Writers have two major tools for including the supporting network in a document:


Quotations use the exact phrasing of a source. The language is identical, and the material is placed inside quotation marks. The quotation’s source must be named.


Paraphrases put the materials of others in new words. The revised language keeps key concepts. Often, the phrasing integrates the source with the style and voice of the paper that uses it. Like quotation, paraphrase requires that the source be named.

Citing Sources                                                                                                                             

Readers expect to see the connections to the existing map of the topic. They expect that writers will identify those connections in the text of the paper. The references in the text of the paper will enable readers to find complete information about the sources in a list (the “works cited”) at the end of the paper. Writers show that their work connects to expert knowledge by citing sources. A writer who names these links in the document gains credibility. A writer can fail by not naming the network of experts who influence the document. Citations have other advantages:

they recognize the history of the topic

they reveal the various ideas and positions that frame a topic

they highlight the difference between the writer’s own insights and the insights to which it connects

Naming sources through careful citations is so important that expert writers use standardized formats that make it easy to retrace the sources for a work and understand how the new document adds to the understanding of a topic.

Good writers cite sources, and the citations use established formats to simplify the work of readers who want to read more about the topic. Cambridge University Press offers the most simple and clear guidance for formatting citations. It asks that they be “clear, logical, and consistent,” and then offer six pages of clear, logical, and consistent advice. Others find that the creation of consistency and completeness requires detailed systems. Most U. S. professors expect that you will follow one of two systems:

Modern Language Association (MLA): for the humanities

American Psychological Association (APA): for psychology, sociology, business, nursing, economics, and other field

These are the two dominant styles, but others exist. Use the format style required by the professor or employer.