You want to cite an article, a book, or a website in your thesis but are unsure if it’s a trustworthy source? After trying the quick CRAAP test, this page will provide more in-depth information to assist you in your assessment. We will show you how to analyze texts based on formal and content criteria using four questions: Who published a text and where it was published, how the text came into existence and was received by the scientific community, and what it is trying to convey.
Formal quality criteria
Who?
Knowing who published a text can reveal a lot about its quality. Behind every text is at least one person responsible for it: an author, an editor or (journal) editorial team supporting its publication in a collective work or journal. Websites are also published by individuals or institutions. Therefore, you should first ask what qualifies a person or institution to speak on a specific topic and take a closer look at the professional background of those involved in the publication.
- Check if the author has expertise in the relevant field, perhaps through an academic title or affiliation with a research institution or academic association such as a university.
- Examine the author’s other publications: Look for other works published in reputable academic journals or by reputable publishers. How often is the author cited in different works on the same or related topics? A high number of citations can indicate recognition and influence in the academic community.
- Pay attention to potential conflicts of interest: Is the author a member of a group or organization that may have financial or personal interests that could influence their independence and result in a narrow perspective on a topic? If so, you should look for information about the financial and organizational sponsorship of the institution, its goals, and its history. Potential conflicts of interest or a biased point of view should be openly disclosed to ensure the objectivity of a source.
In most cases, these questions can be answered with a simple internet search. Furthermore, many researchers today have an ORCID iD, which can help you identify them and their academic output relatively reliably. This data is particularly trustworthy when it comes from research institutions themselves (you can find more information about this at ORCID).
If, despite this, you still cannot determine whether a person has already conducted research on a specific topic, you can take a look at the WU Catalog or literature databases. Both allow you to search for the names of authors. In the WU Catalog, you can do this by clicking on Advanced Search, choosing Author from the drop-down menu, and entering the author’s name in the first search field.
Most literature databases also offer targeted author searches (the example on the right shows the search in the EconLit database).
Where?
The place where a text is published can also provide insights into its academic quality. This is especially true for publications in traditional media – books and journals – but less so for online sources.
Publications in books
Scientific books are usually published by publishing houses specializing in research literature. It’s not easy to answer which publishing houses are the highest regarded among researchers, as different fields have different conventions. The same applies to series of publications where – usually thematically related – books are continuously published by a publishing house. Such series are very significant in some disciplines, such as law. Precisely because there is no fixed quality criteria here, it’s crucial to examine the individuals involved in a given book.
Tip: Take a look at which publishing houses publish the books you work with during your studies and as part of which series they were published (if applicable).
Publications in journals
Similar to publishing houses, the reputation and quality of journals can vary. It’s best to ask your instructors or to find out yourself which journals are frequently cited in your study literature.
However, this method most likely won’t lead you to discover any new or lesser-known journals. Not being well-known does not necessarily imply poor quality. However, you should gather more information about journals you don’t know. You can do this through online research or by using the database Ulrichsweb, which is accessible online through the WU Library. This database provides basic information about journals, including whether articles undergo a peer review process and information about subject areas and publishers.
Recently, digital-only publishing of journals has become more common. However, in the past, it occasionally led to the appearance of new journals with insufficient quality control. Now there are directories that list publishers and journals that should be regarded with caution:
Additionally, bibliometric data can provide some insight into the reputation of a journal or author. Through this data, you can find out how often articles from a specific journal are cited in other publications.
What is bibliometrics?
Bibliometrics is with the quantitative analysis of scholarly publications. It uses statistical methods and metrics to assess and measure the circulation, influence, and structure of academic literature. Some of the most important bibliometric indicators include:
- Impact Factor: Indicates how often articles in a specific journal are cited on average. It is often used to evaluate the reputation and significance of scientific journals.
- h-Index (Hirsch Factor): Indicates the number of publications (h) by an author that have received at least h citations. The advantage of the h-index is its stability: compared to other metrics, individual highly-cited publications by an author cannot distort the result. You can find more information on how the h-index is calculated here.
- Altmetrics: An addition to traditional bibliometrics, measuring the impact of research output in social media, blogs, news websites, online discussion forums, and other online platforms.
These indicators can serve as a framework to assess the quality of scientific publications. However, they also have their limitations. For instance, the exact calculation of some citation numbers can sometimes lack transparency, and the general significance of citations can vary significantly in different disciplines. Furthermore, such methods of measurement often disadvantage young researchers.
Online publications
The threshold for publishing on the internet is still lower than in traditional media. However, even online, there are now media formats in almost every scientific discipline that are of equal quality to traditional ones. Examples of these include the Verfassungsblog in law (only available in German), Soziopolis in the social sciences (only available in German), and VoxEU in the field of economic policy.
Whether this is the case for a specific online medium can ultimately only be judged by the content it publishes. However, there are a few indicators that can provide you with initial orientation:
- Is there an author? Sometimes, it is not easy to determine who is responsible for the content on a website. Check who is behind the information presented there and what professional qualifications that person or organization has.
- What conclusions can you draw from the website’s address (URL)? In some cases, the domain can reveal the type of website it is: .com (commercial sites), .edu (educational institutions), .org (non-profit organizations), .gv.at, or .gov (government-run websites, e.g., authorities or ministries).
- Was the website established for commercial or informational purposes? Are there obvious sponsors? Keep in mind that such factors can influence the content of a website.
- Is there a date of the last update? How current are the links leading away from the website?
Of course, the quality of a text is not solely determined by who wrote it and where. It is most clearly visible in its content, that is, in how it was produced and how it engages with its subject matter.
Content quality criteria
How?
Quality control before publication
In academia, it is customary for texts to undergo quality control before their publication. This process is carried out by other researchers in the respective discipline to ensure the reliability and soundness of the content. Such quality control measures can vary depending on the medium and specific scientific discipline.
Books are often published in series where editors make decisions about publication. In other cases, the decision regarding the content of a book is made by knowledgeable publishing professionals. If a book is based on a scholarly qualification work (e.g., a dissertation or habilitation), the academic assessment already happens at the university.
In comparison to books, scholarly journals typically have more standardized quality control processes. Authors submit their texts to an editorial board, which then forwards them to peer reviewers for expert evaluation. The submitted work is reviewed by them for its originality, methodology, results, and conclusions. The reviewers assess the strengths and weaknesses of the paper, provide feedback, and may offer comments or suggestions for improvement. Only after receiving a favorable quality assessment is the text published.
Depending on the academic discipline, there may be differences in this process. These differences mainly concern the selection of reviewers: Are they the editors or members of the editorial board? Are they external researchers? Do the parties involved know each other’s identities? Models in which reviewers are not permanent staff of the journal but rather scholars working in the same field as the author of the article to be reviewed are known as peer review.
The WU CatalogPLUS and most databases can assist you in determining whether a text has undergone such a peer reviewing process. You can filter search results to display only relevant hits, and the CatalogPLUS also highlights peer reviewed articles graphically.
Quality control after publication
Within academia, there are mechanisms that continue to enforce scholarly standards even after publication. One such means of post-publication quality control for books is reviews. These are scholarly texts in which researchers engage with the works of their academic peers. Through reviews, you gain insight into how the academic community thinks of a particular book.
The WU Catalog can also assist you in your search here. In many cases, it indicates whether reviews for a book are available:
Occasionally, an article may need to be corrected or even retracted due to scientific misconduct. The Retraction Database documents such events. Therefore, you should always search this database for articles you intend to use in your work. Reference management programs can assist you with this (and many other tasks): Zotero and Juris-M, and EndNote automatically flag texts if they have been retracted according to the Retraction Database.
What?
Even after examining all these criteria, you won’t have absolute certainty about the quality of a text. Ultimately, it is crucial to engage with the content more closely. This can be demanding and time-consuming, but, in the end, it’s what academic work is all about. Use the following questions to guide you:
- Is the text clearly structured? Does one thought logically lead to the next, or does it jump between ideas and repeat itself?
- Does the text disclose its research question and the methods used?
- What is the intended audience? Does the text have a specific intention, and if so, what is it? Does it present different viewpoints and treat them with respect? Academic texts usually also engage with opposing results and avoid bias and prejudice; where they might exist, they disclose them.
- How does the author support the proposed theses? With emotions and anecdotes, or with relevant and well-founded arguments? Does the author primarily cite themselves or individuals from their own academic circle?
- Does the text provide proper citations and references for quotes, tables, and figures?
- Is the text current or at least still relevant for your work?
Do you still feel uncertain at this point? Don’t hesitate to discuss a text with your fellow students or talk to your instructors. This is also part of academic work and can provide you with new perspectives you might not have considered before.
Quiz
Got it? Test your knowledge with this quiz on evaluating academic sources!
Literature and further reading
Berkeley Library: Evaluating Resources
Concordia University Library: How to evaluate research materials and resources (gegliedert nach Materialart)
Hacker, D. & Sommers, N. (2015). A pocket style manual (Seventh edition). Boston; New York: Bedford/St. Martin’s.
North Carolina State University Library: Evaluating Sources for Credibility (CC BY-NC-SA; with transcript)
UB Münster: Suchergebnisse prüfen & bewerten (German only)
UB Münster: Fake News & Desinformation (German only)
Wikisource: Biographische Recherche (German only)
Yale Library: Evaluating Sources
Yale Library: Choosing a Journal for Publication of an Article