A smart and effective method for undertaking literature reviews particularly for research students and others new to a discipline.
Narrative methods that are commonly used in many research theses, rely on the expertise and experience of the author, making them challenging for novices. In contrast, the method we use and recommend involves systematically searching the literature using online database and other sources to find all relevant papers that fit specific criteria (systematically identifying the literature), entering information about each study into a personal database, then compiling tables that summarise the current status of the literature (quantifying the literature). The results are reliable, quantifiable and reproducible.
Using this method, it’s also possible to determine if there are suitable datasets for meta-analysis. By mapping the literature we can not only identify what is known, but also, but where there are gaps: a critical issue in advancing research and designing PhD research programs.
Reliable, quantifiable and reproducible
The method works well for specific topics, but also for summarising diverse inter-disciplinary research. Using this method many of our students and others have gone on to publish their reviews. Importantly for PhD students, the database can be updated during the PhD thesis allowing them to easily identify relevant papers and produce their final thesis without having to re-read all the literature.
Resources
- Slides from workshop on the method (PDF 4MB)
- Slides from the advanced workshop on the method (PDF 5MB)
- More resources on doing your PhD as a series of papers
The method and its benefits
- Pickering, C.M. and Byrne, J. (2014). The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early career researchers. Higher Education Research and Development. 33: 534-548
- Pickering, C., Grignon, J., Steven, R., Guitart, D. and Byrne. J. (2015). Publishing not perishing: How research students transition from novice to knowledgeable using systematic quantitative literature reviews. Studies in Higher Education. 40:10, 1756-1769, DOI: 10.1080/03075079.2014.914907. A pre-print copy of the paper is available here, or the final published version from the publisher's website.
Videos about the method
There are now hundreds of papers published using this method. A full list of them is available from google scholar.
Some select examples showing how they have been done, including searching strategies, ways to analysis the data and address some concerns regarding use/non-use of grey literature, factors affecting demand for, and supply of research by country etc, addressed in the advanced videos include:
- Guitart, D., Pickering, C.M. and Byrne, J. (2012). Past results and future directions in urban community gardens research. Urban Forestry and Urban Greening. 11: 364-373 — This was one of the original reviews using our methods. It highlights the importance of gap analysis and dealing with reviewing a very diverse literature including disciplines and methods used, and the capacity to review both quantitative and qualitative literature.
- Steven, R. Pickering, C.M. and Castley, G. (2011). A review of the impacts of nature based recreation on birds. Journal of Environmental Management. 92: 2287-2294 — This early SQLR provides a detailed example of why gap analysis is important and ways of quantifying differences between the supply of literature and need for research by calculating bird diversity vs number of studies per region.
- Pickering, C.M., Rossi, S.D., Hernando, A. and Barros, A. (2018). Current knowledge and future research directions for monitoring and management of visitors in recreational and protected areas. Journal of Outdoor Recreation and Tourism. 21: 10-18 — This SQLR of abstracts from a conference, includes a detailed examination of factors affecting the supply of research including why research is often dominated by literature from North America and Europe, and why that can be an issue. It also includes more sophisticated ways to access the results by using multi-dimensional analysis.
- Verrall, B. and Pickering, C.M. (2020). Alpine vegetation in the context of climate change: A global review of past research and future directions. Science of the Total Environment. 748:141344. This is an example of a bibliometric/scientometric literature review – an alternative method using vosViewer to visuals patterns in keywords over time in the literature and changes in who is publishing on a topic and where.
- Thomas, S. (2014). Blue carbon: Knowledge gaps, critical issues and novel approaches — This review also uses Leximancer analysis (text analysis) of themes to map concepts in the literature based on keywords and phrases in the papers.
- Riebe, L., Girardi, A. and Whitsed, C. (2016). A systematic literature review of teamwork pedagogy in higher education. Small Group Research. 47: 619-664 — This paper from the education area includes an excellent example of how to use keywords and search terms in a stepped down way to identify a complex literature.
- Liao, Y., Deschamps, F., de Freitas Rocha Loures, E. and Ramos, L.F.P. (2017). Past, present and future of Industry 4.0- a systematic literature review and research agenda proposal. International Journal of Production Research. 55: 3609-3629 — This paper includes a good example about how to set out the justification for which papers to include and exclude including less clear-cut topics. It also includes an excellent way to quantify the disciplines represented by papers using well recognised (SCImago) data for each journal.
- Turner, J.A., Babcock, R.C., Hovey, R., and Kendrick, G.A. (2017). Deep thinking: a systematic review of mesophotic coral ecosystems. ICES Journal of Marine Science — This review uses column diagrams to clearly illustrate how over time the methods used in the discipline and the types of analysis conducted become more complex as the literature develops.
- Pahlevan-Sharif, S. Mura, P., and Wijesinghe, S.N.R. (2019). A systematic review of systematic reviews in tourism. Journal of Hospitality and Tourism Management. 39: 158.165 — This recent paper provides an overview review of other literature reviews. It highlights the types of methods used in reviews in tourism including recommendations for future reviews.
- Bezerra, M.F., Lacerda, L.D. and Lai, C-T. (2019). Trace metals and persistent organic pollutants contamination in batoids (Chondrichthyes: Batoidea): A systematic review. Environmental Pollution. 248: 684-695 — This recent review includes a broad SQLR and meta-analysis of a science/quantitative area.
- Nikulina, V., Simon, D., Ny, H. and Baumann, H. (2019). Context-adapted urban planning for rapid transitioning of personal mobility towards sustainability: A systematic literature review. Sustainability: 11: — This paper combines a SQLR, with narrative commentary on themes, and a bibliometric analysis.
Further resources and contacts
Article in The Conversation:
Important reference for how to report systematic literature reviews required by many journals:
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71 - with more details available here.
- Boote, B.N. and Beile P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher. 34: 3-15.
- Crisp, B.R. (2015) Systematic reviews: A social work perspective. Australian Social Work, 68:3, 284-295.
- Murray, R. (2011). How to Write a Thesis. McGraw Hill Open University Press. Maidenhead, England (Chapter on writing a literature reviews).
- Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing, Oxford, England.
- Randolph J.J. (2009). A guide to writing the dissertation literature review. Practical Assessment, Research and Evaluation. 14: 1-13.
Here are examples of the types of excel databases used in some Systematic Quantitative Literature Reviews:
Some of the journals publishing SQLR include:
- Ambio
- Analyse und Kritik
- Anatolia
- ASEE Annual Conference and Exposition, Conference Proceedings
- Asia and the Pacific Policy Studies
- Asian Journal of Criminology
- Austral Entomology
- Australian Social Work
- Behaviour Change
- Biological Conservation
- BMC Health Services Research
- CIRP Journal of Manufacturing Science and Technology
- Climatic Change
- Conservation Biology
- Corruption in Sport: Causes, Consequences, and Reform
- Crop Protection
- Crystal Research and Technology
- Cuadernos de Desarrollo Rural
- Cuadernos de Turismo
- Current Issues in Tourism
- Discourse
- Documents d'Analisi Geografica
- Ecologia Austral
- Ecological Economics
- Ecology and Society
- Education Sciences
- Educational Media International
- Energy and Buildings
- Environment International
- Environment Systems and Decisions
- Environmental Modelling and Software
- Environmental Pollution
- Environmental Science and Policy
- European Journal of Higher Education
- European Journal of Information Systems
- Evaluation and Program Planning
- Event Management
- Fish and Fisheries
- Frontiers in Ecology and the Environment
- Global Environmental Change
- Habitat International
- Higher Education Research and Development
- ICES Journal of Marine Science
- IEEE Internet of Things Journal
- IFAC-PapersOnLine
- Innovar
- International Journal of Disaster Risk Reduction
- International Journal of Language and Communication Disorders
- International Journal of Managing Projects in Business
- International Journal of Mental Health Nursing
- International Journal of Production Research
- International Journal of the Commons
- International Journal of Tourism Research
- International Social Work
- Issues in Educational Research
- Journal of Business Research
- Journal of Cleaner Production
- Journal of Environmental Management
- Journal of Healthcare Leadership
- Journal of Hospitality and Tourism Research
- Journal of Hospitality Marketing and Management
- Journal of Information Technology in Construction
- Journal of Outdoor Recreation and Tourism
- Journal of Place Management and Development
- Journal of Reproductive and Infant Psychology
- Journal of Sustainable Tourism
- Journal of Technical Education and Training
- Journal of the Medical Library Association
- Journal of Travel and Tourism Marketing
- Journal of Urbanism
- Journal of Vocational Rehabilitation
- Landscape and Urban Planning
- Local Environment
- Managing Sport and Leisure
- Natural Hazards and Earth System Sciences
- Nurse Education in Practice
- Ocean and Coastal Management
- Procedia CIRP
- Procedia Engineering
- Procedia Manufacturing
- Proceedings of AISB Annual Convention 2018
- Proceedings of the International Conference on e-Learning, ICEL
- Producao
- Progress in Transplantation
- Reliability Engineering and System Safety
- Renewable and Sustainable Energy Reviews
- Restoration Ecology
- Scientometrics
- Sex Education
- Small Group Research
- Solar Energy
- South African Computer Journal
- Speech, Language and Hearing
- Studies in Higher Education
- Sustainability (Switzerland)
- Teaching and Learning in Medicine
- Teaching and Teacher Education
- The Routledge Companion to Management Information Systems
- Tourism
- Tourism Management
- Tourism Management Perspectives
- Training and Education in Professional Psychology
- Transportation Research Record
- Urban Ecosystems
- Urban Forestry and Urban Greening
- Water Research
- Wildlife Research
- Wiley Interdisciplinary Reviews: Climate Change
- World Leisure Journal