MIXED METHODS APPRAISAL TOOL (MMAT)VERSION 2018User guidePrepared byQuan Nha HONGa, Pierre PLUYEa,, Sergi FÀBREGUESb, Gillian BARTLETTa, Felicity BOARDMANc,Margaret CARGOd, Pierre DAGENAISe, Marie‐Pierre GAGNONf, Frances GRIFFITHSc, Belinda NICOLAUa,Alicia O’CATHAINg, Marie‐Claude ROUSSEAUh, & Isabelle VEDELaaMcGill University, Montréal, Canada; bUniversitat Oberta de Catalunya, Barcelona, Spain; cUniversity of Warwick, Coventry, England;dUniversity of Canberra, Canberra, Australia; eUniversité de Sherbrooke, Sherbrooke, Canada; fUniversité Laval, Québec, Canada;gUniversity of Sheffield, Sheffield, England; hInstitut Armand‐Frappier Research Centre, Laval, CanadaLast update: August 1st, 2018

What is the MMAT?The MMAT is a critical appraisal tool that is designed for the appraisal stage ofsystematic mixed studies reviews, i.e., reviews that include qualitative, quantitative andmixed methods studies. It permits to appraise the methodological quality of fivecategories to studies: qualitative research, randomized controlled trials, non-randomizedstudies, quantitative descriptive studies, and mixed methods studies.How was the MMAT developed?The MMAT was developed in 2006 (Pluye et al., 2009a) and was revised in 2011 (Paceet al., 2012). The present version 2018 was developed on the basis of findings from aliterature review of critical appraisal tools, interviews with MMAT users, and an eDelphi study with international experts (Hong, 2018). The MMAT developers arecontinuously seeking for improvement and testing of this tool. Users’ feedback is alwaysappreciated.What the MMAT can be used for?The MMAT can be used to appraise the quality of empirical studies, i.e., primaryresearch based on experiment, observation or simulation (Abbott, 1998; Porta et al.,2014). It cannot be used for non-empirical papers such as review and theoretical papers.Also, the MMAT allows the appraisal of most common types of study methodologiesand designs. However, some specific designs such as economic and diagnostic accuracystudies cannot be assessed with the MMAT. Other critical appraisal tools might berelevant for these designs.What are the requirements?Because critical appraisal is about judgment making, it is advised to have at least tworeviewers independently involved in the appraisal process. Also, using the MMATrequires experience or training in these domains. For instance, MMAT users may behelped by a colleague with specific expertise when needed.How to use the MMAT?This document comprises two parts: checklist (Part I) and explanation of the criteria(Part II).1. Respond to the two screening questions. Responding ‘No’ or ‘Can’t tell’ to one orboth questions might indicate that the paper is not an empirical study, and thuscannot be appraised using the MMAT. MMAT users might decide not to use thesequestions, especially if the selection criteria of their review are limited to empiricalstudies.2. For each included study, choose the appropriate category of studies to appraise. Lookat the description of the methods used in the included studies. If needed, use thealgorithm at the end of this document.3. Rate the criteria of the chosen category. For example, if the paper is a qualitativestudy, only rate the five criteria in the qualitative category. The ‘Can’t tell’ responsecategory means that the paper do not report appropriate information to answer ‘Yes’or ‘No’, or that report unclear information related to the criterion. Rating ‘Can’t tell’could lead to look for companion papers, or contact authors to ask more informationor clarification when needed. In Part II of this document, indicators are added forsome criteria. The list is not exhaustive and not all indicators are necessary. Youshould agree among your team which ones are important to consider for your fieldand apply them uniformly across all included studies from the same category.How to score?It is discouraged to calculate an overall score from the ratings of each criterion. Instead,it is advised to provide a more detailed presentation of the ratings of each criterion tobetter inform the quality of the included studies. This may lead to perform a sensitivityanalysis (i.e., to consider the quality of studies by contrasting their results). Excludingstudies with low methodological quality is usually discouraged.How to cite this document?Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, GagnonM-P, Griffiths F, Nicolau B, O’Cathain A, Rousseau M-C, Vedel I. Mixed MethodsAppraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), CanadianIntellectual Property Office, Industry Canada.For dissemination, application, and feedback: Please contact [email protected] more information: /1

Part I: Mixed Methods Appraisal Tool (MMAT), version 2018Category of studydesignsScreening questions(for all types)1. Qualitative2. Quantitativerandomized controlledtrials3. Quantitative nonrandomized4. Quantitativedescriptive5. Mixed methodsMethodological quality criteriaYesNoResponsesCan’t tellCommentsS1. Are there clear research questions?S2. Do the collected data allow to address the research questions?Further appraisal may not be feasible or appropriate when the answer is ‘No’ or ‘Can’t tell’ to one or both screening questions.1.1. Is the qualitative approach appropriate to answer the research question?1.2. Are the qualitative data collection methods adequate to address the research question?1.3. Are the findings adequately derived from the data?1.4. Is the interpretation of results sufficiently substantiated by data?1.5. Is there coherence between qualitative data sources, collection, analysis and interpretation?2.1. Is randomization appropriately performed?2.2. Are the groups comparable at baseline?2.3. Are there complete outcome data?2.4. Are outcome assessors blinded to the intervention provided?2.5 Did the participants adhere to the assigned intervention?3.1. Are the participants representative of the target population?3.2. Are measurements appropriate regarding both the outcome and intervention (or exposure)?3.3. Are there complete outcome data?3.4. Are the confounders accounted for in the design and analysis?3.5. During the study period, is the intervention administered (or exposure occurred) as intended?4.1. Is the sampling strategy relevant to address the research question?4.2. Is the sample representative of the target population?4.3. Are the measurements appropriate?4.4. Is the risk of nonresponse bias low?4.5. Is the statistical analysis appropriate to answer the research question?5.1. Is there an adequate rationale for using a mixed methods design to address the research question?5.2. Are the different components of the study effectively integrated to answer the research question?5.3. Are the outputs of the integration of qualitative and quantitative components adequately interpreted?5.4. Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?5.5. Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?2

Part II: Explanations1. Qualitative studiesMethodological quality criteria“Qualitative research is an approach for exploring and understanding themeaning individuals or groups ascribe to a social or human problem”(Creswell, 2013b, p. 3).1.1. Is the qualitative approach appropriate to answer the research question?Common qualitative research approaches include (this list if notexhaustive):ExplanationsThe qualitative approach used in a study (see non-exhaustive list on the left side of this table) should be appropriate for theresearch question and problem. For example, the use of a grounded theory approach should address the development of atheory and ethnography should study human cultures and societies.EthnographyThe aim of the study is to describe and interpret the shared culturalbehaviour of a group of individuals.This criterion was considered important to add in the MMAT since there is only one category of criteria for qualitative studies(compared to three for quantitative studies).1.2. Are the qualitative data collection methods adequate to address the research question?PhenomenologyThe study focuses on the subjective experiences and interpretations of aphenomenon encountered by individuals.ExplanationsThis criterion is related to data collection method, including data sources (e.g., archives, documents), used to address theresearch question. To judge this criterion, consider whether the method of data collection (e.g., in depth interviews and/orgroup interviews, and/or observations) and the form of the data (e.g., tape recording, video material, diary, photo, and/or fieldnotes) are adequate. Also, clear justifications are needed when data collection methods are modified during the study.1.3. Are the findings adequately derived from the data?Narrative researchThe study analyzes life experiences of an individual or a group.Grounded theoryGeneration of theory from data in the process of conducting research (datacollection occurs first).Case studyIn-depth exploration and/or explanation of issues intrinsic to a particularcase. A case can be anything from a decision-making process, to a person,an organization, or a country.Qualitative descriptionThere is no specific methodology, but a qualitative data collection andanalysis, e.g., in-depth interviews or focus groups, and hybrid thematicanalysis (inductive and deductive).ExplanationsThis criterion is related to the data analysis used. Several data analysis methods have been developed and their use depends onthe research question and qualitative approach. For example, open, axial and selective coding is often associated with groundedtheory, and within- and cross-case analysis is often seen in case study.1.4. Is the interpretation of results sufficiently substantiated by data?ExplanationsThe interpretation of results should be supported by the data collected. For example, the quotes provided to justify the themesshould be adequate.1.5. Is there coherence between qualitative data sources, collection, analysis and interpretation?ExplanationsThere should be clear links between data sources, collection, analysis and interpretation.Key references: Creswell (2013a); Sandelowski (2010); Schwandt (2015)3

2. Quantitativerandomizedcontrolled trialsMethodological quality criteriaRandomized controlledclinical trial: A clinicalstudy in which individualparticipants are allocatedto intervention or controlgroups by randomization(intervention assigned byresearchers).2.1. Is randomization appropriately performed?Key references: Higginsand Green (2008);Higgins et al. (2016);Oxford Centre forEvidence-basedMedicine (2016); Portaet al. (2014)ExplanationsIn a randomized controlled trial, the allocation of a participant (or a data collection unit, e.g., a school) into the intervention or control group is based solely on chance.Researchers should describe how the randomization schedule was generated. A simple statement such as ‘we randomly allocated’ or ‘using a randomized design’ is insufficientto judge if randomization was appropriately performed. Also, assignment that is predictable such as using odd and even record numbers or dates is not appropriate. At minimum,a simple allocation (or unrestricted allocation) should be performed by following a predetermined plan/sequence. It is usually achieved by referring to a published list of randomnumbers, or to a list of random assignments generated by a computer. Also, restricted allocation can be performed such as blocked randomization (to ensure particular allocationratios to the intervention groups), stratified randomization (randomization performed separately within strata), or minimization (to make small groups closely similar withrespect to several characteristics). Another important characteristic to judge if randomization was appropriately performed is allocation concealment that protects assignmentsequence until allocation. Researchers and participants should be unaware of the assignment sequence up to the point of allocation. Several strategies can be used to ensureallocation concealment such relying on a central randomization by a third party, or the use of sequentially numbered, opaque, sealed envelopes (Higgins et al., 2016).2.2. Are the groups comparable at baseline?ExplanationsBaseline imbalance between groups suggests that there are problems with the randomization. Indicators from baseline imbalance include: “(1) unusually large differencesbetween intervention group sizes; (2) a substantial excess in statistically significant differences in baseline characteristics than would be expected by chance alone; (3) imbalancein key prognostic factors (or baseline measures of outcome variables) that are unlikely to be due to chance; (4) excessive similarity in baseline characteristics that is notcompatible with chance; (5) surprising absence of one or more key characteristics that would be expected to be reported” (Higgins et al., 2016, p. 10).2.3. Are there complete outcome data?ExplanationsAlmost all the participants contributed to almost all measures. There is no absolute and standard cut-off value for acceptable complete outcome data. Agree among your teamwhat is considered complete outcome data in your field and apply this uniformly across all the included studies. For instance, in the literature, acceptable complete data valueranged from 80% (Thomas et al., 2004; Zaza et al., 2000) to 95% (Higgins et al., 2016). Similarly, different acceptable withdrawal/dropouts rates have been suggested: 5% (deVet et al., 1997; MacLehose et al., 2000), 20% (Sindhu et al., 1997; Van Tulder et al., 2003) and 30% for a follow-up of more than one year (Viswanathan and Berkman, 2012).2.4. Are outcome assessors blinded to the intervention provided?ExplanationsOutcome assessors should be unaware of who is receiving which int