2.2 How to evaluate QRM?

Author(s): 
Laurence.Kohn
Author(s): 
Wendy.Christiaens

In this section we want to address quality criteria for the use and evaluation of qualitative research. At the one hand it should guide those who want to apply QRM in their research project(s), at the other hand KCE researchers asked for criteria that allow them to evaluate existing qualitative studies or publications resulting from qualitative studies, for example in function of a systematic review.

2.2.1       Usefulness of quality criteria to evaluate qualitative research

Whatever the method, it needs to be well-defined, well-argued, and well-executed” (Snijders, 2007)

The increasing demand for qualitative research within health and health services research has emerged alongside an increasing demand for the demonstration of methodological rigor and justification of research findings (Reynolds, 2011) . Not only is qualitative research challenged by the current evidence-based practice (EPB) movement in healthcare, also the emergence of meta-analyses (e.g. meta-synthesis) of qualitative research findings urges for quality criteria. Although in quantitative health sciences research, there exist widely-recognized guidelines, no comparable standardized guidelines exist for qualitative research. This can be explained by a lack of consensus related to how to best evaluate “rigor” in qualitative research (Nelson, 2008). Every qualitative paradigm has its own implications regarding the definition of good quality research. First, we  introduce the reader briefly in the debate about quality criteria, second, we present the framework of Walsh and Downe (Walsh, 2006) as the most complete and comprehensible list of quality criteria to appraise qualitative research studies, and the framework of Côté and Turgeon as a shorter and practical alternative. For other checklists we refer to Appendix 1.

Among qualitative researchers there is a debate going on between those demanding for explicit criteria, for example in order to serve systematic reviewing and evidence-based practice, and those who argue that such criteria are neither necessary nor desirable (Hammersley, 2007). The quest for quality criteria assumes that qualitative research is a unified field, but this image does not fit reality. In fact, apart from a variety of other positions (e.g. symbolic interactionism, hermeneutics, phenomenology, ethnography) three main paradigms can be discerned in relation to this discussion:

  • The interpretativist paradigm assumes that social realities are multiple, fluid and constructed. This framework values research that illuminates subjective meanings and multiple ways of seeing a phenomenon. These researchers question the need for and the utility of quality criteria for qualitative research or apply specific criteria for qualitative research, such as clear delineation of the research process, evidence of immersion and self-reflection, demonstration of the researcher’s way of knowing (e.g. tacit knowledge) (Cohen, 2008).
  • The positivist approach stands at the other end of the continuum and assumes that there is a single objective reality that is knowable. Positivists apply traditional quantitative criteria, such as validity and reliability to qualitative work.
  • The realist perspective is positioned in between. It maintains a belief in an objective reality, but knowledge of reality is always imperfect (Cohen, 2008). Realists use techniques such as triangulation, member validation of findings, peer review of findings, deviant or negative case analysis and multiple coders of data, to promote to verify findings. The realist perspective adopts a philosophy of science that is in line with positivism, but at the same time embracing the complexity of social life and recognizing the importance of social meanings. “By maintaining a belief in an objective reality and positing truth as an ideal qualitative researchers should strive for, realists have succeeded at positioning the qualitative research enterprise as one that can produce research which is valid, reliable, and generalizable, and therefore, of value and import equal to quantitative biomedical research” (Cohen, 2008, p. 336).

The position one takes in the debate about quality criteria is heavily influenced by the paradigm one feels most attracted to, or identifies with.

2.2.2       General quality criteria

Most of the quality criteria are applicable to all research, both quantitative and qualitative. For example in 2008, Cohen and Crabtree (Cohen, 2008) reviewed and synthesized published criteria for good qualitative research. They identified the following general evaluative criteria: 1) ethical research, 2) importance of the research, 3) clarity and coherence of the research report, 4) use of appropriate and rigorous methods, 5) importance of reflexivity or attending to researcher bias, 6) importance of establishing validity or credibility, 7) Importance of verification or reliability. Researcher bias, validity, and reliability are most heavily influenced by quantitative approaches. Table 6 bridges quantitative and qualitative research by illustrating the parallels between criteria for conventional quantitative inquiries and qualitative research.

Table 6 – Lincoln and Guba’s translation of terms

Quantitative research

Qualitative research

Methods to ensure quality

Internal validity

Credibility:

Are the findings credible?

Member checks[a]; prolonged engagement in the field; data triangulation

External validity

Transferability:

Are the findings applicable in other contexts?

Thick description[b] of setting and/or participants

Reliability

Dependability:

Are the findings consistent and could they be repeated?

Audit – researcher’s documentation of data, methods and decisions; researcher triangulation

Objectivity

Confirmability:

To which extend are the findings shaped by the respondents and not researcher bias, motivation or interests?

Audit and reflexivity – e.g. awareness of position as a researcher and its influence on the data and findings

Source: Adapted from Finley,2006

In what follows we pay attention to some keywords appearing in Table 6.

Reflexivity

“Reflexivity is an awareness of the self in the situation of action and of the role of the self in constructing that situation.” (Bloor and Wood, 2006, p. 145)

Because in qualitative research, the researcher could not be ‘blinded’, he/she has to take into account subjectivity in an explicit way. To demonstrate this reflexive awareness during the research process, the following ‘good practices’ can be used (Green, 2009, p. 195):

  • Methodological openness: report steps taken in data production and analysis, the decisions made, and the alternatives not pursued.
  • Theoretical openess: theoretical starting points and assumptions should be adressed.
  • Awareness of the social setting of the research itself: be aware of the interactivity between the researcher and the researched.
  • Awareness of the wider social context, including historical and policy contexts and social values.

Triangulation

Qualitative research is inherently multimethod in focus (Flick, 2002, p.226-227). However, the use of multiple methods, or triangulation, reflects an attempt to secure an in-depth understanding of the phenomenon in question. Objective reality can never be captured. We know a thing only through its representations. Triangulation is not a tool or a strategy of validation, but an alternative to validation (Flick, 2002, p. 227). The combination of multiple methodological practices, empirical materials, perspectives, and observers in a single study is best understood, then, as a strategy that adds rigor, breadth, complexity, richness, and depth to any inquiry (See Flick, 2002, p. 229)” (Denzin and Lincoln, 2008, p. 7).

Triangulation is the use of several scientific methods, both qualitative and quantitative, to answer the same research question (Bloor, 2006. Often triangulation is understood as producing the same results by means of several methods, sources or analysts. However, different methods or types of inquiry are sensitive to different nuances, so that they may lead to somewhat different results. In fact, triangulation is more about finding inconsistencies to gain deeper insight into the relationship between the inquiry approach and the subject under study. Thus, finding inconsistencies do not weaken the credibility of the results, but rather strengthen it (Patton, 1999).

Five kinds of triangulation can contribute to the quality and consistency of qualitative data analysis:

  1. Methods triangulation: Information obtained through several methods is compared. These methods can be qualitative, or quantitative or both. Often qualitative and quantitative data can be fruitfully combined as they mostly elucidate complementary aspects of the same phenomenon (Patton, 1999) .
  2. Triangulation of sources: Information derived at different times and by different means is compared, e.g. comparing observational data with interview data, but also comparing what people say in public with what they say in private (Patton, 1999) .
  3. Analyst triangulation: Several observers, interviewers, researchers or analysts are used. By this way the potential bias that comes from a single person doing all the data collection and/or data analysis is reduced. In addition to several researchers or data analysts, analytical triangulation may also be to have those who were studied review the findings (Patton, 1999) .
  4. Theory/perspective triangulation:  It involves the use of different theoretical perspectives to look at the same data. Also, for example, data can be examined from the perspective of various stakeholder positions (Patton, 1999) .
  5. Member validation: It is a popular kind of triangulation that consists of “checking the accuracy of early findings with research respondents” (Bloor and Wood, 2006, p. 170).

These kinds of triangulation protect the researcher against the accusation that findings are an artifact of a single method, or source or investigator’s biases (Patton, 1999).

Transferability

Earlier in this report we argued that qualitative research is context sensitive and it is not aimed at making generalizations to the wider population. This may appear to contradict with the notion of transferability which is just about the extent to which findings of one study can be applied to other situations (external validity) (Merriam, 1998).

Transferability refers to the responsibility of the researcher to provide sufficient contextual information about the fieldwork to enable the reader to determine how far he can be confident in transferring the findings to other situations (Firestone, 1993). However, the situation might be complicated by the possibility that factors considered by the researcher to be unimportant, and consequently unaddressed in the research report, may be critical in the eyes of a reader(Firestone, 1993) .


2.2.3       Checklists

We have found four papers (Reynolds, 2011; Walsh, 2006; Cohen, 2008; Côté and Turgeon, 2005) reviewing the literature on quality criteria or guidelines for qualitative research. One of them (Walsh, 2006) provides us with a synthesis of eight existing checklists and summary frameworks (see Table 7). This checklist is quite detailed and is designed in function of meta-synthesis, which is a kind of systematic review of qualitative research papers.

The list of criteria was built in order to rigorously appraise studies first before submitting them to the meta-synthesis technique. Agreement on criteria to judge rigor was necessary in order to decide which studies to include in the meta-synthesis. Walsh and Downe (Walsh, 2006) tabulated the characteristics mentioned in each of the papers in their review. Then they mapped together the characteristics given in all the included papers, sorting them by the number of checklists in which they appeared. In the next step both authors independently attempted a synthesis before coming together to discuss. Redundant criteria were excluded if both authors agreed that the exclusion would not change the final judgment on the meaningfulness and applicability of a piece of qualitative research. Finally the table below was constructed, structured into three columns, namely stages, essential criteria and specific prompts. Although some criteria may seem self-evident, others are less obviously fundamental (Walsh, 2006). This list of criteria is very detailed. In some studies, especially those with short time frame, a shorter and more pragmatic hands-on list could be practical. Therefore we also added the grid of  Côté and Turgeon [c] (Table 8) which is shorter, adapted to the specific context of heath care and easier to use for researchers who are less familiar with qualitative research. Other checklists are described in Appendix 1.

The use of a checklist may improve qualitative research, however they should be used critically: not every criterion is appropriate to every research context (Barbour, 2001). For example the list of Coté and Turgeon mentions interpretation of results in an innovative way as a quality criterion (point 10, Table 8), while this is not necessarily the case. Most important is a systematic approach during research process. For example the credibility of data analysis could encompass the use of software (Table 7), triangulation and/or member checking (point 7, Table 8), whereas a systematic approach with a detailed description of each step in the research process could have been sufficient.

 

Table 7 – Summary criteria for appraising qualitative research studies

Stages

Essential criteria

Specific prompts

Scope and purpose

Clear statement of, and rationale for, research question / aims / purposes

  • Clarity of focus demonstrated
  • Explicit purpose given, such as descriptive/explanatory intent, theory building, hypothesis testing
  • Link between research and existing knowledge demonstrated

 

Study thoroughly contextualized by existing literature

  • Evidence of systematic approach to literature review, location of literature to contextualise the findings, or both

Design

Method/design apparent, and consistent with research intent

  • Rationale given for use of qualitative design
  • Discussion of epistemological/ontological grounding
  • Rationale explored for specific qualitative method (e.g. ethnography, grounded theory, phenomenology)
  • Discussion of why particular method chosen is most appropriate/sensitive/relevant for research question/aims
  • Setting appropriate

 

Data collection strategy apparent and appropriate

  • Were data collection methods appropriate for type of data required and for specific qualitative method?
  • Were they likely to capture the complexity/diversity of expereince and illuminate context in sufficient detail?
  • Was triangulation of data sources used if appropriate?

Sampling strategy

Sample and sampling method appropriate

  • Selection criteria detailed, and description of how sampling was undertaken
  • Justification for sampling strategy given
  • Thickness of description likely to be achieved from sampling 
  • Any disparity between planned and actual sample explained 

Analysis

Analytic approach appropriate

  • Approach made explicit (e.g. thematic distillation, constant comparative method, grounded theory)
  • Was it appropriate for the qualitative method chosen?
  • Was data managed by software package of by hand and why?
  • Discussion of how coding systems/conceptual frameworks evolved
  • How was context of data retained during analysis
  • Evidence that the subjective meanings of participants were portrayed
  • Evidence of more than one researcher involved in stages if appropriate to epistemological/theoretical stance
  • Did research participants have any involvement in analysis (e.g. member checking)
  • Evidence provided that data reached saturation or discussion/rationale if it did not
  • Evidence that deviant data was sought, or discussion/rationale if it was not

Interpretation

Context described and taken account of in interpretation

  • Description of social/physical and interpersonal contexts of data collection
  • Evidence that researcher spent time ‘dwelling with the data’, interrogating it for competing/alternative explanations of phenomena

 

Clear audit trail given

  • Sufficient discussion of research processes such that others can follow ‘decision trail’

 

Data used to support interpretation

  • Extensive use of field notes entries/verbatim interview quotes in discussion of findings
  • Clear exposition of how interpretation led to conclusions

Reflexivity

Researcher reflexivity demonstrated

  • Discussion of relationship between researcher and participants during fieldwork
  • Demonstration of researcher’s influence on stages of research process
  • Evidence of self-awareness/insight
  • Documentation of effects of the research on researcher
  • Evidence of how problems/complications met were dealt with

Ethical dimensions

Demonstration of sensitivity to ethical concerns

  • Ethical committee approval granted
  • Clear commitment to integrity, honesty, transparency, equality and mutual respect in relationships with participants
  • Evidence of fair dealing with all research participants
  • Recording of dilemmas met and how resolved in relation to ethical issues
  • Documentation of how autonomy, consent, confidentiality, anonymity were managed

Relevance and transferability

Relevance and transferability evident

  • Sufficient evidence for typicality specificity to be assessed
  • Analysis interwoven with existing theories and other relevant explanatory literature drawn from similar settings and studies
  • Discussion of how explanatory propositions/emergent theory may fit other contexts
  • Limitations/weaknesses of study clearly outlined
  • Clearly resonates with other knowledge and experience
  • Results/conclusions obviously supported by evidence
  • Interpretation plausible and ‘makes sense’
  • Provides new insights and increases understanding
  • Significance for current policy and practice outlined
  • Assessment of value/empowerment for participants
  • Outlines further directions for investigation
  • Comment on whether aims/purposes of research were achieved

Source: Walsh and Downe, 2006

Table 8 – Grid for the critical appraisal of qualitative research articles in medicine and medical education

 

Yes

+/-

No

Introduction

     

1. The issue is described clearly and corresponds to the current state of knowledge.

     

2. The research question and objectives are clearly stated and are relevant to qualitative research (e.g. the process of clinical or pedagogical decision-making).

     

Methods

     

3. The context of the study and the researchers’ roles are clearly described (e.g. setting in which the study takes place, bias).

     

4. The method is appropriate for the research question (e.g. phenomenology, grounded theory, ethnography).

     

5. The selection of participants is appropriate to the research question and to the method selected (e.g. key participants, deviant cases).

     

6. The process for collecting data is clear and relevant (e.g. interview, focus group, data saturation).

     

7. Data analysis is credible (e.g. triangulation, member checking).

     

Results

     

8. The main results are presented clearly.

     

9. The quotations make it easier to understand the results.

     

Discussion

     

10. The results are interpreted in credible and innovative ways.

     

11. The limitations of the study are presented (e.g. transferability).

     

Conclusion

     

12. The conclusion presents a synthesis of the study and proposes avenues for further research.

     

Source: Côté and Turgeon,2005


2.2.4       Conclusion

To conclude this chapter on quality criteria we wish to warn against a rigid use of checklists and quality criteria in qualitative research and to argue instead for flexible use. Moreover this also applies to quantitative research.

Barbour criticizes the widespread use and description of assumed quality indicators like theoretical sampling, grounded theory, multiple coding, and triangulation in scientific articles, as an unequivocal guarantee of robustness. These dimensions of qualitative research should be embedded within a broader understanding of the qualitative research design and not “stuck on as a badge of merit” (Barbour, 2001, p. 1115).

We agree with Walsh and Downe (Walsh, 2006) that a checklist is indicative of good quality research, but not a guarantee.

Key messages

  • Although in quantitative health sciences research, there exist widely-recognised guidelines, no comparable standardised guidelines exist for qualitative research.
  • Among qualitative researchers there is a debate going on between those demanding for explicit criteria, for example in order to serve systematic reviewing and Evidence-Based Practice, and those who argue that such criteria are neither necessary nor desirable.
  • The framework of Walsh and Downe as an comprehensible example of quality criteria checklist to appraise qualitative research studies. The grid of Côté and Turgeon is more simple and could be recommended as tool for evaluation in KCE reports.

 


[a]           Informants may be asked to read transcripts of dialogues in which they have participated to check whether their words match with what they actually intended (Shenton 2004), or they may be asked to check the accuracy of early findings (Bloor 2006) 35.

[b]           Thick description refers to rich qualitative data allowing not only the description of social behaviour, but also to connect it to the broader context in which it occurred (Mortelmans 2009).

[c]           A French-speaking version is also available (Côte 2002)