February 2008, Volume 12, Number 1

 

Article

Measuring Patient-Oriented Outcomes in Palliative Care: Functionality and Quality of Life

Clara Granda-Cameron, RN, MSN, CRNP, AOCN®, Sara R. Viola, Mary Pat Lynch, MSN, CRNP, AOCN®, and Rosemary C. Polomano, RN, PhD, FAAN

 

 

Outcomes measurement is necessary to evaluate quality of care, increase knowledge about experiences with cancer and therapies, and determine the effectiveness of interventions directed toward improving symptoms and quality of life (QOL) in research and clinical care. Recent attention on outcomes measurement and research in palliative care settings has emphasized the need to incorporate patient-reported outcomes. Unlike other areas of research in oncology, palliative care research is comprised largely of descriptive studies elucidating the process involved with palliative care, with a notable void in well-designed patient-oriented studies employing standard instruments for measuring functional status, QOL, symptoms, and psychosocial well-being. Outcomes programs in practice settings where palliative care is an integral part of clinical services can offer important information about patient experiences across the continuum of care and help to identify patients most likely to benefit from palliative care interventions. Therefore, oncology nurses must be informed about outcome-measurement issues, including ways to select reliable and valid instruments and determine which ones are appropriate for palliative care populations. Content related to the measurement of patient-oriented outcomes is presented to assist nurses in developing outcomes programs in palliative care settings.

 

At a Glance

 

·         Measurement of patient outcomes with palliative care is important to understand patient experiences and evaluate the effectiveness of interventions.

·         Numerous reliable and valid patient-reported instruments can be used in palliative care settings.

·         Knowledge of measurement and instruments can assist nurses in selecting the most appropriate outcome measures for clinical care and research.

 

An outpatient palliative care service, the Pain and Supportive Care Program, has been established at the Joan Karnell Cancer Center (JKCC) at Pennsylvania Hospital in Philadelphia. The three-year-old program serves patients with cancer at the nation’s first hospital with an enrollment average of 400 patients in various phases of disease from newly diagnosed to end stage. As part of the program, an interdisciplinary team comprised of oncology nurses, advanced practice nurses, physicians, social workers, physical therapists, a psychologist, music therapists, and other health professionals is designing an outcomes research program to measure the effectiveness of targeted palliative care interventions (e.g., symptom management strategies, structured counseling, music therapy sessions). The team has been challenged to find ways to document how a palliative care program achieves optimal outcomes for patients and families. Recognizing the importance of outcomes measurement, advanced practice nurses at JKCC partnered with two nurse researchers in the School of Nursing at the University of Pennsylvania to review standard instruments to capture patient outcomes throughout the cancer experience. A detailed summary of the team’s work, focused on the measurement of functionality and quality of life (QOL) in palliative care, is presented, and a thorough appraisal of standard instruments and their application in palliative care practice settings is provided.

 

Oncology nurses in inpatient, outpatient, and homecare settings play an important role in patient-oriented outcomes measurement and research. First, oncology nurses can determine the feasibility of collecting patient-oriented outcomes in clinical practice. Is there enough time to obtain patient-reported outcomes? Is there a place where patients can complete questionnaires or instruments? Who is available to assist patients? Second, nurses have unique knowledge of their patients and can help to select the appropriate tools for specific patient populations. Third, nurses appreciate how outcomes can be used to direct clinical care. For example, nurses routinely assess pain and symptoms and use responses from patients to intervene and ensure the appropriate level of care. Last, nurses always are searching for ways to document that their care truly makes a difference in the lives of patients.

 

To participate in outcomes programs, oncology nurses must (1) understand the underlying concepts of outcomes measurement, (2) have knowledge of indicators of quality of care, (3) be familiar with criteria used to evaluate the strengths and weaknesses of instruments, and (4) be able to determine measures that are best suited for their patient populations and practice settings. After all, issues related to measuring patient perceptions and experiences are not new to many nurses because undergraduate and graduate courses in clinical practice, research, and statistics emphasize the reliability and validity of outcomes measures. In fact, many nurses participate in journal clubs, are involved in critiquing research when developing evidence-based practice protocols and guidelines, and are responsible for conducting performance-improvement projects linking care with outcomes. Such activities require an understanding of methods and outcomes of research studies, in addition to how conclusions are formed from measurement indicators. Oncology nurses who work in settings that conduct clinical trials are exposed to a variety of standard patient-reported instruments for functionality and QOL, and some even explain and administer the questionnaires to patients.

 

Oncology nurses continually are challenged to identify ways to measure patient responses to treatments and supportive care interventions such as palliative care. Although considerable progress has been made in developing and testing patient-specific instruments for outcomes research and performance improvement, the instruments seldom are used in clinical practice to build outcomes programs (Clancy & Lawrence, 2002). Outcomes measurement is essential to assess quality of care, increase knowledge about patient experiences with cancer, and evaluate the effectiveness of interventions directed toward improving QOL and symptoms. Demands for greater accountability for quality performance imposed by administrators and national accreditation agencies and organizations have warranted more formalized approaches to gathering data from patients. Outcomes research in oncology is an emerging area of investigation that encompasses the study of quality of adjustment, QOL, satisfaction with care, and the societal impact of cancer. From the National Cancer Institute’s perspective, “Outcomes research describes, interprets, and predicts the impact of various influences, especially (but not exclusively) interventions on ‘final’ endpoints that matter to decision makers, patients, providers, private payers, government agencies, accrediting organizations, or society at large” (Lipscomb & Snyder, 2002, p. 3). Recent attention has focused on outcomes in palliative care settings, emphasizing the need to design comprehensive outcomes programs. Oncology nurses can use patient outcomes to understand individual experiences, direct care, and gauge the success of palliative or supportive care interventions. As such, oncology nurses need to know how to obtain reliable and valid patient-reported data and how to do so in clinical practice and with research. This article informs nurses about the psychometric properties of standard instruments for measuring functionality and QOL, criteria for selecting appropriate instruments for various populations, and strategies for building outcomes programs based on patient-reported measures. Supporting evidence on the utility of the tools in palliative care settings forms the basis for making decisions regarding the most appropriate tools to employ in routine clinical practice and for the purposes of research.   

 

Outcomes Measurement in Palliative Care: A Call for Action

 

In the United States, palliative care has expanded beyond just caring for people at the end of life to improving QOL for all people living with serious, complex, and eventually terminal illnesses; it is offered simultaneously with life-prolonging and curative therapies (National Palliative Care Research Center, 2007). According to the World Health Organization (2002), “Palliative care is an interdisciplinary team approach that improves the quality of life of patients and families living with a life-threatening illness through early identification, assessment, and treatment of pain and other physical, psychological, and spiritual problems.” From 2001–2003, the number of hospital-based palliative care programs grew from 632 to 1,027, a 60% increase (Morrison, Maroney-Galin, Kralovec, & Meier, 2005). Despite the proliferation, limited research demonstrates the benefits of interventions by palliative care teams on patient and care outcomes (Francke, 2000; Higginson et al., 2002; Jack, Hillier, Williams, & Oldham, 2003). Unlike other areas of oncology research, palliative care research is comprised largely of descriptive studies elucidating the process involved in delivering care. An earlier review of 16 studies investigating the effectiveness of palliative care teams showed encouraging results for physical symptoms but no clear and somewhat contradictory findings for other areas such as psychosocial and spiritual outcomes, costs of care, and resource consumption (Francke).

 

Two nationally publicized documents call for greater emphasis on practice-based models that incorporate outcomes measurement in routine clinical care. At the December 2004 National Institute of Health (NIH) State-of-the-Science Consensus Conference on Improving End-of-Life Care, consensus was reached about the significant gaps in science and knowledge related to patient-reported outcomes with palliative care (“NIH State-of-the-Science Conference Statement,” 2004). Similarly, recommendations put forth by experts at the July 2002 NIH State-of-the-Science Conference on Symptom Management in Cancer Pain, Depression and Fatigue drew attention to the lack of focused approaches to assess and treat symptom clusters, specifically pain, fatigue, and depression (Patrick et al., 2003).

 

The message from the National Palliative Care Research Center (2007) is that “without research, palliative care is an art, not a science.” Well-designed studies are needed to determine the effectiveness of palliative care and to identify patient populations likely to benefit most from interventions. Moreover, outcomes research programs in practice settings where palliative care is an integral part of clinical services can provide valuable information about patient experiences across the continuum of care. Because of more broadly defined palliative care, standardized instruments originally designed to measure outcomes with treatment protocols have been adapted and tested specifically for palliative care populations. With so many of the questionnaires and tools, clinicians struggle with when, how, and what to measure in palliative care programs.

 

Considerations in Outcomes Measurement

 

Instrument Selection

 

Before selecting outcomes measurement tools for use in palliative care settings, oncology nurses must consider the following. First, the intent or purpose for outcomes measurement in palliative care has important implications for selecting measurement tools. In research, outcome measures must be capable of generating data to answer questions and test hypotheses. Performance-improvement projects often involve careful attention to measuring outcome(s) related to indicators that monitor quality of care. Outcomes measurement in routine clinical practice may use various approaches, from capturing multiple patient outcomes to a more focused effort of measuring the effectiveness of specific treatments and supportive care approaches. Patient outcomes studies or projects that are conducted to generate information or data for internal institutional use (e.g., performance or quality improvement) in the course of routine clinical practice may not require prior approval from an institutional review board (IRB). On the other hand, when the intent of a study or project is to disseminate findings externally through professional venues (e.g., presentations, publications), researchers must check with the institution’s office for human subjects protection or IRB to determine whether IRB approval may be required. For outcomes research that is an integral part of routine clinical practice, nurses may seek authorization for waiver of informed consent with imposed stipulations on access to protected health information and de-identification of results. Regardless of how data are used or the scope of data collection, appropriate instruments are selected based on their measurement domains, psychometric properties, applicability to patient populations, and feasibility of use.

 

Second, careful attention must be paid to the conceptual basis and domains of an instrument and whether the instrument reflects the outcomes that are of greatest importance to measure. Theory-derived instruments are guided by theoretical constructs and concepts from theories in the construction of items or groups of items. The theoretical constructs or conceptual underpinnings of an instrument generally are cited in the original work describing how the instrument was developed. Conceptual domains, part of a theory or conceptual framework, often are represented by an instrument’s subscales or item groupings measuring a similar concept. Multidimensional measures such as QOL instruments typically have several measurement domains, such as functional status and ability, physical symptoms (e.g., pain, fatigue, nausea, anorexia), psychosocial issues (e.g., psychological distress, occupational concerns, social relationships, sexuality and intimacy, financial concerns), and spirituality.

 

Third, outcomes measures must be used for the appropriate group of respondents, especially for patient-reported instruments. For example, instruments centered on treatment side effects, occupational-related functioning, or rehabilitation may not be appropriate for patients nearing the end of life who have exhausted all treatment options. Disease- and treatment-specific instruments may address unique experiences or aspects of care that are relevant to a given population and, therefore, may be applicable only to certain patients. Last and importantly, outcome measures must be reliable and valid in palliative care. Instruments are selected for their ease of use, number of items, time for completion, readability and interpretability of items or questions by responders, timing and sequencing of administration, and scoring methods. When evaluating culturally diverse populations, nurses must take into account availability of instruments in multiple languages and cultural variations.

 

Psychometric Properties of Patient-Reported Instruments

 

Psychometric properties of instruments are defined by measures of reliability and validity. Estimates of reliability and validity are obtained from results of either homogeneous (similar) or heterogeneous (diverse) study samples and are interpreted in relation to the samples. Psychometric properties of an instrument may not hold up across various groups of patients; therefore, clinicians should select tools that have demonstrated reliability and validity in populations similar to the ones they wish to study.

 

Reliability refers to consistency in measurement, consistency over time, and an instrument’s reproducibility in results. The reliability coefficient is an index of stability and is represented by a value assigned using a particular method, usually a correlation coefficient between 0 and 1; the higher the correlation coefficient, the higher the reliability. The threshold for acceptable reliability is generally 0.8; however, slightly lower values are tolerated depending on the numbers of items assessed and methods used (Nunnally & Bernstein, 1994). Reliability may be higher for the overall items of an instrument compared to item groups and subscales. Table 1 describes several forms of reliability; meeting the requirements for reliability using only one method is sufficient.

 

Validity indicates how well an instrument measures what it is supposed to measure. Judgments are based on evidence and statistical procedures to ascertain the appropriateness of inferences drawn from the results of scores obtained from a group of respondents and can be “acceptable” or “weak.” An instrument that is not reliable is not considered valid and consequently would not meet the rigor required for acceptable psychometric properties (Polit & Beck, 2007). The types of validity and the ways in which validity can be determined are detailed in Table 1. Unlike reliability, no single validity coefficient or indicator exists for an instrument; rather, techniques are based on estimates and judgments of what scores on a measure really mean.

 

Item Burden

 

Attention to respondent item burden is an important consideration in selecting an instrument or combining instruments or tools, especially when studying patients who must contend with complex treatment regimens, progressive disease, and the end of life. Little research has been done to address item burden, but some factors that contribute are: (a) relevance of measurement domains and items to respondents, (b) length of a questionnaire and time for completion, (c) sequence and timing of administration, and (d) overall impact on the respondent, such as time commitment, physical exhaustion, and emotional distress from answering questions that may be upsetting. Equally important is determining the optimal point during the course of the disease trajectory, the individual readiness of patients, and their cognitive capacity for completing questionnaires. Figure 1 outlines strategies to help researchers overcome item burden.

 

Palliative Care Outcome Measurement Tools

 

The Center to Advance Palliative Care ([CAPC], n.d.) and National Council for Palliative Care Web sites are excellent places to find outcome measures that are suitable for use in palliative care settings. Many of the tools can be downloaded as PDF files so that clinicians and researchers can view the items that are included in the measure. However, practicing clinicians also must evaluate the measurement domains and consider the relative contributions that the instruments offer in capturing the outcomes of greatest interest. This article presents several instruments for measuring functionality and QOL, with relevant research supporting their psychometric properties and suitability for palliative care populations. A list of the instruments and practical information about their structures, measurement formats, and time required for completion appear in Table 2.  

 

Functional Status Instruments

 

Karnofsky Performance Scale

 

The Karnofsky Performance Scale (KPS) is a single-item, unidimensional functional status scale used to obtain a global measure of level of activity, especially for patients undergoing cancer treatment (Hwang et al., 2004). Level of functionality is rated by a healthcare provider as a percentage ranging from 100% (normal, no complaints, no evidence of disease) down to 0% (dead). Acceptable reliability and validity have been established in research and clinical practice, especially when standardized scoring guidelines are used and opportunities to interview patients are possible (Mor, Laliberte, Morris, & Wiemann, 1984; Schag, Heinrich, & Ganz, 1984). A low KPS score is associated with high symptom distress (Hwang, Chang, Fairclough, Cogswell, & Kasimis, 2003). Because it correlates closely with levels of symptom distress, the KPS often is used as a proxy measure for a global evaluation of a patient’s status and most appropriately as a prognostication tool to predict life expectancy (Maltoni & Amadori, 2002; Stanley, 1980). A clear disadvantage of the KPS is its lack of specificity for defining aspects of function from the patient’s perspective.

 

Palliative Performance Scale

 

The Palliative Performance Scale (PPS) is based on a similar premise as the KPS and is completed by a healthcare provider. The original version, consisting of a unidimensional scale (Anderson, Downing, Hill, Casorso, & Lerch, 1996), has been expanded to include dimensions of ambulation, activity, evidence of disease, self-care, intake, and consciousness level. A healthcare professional scores each dimension by assigning a value from 100% to 0% (death), with 10% denoting the lowest level of functioning (PPS version 2) (Victoria Hospice, 2001). Ratings of ambulation and activity and evidence of disease are dominant over the latter variables. For example, a patient who mainly sits or lies down all day (50% ambulation score) but has normal intake and normal consciousness level (100% intake and conscious level scores) has an overall PPS score of 50%. Therefore, researchers and clinicians who plan to use the PPS version 2 should consider whether the hierarchy reflects the priorities for assessing patients.

 

Acceptable reliability and validity have been established for the PPS version 2, and studies have shown strong agreement with the KPS (Bradley, Davis, & Chow, 2005; Hwang et al., 2004). Studies differ in the value of PPS version 2 scores as prognostic indicators. PPS version 2 scores on admission have been shown to be significant predictors of survival among patients receiving palliative care and hospice care, but studies differ in measuring the impact of diagnosis on survival. Harrold et al. (2005) found the PPS to be a stronger prognostic tool for nursing home residents with noncancer diagnoses than for those with cancer, whereas Lau, Downing, Lesperance, Shaw, and Kuziemsky (2006) reported no appreciable impact of diagnosis on survival of patients admitted to a palliative care unit. Such differences likely are related to sample size and characteristics and study settings. The scale has successfully predicted length of stay in hospice; declining scores are associated with worsening condition and death, whereas stable scores are associated with discharge from hospice care (Head, Ritchie, & Smoot, 2005). However, the PPS version 2 demonstrates insufficient sensitivity to distinguish outcomes between patients in the 30% category from those at 40% or between those in the 50% category from those in at 60% (Head et al.). Variations in performance of the scale may be attributed to discrepancies among users in interpreting and applying criteria from the scale, rather than an inherent problem with the scale itself.

 

Short-Form Health Survey

 

The Short-Form Health Survey (SF-36) is one of the most widely used health outcomes survey. It consists of eight subscales measuring limitations in physical activities, social activities, and role activities caused by physical health and emotional problems, in addition to physical pain, general mental health, vitality, and general health perceptions (Stewart, Hays, & Ware, 1988). Although it commonly is touted as a functional status and QOL instrument, it sometimes is described as a measurement of global health status. The SF-36 evolved from the 20-item Medical Outcomes Study (MOS) short form, which was inadequate in representing its domains and had less measurement sensitivity (Ware & Sherbourne, 1992). The survey can be completed by the patient or an interviewer. The SF-36 is capable of detecting outcomes related to QOL independent of mood, as measured by the Profile of Mood States (POMS), among patients with neuropathic pain (Deshpande, Holden, & Gilron, 2006). When compared to the Functional Assessment of Cancer Therapy-General (FACT-G), a QOL instrument, and Symptom Distress Scale (SDS), the SF-36 ranked first in achieving a 100% completion rate, and patients with lymphoma, early-stage cancer, or recent treatment had a higher preference for the SF-36 (Cooley et al., 2005). However, most patients favored the FACT-G and the SDS over the SF-36. Nevertheless, the SF-36 is a valid and reliable instrument that has been used extensively with patients with cancer (Golden-Kreutz et al., 2005; Ware, Kosinski, & Keller, 1994; Ware, Snow, & Kosinski, 2000). Shorter versions of the SF-36, such as the SF-12, are available, but they have not been adequately studied in palliative care settings (Radbruch et al., 2000).

 

Quality-of-Life Instruments

 

European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30

 

The European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire Core 30 (QLQ-C30) consists of five multi-item scales measuring functionality in physical, role, social, emotional, and cognitive dimensions; three symptom scales measuring fatigue, pain, and nausea and vomiting; six single-item symptom measures; and a global health and QOL scale. Bruley (1999) raised concern about the appropriateness of the EORTC QLQ-C30 for patients in palliative care settings because it conceptualizes QOL in terms of normal life for a healthy individual. Researchers have concerns that the length of the instrument may be too demanding for respondents and that the items are not well suited for palliative care (Groenvold et al., 2006a). The EORTC QLQ-C30 is most appropriate for use during and directly after cancer treatment because of its emphasis on physical and functional aspects of QOL, which are most relevant at that time (Kopp et al., 2000). Several studies have demonstrated reliability and validity for the EORTC QLQ-C30 and its sensitivity to changes in QOL over time (Bjorner et al., 2004; Groenvold, Klee, Sprangers, & Aaronson, 1997; Hjermstad, Fossa, Bjordal, & Kaasa, 1995; Kaasa et al., 1995; Niezgoda & Pater, 1993). The EORTC QLQ-C30 has been translated and validated in 81 languages, and several disease-specific supplements exist, including breast, lung, head and neck, esophagus, ovary, gastric, and cervical cancers and multiple myeloma (EORTC Quality of Life Questionnaires, 2006).

 

European Organization for Research and Treatment of Cancer Quality of Life Questionnaire for Palliative Care

 

To reduce respondent burden in multi-instrument studies, researchers and clinicians should consider using the shorter version of the EORTC QLQ-C30, the EORTC Quality of Life Questionnaire for Palliative Care (QLQ-C15-PAL), which has 15 items specifically designed for palliative care (Groenvold et al., 2006b). The elimination of items from the EORTC QLQ-C30 such as “Were you limited in pursuing your hobbies or other leisure time activities?” may have modified the original version in such a way to be better suited for end-of-life populations, limiting its utility with patients across the continuum of care. Because items were not respondent generated, the EORTC QLQ-C15-PAL fails to address existential and spiritual issues (Echteld, Deliens, Onwuteaka-Philipsen, Klein, & van der Wal, 2006). Similar to the EORTC QLQ-C30, the EORTC QLQ-C15-PAL captures the physical domains of QOL, favoring its use for measuring functionality. Although the EORTC QLQ-C15-PAL is relatively new, it has considerable potential for use with patients with progressive disease.

 

Functional Assessment of Cancer Therapy–General and Other Versions

 

The Functional Assessment of Cancer Therapy-General (FACT-G) is a 27-item QOL assessment tool with a five-point rating scale and measured well-being in physical, social, emotional, and functional dimensions. Its reliability, validity, and sensitivity to changes over time have been documented consistently (Cella et al., 1993). The development of the instrument was guided by respondent-generated information prioritizing the content of greatest importance to patients. Another major strength of the FACT-G is its minimal variability, which requires fewer respondents than instruments with large variability. In comparison to the EORTC QLQ-C30, smaller sample sizes may be required for FACT-G versions because it has greater ability to detect treatment differences in differentiating patient groups based on performance status (Cheung, Goh, Thumboo, Khoo, & Wee, 2005). Disease-specific and treatment-specific versions of the FACT tools differ from the EORTC QLQ modules in their QOL evaluation strategies. Whereas FACT tools seem to address the multitude of facets of QOL equally, the EORTC QLQ modules focus more on the physical aspects of patients’ lives that are likely to influence QOL (Kopp et al., 2000). The FACT-G is part of an entire collection of questionnaires, the Functional Assessment of Chronic Illness Therapy (FACIT), which include several validated disease-specific, symptom-specific, and treatment-specific versions, available at www.facit.org (Cella, 1997, n.d.).

 

Spitzer Quality of Life Index

 

The Quality of Life index (QLI) measures health-related QOL with only five items rated on a scale from 0 (indicating best QOL) to 2 (indicating worst QOL) (Perez, McGee, Campbell, Christensen, & Williams, 1997; Spitzer et al., 1981). The instrument measures activity, daily life, health perceptions, social support, and behavior. Along with the FACT-G and the Spitzer Uniscale (not reviewed here), the QLI was one of the most frequently used QOL assessments in the late 1990s (Buchanan, O’Mara, Kelaghan, & Minasian, 2005). A low QLI score has been associated with a greater likelihood of death within six months compared with higher scores; however, scores from the limited item set have not been able to predict the type or length of treatment required for terminally ill patients (Addington-Hall, MacDonald, & Anderson, 1990). Its brevity and ease of administration and scoring certainly are desirable for use in clinical practice, but the QLI is a global measure and thus not recommended as a comprehensive measure to evaluate QOL.

 

Cancer Rehabilitation Evaluation System and Its Short Form

 

The Cancer Rehabilitation Evaluation System (CARES) is designed to evaluate the rehabilitation status of patients with cancer. The scale has five domains: physical, psychosocial, medical interaction, marital, and sexual (Schag, Heinrich, & Ganz, 1983). Every patient answers the first 88 of 139 questions, but only certain patients answer the remaining 51 items, which relate to population-specific concerns. Respondents rate the extent to which each statement applies to them during the past month on a five-point scale. Internal consistency reliability and test-retest reliability are excellent for the CARES, and validity has been established (Schag et al., 1983; Schag, Heinrich, Aadland, & Ganz, 1990). The one-month time frame may be difficult for patients with fluctuating symptoms and dramatic changes in status. The CARES may not be ideal for use in routine clinical practice because of the number of items, may not have sufficient ability to determine how a patient’s life is affected by treatment interventions, and may not be appropriate for patients nearing the end-of-life.

 

A short-form, 59-item version of the CARES has been created, the CARES-SF (Schag, Ganz, & Heinrich, 1991). The CARES and CARES-SF have a unique set of subscales, which generally are not emphasized in other QOL instruments, namely medical interactions, marital, and sexual. Concurrent validity has been demonstrated for the CARES-SF. In a study of patients newly diagnosed with breast cancer, rehabilitation scores correlated with results from the FLIC (Schag et al., 1991). Another study of 130 patients after stem cell transplantation showed a strong and statistically significant correlation between the physical subscales of the CARES-SF and the EORTC QLQ-C30 (Hjermstad et al., 2003). With 80 patients with metastatic breast cancer, a significant correlation was found between mood disturbance, as measured by the Hospital Anxiety and Depression Scale, and physical rehabilitation status, as measured by the CARES-SF (Fulton, 1999). The CARES and CARES-SF may be appropriate for patients with cancer receiving palliative care rehabilitation interventions and when specific aspects of physical and psychosocial functioning are important to measure.

 

McGill Quality of Life Questionnaire

 

The 17-item McGill Quality of Life Questionnaire (MQOL) incorporates a unique dimension: the existential domain defined as a patient’s subjective well-being as affected by imminent death, loss of freedom, isolation, and meaning or purpose in life. Additional domains include physical, psychological, and support along with a global QOL item. Reliability and validity have been established in patients with cancer at all stages of disease and with individuals with varying prognoses. The MQOL has been used and tested in hospice and palliative care settings. Because of its brevity, the MQOL may not be the best measure to monitor QOL for any single patient over time. However, it is sufficiently sensitive to detect differences among groups receiving various treatments and services (Cohen & Mount, 2000). When compared to the Patient Evaluated Problem Scores (PEPS) (an individualized questionnaire requiring patients to identify and rate major problems affecting their QOL), using a small sample of 36 patients, the MQOL was acceptable to 95% of respondents and favored by 60% because they believed that it was comprehensive (Pratheepawanit, Salek, & Finlay, 1999). However, it did not detect some of the problems noted in the PEPS, such as isolation.

 

Schedule for Evaluation of Individual Quality of Life

 

Some instruments, such as the Schedule for Evaluation of Individual Quality of Life (SEIQoL), are considered respondent generated because they are constructed to allow respondents to define the domains of QOL from their own perspectives (Macduff, 2000). The instrument’s underlying measurement framework focuses on the areas of life that are important to respondents, how they currently are doing in each of area, and the perceived importance of the areas to overall QOL (O’Boyle & Waldron, 1997). A trained interviewer must conduct semistructured interviews to elicit responses on 46 items, first by asking respondents to draw a bar from bottom to top indicating how good or bad life is with respect to each domain (bottom being worst, top being best). Next, respondents rate their overall QOL on a visual analog scale, placing a mark on 100 mm line from poorest QOL to best QOL. Finally, they complete a “judgment analysis procedure,” in which 30 hypothetical scenarios are presented and expected QOL is rated.

 

The original SEIQoL is valid and reliable in relatively healthy patient populations. However, it may be too burdensome for repetitive clinical use and takes about 40 minutes to complete. In one study, 22% of patients failed to finish the SEIQoL because of fatigue (Waldron, O’Boyle, Kearney, Moriarty, & Carney, 1999). The SEIQoL-Direct Weighting (SEIQoL-DW) is a shortened version, also validated, that requires about 15 minutes and thus may be more acceptable for patients with progressive cancer (Campbell & Whyte, 1999; Kaasa & Loge, 2003; Willener & Hantikainen, 2005). With the SEIQoL-DW, the “judgment analysis procedure” is replaced with interlocking discs representing the weight given to each nominated domain of QOL. A patient stacks them in such a way that the weight of each domain can be calculated and analyzed in combination with the other sections of the assessment (Lhussier, Watson, Reed, & Clarke, 2005). Insignificant correlation exists among the SEIQoL-DW, KPS, and EORTC QLQ-C30 for patients with leukemia undergoing autologous stem cell transplantation, which may illustrate its conceptual departure from traditional health-related QOL assessments (Frick, Borasio, Zehentner, Fischer, & Bumeder, 2004). Researchers warn that an interviewer’s behavior and wording of instructions, as well as an optional list of prompts for patients who fail to think of five domains on their own, may introduce bias in the SEIQoL and SEIQoL-DW. The long and short versions of the SEIQoL can yield valuable information about a patient’s QOL; however, the facts that interviewers must be trained carefully, administration is complex, and specific software is needed to analyze responses preclude its feasibility in routine clinical practice.

 

Functional Living Index–Cancer

 

The Functional Living Index–Cancer (FLIC) is a 22-item, seven-point rating QOL measure. Construct validity for the subscale structure of measurement domains (physical well-being and ability, emotional state, sociability, family situation, and nausea) has been supported in three cancer populations (King, Dobson, & Harnett, 1996; Schipper, Clinch, McMurray, & Levitt, 1984). A unique feature of the FLIC is the phrasing of items. For example, social functioning is measured in terms of a patient’s “willingness to see and spend time with relatives and friends,” as opposed to the extent to which disease and treatment interfere with social interactions (Kuenstner et al., 2002). Responses to items for the FLIC are framed in the context of how patients feel the day of completing the questionnaire. As a result, responses on the FLIC may differ from answers to items from QOL measures that ask patients how they feel in general, recently, or within the past two weeks. This may explain the problem of construct divergence or differences in results noted between the FLIC and SF-36 (Wilson, Hutson, & VanStry, 2005).

 

Putting Outcomes Measurement Into Practice

 

To illustrate how oncology nurses can use standardized measurement tools in practice, a few case-based examples are provided throughout the text of this article. Selecting the most suitable instrument to capture the most critical information about patients’ status and experiences is paramount. The case scenarios make clear just how valuable patient-assessment criteria and patient-reported information can be in clinical practice. Moreover, the use of outcomes measures provides a useful way to quantify, communicate, and track relevant data about patients.

 

Conclusions

 

The viability and expansion of palliative care programs no doubt will rely on healthcare professionals’ ability to demonstrate the programs’ unique contributions to high-quality patient care. Therefore, a system that tracks patient outcomes over time is critical. Nurse leaders in palliative care should be accountable for developing a comprehensive plan for measuring and monitoring the impact of services and interventions that are part of palliative care programs. However, that is not easy and may require additional time and resources to accomplish data collection, analysis of findings, and reporting mechanisms. A major challenge can be identifying the most appropriate indicators to quantify the benefits that palliative care interdisciplinary teams bring to patient care. Nurses who are knowledgeable about outcomes measurement are in a better position to plan and make decisions regarding the best options for measuring the effectiveness of interventions and the overall well-being of their patients. Collaborations between clinicians and researchers greatly enhance the feasibility and capacity for implementing outcomes programs in routine clinical practice and for research.

 

The authors gratefully acknowledge Theresa Alcorn, BSA, assistant to the Palliative Care Program in the Joan Karnell Cancer Center at Pennsylvania Hospital in Philadelphia, for her dedication and enthusiasm in the development of this article.

 

References

 

Addington-Hall, J.M., MacDonald, L.D., & Anderson, H.R. (1990). Can the Spitzer Quality of Life Index help to reduce prognostic uncertainty in terminal care? British Journal of Cancer, 62(4), 695–699.

 

Anderson, F., Downing, G.M., Hill, J., Casorso, L., & Lerch, N. (1996). Palliative Performance Scale (PPS): A new tool. Journal of Palliative Care, 12(1), 5–11.

 

Bjorner, J.B., Petersen, M.A., Groenvold, M., Aaronson, N., Ahlner-Elmqvist, M., Arraras, J.I., et al. (2004). Use of item response theory to develop a shortened version of the EORTC QLQ-C30 emotional functioning scale. Quality of Life Research, 13(10), 1683–1697. [CrossRef]

 

Bradley, N., Davis, L., & Chow, E. (2005). Symptom distress in patients attending an outpatient palliative radiotherapy clinic. Journal of Pain and Symptom Management, 30(2), 123–131. [CrossRef]

 

Bruley, D.K. (1999). Beyond reliability and validity: Analysis of selected quality-of-life instruments for use in palliative care. Journal of Palliative Medicine, 2(3), 299–309. [CrossRef]

 

Buchanan, D.R., O’Mara, A.M., Kelaghan, J.W., & Minasian L.M. (2005). Quality-of-life assessment in the symptom management trials of the National Cancer Institute–Supported

Community Clinical Oncology Program. Journal of Clinical Oncology, 23(3), 591–598. [CrossRef]

 

Campbell, S., & Whyte, F. (1999). The quality of life of cancer patients participating in phase I clinical trials using SEIQoL-DW. Journal of Advanced Nursing, 30(2), 335–343. [CrossRef]

 

Cella, D. (n.d.). Functional Assessment of Chronic Illness Therapy. Retrieved September 4, 2006, from http://www.facit.org

 

Cella D. (1997). Functional Assessment of Chronic Illness Therapy (FACIT) measurement system. FACIT manual. Evanston, IL: Center on Outcomes Research and Education, Evanston Northwestern Healthcare, and Northwestern University.

 

Cella, D.F., Tulsky, D.S., Gray, G., Sarafian, B., Linn, E., Bonomi, A., et al. (1993). The Functional Assessment of Cancer Therapy scale: Development and validation of the general measure. Journal of Clinical Oncology, 11(3), 570–579.

Center to Advance Palliative Care. (n.d.). Additional resources and Websites. Retrieved January 25, 2007, from http://www.capc.org/research-and-references-for-palliative-care/add-resources
-websites

 

Cheung, Y.B., Goh, C., Thumboo, J., Khoo, K.S., & Wee, J. (2005). Variability and sample size requirements of quality-of-life measures: A randomized study of three major questionnaires. Journal of Clinical Oncology, 23(22), 4936–4944. [CrossRef]

 

Clancy, C.M., & Lawrence, W. (2002). Is outcomes research on cancer ready for prime time? Medical Care, 40(6, Suppl.), 92–100. [CrossRef]

 

Cohen, S.R., Balfour, B.M., Tomas, J.J., & Mount, L.F. (1996). Existential well-being is an important determinant of quality of life: Evidence from the McGill Quality of Life Questionnaire. Cancer, 77(3), 576–586. [CrossRef]

 

Cohen, S.R., & Mount, B.M. (2000). Living with cancer: “Good” days and “bad” days—What produces them? Can the McGill Quality of Life Questionnaire distinguish between them? Cancer, 89(8), 1854–1865.

 

Cooley, M.E., McCorkle, R., Knafl, G.J., Rimar, J., Barbieri, M.J., Davies, M., et al. (2005). Comparison of health-related quality of life questionnaires in ambulatory oncology. Quality of Life Research, 14(5), 1239–1249. [CrossRef]

 

Deshpande, M.A., Holden, R.R., & Gilron, I. (2006). The impact of therapy on quality of life and mood in neuropathic pain: What is the effect of pain reduction? Anesthesia and Analgesia, 102(5), 1473–1479. [CrossRef]

 

DeVellis, R.F. (1991). Scale development: Theory and applications (Applied Social Research Methods Series, vol. 26). Newbury Park, CA: Sage.

 

Echteld, M.A., Deliens, L., Onwuteaka-Philipsen, B., Klein, M., & van der Wal, G. (2006). EORTC QLQ-C15-PAL: The new standard in the assessment of health-related quality of life in advanced cancer? Palliative Medicine, 20(1), 1–2. [CrossRef]

 

European Organization for Research and Treatment of Cancer. (2006). EORTC Quality of Life Questionnaires. Retrieved September 8, 2006, from http://www.eortc.be/home/qol/ExplQLQ
-C30.htm

 

Francke, A.L. (2000). Evaluating research on palliative support teams: A literature review. Patient Education and Counseling, 41(1), 83–91. [CrossRef]

 

Frank-Stromborg, M., & Olsen, S.J. (Eds.). (2004). Instruments for clinical health-care research. Sudbury, MA: Jones and Bartlett.

 

Frick, E., Borasio, G.D., Zehentner, H., Fischer, N., & Bumeder, I. (2004). Individual quality of life of patients undergoing autologous peripheral blood stem cell transplantation. Psycho-Oncology, 13(2), 116–124. [CrossRef]

 

Fulton, C. (1999). Patients with metastatic breast cancer: Their physical and psychological rehabilitation needs. International Journal of Rehabilitation Research, 22(4), 291–301.

 

Golden-Kreutz, D.M., Thornton, L.M., Wells-Di Gregorio, S., Frierson, G.M., Jim, H.S., Carpenter, K.M., et al. (2005). Traumatic stress, perceived global stress, and life events: Prospectively predicting quality of life in breast cancer patients. Health Psychology, 24(3), 288–296. [CrossRef]

 

Groenvold, M., Klee, M.C., Sprangers, M.A., & Aaronson, N.K. (1997). Validation of the EORTC QLQ-C30 quality of life questionnaire through combined qualitative and quantitative assessment of patient-observer agreement. Journal of Clinical Epidemiology, 50(4), 441–450. [CrossRef]

 

Groenvold, M., Petersen, M.A., Aaronson, N.K., Arraras, J.I., Blazeby, J.M., Bottomley, A., et al. (2006a). EORTC QLQ-C15-PAL: The new standard in the assessment of health-related quality of life in advanced cancer? Palliative Medicine, 20(1), 5961. [CrossRef]

 

Groenvold, M., Petersen, M.A., Aaronson, N.K., Arraras, J.I., Blazeby, J.M., Bottomley, A., et al. (2006b). The development of the EORTC QLQ-C15-PAL: A shortened questionnaire for cancer patients in palliative care. European Journal of Cancer, 42(1), 55–64. [CrossRef]

 

Harrold, J., Rickerson, E., Carroll, J.T., McGrath, J., Morales, K., Kapo, J., et al. (2005). Is the Palliative Performance Scale a useful predictor of mortality in a heterogeneous hospice population? Journal of Palliative Medicine, 8(3), 503–509. [CrossRef]

 

Head, B., Ritchie, C.S., & Smoot, T.M. (2005). Prognostication in hospice care: Can the Palliative Performance Scale help? Journal of Palliative Medicine, 8(3), 492–502. [CrossRef]

 

Higginson, I.J., Finlay, I., Goodwin, D.M., Cook, A.M., Hood, K., Edwards, A.G., et al. (2002). Do hospital-based palliative teams improve care for patients or families at the end of life? Journal of Pain and Symptom Management, 23(2), 96–106. [CrossRef]

 

Hjermstad, M.J., Evensen, S.A., Kvaloy, S.O., Loge, J.H., Fayers, P.M., & Kaasa, S. (2003). The CARES-SF used for prospective assessment of health-related quality of life after stem cell transplantation. Psycho-Oncology, 12(8), 803–813. [CrossRef]

 

Hjermstad, M.J., Fossa, S.D., Bjordal, K., & Kaasa, S. (1995). Test/retest study of the European Organization for Research and Treatment of Cancer Core Quality-of-Life Questionnaire. Journal of Clinical Oncology, 13(5), 1249–1254.

 

Hwang, S.S., Chang, V.T., Fairclough, D.L., Cogswell, J., & Kasimis B. (2003). Longitudinal quality of life in advanced cancer patients: Pilot study results from a VA medical cancer center. Journal of Pain and Symptom Management, 25(3), 225–235. [CrossRef]

 

Hwang, S.S., Scott, C.B., Chang, V.T., Cogswell, J., Srinivas, S., & Kasimis, B. (2004). Prediction of survival for advanced cancer patients by recursive partitioning analysis: Role of Karnofsky performance status, quality of life, and symptom distress. Cancer Investigation, 22(5), 678–687. [CrossRef]

 

Jack, B., Hillier, V., Williams, A., & Oldham, J. (2003). Hospital based palliative care teams improve the symptoms of cancer patients. Palliative Medicine, 17(6), 498–502. [CrossRef]

 

Kaasa, S., Bjordal, K., Aaronson, N., Moum, T., Wist, E., Hagen, S., et al. (1995). The EORTC Core Quality of Life Questionnaire (QLQ-C30): Validity and reliability when analysed with patients treated with palliative radiotherapy. European Journal of Cancer, 31A(13-14), 2260–2263. [CrossRef]

 

Kaasa, S., & Loge, J.H. (2003). Quality of life in palliative care: Principles and practice. Palliative Medicine, 17(1), 11–20. [CrossRef]

 

King, M.T., Dobson, A.J., & Harnett, P.R. (1996). A comparison of two quality-of-life questionnaires for cancer clinical trials: The Functional Living Index-Cancer (FLIC) and the Quality of Life Questionnaire Core Module (QLQ-C30). Journal of Clinical Epidemiology, 49(1), 21–29. [CrossRef]

 

Kopp, M., Schweigkofler, H., Holzner B., Nachbaur, D., Niederwieser, D., Fleischhacker, W.W., et al. (2000). EORTC QLQ-C30 and FACT-BMT for the measurement of quality of life in bone marrow transplant recipients: A comparison. European Journal of Haematology, 65(2), 97–103. [CrossRef]

 

Kuenstner, S., Langelotz, C., Budach, V., Possinger, K., Krause, B., & Sezer O. (2002). The comparability of quality of life scores: A multitrait multimethod analysis of the EORTC QLQ-C30, SF-36 and FLIC questionnaires. European Journal of Cancer, 38(3), 339–348. [CrossRef]

 

Lau, F., Downing, G.M., Lesperance, M., Shaw, J., & Kuziemsky, C. (2006). Use of Palliative Performance Scale in end-of-life prognostication. Journal of Palliative Medicine, 9(5), 1066–1075. [CrossRef]

 

Lhussier, M., Watson, B., Reed, J., & Clarke, C.L. (2005). The SEIQoL and functional status: How do they relate? Scandinavian Journal of Caring Sciences, 19(4), 403–409. [CrossRef]

 

Lipscomb, J., & Snyder, C.F. (2002). The outcomes of cancer outcomes research: Focusing on the National Cancer Institute’s quality-of-care initiative. Medical Care, 40(6, Suppl.), 3–10. [CrossRef]

 

Macduff, C. (2000). Respondent-generated quality of life measures: Useful tools for nursing or more fool’s gold? Journal of Advanced Nursing, 32(2), 375–382. [CrossRef]

 

Maltoni, M., & Amadori, D. (2002). Prognosis in advanced cancer. Hematology/Oncology Clinics of North America, 16(3), 715–729. [CrossRef]

 

Mor, V., Laliberte, L., Morris, J.N., & Wiemann, M. (1984). The Karnofsky Performance Status Scale. An examination of its reliability and validity in a research setting. Cancer, 53(9), 2002–2007. [CrossRef]

 

Morrison, R.S., Maroney-Galin, C., Kralovec, P.D., & Meier, D.E. (2005). The growth of palliative care programs in United States hospitals. Journal of Palliative Medicine, 8(6), 1127–1134. [CrossRef]

 

National Institutes of Health State-of-the-Science Conference Statement. Improving end-of-life care. (December 6–8, 2004). Journal of Pain and Palliative Care Pharmacotherapy, 19(3), 75–83.

 

National Palliative Care Research Center. (2007). Retrieved January 25, 2007, from http://www.npcrc.org/about/about_show.htm?doc_id=374963

 

Niezgoda, H.E., & Pater, J.L. (1993). A validation study of the domains of the core EORTC quality of life questionnaire. Quality of Life Research, 2(5), 319–325. [CrossRef]

 

Nunnally, J.C., & Bernstein, I. (1994). Psychometric theory (3rd ed). New York: McGraw Hill.

 

O’Boyle, C.A., & Waldron, D. (1997). Quality of life issues in palliative medicine. Journal of Neurology, 244(Suppl. 4), S18–S25. [CrossRef]

 

Patrick, D.L., Ferketich, S.L., Frame, P.S., Harris, J.J., Hendricks, C.B., Levin, B., et al. (2003). National Institutes of Health State-of-the-Science Conference Statement: Symptom management in cancer: Pain, depression, and fatigue. Journal of the National Cancer Institute, 95(15), 1110–1117.

 

Perez, D.J., McGee, R., Campbell, A.V., Christensen, E.A., & Williams, S. (1997). A comparison of time trade-off and quality of life measures in patients with advanced cancer. Quality of Life Research, 6(2), 133–138. [CrossRef]

 

Polit, D.E., & Beck, C.T. (2007). Assessing measurement quality in quantitative studies. In Nursing research: Generating and assessing evidence for nursing practice (8th ed., pp. 449–473). Philadelphia: Lippincott Williams and Wilkins.  

 

Pratheepawanit, N., Salek, M.S., & Finlay, I.G. (1999). The applicability of quality-of-life assessment in palliative care: Comparing two quality-of-life measures. Palliative Medicine, 13(4), 325334. [CrossRef]

 

Radbruch, L., Sabatowski, R., Loick, G., Jonen-Thielemann, I., Kasper, M., Gondek, B., et al. (2000). Cognitive impairment and its influence on pain and symptom assessment in a palliative care unit: Development of a Minimal Documentation System. Palliative Medicine, 14(4), 266–276. [CrossRef]

 

Schag, C.A., Ganz, P.A., & Heinrich, R.L. (1991). Cancer Rehabilitation Evaluation System-Short Form (CARES-SF). A cancer specific rehabilitation and quality of life instrument. Cancer, 68(6), 14061413. [CrossRef]

 

Schag, C.A., Heinrich, R.L., Aadland, R.L., & Ganz, P.A. (1990). Assessing problems of cancer patients: Psychometric properties of the Cancer Inventory of Problem Situations. Health Psychology, 9(1), 83–102. [CrossRef]

 

Schag, C.A., Heinrich, R.L., & Ganz, P.A. (1983). The Cancer Inventory of Problem Situations: An instrument for assessing cancer patients’ rehabilitation needs. Journal of Psychosocial Oncology, 1(4), 11–24.

 

Schag, C.C., Heinrich, R.L., & Ganz, P.A. (1984). Karnofsky performance status revisited: Reliability, validity, and guidelines. Journal of Clinical Oncology, 2(3), 187–193.

 

Schipper, H., Clinch, J., McMurray, A., & Levitt, M. (1984). Measuring the quality of life of cancer patients: The Functional Living Index-Cancer: Development and validation. Journal of Clinical Oncology, 2(5), 472–483.

 

Spitzer, W.O., Dobson, A.J., Hall, J., Chesterman, E., Levi, J., Shepherd, R., et al. (1981). Measuring the quality of life of cancer patients: A concise QL-index for use by physicians. Journal of Chronic Diseases, 34(12), 585–597. [CrossRef]

 

Stanley, K.E. (1980). Prognostic factors for survival in patients with inoperable lung cancer. Journal of the National Cancer Institute, 65(1), 25–32.

 

Stewart, A.L., Hays, R.D., & Ware, J.E., Jr. (1988). The MOS short-form general health survey: Reliability and validity in a patient population. Medical Care, 26(7), 724–735.

 

Victoria Hospice. (2001). Palliative Performance Scale (PPSv2) version 2. Retrieved January 25, 2007, from http://www.victoriahospice.org/Revised%20PPSv2%20ScaleJuly06.pdf

 

Waldron, D., O’Boyle, C.A., Kearney, M., Moriarty, M., & Carney, D. (1999). Quality-of-life in advanced cancer: Assessing the individual. Journal of Clinical Oncology, 17(11), 3603–3611.

 

Waltz, C. F., Strickland, O. L., & Lenz, E. R. (Eds.). 2003. Measurement in nursing and health research (3rd ed.). New York: Springer.

 

Ware, J.E., Kosinski, M., & Keller, S.D. (1994). SF-36 Physical and mental health summary scales: A user’s manual. Boston: Health Assessment Lab.

 

Ware, J.E., Snow, K.K., & Kosinski, M. (2000). SF-36 health survey: Manual and interpretation guide. Lincoln, RI: Quality Metric Incorporated.

 

Ware, J.E., Jr., & Sherbourne, C.D. (1992). The MOS 36-Item Short-Form Health Survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30(6), 473–483.

 

Willener, R., & Hantikainen, V. (2005). Individual quality of life following radical prostatectomy in men with prostate cancer. Urologic Nursing, 25(2), 88–90, 95–100.

 

Wilson, R.W., Hutson, L.M., & VanStry, D. (2005). Comparison of 2 quality-of-life questionnaires in women treated for breast cancer: The RAND 36-item Health Survey and the Functional Living Index-Cancer. Physical Therapy, 85(9), 851–860.

 

World Health Organization. (2002). Cancer pain relief and palliative care. Geneva, Switzerland: Author.

 

Clara Granda-Cameron, RN, MSN, CRNP, AOCN®, is a coordinator of the pain and supportive care program in the Joan Karnell Cancer Center at Pennsylvania Hospital in Philadelphia; Sara R. Viola is an undergraduate researcher in the Department of Molecular Biology at Princeton University in New Jersey; Mary Pat Lynch, MSN, CRNP, AOCN®, is an administrator in the Joan Karnell Cancer Center at Pennsylvania Hospital; and Rosemary C. Polomano, RN, PhD, FAAN, is an associate professor of pain practice in the School of Nursing at the University of Pennsylvania in Philadelphia. No financial relationships to disclose. (Submitted February 2007. Accepted for publication July 23, 2007.)

 

Author Contact: Rosemary C. Polomano, RN, PhD, FAAN, can be reached at polomanr@nursing.upenn.edu, with copy to editor at CJONEditor@ons.org.

 

Digital Object Identifier:10.1188/08.CJON.65-77