Primary Phase - Final Report
The Mental Health Performance Measures Pilot Project is a joint effort of the Alaska Mental Health Board and the Division of Mental Health and Development Disabilities (DMHDD) that began in February 2001. The goal of this project is to develop standardized outcome measures to be used at all state-funded mental health treatment facilities in Alaska. To finalize these questionnaires, DMHDD contracted with the Alaska Comprehensive and Specialized Evaluation Services (ACSES) to serve as independent consultants to conduct a pilot project on draft questionnaires. The pilot project has been approached in two phases, a preliminary phase and a primary phase. The work of the Preliminary Phase, to collect clinician and consumer feedback about the ease of use, structure, and utility of the instruments, has been completed and results were submitted to the Oversight Committee. The goal of the Primary Phase, to evaluate the sensitivity of the revised versions of the instruments to consumer change across time, has been achieved and results are presented in this report.
Reports Prepared by ACSES and Submitted to the Oversight Committee:
Three instruments were used in the Preliminary Phase, namely, the Client Assessment Worksheet, Mental Health Consumer Satisfaction Survey, and Mental Health Statistical Improvement Program Consumer Survey (MHSIP). Based on feedback from clinicians and consumers, the Oversight Committee made additional revisions to these instruments after the Preliminary Phase. The resultant instruments are the Client Status Review, Demographics Questionnaire, and the MHSIP.
Client Status Review
For the Primary Phase, the revised Client Assessment Worksheet was renamed the Client Status Review. Appendix Seven provides the original Client Assessment Worksheet; Appendix Eight provides the Client Status Review used in the primary phase of this pilot project.
For the Primary Phase, the revised Mental Health Consumer Satisfaction Survey was renamed the Demographic Questionnaire. Two versions of the Demographic Questionnaire were developed, an Adult Services version and a Child and Family Services version. Appendix Nine provides the original Mental Health Consumer Satisfaction Survey; Appendix Ten provides the Adult Demographic Questionnaire; Appendix Eleven provides the Youth Demographics Questionnaire.
Mental Health Statistical Improvement Program Consumer Surveys
Three versions of this instrument were used, namely, Adult Survey, Youth Services Survey, and Youth Services Survey for Families. Appendix Twelve provides the original Adult MHSIP. The original youth surveys consisted of 26 items; subsequent revisions (by the national group that is developing and refining this instrument) have resulted in 21-item questionnaires. Appendix Thirteen provides the original Youth Services Survey; Appendix Fourteen provides the Youth Services Survey for Families.
Global Assessment of Functioning
In addition to administering these questionnaires, clinicians were asked to provide a current Global Assessment of Functioning (GAF) for each client. GAFs were collected to provide an established measure of client functioning against which to compare the overall CSR results. It should be noted that GAFs are not precise measures; however, clinicians are trained in their use and they do provide some data for comparison purposes.
Prior to data collection, an ACSES staff member visited each of the participating agencies to provide training regarding timing and procedures, including information on research design, data submission procedures, and confidentiality issues. Prior to administering the instruments, clinicians were to ask consumers if they would be willing to help with the project by answering the questionnaires, providing opinions about the questions, and making recommendations about possible changes.
Time One Data Collection
At Time One, after receiving consumer consent, clinicians administered the CSR in a structured interview format to as many consumers as possible over a one-month time period (later extended to last six weeks), and asked the same consumers to complete and return the Demographic Questionnaire and MHSIP independently. The one-month time period was selected to allow for missed appointments, to allow inclusion of consumers who are on a once-a-month schedule, and similar circumstances. To match Time One and Time Two responses, questionnaires were coded with consumers’ ARORA or agency identification number (no names were used anywhere on the research protocols). Due to two agencies getting started late, the data collection period was extended for an additional two weeks to maximize data collection opportunities; all sites were notified of and invited to participate in this change of timeframes. Actual Time One dates differed from administration site to administration site and are provided in the individual agency descriptions below.
Subsequent to completing the CSR with the consumer, clinicians handed the consumer a packet containing the Demographic Questionnaire and the MHSIP to complete independently and gave them a brief overview of the tasks involved. A cover letter from ACSES to the consumer was included in the packet to explain the purpose of the project and the procedures used to insure confidentiality and anonymity. On behalf of ACSES, clinicians also provided stamped, self-addressed envelopes to the consumers, who had the option either to mail the questionnaires directly to ACSES or to seal the envelope and give it to an agency staff member (e.g., a receptionist) who would mail it for them. Consumers were also given the option of asking clinicians to help them complete the questionnaires if they needed assistance. Due to the low rate of return by mail from consumers during the Preliminary Phase, clinicians were asked to encourage their consumers to exercise their right to give feedback about the services they receive and, if possible, to complete the questionnaires while at the agencies and return them in sealed envelopes to a staff person to be mailed for them.
Time Two Data Collection
Time Two administrations were conducted four to six weeks after Time One, following the same procedures outlined for Time One and using the same questionnaires. During Time Two, clinicians administered the questionnaires to the same consumers who responded during Time One. Any consumers who completed the questionnaires during Time One and were discharged before Time Two began, were to be administered the questionnaires the second time at their time of discharge. Ideally, consumers were to complete the questionnaires during a termination session. However, if the consumers did not attend a termination session, the clinicians were asked to mail all three instruments to the consumers to complete and return in a pre-paid, self-addressed envelope to ACSES. Time Two data collection began on July 16 and was completed for all agencies on August 13, 2001.
Due to the low participation rate at several of the sites, as reported in the Interim Report, the Oversight Committee requested that more Time One data be collected. This was particularly important for gathering rural consumer and clinician feedback, as well as additional input from child and youth consumers. Arrangements for additional Time One data collection were made; agencies did not collect Time Two data for those consumers.
Based on community size, region, willingness to participate, number of consumers serviced, and informed consent procedures in place, six pilot sites were selected by DMHDD to participate in this pilot project. These agencies were as follows: Southcentral Counseling Center, 4Rivers Counseling Service, Alternatives Community Mental Health Center, Bristol Bay Mental Health Center, LifeQuest, and Norton Sound Behavioral Health Service. A brief description of their level of participation during the Primary Phase follows.
Staff training for the Time Two administration phase was conducted with the clinical associates on July 5. Time Two data was collected from July 16 through August 13. It is notable that throughout this pilot project participation at this agency was higher than at any of the other agencies. This may be attributable to the Executive Director’s strategy to solicit volunteers, set a minimum level of participation, and offer monetary and compensatory incentives to each staff participant who achieved the requisite participation level. SCC staff members are to be commended for their efforts in this project.
4Rivers Counseling Service, McGrath
Staff training for Time One of the primary phase was conducted on May 4 with the Executive Director. The primary clinician at this site continued to have strong personal objections to the overall project and called in sick on the day of the training. However, on the same day as the training, an ACSES staff member observed this clinician to be in good health on the flight from Anchorage to McGrath. The Director prepared detailed written instructions for the clinician outlining the administration process and a directive that she was to participate by administering the questionnaires to her consumers according to the research design procedures.
The clinician called an ACSES staff person stating that, because of to 4Rivers policies and procedures, the clients’ bill of rights, and her ethical practice, she could not administer the surveys to clients aged 13-18 without written informed consent of the parents. She reported that this was consistent with her need to obtain written informed consent from parents each time she traveled to her assigned villages to provide mental health services. She expressed a great deal of concern that asking her clients to complete the questionnaires was not part of their treatment plans and, therefore, outside the realm of professional practice. She also reported being unable to administer the questionnaires to children and youth because she saw these consumers in their school environment and therefore did not have contact with the parent/primary caregiver to obtain informed consent. The ACSES staff person notified the Executive Director of the clinician’s request for an informed consent form and the need to coordinate this request with DMHDD (as other agencies had successfully done). The informed consent form developed for this project was provided to the Executive Director. Anne Henry, DMHDD, also talked with the clinician to impress upon her the importance of gathering information from her client population. The clinician stated that she was participating; however, she did not schedule further trips to the villages during Time One administration and was on vacation for the month of June.
The Director prepared a short introduction letter from his agency to explain the project to consumers and to emphasize the importance of rural input into the project. Data collection was scheduled to begin on May 7 and to be completed on June 15. Because the Director was out of McGrath for most of the Time One administration period and the clinician did not administer any questionnaires, no data was received from this site. The lack of data from Time One precluded the agency’s participation during Time Two administration.
Alternatives CMHC, Anchorage
Staff training for Time One of the primary phase was conducted on May 10 with clinicians and case managers, with much discussion about the administration process. Staff wanted to administer the questionnaires to their consumers aged 18-21 who were involved with the Division of Juvenile Justice. For this group of consumers, the adult questionnaires were used, with the clinicians noting the consumer’s age and custody status on the forms.
The Utilization Review Manager expressed the need for an informed consent prior to administering the questionnaires. Although it was explained that the consent form developed for the Preliminary Phase was sufficient, the manager remained firm in the need for a different consent form. This issue was referred to Executive Director to resolve with DMHDD. A staff person noted that the Division of Family and Youth Services (DFYS) has its own regulations about consumers participating in this type of project. The Director decided that they would contact DFYS about giving them a general consent for their consumers to participate in the project, which might eliminate the need for youths to complete the consent forms. Developing and coordinating another consent form with DMHDD and other projects with which the agency was involved led to delay in data collection. Data collection started the week of May 21 and was completed on June 15.
Time Two staff training was conducted on July 10, and data were collected from July 16 through August 13. Staff also collected additional Time One data during this phase from consumers who had not yet participated in the project. During the Primary Phase, the agency experienced significant staff turnover that negatively affected staff’s ability to collect an adequate sample size of consumers. The Executive Director and the Utilization Review Coordinator remained interested, cooperative, and enthusiastic about the project, and did their best to collect data.
Bristol Bay Mental Health Center, Dillingham
Staff training for Time One of the primary phase was conducted on May 1, with data collection for adult consumers starting on May 2 and scheduled to end May 30. The Executive Director requested that staff administer the questionnaires to child and youth clients as well. However, no child or youth surveys were administered because the assigned clinician felt the CSR was not age-appropriate. In some of the villages, village-based family social workers were to be asked to administer the questionnaires to their clients, instead of agency staff. A ‘village code’ number on the form would identify this. Time One data collection was conducted from May 8 through June 15.
Staff training for Time Two was held on July 9 with data collection occurring from July 16 through August 13. Staff also collected additional Time One data during this phase from consumers who had not yet participated in the project.
Staff training for Time One of the primary phase was conducted on May 3 with case managers and the Medical Services Team; staff training for outpatient clinicians was conducted on May 7. The Medical Services Team opted out of administering the questionnaires due to the ‘non-billable’ nature of the service and concerns that the data would not show improvement unless administered at intake, since the majority of their consumers were already stabilized on medications. Data collection by case managers began on May 7; outpatient clinicians started data collection on May 8. Data collection was completed on June 15.
Time Two staff training was conducted on July 12 with case managers and outpatient clinicians, with data collection scheduled to begin on July 16 and to be completed on August 13. When no data had been received from the agency by August 27, an ACSES staff member contacted one of the managers coordinating the project at LifeQuest. This individual reported being unsure what had happened, stating that the departure of the administrative person coordinating the project and staff’s busy schedules were the most likely contributing factors to the lack of data. The manager reported that staff was ‘so pushed’ with other demands that she had not asked them to do more. The Executive Director, however, stressed the importance of gathering the Time Two data to the managers and the data collection period was extended until September 14 to permit staff to re-administer as many surveys as possible. This deadline was extended again until September 21, when data had still not been received. Due to the need to establish a cut-off date that would permit adequate time to evaluate the data, the deadline was changed to September 19, and an ACSES staff person traveled to Wasilla to pick up any data that had been collected.
Norton Sound Behavioral Health Services (BHS), Nome
Staff training for Time One of the primary phase was conducted on May 15. For the first week during the administration period, no surveys were administered because most of the staff was attending a local training conference. In some of the villages, staff planned to ask the village-based counselors to administer the questionnaires. Data was collected from May 21 through June 15.
Time Two staff training was conducted on July 13, with data collection occurring from July 16 through August 13. Staff also gathered additional Time One data during this phase from consumers who had not yet participated in the project. Despite the Executive Director’s continued interest and enthusiasm about the project, the agency’s participation rate continued to be lower than expected. This significantly limited the rural input that was so important to the project.
Findings are presented in three sections: 1) staff observations, which outline observations made by ACSES staff regarding the level of cooperation and enthusiasm about the pilot project among participating agency staff; 2) quantitative results, which provide data derived from the questionnaires; and 3) qualitative results, which summarize written comments made by consumers and clinicians about the questionnaires. Individualized reports regarding consumer satisfaction at Time One were prepared for three agencies for which adequate data were available. These reports were distributed directly to these agencies and are not part of this report.
Clinicians’ responses to participating in the pilot project were obtained from direct observation by an ACSES staff member during staff training and from direct comments by clinicians. As in the Preliminary Phase, responses ranged from enthusiastic acceptance of the project to significant resistance. Staff at most of the agencies were professional and cooperative. However, at one agency, some staff did not even look at the questionnaires during the training; at another agency, one clinician avoided the training by calling in sick. All agency directors continued to be very cooperative and interested in participating. Specific levels of staff interest and cooperation at the various agencies can be inferred from the response rates contained in Table One.
Several agencies reported difficulties obtaining consumer participation for Time Two administrations. Reasons cited included not being able to contact consumers by phone, consumers being unwilling to meet with the clinician, and consumers being on extended vacations or absences from home during late summer months or relocating out of state. At two agencies, staff turnover and workloads were reported as obstacles to completing the Time Two administrations. Another concern had to do with language, with some staff members reporting that, for many of their clients, English was a second language, creating a significant barrier to data collection. Another concern expressed by some staff members was that administering the instruments could have a negative impact on therapeutic relationships. These individuals suggested that an independent party, rather than agency staff, should administer these types of questionnaires. Finally, several expressed concerns that the State would cut funding if consumers did not show improvement over the two administrations.
Response and Match Rates
Table 1 provides the number of instruments completed at Time One and Time Two, broken down by agency. Across all agencies, 150 completed CSRs were submitted to ACSES for Time One and 69 CSRs were submitted for Time Two. In addition to the 150 CSRs returned at Time One, 21 CSRs were collected out of phase with no plans for a Time Two administration; these 21 CSRs were included in Time One analyses. Of the total 171 Time One CSR protocols, 37.4% (n=64) were submitted by SCC, 33.9% (n=58) by LifeQuest, 16.4% (n=28) by Bristol Bay, 7% (n=12) by Norton Sound, and 5.3% (n=9) by Alternatives.
The length of time between Time One and Time Two administrations ranged from 37 to 135 days, with a mean of 72.6 days (SD=26.9). Based on ARORA numbers provided on the CSR, it was possible to match 67 Time One (n=150) and Time Two (n=69) CSRs, representing an overall match rate of 44.7% (67/150). Two of the Time Two CSRs included ARORA numbers that did not match with any Time One ARORA numbers. It is unknown whether this means that two consumers were administered Time Two protocols without having received Time One protocols, or whether ARORA coding was incorrect. The five participating agencies varied in the overall match rate between Time One and Time Two as follows: Southcentral Counseling Center, 75% (48/64); Bristol Bay, 23.5% (4/17); Life Quest, 22.4% (13/58); Norton Sound, 20% (1/5); and Alternatives, 16.7% (1/6).
For Time One, 106 Demographic Questionnaires and 106 MHSIPs were received from consumers. For Time Two, 54 Demographic Questionnaires and 54 MHSIPs were received. An additional 15 Demographic Questionnaires and MHSIPs were received for and used in subsequent Time One analyses. These additional protocols had no matching Time Two data. Of these received Time One forms, 112 were adult versions and 19 were youth versions; of the 54 Time Two forms, 53 were from adults and 1 from a youth. On the consumer-completed questionnaires, to insure confidentiality of responses, no means to match Time One and Time Two information was requested. However, by comparing responses to demographic items, it was possible to match 42 Time One and Time Two Demographic Questionnaires and MHSIPs, for an overall match rate of 39.6% (42/106).
Based on the place of administration of the 171 Time One CSRs, it was possible to determine that 76.6% of the participants were from urban areas and 23.4% were from rural areas. Unfortunately, of the 67 matched CSRs, only 7.5% (n=5) were received from rural areas. Table 2 provides Time One and Time Two consumer and clinician responses for the adult version of the Demographic Questionnaires. Based on Time One data, 51 (45.5%) were female, 46 (41.1%) were male, and 15 (13.4%) gave no response. The majority of adult respondents were in the 23-59 years old category (83.9%, n=94). Of these consumers, 58 (51.8%) were Caucasian, 40 (35.7%) Alaska Native/American Indian, five (4.5%) African American, three (2.7%) Hispanic/Latino, two (1.8%) Asian/Pacific Islander, one (0.9%) did not know, and three (2.7%) provided no response. Of the consumers, 67% indicated receiving case management services, 76.8% psychiatric services and medications, 33.9% transportation services, and 19.6% housing. These figures suggest that the typical consumers who participated in this pilot study were persons with persistent and severe mental illness.
Table 3 provides Time One and Time Two consumer responses for 19 youth and child version Demographic Questionnaires. Based on Time One data, the majority of minor respondents were in the 6-12 years old category (63.2%, n=12). Of the respondents, 13 (68.4%) were Caucasian, three (15.8%) Alaska Native/American Indian, one (5.3%) African-American, and two (10.5%) Other.
Client Status Review
On average, clinicians reported that it took 10-30 minutes to administer the CSR, with the most commonly reported timeframe being 10-15 minutes. A few clinicians reported taking longer, depending on the client’s intellectual level or diagnosis. One clinician noted that ‘scheduling time to meet with the clients was more of a problem’ than administration time.
Of the 171 CSRs received during Time One administration, five questionnaires were missing the consumer’s ARORA or agency number. Most of the missing ARORA (or agency) numbers were later obtained by an ACSES staff member; however, this took additional staff time. Almost three-quarters of the Time One instruments (70.8%, n=121) were returned without the Sum of Ratings section completed. Of the 50 CSRs returned with the Sum of Ratings scored, eight (16.0%) were scored incorrectly. Of the 69 CSRs received during Time Two, all included an ARORA or agency number. Of these 69 Time Two CSRs, 18 (26.1%) did not provide calculated Sum of Ratings. Of the 41 returned with the Sum of Ratings scored, three (7.3%) were scored incorrectly.
Initial review of the CSR data revealed some missing responses, particularly on the five components of Question #6. At Time One, 7.0% of protocols had at least one of the 10 responses missing; at Time Two, 4.3% had at least one item missing. Table Four provides the specific number of missing responses for each CSR item. As the Sum of Ratings is calculated based on responses to all 10 items, any missing data will yield a Sum of Rating that is not indicative of the consumer’s actual level of functioning. An alternative would be to calculate a Sum of Rating based on the average score to the responses provided; this would then yield a score that is comparable to other Sum of Ratings received from the same or other individuals. However, scoring the questionnaire in this matter is more complicated and prone to error.
Table Five provides Time One and Time Two responses for the 67 matched CSRs. Table Six provides means and standard deviations for each CSR item and the Sum of Ratings for Times One and Two. As a way of reminder, on the CSR lower scores indicate higher level of functioning. Thus, the desired change is one of lower scores at Time Two as compared to Time One. Thus, a difference score (typically calculated as Time Two minus Time One) that is negative indicates improvement, whereas a difference score that is positive indicates worsening. Table Six provides these mean difference scores and results of t-tests comparing scores at Time One and Two. For the 10 primary items, four had negative (improved) difference scores means (with difference scores ranging from -.03 to- .25), five had positive (worsened) difference score means (with difference scores ranging from .01 to .15), and one had equal means (i.e., a difference score of 0) at Time Two as compared to Time One. The Sum of Ratings was negligibly higher (worse) at Time Two than Time One. Results of the t-tests revealed that the differences between Time One and Time Two failed to reach statistical significance on all individual items and the Sum of Ratings.
To look at change from Time One to Time Two for each item and the Sum of Ratings in yet another way, Table Seven provides the number of consumers who, at Time One versus Time Two, had either the same, a lower, or a higher score. Table Seven also provides results of McNemar’s change tests (Siegal, 1956), used to determine whether changes in responses to these items increased or decreased in a consistent manner across consumers from Time One to Time Two. For all of the individual items, the highest number of respondents had no change on the individual items. For those individuals for whom change was reported, on a majority of the items the change was in the direction of improvement (i.e., a decrease in item score). However, when considering the Sum of Ratings, the proportion of consumers who had the same scores was very low; the proportions who had an increase or decrease in functioning was nearly identical.
Results of McNemar’s tests revealed significant decreases in scores (improvements of functioning) on four items, namely, “How often can you currently get the physical health care that you need?”, “Have friends/relatives asked you to cut down on alcohol or other drugs or to quit entirely?”; “Have you experienced guilt because of your drinking or use?”; and “Do you need an eye opener in the morning to get started?” Note that for the three drinking related items, decreases (improvements) meant that the individuals stated “yes” at Time One and “no” at Time Two.
In terms of overall functioning, at Time One, the Sum of Ratings ranged from 8 to 34, with a mean of 19.4 (SD=4.3) and with lower scores indicating better functioning. GAFs obtained at Time One ranged from 21 to 95, with a mean of 48.6 (SD=12.0) and with higher scores indicating better functioning. The correlation between the Time One Sum of Ratings and GAF scores was -.37. At Time Two, the Sum of Ratings ranged from 14 to 42, with a mean of 19.5 (SD=5.0). GAFs obtained at Time Two ranged from 21 to 80, with a mean of 42.3 (SD=10.8). The correlation between Time Two Sum of Ratings and GAF scores was -.22. These levels of correlations suggest that the GAF and Sum of Ratings measure different aspects of consumer functioning.
Table Eight provides Time One and Time Two responses to the adult version of the MHSIP. Table Nine provides means and standard deviations for each MHSIP item and the Domain Subscales for Times One and Two. Table Nine also provides the mean difference scores and results of t-tests comparing Times One and Two. Of the means for the 28 items, 20 were lower at Time Two than Time One (indicating higher levels of satisfaction), seven were higher (indicating lower level of satisfaction), and one was the same. Of these differences, t-tests revealed statistical significance for only one item, namely, “Encouraged to use consumer programs”, with greater satisfaction being expressed with this item at Time Two. Of the four subscales, all were somewhat lower at Time Two than Time One, indicating slightly higher levels of satisfaction at Time Two; however, the differences were not statistically significant.
Table Ten provides the number of consumers who had the same, lower, or higher scores at Time Two as compares to Time One for each item and the domain subscales. Table Ten also provides results of McNemar’s change tests (Siegal, 1956), used to determine whether changes in responses to these items increased or decreased in a consistent manner from Time One to Time Two. For most of the items, the highest number of respondents had no change on the individual items. For those individuals for whom change was reported, on 22 of the items, the change was in the direction of greater satisfaction (decrease in score); on four of the items, change was in the direction of lesser satisfaction, and on two items the number of consumers whose satisfaction decreased and increased were the same. Results of the McNemar’s tests revealed significant changes in scores on only four items, all in the direction of greater satisfaction: “Staff were willing to see me as often as I felt necessary”; “Encouraged to use consumer-run programs”; “I do better in school and/or work”; and “My housing situation has improved”. When considering the four domain subscales, the majority of consumers reported slightly higher levels of satisfaction at Time Two than Time One, although these differences did not reach statistical significance.
Table Eleven provides Time One responses to the Youth Services Survey for Families. No Time Two data were provided on this instrument. Table Twelve provides Time One and Time Two responses to the Youth Services Survey. Given that only one Time Two questionnaire was collected, no further analyses were conducted.
At the request of the Oversight Committee, additional analyses were conducted to assess whether rural and urban respondents differed from one another in their responses to the instruments. Additionally, analyses were requested regarding potential differences between consumers of Alaska Native heritage as compared to those of other ethnic backgrounds. The rural versus urban comparisons were based on location of data collection and hence were possible for all instruments. Ethnicity was not assessed on the CSR, making such analyses impossible for this instrument. As ethnicity information is collected on the Demographic Questionnaires, it was possible to conduct analyses on ethnicity for the MHSIP and Demographic Questionnaire.
Rural versus Urban Analyses: Initial review of the Demographic Questionnaire revealed that consumers from urban and rural agencies differed significantly on a number of critical variables related to type of clientele served. Specifically, urban consumers represented in the current sample were more likely than their rural counterparts to report that they are:
· receiving psychiatric services/medications (91% urban; 56% rural);
· receiving case management (86% urban; 25% rural)
· receiving long-term care (2 years or more; 72% urban; 34% rural)
· non-Native versus Alaska Native (79% urban; 39% rural)
These findings strongly suggest that the urban and rural samples included in this project were not equivalent, but rather were comprised of significantly different consumers. Specifically, it appears likely that the urban sample was comprised of a relatively homogeneous group of consumers who received intensive long-term care and who were likely to represent a persistently and severely mentally ill population. The rural sample appeared to be comprised of a much more heterogeneous group of consumers. Further, rural versus urban location of assessment was highly correlated with ethnic background, making it impossible to tease out the different effects of these two ways of grouping the sample.
Given the dissimilarity of the urban versus rural sample and the confounding of location with ethnicity, additional analyses are not appropriate (thus, although they were calculated, they are not reported here, lest they be misunderstood as usable and appropriate). For example, if differences were revealed in urban versus rural responses, it would be impossible to ascertain whether these differences were caused by urban versus rural factors, ethnic backgrounds, or type of consumer. To address questions regarding location in the state and ethnicity, future data collection must incorporate careful stratified sampling procedures.
A complete listing of the qualitative comments provided by clinicians and consumers is provided in Appendices One to Six, categorized by instrument and site. Following is a summary of the most commonly expressed concerns, provided separately for each questionnaire. This summary should not be used in place of reading the comments. It is evident that the clinicians and consumers put a lot of thought into their comments and a careful review of the comments is crucial to the credibility of any additional instrument revision or replacement effort. Not surprisingly, given that all clinicians and many consumers had encountered the questionnaires previously, most consumer and clinician responses to the follow-up questions about the questionnaires themselves were brief, and significantly fewer responses were obtained than in the Preliminary Phase.
According to clinicians, consumers were willing to complete the questionnaires in the structured interview format and most who were asked to participate did so voluntarily. One consumer who declined to participate did so because of ‘too much asking and too much paperwork’ (as recorded by the clinician). One rural agency reported that three consumers declined to participate. One clinician offered this general comment about all of the questionnaires: ‘Many of the participants requested help with the forms, which were created with the intention for independent completion from the consumers. Keeping this in mind, I think it is important to plan the wording carefully to encompass the wide range of education barriers that many people struggle with.’ A number of clinicians noted that consumers needed clarification on many questions. At one rural agency, clinicians noted that about one-half of their consumers needed help completing the Demographic Questionnaire and MHSIP. They also noted that most consumers did not want to complete the forms while at the agency, preferring to take and mail them later. Also of note were concerns expressed by consumers about inadequate staffing, overworked staff, and Medicaid funding cuts that have decreased services and make filling out additional forms a significant hardship.
Client Status Review: Overall Comments
Appendix One provides clinician and consumer comments in response to the CSR. Several consumers expressed positive opinions about the process: ‘I enjoy the questions. I support this questionnaire. Good job.’ However, a number of consumers also expressed negative comments about the CSR, such as ‘I didn’t like intrusive questions; I don’t think these questions are necessary; don’t like nosey people.’ One consumer commented that ‘the questions assuming substance abuse and problems with the law are offensive.’
A significant number of clinicians and consumers commented that the CSR was not appropriate for children and youth, and recommended developing a separate questionnaire that is more applicable to a younger consumer population. Comments in this regard included ‘doesn’t apply to consumer’s age’, ‘none applied’, ‘not worded for parents to answer for a child’, and ‘should make a form for the parent specifically to answer for the child.’ Several clinicians noted having to reword questions, using consumers’ names, since they were requesting the information from the consumers’ parents. The overall recommendation from consumers and clinicians alike was to develop a separate questionnaire for child and youth consumers structured in such a manner as to be completed by a parent or primary caregiver.
One rural provider commented, representatively so, in conversation that rural services differ significantly from urban services in that they tend to be much more short-term (one or two sessions), crisis-focused, or intensive in nature. Systemic interventions at the family and community level are more common and will not be captured by the current CSR. This provider suggested alternative means for assessing the success of rural mental health programs, not focusing on individual clients, but focusing on community perceptions and use of the local mental health agency.
The majority of consumers reported that the CSR questions were neither hard to answer (64.2%), nor unclear (68.9%). One consumer suggested adding the question, ‘What do you feel has been helping the most?’ Another suggested asking, ‘Is treatment confidential (private)?’, while noting the lack of privacy in a case manager’s office. Another consumer indicated a ‘Need to ask if [services are] adequate after hospitalization.’ Clinician comments included adding questions about client satisfaction and complaints about medications and services, treatment results over time, and lack of improvement and its reasons. One clinician indicated that the questionnaire should have asked more about mental health. Another clinician asked, ‘What are you attempting to determine?’
Specific CSR Item Comments
Following is a summary of comments made regarding specific CSR items. The reader again is urged to read the detailed comments in the Appendix and not to rely only upon the summaries.
Comments about CSR Question One: On Question 1 (regarding activity), one consumer commented ‘If you weren’t doing one of those things, you’d be dead.’ Along the same line, one clinician noted that the response ‘not active’ makes it appear ‘as if the consumer never does any activity’, suggesting ‘It may be better to state a choice such as rarely active, minimally active, etc.’ Another asked ‘level of activity, how defined for a home-schooled child?’ Another consumer described the question as ‘nebulous.’ There were several clinician comments that adding agency group activities should be part of meaningful activity since this was ‘an active area’ for many consumers.
Comments about CSR Question Two: Several consumers commented that Question 2 (regarding physical health problems) was difficult to answer or unclear due to the wording and being ‘unsure about physical.’ One consumer attributed weight problems that negatively affected her mental health as the factor that kept her from doing normal activities.
Comments about CSR Question Three: For Question 3 (regarding physical health care access), a number of consumers noted services that they needed, such as eye and/or dental care. One consumer wrote ‘I need dental care and just filled out a grant through the dental program.’
Comments about CSR Question Four: Numerous comments about Question 4 (regarding payment for health care) indicated that consumers had difficulty choosing between ‘fully insured’ or ‘well-insured’ because Medicaid and Medicare were not identified and do not cover all services. Other consumer comments included ‘confusing; Medicaid & Medicare needed to be specified.’ One consumer asked ‘Is health care the same as mental health care?’
Comments about CSR Question Five: There were several comments about including crisis respite services on Question 5 (regarding hospitalization for mental health care) since consumers had used this type of service with the past six months.
Comments about CSR Question Six: Question 6 (regarding drug and alcohol use) received the most criticism, with a recurrent theme from both clinicians and consumers being that the question presumed a substance use problem. One consumer captured the predominant feeling about this question: ‘I don’t drink and you should have asked if I did before you implied that I needed to answer these questions.’ Other consumer comments included: ‘I felt it was intrusive; assumes I have drug/alcohol problem; where to indicate no longer using? You assume I’m alcoholic; I have 3 years clean and sober and nowhere to note it; Don’t drink!’ Clinician comments echoed those of consumers: ‘#6 continues to be a poorly worded question as many of those who participated do not use drugs or alcohol and felt this question assumed that they did; Consumers find it difficult to differentiate whether the question is pertaining to past or present use; Need to ask whether client drinks/drugs.’ There also seemed to be some confusion on the part of consumers about how to answer the questions. One clinician commented that a consumer had answered ‘yes’ to the question about feeling guilty because of one relapse with alcohol use. Another consumer answered ‘yes’ to ‘are you annoyed by friends or relatives who question your use?’ and wrote in ‘annoyed by other people pressuring (me) to use.’ There were also a number of comments wondering if caffeine and tobacco were considered drugs. One clinician noted that response option one had ‘weird phrasing – double negative.’
Comments about CSR Question Seven: On Question 7 (regarding financial situation), one parent indicated that the child has Medicaid, but the parent herself or himself faces ‘extreme financial hardship.’ One clinician reported that a consumer had problems with this question, stating ‘Kids questioned if this relates to their finances or their parents’?’
Comments about CSR Question Eight: On Question 8 (regarding housing), several consumers wrote in ‘pay rent’ as a clarification to ‘live with others.’ One consumer was unsure about being independent or not, commenting ‘I’m very dependent on various agencies.’ One clinician stated that the question ‘continues to not address transitional housing’ and asked ‘Does this fit into the homeless category?’ A clinician commented that ‘the parents own; the kids live with their parents. Question doesn’t make sense.’
Comments about CSR Question Nine: For Question 9 (regarding general safety), consumers’ comments indicated that mental health issues, rather than environmental factors, primarily determined their feelings of safety. One parent added an explanation to response option three to indicate that the child is fearful of being teased or picked on. Several consumers noted feeling fearful or paranoid as a result of their symptoms or their own actions, acknowledging that there were no identifiable external threats. Several clinicians noted that consumers felt safe and were aware of suffering from paranoia. Only one consumer referred to environmental factors, stating ‘It’s hard to feel safe when hearing about community violence that may not directly affect me.’
Comments about CSR Question Ten: One clinician had several comments for Question 10 (regarding involvement with police, court, jail) about the redundancy of ‘legal issues pending’ and ‘felony charges pending’, and the need to clarify terms, such as ‘extreme impact’ and ‘non-lock-up facility.’ Another clinician commented that ‘non-criminal involvement’ was unclear terminology. One clinician added the comment ‘unknown, if any’ regarding the consumer’s legal involvement.
Demographic Questionnaire (Adult, Child and Families)
Of the consumers who completed a Demographic Questionnaire and MHSIP, 95.5% returned them directly to ACSES via mail. The vast majority of consumers (60.1%) reported that they understood the questions and that they were clear (61.5%). Only two consumers stated that the questionnaire was too long.
Appendix Two provides comments on the adult version of the Demographics Questionnaire; Appendix Three provides comments on the youth version. Many consumers took the opportunity to express appreciation for staff and services they receive, as well as to identify some problem areas. Comments from a number of consumers suggested the need to ask open-ended questions regarding how they felt about the quality of the services. Suggestions were as follows:
· Do you need more help?
· Are you happy with your counselor or doctor?
· Are you happy with the programs?
· How are the meds working?
· If you don’t like your counselor, what are your options?
· Can you choose providers?
· Does the system adequately help you completely control your mental illness?
One consumer wondered if “… this [is] a test of our competence?’ Another consumer stated ‘the purpose isn’t clear…’ A few consumers noted that the questionnaire did not ask everything that they thought was important about mental health services and that questions should be added, although they did not elaborate.
Two respondents completed one questionnaire each for two children receiving services (completed by biological parent and adoptive parent for one child; completed by a biological parent and a foster parent for the other child). On the question asking about guardian, conservator, or payee, several consumers wrote in the name of the person, instead of selecting the job title of the person.
Several consumers selected or wrote in more than one response for the question about ethnicity. When this occurred, the minority ethnicity was recorded in the data tables. One clinician recommended using the new United States Census Bureau categories for the ethnic group.
For the question regarding services, one clinician noted that ‘housing, transportation, employment and advocacy services are not applicable’ in rural location. Several clinicians noted that consumers did not know what ‘advocacy’ meant. Some consumers were unclear that group therapy or group counseling were included in the ‘therapy/counseling’ choice, as indicated by their addition of the name of specific treatment groups in the ‘Other’ category. A couple of consumers selected ‘Other’ and wrote in comments, such as ‘new client’ or ‘just started’.
The question concerning frequency of services received the largest number of write-in consumer responses, suggesting the need to more clearly define the answer options. One consumer selected two responses: ‘daily’ and wrote ‘occasionally’ in the ‘Other’ category. Consumers wrote a variety of responses in the ‘Other’ category (e.g., ‘for years’, ‘varies’, ‘3-5 days/week’, ‘3x a week’, ‘4x a week’, ‘rarely’, ‘5 days/week’). One consumer selected three responses, commenting that ‘some services are monthly, others may be weekly – to lump them together makes it more difficult to answer.’
Over three-fourths of the consumers found the questions understandable and clear (77.4%) and inclusive of issues that they thought were important to ask about mental health services (61.9%). Appendix Four provides the comments provided by consumers in response to the adult version of the MHSIP. A few consumers responded that the questionnaire did not ask everything that is important about mental health services, but did not elaborate on what was missing. Of note were consumer comments regarding the need to ask about other agency staff and about billing techniques. Only three respondents felt that the questionnaire was too long. One consumer asked ‘Will I get the outcome of this survey?’ and wondered ‘if this survey helps in funding.’ Another consumer suggested adding a question about ‘What programs or areas of your life would you like more help in…?’ Another consumer commented ‘This questionnaire is a good tool if your ears and hearts are open and compassionate.’ A clinician reported continuing to have to provide assistance in understanding the questions and clarifying the wording to simplify questions to many of the consumers. The clinician reported that ‘many of the participants requested assistance with the Demographic and MHSIP survey questionnaires which were intended for independent completion from the consumers. Keeping this in mind, I think it is important to plan the wording carefully to encompass the wide range of education barriers that many people struggle with so that, in the event this form is completed independently, it will be easy to follow and answer.’ One clinician reported that many of the consumers read the word ‘options’ in Question 2 as ‘opinion’ and needed assistance with the question. The suggestion was to change the wording of the question to ‘choice’ or ‘preference.’ Several consumers noted that Question 17 was difficult to understand and unclear, with one consumer commenting that ‘both therapist and I decided treatment goals’ and another ‘difficult to know if you want the end results or the actual therapy/participation.’ Question 19 was also identified as problematic for several consumers, but no elaboration was provided.
MHSIP (Youth and Families)
The majority of respondents (70.6%) thought the questionnaires were understandable, clear, and inclusive of issues that were important to ask about mental health services. None reported that the questionnaires were too long. Appendix Five provides consumers’ comments in response to the Youth Services Survey for Families; Appendix Six provides consumers comments in response to the Youth Services Survey. However, few comments were provided by consumers. One consumer suggested adding the question ‘what kind of services do you feel were lacking?’ One consumer noted the ‘high turnover in manpower’ and having been ‘treated disparately’ at an agency. One youth consumer selected N/A for 18 of the 26 responses. Another youth consumer selected two responses for the question asking if staff were sensitive to cultural/ethnic background, writing in the comment ‘1 person’ for Agree and ‘all others’ for Disagree.
Two sets of recommendations will be offered. The first set focuses on recommendations for revising the instruments; the second set focuses on recommendations for the next phase of pilot testing.
Assuming that DMHDD and the AMHB wish to revise the CSR, Demographic Questionnaires, and the MHSIPs to incorporate Primary Phase feedback from consumers and clinicians, a number of recommendations follow for such revisions. These recommendations were developed based upon consumers’ and clinicians’ feedback.
Client Status Review
· Develop a separate questionnaire that is age-appropriate for child/youth consumers
· Question 1: Answer 5: change to ‘Rarely active’
· Question 2: Answer 1: change to ‘Rarely’
Answer 5: change to ‘Almost always’
· Question 3: Reword parenthetical description to read “Does not include dental and eye care, and is defined as care received from your regular, local, or visiting health care provider”
· Question 4: Specify in the question whether health care refers to mental or physical health, or both
Answer 1: change ‘ Fully insured’ to ‘free’
Answer 2: change ‘Well-insured’ to ‘Good insurance’; add ‘, such as Medicaid/Medicare/insurance co-pay’ to end of parenthetical sentence
· Questions 6: Given considerable missing data on this item, and clinician and consumer criticisms, consider a major rewrite of this item. One possibility is to rewrite question to be a more general query of substance use/abuse that highlights the need for further evaluation, rather than using a standardized assessment tool designed to access degree of SA
· Question 8:
Answer 1: delete ‘or I chose to live with others’
Answer 2: delete ‘family or others’
Answer 4: change ‘temporary’ to ‘transitional’ and add ‘temporarily’ to ‘staying with family/friends’
· Question 10:
Answer 3: change ‘legal issues now pending’ to ‘misdemeanor changes pending or conviction’
Answer 4: after ‘pending’, add ‘or conviction’; delete ‘or extreme impact’; change ‘contested divorce, contested custody issues’ to ‘contested divorce and/or custody issues’
Answer 5: change ‘non-lock-up facility’ to ‘half-way house’; add ‘Title 12’ before ‘mandatory’
· For all versions, incorporate the questions at the end of the MHSIP
· Question 2a: Change ‘If yes,’ to check boxes for guardian, payee, conservator
· Question 3:
Change stem to read: ‘Which one of …’
Use new US census ethnic categories
· Question 5, answer 1 – add ‘(includes individual & group)’
· Question 8, answer 1 – after ‘Daily’, add ‘(5-7 days/week)’
Demographic Questionnaire (Child & Family Services)
· Completed by: change ‘biological parent’ to ‘biological/adoptive parent’
· Question 2: change stem to read ‘Which one of …’
· Question 6: after stem, add ‘(Please select only one.)’
· Question 7, answer 1 – add ‘(includes individual & group)’
· Question 10, answer 1 – after ‘Daily’, add ‘(5-7 days/week)’
· Place demographic questions at end of MHSIP questions, similar to that of Youth & Family MHSIPs
· Include open-ended questions in the Adult version:
What has been the most helpful thing about the services you have received over the last 6 months?
What would improve services here?
· Use the revised 21-item versions of the Youth Services Survey and Youth Services Survey for Families
For any future investigations into the utility and psychometric characteristics of these instruments, we make the following recommendations:
1) The Oversight Committee may be interested in gathering feedback from staff members at mental health agencies around the state via focus group and key informant interview formats to identify problem areas and possible solutions that could help further refine the performance measures instruments for youth and rural consumers. It appears that rural providers perceive the instruments as less applicable to their clientele than urban providers, which has likely contributed to the resistance by rural providers to participating in this pilot project and may also lead to resistance to using the measures in future. The suggestion of rural providers to develop alternate means of measuring outcomes in rural areas in this regard appears particularly pertinent.
2) Using the same focus group and key informant interview formats suggested in item 1), it would be helpful to gather feedback from child and youth mental health care providers before beginning construction of a child/youth version of the CSR. Merely adapting the current CSR downward in age may not fully meet the needs and concerns of these providers, as completely different topics may need to be assessed to measure outcomes among a younger consumer population.
3) To gain further insight into the current questionnaires and any newly developed questionnaires (such as a child or rural version), administration of instruments should be repeated at a different set of agencies (or at different programs within agencies that participated this time). This re-testing phase is particularly important for determining the utility of new versions (youth, rural) in rural areas, with ethnically diverse populations, and with youth and adolescent populations. Stratified sampling (by primary diagnosis, ethnicity, gender, and rural/urban residence) may be necessary to receive definitive answers about the instruments’ utility across geographic locations, ethnic backgrounds, and types of clientele. Such stratified sampling would require extreme cooperation by agencies and may be difficult to realize.
4) This phase ideally would allow sufficient time to include a trial of reporting collected data back to clinical staff to let them see what types of summary reports can be expected based on the data collection. Such feedback would serve two purposes. First, it would demonstrate to clinicians that useful data comes back to the agency and that data do not just disappear into a “black hole.” Second, such reporting back would allow for double-checking whether the data are indeed useful in meeting their purpose of improving services. Clinicians could look over the data report to ascertain if the data would, in fact, help them improve the services they are currently providing to their consumers.
For future investigations into the statewide implementation of outcome measures, we make the following recommendations:
6) It appears necessary to collect more input from direct service provider about possible barriers, resistances, and solutions to the actual process of gathering and reporting outcome data (independent of what instruments are being used). It appears that rural versus urban staff members had different concerns and that different procedures (and perhaps even instruments) are needed to accommodate rural versus urban settings. To gather this input, agency staff and consumers could meet in (separate) focus groups to answer specific questions about the process of gathering and using outcome data. These focus groups could be combined with the focus groups mentioned in item 1) above.
7) Given concerns and the lack of clarity expressed by clinicians and consumers about the purpose of these instruments, it may help to use the terminology of “outcome” rather than “performance” measures.