Wrong Assessment, Wrong findings, Wrong Intervention
(The applicability of standardised measures to other cultures: case example of assessing individuals with an alcohol problem in Saudi Arabia)
Introduction
‘Standardised measures’ (SM) are tools or instruments that are pre-tested for their validity, reliability, sensitivity and specificity. They can be used by researchers and / or professionals to assess, screen or diagnose problems, events or people. The use of standardised measures in other cultures is widely recognised and acceptable. However, little attention has been given to their efficiency, applicability and / or the validity of their results when applied to different cultures. It is the aim of this paper to draw attention to the limitations of using what I call the cutting and pasting of standardised measures without giving enough consideration to their applicability to a given culture.
The misuse of SM can give rise to both methodological and ethical problems, as well as producing misleading results.
These problems can be avoided if cultural variables and differences are taken into account.
In an attempt to address the issue of applying SM to other cultures, some SM relating to alcoholism are presented and used as an example of how their use can be problematic when applied to Saudi Arabian society.
There are more than 70 standardised instruments that can be directly related to alcohol. These instruments can be used to screen, assess or diagnose (Cooney, Zw eben, & Fleming, 1995) many alcohol-related symptoms.
Some of the most popular measures are the Michigan Alcoholism Screening Test (MAST) (Selzer, 1971), Short Michigan Alcoholism Screening Test (mast) (Selzer, Vino kur, & van Rooijen, 1975), CAGE (Mayfield, McLeod, & Hall, 1974), Munich Alcoholism Test (MALT) (Feuerlein, Ringer, Kufner, & An tons, 1979), and Alcohol Use Disorder Identification Test (AUDIT) (Babor, de la Fuente, Saunders, & Grant, 1992). These tests are widely used for research and clinical purposes. Other alcohol-related instruments are available but they are less commonly used. Some of the standardized instruments are designed to measure one or more aspect of alcohol consumption in particular situations or types of population. Examples of these special instruments are the (TWEAK 1) (Russell et al.
, 1994), specifically developed for pregnant women, and Problem-Oriented Screening Instrument for Teenagers (POSIT) (Grunewald, Stewart, & Klitzner, 1990).
The term ‘standardized measures’ refers to those instruments, scales, inventories, questionnaires or tests that are used to measure certain objects or events (clients or problems) (Bloom & Fischer, 1982).
The development of an SM involves complex procedures that are meant to test the tool and assure the validity, reliability, sensitivity, specificity and interpret ability of the instrument. The validity of an instrument has to do with whether the instrument is really measuring what it is supposed to measure (Meenaghan, Powers, & Toomey, 1985). Its reliability, on the other hand, is concerned with how consistent and stable it is (Meenaghan et al. , 1985).
An SM needs be tested for its sensitivity and comparability (Bloom & Fischer, 1982). Its sensitivity is the extent to which it correctly detects positive cases, while its specificity is the extent to which it correctly detects other cases as not having the condition being tested (Miller et al. , 1995).
The validity of an instrument is usually based on a theoretical model that defines the problem to be measured (Berger & Patchner, 1988). The adoption of a theoretical model is determined by the developer of the instrument according to how he or she defines the problem to be measured (Berger & Patchner, 1988).
Thus we can conclude that an instrument is the result of the developer’s views, which are based on the theoretical model and the society in which it is generated.
Literature review
It is observed that many research studies have used western standardized instruments to measure problems in other cultures (e. g. Melendez, 1994). Some of these research studies claim to have achieved positive results and that the instruments they used were valid when applied to other cultures.
For example, Rubinstein (1995) used the Right-Wing Authoritarianism (RWA) scale to evaluate conservatism in Jewish people in Israel. The study findings confirmed the construct validity of the RWA scale with respect to Israeli culture. Another example is the comparison study conducted by Suzuki and Rance r (1994), who attempted to test the argumentativeness of college students in both the United States and Japan using two western scales, the Argumentative Scale and the Verbal Aggressiveness Scale. The authors concluded that both scales provided a reasonable overall fit to both samples, and that these scales had satisfactory construct validity for the Japanese sample.
In contrast, Dolan and Ford (1991) examined the validity of the Binge Scale Questionnaire (BS) and the Restraint Scale (RS) if applied to Arab culture, as in Egypt for example.
The authors’ findings suggested poor validity of the scales, especially when applied to Egyptian men. Moreover, Chattopadhyaya, Biswas, Bhattacharya, & Chattoraj (1990) administered the English form of the Eysenck Personality Questionnaire (EPQ) to 180 male and 124 female, educated, middle-class Bengalis to determine the applicability of the EPQ to this population. The authors found that, in contrast to English norms, the Bengali sexes did not differ significantly on any of the four EPQ scales. The study suggested limited applicability of the EPQ to Bengali culture.
Although some studies support the applicability of some standardised measures to other cultures, this is not the case for all standardised measures.
To clarify the problem, four western measures will be discussed. These are the Michigan Alcoholism Screening Test (MAST), CAGE, the Munich Alcoholism Test (MALT), and the Alcohol Use Disorder Identification Test (AUDIT). These scales are used to assess alcoholism, and they are believed to be objective and have high validity and reliability (Kinney & Leaton, 1991). In the following section, the validity of these measures in respect to Saudi Arabian culture is examined.
Michigan Alcoholism Screening Test (MAST) 2
Michigan Alcoholism Screening Test (MAST) (see appendix 4) is used to assess alcoholic problems in clients.
It consists of 24 questions to be answered ‘yes’ or ‘no’. The areas addressed by MAST include the person’s perception of his / her drinking behaviour, family problems related to alcohol consumption, loss of control, prior treatment, job impairment, problems with physical health, and the presence of legal difficulties (Kinney & Leaton, 1991). Scoring five points or more would categorise the subject as an alcoholic, four points would be suggestive of alcoholism, and three points or less would indicate the subject was not alcoholic.
Applying this instrument to Saudi Arabia may result in classifying many Saudis who drink occasionally or slightly (say once a month) as alcoholic. It might not yield any discrimination among Saudis who have experienced alcohol, no matter what their drinking behavior has been. A Saudi individual who drinks once a month would easily score more than five points on this scale, whereas an American, for example, who is in the same state, would score lower points.
For example, let us consider the following questions asked by MAST:
1. Does your wife, husband, parent, or other near relative ever worry or complain about your drinking?
2. Do you ever feel guilty about your drinking?
3. Has your drinking ever created problems between you and your wife, husband, a parent or other relative?
4.
Has your wife, husband, or another family member ever gone to anyone for help about your drinking?
A Saudi who drinks alcohol even slightly, might answer ‘yes’ to the above questions since the drinking of alcohol is religiously, legally, and socially unacceptable, and is considered deviant behaviour. Such behaviour could lead to family problems and even divorce. A family would seek help from available resources to solve the problem. The person concerned would be likely to experience a feeling of guilt since he or she would have done something contrary to social norms, the law and, above all, religion. This would be especially likely if the questions asked whether the feeling of guilt about drinking alcohol had ever occurred (i. e.
in the person’s entire life).
Another set of questions included in the instrument is related to the legal aspect of drinking. These are likely to be answered positively by Saudis since drinking alcohol is illegal in Saudi Arabia. For example, one of MAST questions asks if the respondent has ever been arrested for driving while drunk. In Saudi Arabia, however, the individual in such a situation would be arrested, not for driving while drunk, but for drinking whether driving or not.
Other questions of MAST are:
1.
Have you ever been seen at psychiatric or mental health clinic or gone to any doctor, social worker, or clergyman for help with any emotional problem, where drinking was part of the problem?
2. Have you ever gone to anyone for help about your drinking?
3. Have you ever been in a hospital because of drinking?
The expected answers to these questions, if asked in Saudi Arabia, are likely to be ‘yes’ because drinking alone (as opposed to problem drinking or alcohol dependency) is enough to get the individual into a treatment facility. Referral for treatment can be suggested by a family, an authority, an employer or any party to the act of drinking. Individuals who are caught drinking are directed or pressurised to enter an alcohol treatment centre even if they were drinking for the first time.
Also, what ever a Saudi’s level of drinking might be, it would cause him or her a lot of trouble either in the work place or in school.
Thus the answer might be a ‘yes’ if a Saudi respondent is asked a question like ‘Have you ever been in trouble at work or school because of drinking?’ .
In addition, the MAST instrument contains questions that do not apply or are inappropriate to be asked. For example, one of the questions asks ‘Do you feel that you are a normal drinker?’ Such a question is hard to be asked since any kind of alcohol drinking is abnormal in Saudi Arabia.
The CAGE Questionnaire
An even simpler tool for diagnosing alcoholism than MAST is the CAGE Questionnaire (see appendix 5).
CAGE offers a simple and rapid means of self-evaluation, which anyone can use to evaluate their own alcohol consumption. It can also be used by a physician or other health worker to identify excessive alcohol consumption, with the aim of prevention and early treatment of alcohol problems, and has proved effective as a screening tool for physicians in general practice (Sepp ” ea, M’eakel ” ea, & Sillanaukee, 1995). It consists of questions that focus on four main ideas: cutting down alcohol consumption, annoyance by criticism, guilty feelings associated with alcohol use, and eye-openers, or early-morning drinking. The title of the questionnaire was derived by extracting the first letter of the four main ideas and formulating the word CAGE.
According to Kinney & Leaton (1991), it has been tested for validity among a number of populations of possible alcoholics. Two or more positive answers would strongly suggest the presence of alcoholism (Mayfield, McLeod, & Hall, 1974).
If the CAGE test is applied to many Saudis who drink, in western standards, lightly, perhaps most of the responses would be positive answers. Take, for example, annoyance by others or feeling guilty because of drinking. The answers to these two questions might be ‘yes’ for a Saudi who consumes alcohol, however little. Consequently the test may provide incorrect and misleading results if applied in a Saudi context.
However, practical testing of the validity of these measurements in Saudi culture is needed.
It seems that neither measure is a valid gauge of alcoholism in respect to the Saudi culture. Part of the problem is that these measures do not directly measure the quantity and frequency of alcohol consumption in order to determine the existence of alcohol dependency but instead rely on indirect indicators. These indirect indicators or variables, however, are not valid in Saudi Arabia, especially if the instrument is being used to measure alcoholism or alcohol dependency. In other words, a Saudi could be asked if he / she has ever been referred to a treatment centre.
But this question (along with other questions) is not a good indicator of alcohol dependency. The subject could have been referred to a treatment centre only because he consumes alcohol.
Munich Alcoholism Test (MALT)
The MALT questionnaire (see Appendix 6) is a composite test which measures consumption, physical signs and symptoms, and biological abnormalities. It was developed by Feuerlein et al.
(1979), who selected 250 from more than 1000 items relating to alcohol that they found in the literature. The final form of the questionnaire makes use of 31 items which have been tested and validated both separately and together. The MALT test consists of two sections: the first, with seven items, requires clinical evaluation by a physician (MALT-P); the second is a questionnaire of 24 items to be filled in by the patient (MALT-S) (Feuerlein et al. , 1979).
According to the MALT test’s developers, the validity of the test is 0.
94 with a reliability of 0. 84. In addition, it is a cost effective and easy-to-use instrument for identifying alcoholic subjects in groups of all types of patients, and for confirming the diagnosis of alcoholism.
Each positive response in the physician’s section scores four points, whereas a positive response in the self-assessment section scores only one point.
Patients scoring six to ten, even in the absence of clinical symptoms, are suspected of alcoholism or having an alcohol problem. It can be assumed that patients with a weighted score of eleven should be diagnosed as alcoholic (Feuerlein et al. , 1979).
In examining the applicability of the MALT test to Saudi alcohol drinkers, only one out of the 31 items can be said to be problematic. This item is Question 7 in the physician’s section: ‘Spouse, family members or good friends have sought help because of alcohol-related problems of the patient.’ Since drinking alcohol is a problem in itself, the respondent may answer ‘yes’ to this question even if the patient does not have alcohol-related problems. Having said this, the misinterpretation of this question is less likely to occur since the question is clearly asking about alcohol-related problems rather than alcohol drinking itself.
Apart from Question 7, which would require minor modification, the MALT test can be said to be more applicable to the measurement of Saudi alcoholics than the previous measures.
Alcohol Use Disorder Identification Test (AUDIT)
AUDIT (see appendix 2 A) is questionnaire developed as part of the six-country World Health Organisation (WHO) collaborative project on identification and management of alcohol-related problems (Babor, de la Fuente, Saunders, & Grant, 1992). By placing emphasis on heavy drinking and frequency of intoxication rather than on signs of dependency, it is designed specifically to detect problem drinkers rather than alcoholics. The questions refer to lifetime alcohol experiences as well as those in the past year, thus distinguishing between current and previous problems.
Its development in a broad range of cultures is thought to enhance cross-cultural validity (Haggerty, 1994).
It is a ten-item (three sub scales) screening questionnaire with three questions on the amount and frequency of drinking, three questions on alcohol dependence, and four questions on problems caused by alcohol. Responses to each question are scored from 0-4 and a total score of 8 or more is taken to indicate hazardous or harmful alcohol use (Conigrave, Saunders, & Rez nik, 1995).
According to Babor et al.
(1992) AUDIT is aimed at screening adult populations. It can be used in a variety of settings and programmes, including primary care, the emergency room, surgery and psychiatry. It can also be used on a variety of subjects, such as: driving while intoxicated (DWI) offenders; criminals in court, jail, and prisons; enlisted men in the Armed forces; and workers encountered in employee assistance programmes and industrial settings. In addition to clinical use, AUDIT can also be used in a variety of research projects and epidemiological studies (Saunders, Aas land, Babor, de la Fuente, & Grant, 1993).
AUDIT has been normed on heavy drinkers and alcoholics. Since it was first established, it has been widely adopted as a screening instrument (Conigrave et al.
, 1995). It has been shown that AUDIT detected a higher portion of problem drinkers than the short Michigan Alcoholism Screening Test (Barry & Fleming, 1993).
As the sensitivity and specificity of the instrument, AUDIT was reported to have a sensitivity of 92% and a specificity of 94% in detecting hazardous or harmful alcohol use in a sample of 913 people (Conigrave et al. , 1995).
Tested theoretically for possible use in Saudi Culture, the AUDIT questionnaire seems more promising than the measures discussed so far. However, to allow more promising results, a minor modification is needed to Question 10, which asks about relatives’ and friends’ concerns about the person’s drinking.
So, instead of asking ‘Has a relative or friend or a doctor or other health worker been concerned about your drinking or suggested you cut down?’ , a suggested alternative might be: ‘Has a relative or friend or a doctor or other health worker been concerned about your drinking or suggested you quit drinking?’ . The remaining questions are highly applicable to Saudi alcohol drinkers. However, as it has been said earlier, in order to reach a concrete conclusion, a practical testing of all of these measures is needed.
Discussion
The above examples are used to illustrate the issue of applying standardized measures to other cultures. Two things should be noted: first, that not all measures are invalid or inapplicable for cross cultural studies, and, second, that the first two examples are too extreme. They show clearly how misleading some questions are.
However, the situation is even more confusing because the unsuitability of some of the questions is less obvious in some cultures than in others…
Using an SM in another culture without paying enough consideration to its applicability and usability in this context can result in misleading findings and / or a wrong diagnosis, which in turn can lead to a wrong intervention or treatment. Another important issue that might emerge from such misuse is the ethical dilemma that can result from wrong findings. Ethics in research have been the subject matter of many disciplines including social science. Research ethics are codes that provide guidelines for the proper conduct of research (Gillespie, 1995). One important purpose of these codes is to protect the participant subjects, organisations, and society from potential risks which may result from research studies (American Psychological Association, 1990).
For example, it is unethical for a researcher or a professional to present wrong data about his subjects or clients. Such presentation is a violation of the code of ethics of most research and professional societies.
In conducting research studies, the researcher should make sure that the instrument he / she uses is truly measuring what it is supposed to measure and should give full consideration to the cultural differences that may be present. Different subjects may understand a question differently and, whether we analyse the data within or between cultures, we often uncover different interpretations of the question (Tri andis, 1990).