Sampling problems: the influence of age and literacy
Psychometric artefacts: test selection and administration
Statistical problems: are the differences real?
Some constructive conclusions
Author: Jack FLETCHER , Paul SATZ
Pages: 29 to 34
Creation Date: 1977/01/01
The Egyptian study of chronic hashish users comprises the largest study of psychological function in chronic users to date. It is also alone among recent studies of chronic cannabis users in suggesting impairment in psychological function as a consequence of prolonged cannabis use. Unfortunately, the Egyptian study contains several sampling and psychometric shortcomings which limit conclusions made from the study. The present paper is addressed to a critical and constructive assessment of these methodological shortcomings so that the impact of the Egyptian study can be properly evaluated.
In a series of papers Soueif (1967, 1971, 1975, 1976 a, 1976 b) has presented the results of a large-scale study of chronic hashish use in Egypt. While the study examined many behavioural correlates of long-term cannabis use, including sociocultural phenomena, subjective mood effects, and attitudes towards cannabis, the focus of the present paper is the battery of psychological tests administered to cannabis users and non-users. The tests included a variety of measures assessing perceptual-motor skill (4 GATB subtests, Trail-Making Test, and Bender Gestalt), memory (Digit Span), reaction time (cards I, IV, V, and VI) from the Rorschach), and time and length estimation. Administration of these tests revealed significant differences favouring the control group on 3 GATB subtests, Trail-Making, Digit Span Backwards, and Bender Gestalt in "Copy" and "Recall" conditions. Users performed better than controls on distance and time estimation.
These results led Soueif (1971; 1976 a,b) to argue for a "differential association'' of "psychological function deficits" with chronic cannabis use. The nature of these deficits, it was argued, suggested impairment in "level of cortical arousal" attributable to prolonged cannabis use. Impairment varied, however, according to the literacy, age, and residence (urban v. rural) of the user. To explain the differential association of these deficits with socio-cultural variables, 3 hypotheses were produced and apparently confirmed:
The lower the level of literacy the smaller the size of the deficit;
The older the S the smaller the size of the deficit;
The more rural an S the smaller the impairment.
The Egyptian study is quite important, if only for the reason that of several more recent cross-cultural neuropsychological studies, only the Egyptian study has produced evidence for impairment in higher adaptive cortical functions attributable to prolonged cannabis use. Studies in Jamaica (Bowman and Pihl, 1973; Knights, 1975), Greece (Dornbush and Kokkevi, 1976), and Costa Rica (Satz, Fletcher, and Sutker, 1976) have uniformly failed to show any differences between chronic cannabis users and controls of measures of higher integrative adaptive functions. The present paper is addressed to a critical and hopefully constructive assessment of the Egyptian study in an effort to reconcile these apparently discrepant studies. We hope to show that the Egyptian study, despite its size, is hampered by sampling and statistical problems of sufficient magnitude to raise questions concerning conclusions suggesting the presence of brain dysfunction. In addition, an attempt will be made to offer constructive suggestions so that the Egyptian data can be reanalysed and applied to the important question of the long-term effects of chronic cannabis use.
In comparison with the NIDA funded Jamaican (N=58), Greek (N=48) and Costa Rican (N=82) studies, the Egyptian projects is much larger. The Egyptian study comprised the administration of 12 psychological tests to 850 hashish users and 839 non-users. All Ss were selected from Egyptian prisons, with the user group composed of all males convicted for hashish use and imprisoned during the period from June 1967 to March 1968. Controls were defined as "those subjects who never took hashish or any other narcotic substance" (Soueif, 1971, p. 17). The mean age of the user group was 39 years, as compared with 33 years for the control group. The distribution of urban/rural Ss and percentage of skilled labourers was approximately equal between the groups. However, 60 per cent of the user group and 55 per cent of the non-user group were illiterate. Forty-four of the control group possessed high school certificates and an additional 9 had some type of university degree. Only 6 users had a high school degree. These descriptive data suggest that the two groups differed on age and literacy, with the difference favouring the non-user group.
The difference in age and literacy between users and non-users may be quite important and the influence of these sampling factors is difficult to ascertain. The significance of these literacy differences is readily apparent when specific hypotheses concerning the relationship between literacy and cannabis use are examined. The first of Soueif's 3 hypotheses states that deficits in brain function are associated only with higher literacy levels. Although Soueif does seem to demonstrate this phenomena statistically, the emergence of these differences only for higher literacy levels may reflect nothing more than the greater number of high school (44) and university graduates (9) at the higher literacy level in the non-user group. Only 6 users had even a high school degree! The potential influence of literacy differences on the over-all direction of the results between the groups is highlighted by the fact that no significant differences were apparent when literacy levels favoured the user group (in the Upper Egyptian analysis; Soueif, 1971)! With a sample as large as that employed in Egypt, it is difficult to understand why appropriate matching on variables such as literacy (and age) was not done.
Other problems are apparent when one examines the age differences between user groups. Another relationship suggested by Soueif (1971; 1976 a,b) is that deficits are more likely to emerge with younger users. The importance of this relationship is difficult to comprehend. It implies that the more chronic the user, the less likely is the presence of a deficit! This conceptual difficulty emerges because chronicity (often defined as a period of intake of at least 10 years) is confined largely to an older age group. If the chronic effects of cannabis disappear with age, it is difficult to see how this finding reflects changes in brain function.
Claims made by Soueif about the relationship of age, literacy, and residence are particularly interesting to evaluate in light of other studies. Soueif (1976 a) argued that no differences are found when Ss are "illiterates, rurals and/or old" (p. 25) and uses this explanation to explain discrepant findings in Jamaica (Bowman and Phil, 1973). Such an argument receives no empirical support in at least one other culture. In Costa Rica (Satz, Fletcher and Sutker, 1976; Carter and Doughty, 1976) users were all urban (from San Jose), and with an average age of 28 years (range = 18-50). When the Costa Rican user groups were divided on the basis of age, no differences were apparent either between younger users and matched controls or older users and matched controls. However, sample size is much smaller in this study than for the Egyptian study.
Other serious sampling problems are apparent in the Egyptian study. For example, how representative is a prison population of the Egyptian culture as a whole? In the absence of relevant data, it may be that the results with this sample are not generalizable to the Egyptian population. In addition, consumption of other narcotics in the user group was uncontrolled (30 per cent used opium). Within this context, a higher percentage of users regularly consumed alcohol (22 per cent v. 9 per cent). These differences in sample characteristics, all of which favour the non-user group, suggest alternative explanations of the differences in test performance.
The possibility of additional psychometric and/or sampling artefacts operating in the Egyptian study is apparent in more recent papers by Soueif (1976 a,b). In particular, the question of floor effects arises. A floor effect occurs when a test is too difficult for the sample to which it is administered. If a test produces a floor effect, then potential between group differences may be masked because of the difficulty of the test. This problem is particularly salient when tests developed in one country are used in another without adequate standardization within the new culture (cf. Satz, Fletcher and Sutker, 1976). In the Egyptian study, Soueif (1976 b) professes bemusement at the fact that fewer differences are found between illiterate and older user groups. Rather than representing a relationship between cannabis use and general level of proficiency, however, these findings may represent a floor effect. The very low performance of illiterate Ss on the majority of tests is a case in point. Furthermore, it is difficult to understand why a deficit would be smaller in illiterate or older Ss unless a floor effect was present on some measures, especially when performance is inverselycorrelated with these tests.
The possibility of floor effects is quite important when the fact that over half the Egyptian sample is illiterate is considered. This again points to serious sampling biases in the Egyptian study. Indeed, in addition to washing out potential between group differences, the presence of floor effects in just certain segments of the sample (i.e. older or illiterate Ss) can increase Type I error rates. This is because performance is no longer adequately sampled across the entire spectrum of the potential population. Thus, the differences reported for younger, literate Ss might have been minimized had the tests adequately reflected performance of illiterate, older Ss. Again, the problem of literacy and age factors has not been sufficiently addressed in the Egyptian study.
Other problems are apparent when one examines test administration. It is not apparent that examiners were unaware of Ss group membership when tested. Indeed, separate interviewing schedules were developed and conducted for the two groups. The influence of expectancy effects on the direction of results in psychological research is well known (Rosenthal, 1966). The fact that certain of the psychological tests, particularly the Bender Gestalt and Rorschach, were scored according to subjective (often unreliable) methods and criteria makes this point more salient.
Problems of a statistical nature are apparent throughout the Egyptian study. In no place that we are aware of does Soueif present basic univariate t-test results between means for the entire user and non-user groups. While such findings would undoubtedly be distorted because of the absence of control over sampling biases in the two groups, it would be helpful to know the results of the t-tests so that more accurate assessment of the Egyptian data could be made. Soueif (1976 b) does present t-tests between median scores on the tests, ostensibly for the purpose of avoiding violations of sampling assumption underlying univariate tests. However, the sampling distribution of these tests is sufficiently robust so that deviations from sampling assumptions have little influence on test outcomes (Harris, 1975). If it is suspected that variances are different between the 2 groups, then tests should be made for this possibility.
Another source of difficulty arises in looking yet again at the age, literacy, and urban/rural analyses. The accuracy of these analyses is difficult to evaluate, for Soueif (1971; 1976) reports different degrees of freedom for each variable in these analyses. This fact, acknowledged by Soueif (1975), suggests that different Ss composed the analysis for each variable. Such a procedure makes difficult comparison of results on different tests, in as much as an essentially different set of Ss constituted groups for each statistical analysis. Certainly, on the basis of the information reported by Soueif (1971, 1976 a,b), it is difficult to establish the basis for these analyses and the degree to which his contentions are supported statistically.
An additional statistical problem concerns the considerable scientific and theoretical credence lent to the magnitude of the p-values used to establish statistical significance. It has long been known that the magnitude of a p-value increases as sample size becomes larger, such that small differences in means between groups are likely to be significant if the sample is sufficiently large (Neyman and Pearson, 1933; Bakan, 1971). With samples as large as those employed in Egypt, statistical significance is virtually ensured and therefore a moot point. Furthermore, as sample size is increased, Type I error rates become higher (Hays, 1963), increasing the probability of "significant" differences because of chance. This problem poses particular difficulty for the Egyptian study because of different group composition for each variable. More sophisticated regression analyses designed to measure the amount of variance between groups accounted for by each test and to control Type I error rates would add to the psychological significance of the Egyptian data. This is particularly important in view of the small mean differences between groups on some variables. These differences, while possibly statistically significant, might be explained in terms of sample size and the differences in age and literacy between the groups.
On a final note, the Egyptian study can be faulted on a number of methodological levels. While the sample size is much larger than any other recent cross-cultural study on chronic cannabis use, adequate sampling and statistical procedures that would take advantage of the resources apparently available for chronic cannabis research in Egypt were not employed. Nonetheless, the fact that the results of the Jamaican, Greek, and Costa Rican studies were not replicated in Egypt is clearly cause for concern. Can it be, as Soueif claims, that very large samples are needed to ferret out the subtle effects of cannabis on higher brain function? Because of the potential health hazards and legal implications associated with cannabis use (Hollister, 1971), additional work with the Egyptian data should be attempted. While some problems cannot be corrected, e.g. the apparent absence of a double blind test administration and the exclusive selection of prisoners in the sample, other problems can be dealt with. User and non-user Ss should be carefully matched subject by subject on age, literacy, residence and use of other drugs. Subjects on whom data are incomplete should be discarded. If medical records were obtained, Ss with a major disease syndrome (e.g. syphilis or epilepsy) not associated with cannabis use could be excluded. Appropriate univariate tests should be employed. More sophisticated statistical analyses could be conducted along with the more simple multiple univariate tests. For example, multivariate analyses of variance which would take advantage of the large sample and help control for Type I error rates (Hummel and Sligo, 1971) would be most appropriate. Linear regression techniques which employed as predictor variables age, literacy, and residency would assist in exploring the relationship of these variables with chronic cannabis use. Not only would such an approach make possible clarification of the nature of the Egyptian data, morespecific statements about the influence of cannabis use on behaviour would be possible. Because of the sample size, the Egyptian data is a potentially fruitful source of information pertaining to socio-cultural and psychological factors involved in long term cannabis use. The results of a more careful analysis can shed considerable light on this important issue.
Bakan, D., The test of significance in psychological research. In B. Lieberman (ed.), Contemporary problems in statistics, New York, Oxford University Press, 147-162, 1971.
Bowman, M. and R. Pihl, Cannabis: Psychological effects of heavy use, Psychopharmacologia, 29, 153-170, 1973.
Carter, W. E. and P. Doughty, Social and cultural aspects of cannabis use in Costa Rica. In R. Dornbush, M. Fink, and A. Freedman (eds.), Chronic cannabis use, Annals of the New York Academy of Science, 1976. In press.
Dornbush, R. and A. Kokkevi. Cognitive, perceptual, and motor performance in acute and chronic hashish use. In R. Dornbush, M. Fink, and A. Freedman (eds.),Chronic cannabis use, Annals of the New York Academy of Science, 1976. In press.
Harris, R., A primer of multivariate statistics, New York, Academic Press, 1975.
Hays, W., Statistics for psychologists, New York, Holt, Rinehart, and Winston, 1963.
Hollister, L., Marihuana in man: Three years later, Science, 172, 21-29, 1971.
Hummel, T. and J. Sligo. Empirical comparisons of univariate and multivariate analyses of variance, Psychological Bulletin, 76, 49-57, 1971.
Knights, R.M., Psychological test results. In V. Rubin and L. Comitas (eds.), Ganja in Jamaica, The Hague, Mouton, 111-120, 1975.
Neyman, J. and E. Pearson. On the problem of the most efficient tests of statistical hypotheses, Philosophical transactions of the Royal Society (A), 231, 289-337, 1933.
Rosenthal, R., Experimenter effects in behavioral research,New York, Appleton-Century-Crofts, 1966.
Satz, P., J. M. Fletcher and L. Sutker. Neuropsychological, intellectual, and personality correlates of chronic marihuana use in native Costa Ricans. In R. Dornbush, M. Fink, and A. Freedman (eds.), Chronic cannabis use, Annals of the New York Academy of Science, 1976. In press.
Soueif, M. I., Hashish consumption in Egypt, with special reference to psychosocial problems, Bulletin on Narcotics, XIX: 2, 1-12, 1967.
---, The use of cannabis in Egypt: A behavioural study, Bulletin on Narcotics, XXIII: 4, 17-28, 1971.
---, Chronic cannabis users: Further analysis of objective test results, Bulletin on Narcotics, XXVII: 4, 1-26, 1975 .
---, The differential association between chronic cannabism and psychological function deficits. In R. Dornbush, M. Fink, and A. Freedman (eds.), Chronic cannabis use, Annals of the New York Academy of Science, 1976( a). In press.
---, Some determinants of psychological deficits associated with chronic cannabis consumption. Bulletin on Narcotics, XXVIII: 1, 25-42, 1976 b.