ABSTRACT
Introduction
The questionnaire
Test-retest reliability study
Validity analysis
Discussion and conclusion
Acknowledgements
Author: M. E. MEDINA-MORA, S. CASTRO, C. CAMPILLO-SERRANO, F. A. GÓMEZ-MONT
Pages: 67 to 76
Creation Date: 1981/01/01
This study was carried out to ascertain the validity and reliability of data generated through an internationally developed self-administered questionnaire. Data were collected from two student populations in which prevalence of drug use was known (high and low prevalence rates) and information obtained from the questionnaire checked through personal interview. The questionnaire covered both demographic characteristics and drug use items. A total of 474 students were tested and 335 students (70.7 per cent) retested. Although percentages of missing data and inconsistent responses were high, the questionnaire appeared valid and reliable when analysed on a group basis. It was less reliable on an individual basis. In conclusion, it is considered that with minor modifications the questionnaire can be used in the populations studied with enough confidence in validity and reliability to allow comparisons.
The most common approach to the epidemiological study of drug use among students has been the self-administered questionnaire survey. Self-administered questionnaires are used with the assumption that they will be honestly answered by students because they are anonymous. However, insufficient studies have been made on this subject. The issue has been addressed in some developed countries (Smart, 1975; Whitehead and Smart, 1972), but there are (to the author's knowledge) no published reliability and validity studies of survey instruments used in developing countries. Surveys and census activities are less common in such countries and the population is not as accustomed to questionnaires.
This issue is of more than academic interest because without some idea of the quality of data generated in these surveys, it is not possible for planners to take due consideration of their findings or to assess accurately data on youthful drug use trends which are made available to them.
Smart (1975) reviewed the validity and reliability of studies of self-reported drug use conducted in the period 1970-1974 in Canada, England and the United States. He found numerous validity studies but only one reliability study using high-school students and few using any other populations. In this reliability study made by Haberman in 1971, a high degree of consistency was found for all drugs with no more than a 4 per cent difference between the two estimates. However, this study apparently did not match individual responses at two points in time.
Numerous surveys of drug use among Mexican students have been made in recent years (Lafarga, 1972; Cabildo and others, 1972; Carranza-Acevedo and others, 1972; De la Fuente, 1972; Wellisch and Hays, 1974; Castro and Chao, 1976) but only one addresses the reliability and validity of data found or the measurement instruments used. This unpublished pilot study by Chao and Castro, 1975, found the same rate of amphetamine use when questionnaire results were compared with urine test examinations of the same students. Despite promising results, the pilot nature of this study does not permit general conclusions on the reliability and validity of drug use survey data on Mexican students.
The present study is the first which examines the validity and the test-retest reliability of drug use questions in a developing country with an internationally developed self-administered questionnaire, where variation is assessed in both individual and group responses. It includes two validity studies. In the first, data were gathered from two populations in which prevalence of drug use was known (high and low prevalence rates). In the second, two data-collecting procedures were used on the same population: self-administered questionnaire and interview. The validity was tested in terms of whole group characteristics rather than on a person-by-person basis.
A questionnaire was developed for use in a variety of countries and it is described in detail in Smart and others, 1980. The questionnaire was originally developed in English and was translated into Spanish independently by two individuals. The most common "street names" of the different pharmacological agents were added to the questionnaire. A translation back into English was made by a third translator whose mother tongue was English and who was not familiar with the original English version; this did not show important differences.
The questionnaire included questions on ( a) demographic characteristics such as: age, sex, education, urban or rural residence and parents' education;
drug use during the students' lifetime, the past 12 months and the past 30 days, and age when drugs were first used for the following drug types: tobacco, alcohol, cannabis, amphetamines or other stimulants, cocaine, hallucinogens, solvents, tranquillizers, sedatives, opium, heroin and other opiates; ( c) whether the students would answer honestly if they had used cannabis, opium or heroin.
Twelve groups of students were chosen from seven high-risk schools: secondary, 7 to 9 years of schooling, both urban ( n=99), and rural ( n=45); preparatory, urban, 10 to 12 years of schooling ( n= 128) and two urban university departments, philosophy and arts ( n = 63). The high-risk schools were known to have more drug users in comparison with other schools as indicated by the results of a previous survey (Castro and Chao, 1976). In addition, the Subsecretaria de Educación Publica de México was consulted. A total of 474 students were tested and 70.7 per cent of them were retested and gave a test-retest sample of 335 students. Two groups of 73 (15.4 per cent) students were not retested because of the official holiday period between semesters. A further 66 (13.9 per cent) were not present on the day of the second visit. Students did not know in advance the day of the retest.
Data were collected on both occasions in a group setting (classrooms of 40 to 70 students) with an interval of four weeks. Students did not sign their names on questionnaires. However, in order to ensure comparability in both phases of the study, each subject was assigned a number that was shown on both questionnaires and referred to a list of students' names. The questionnaires were handed to the students according to the list. They were informed that it was important to know which of them were present in order to use the same people on a second occasion. They were also assured that no names would be used and that data would be analysed on a group basis.
The above procedure was followed in 10 groups. In the other 2 groups each student was asked to identify himself in both test and retest with any three letters of the alphabet. The students were asked to use the same set of letters on both occasions. Only two questionnaires (out of 63) could not be matched.
Forty-one subjects (12 per cent) were excluded from the reliability analysis due to inconsistencies (four or more inconsistent responses, additional responses, non-legitimate blanks or a combination of all three). Test and retest data were analysed separately and when either questionnaire showed inconsistencies both questionnaires were removed from the sample.
One quarter of the subjects attended rural schools where it was expected that the comprehension level would be lower. However, the highest percentage of inconsistent replies was observed at the higher levels of education (10 to 12 years of school) and in the higher age levels. Part of the inconsistency can be interpreted as a lack of co-operation rather than a lack of understanding of the questionnaire.
Few drug users were excluded from the analysis due to inconsistencies, except in the case of use of stimulants (43 per cent, n = 7) and of barbiturates (60 per cent, n = 5) where there was a large proportion of users who replied inconsistently. Reported use of hallucinogens, opium and heroin observed were consistent.
The remaining students were aged 11 to 21 or over with most aged 13 or 14 (35 per cent). Both sexes were equally represented. About 42 per cent had completed 7 to 9 years of school and 35 per cent from 10 to 12 years. About 74 per cent were full-time students and 56 per cent had not worked during the previous year.
Percentages of drug use ever obtained in both test and retest were examined. Small differences were found between both applications. The difference was around 1 per cent for most drugs except in the case of tranquillizers where a difference of 3 per cent was observed, and of alcohol where there was a 2 per cent difference between both applications. The only cases that reported opium and heroin use in the test did not report use in the retest.
Substance |
Test (%) |
Retest (%) |
---|---|---|
Tobacco
|
44.7 | 45.6 |
Alcohol
|
42.9 | 41.1 |
Cannabis
|
5. 1
|
4.8 |
Stimulants
|
2.0 | 2.7 |
Cocaine
|
1.2 | 2.1 |
Hallucinogens
|
0.3 | 0.3 |
Solvents
|
1.5 | 2.7 |
Tranquillizers
|
6.9 | 3.9 |
Barbiturates
|
1.5 | 0.6 |
Opium
|
0.3 |
-
|
Heroin
|
0.3 |
-
|
Other opiates
|
-
|
-
|
Other drugs
|
1.5 | 1.5 |
The following items were examined in this analysis; ( a) those with no answer in both applications; ( b) where students responded identically across
Validity and reliability of a high school drug use questionnaire 71 both administrations; ( c) where there was no answer in one of the applications, either test or retest; ( d) where inconsistencies were found, defined as different responses between test and retest.
In the analysis of missing data it can be seen that percentages are higher in those items related to drug use (ranging from 16.3 to 21.4 per cent) than in those related to demographic items (ranging from 0.7 to 8.5 per cent).
Item |
No answer in both test and retest |
Answer provided in test and missing in retest |
Anwer provided in retest and missing in test |
Total |
Fraction of n (%) |
---|---|---|---|---|---|
Sex
|
0 | 7 | 2 | 9 | 3.1 |
Age
|
0 | 1 | 1 | 2 | 0.7 |
Years of school completed
|
1 | 0 | 4 | 5 | 1.7 |
Level of urbanity
|
0 | 1 | 6 | 2.0 | |
School status
|
0 | 2 | 1 | 3 | 1.0 |
Work status
|
7 | 8 | 10 | 25 | 8.5 |
Father's education
|
0 | 1 | 4 | 5 | 1.7 |
Mother's education
|
0 | 1 | 1 | 2 | 0.7 |
Tobacco
|
10 | 20 | 18 | 48 | 16.3 |
Alcohol
|
11 | 20 | 17 | 48 | 16.3 |
Cannabis
|
11 | 27 | 11 | 49 | 16.7 |
Stimulants
|
13 | 30 | 13 | 56 | 19.1 |
Cocaine
|
17 | 27 | 11 | 55 | 18.7 |
Hallucinogens
|
17 | 30 | 16 | 63 | 21.4 |
Solvents
|
16 | 26 | 8 | 50 | 17.0 |
Tranquillizers
|
15 | 28 | 9 | 52 | 17.7 |
Barbiturates
|
18 | 29 | 12 | 59 | 20.1 |
Opium
|
16 | 32 | 14 | 62 | 21.1 |
Heroin
|
14 | 27 | 9 | 50 | 17.0 |
Other opiates
|
17 | 26 | 10 | 53 | 18.0 |
Listed substances refer to their life-time use.
Inconsistent responses to demographic variables ranged from 4.8 to 29.6 per cent. Answers to questions about age, years of school completed, and about home location were the least consistent. The range of inconsistent responses was higher for demographic items than for drug items (ranging from 0.3 to 10.5 per cent).
Students were asked on both tests, for each of 12 substances including alcohol and tobacco: ( a) if they had ever used the substance; ( b) if they had used it in the past year; ( c) if they had used it in the past month; ( d) their age at first use. The range for consistent reporting for use ever of one or more of these substances was 73 to 84 per cent (mean, 80 per cent). For the question whether they had used drugs in the past year, the range was between 70 and 78 per cent (mean, 73 per cent). For the question on use of any of these substances in the past month, the range of consistent replies was between 61 and 78 per cent (mean, 73 per cent). For the question on age at first use of these drugs, the range of consistent responses was between 61 and 77 per cent (mean, 72 per cent). Alcohol and tobacco turned out to be the least consistent items in all four questions.
Item |
Number |
Fraction of n (%) |
---|---|---|
Sex
|
14 | 4.8 |
Age
|
86 | 29.3 |
Years of school completed
|
87 | 29.6 |
Level of urbanity
|
67 | 22.8 |
School status
|
50 | 17.0 |
Work status
|
57 | 19.4 |
Father's education
|
48 | 16.3 |
Mother's education
|
42 | 14.3 |
Tobacco
|
31 | 10.5 |
Alcohol
|
28 | 9.5 |
Cannabis
|
7 | 2.4 |
Stimulants
|
6 | 2.0 |
Cocaine
|
3 | 1.0 |
Hallucinogens
|
0 | 0.0 |
Solvents
|
4 | 1.4 |
Tranquillizers
|
13 | 4.4 |
Barbiturates
|
1 | 0.3 |
Opium
|
1 | 0.3 |
Heroin
|
1 | 0.3 |
Other opiates
|
0 | 0.0 |
a. Listed substances refer to their life-time use.
Missing values on either application were responsible for 59 to 75 per cent of the inconsistencies found in the answer to alcohol and tobacco use and from 91 to 100 per cent of the inconsistencies found in the replies on the other drugs studied.
These findings suggest that the questionnaire was not as reliable as hoped for. The percentage of missing data and inconsistent responses was high, and missing data on drug questions were particularly high. However, in other studies using a self-reporting questionnaire in Mexico (Chao and Castro, 1975; Castro and Chao, 1976), it was found that missing data were characteristic of answers from student populations.
Validity and reliability of a high school drug use questionnaire 73
Why do students leave questions unanswered? Is it because the questionnaire is unreliable, or do they simply feel distrustful about answering questionnaires in general?
If missing data are a function of reliability our concern should be to improve the instrument, but if missing data stem from student distrust our job is much more difficult. The fact that percentages of missing data were lower for demographic items than for drug items seems to point to distrust, perhaps because of the topic. If we take into account that global percentages of drug use obtained are consistent with those found in other surveys of Mexican school populations, then we might conclude that although we have under-reporting, the responses might be reliable.
The results also show the need to make individual comparisons in test-retest studies. In this study variations in reported drug use are small (3 per cent being the biggest difference). Nevertheless, this method of comparison does not show such problems as missing responses and individual variations which may be sources of unreliability.
The questionnaire described above was also administered to a sample of 47 students known from previous studies (Salinas and Gómez, 1975) as a group with a high proportion of drug users. This sample was selected from a delinquent student population, i.e. minors convicted of different criminal offences. The sample included persons aged 14 to 18 years, of both sexes, from low socio-economic levels and with 3 to 5 years of completed schooling. In addition, the questionnaire was administered to a sample of 49 students known as a population with a low frequency of drug use (Castro and Chao, 1976). Both groups were similar in the four demographic characteristics considered, except that one group were prisoners and had different expected rates of drug usage.
The prevalence obtained from the two populations was in the expected direction with the "high prevalence" group reporting far more use of all drugs than did the low prevalence group.
As described above, the questionnaire was administered to a high prevalence prison population. Respondents did not identify themselves on the questionnaire. Two weeks later, 43 of the 47 original subjects were interviewed; the other four could not be interviewed because they were no longer in prison. The interviewer read each question from the self-administered questionnaire and recorded the subjects' answers on the form.
The prevalence rates obtained through the two methods were similar. A chi-square analysis showed no significant differences between them.
Substance |
Group with prevalance (n=47) |
Group with low prevalance (n=49) |
---|---|---|
Tobacco
|
79 | 12 |
Alcohol
|
72 | 12 |
Cannabis
|
77 | 2 |
Stimulants
|
34 |
-
|
Cocaine
|
13 | 2 |
Hallucinogens
|
11 |
-
|
Solvents
|
51 | 4 |
Tranquillizers
|
26 | 2 |
Barbiturates
|
30 | 2 |
Opium
|
6 | 2 |
Heroin
|
6 |
-
|
Other opiates
|
11 | 2 |
Substance |
Self-administered questionnaire |
Interview |
---|---|---|
Tobacco
|
79 | 86 |
Alcohol
|
72 | 74 |
Cannabis
|
77 | 63 |
Stimulants
|
34 | 26 |
Cocaine
|
13 | 12 |
Hallucinogens
|
11 | 19 |
Solvents
|
51 | 58 |
Tranquillizers
|
26 | 26 |
Barbiturates
|
30 | 21 |
Opium
|
6 | 7 |
Heroin
|
6 | 9 |
Other opiates
|
11 | 10 |
From an analysis of the replies regarding self-reported honesty (Would the respondents "feel free to admit marijuana and heroin use if they were users of these drugs"?) the following findings are noteworthy: 81 per cent of the respondents answered that they would feel free to admit marijuana use, 14 per cent gave a negative response and 3 per cent did not reply. The 17 students who said that they had used cannabis at least once gave a positive answer. About 78 per cent of the respondents answered that they would feel free to admit heroin use, 16 per cent gave a negative response and 6 per cent did not reply. The only student who said that he had used heroin at least once gave a negative response to this question.
The percentages of students who reported that they would feel free to answer are high for both drugs considered. Cannabis users were consistent in their answers to both questions. Unfortunately, the low rate of heroin users does not permit any conclusion to be drawn about self-reported honesty related to the use of this drug.
It is clear that the questionnaire turned out to be highly valid and reliable when analysed on a group basis. However, when the analysis was done on an individual basis, data showed less reliability. Demographic variables were less consistent than drug items. This fact stressed the need to revise and adapt such items to the Mexican milieu.
Drug use items were most consistent for "use ever", somewhat less consistent for "use in the past year" and still less consistent for "use in the past month". Such differences were expected because of the period of time between the two questionnaires (four weeks). "Age at first use" presented the lowest consistency rate.
Missing values turned out to be an important source of unreliability and were more frequent in drug items than in demographic variables. The avoidance of skip patterns might reduce the rate of no answers. Nevertheless, further research should be done in this area and the data suggest that distrust is an important factor.
In spite of the difficulties the results are promising. From Mexican School populations the validity data from these studies indicated an acceptable instrument. With minor modifications this internationally developed instrument can be used in the populations studied with enough confidence in validity and reliability to allow comparisons. In view of the problems that differences in languages and cultures can raise, a pretest of the questionnaire in other countries would be useful in order to assess variations in individual responses and ensure comprehension of the instructions and wording of the instrument. The use of local or previously proved validity control scales such as lie scales or social desirability scales to assess the validity of the information is also recommended.
Self-administered, anonymous questionnaires are less costly and time-consuming than other techniques; an important issue, especially in developing countries. In this survey no significant differences in rates of drug use reported were found between the method of self-administration and personal interview. This finding supports the use of the first method in this culture. The advantage of the second method is that it reduces the rate of no answer but if this problem can be lessened through additional controls, the self-administered questionnaire is the recommended approach.
The questionnaire was developed by the WHO Research and Reporting Project on the Epidemiology of Drug Dependence. Acknowledgement is given to the World Health Organization and the United Nations Fund for Drug Abuse Control for their support, and to Dr. R. Smart, Addiction Research Foundation, Toronto, Canada, and Dr. P. H. Hughes, WHO Project Manager, for their comments on the methodological aspects of this study.
A methodology for student drug use surveys. By R. Smart and others. No. 50. Geneva, World Health Organization, 1980.
Castro, M. E. and Z. Chao. Encuesta nacional sobre consumo de drogas en población estudiantil entre 14 y 18 años de edad. CEMEF internal report. México, 1976.
Chao, Z. and M. E. Castro. Epidemiología y actitudes en la población escolar; estudio piloto. CEMEF internal report. México, 1975.
De la Fuente, R. El problema de la farmacodependencia. Gaceta médica de México, 103:101-123, 1972.
Farmacodependencia en estudiantes de enseñanza media del D. F. Paper presented at the Consejo Nacional de Problemas en Farmacodependencia. By J. Carranza-Acevedo and others. Quoted in Farmacodependencia(Mexico), 4:1 - 16, 1972.
Haberman, quoted by R. G. Smart. Recent studies on the validity and reliability of self-reported drug use, 1970-1974. Canadian journal of criminology and corrections, 17:326-333, 1975.
Investigación sobre el use de sustancias intoxicantes entre menores y jóvenes del Distrito Sanitario XVI. By H. Cabildo and others. Revista de salud pública de México, 1:14:17-58, 1972.
Lafarga, J. Encuesta sobre las actitudes del estudiante en una universidad privada de México, ante el uso de drogas. Comunidad, VI:35, 1972.
Salinas, V. O. and C. A. Gómez. Correlación entre el estudio epidemiológico en fase piloto y el estudio epidemiológico realizado en el Tribunal para Menores y Escuelas de Orientación en el Distrito Federal. Cuadernos cientificos (CEMEF), 2:59-71, 1975.
Smart, R. Recent studies of the validity and reliability of self-reported drug use, 1970-1974. Canadian journal of criminology and corrections, 17, 326-333, 1975.
Wellisch, D. and J. R. Hays. A cross-cultural study of the prevalence and correlates of student drug use in the United States and Mexico. Bulletin on narcotics, 26:1:31-42, 1974.
Whitehead, P. C. and R. G. Smart. Validity and reliability of self-reported drug use. Canadian journal of criminology and corrections, 14: 1-7, 1972.