```Lecture 7
HO 7
-1-
Reliability,Validity and Data
We saw in the previous lectures that the quality of result of
research is –amongst other things – dependent on the quality of the
statistical analysis we perform to obtain such results. Statistical
analyses are however performed on data. Without data having been
collected, no statistical analysis is possible.
So, data collection is a central aspect of research.
Data that is to be collected however is the basis for all the
inferences we would eventually make. As such these data must be:
1. Reliable
2. Valid
CSCI 6960- Research Methods
Lecture 7
HO 7
-2-
Reliability,Validity and Data
We should remember that:
Reliability is the property of independent reproducibility of the
data.
Validity is the quality of a measure being an adequate and
acceptable representative of what it is supposed to represent.
We can not have validity without reliability, but a measure can be
reliable without being valid.
For example,measuring the number of new-line characters in a
program as a measure of its quality is reliable but certainly not
valid.
CSCI 6960- Research Methods
Lecture 7
HO 7
-3-
Reliability,Validity and Data
Therefore we must ensure that the data we collect are (amongst
other characteristics) both reliable and valid. To do so, we need to
pay special attention to all the steps leading up to data collection.
But particularly to the:
1. Formulation of the operational definition, and
2. The design of the procedure as to remove
threats to validity.
In other words, threats to reliability and validity are best handled
procedurally, and BEFORE data collection begins.
CSCI 6960- Research Methods
Lecture 7
HO 7
-4-
Reliability,Validity and Data
Ensuring reliability:
We mentioned three different types of reliability considerations:
1. Test-retest reliability
2. Inter-rater reliability
3. Internal-consistency reliability
We now must ensure that such reliability is evidenced in our
measurement by designing our procedures in such way that
provides such confidence.
CSCI 6960- Research Methods
Lecture 7
HO 7
-5-
Reliability,Validity and Data
The best course of action to take is to provide for each measure
as precise an operational definition as possible, and for all the
measurements, a precise operational procedure. Doing so
significantly increases the probability of scoring higher on testretest and inter-rater types of reliability evaluations.
Example:
The ISO15504 (SPICE) standard for software process
improvement and capability determination has a very precisely
devised and followed procedure for data collection. This is solely
to increase the inter-rater and test-retest reliability of the
measure.
CSCI 6960- Research Methods
Lecture 7
HO 7
-6-
Reliability,Validity and Data
Another way of increasing test-retest and inter-rater reliability is to
reduce the scale and the levels of measurement.
Examples:
Would a number of raters have better agreement when putting a
program into a category clean versus buggy (nominal) or into an
ordinal scale of “bugginess” from 1-10 or to predict its exact defect
density? (given the same operational definition in all three cases)
CSCI 6960- Research Methods
Lecture 7
HO 7
-7-
Reliability,Validity and Data
To increase internal consistency, several compatible operational
definitions are needed. However, internal consistency cannot be
assured unless a series of pilot measures are made.
Example:
If whilst measuring the number of lines of code in a program by
hand, we get a number say 120, doing it via an automated line
counter we get 127, and doing it by using the compiler’s line
numbering mechanism we get 112, then our measures would not
be consistent. To make them consistent, all three approaches must
be made to conform in the way they measure lines of code. This
means having compatible operational definitions.
CSCI 6960- Research Methods
Lecture 7
HO 7
-8-
Reliability,Validity and Data
Ensuring Validity:
Validity, we said, came in a number of “flavors”. These were:
1. Statistical Validity
2. Construct Validity
3. External Validity, and
4. Internal validity
CSCI 6960- Research Methods
Lecture 7
HO 7
-9-
Reliability,Validity and Data
Statistical Validity:
We say that a measure is statistically valid when we can
demonstrate that they did not arise by chance.
One threat to statistical validity is when the data is not reliable.
We have already discussed how to increase reliability.
Another threat is the researcher’s violation of the assumptions
that underlie statistical tests. For example when the researcher
uses a test appropriate for independent groups on data that is
internally correlated.
Use of appropriate statistical procedures on reliable data is the
best way of improving statistical validity.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 10 -
Reliability,Validity and Data
Construct Validity:
Construct validity refers to how well the study’s result (or data)
support the underlying principles relevant to the work. Construct
validity would be in question when the evidence can be
explained in more than one way; that is according to more than
one hypothesis or theory.
Example:
A research project “showed” that female programmers score
consistently lower in their annual appraisals compared to their
male counterparts; thus females must be poorer programmers.
This is a prime example of dubious construct validity.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 11 -
Reliability,Validity and Data
Because:
1. It may be that the hypothesis is true and females are
indeed poorer programmers than males, or that
2. Females are discriminated against and are not rated fairly
in a largely male-dominated and sexist work
environment, or that
3. In the current society, those females with talent for
programming would be first attracted to other
professions, or that
4. Females do not care as much about performance
evaluations as males do and therefore do not argue for a
higher evaluation score during annual evaluation.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 12 -
Reliability,Validity and Data
For the researcher to have his (could not be hers, could it?)
hypothesis accepted (hypothesis 1), he has to design, perform
and publish research that refutes each and every of the
remaining hypotheses. Those listed here, or any other that
might emerge.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 13 -
Reliability,Validity and Data
External Validity:
External validity refers to the degree to which we are able to
generalize the results of a study to other subjects, conditions,
environments, times and place.
To make a generalization from the sample to any population, the
sample must be an adequate and acceptable sample of THAT
population.
Example: In a research project rates of defect detection of
particular testing schemes were calculated and contrasted. It
turned out that Scheme A was better than scheme B in
discovering defects of functionality and usability.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 14 -
Reliability,Validity and Data
To do so, the researchers used 30 programs written by masters
students and using the C programming language. Each program
was of between 20 and 1000 lines long.
How well would the result that method A is better than B for
discovering functionality and usability defects transfer to
programs written in the object-paradigm, in Eiffel, and by
professionals?
we must first find if the sample is representative of the population.
Despite the differences in language, programmer background and
paradigm, the sample may STLL BE representative!!!
CSCI 6960- Research Methods
Lecture 7
HO 7
- 15 -
Reliability,Validity and Data
The problem of generalization from sample to population is often
best controlled by random and adequate selection of subjects from
the population. The researcher must be careful that if he or she
wishes to extend the finding to a particular group, that such group
should at least be represented.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 16 -
Reliability,Validity and Data
Internal Validity:
Internal validity deals with the concern whether there was
causality at play. In other words; “Was the independent variable
and not some extraneous variable responsible for the observed
changes in the dependent variable?”.
There are many factors that can interfere with internal validity.
These are collectively called confounding factors. We must
minimize the effect of these factors in order to increase the
internal validity of our work.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 17 -
Reliability,Validity and Data
Threats to internal Validity:
1. Attrition: Loss of subjects during study. Differential loss is
particularly problematic as those who drop out are usually
the interesting ones.
2. Diffusion: When information “leaks” from one subject or
group to another and thus modifies behavior.
3. Experimenter effects: The inadvertent or intentional action of
the experimenter that might compromise the study.
4. History: Changes in the dependent variable that are due to
historical or time-based events but are not related to the
study.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 18 -
Reliability,Validity and Data
5. Instrumentation: Any change or change in calibration of the
instruments.
6. Learning : Changes in the dependent variable that occur due
to learning done as a result of participation in the study.
7. Maturation: Changes in the dependent variable that occur
during the course of study due to normal passage of time and
maturation/development of the subject.
8. Placebo effect: The effect that the subjects might compromise
the results by behaving in a certain controlled way through
knowledge of the result being sought. E.g. when subjects
feign drunk even when given unlaced tonic.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 19 -
Reliability,Validity and Data
9. Regression to the mean: The tendency for subjects that had
extreme scores in earlier phases to be less extreme in followup scoring.
10. Sequencing effect: The impact of the experience a subject had
in one situation on the next situation.
11. Testing: The impact of the subject having been tested before.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 20 -
Reliability,Validity and Data
Some controls to threats of validity include:
1. Use of calibrated and proper preparation of equipment.
2. Replication
3. Single and double blind procedures
4. Automation
5. Multiple observers
6. Use of deception (within the bounds of ethics)
7. Random subject selection
8. Control of subject-to-subject communication
CSCI 6960- Research Methods
Lecture 7
HO 7
- 21 -
Reliability,Validity and Data
1. Have a clear, literature supported initial idea
2. Have a clear and identifiable statement of problem.
3. Ensure that all variables are identified and operationally
defined.
4. Develop a clear research hypothesis
5. Select your statistical analysis procedures
6. Clearly identify the theoretical bases of your intended study
7. Identify if the hypothesis and procedures address the issue.
CSCI 6960- Research Methods
Lecture 7
HO 7
- 22 -
Reliability,Validity and Data
8. Ensure that the independent variable manipulation has been
carefully planned to ensure reliability and validity
9. Pre-test (pilot) the manipulations. Make any changes necessary
10. Ensure that all dependent variables are adequately defined
through operational definitions.
11. Pres-test and pilot dependent variables
12. Put all controls of reliability and validity in place
13. Ensure the sample is representative
14. Ensure the sample is sufficiently large
CSCI 6960- Research Methods
Lecture 7
HO 7
- 23 -
Reliability,Validity and Data
15. Ensure correct assignment of subjects in accordance to the
conditions in the research design
16. Ensure subject availability and produce a data collection
schedule.
17. Ensure all ethical issues have been addressed and that all ethics
preserving procedures are in place.
18. Ensure the logistics of the study. E.g space, equipment,
personnel, instruction lists, labels, etc.
19. Go for it.
CSCI 6960- Research Methods
```