Experimental Design for Linguists Charles Clifton, Jr. University of Massachusetts Amherst Slides available at http://people.umass.edu/cec/teaching.html Goals of Course ► Why should linguists do experiments? ► How should linguists do experiments? Part 1: General principles of experimental design ► How should linguists do experiments? Part 2: Specific techniques for (psycho)linguistic experiments Schütze, C. (1996). The empirical basis of linguistics. Chicago: University of Chicago Press. Cowart, W. (1997). Experimental syntax: Applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications Inc. Myers, J. L., & Well, A. D. (in preparation). Research design and statistical analysis (3d ed.). Mahwah, NJ: Erlbaum. 1. Acceptability judgments ► Check theorists’ intuitions about acceptability of sentences Acceptability, grammaticality, naturalness, comprehensibility, felicity, appropriateness… ► Aren’t theorists’ intuitions solid? Example of acceptability judgment: Cowart, 1997 ► Subject extraction: I wonder who you think (that) likes John. ► Object extraction: I wonder who you think (that) Mean judged acceptability (z-score) John likes. 0.6 0.4 0.2 0 -0.2 -0.4 -0.6 -0.8 No-That No-That That That Subject Extraction Object Extraction Stability of ratings (Cowart,1997) 2. Sometimes linguists are wrong… ► Superiority effects I’d like to know who hid it where. *I’d like to know where who hid it. ► Ameliorated by a third wh-phrase? ?I’d like to know where who hid it when. …maybe. Paired-comparison preference judgments a. I’d like to know who hid it where. 86% b. (*)I’d like to know where who hid it. 14% c. (*)I’d like to know where who hid it when. d. I’d like to know who hid it where when. 76% 24%. 49% 51% a-b basic superiority violation b-c heads-on comparison, extra wh “when” hurts, doesn’t help c-d the “ameliorated” superiority violation, c, seems good when compared to its non-superiority-violation counterpart Clifton, C. Jr., Fanselow, G., & Frazier, L. (2006). Amnestying superiority violations: Processing multiple questions. Linguistic Inguiry, 37(51-68). Another instance… Question: is the antecedent of an ellipsis a syntactic or a semantic object? Why is (a) good and (b) bad? (a) The problem was to have been looked into, but obviously nobody did. (b) #The problem was looked into by John, and Bob did too. Andrew Kehler’s suggestion: semantic objects for causeeffect discourse relations, syntactic objects for resemblance relations. Corpus data bear his suggestion out. But an experimental approach… Kim looked into the problem even though Lee did. (causal, syntactic parallel) Kim looked into the problem just like Lee did. (resemblance) The problem was looked into by Kim even though Lee did. (causal, nonparallel) The problem was looked into by Kim just like Lee did. (resemblance) Mean Acceptability Rating (5 = good) 4.5 4 Causal Resemblance 3.5 3 2.5 Parallel NonParallel Frazier, L., & Clifton, C. J. (2006). Ellipsis and discourse coherence. Linguistics and Philosophy, 29, 315-346. Context effects ► Linguists: think of minimal pairs ► The contrast between a pair may affect judgments ► Hirotani: Production of Japanese sentences The experimental context in which sentences are produced affects their prosody Hirotani experiment a. Embedded wh-question (ka associated to na’ni-o) (# = Major phrase boundary) Mi’nako-san-wa Ya’tabe-kun-ga Minako-Ms.-TOP Yatabe-Mr.-NOM na’ni-o moyasita’ka (#) gumon-sita’-nokai? what-ACC burned-Q stupid question-did-Q (-wh) ‘Did Minako ask stupidly what Yatabe burned?’ (Yes, it seems (she) asked such a question.’) b. Matrix wh-question (ndai associated to na’ni-o) Mi’nako-san-wa Ya’tabe-kun-ga Minako-Ms.-TOP Yatabe-Mr.-NOM na’ni-o moyasita’ka (#) gumon-sita’-ndai? what-ACC burned-Q stupid question-did-Q (+wh) ‘What did Minako ask stupidly whether Yatabe burned?” (‘The letters (he) received from (his) ex-girlfriend.’) Hirotani results Percentage of insertion of MaP before phrase with question particle Initial Block (pure) Final Block (pair contrast) Embedded question 100% 100% Matrix question 57% 15% Hirotani, Mako. (submitted). Prosodic phrasing of wh-questions in Japanese 3. Unacceptable grammaticality ► Old multiple self-embedding sentence experiments Miller & Isard 1964: sentence recall, right-branching vs. self-embedded (1-4) ► She liked the man that visited the jeweler that made the ring that won the prize that was given at the fair. ► The prize that the ring that the jeweler that the man that she liked visited made won was given at the fair. ► Median trial of first perfect recall: 2.25 vs never Stolz 1967, clausal paraphrases: subjects never understood the self-embedded sentences anyway Miller, G. A., & Isard, S. (1964). Free recall of self-embedded English sentences. Information and Control, 4, 292-303. Stolz, W. (1967). A study of the ability to decode grammatically novel sentences. Journal of verbal Learning and verbal Behavior, 6, 867-873.. 3’. Acceptable ungrammaticality Speeded acceptability judgment and acceptability rating %OK Rating a. OK None of the astronomers saw the comet, but John did. 83% 4.36 B. Embedded VP Seeing the comet was nearly impossible, but John did. 66% 3.71 The comet was nearly impossible to see, / but John did. 44% 3.27 C. VP w/ trace D. Neg adj The comet was nearly unseeable, / but John did. 17% 2.21 Arregui, A., Clifton, C. J., Frazier, L., & Moulton, K. (2006). Processing elided verb phrases with flawed antecedents: The recycling hypothesis. Journal of Memory and Language, 55, 232-246. 4. Provide additional evidence about linguistic structure ►A direct experimental reflex of structure would be nice But we don’t have one ► Are traces real? Filled gap effect: reading slowed at us in My brother wanted to know who Ruth will bring (t) us home to at Christmas. Compared to My brother wanted to know if Ruth will bring us home to at Christmas. Stowe, L. (1986). Parsing wh-constructions: Evidence for on-line gap location. Language and Cognitive Processes, 1, 227-246. Are traces real, cont. ► Pickering and Barry. “no.” ► Possible evidence That’s the pistol with which the heartless killer shot the hapless man yesterday afternoon t. That’s the garage with which the heartless killer shot the hapless man yesterday afternoon t. disrupted at shot in the second example, far before the trace position ► Reading But who’s to say that the parser has to wait to project the trace? Pickering, M., & Barry, G. (1991). Sentence processing without empty categories. Language and Cognitive Processes, 6, 229-259. Traxler, M. J., & Pickering, M. J. (1996). Plausibility and the processing of unbounded dependencies: An eye-tracking study. Journal of Memory and Language, 35, 454-475. 5. Is grammatical knowledge used? ► Serious question early on “psychological reality” experiments ► Direct experimental attack did not succeed Derivational theory of complexity ► Indirect experimental attack has succeeded Build experimentally-based theory of processing 6. Test theories of how grammatical knowledge is used ► Moving beyond modularity debate – more articulated questions about real-time use of grammar ► Phillips: parasitic gaps, selfpaced reading The superintendent learned which schools/students the plan to expand _ … overburdened _. (slowed at expand after students – plausibility effect) The superintendent learned which schools/students the plan that expanded _ … overburdened _. (no differential slowing at expand – no plausibility effect) Phillips, C. (2006) The real-time status of island phenomena. Language, 82, 795-823. II: How to do experiments. Part 1, General design principles ► Dictum 1: Formulate your question clearly ► Dictum 2: Keep everything constant that you don’t want to vary ► Dictum 3: Know how to deal with unavoidable extraneous variability ► Dictum 4: Have enough power in your experiment ► Dictum 5: Pay attention to your data, not just your statistical tests Dictum 1: Formulate your question clearly ► Independent variable: variation controlled be experimenter, not by what subject does ► Dependent variable: variation observed in subject’s behavior, perhaps dependent on IV ► Operationalization of variables Formulate your question ► Question: Do you identify a focused word faster than a non-focused word? Must clarify: Syntactic focus? Prosodic focus? Semantic focus? Must operationalize ►Syntactic focus – Clefting? Fronting? Other device? ►Prosodic focus – Natural speech? Manipulated speech? Synthetic speech? Target word or context? Formulate your question ► Question: does discourse context guide or filter parsing decisions? Clarify question: does discourse satisfy reference? establish plausibility? set up pragmatic implications? create syntactic structure biases? Operationalize IV: Lots of choices here ►But also have to worry about dependent variable… Choose appropriate task, DV ► Question about focus: need measure of speed of word identification Conventional possibilities: lexical decision, naming, phoneme detection, reading time ► Question about “guide vs filter:” probably need explicit theory of your task Tanenhaus: linking hypothesis E.g. eye movements in reading: tempting to think that “guide” implicates “early measures,” “filter” implicated “late measures.” ► But what’s early, what’s late? Need model of eye movement control in parsing. Subdictum A: Never leave your subjects to their own devices ► It may not matter a lot Cowart example: 5-point acceptability rating ►A. “….base your responses solely on your gut reaction” ►B. “…would you expect the professor to accept this sentence [for a term paper in an advanced English course]?” ► But sometimes it does matter… Cowart 1997 Dictum 2: Try to keep everything constant except what you want to vary ► Try to hold extraneous variables constant through norms, pretests, corpora… ► When you can’t hold them constant, make sure they are not associated (confounded) with your IV An example: Staub, in press Eyetracking: does the reader honor intransitivity? Compare unaccusative (a), unergative (b), and optionally transitive) a. When the dog arrived the vet1 and his new assistant took off the muzzle2. b. When the dog struggled the vet1 and his new assistant took off the muzzle2. c. When the dog scratched the vet1 and his new assistant took off the muzzle2. Critical regions: held constant (the vet…; took off the muzzle). Manipulated variable (verb): conditions equated on average length and average word frequency of occurrence. Better: match on additional factors (number of stressed syllables, concreteness, plausibility as intransitive, ….) Better: don’t just have overall match, but match the items in each triple. Staub, A. (in press). The parser doesn't ignore intransitivity, after all. Journal of Experimental Psychology: Learning, Memory and Cognition. Another example: NP vs S-comp bias Kennison (2001), eyetracking during reading of sentences like: a. The athlete admitted/revealed (that) his problem worried his parents…. b. The athlete admitted/revealed his problem because his parents worried… Conflicting results from previous research (Ferreira & Henderson, 1990; Trueswell, Tanenhaus, & Kello, 1993): does a bias toward use as S-complement (admit) reduce the disruption at the disambiguating word worried? Problems in previous research: plausibility of direct object analysis not controlled (e.g., Trueswell et al., ambiguous NP (his problem) rated as implausible as direct object of S-biased verb) Kennison, normed material, equated plausibility of subject-verb-object fragment for NP- and S-comp biased verbs; found reading disrupted equally at disambiguating verb worried for both types of verbs. Kennison, S. M. (2001). Limitations on the use of verb information during sentence comprehension. Psychonomic Bulletin & Review, 8, 132-137. What happens when there is unavoidable variation? ► Subdictum B: When in doubt, randomize Random assignment of subjects to conditions Questionnaire: order of presentation of items? ► Single randomization: problems ► Different randomization for each subject ► Constrained randomizations ► Equate confounds by balancing and counterbalancing Alternative to random assignment of subject to conditions: match squads of subjects Counterbalancing of materials ► Counterbalancing Ensure that each item is tested equally often in each condition. Ensure that each subject receives an equal number of items in each condition. ► Why is it necessary? Since items and subjects may differ in ways that affect your DV, you can’t have some items (or subjects) contribute more to one level of your IV than another level. Sometimes you don’t have to counterbalance ► If you can test each subject on each item in each condition, life is sweet ► E.g., Ganong effect (identification of consonant in context) Vary VOT in 8 5-ms steps ► /dais/ - /tais/ ► /daip/ - /taip/ Classify initial segment as /d/ or /t/ ► Present each of the 80 items to each subject 10 times ► Ganong effect: biased toward /t/ in “type,” /d/ in “dice” Connine, C. M., & Clifton, C., Jr. (1987). Interactive use of information in speech perception. Journal of Experimental Psychology: Human Perception and Performance, 13, 291-299. If you have to counterbalance… ► Simple example Questionnaire, 2 conditions, N items Need 2 versions, each with N items, N/2 in condition 1, remaining half in condition 2 ► Versions ► More 1 and 2, opposite assignment of items to conditions general version M conditions, need some multiple of M items, and need M different versions ► Embarrassing if you have 15 items, 4 conditions… ► That means that some subjects contributed more to some conditions than others did; bad, if there are true differences among subjects Counterbalancing things besides items ► Order of testing Don’t test all Ss in one condition, then the next condition… At least, cycle through one condition before testing a second subject Fancier, latin square ► Avoid minor confound if always test cond 1 before cond 2 etc. ► N x n square, sequence x squad, containing condition numbers, such that each condition occurs once in each column, each order ► Location of testing E.g., 2 experiment stations Experimental Design for Linguists Charles Clifton, Jr. University of Massachusetts Amherst Slides available at http://people.umass.edu/cec/teaching.html and at http://coursework.stanford.edu Goals of Course ► Why should linguists do experiments? ► How should linguists do experiments? Part 1: General principles of experimental design ► How should linguists do experiments? Part 2: Specific techniques for (psycho)linguistic experiments Schütze, C. (1996). The empirical basis of linguistics. Chicago: University of Chicago Press. Cowart, W. (1997). Experimental syntax: Applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications Inc. Myers, J. L., & Well, A. D. (in preparation). Research design and statistical analysis (3d ed.). Mahwah, NJ: Erlbaum. II: How to do experiments. Part 1, General design principles ► Dictum 1: Formulate your question clearly ► Dictum 2: Keep everything constant that you don’t want to vary ► Dictum 3: Know how to deal with unavoidable extraneous variability ► Dictum 4: Have enough power in your experiment ► Dictum 5: Pay attention to your data, not just your statistical tests So how do you randomize? ► E-mail me (email@example.com) and I’ll send you a powerful program ► But for most purposes, check out http://www-users.york.ac.uk/~mb55/guide/randsery.htm Or http://www.randomizer.org/index.htm Factor out confounds ► Factorial design An example, discussed earlier: Arregui et al., 2006 Initial experiment contained a confound; corrected in second experiment by adding a second factor Arregui et al., rating study Acceptability rating Rating Rating clause 1 a. OK None of the astronomers saw the comet, but John did. 4.36 4.53 B. Embedded VP Seeing the comet was nearly impossible, but John did. 3.71 4.41 C. VP w/ trace The comet was nearly impossible to see, but John did. 3.27 4.81 D. Neg adj The comet was nearly unseeable, but John did. 2.21 4.39 Arregui, A., Clifton, C. J., Frazier, L., & Moulton, K. (2006). Processing elided verb phrases with flawed antecedents: The recycling hypothesis. Journal of Memory and Language, 55, 232-246. Factorial Design First clause Ellipsis absent Ellipsis present Syntactically OK None of the astronomers saw the comet. ..but John did. Embedded VP Seeing the comet was nearly impossible. ..but John did. VP with trace The comet was nearly impossible to see. ..but John did. Nominalization The comet was nearly unseeable. ..but John did. Factor 1: syntactic form of initial clause (4 levels) Factor 2: presence or absence of ellipsis (2 levels) An interaction Interaction: The size of the effect of one factor differs among the different levels of the other factor. Mean Acceptability Rating 5 4 3 OK Embed VP VP trace Nominal 2 1 0 Ellipsis absent Ellipsis present Factorial Designs in Hypothesis Testing ► Cowart (1997), that-trace effect Question: is it bad to extract a subject over that ► ?I wonder who you think (that) t likes John. Acceptability judgment: worse with that ► But: underlying theory talks just about extracting a subject. Does acceptability suffer with extraction of object over that? ►I wonder who you think (that) John likes t. Need to do factorial experiment 1: presence vs. absence of that ► Factor 2: subject vs. object extraction ► Factor Mean judged acceptability (z-score) The results (from before) 0.6 0.4 0.2 No-That 0 -0.2 -0.4 -0.6 -0.8 No-That That That Subject Extraction A clear interaction. Object Extraction A worry about scales ► Interactions of the form “the effect of Factor A is bigger at Level 1 than at Level 2 of Factor B. Cowart, effect of that bigger at subject than object extraction ► Types of scales Ratio: true zero, equal intervals, can talk about ratios (time, distance, weight) Interval: equal intervals, but no true zero (temperature, dates on a calendar) Ordinal: only more or less (ratings on rating scale, measures of acceptability, measures of difficulty) Original Scale 1000 800 600 Factor 1-1 Factor 1-2 400 200 0 Factor 2-1 Factor 2-2 Log scale 3 2 Factor 1-1 Factor 1-2 1 0 Factor 2-1 Is there really an interaction? Factor 2-2 Disordinal and crossover interactions Factor 1-1 Factor 1-2 35 30 30 25 25 20 20 15 15 10 10 5 0 Factor 1-1 Factor 1-2 5 Factor 2-1 Factor 2-2 0 Factor 2-1 Factor 2-2 An example of an important but problematic experiment: Frazier & Rayner, 1982 Closure: LC: Since Jay always jogs a mile and a half this seems like a short distance to him. 40 40 ms/ch EC: Since Jay always jogs a mile and a half seems like a very short distance to him. 35 54 ms/ch Attachment: MA: The lawyers think his second wife will claim the entire family inheritance. 36 ms/ch NMA: The second wife will claim the entire family inheritance belongs to her. 37 51 ms/ch Data shown: ms/character first pass times for the colored regions. Problems??? Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178-210. Dictum 3: Know how to deal with unavoidable extraneous variability ► i.e., know some statistics ► Measures of central tendency (“typical”) Mean (average, sum/N) Median (middle value) Mode (most frequent value) ► Measures of variability Variance (Average squared deviation from mean) Average deviation (Average absolute deviation from median) Computation of Variance Distr 1 X Mean-X Sq’d 7 9 10 6 X Mean-X Sq’d 81 12 4 16 36 14 2 4 … Distr 2 … 30 -14 196 21 -5 25 17 -1 1 17 -1 1 314 64 78.5 16 Sum 64 Mean 16 Variance 46 Variance 11.5 Variance in an experiment ► Systematic variance: variability due to manipulation of IV and other variables you can identify ► Random variance: variability whose origin you’re ignorant of ► Point of inferential statistics: is there really variability associated with IV, on top of other variability? Is there a signal in the noise? Best way to deal with extraneous variability: Minimize it! ► Keep everything constant Reduce experimental noise ►See the signal easier Keep environment, instructions, distractions, experimenter, response manipulanda, etc. constant Pretest subjects and select homogeneous ones, if that suits your purposes One way to minimize extraneous variance: Within-subject designs ► Subjects differ …a lot, in some measures, eg. Reading speed, reaction time ► Present all levels of your IV to each subject Assume the subject effect is a constant across all the levels. Differences among conditions thus abstracted from subject differences ► Counterbalancing necessary Test each item in each condition for an equal number of subjects. ► Worry about experience changing what your subject did E.g., will reading an unreduced relative clause (The horse that was raced past the barn fell) affect reading of a reduced relative clause sentence? Statistical tests/statistical inference ► Never expect observed condition means to be exactly the same Just noise? Or signal + noise? ► Statistical inference: is there really a signal? p value: the probability you’d obtain a difference among the means that is as large as what you observed, if the true signal is zero “null hypothesis” test Basic logic of statistical tests (t, F, etc.) ► Get one estimate of the variabilty due to noise + any signal Estimate from the variation among the observed mean values in the different conditions ► Get another estimate of the variabilty due to noise alone Estimate from how much variation there is among subjects, within a condition ► If signal = 0, ratio is expected to be 1 If it’s enough bigger than 1, then the signal is likely to be non-zero Underlying model ► Subjects are a random sample from some population ► You can make inferences about variability in the population from the observed variability in the sample ► Logical inference: “if the size of the signal in the population is zero, the probability of getting a difference among the means that is as big as we observed is p” where p is the level of significance If p is small enough, reject the proposal that the population signal is zero Between-subject design ► Estimate of signal + noise: variability among the condition means ► Estimate of noise alone: variability among the subject means in each condition ► F = MSbetween conds/MSwithin cond MS, not exactly variance; must divide sum of squares by df, not by N Within-subject design ► Estimate of signal + noise: variability between the condition means ► Estimate of noise alone Get a measure of the variability among condition means for each subject Calculate the variability among these measures Subjects x treatment interaction ► How much the the size of the treatment effect differs among subjects is an estimate of error variability. ►F = MSbetween conditions/MSsubjects x treatments Advanced topics ► Multi-factor designs, tests for interactions ► Treat counterbalancing factors as factors in ANOVA E.g., if have 4 conditions, 4 counterbalancing groups, differing in assignment of items to conditions, you can treat groups as a between-subject factor and pull out variability due to items from the subjects x treatment error term ► Statistical accommodation of extraneous variation Analysis of covariance Multi-level, hierarchical designs Pollatsek, A., & Well, A. D. (1995). On the use of counterbalanced designs in cognitive research: A suggestion for a better and more powerful analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 785-794. Forthcoming special issue of the Journal of Memory and Language on new and alternative data analyses. Dictum 4: Have enough power to overcome extraneous variability ► Add more data! Minimizes noise component of differences among condition means ► Law of large numbers The larger the sample size, the more probable it is that the sample mean comes arbitrarily close to the population mean If you’re (almost) looking at population means, any differences have to be real – not sampling error Law of large numbers a population with a variance v2. ► Imagine you take a bunch of independent samples from this population, each sample of size N. ► Each sample will have a mean value. ► These mean values will have a variance, which turns out to be v2/N. ► This variance will be smaller as N gets larger. ► Imagine A sampling simulation ► The effect of sample size on the variability of sample means Bigger samples, smaller variability Standard deviation = square root of variance N=5 http://onlinestatbook.com/stat_sim/index.html N = 25 Means from larger Ns have less noise ► Holds for subject means More subjects, means reflect vagaries of sample less; means have less noise ► Holds for item means too More items, means less affected by peculiarity of individual items ► OK, you can have too many items and burn out your subjects Have enough power…. ► Back to holding everything constant First reason: don’t want variables confounded with our independent variable Second reason: minimize noise. Less noise, more power. Dictum 5: Pay attention to your data, not just your statistical tests ► Look at your data, graph them, try to make sense out of them Don’t just look for p < .05! ► Examine confidence intervals ► Look at your data distributions Stem and leaf graphs By subjects… Confidence intervals ► Confidence subjects) intervals (of means over items and If you have a sample mean and you know the true population standard deviation of the sample σM , you can say that there is a 95% chance that the true population mean is within +/- 1.96 * σM your sample mean. But of course you don’t know σM so you have to estimate it from your data and use the t distribution. But then you can present your means as X +/- CI ►A simulation: http://onlinestatbook.com/stat_sim/index.html Confidence Intervals ► Do you want to look at individual item data? Don’t make too much of the tea leaves Consider getting a confidence interval on the individual item means Example: Self-paced reading time ► Cond 1: This table is slightly dirty and the manager wants it removed. (minimum standard adjective) ► Cond 1: This table is slightly clean and the manager wants it removed. (maximum standard adjective) ► Reading time, clause 2, slower for maximum than minimum standard adjective Are some items more effective than others? Confidence intervals, individual items ► Each item has 12 different observations (different subjects) in each condition. ► Can measure the variability among these subject data points for max std and min std adjective And from that, estimate the variability of the difference, and from that, the confidence interval of the difference Max std adj Min std adj Mean Max Mean Min Diff 95% CI Diff Clean Dirty 1960 1486 474 +/- 662 Safe Dangerous 1572 1258 314 +/- 303 Healthy Sick 1635 1164 471 +/- 485 Dry Wet 1130 1229 -99 +/- 427 Complete Incomplete 1196 1617 -421 +/- 621 Dictum 5: Pay attention to your data, not just your statistical tests ► Graph your data ► Examine confidence intervals ► Look at the distributions of your means Stem and leaf graphs By subjects…and by items Maria asked Bob to invite Fred or Sam to the barbecue. She didn't have enough room to invite both. Frequency Stem & Leaf 3.00 0 . 778 17.00 1 . 01111222333334444 15.00 1 . 556677788889999 11.00 2 . 00011122333 1.00 2. 7 1.00 Extremes (>=3286) Stem width: 1000.00 Each leaf: 1 case(s) Maria asked Bob not to invite Fred or Sam to the barbecue. She didn't have enough room to invite both. Frequency Stem & Leaf 1.00 0. 9 13.00 1 . 1112223334444 22.00 1 . 5555555666677777777788 9.00 2 . 000111344 1.00 2. 7 2.00 Extremes (>=3120) Stem width: 1000.00 Each leaf: 1 case(s) Maria asked Bob to invite Fred or Sam to the barbecue. She didn't have enough room to invite both. Maria asked Bob not to invite Fred or Sam to the barbecue. She didn't have enough room to invite both. By items Frequency Stem & Leaf 7.00 1 . 1122234 12.00 1 . 556666667788 4.00 2 . 1123 1.00 Extremes (>=2629) Stem width: Each leaf: 1000 1 case(s) Frequency Stem & Leaf 1.00 Extremes (=<950) 4.00 1 . 3334 14.00 1 . 56666777777888 4.00 2 . 1222 1.00 Extremes (>=2562) Stem width: 1000.00 Each leaf: 1 case(s) Variation among items ► Treat items as a random sample from some population. Just like we treat subjects as a random sample ► Then do statistical tests to generalize to this population of items. “F1” and “F2” ► Criticisms Should generalize simultaneously to subjects and items, using F’. ► But must estimate F’ unless every you have data from every subject on every condition of every item (min F’; Clark, 1973) We’re fooling ourselves when we view items as anything like a random sample from a population. Alternatives to F1 and F2 ► Some conventional ANOVA designs do permit generalization to subjects and items without full data But generally lack power ► Coming designs trend: multilevel, hierarchical Complex regression-based analyses of individual data points, not subject- or item-means. But what if you recognize that random sampling from a population of items is nutty? ► What you really want is to show that your effects hold for most or all of your items and aren’t due to a couple of oddballs F2 tests a crude attempt to do this. ► People struggling to get a better way. One possibility, from Ken Forster: plot effect size vs effect rank, see if it is pleasingly regular. Forster, “What is F2 good for” ► Plot effect size (difference between two conditions) against rank of effect size (suggested by Peter Killeen) Both cases: a 5 msec mean effect size Left panel: a limited effect (add 100 ms to 5 items) Right panel: a general effect (add 5 ms to 100 items) 20 120 100 R2 = 0.9502 10 R2 = 0.4491 EFFECT SIZE EFFECT SIZE 80 60 40 20 0 0 0 10 20 30 40 -10 0 10 20 30 40 50 -20 -40 -20 RANK RANK Forster, K. (2007). What is F2 good for? Round 2. Unpublished ms, University of Arizona. 50 Bogartz, 2007 ► Effect size vs. rank effect size, Clifton et al. JML 2003 Effect of ambiguity (absence of relative pronoun) on sentences with relative clauses (The man [who was] paid by the parents was unreasonable) Contrasted with Monte Carlo data based on same mean and variance as experimental data Bogartz, R. (2007). Fixed vs. random effects, extrastatistical inference, and multilevel modeling. Unpublished manuscript, University of Massachusetts. III. How to do experiments, Part 2: Experimental procedures ► Acceptability judgment ► Interpretive choices ► Stops making sense ► Self-paced reading ► Eyetracking during reading ► ERP ► Secondary tasks ► Speed-accuracy tradeoff tasks ► Eyetracking during listening (“visual world”) Choose task that is appropriate for your question ► Is this really a sentence of English? ► Does some variable affect how a sentence is understood? ► Is there some difficulty in understanding this sentence? ► Just where in the sentence does the difficulty appear? ► Where in processing does the difficulty appear? ► Can we observe consequences of processing other that difficulty? ► and more…. Acceptability judgment ► Simple written questionnaire See Schütze, Cowart for lots of examples Worry about instructions Rating scales ►Is seven the magical number? ► Magnitude estimation Basis in psychophysics – attempt to build an interval scale Magnitude estimation: an example Which man did you wonder when to meet? Assign an arbitrary number to that item, greater than zero. Now, for each of the following items, assign a number. If the item is better than the first one, use a larger number; if it’s worse, smaller. Make the item proportional to how much better or worse the item is than the original – if twice as good, make the number 2x the start; if 1/3 as good, make the number 1/3 as big as the start. Magnitude estimation : an example ► Which man did you wonder when to meet? Assign an arbitrary number, greater than 0, to this first item. Now, for each successive item, assign a number – bigger if the item is better, smaller if worse, and proportional – if the item is 2x as good, make the number 2x the original; if ¼ as good, make the number ¼ as big as the original. Which book would you recommend reading? ► When do you know the man whom Mary invited? ► This is a paper that we need someone who understands. ► With which pen do you wonder when to write. ► Who did Bill buy the car to please? ► Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72. On-line and web-based questionnaires ► WebExp: http://www.webexp.info ► Subject scheduling systems option ► Advantages: Big N, easy, broader population ► Disadvantages: you have to worry about control Speeded acceptability judgment ► Time pressure; discourage navel-examining ► Measure reaction time and acceptability ► Example: is given-new order more acceptable than new-given? Maybe so. Maybe not always. Given-New: DefNP-IndefNP • All the players were watching an umpire. The pitcher threw the umpire a ball. New-Given: IndefNP-DefNP b. The catcher tossed a ball to the mound. The pitcher threw an umpire the ball. Given-New: DefNP-IndefPP c. The catcher tossed a ball to the mound. The pitcher threw the ball to an umpire. New-Given: IndefNP-DefPP d. All the players were watching an umpire. The pitcher threw a ball to the umpire. Given-New New-Given Given-New 3600 100 3200 90 Reaction 2800 Time, ms Percent Accepted 2400 2000 New-Given 80 70 NP-NP NP-PP 60 NP-NP NP-PP Clifton, C. J., & Frazier, L. (2004). Should given information come before new? Yes and no. Memory & Cognition, 32, 886-895. Choice of interpretation ► Paper and pencil or speeded ► Multiple-choice or paraphrase ► Example: interpretation of ellipsis Full stop effect ► Auditory questionnaire Relative size of intonational phrase boundary ► Strengths: does indicate whether a variable has an effect or not ► Weaknesses: don’t know when the effect operates Worst case: subject says sentence to self, mulls it over, reacts to the prosody s/he happened to impose Example of interpretation questionnaire: VPE John said Fred went to Europe and Mary did too. What did Mary do? …went to Europe 60% …said Fred went to Europe 40% John said Fred went to Europe. Mary did too. What did Mary do? …went to Europe 45% …said Fred went to Europe 55% Frazier, L., & Clifton, C. Jr. (2005). The syntax-discourse divide: Processing ellipsis. Syntax, 8, 154-207. Who arrived? Johnny and Sharon’sip inlaws. (0 ip) Who arrived? Johnnyip and Sharon’sip inlaws (ip ip) Who arrived? JohnnyIPh and Sharon’sip inlaws (IPh ip) Alternative answers: Sharon’s inlaws and Johnny; Sharon’s and Johnny’s inlaws Clifton, C. J., Carlson, K., & Frazier, L. (2002). Informative prosodic boundaries. Language and Speech, 45, 87-114. Stops-making-sense task ► Word-by-word, self-paced, but each word make one of two responses: OK, BAD ► Get cumulative proportion of BAD responses and OK RT ► Sensitive to point of difficulty in a sentence Example of SMS Which client/prize did the salesman visit while in the city? (transitive) Which child/movie did your brother remind to watch the show? (object control) Boland, J., Tanenhaus, M., Garnsey, S., & Carlson, G. (1995). Verb argument structure in parsing and interpretation: Evidence from wh-questions. Journal of Memory and Language, 34, 774-806. Stops-making-sense task ► Strengths Begins to address processing dynamics questions Can get both time and choice as relevant data ► Weaknesses Very slow reading time – 500 to 900 ms/word typically Permits more analysis than is done in normal reading Self-paced reading ► Word by word self-paced reading Generally noncumulative Sometimes in place (“RSVP”), sometimes moving across screen Time strongly affected by length of word, frequency of word ►Can ► Variant: statistically adjust reading. phrase by phrase self-paced SPR methods ► Computer programs E-prime (www.pstnet.com) Dmastr/DMDX (http://www.u.arizona.edu/~kforster/dmastr/d mastr.htm) Others (PsyScope, Superlab, various homemade systems) SPR Evaluation ► Cheap and effective Don Mitchell, trailblazing technique ► Slower than normal reading Perhaps 180 words per minute reading Unless reader clicks fast and buffers…. ► Often get effect on word following critical word Spillover ► Phrase-by-phrase: overcomes these difficulties, but you lose precision Mitchell, D. C. (2004). On-line methods in language processing: Introduction and historical review. In M. Carreiras & C. J. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERPs, and beyond. Brighton, UK: Psychology Press. More SPR evaluation ► Does SPR hide subtle details Maybe: Clifton, Speer, & Abney 1991 JML; Schütze & Gibson 1999 JML attachment: The man expressed his interest in a hurry during the storewide sale… (VP adjunct) ► NP attachment: The man expressed his interest in a wallet during the storewide sale… (NP argument) ► Verb Clifton et al: eyetracking, slow first-pass time in NPattached PP (followed by faster reading for argument than adjunct) Schütze & Gibson, word by word SPR, only the argument advantage ► Better materials ► Worse technique Even more SPR evaluation ► Does SPR introduce unnatural effects? Maybe: Tabor, Galantucci, Richardson, 2004, local coherence effects The coach smiled at the player tossed/thrown the frisbee by the… slowed reading at tossed as if reader considering grammatically illegal main clause interpretation of “the player tossed the…” ► Result: But: scuttlebutt, may not show up in eyetracking ► Global SPR reading speed, 412 ms/word, 145 wpm Eyetracking during reading ► Eye movement measurement Fixations and saccades Reading time affected by word length, frequency, other lexical factors Word-based measures of eye movements (ms) Most cowboys hate to live in houses so they 1 223 2 3 235 178 cowboys SFD: FFD: 235 ms GAZE: 413 ms Go-P: 413 ms 4 6 5 7 301 179 267 199 hate 301 ms 301 ms 301 ms 301 ms houses 267 ms 267 ms 267 ms 436 ms Region-based measures While Mary/ was mending/ the sock/ fell off/ * * * * * * ** * 1 2 3 6 4 7 58 9 277 213 233 277 445 289 401 233 314 First pass: 510 ms Second pass: 401 ms Go-Past: 510 ms Total Time: 911 ms 445 ms 547 ms 1393 ms 992 ms Interpretation of the measures ► “Early” vs. “late” measures Debates about modularity Some measures clearly late – second pass time But early: need explicit model of eye movement control ► Rayner, Pollatsek, Reichle, colleagues – EZ Reader Good model of lexical effects Says little or nothing about parsing & intepretation ERP (event-related potentials) ► Measure electrical activity on scalp Reflect electrical activity of bundles of cortical neurons Good time resolution, questionable spatial resolution ► Standard effects: LAN, N400, P600 Typical peak time, polarity “Standard” ERP effects (Osterhout, 2004) N400 P600 The cat will EAT The cat will BAKE The cat will EAT *The cat will EATING Osterhout, L. et al. (2004). Sentences in the brain…. In M. Carreiras and C. Clifton, Jr., The on-line study of sentence comprehension. New York: Psychology Press, pp 271-308. Secondary tasks: Load effects ► Limited capacity models ► Desire: measure of auditory processing difficulty ► Phoneme monitoring Eg: Cutler & Fodor, 1979 ► Which man was wearing the hat? The man on the corner was wearing the blue hat. ► Which hat was the man wearing? The man on the corner was wearing the blue hat. ► Target: /k/ or /b/; when target started focused word, 360 ms; when started non-focused word, 403 ms. Interpretive difficulties Secondary tasks: Load effects II ► Lexical decision (or naming, or semantic decision, or….) Word unrelated to sentence; measure of available capacity Piñango et al., auditory presentation, visual probe ►The man examined the little bundle of fur for a long time aspect to see if it was… 743 ms ►The man kicked the little bundle of fur for a long time aspect to see if it was… 782 ms Pinango, M. M., Zurif, E., & Jackendoff, R. (1999). Real-time processing implications of enriched composition at the syntaxsemantics interface. Journal of Psycholinguistic Research, 28, 395-414. Secondary tasks: Probe for activation ► Auditory (or visual) presentation Probe semantically related to word in sentence whose activiation you want to measure ► E.g., activation of “filler” at “gap” in longdistance dependency The policeman saw the boy who the crowd at the party1 accused2 of the crime. probe girl or matched unrelated word at point 1 or 2; girl faster at 2. ►Present Worries, criticisms… Nicol, J., Swinney, D., Love, T., & Hald, L. (2006). The on-line study of sentence comprehension: An examination of dual task paradigms. Journal of Psycholinguistic Research, 35, 215-231. Speed-accuracy tradeoff ► Present sentence (usually RSVP), subject to make judgment (grammaticality, etc.) ► But judgment is made in response to a signal that is presented some time after a critical point. ► Accuracy increases with time after the critical point. ► Note, current procedure, multiple signals and multiple responses, e.g., every 350 ms Early procedure: just one signal, one response, per trial McElree, B., Pylkkanen, L., Pickering, M., & Traxler, M. (2006). A time course analysis of enriched composition. Psychonomic Bulletin & Review, 13, 53-59. McElree et al. data Best fit: coercion lowered asymptote and lowered rate of approach to asymptote. Visual World (“head-mounted eyetracking”) ► Measure where you look when you are listening to speech. Cooper, 1974. About 40% probability of fixating on referent, 30% fixating on related picture ►About 10% in control group. ► Permits on-line measure of processing during listening. Not just difficulty – actual content Both incidental looks and controlled reaching Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107. Cooper, 1974 While on a photographic safari in Africa, I managed to get a number of breathtaking shots of the wild terrain. These included pictures of rugged mountains and forests as well as muddy streams winding their way through big game country. One of my best shots thought was ruined by my scatterbrained dog Scotty. Just as I had slowly wormed my way on my stomach to within range of a flock…. Allopenna, Magnuson & Tanenhaus (1998) Eye camera Scene camera Pick up the beaker Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38, 419-439. Allopenna et al. Results 200 ms after coarticulatory information in vowel Thanks! Enjoy the Institute!