University of Washington EMBA Program Regional 20 Marketing Management “Marketing Research” Instructor: Elizabeth Stearns With Survey review from Professor Dan Turner Marketing Research Agenda • Description and Overview • Choosing to do Marketing Research • Survey & Questionnaire Design • Conjoint Analysis • Advertising Research Marketing Research • A disciplined approach to the development and provision of information relevant to marketing decision-making. • Research should be focused on addressing questions where different answers would alter the decision a rational manager makes What is the first thing that comes to mind… Fit the research to the problem! Open Blind Prefer Pepsi 51% Prefer Coke 44% Equal/Can’t Say 5% Understand your what your customers value. Open Blind Prefer Pepsi 51% Prefer Coke 44% Equal/Can’t Say 5% Two Layers of Decision Problems • Management Decision Problem – What should the decision maker do? Should the price of the brand be increased? Should we change the advertising campaign? • Marketing Research Problem – What information is needed and how can it be obtained? What is the price elasticity of demand? What is the anticipated impact on sales and profits of the price change? How effective is the current campaign in generating awareness? Should Research be Conducted? Time Constraints Is there sufficient time available before a managerial decision must be made? N Availability of Data Y Is the information already on hand inadequate for Y making the decision? N Decision Importance Is the decision of considerable strategic or Y tactical importance? N Value vs. Costs Does the value of the research information exceed the cost of Y conducting research? N Marketing Research Should Not be Conducted Conduct Marketing Research Backward Market Research • Start the process where it usually ends and then work backward – Determine how the research will be implemented – Determine what the final report should contain and how it should look – Begin with an end in mind • Close collaboration between the researcher and the organizational decision maker is the single most important factor in obtaining a desirable outcome Marketing Research • Qualitative: Exploratory/Developmental – Help frame the problem • Provide insight and ideas • Generate testable hypotheses • Choose variables for inclusion in a data set • Add richness to interpretation of relationships that are uncovered through quantitative analysis • Quantitative: Descriptive/Evaluative – Detect & describe relationships between variables in data – Sophisticated quantitative analysis cannot compensate for framing the problem inappropriately and, hence, examining the wrong set of data. Nor can it insure the correct interpretation of a relationship that is uncovered. • Causal: Experimental Choosing Research Designs Research Design EXPLORATORY (Qualitative) * Identifies motivations underlying behavior * Yields initial hunches & insights for future research DESCRIPTIVE (Quantitative) * Describes consumer behavior Who, when, where, how much * Verifies insights with numerical data for action CAUSAL (Experimental) *Identify consumer response to marketing strategies * Formally establishes cause and effect relationship Data Collection Examples • Focus Groups • Ethnographies • Depth Interviews • Projective Techniques Sample Marketing Application Opportunity Analysis Issue Identification • Observational Studies Target Profile • Surveys (one time) Identification • Consumer Panels(longetudinal) •A & U Studies Demand Estimation • Lab Experiments Strategy Formulation • Field Experiments • Quasi-Experiments Strategy Evaluation Sources of Marketing Data PRIMARY DATA SOURCES External Sources Internal Sources • Market research firms • Advertising agencies • In-House marketing research department SECONDARY DATA SOURCES External Sources Internal Sources • Government agencies • Accounting data • Competitors • Sales reports • Trade associations • Factory shipments • Business publications • Syndicated sources – Store audits – UPC scanner data – Advtg. exposure data – Single source data – Specialized syndicated products/expert systems Ethnography Survey Basics • Background: Surveys as part of the marketing research efforts • Managing the Survey Effort • Item Development • Questionnaire Organization • Critical Reviews & Pretests • Additional Resources Background—Surveys as Part of the Marketing Research Effort • Form of primary data collection for descriptive research • Can be either cross-sectional (snapshot/photograph) or longitudinal (movie) – Easier to obtain representativeness with cross-section but more difficult to detect changes over time • Surveys administered via interview (personal, phone) or “writing” (paper & pencil, web/email) Key Questions in Survey Research • What managerial decision is to be made? • What information will best help us make the decision? Begin with an end in mind! • What survey questions/items can get us that information from respondents? • How should those questions/items be phrased? • Who should our respondents be? • How are we going to contact respondents? • How many should we get? • What will we revise based upon the pretest? • How do we analyze and draw inferences from the data? Managing the Survey Effort : Survey Mode • Primary considerations are effectiveness, speed, monetary cost – Personal interview/intercept • Negatives—Expensive, respondent/interviewer biases • Positive—Flexibility – Telephone • Negatives—Requires very simple items, representativeness issues • Positive—Quick – Paper & pencil/Mail • Negative—Requires simple items • Positive—Respondent privacy/anonymity – Email/web • Negative—Sampling frame bias, requires simple items • Positives—Quick, cheap Managing the Survey Effort : Sampling • Probability sampling—Every element has a known (or knowable) non-zero probability of being included in the sample • Non-probability sampling—Sampling without random selection • Increasing sample size reduces random sampling error but does nothing to reduce biases – To reduce sampling error you have to be rich, and to reduce sample bias you have to be smart Managing the Survey Effort: Total Survey Error • Sampling error—Unless you measure the whole population (a census) your sample estimate will not likely equal the exact population value – “Margin of error” noted with survey results is typically an estimate of the sampling error • Nonsampling error – Nonresponse bias—Certain members of the original sample do not provide responses – Measurement error—Instruments used to gather observations produce systematic error Managing the Survey Effort : Controlling Nonresponse Bias • Two basic determinants of who responds to surveys and who does not – Availability • Choose data collection method that is best suited to the respondents – Motivation • Some element is exogenous, e.g., BCC survey vs. canned soup survey for UW MBA students • Controllable factors – Minimize cost of responding – Maximize the rewards – Establish trust that rewards will be delivered Managing the Survey Effort : Controlling Nonresponse Bias • Reward respondents – – – – Show positive regard and give verbal appreciation Support respondent values Offer tangible rewards (including possibly sharing results/insights) Write an interesting questionnaire • Reduce costs to respondents – – – – – Make the task appear brief Reduce physical and mental effort requirements Minimize the possibility for embarrassment Eliminate direct and indirect monetary costs Follow-up with nonrespondents in a second survey wave • Establish trust – Provide a token of appreciation in advance – Borrow legitimacy by establishing ties to a trusted organization Managing the Survey Effort : Measurement Error • Reliability—Stability, consistency, or reproducibility of a measure • Validity—Measurement of the construct or concept intended • Examples – “Short” ruler to measure distance – GMAT to measure business school acumen • Insights regarding measurement error – The greater the degree of structure the lower the magnitude of measurement error – The greater the self-presentation elicited by the question the greater the risk of measurement error Item Selection: Laws of Questionnaire Design • Information Test: Don’t ask a question unless truthful answers to it will provide useful information in making the decision at hand. – How (specifically) will I use the data from this question? • If the answer is no more precise than “I’ll analyze it” you probably don’t need that question. • If there is more than one way to get a particular piece of information, pick the items for which respondents are more likely to both… • know the answer • be willing to tell you the answer AND Item Design: Types of Items • • • • • • Closed-ended questions with categories Semantic differential Likert scale Object ranking Constant sum scale/Point Allocation Open-ended – Numeric – Textual Item Design: Types of Items • Closed-ended questions with categories What marathon training programs did you consider before choosing Team in Training? ___ Joints in Motion ___ Team Diabetes ___ Official Asics Training ___ Always Running ___ Chuckit ___ Team Danskin ___ Other ________________ (please specify) • Make sure categories are mutually exclusive and collectively exhaustive Item Design: Closed-ended Purchase Intent Scale If a set of three ant traps sold for approximately $1.00 and was available in stores where you normally shop, would you: ___ Definitely buy the traps ___ Probably buy ___ Probably not buy ___ Definitely not buy Item Design: Types of Items • Semantic Differential – – – – Most popular way in marketing research to obtain attitude toward an object Very useful for rating several alternatives on a given set of attributes Anchored at each end by an adjective • Often bipolar (Sweet-Sour) • Sometimes monopolar (Sweet-Not Sweet) Respondent marks the point which best describes the object(s) Old New Item Design: Semantic Differential Scale Lexus IS300 and BMW 3 Series Unattractive Appearance --- --- --- --- --- --- --- Attractive Appearance Noisy --- --- --- --- --- --- --- Quiet Reliable --- --- --- --- --- --- --- Unreliable Built well --- --- --- --- --- --- --- Built poorly Good value for money --- --- --- --- --- --- --- Poor value for the the money Item Design: Likert Scale Strongly Disagree Strongly Agree They might make my feet feel hot 1 2 3 4 5 I am satisfied with what I am using 1 2 3 4 5 My problem is not serious enough 1 2 3 4 5 Too much trouble to cut them to fit 1 2 3 4 5 Price is too expensive 1 2 3 4 5 Might make my shoes too tight 1 2 3 4 5 I’m embarrassed to buy them 1 2 3 4 5 Item Design: Constant Sum Scale/Point Allocation Below are ten characteristics of women’s tennis sportswear. Please allocate 100 points among the characteristics such that the allocation represents the importance of each characteristic to you. The more points that you assign to a characteristic, the more important it is. If the characteristic is totally unimportant, you should no allocate any points to it. When you’ve finished, please double check to make sure that your total adds to 100. • Disadvantage: High level of respondent effort • Advantage: Ratio level measurement Item Design: Point Allocation Example Characteristics of Tennis Sportswear Number of Points Is comfortable to wear _______ Is durable _______ Has the endorsement of a famous athlete _______ Is made by well-known brand or sports manufacturers _______ Is made in the U.S.A _______ Has up-to-date styling _______ Is flattering to the body _______ Gives freedom to the body _______ Is a good value for the money _______ Authentic, like the pros wear _______ _______ 100 Points Item Design: Key Issues Checklist • Can the research objective be fulfilled without asking this question? – There is a cost to each item. • Do respondents have the information and motivation to answer each item? – Do not ask what respondents cannot or will not divulge. • Are the questions clear and unequivocal? Will the words be universally and uniformly understood? – For example, “How many members are there in your family?” • Does the question use a double negative? – “Are there no circumstances under which you would not use a stain removal detergent enhancer?” Item Design: Key Issues Checklist • Are the questions leading/loaded? – Do you think the US should allow public speeches against democracy? – Do you think the US should forbid public speeches against democracy? • 44% of respondents replied “No” to the first question • while 28% of respondents in a similar sample said “Yes” to the second question Item Design: Key Issues Checklist • Are the questions double-barreled? – Do you believe McDonald’s offers fast and courteous service? • Are responses mutually exclusive and collectively exhaustive? • Are the questions too complex? – Of the total number of miles you have driven during the past month, approximately what percentage was for driving to and from work? • Are the question implications implicit or explicit? – Are you in favor of UW Business School providing every student with a laptop? • …even though it will mean a substantial increase in tuition? Questionnaire Organization • Introduction: Basic who, what, when, where, why, how We are a group of UW Executive Business School students working on a school-related project. We are conducting a marketing research survey to determine interest in a new child safety product. Your attitudes and opinions will be valuable in designing a child safety product that is best suited to satisfying market needs. We are not selling anything, and you will not be solicited later. Your responses will be used for research purposes only, and the input you provide will be both completely anonymous and confidential. We anticipate that this survey will take you less than 5 minutes to complete. THANKS IN ADVANCE FOR YOUR PARTICIPATION Questionnaire Organization • Body—Theories of Organization – Typically begin with one or two interesting but nontaxing/non-invasive “warm-up” questions – In general order items in terms of descending usefulness or importance to you – Group questions that have similar content or similar form (e.g., Likert scale items) together – Leave potentially objectionable and sensitive items to the end of the survey – Thank respondents for participating Questionnaire Organization • Body: Typical Item Sequence – Screeners/qualifying questions “Have you been skiing in the past 12 months?” – Warm-ups “What brand of skis do you own?” – Beginning body/transitions “What features do you look for in skis?” – Middle—more complicated/difficult questions “Please rate the importance of the following ski attributes in your ski purchase decisions.” – End—classifications, demographics, and sensitive items “What is the highest level of education you have obtained?” Critical Reviews and Pretests • Always pretest among a small group of people similar to those future respondents of the actual survey – Get qualitative feedback on troublesome questions, ambiguities, etc. after the pretest group completes the survey If you don’t have the resources to pilot test your survey then don’t do the study. • For each item, ask “Is this question really necessary or merely interesting?” • Keep it short and simple Conjoint Analysis • Technique to understand how consumers make trade-offs among attributes or characteristics of products or services which deliver the desired benefit. • To help management understand the implications of such trade-offs for design of product offerings • Conjoint Analysis also provides a measure of how important the attributes are to the customer • Conjoint Analysis is now quite widely used by marketing research companies, consulting companies, etc. How Does It Work? Suppose I want to know what factors are important to you in your evaluation of cars. You can think of a car as a bundle of features or attributes e.g. size - small, mid-size, full size; fwd/rwd; mpg - 20, 25, 35; engine - 4 cl, 6 cl; American/Japanese/European; many dealers/few dealers; $15k/20k/25k; low interest financing or cash back, etc. Now how do you go about buying a car - you have an approximate budget, you want certain features, you would pay more for a fwd than a rwd, more for an automatic than manual, etc. Assume there is no optimum car or if there is one it is out of your budget. Some of the attributes listed above are not directly comparable. You like the prestige of a German car (say) but you may have to put up with the inconvenience of longer down-time if the car has to be serviced. You like fuel efficiency but you don't like the cramped feeling of a small car. And so on. HowDoes It Work? If you could assign a $ value to every attribute - how much more would you pay for a bigger engine, for prestige, for fuel efficiency, for size, etc. -then you could compute the equivalent $ value of the cars in your set and make a choice. However, in many cases this is not possible. You may make some mental computations and come up with an overall value or utility preference for the cars in your set. This overall value is presumably based on component values assigned to the attributes. How Does It Work? It is much easier for the consumer to provide overall preferences than the component utilities or values. Conjoint analysis takes these overall preferences and the attribute descriptions and decomposes the overall preferences into utilities for the attributes. The utilities of different attributes are comparable. These utility values provide the trade-offs among the attributes. From these values we can construct an importance measure for each attribute. Managerial Uses of Conjoint • After determining the contribution of each attribute, the researcher could – Define the object with the optimum combination of features – Predict market shares among objects with different sets of features – Isolate groups of consumers who place differing importance on features – Identify market potential for product concepts not yet available Commercial Applications • Commercial applications are across the board: – Consumer durables • Automobiles, refrigerators, car stereos, condominiums – Consumer non-durables • Bar soaps, hair shampoos, disposable diapers, clear gravy (!) – Industrial Products • Copy machines, forklift trucks, computer software, aircraft – Services • Car rental, credit cards, hotels, mass transit Steps in Conjoint Analysis 1. 2. 3. 4. 5. 6. 7. 8. 9. Develop the set of attributes Select the levels of each attribute Develop the set of stimuli to be used Obtain an evaluation (rating or ranking) of the stimuli from consumers Estimate the utility values for each level of each attribute Compute importance weights for each attribute Evaluate the trade-offs among attributes and which combination of features is most preferred Affiliate results across consumers Conduct market simulation Desirable Problem Situations for Conjoint • Product must be realistically decomposable into basic attributes. • Product choice tends to be a reasoned decision. • Factorial combinations of basic attribute levels are reasonable (i.e., some combinations are not unreasonable). Example: Carpet Cleaner Attributes Package Design Brand Name Price Good Housekeeping Seal Money-back Guarantee Levels A, B, C K2R, Glory, Bissell $1.19, $1.39, $1.59 Yes, No Yes, No Total number of possible product combinations = 108 A subset of 18 product combinations is selected for use such that attributes are uncorrelated (orthogonal). •Which 18? Commercial software available to pick subset. •Why 18? To allow sufficient degrees of freedom to estimate the model for each individual respondent. Experiment Design 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. P ack ag e D esig n B rand N am e P rice $ Good H o u sek eep ing S eal? A A A B B B C C C A A A B B B C C C K2R G lo ry B issell K2R G lo ry B issell K2R G lo ry B issell K2R G lo ry B issell K2R G lo ry B issell K2R G lo ry B issell 1 .1 9 1 .3 9 1 .5 9 1 .3 9 1 .5 9 1 .1 9 1 .5 9 1 .1 9 1 .3 9 1 .5 9 1 .1 9 1 .3 9 1 .1 9 1 .3 9 1 .5 9 1 .3 9 1 .5 9 1 .1 9 No No Y es Y es No No No Y es No Y es No No No Y es No No No Y es M o ney B ack G u arantee R esp o nd ent’s evalu atio n (rank ) No Y es No Y es No No Y es No No No Y es No No No Y es No No Y es 13 11 17 2 14 3 12 7 9 18 8 15 4 6 5 10 16 1 Conjoint Analysis Brand name | Glory Bissell | | $1.39 $1.59 Good Housekeeping Seal? 1.0-----| | 0 No Yes Money-Back Guarantee? 1.0-----| | 0 No Yes Utility | Utility ©2000 Prentice Hall 1.0 -----| 0 $1.19 Utility Utility 1.0 -----0 K2R| Retail Price Desirable Problem Situations for Conjoint • Product can be realistically described, verbally or pictorially. • Desirable new product alternatives can be synthesized from basic attributes. Conjoint Analysis Review • Tuesday, October 8, at 6:30 • To prepare: – HBR Article “Conjoint Analysis, A Manager’s Guide – Exercise, “Initech” (on the website) Advertising Research Three areas that should be addressed in the assessment of advertising effectiveness: I. the target (prospect) II. An understanding of the appropriate media and vehicles within media type for gaining the optimal reach and frequency within each target (prospect) group. Advertising Research III. An effective and actionable measurement system designed to assess: A) The quality of a particular advertisement (a.k.a. copy test) B) Baseline awareness levels and “lift” attributed to an advertisement or campaign, C) Media vehicles and weight of each, D) “Wear-in” and “wear-out” of an ad or campaign E) Predicted and actual sales responses to advertising The Target or Prospect • Challenge: Can the marketing department get to know the prospect so well that we are able to introduce him/her to the creative staff as though we were introducing a close friend? – The point is to give the creative staff an understanding of the prospect that will enable them to create an ad or campaign with a message that uses the prospects’ language and speaks to situations and needs of that group. The Target or Prospect Media Choices • Media choices for the advertisement or campaign will be based on the media habits of the target audience and prospects and the content of the communication. – The best choices are those media most often used by the prospect group The Target or Prospect Media Choices • Media habits are best determined through syndicated data such as Simmons Market Research Bureau. – The clearer the prospect profile is, the easier it will be to determine media habits. • Once the appropriate vehicles are chosen, a plan optimizing reach and frequency in light of the media budget and marketing strategy should be developed. Copy Testing • Important elements for evaluation include: – Reception of the advertisement by the prospect group, • Did they see or hear the ad at all? – Comprehension of the advertisement and, • Did they understand the message? Was it compelling? – A behavioral response (i.e. purchase). • Will it motivate a purchase? Copy Testing • Developmental Tool – Copy testing at an early stage in the development of a specific advertisement or campaign can assist in the improvement and/or redirection of copy and creative execution. • Developmental copy tests usually rely on qualitative research among the specified prospects for an ad or campaign. The stimuli are usually in rough or pre-finished form. Developmental Copy Testing Methods • Most developmental copy testing methods are qualitative in nature and rely on forced exposure techniques. The methods most commonly employed include: – Focus Groups – One-on-One Interviews (IDI’s) – Cluster Interviews-small groups (n=+/-3) – Ethnographies Developmental Copy Testing Methods • The goal of developmental copy testing methods is to evaluate: – Narrative playback (story line, action) – Message playback (attributes and benefits) – Brand name playback – Commercial or print ad comments (evaluative) – Product comments (evaluative) – Other diagnostic information as needed Copy Testing • Evaluative Tool – As an evaluative tool, copy testing allows comparisons among alternative ads, normative comparisons (depending on the firm conducting the research), and go/no go decisions. Copy Testing – Evaluative copy testing usually has a quantitative component and results in some form of scoring (depending on the method chosen). Again, the appropriate sample for the test will be drawn from among the specified prospects for an ad or campaign. The stimuli are usually in finished form. Evaluative Copy Testing Methods • Most evaluative copy testing methods are quantitative in nature and rely either on natural or forced exposure techniques. • The goals for evaluative copy testing are: – To measure recall of the advertisement and message following a defined period of time and/or, – Persuasive nature of the copy/creative execution – Other diagnostic information as needed Evaluative Copy Testing Methods - Print • Methods most commonly used for print advertising include: – Normal Exposure in Newspaper or Magazine – Simulated Normal Exposure-e.g.: dummy book or paper, “tipped” into magazine – Forced Exposure-e.g.: central location Evaluative Copy Testing Methods - Broadcast • The methods most commonly employed for broadcast advertising include: – Normal On-Air Exposure – Simulated Normal Exposure-e.g.: theater, cable – Forced Exposure-e.g.: central location Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Pros • Natural Exposure – Natural environment – Natural exposure – Can assess attention getting through clutter – Allows measures of recall and persuasion Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Pros • Simulated Natural Exposure – Near natural environment and exposure conditions – Less expensive than natural exposure – More control than natural exposure – Allows measures of recall and persuasion Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Pros • Forced Exposure – Excellent diagnostics – Suitable for testing rough ideas or executions – Ample opportunity to probe Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Cons • Natural Exposure – Uncontrolled environment – Security can be a problem – Personal interviewing can be expensive (sampling costs) – Pragmatic issues-schedules, costs – May not be able to collect extensive diagnostics Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Cons • Simulated Natural Exposure – The environment is not natural – Respondents may be sensitized to advertising – May not account for multiple exposure effects – May not be able to collect extensive diagnostics Evaluative Copy Testing Methods Broadcast and Print Pros and Cons Cons • Forced Exposure – No measure of attention getting ability – No measure of recall – No accurate measurement of persuasion – Artificial environment and exposure conditions – Consumer becomes critic – Assumes single exposure for brand preference change – Sensitized respondents Copy Testing - Other Issues • Anticipated “wear-in” and “wear-out” of the advertising is an issue to consider in the evaluation process. Some ads, depending on their nature and attention-getting ability do not have an immediate effect on recall or persuasion. There may be a period of time, the cumulative effect of multiple exposures, that is required before the ad becomes salient in the marketplace. – How long and what is the slope of the initial awareness curve? Copy Testing - Other Issues • Similarly, ads “wear-out” at differential rates, depending on the nature and saliency of each execution. – How long and what is the slope of the decay of the effectiveness of the advertising both on awareness and sales? Predicted and actual sales responses to advertising (Bruce Faaland’s favorite slide!) • Sales measures as a function of media spending, weight, base awareness and purchase rates, copy effectiveness, decay of advertising effect over time. • Relies on continuous tracking data in conjunction with data re: media weights, schedules, and copy evaluation. Decay Effect Base Media Flight Copy Testing-Other Methods • In recent years a variety of nontraditional methods of measuring responses to advertising have been developed. Copy Testing-Other Methods • Most of these methods are forced exposure methods and are developmental in nature. These include such techniques as: – Eye tracking measurement-Behavioral measure of attention given to the elements of an advertisement/print and broadcast – Galvanic skin response-Behavioral measure of reaction to broadcast advertising-measures changes much the same way a lie detector does. Problem-is a change an indication of a positive or negative emotional reaction? Copy Testing-Other Methods – Voice response (e.g.: VOPAN)-Behavioral measure of changes in voice inflection when talking about an advertisement. – Keypad or joystick second by second conscious reactions to advertising-Respondent responds to each element of the broadcast advertisement by moving a joystick or typing on a keypad to indicate the direction and degree of emotion s/he experiences continuously during exposure to the ad. Additional Resources • Marketing Research Textbooks – “Marketing Research: Methodological Foundations”— Churchill – “Marketing Research: An Applied Orientation”—Malhotra • Business Press – “Survey Questions: Handcrafting the Standardized Questionnaire”—Converse & Presser – “Mail and Internet Surveys: The Tailored Design Method”—Dillman – “The Survey Research Handbook”—Alreck & Settle • Harvard Business School Publishing Notes (www.harvardbusinessonline.hbsp.harvard.edu) – “Questionnaire Design and Development” (9-590-015) – “Market Research” (9-592-034)” Thank you!