{"id":28,"date":"2020-08-26T15:23:07","date_gmt":"2020-08-26T15:23:07","guid":{"rendered":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/?post_type=chapter&p=28"},"modified":"2020-08-26T15:44:41","modified_gmt":"2020-08-26T15:44:41","slug":"chapter-5-measurement-of-psychological-variables","status":"publish","type":"chapter","link":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/chapter\/chapter-5-measurement-of-psychological-variables\/","title":{"raw":"Chapter 5: Measurement of Psychological Variables","rendered":"Chapter 5: Measurement of Psychological Variables"},"content":{"raw":"

Measurement of Psychological Variables<\/h1>\r\nWhat Is Measurement?<\/strong>\r\n\r\nMeasurement is the assignment of scores to individuals so that the scores represent some characteristic of the individuals. This very general definition is consistent with the kinds of measurement that everyone is familiar with\u2014for example, weighing oneself by stepping onto a bathroom scale, or checking the internal temperature of a roasting turkey by inserting a meat thermometer. It is also consistent with measurement throughout the sciences. In physics, for example, one might measure the potential energy of an object in Earth\u2019s gravitational field by finding its mass and height (which of course requires measuring those variables) and then multiplying them together along with the gravitational acceleration of Earth (9.8 m\/s2). The result of this procedure is a score that represents the object\u2019s potential energy.\r\n\r\n \r\n\r\nOf course this general definition of measurement is consistent with measurement in psychology too. (Psychological measurement is often referred to as psychometrics.) Imagine, for example, that a cognitive psychologist wants to measure a person\u2019s working memory capacity\u2014his or her ability to hold in mind and think about several pieces of information all at the same time. To do this, she might use a backward digit span task, where she reads a list of two digits to the person and asks him or her to repeat them in reverse order. She then repeats this several times, increasing the length of the list by one digit each time, until the person makes an error. The length of the longest list for which the person responds correctly is the score and represents his or her working memory capacity. Or imagine a clinical psychologist who is interested in how depressed a person is. He administers the Beck Depression Inventory, which is a 21-item self-report questionnaire in which the person rates the extent to which he or she has felt sad, lost energy, and experienced other symptoms of depression over the past 2 weeks. The sum of these 21 ratings is the score and represents his or her current level of depression.\r\n\r\n \r\n\r\nThe important point here is that measurement does not require any particular instruments or procedures. It does not require placing individuals or objects on bathroom scales, holding rulers up to them, or inserting thermometers into them. What it does require is some systematic procedure for assigning scores to individuals or objects so that those scores represent the characteristic of interest.\r\n

5.1\u00a0 Psychological Constructs<\/h2>\r\nMany variables studied by psychologists are straightforward and simple to measure. These include sex, age, height, weight, and birth order. You can almost always tell whether someone is male or female just by looking. You can ask people how old they are and be reasonably sure that they know and will tell you. Although people might not know or want to tell you how much they weigh, you can have them step onto a bathroom scale. Other variables studied by psychologists\u2014perhaps the majority\u2014are not so straightforward or simple to measure. We cannot accurately assess people\u2019s level of intelligence by looking at them, and we certainly cannot put their self-esteem on a bathroom scale. These kinds of variables are called constructs (pronounced CON-structs) and include personality traits (e.g., extroversion), emotional states (e.g., fear), attitudes (e.g., toward taxes), and abilities (e.g., athleticism).\r\n\r\n \r\n\r\nPsychological constructs cannot be observed directly. One reason is that they often represent tendencies to think, feel, or act in certain ways. For example, to say that a particular college student is highly extroverted (see below) does not necessarily mean that she is behaving in an extroverted way right now. In fact, she might be sitting quietly by herself, reading a book. Instead, it means that she has a general tendency to behave in extroverted ways (talking, laughing, etc.) across a variety of situations. Another reason psychological constructs cannot be observed directly is that they often involve internal processes. Fear, for example, involves the activation of certain central and peripheral nervous system structures, along with certain kinds of thoughts, feelings, and behaviors\u2014none of which is necessarily obvious to an outside observer. Notice also that neither extroversion nor fear \u201creduces to\u201d any particular thought, feeling, act, or physiological structure or process. Instead, each is a kind of summary of a complex set of behaviors and internal processes.\r\n\r\n \r\n\r\nThe Big Five<\/strong>\r\n\r\nThe Big Five is a set of five broad dimensions that capture much of the variation in human personality. Each of the Big Five can even be defined in terms of six more specific constructs called \u201cfacets\u201d (Costa & McCrae, 1992).\r\n
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Big Five Dimension<\/td>\r\nFacets<\/td>\r\n<\/td>\r\n<\/td>\r\n<\/td>\r\n<\/td>\r\n<\/td>\r\n<\/tr>\r\n
Openness to Experience<\/td>\r\nFantasy<\/td>\r\nAesthetics<\/td>\r\nFeelings<\/td>\r\nActions<\/td>\r\nIdeas<\/td>\r\nValues<\/td>\r\n<\/tr>\r\n
Conscientiousness<\/td>\r\nCompetence<\/td>\r\nOrder<\/td>\r\nDutifulness<\/td>\r\nAchievement Striving<\/td>\r\nSelf-Discipline<\/td>\r\nDeliberation<\/td>\r\n<\/tr>\r\n
Extraversion<\/td>\r\nWarmth<\/td>\r\nGregariousness<\/td>\r\nAssertiveness<\/td>\r\nActivity<\/td>\r\nExcitement Seeking<\/td>\r\nPositive Emotions<\/td>\r\n<\/tr>\r\n
Agreeableness<\/td>\r\nTrust<\/td>\r\nStraight-forwardness<\/td>\r\nAltruism<\/td>\r\nCompliance<\/td>\r\nModesty<\/td>\r\nTender-Mindedness<\/td>\r\n<\/tr>\r\n
Neuroticism<\/td>\r\nWorry<\/td>\r\nAnger<\/td>\r\nDiscouragement<\/td>\r\nSelf-Consciousness<\/td>\r\nImpulsivity<\/td>\r\nVulnerability<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/figure>\r\nThe conceptual definition of a psychological construct describes the behaviors and internal processes that make up that construct, along with how it relates to other variables. For example, a conceptual definition of neuroticism (another one of the Big Five) would be that it is people\u2019s tendency to experience negative emotions such as anxiety, anger, and sadness across a variety of situations. This definition might also include that it has a strong genetic component, remains fairly stable over time, and is positively correlated with the tendency to experience pain and other physical symptoms.\r\n\r\n \r\n\r\nStudents sometimes wonder why, when researchers want to understand a construct like self-esteem or neuroticism, they do not simply look it up in the dictionary. One reason is that many scientific constructs do not have counterparts in everyday language (e.g., working memory capacity). More important, researchers are in the business of developing definitions that are more detailed and precise\u2014and that more accurately describe the way the world is\u2014than the informal definitions in the dictionary. As we will see, they do this by proposing conceptual definitions, testing them empirically, and revising them as necessary. Sometimes they throw them out altogether. This is why the research literature often includes different conceptual definitions of the same construct. In some cases, an older conceptual definition has been replaced by a newer one that works better. In others, researchers are still in the process of deciding which of various conceptual definitions is the best.\r\n

5.2\u00a0 Operational Definitions<\/h2>\r\nAn operational definition is a definition of a variable in terms of precisely how it is to be measured. These measures generally fall into one of three broad categories. Self-report measures are those in which participants report on their own thoughts, feelings, and actions, as with the Rosenberg Self-Esteem Scale. Behavioral measures are those in which some other aspect of participants\u2019 behavior is observed and recorded. This is an extremely broad category that includes the observation of people\u2019s behavior both in highly structured laboratory tasks and in more natural settings. A good example of the former would be measuring working memory capacity using the backward digit span task. A good example of the latter is a famous operational definition of physical aggression from researcher Albert Bandura and his colleagues (Bandura, Ross, & Ross, 1961). They let each of several children play for 20 minutes in a room that contained a clown-shaped punching bag called a Bobo doll. They filmed each child and counted the number of acts of physical aggression he or she committed. These included hitting the doll with a mallet, punching it, and kicking it. Their operational definition, then, was the number of these specifically defined acts that the child committed in the 20-minute period. Finally, physiological measures are those that involve recording any of a wide variety of physiological processes, including heart rate and blood pressure, galvanic skin response, hormone levels, and electrical activity and blood flow in the brain.\r\n\r\n \r\n\r\nFor any given variable or construct, there will be multiple operational definitions. Stress is a good example. A rough conceptual definition is that stress is an adaptive response to a perceived danger or threat that involves physiological, cognitive, affective, and behavioral components. But researchers have operationally defined it in several ways. The Social Readjustment Rating Scale is a self-report questionnaire on which people identify stressful events that they have experienced in the past year and assigns points for each one depending on its severity. For example, a man who has been divorced (73 points), changed jobs (36 points), and had a change in sleeping habits (16 points) in the past year would have a total score of 125. The Daily Hassles and Uplifts Scale is similar but focuses on everyday stressors like misplacing things and being concerned about one\u2019s weight. The Perceived Stress Scale is another self-report measure that focuses on people\u2019s feelings of stress (e.g., \u201cHow often have you felt nervous and stressed?\u201d). Researchers have also operationally defined stress in terms of several physiological variables including blood pressure and levels of the stress hormone cortisol.\r\n\r\n \r\n\r\nWhen psychologists use multiple operational definitions of the same construct\u2014either within a study or across studies\u2014they are using converging operations. The idea is that the various operational definitions are \u201cconverging\u201d on the same construct. When scores based on several different operational definitions are closely related to each other and produce similar patterns of results, this constitutes good evidence that the construct is being measured effectively and that it is useful. The various measures of stress, for example, are all correlated with each other and have all been shown to be correlated with other variables such as immune system functioning (also measured in a variety of ways) (Segerstrom & Miller, 2004). This is what allows researchers eventually to draw useful general conclusions, such as \u201cstress is negatively correlated with immune system functioning,\u201d as opposed to more specific and less useful ones, such as \u201cpeople\u2019s scores on the Perceived Stress Scale are negatively correlated with their white blood counts.\u201d\r\n

5.3\u00a0 Reliability<\/h2>\r\nMeasurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate that they work. If their research does not demonstrate that a measure works, they stop using it.\r\n\r\n \r\n\r\nAs an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions: reliability and validity.\r\n\r\n \r\n\r\nReliability<\/strong>\r\n\r\nReliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).\r\n\r\n \r\n\r\nTest-Retest Reliability<\/strong>\r\n\r\nWhen researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.\r\n\r\n \r\n\r\nAssessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at test-retest correlation between the two sets of scores. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.\r\n\r\n \r\n\r\nAgain, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.\r\n\r\n \r\n\r\nInternal Consistency<\/strong>\r\n\r\nA second kind of reliability is internal consistency, which is the consistency of people\u2019s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people\u2019s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities. If people\u2019s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants\u2019 bets were consistently high or low across trials.\r\n\r\n \r\n\r\nLike test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a split-half correlation. This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. A split-half correlation of +.80 or greater is generally considered good internal consistency.\r\n\r\n \r\n\r\nPerhaps the most common measure of internal consistency used by researchers in psychology is a statistic called Cronbach\u2019s \u03b1 (the Greek letter alpha). Conceptually, \u03b1 is the mean of all possible split-half correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of five. Cronbach\u2019s \u03b1 would be the mean of the 252 split-half correlations. Note that this is not how \u03b1 is actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of +.80 or greater is generally taken to indicate good internal consistency.\r\n\r\n \r\n\r\nInter-rater Reliability<\/strong>\r\n\r\nMany behavioral measures involve significant judgment on the part of an observer or a rater. Inter-rater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring college students\u2019 social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student\u2019s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers\u2019 ratings should be highly correlated with each other. If they were not, then those ratings could not be an accurate representation of participants\u2019 social skills. Inter-rater reliability is often assessed using Cronbach\u2019s \u03b1 when the judgments are quantitative or an analogous statistic called Cohen\u2019s \u03ba (the Greek letter kappa) when they are categorical.\r\n

5.4\u00a0 Validity<\/h2>\r\nValidity is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account\u2014reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people\u2019s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people\u2019s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person\u2019s index finger is a centimeter longer than another\u2019s would indicate nothing about which one had higher self-esteem.\r\n\r\n \r\n\r\nTextbook presentations of validity usually divide it into several distinct \u201ctypes.\u201d But a good way to interpret these types is that they are other kinds of evidence\u2014in addition to reliability\u2014that should be taken into account when judging the validity of a measure. Here we consider four basic kinds: face validity, content validity, criterion validity, and discriminant validity.\r\n\r\n \r\n\r\nFace Validity<\/strong>\r\n\r\nFace validity is the extent to which a measurement method appears \u201con its face\u201d to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively\u2014for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to\u2014it is usually assessed informally.\r\n\r\n \r\n\r\nFace validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people\u2019s intuitions about human behaviour, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory (MMPI) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them\u2014where many of the statements do not have any obvious relationship to the construct that they measure. Another example is the Implicit Association Test, which measures prejudice in a way that is nonintuitive to most people.\r\n\r\n \r\n\r\nContent Validity<\/strong>\r\n\r\nContent validity is the extent to which a measure \u201ccovers\u201d the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people\u2019s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.\r\n\r\n \r\n\r\nCriterion Validity<\/strong>\r\n\r\nCriterion validity is the extent to which people\u2019s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. For example, people\u2019s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people\u2019s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people\u2019s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.\r\n\r\n \r\n\r\nA criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People\u2019s scores on this measure should be correlated with their participation in \u201cextreme\u201d activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. So the use of converging operations is one way to examine criterion validity.\r\n\r\n \r\n\r\nAssessing criterion validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982). In a series of studies, they showed that college faculty scored higher than assembly-line workers, that people\u2019s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty, Bri\u00f1ol, Loersch, & McCaslin, 2009).\r\n\r\n \r\n\r\nDiscriminant Validity<\/strong>\r\n\r\nDiscriminant validity is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people\u2019s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.\r\n\r\n \r\n\r\nWhen they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people\u2019s scores were not correlated with certain other variables. For example, they found only a weak correlation between people\u2019s need for cognition and a measure of their cognitive style\u2014the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of \u201cthe big picture.\u201d They also found no correlation between people\u2019s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.\r\n\r\n \r\n\r\nKey Takeaways<\/strong>\r\n\r\n\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.\r\n\r\n\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (inter-rater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.\r\n\r\n\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Validity is a judgment based on various types of evidence. The relevant evidence includes the measure\u2019s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.\r\n\r\n\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.\r\n\r\n \r\n

References from Chapter 5<\/h2>\r\nCacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116\u2013131.\r\n\r\n \r\n\r\nNosek, B. A., Greenwald, A. G., & Banaji, M. R. (2006). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265\u2013292). London, England: Psychology Press.\r\n\r\n \r\n\r\nPetty, R. E, Bri\u00f1ol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary and R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 318\u2013329). New York, NY: Guilford Press.","rendered":"

Measurement of Psychological Variables<\/h1>\n

What Is Measurement?<\/strong><\/p>\n

Measurement is the assignment of scores to individuals so that the scores represent some characteristic of the individuals. This very general definition is consistent with the kinds of measurement that everyone is familiar with\u2014for example, weighing oneself by stepping onto a bathroom scale, or checking the internal temperature of a roasting turkey by inserting a meat thermometer. It is also consistent with measurement throughout the sciences. In physics, for example, one might measure the potential energy of an object in Earth\u2019s gravitational field by finding its mass and height (which of course requires measuring those variables) and then multiplying them together along with the gravitational acceleration of Earth (9.8 m\/s2). The result of this procedure is a score that represents the object\u2019s potential energy.<\/p>\n

 <\/p>\n

Of course this general definition of measurement is consistent with measurement in psychology too. (Psychological measurement is often referred to as psychometrics.) Imagine, for example, that a cognitive psychologist wants to measure a person\u2019s working memory capacity\u2014his or her ability to hold in mind and think about several pieces of information all at the same time. To do this, she might use a backward digit span task, where she reads a list of two digits to the person and asks him or her to repeat them in reverse order. She then repeats this several times, increasing the length of the list by one digit each time, until the person makes an error. The length of the longest list for which the person responds correctly is the score and represents his or her working memory capacity. Or imagine a clinical psychologist who is interested in how depressed a person is. He administers the Beck Depression Inventory, which is a 21-item self-report questionnaire in which the person rates the extent to which he or she has felt sad, lost energy, and experienced other symptoms of depression over the past 2 weeks. The sum of these 21 ratings is the score and represents his or her current level of depression.<\/p>\n

 <\/p>\n

The important point here is that measurement does not require any particular instruments or procedures. It does not require placing individuals or objects on bathroom scales, holding rulers up to them, or inserting thermometers into them. What it does require is some systematic procedure for assigning scores to individuals or objects so that those scores represent the characteristic of interest.<\/p>\n

5.1\u00a0 Psychological Constructs<\/h2>\n

Many variables studied by psychologists are straightforward and simple to measure. These include sex, age, height, weight, and birth order. You can almost always tell whether someone is male or female just by looking. You can ask people how old they are and be reasonably sure that they know and will tell you. Although people might not know or want to tell you how much they weigh, you can have them step onto a bathroom scale. Other variables studied by psychologists\u2014perhaps the majority\u2014are not so straightforward or simple to measure. We cannot accurately assess people\u2019s level of intelligence by looking at them, and we certainly cannot put their self-esteem on a bathroom scale. These kinds of variables are called constructs (pronounced CON-structs) and include personality traits (e.g., extroversion), emotional states (e.g., fear), attitudes (e.g., toward taxes), and abilities (e.g., athleticism).<\/p>\n

 <\/p>\n

Psychological constructs cannot be observed directly. One reason is that they often represent tendencies to think, feel, or act in certain ways. For example, to say that a particular college student is highly extroverted (see below) does not necessarily mean that she is behaving in an extroverted way right now. In fact, she might be sitting quietly by herself, reading a book. Instead, it means that she has a general tendency to behave in extroverted ways (talking, laughing, etc.) across a variety of situations. Another reason psychological constructs cannot be observed directly is that they often involve internal processes. Fear, for example, involves the activation of certain central and peripheral nervous system structures, along with certain kinds of thoughts, feelings, and behaviors\u2014none of which is necessarily obvious to an outside observer. Notice also that neither extroversion nor fear \u201creduces to\u201d any particular thought, feeling, act, or physiological structure or process. Instead, each is a kind of summary of a complex set of behaviors and internal processes.<\/p>\n

 <\/p>\n

The Big Five<\/strong><\/p>\n

The Big Five is a set of five broad dimensions that capture much of the variation in human personality. Each of the Big Five can even be defined in terms of six more specific constructs called \u201cfacets\u201d (Costa & McCrae, 1992).<\/p>\n

\n\n\n\n\n\n\n\n\n
Big Five Dimension<\/td>\nFacets<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Openness to Experience<\/td>\nFantasy<\/td>\nAesthetics<\/td>\nFeelings<\/td>\nActions<\/td>\nIdeas<\/td>\nValues<\/td>\n<\/tr>\n
Conscientiousness<\/td>\nCompetence<\/td>\nOrder<\/td>\nDutifulness<\/td>\nAchievement Striving<\/td>\nSelf-Discipline<\/td>\nDeliberation<\/td>\n<\/tr>\n
Extraversion<\/td>\nWarmth<\/td>\nGregariousness<\/td>\nAssertiveness<\/td>\nActivity<\/td>\nExcitement Seeking<\/td>\nPositive Emotions<\/td>\n<\/tr>\n
Agreeableness<\/td>\nTrust<\/td>\nStraight-forwardness<\/td>\nAltruism<\/td>\nCompliance<\/td>\nModesty<\/td>\nTender-Mindedness<\/td>\n<\/tr>\n
Neuroticism<\/td>\nWorry<\/td>\nAnger<\/td>\nDiscouragement<\/td>\nSelf-Consciousness<\/td>\nImpulsivity<\/td>\nVulnerability<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

The conceptual definition of a psychological construct describes the behaviors and internal processes that make up that construct, along with how it relates to other variables. For example, a conceptual definition of neuroticism (another one of the Big Five) would be that it is people\u2019s tendency to experience negative emotions such as anxiety, anger, and sadness across a variety of situations. This definition might also include that it has a strong genetic component, remains fairly stable over time, and is positively correlated with the tendency to experience pain and other physical symptoms.<\/p>\n

 <\/p>\n

Students sometimes wonder why, when researchers want to understand a construct like self-esteem or neuroticism, they do not simply look it up in the dictionary. One reason is that many scientific constructs do not have counterparts in everyday language (e.g., working memory capacity). More important, researchers are in the business of developing definitions that are more detailed and precise\u2014and that more accurately describe the way the world is\u2014than the informal definitions in the dictionary. As we will see, they do this by proposing conceptual definitions, testing them empirically, and revising them as necessary. Sometimes they throw them out altogether. This is why the research literature often includes different conceptual definitions of the same construct. In some cases, an older conceptual definition has been replaced by a newer one that works better. In others, researchers are still in the process of deciding which of various conceptual definitions is the best.<\/p>\n

5.2\u00a0 Operational Definitions<\/h2>\n

An operational definition is a definition of a variable in terms of precisely how it is to be measured. These measures generally fall into one of three broad categories. Self-report measures are those in which participants report on their own thoughts, feelings, and actions, as with the Rosenberg Self-Esteem Scale. Behavioral measures are those in which some other aspect of participants\u2019 behavior is observed and recorded. This is an extremely broad category that includes the observation of people\u2019s behavior both in highly structured laboratory tasks and in more natural settings. A good example of the former would be measuring working memory capacity using the backward digit span task. A good example of the latter is a famous operational definition of physical aggression from researcher Albert Bandura and his colleagues (Bandura, Ross, & Ross, 1961). They let each of several children play for 20 minutes in a room that contained a clown-shaped punching bag called a Bobo doll. They filmed each child and counted the number of acts of physical aggression he or she committed. These included hitting the doll with a mallet, punching it, and kicking it. Their operational definition, then, was the number of these specifically defined acts that the child committed in the 20-minute period. Finally, physiological measures are those that involve recording any of a wide variety of physiological processes, including heart rate and blood pressure, galvanic skin response, hormone levels, and electrical activity and blood flow in the brain.<\/p>\n

 <\/p>\n

For any given variable or construct, there will be multiple operational definitions. Stress is a good example. A rough conceptual definition is that stress is an adaptive response to a perceived danger or threat that involves physiological, cognitive, affective, and behavioral components. But researchers have operationally defined it in several ways. The Social Readjustment Rating Scale is a self-report questionnaire on which people identify stressful events that they have experienced in the past year and assigns points for each one depending on its severity. For example, a man who has been divorced (73 points), changed jobs (36 points), and had a change in sleeping habits (16 points) in the past year would have a total score of 125. The Daily Hassles and Uplifts Scale is similar but focuses on everyday stressors like misplacing things and being concerned about one\u2019s weight. The Perceived Stress Scale is another self-report measure that focuses on people\u2019s feelings of stress (e.g., \u201cHow often have you felt nervous and stressed?\u201d). Researchers have also operationally defined stress in terms of several physiological variables including blood pressure and levels of the stress hormone cortisol.<\/p>\n

 <\/p>\n

When psychologists use multiple operational definitions of the same construct\u2014either within a study or across studies\u2014they are using converging operations. The idea is that the various operational definitions are \u201cconverging\u201d on the same construct. When scores based on several different operational definitions are closely related to each other and produce similar patterns of results, this constitutes good evidence that the construct is being measured effectively and that it is useful. The various measures of stress, for example, are all correlated with each other and have all been shown to be correlated with other variables such as immune system functioning (also measured in a variety of ways) (Segerstrom & Miller, 2004). This is what allows researchers eventually to draw useful general conclusions, such as \u201cstress is negatively correlated with immune system functioning,\u201d as opposed to more specific and less useful ones, such as \u201cpeople\u2019s scores on the Perceived Stress Scale are negatively correlated with their white blood counts.\u201d<\/p>\n

5.3\u00a0 Reliability<\/h2>\n

Measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate that they work. If their research does not demonstrate that a measure works, they stop using it.<\/p>\n

 <\/p>\n

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions: reliability and validity.<\/p>\n

 <\/p>\n

Reliability<\/strong><\/p>\n

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).<\/p>\n

 <\/p>\n

Test-Retest Reliability<\/strong><\/p>\n

When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.<\/p>\n

 <\/p>\n

Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at test-retest correlation between the two sets of scores. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.<\/p>\n

 <\/p>\n

Again, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.<\/p>\n

 <\/p>\n

Internal Consistency<\/strong><\/p>\n

A second kind of reliability is internal consistency, which is the consistency of people\u2019s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people\u2019s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that that they have a number of good qualities. If people\u2019s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants\u2019 bets were consistently high or low across trials.<\/p>\n

 <\/p>\n

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a split-half correlation. This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. A split-half correlation of +.80 or greater is generally considered good internal consistency.<\/p>\n

 <\/p>\n

Perhaps the most common measure of internal consistency used by researchers in psychology is a statistic called Cronbach\u2019s \u03b1 (the Greek letter alpha). Conceptually, \u03b1 is the mean of all possible split-half correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of five. Cronbach\u2019s \u03b1 would be the mean of the 252 split-half correlations. Note that this is not how \u03b1 is actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of +.80 or greater is generally taken to indicate good internal consistency.<\/p>\n

 <\/p>\n

Inter-rater Reliability<\/strong><\/p>\n

Many behavioral measures involve significant judgment on the part of an observer or a rater. Inter-rater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring college students\u2019 social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student\u2019s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers\u2019 ratings should be highly correlated with each other. If they were not, then those ratings could not be an accurate representation of participants\u2019 social skills. Inter-rater reliability is often assessed using Cronbach\u2019s \u03b1 when the judgments are quantitative or an analogous statistic called Cohen\u2019s \u03ba (the Greek letter kappa) when they are categorical.<\/p>\n

5.4\u00a0 Validity<\/h2>\n

Validity is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account\u2014reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people\u2019s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people\u2019s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person\u2019s index finger is a centimeter longer than another\u2019s would indicate nothing about which one had higher self-esteem.<\/p>\n

 <\/p>\n

Textbook presentations of validity usually divide it into several distinct \u201ctypes.\u201d But a good way to interpret these types is that they are other kinds of evidence\u2014in addition to reliability\u2014that should be taken into account when judging the validity of a measure. Here we consider four basic kinds: face validity, content validity, criterion validity, and discriminant validity.<\/p>\n

 <\/p>\n

Face Validity<\/strong><\/p>\n

Face validity is the extent to which a measurement method appears \u201con its face\u201d to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively\u2014for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to\u2014it is usually assessed informally.<\/p>\n

 <\/p>\n

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people\u2019s intuitions about human behaviour, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory (MMPI) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them\u2014where many of the statements do not have any obvious relationship to the construct that they measure. Another example is the Implicit Association Test, which measures prejudice in a way that is nonintuitive to most people.<\/p>\n

 <\/p>\n

Content Validity<\/strong><\/p>\n

Content validity is the extent to which a measure \u201ccovers\u201d the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people\u2019s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.<\/p>\n

 <\/p>\n

Criterion Validity<\/strong><\/p>\n

Criterion validity is the extent to which people\u2019s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. For example, people\u2019s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people\u2019s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people\u2019s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.<\/p>\n

 <\/p>\n

A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People\u2019s scores on this measure should be correlated with their participation in \u201cextreme\u201d activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. So the use of converging operations is one way to examine criterion validity.<\/p>\n

 <\/p>\n

Assessing criterion validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982). In a series of studies, they showed that college faculty scored higher than assembly-line workers, that people\u2019s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty, Bri\u00f1ol, Loersch, & McCaslin, 2009).<\/p>\n

 <\/p>\n

Discriminant Validity<\/strong><\/p>\n

Discriminant validity is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people\u2019s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.<\/p>\n

 <\/p>\n

When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people\u2019s scores were not correlated with certain other variables. For example, they found only a weak correlation between people\u2019s need for cognition and a measure of their cognitive style\u2014the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of \u201cthe big picture.\u201d They also found no correlation between people\u2019s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.<\/p>\n

 <\/p>\n

Key Takeaways<\/strong><\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (inter-rater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Validity is a judgment based on various types of evidence. The relevant evidence includes the measure\u2019s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 The reliability and validity of a measure is not established by any single study but by the pattern of results across multiple studies. The assessment of reliability and validity is an ongoing process.<\/p>\n

 <\/p>\n

References from Chapter 5<\/h2>\n

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116\u2013131.<\/p>\n

 <\/p>\n

Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2006). The Implicit Association Test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp. 265\u2013292). London, England: Psychology Press.<\/p>\n

 <\/p>\n

Petty, R. E, Bri\u00f1ol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary and R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 318\u2013329). New York, NY: Guilford Press.<\/p>\n","protected":false},"author":13,"menu_order":5,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"part":3,"_links":{"self":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapters\/28"}],"collection":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/wp\/v2\/users\/13"}],"version-history":[{"count":3,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapters\/28\/revisions"}],"predecessor-version":[{"id":52,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapters\/28\/revisions\/52"}],"part":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapters\/28\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/wp\/v2\/media?parent=28"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/pressbooks\/v2\/chapter-type?post=28"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/wp\/v2\/contributor?post=28"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/digitaleditions.library.dal.ca\/researchmethodspsychneuro\/wp-json\/wp\/v2\/license?post=28"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}