This transcript is automatically generated by Descript, please excuse any mistakes.
Turi: We could not be more pleased than to be talking to John Russ today. John rust is one of the foremost experts. Anywhere in the world on measuring opinions, measuring how we think. Um, John, it’s a tremendous pleasure to, to have you here.
Um, you’re a fellow of the liver whom center for the future of intelligence and critically for our case, the author of modern psychometrics, the science of psychological assessment in its fourth edition, no less, um, for which congratulations, but thank you very much for joining us.
John: Well, thank you very much.
Turi: John. Can I start off by asking you what psychometrics are?
John: Well, psychometrics is the science of psychological assessment. It’s very old. It’s been around for thousands of years in ancient China. The emperor used to use psychometric methods. As we now know they were in order to assess the candidates who wanted to serve in this court.
Uh, it’s based on a set of scientific principles, which I’ll briefly say that reliability artists want to know. If you take a measure of a human characteristic, is it free from error with different people make the same judgment or is it likely to change dramatically? What day you make it? The second characteristic is validity. We may all agree that a person has a certain characteristic and say, well, people with this characteristic are going to be really good at its job. I’m recruiting for this job, but we may be wrong just because we both agree with each other. It’s a reliable measure. It doesn’t mean that it’s valid. And thirdly, standard orientation. When we go as a result of a assessment, what does it mean? What will someone with this particular score on a test actually be able to do? How do they compare with other people? Who’ve also got scores on this test. Are they in the highest group or the lowest group and not just which group of, and whether it’s more or less so than other people, but what skills are they likely to do?
Could they actually do the job even though they may not be the best person at it? Um, last but not least, certainly freedom from bias. Uh, all assessments have the potential to be biased, whether it be in terms of gender, ethnicity, race, social class, or whatever, and the consequences of using any form of assessment, even if it’s just my personal judgment in a biased way, maintenance search, we’re going to beat stopped, being unfair to people, depending on what group they’re in. And of course this is wrong in itself, but it’s also illegal in most countries to have this form of bias. So within psychometrics, we have the scientific methods of assessing all these four characteristics and ensuring that when we make decisions about other people, whether it’s in education work or any other framework, we aren’t doing it in the best possible way. So there have been enormous advances in science over the last few hundred years. And psychometrics is in many ways, one of the most important or influential areas of applied psychology after all the way assessed from the cradle to the grave in terms of our job, our education, our mental health, and so on. So the potential, if you get it wrong, can be quite serious. And there’ve been many examples of where people have got it wrong. Uh, some of the earliest psychometricians, for example, were influential in statistics and also quite involved with Charles Darwin. Uh, in many ways, pharmacists golden Darwin’s cousin is seen as the origin of psychometrics, but he was in many ways of dreadful influence. He was the developer of the idea of eugenics that is that we should be, um, making sure that we improve the human race. He thought, but its way of improving the human rights was trying to get rid of all the people he thought, what a bridge get. Uh, so a lot of these policies, he was suggesting sterilization and so on.
Have really, really bad effects. On the other hand, it’s a science, which has improved our educational system, no ends. Um, before a hundred years ago, it was just, um, who your family were, uh, while you will know how much money your parents had would decide to what to school or not. It was the introduction of the safety t-tests or talk with you. And so on, it gave everybody an opportunity to get an education. So it’s an remarkably important area of science, particularly of psychology. We can get it right. We can do lots of good. If you get it wrong, there can be a disaster.
Turi: I want to dig a little bit deeper into both the getting it right. And the getting it really tremendously wrong as we go through. And it’s a huge concern. An interest of yours, the ethics of, of psychometrics. But can I take you back, um, to that, to, to ancient China, um, and ask you to give us a little bit of the history of psychometrics, cause it’s fascinating, you, there, there’s a suggestions that Hippocrates that Galen, that Aristotle thought about humans in this sort of graph the way this way of sort of measuring them against scales. Um, of course it’s instrumentalized and, and used as an, as an, as a, as a weapon of state. Was a tool of state by the Chinese, but can you give us a little bit of that history?
John: Yes. Well, it’s interesting. The ideas have developed both in the east and the west coast. Um, uh, Galen Hippocrates, even Aristotle work several hundred years in many ways, almost a thousand years later than the early Chinese, um, thinkers on this topic. Uh, the idea of assessment for the entrance coped with fairly central those few thousand years ago. They would assess skills, which weren’t just reading and writing and arithmetic, but they were there that also included things like archery and horsemanship. So these were seen as important. There were, uh, characteristics such as honesty, um, loyalty and so on, which are recognized as important and also were assessed and virtues.
Such as honor being honorable and so on. Uh, there were six of each of these, so they even had a classification system, but then of three things that 18 different characteristics. So, uh, All of these were assessed. And in order to do this, they had examination halls. They had a principal, which said there had to be two people assessing each of the characteristics and they had to assess the extent to which they degree agreed with each other. They made sure that there was no cheating. So a lot of these principles, which began then followed on. And rather interestingly, it was the British east India company in Shanghai, but learnt about the Chinese methodology. They introduced it, that their own coven is in Bangor and it was on the British nationalized, the east India company, they were introduced and formed the basis of recruitment to be Indian civil service system that was subsequently copied by the Americans and the French.
Turi: Which brings us to the meritocratic fuel furiously meritocratic French administration system to get into the . It sounds like an exact repeat of the Chinese Imperial service. Um, the civil service exams in the UK that, that not much has changed in a few thousand years.
John: Uh, well, that hasn’t changed what people are trying to do. They have some major errors which are made. For example, these systems are good at looking individual differences within a group. So if you have a lots of people who have to say education or opportunities for education, they can quite easily discriminate between those who are able to learn and make.
Good progress. Um, those who are having difficulty learning lots of the early tests of this to identify learning difficulties where it goes wrong is that if groups have different education, then the two things get confounded. So if I apply a meritocratic system, say to two groups, which are called a and B any arguments, uh, and I have the same. A pass mark for each. They’re not going to finish up selecting more people in one group than I am in the other. And this has nothing to do with individual differences in the. Ability to learn within the group is because one of the groups may well have a lower overall level, and it can be the case of quite small differences between these groups. It can suddenly lead to quite large inequalities in recruitment and recruitment at primary school, secondary school, university to job, and eventually it’s into a career and promotion and a career. Uh, it has the potential to introduce quite big. Um, disadvantages for the group, which has been seen less preferably by the society in which they live, because one society to another, or one group may be successful in one society, but the, at the bottom of the heap and then other society.
Turi: Can we jump into this question of intelligence, which is most in the, in the instances that you’ve brought up the theme that’s being measured, um, before we get to personality type tests and the Myers Briggs, or the big five and the others. But if we can, we start with intelligence and look at sort of what the history of. It’s is peculiar question to ask. What is the history of intelligence? What I think I’m asking is how have we understood intelligence and how has that changed over the period that psychometrics around intelligence have been around?
John: Well, Darwin, um, with, I have to say unashamedly racist, I mean, you only have to read the defense of maps. Let’s see quite clearly states that he thinks that the civilized nations are more ahead than the lesser races. He actually uses that. Um, as opposed to savages, uh, People have argued that this was sort of justified at the time, because that’s the way Europeans were thinking at that time. But in fact, that’s not the case. There were plenty of other people who thought differently about it, but he of course, was very influential. Uh, Colton picked up this idea. The T also we’re very into evolution and you can still see the impact of the variety of today in an area called evolutionary psychology, which if I popular, we have a certain group of bull right-wing psychologist.
I would have to say. Um, there’s a certain compulsion to it. You can see why they believe this, but most of us treat it as a joke. Lots of people it’s very serious. So the idea that, um, men in the ancient world, or in the jungle had to run about and chase things to catch animals or women would sit at home cooking. And this means that men are genetically better at running around catching things. And women are better at sitting at home. Cook it. on. I spoke to the idea that men are better at fighting, so they should be doing all the fighting and the women are better at chatting because they set up home cooking. And from the, it’s a small leap to be ideal.
That’s what they’re meant to do. And therefore we shouldn’t try and change things because of the genes are said, that’s how we are meant to be. Um, well, We now know that’s wrong. I mean, many people a hundred years ago said, well, God meant us to fly. He’d have given us wings. They shouldn’t be inventing airplanes. I remember when I was a kid, my vicar said, there’s no way humans will ever get into space. This was a long time ago because they weren’t. But to be there, it seems a very compulsive way. A lot of people find satisfying clearly golden and Darwin fit into that category. So once you bring evolution into it, you then had a group of people who said, well, that’s really interesting because that explains why people like me are so clever. And my all leaf foreigners also stupid. We are meant to be in charge of everything because we’ve inherited the appropriate genes to be in charge. So you can see it’s the foundation of a lot of right wing and outright views. And it’s ideas of intelligence studies of intelligence differences between races have been really quite central to a lot of the right wing thinking it still is today.
It’s still quite so that’s a certain with obsession with them. Um, even as recently, as 30 years ago, giving by HOA again, um, the bell curve with written tried to show that’s, um, people setters were breeding too much and lowering the average IQ of society and how dreadful it was that stupid people were allowed to have more children. And the intelligent people were somehow restricting the number of children. They had 30 minutes only, well, almost into the 21st century. Uh, we now know really that’s, can’t be true. Um, I’m not saying this just from a bias point of view. I I’m biased. Of course, of course, because I come from a fairly large multi-racial family, but, uh, We know it’s not true because. The twin studies, which they fed supported. The idea that it’s identical twins are more similar than non identical twins and their IQ scores, which proves it’s genetic. And they argued that not only is it genetic, but it’s largely genetic and the differences you’ll get between groups are twos. Are too big to have been explained by our environmental factors. We now know from the work of Jim Flynn, who sadly died a few months ago, but, um, this can’t be true. The Flynn effect has shown that scores on IQ tests. I’m not saying intelligence scores on IQ tests have gone up by about three IQ points per decade for the last hundred years. This means effectively that we’re now 30 IQ points higher than we were a hundred years ago. Uh, a lot of people don’t really get that. So I think we really have become more intelligent. And of course the young people say, well, of course we more, some grandparents who would
Turi: probably contest this across, but,
John: but Jim was a bit more insightful of that to the circle. If their scores are true, that would mean that. 50% of my grandparents or great grandparents would have been, uh, uh, big diagnosed of being intellectually subnormal and he just doesn’t believe that with true free Sebastopol reasons. Uh, the more immediate point is that looked at it in this way can take the American example to where most of the race IQ debates come. The average IQ of black Americans in the year, 2000 is higher than the average IQ of white Americans in the year 1917. Now this has to be an environmental effect. It’s much bigger than any differences they were claiming they had between the races there’s fat on it. So it’s sufficient to show that it’s not the case, that we need to have an genetic explanation for differences between groups and intelligence. I know I’ve labored this point, but I think it’s such an important one as well, saying over and over again. Um, it’s important, partly because as a reaction to the pressure of the race IQ debate in the 1980s and nineties, a lot of psychology degrees stopped teaching psychometrics altogether. Uh, and that was not good because how are we going to prove people wrong? If you don’t actually discuss the issue, which they’re putting that it left. Um, lots of prejudices, um, challenged. Um, we have a situation in the UK, for example, where black kids until at least a few years ago and still to some extent, and then let’s set it particularly black boys underachieve at school, particularly in mathematics.
It was a huge problem. And it was quite a big one. Um, unless you actually are willing to. Dismiss the genetic argument, you’re less likely to anyone or do anything about it. If you leave this unchallenged, I’ll have the white. Um, majority race, even the kids say, well, of course they are, you would expect that because they’re black. And we write these studies, which say they won’t do so well if you take the alternative view, but it’s not genetic, it’s environmental. Um, think in a different way you think, well, how is this injustice come about? How can we be treating a group in our society so badly? What can we do about it? This really had to be challenged. And increasingly now it is, I think we’ve moved on since the 1980s and nineties, but it’s certainly been part of my fight for the last 40 years or trying to get these issues addressed and certainly not talking about them. Hasn’t helped
Turi: John. You, you, you couldn’t have given a more empowered impassioned, but also clear description of this important fourth. Factor of, of psychometrics, which is the freedom from bias. It’s the perfect example of what happens when your bias creeps into notions of validity or where your notions of validity are, are wrong. Um, what tremendous social, cultural, political impact. Can I ask us to move on to the other area of psychometry, which would be. Personality tests. Um, again, a very messy fraught, ethically challenged, um, and fascinating space. Can I ask you to kick off if that’s okay, John, by helping us understand the four big buckets of theories -you’ve got psychoanalytical was Sigmund Freud. You have the humanistic, the social learning approach, and then the genetic, which obviously feeds into much of the work that you do. Can you give us a quick sort of, sort of the horizon of those four ideas?
John: When you’re assessing abilities, don’t call it intelligence called disabilities. Right? That’s much more of a consensus. So when you’re assessing personality, personality, theories are. Part of our philosophy. So the way in which we look at personality, it depends very much on who we are on our religion, our temperament on various aspects of it and the consequences, Sarah, very large number of different personality theories and the different personality theories, not as simply a case of one being true and another false it’s often the question of different layerings that is people are looking at human individual differences in different ways.
Uh, the, um, psychoanalyst, for example, under Sigmund Freud had a very, still very influential model. Freudian psychology is actually pretty influential as well as psychometrics, uh, not in psychology anymore, but in advertising and marketing, you’ll see endless influence of these types of ideas and in social sociology as well. They Freud’s ideas. Um, Took the idea of there being an it, that is a basic drive that was, we were born with, which was something that was aggressive, angry, sexual. It was something that was out of control. And in the first or three or four years of life, a child has to learn to control this. It does it with the development of what’s called the ego. The reality principle that enables the child to bring its unconscious motivations from the ed under control. So that’s the model that says, we start off with the forces, which are trying to make us have sex or eat or drink, or look after ourselves. We learn to control them and a good society as well, in which we do this properly, you can contrast that with Carl Rogers, for example, and another group of early psychologists who have a completely different approach to it. Carl Rogers also started off as a therapist. But he take like the politics though. The authoritarian nature, Freudian psychoanalysis, they thought, why should we be. Telling people what to do. Why is it the job of parents to say, this is what you’ve got to do? Why is it that the parents are somehow all the things that keep these evil forces coming from it under control Freud’s idea.
According to rod with something like a regional standard on people are born evil and then become good. So Roger’s idea was the opposite. He had this idea. That the evil that we do is taught to us by our parents. If we’re allowed to evolve in a sort of California open source environment, we would just naturally be good. We are born good and not unconditioned to be evil. By society. So that was the origin of a lot of, um, free schooling techniques. You know, the pretty popular ones that have been around where children are allowed to do what they want. If you let them do what they want, they will be good, massive difference in philosophy. And it’s associated with different ideas of what personality is, how it’s a false personality theory is just so fast to subject. And each one of these has come up with a different approach.
Turi: So that leads into the humanistic, um, understanding of personality. How would you describe the social learning approach? I suppose it’s a continuation of that.
John: Uh, it sends a continuation, but it has less of the as that, but that’s how it seems to be, uh, social learning. It’s all about learning or imitation. So this means that society is entirely responsible. It’s similar in that way, uh, to how we grow up. So it’s nothing to do with our genes. It’s nothing to do with the first, even the first five years of life, uh, whatever we do, um, we can learn to do differently. And it’s the job of society to find good ways of training people. We get that, for example, cognitive behavioral therapy. Um, just based on the seminar idea that if someone is depressed, a psychologist, let’s talk, the psychologists say that depressed, except they have negative cognition. The psychological way of saying you’re also thinking things like, oh, life is not worth living to doing that.
And you have to replace it with positive cognitions. Such as things are getting better all the time. Right. Um, it may have been a bad day, but tomorrow will be fine. And so on. So that’s a question of imposing, um, learning particular ways of thinking.
Turi: And an example of that would be. This idea of the social construct is particularly important for theories, personality, theories, based on social learning. So the idea that boys are brought up and reinforced with trucks and footballs and girls are brought up and reinforced with Barbies and pink dresses, et cetera. Yes. Is nothing gendered in our personality. And the fourth would be this genetic approach, which would be, which has, as you say, is deeply problematic in all sorts of different ways. Um, but which forms the basis or at least part of the basis of quite a lot of the psychometric work that you’ve been doing it as a psych psychometry has been involved with these last, these last few years, what would its underlying principles? How would you describe its underlying principles?
John: Well, yeah, it’s evolved I mean, it’s a really important because it’s the field in which psychometric, the mathematics of Sacramento, I should say evolved. It was Golten who invented statistics invented the standard deviation, Cole Pearson. All these people are now bound at university college in dialogue, having the lecture theaters named after them were the early psychometricians things like correlation, coefficients, and so on all day out to them.
So they were very influential. In developing our methodology. Uh, and there’s no way psychometrics to go to a Polish, you know, regression and statistical modeling cause toilets foundation of practically all statistical science today. Uh, but yes, it has been influential in psychometric, but it’s, uh, you can use the models without having the theory.
Turi: So, um, Golden’s fascinating because. I suppose for most laypeople, our idea of psychometric tests or psychographic tests are things like the Myers-Briggs test, which you could find online, I think is, is downloaded or done tens of millions of times a year by the broader internet that people discover whether they are an IMTJ or an NFP or whatever it is. Um, every time I do the test, it very slightly, and I, I like my, whatever. Category I’m put into less, but, um, there is, there’s an interesting idea here, which while there’s a, perhaps an element of Hocus Pocus around the way that Maya’s Briggs has got this sort of, um, place on this particular pedestal, there has been lots of work around personalities using those kinds of tools. You, you, you perhaps you might talk to us about Cacharel perhaps you might also talk to us about Golten who, um, who starts off trying to. Um, build a model of what a personality might look like by looking at all the words and adjectives that humans use to describe each other. It seems like such a, an inventive, imaginative way of trying to decode what personality might be.
John: Yes, exactly. The Golden’s lexical hypothesis. The idea is quite simply this, that if people vary for one another, we should have a word to describe what the variation is. If we’ve got a word and it exists in more than one language, that is a written prescription, then how come. It existed all these languages, if it’s not important globally important, regardless of your background and culture. So if you can actually identify all of the words, which everybody uses such as sociable, outgoing, um, anxious, uh, Accurate and so on and somehow develop a statistical model or a test, which will cluster these together in a particular way, or have the foundation really of modern sacrament of a psychometric methods. And it is actually the basis of most modern psychometric testing methods of personality that interestingly. Because this all developed in the early 19th century, they were worried at the time that psychology wasn’t being treated properly, as a science said, uh, you’re not willing to evolve out of philosophy.
That’s called moral philosophy at one time.
Turi: That’s the perennial concern of psychology that it’s not being treated as a proper science.
John: Yeah. I mean, physics used to be called natural philosophy as well. Borrowed philosophy came later, but they, they thought people, many people thought it was the job of the church, uh, or any religion, whatever religion you’re in to, um, deal with these ethical issues. So there were plenty of words which are used to describe people particularly good and bad, which they deliberately left out of this particular analysis. Which I think is a rather intriguing aspects of so-called objectivity because after all these are the words we use, um, Skinner argued that uh, psychology was the science of prediction and control of behavior. Well, okay. Saying someone is an extrovert enables you to protect a good behavior, but also saying that they’ve got all bad, also enables you to protect or control behavior. So we notice psychologists that deliberately left out one of the main predictors, not necessarily off web the people’s behavior was good or bad, but whether people would treat them in a particular way, it was deliberately excluding some of its own subject. This is interesting because modern AI techniques, um, don’t do that. And AI doesn’t cannot ignore things. It’s not a human being, it doesn’t have our ethics, so it can quite rapidly identify whether someone’s behavior it’s up to, that would be considered to be good or bad. Because it enables it to predict and control what they’re likely to do has no ethics and therefore is better at predicting suburbs ethics that people who have ethics,
Turi: um, so Goldman starts with thousands and thousands of words, which she groups into synonym buckets, one, whatever it might be friendly, agreeable social. Gentle kind. I don’t know what I’m, I’m sort of aggregating them perhaps wrongly. Um, and eventually we end up 30 years ago or so with, um, with a model which suggests that there in fact, 16 personality types now. Ray. Cattell um, Which is a model which still persists today. The Myers-Briggs is based on, uh, on 16 factors or 16 types of personality as well. But actually what has emerged sort of bubbled up to the top and sort of perhaps, because the only psychometric personality test that there are around which there is broad agreement, is this thing called ocean or big five? Can I ask you to describe what it is and also tell us why, why it’s one.
John: Well it’s well, uh, yes, well, it costs, there were so many different psychological theories around for the auction model. It started about 1990, actually. Um, all the theories were producing different tests. Uh, um, there were a large number of different tests in these, uh, in the septic native people are starting to realize that ability tests on their own what, the best way of recruiting staff you wanted to on the tests. But as obvious you can give it to a large number of people. So we can’t interview everyone necessarily. So you want to have some tests which are quick and easy to administer. Um, there was starting as being carried out to find out which of these personality tests were the most effective at cruising people for jobs. What character traits say extroversion was that necessary for sales was attention to detail important for airline pilots under thriller obvious, but you can see that they’re all these basic things rules running here. Uh, they. Looked at the literature. And I found that there was a huge diversity of literature.
And some moment at that time introduced the idea of what’s now called Metro analysis, which is a way of taking all the data together from say a hundred different studies and combining it. It’s a bit like what journalists do. I mean, journalists just tend to read all the scientific studies, then write a paper and say, this is my idea, which is sort of a better analysis, but we’re trying to take it a bit further. We’ll say we’ll actually take the data from all these studies and statistically combine it. Now, in order to do this, you need to there to measure all of the personality traits on enough of these studies to be able to make it work. And there were quite a lot of these tests. All of them have something that looked like an extroversion scale in them.
So they would say, well, regardless if it’s the Myers-Briggs or the 16 PF or any other scale of BQ, we’ll just take their extroversion scale and melt all of the extroversion of calls from these various studies, uh, giving us a sample size of several thousand and see whether overall taking these studies together. It does predict success in sales. And you can do the same with stress ability to work under stress. It’s a, quite a common personality traits. It’s actually every single personality test in one way or another, regardless of the theory. So they managed to do that as well. They found that, yes, it was the case that people who are able to operate on the stress. Well, I’m more likely to, less likely to cause accidents. Basically if they’re flying airplanes or doing a dangerous job. And they went on looking for more and more trades, uh, and some were easy. Uh, openness to experience turned out to be quite important. Can you consider new things or are you very conventional? Just prefer to do things in the usual way. Uh, can you attend to detail or are you sort of tactical or strategic? Uh, but by the time they got beyond five of these trays, they found they were coming from. Left left field all the time. There was no consensus on what the sex of the seventies tray would be.
My mom, all these personality tests. So they said, right, we’re going to analyze these five of this somehow just became the accepted way of doing it. There’s nothing magic about it. Anyone ever said there are only five personality traits or rather they’re Machiavellianism or narcissism that, uh, They stopped at five. And because these were the most useful five for use at work settings, because that’s where the research was carried out. These just became an all understood. Nobody was saying there are plenty of others and it’s actually, it’s utilitarian.
Turi: It’s that what’s the most efficient way of getting to information, which is useful for making. Predominantly hiring decisions. Let’s just, I realize we haven’t been through them, but open, but ocean or the big five personality tests, which many of you will have listed, have done is openness to experience C for conscientiousness, extroversion, agreeableness, and neuroticism. You’ve talked, you’ve talked about openness to experience and conscientiousness for pilots. For example, extroversion we’ve touched on. Can we talk to these two seemingly very loaded terms? The last two agreeableness and neuroticism? What, what I meant by those?
John: Okay. Well, if you look at the agreeableness, that’s the earlier test psychometric tests were largely developed by academics and they were developed using students. Uh, and agreeableness really is how well you get up with the other students. Right. Not how sociable, but you know, whether you thought they should be allowed to do what they want and, uh, you shouldn’t be telling them what to do. Uh, the opposite with what’s sometimes called tender mindedness. The, the opposite. That’s called tough mindedness. That is the ability to make tough decisions about people. When we started, um, applying this model within occupational settings, we found that quite a lot of very successful people were not agreeable. They were tough minded, particularly human resource professionals got low scores on agreeableness.
And if you actually look at the items in the test and think these are the people who have to fire you, of course, they have to be able to make tough decisions. So there again, um, we tend not to use the word agreeable, we’ll be using it in work-based settings. Most occupational work based tests to don’t call it agreeableness in, in the, um, ODPi for example, the Orpheus personality test it’s called authority. and neuroticism of course. Well, at that developed Isaac developed the term for work in a psychiatric hospital to identify neurotic patients. Right. Uh, because again, it was his model that was working and that we’ve done on students. It’s still called neuroticism rather, oddly, it still appears in the ocean, but in work-based settings, it’s a motion because that’s makes more sense.
Um, neurotic people clearly are more emotional. And it also enables you to say, well, it’s not big, not neurotic, isn’t it? The opposite. It’s actually big. Non-emotional it’s the opposite. So you can argue that people with low scores on this particular tray can’t recognize anxiety or emotion and others because they don’t experience it themselves.
Turi: Fascinating. Um, John, can I. Begin to wrap this with a big question about immutability, because, um, as you say, going back to your four key, um, requirements for a decent psychometric test reliability, validity, standardization, and freedom from bias, um, this reliability element, this standardization piece, the standardization piece here, um, it does it presuppose that these traits are. Profoundly immutable that somebody who scores highly on neuroticism or emotion or schools highly on an IQ test has got that capacity or has got that label forever. Is that, is that, is that right? Or am I, or is that too simplistic? A way of understanding the way we think about personalities,
John: lots of the traditional models would assume that particularly if you take a genetic approach, um, there are other approaches in integrity testing, for example, where things are based on choice. So you can, you could, my people have anger management courses, so you can change or levels of anger, for example, Uh, that said it. One of the ways in which tests are developed is you actually develop tests which are reliable, which means you will get the same test to the same people with a month apart. And you only choose items to go in the test if they are reliable where these of course means you are choosing items that don’t change. So in many ways, or actually design the test to pick up things which are endurable, right. I think a second more interesting aspect of it is, um, that. The personality testing using classical test theory is, is the bottle that came from ability testing, which was your score.
It’s a number of items you get, right. Uh, which is a bit odd in personality testing because you wouldn’t say giving the right answer to being an extrovert. It’s actually good to get. Right, right. Uh, it just means that you’re responding. And so an extrovert would all, if you get it wrong, you’re responding as an introvert with, uh, Modern psychometric method uses computer adaptive testing. Uh, which is a far more sophisticated way of doing it. And if I look at it from that point of view, one thing we can learn is that if someone gets a very high score on an extroversion test or a very low score, we can think that is pretty staple. That’s going to last well, but imagine if somebody gets an average score on it. There’s no way in classical testing of knowing whether or not a person with an average score is someone who is just bog standard, average on everything that is now the extrovert or introverted. You can actually get the same score if sometime on some things you’re extroverted and some things you are introverted because they both cancel each other out. It still feel with average score. You compare that with an ability to Netflix, for example, If you had 10 items in order of difficulty, uh, I think it was a classical test. You could get the first easy items, right. And the difficult problems wrong, and you get a score. I, the metal, you could also get a score of five.
If you got them alternately, that is, you’ve got some of the difficult items wrong and some of the easier sort of, some of the difficult items, right. And some of the easy ones wrong. You can still get to score in the mental. So a lot of psychometric techniques, the machine learning algorithms pick these up quite quickly can differentiate these. We’re getting far better looking at. What characteristics or whether someone’s characteristic is stable or whether it’s likely to be variable than we used to have.
Turi: John, one of the theories in, um, in child-rearing I say this as, as a, as a, as a father, um, has been that we should stop thinking and telling our children that they are clever or good at things we should always focus on, um, the work that they’re doing and their capacity to improve. It’s called the growth mindset in tech Silicon valley terms. I can’t remember what it’s called in parenting terms, but the idea is that as soon as we accept immutability or we embed the notion of immutability in the subject, they themselves limit, they limit themselves. Um, do you think there’s plausible truth there, or do you really actually think of the human being as pretty immutable in their qualities and characteristic?
John: I think the parent is making a lots of. You’ll take it. And lots of confidence, a and the parent and belief in what parents happens to believe is true. I think today that’s possible recognition of diversity. Let’s put it that way. So, I mean, I’m dyslexic, for example. Which meant I could find if accepted today. Um, but less when I was at school, I mean, I couldn’t, uh, Latin or French, so I couldn’t possibly get into Cape bridge. Uh, I couldn’t spell, uh, today I’m honored to be at the capacity of various other distinguished persons, far more distinguished, I should say, who are also suddenly when terms of spectrum disorder, autistic spectrum disorder.
We now recognize, I think a huge number of different ways in which. Um, people think, um, but. All of us make our own contributions by thinking in all these different ways, the sum is greater than the whole. So, yes, I suppose I’m being a bit anarchistic. I’m saying we shouldn’t stop telling parents how they ought to treat their children. I think having some parents don’t get exactly what you said, but I’m sure they’re doing that anyway. And other parents don’t different things. I mean, keep the dark diversity up that way you get more variation. It’s a bit like a biodiversity isn’t it eliminates a goalie species on the planet because they’re thought of as a weak color. And so maybe by spreading our values across the internet, we’re also eliminating ways of thinking too.
Turi: What a wonderful way of. Ducking that question completely. Um, and saying, rather than thinking about whether growth mindsets work or static mindsets work best, um, we should say that actually all these various different mindsets fit into an even broader spectrum of different ways of being different ways of thinking. Um, all of which makes up this great super organism that, uh, that is humankind. John’s such a pleasure talking to you. Thank you so much for walking us through this fascinating, problematic, super exciting, and obviously extremely, um, appropriate current topic of, uh, psychometric.
John: Okay. Thank you very much.