Dr. Jutta Treviranus: The Human Starburst - Why AI Keeps Failing People at the Edges

Hyun Song (00:00) Hello and welcome to This Is HCD. I'm your host, Hyun Song. I'd like to acknowledge that I'm joining you today from the lands of the Gadigal people. I pay my respects to Elders past and present, and extend that respect to any and all Aboriginal and Torres Strait Islander listeners of this podcast. My pronouns are she/her, and my visual description is: I'm a woman with dark, medium-length hair, wearing headphones and a black jumper. And behind me, once again, is my very unexciting living room wall. Today I'm joined by Dr. Jutta Treviranus, founder and director of the Inclusive Design Research Centre at OCAD University in Canada, and one of the most influential voices in inclusive design. Jutta has spent more than four decades challenging the idea that we should design for the average person. In this conversation, we'll explore what this means for AI, data and disability and how we can start shifting the balance towards more benefit and less harm. Jutta Treviranus (01:04) Mm-hmm. Hyun Song (01:13) Jutta, thank you so much for joining me. Jutta Treviranus (01:16) It's a pleasure. And my pronouns are she/her. And as a visual description: I am an older white woman with gray hair and glasses. And a very nondescript background as well. Hyun Song (01:33) I'd love to start with one of the most impactful concepts that you've used in your work. You've spoken about the Human Starburst as a way to understand human difference. So for listeners who might not be familiar with that metaphor, can you explain to us what the Human Starburst is? Jutta Treviranus (01:52) Yeah, so I started in the field in '79 and ever since then I've been collecting this fairly informal ad hoc data set where I ask almost anybody that I meet, what is it that you need to thrive? What do you need to participate fully in society? And when I try to plot that, of course, because it's so multifaceted. And there's so many variations in what people have responded. The only way that I can do that is in what's called a high-dimensional multivariate scatterplot. And what I noticed with any part of the data that I took or any population, it it looks like a starburst. and that's why I call it the human starburst. And what is notable is that about 80% of the needs are clustered in. 20%, roughly 20% of the space right in the middle. And they're very close together, meaning they're very similar. And then the remaining 20% of the needs are spread out throughout the periphery, which is about 80% of that space. And so it's very similar to the pattern that Pareto noticed when he designed or when he talked about the 80-20 principle. It looks like a normal distribution. It looks like multiple bell curves sort of plotted around each other. and the the other thing that is notable is that people who people with disabilities, people who are facing barriers tend to be in that outer region and the edges are very jagged. so there there are many things that are distributed very far from each other towards the edges. And the you're probably gonna ask what the impact of that is. so the the one of the things to note is that because I think because of this pattern and because of economies of scale, because of things like Hyun Song (04:00) You've read my mind. Jutta Treviranus (04:15) majority rules, decision making, et cetera, what you find is whether it's services, products, environments, anything seems to be fairly well designed for anyone that had has needs in that center, gets not very functional or starts to break down as you move as your needs move away from the center, and generally doesn't work if you have needs out at the outer edge. And that perpetuates through all of our systems, whether it's the design of products, the digital inclusion efforts that we have, whether it's employment, academic admissions, how we design our education system, how we design our health system, how we design all of the sort of critical services and systems that keep our society going. And even notions such as democracy are somewhat affected by this because we use one person, one vote, and we frequently forget about human rights. And so the the very, very critical needs of those minorities out at the edges are overwhelmed by the the more trivial needs of people that are in that central core. So it's a pattern that affects every part of our life and it is self perpetuating and it's entrenched in our systems. Hyun Song (05:54) Yeah, thank you for that description. and what does that , what does that mean for data and for how people who are in those, in that periphery, like people with disabilities, are represented in the data that we collect? Jutta Treviranus (06:10) Yeah. So I mean the data of course relates to AI and it's one of the biggest questions we have at the moment. But even before AI, any statistically determined conclusion or generalization that's made or evidence or truth as we say, empirical evidence, whatever we call it, about a population will be true for that average, that core, will be inaccurate as you move away from there and it will be wrong as you get to those edges. So those pronouncements that you see in newspapers or in the news or in any of the studies that say the average man, the average woman or the majority, whatever, of course doesn't hold true if you are out have if your experience or your related data Is out at those jagged edges. The other thing that, though, is somewhat problematic recently is when, or not recently, but has always been somewhat problematic, is there is a drive. I I mean, there's there's a very, very positive drive towards equity and diversity and inclusion. but the assumption there is that there is a homogenous group. So that there are these characteristics that are common to a particular group. And in the data there's a cluster. And you can say, okay, here's a cluster of women, or here's a cluster of a particular race or of a particular language background or specific to a variety of reasons why someone's experiencing barriers or discrimination. And that is true for most other groups that are experiencing that type of discrimination. I mean, not completely true. I mean, it's an if it's an over-generalization in for everybody, but it's definitely an over-generalization if you have a disability, because the only common data characteristic in disability is distance from the average. So if you know somebody with a disability, you know one person with a disability. even in sort of diagnostic categories, the the needs and the barriers experienced by somebody, say who's blind, varies greatly whether you have when whether you you've been blind since birth or you have become blind, the the range of variation amongst I mean any any way in which you want to parse the disability is is so so huge. So there is no cluster. There's no identifiable, distinguishable group that you can use. And the the issue of self identification is also problematic because of the stigma and discrimination that comes with it. And so while it might be there might be a high rate of ident self identifying as having a disability in an area where there isn't as much stereotype or stigma. that isn't that can't be the case in other areas where identifying as having a disability has all sorts of negative implications. Hyun Song (09:44) I think we're starting to get into the territory of the risks of AI when it's being used to make decisions about people, especially people with with disabilities and people who, as you say, are sort of further away from that middle and also have very different needs from each other. So what what does that mean when we start thinking about this from the perspective of AI? What is the particular risk for people with disabilities in that scenario? Jutta Treviranus (10:17) So many people say that the bias of AI against people with disabilities is a data issue, but but it's it's a far more complex story than that. so the AI at its basic, whether it is the I mean all generations of AI, and and AI, of course, is this I mean, it's an industry buzzword that That encompasses so many different systems and so many different processes. But in every instance of it, it is driven by statistical reasoning. So it's mechanized statistical reasoning that will determine the decision that's recommended. It'll determine what text is generated, what the next word is, what images are chosen, etc. And as I mentioned, th that statistical reasoning tends to be wrong if you're far from the average. And AI is basically intensifying or amplifying the the current patterns that are already there. It's using pro probabilistic reasoning to choose the most popular, to choose the the thing with statistical power. and the when I mentioned that most people say, well, why don't we just add date more data about people with disabilities in into the data pool that AI is trained on. but the the issue there is even if we have full proportional representation of everyone with a disability because of the heterogeny, because of the the huge diff diversity within that group, the system is still going to rule with the average or it will try to find some some large cluster. So it isn't that issue. And there's I think there's an anecdote that you wanted me to to relay when we talked earlier. they and this has been oft retold and retold in sometimes not very accurate ways, but the the the time or the incident that caused me to realize what the You know, how complex this problem was was when I tried out a number of automated vehicle learning models. so we were asked by our Ministry of Transportation to try out systems that would be making decisions for automated vehicles and intersections, whether to proceed, stop, turn, etc. and this was back in 2013. So of course they've progressed a bit, but not completely as yet. They still haven't addressed the issue that I'm I discovered back then. and what I tried to do was to test those models with a capture of a friend of mine who pushes her wheelchair backwards. So it's a very unanticipated, odd thing to be doing and and even people that encounter her in the in the intersection will think she's lost control and we'll try to push her back onto the the side of the intersection that she came from. But she's very efficient and very fast. But all of the systems decided to proceed. And if they had been deployed, they would have run her over. and they all s all of the developers of these systems said, not to worry, not to worry. We just need to add additional data about people in wheelchairs and intersections. And so they actually even oversampler in some cases the data. but what happened when I retested them is that they all decided to do the same thing with greater confidence because given the data, the systems were confident that people in wheelchairs moved forward. And so it was safe to proceed because this person was yeah, I was was not yeah. Hyun Song (14:36) So in in that scenario, I mean that so giving it more data just made it go, well, yes, now I have more data to work with and I am now more confident in this false conclusion. How how do you correct for this when the solution isn't simply just to add in the representation? Jutta Treviranus (15:01) Yeah. Well I I think it it correcting it will has many benefits beyond people with disabilities. because well take take for example employment. The way that AI hiring tools are set up is it is taking the success patterns of the past. It's finding the you know best employee perhaps, it's looking for culture fit with the within the organization. It's noting the productivity metrics from before and the characteristic of people that have have been successful in the company before. And then whoever's doing the hiring can add additional things, which tend to be things that are relate to, you know, my favorite university or whatever. All of that leads to a monoculture. So you're going to replicate, you're going to create clones of the people that. Have been hired before. And so you're just going to create copies of everyone that is already within the organization. And we know that monocultures can't adapt very well and that they're they're not going to have choice, adaptive choices when the world changes. And so I think that ruling with the average, with popularity. etc., with the largest customer base, with the aver with what whatever is popular is n is detrimental to our society as a whole. So some of the things that we've been looking at is inverting the AI. So rather than say in the hiring situation, looking for people that are similar to the previous success patterns within the employees that you have, actually looking for skills that are missing, the unusual not current. So turning it from an exp data exploitation algorithm into a data exploration algorithm. So we have a thousand applications rather than looking for the particular people within that thousand applications. Who are going to match the emp the successful employees with our c within our company, we're going to look for people who have skills we don't have currently. similarly, in academic admissions, that's the same thing is happening. those academic admission AI systems that are choosing the the students that should be admitted tend to try to have, you know, the there's this notion of best and brightest, but it tends to also result in a monoculture. And it it verifies some of the the metrics or the the determinants of success that may be questionable. And unfortunately one of the things that's happening within the implementation of AI, there's such a push to adopt AI in every sector. There's sort of this geopolitical push, but also this this idea that If we don't adopt AI and use AI in everything we do, then we're going to be falling behind. But if we simply automate without really questioning why are we doing it? And what do we actually think that this particular thing we're making more efficient is best for everyone? If we do that before critiquing it and automate it. then we take a we lose control of that and we have no opportunity to question and rethink what are we doing? And it it verifies and self perpetuates patterns that are potentially very destructive for our society and for our systems. Hyun Song (19:05) And I think you've raised such a great point about how this sort of blurring the edges is not just a digital trend anymore. By using AI in sort of all sectors of society, it's also blurring the edges in real life. Jutta Treviranus (19:23) Exactly. And the that's one of the so one of the other stories about AI and disability is that AI is actually wonderful at if you want to be more average, if you want to approximate average ability. And so the AI is highly seductive for everybody. I mean it's it's v great to just simply turn over a whole number of tasks to AI, especially the mundane repetitive tasks or the ones where you're feeling insecure and you think that AI might do a better job. And we're certainly finding that with say, well, with anyone that has a that wants to better approximate seeing, hearing, reading, writing, etc. similar to an average person. But in the process of doing that, there there's also a loss of of that individuality and personality. So I I'm seeing that especially in high school and other places where there's already a there's already peer pressure to fit in. And so students are simply accepting whatever the AI offers to in writing and but also if you're using it as an augmentative and alternative communication system. And so you lose that that quirky expression you use, you lose the individuality, the the personality that you put into the content that you're generating, whether it's communication or writing. and that's that's really heartbreaking to some extent. And it But the other effect it has on the classroom or on our society is that we become less comfortable with that quirkiness and diversity. We're pushing it's so much more possible to be what we've deemed to be perfect that it's it's quite jarring w when it doesn't match that homogeneity that that AI is pushing towards. Hyun Song (21:44) Yeah. Different is now being classified as wrong. Jutta Treviranus (21:49) Even yeah. An even greater extent. Yeah. And I see that in talking to professors or other educators, that now their expectations that grammar and spelling and everything should be completely perfect, that all you know, th that that these there's an expectation that there yeah, that everything is to be this this notion of of good quality or or whatever. Rather than questioning our sense of what is quality and and valuing diversity rather than standardization, we have too much emphasis right now on the the well d one of the the the things I'm pushing for in my education system is this notion of the Wabi Sabi in education, the value of the imperfect and permanent and incomplete. because that that's really where we adapt gr and where we learn from our mistakes and we invite greater participation and greater learning. but the AI tools that are used in education are definitely making that harder and harder. Hyun Song (23:12) Now I'd love to kind of talk about the work that's being done to shift that trend and that balance. So the IDRC has in a lot of initiatives that look at AI and inclusion across just a diversity of topics and areas. So everything from employment, education, language, assistive tech, and policy. And one of the projects that I came across in my research is We Count. Can you tell us a little bit more about that project and what it's trying to change? Jutta Treviranus (23:47) Yeah, so in We Count we we're trying to create a knowledge base and a community of practice that is looking at the impact of AI and data systems on people that experience the greatest barriers. so one of the the problems with many of the protections, and of course there's thousands of of guid guidance documents, policies, et cetera, that address things like privacy and data systems and AI. but most of them are looking at it from standard types of metrics. and it the what Tends to get missed is the people that are out of distribution. so meaning that they they are at those edges. the If we look at things like the I mean the best system or policy and regulation at the moment, or the the most advanced at the moment, is the EUAI Act, and it looks at impact and only considers high impact systems, and much of the the metrics regarding compliance have to do with those specific measures that are where you're addressing a particular disaggregated data, etc. So all of the the the not only the the regulations, but also the measures towards protecting people from the harms of AI don't work for people who are tiny minorities. the they will be deemed to be statistically insignificant or the harms that they experience are statistically insignificant. And or from a risk perspective, it's frequently classed as simply anecdotal. It's there is no real evidence to show that this is a pattern that occurs because the the issues are so highly diverse. And that that that's one of the other areas that we count and many of our efforts are looking at. how are we missing out on these harms in the protections that we have, in the certification? similar to the accessibility initiative or like web accessibility, ICT, digital inclusion initiatives, there's a whole industry that has emerged that does evaluation and repair, document remediation, etc. But there's also now a whole industry that has emerged. that is looking at AI protection compliance, et cetera. And unfortunately, it's giving the false assurance that certain things are not harmful. But they're missing the harms to those individuals that are outliers. And all of them have a particular threshold of harm as well, where below this threshold of impact or below this threshold of risk, You don't need to worry, you don't need to do anything. But there's there's also this notion of cumulative harm where if everything is deciding against you, even those things that are deemed to be low impact, then it there's there's definitely harm by a thousand cuts, because they're all somewhat entangled and it it impacts your life in a a cumulative way. so Yeah, so those are some of the things that WeCount is is trying to address. But in addition to that, with We Count and with the other projects, we're also trying to show how AI to some extent is a magnifying mirror. So these things that AI didn't invent this, AI didn't introduce these things. These are things that have been in our society. AI is simply amplifying them and automating them. And so I I I think it behooves us not only in the way that to to think about how we're using AI, but it behooves us to look at what have become conventions and assumptions in our society. And I I know this is a podcast on human centered design. And so even in design, we're saying, wait a sec, you know, this is simply amplifying something that's already there. for example most of like design thinking in and of itself. We have the design thinking squiggle, which where you yes, you ideate and you think about all sorts of imaginative ways in which you can address an an issue, but then it iterates towards a winning solution. And one of the things that happens with that that iterative process is that the individuals we work with tend to be the the losers in that winning and losing. And this idea that you can find a winning solution and then scale it throughout every application. And that's an indicator of success is how far you can replicate a formulaic winning solution. Well that is is quite problematic because everybody has different contexts and Things change and there the winning solution is frequently not very adaptive because you are using majority rules decision systems, all those sticky notes on the one thing that, and here's the winner, and everybody pays attention to it. yeah, continues to, you know, piles on to whatever happens to be the most popular. So in We Count, we've we've done fairly Hyun Song (29:55) Yeah. Jutta Treviranus (30:07) simple exercises like we've we've created an inverted word cloud. So word clouds, there's there's a f a popular exercise in most presentations or in group workshops where you have a question and everybody submits a response and there's a word cloud that comes up that shows the most popular choice and it goes to the middle and it grows in size and people that's then is what people notice and so they they pile on to that and the the minority requ you know answers needs whatever to the question disappear and become invisible so what what we've been doing is we've been inverting that so it's the novel minority words that go to the middle and increase in size it's to get people to think about well you know these are unconscious assumptions and unconscious processes. We would never likely question the the word cloud, the the push to the you know, our our notion that what we seem what we see as empirical evidence as science is something that is has t the greatest statistical power. Well what does that do to people who are in a statistical minority? yeah so we're Through we count we're entering it in data and in AI, but I the the underlying goal of that is to look at well, what is AI showing us about how we are designing our systems, how we're treating different parts of our society, what choices we make, how we decide, who ha who decides, etc. All of those. Those questions. we the inverted clo word cloud, we release everything open source. So you can you can use it. Yeah. Hyun Song (32:12) And we'll include in the podcast description where we can find that information too, so everyone can benefit from this. I think that's pretty genius. And also it's, you raised this really interesting point about how AI itself has an algorithm, it's built around statistical reasoning and it's inherently biased, but then it is also a mirror that reflects on the values of our society. Well if we have existing biases then the AI will just take that bias and proliferate it and scale it and accelerate it. But it didn't come from nowhere. Jutta Treviranus (32:55) Exactly. Yeah. And the people who who design the AI, of course, are already the people in power, which and yeah, it's I mean it's it's it's increasing disparity quite significantly. And disparity is is to some extent at I mean it's it's one of the core drivers of many of the other issues that we're facing as a society. Hyun Song (33:18) Yeah, yeah. And and I guess related to this, I I also understand that the work that you've done through We Count and the Centre has contributed to Canada's accessible and equitable AI standards, which was published in December, I believe. and that you chaired the technical committee involved in that work. So congratulations. What an incredible achievement. and and going through that one Jutta Treviranus (33:42) Thank you. Hyun Song (33:47) One part of that that stood out to me was the call for people with disabilities to be full participants in the AI lifecycle. And my question to you is is what does that need to look like and and why is that important to have full participation? Jutta Treviranus (34:04) Yeah, it's a it and it of course is a very heavy lift because almost anything in the AI lifecycle is not very accessible. the it at the heart of that is our belief that in order to to fully make something inclusive, you need to ha to have the expertise of the people with lived experience of the barriers. there The types of mechanisms or instruments that we have within the design field, like persona, are or the the classificatory diagnostic categories are completely inadequate to to understand just the the complex range of of an interacting range of requirements and the the empathy exercises are also not adequate. So we need in to systemically change our society, we we need people that experience the barriers, people that understand the struggle and understand the type of resourcefulness that we need to do something, they need to be there to frame the problem, to determine whether in fact AI should be used given the the risks that might be experienced and the assumptions and presumptions that are in there, but also to design and create AI systems. if if only the people that are currently doing well with the technologies and have full digital literacy or or have the the privilege of digital literacy because the systems are designed for them and by them, then it We're we're just going to get a repeat of these gadgets that are serving people that are already quite complacent and already doing well. if we're going to create a society, a system, a set of tools and instruments that will work for us in the very chaotic, crisis ridden world that we're living in at the moment, we need people who understand struggle, who understand what needs to be changed to design things, to build things, to make decisions about what we need, to monitor it, to say, okay, this is enough. We need to stop using this because it's it's causing this much harm. so we need that those individuals to to be included in that ecosystem. Because if if we have those individuals participating, people that understand struggle, people that understand the barriers that exist, that need the world to change, then we're going to create an AI system or any system that will work for us when we all struggle. And there will be, I mean, we know the pandemic was one crisis, but everyone is forecasting that that was a minor blip, that we are all destined for a a series of even greater and greater escalating crises. So let's engage a form of design or a participatory design, an inclusive design that prepares us for that, that where we can create adaptive systems that where we have further choices. than we currently have. We're just going to replicate and replicate existing patterns if we we do it the way that we're doing it right now. Hyun Song (38:05) And that goes back to your point about selecting for things that we are missing, that we're not seeing in the the majority or this imagined ideal that that we kind of prioritize in every facet of society. Jutta Treviranus (38:23) Yeah, and there's this myth that it will cost more if we do that, that it will cost more if we consider, you know, that entire spectrum of requirements right from the beginning. But we've shown through economic modeling and through a number of of evaluations, longitudinal evaluations of service design, that in fact that's not the case. It in the medium to long term, it costs a lot less. so that if Because what happens is you create a much more adaptive system. So the longevity is greater. You don't the system doesn't become brittle. If you only design for the middle, if you only listen to the complacent few that already are doing quite well, then it your system you're going to have to patch on and patch on. Because not only will there be people that are missing out and that want changes to the system, but Your environment is going to change, the context is going to change, and so you're going to have to keep modifying the system, provide additional help, ma many more fixes that you need to do. And so the system becomes brittle and and there's end of life very, very quickly. But if you design with that entire spectrum of needs right from the beginning, you you're forced to create a much more adaptive system with many more choices. and greater flexibility and so it has longer much greater longevity. And it costs Hyun Song (39:56) Yeah, cost less, less fixes retroactively. And the whole principle of universal design, when you design for that diversity, so many more people benefit than just the people that you're considering as your outliers or your edge cases. It's actually often that the rest of us benefit too. So Jutta I might wrap up with this question. So for our listeners who work in human centred design and research in product or policy. what's the one thing that we can do now to help shift AI towards more benefit and less harm, especially for people with disabilities? Jutta Treviranus (40:43) okay. there's a I I I don't tend to want to say one thing. There's a diversity of strategies and I and I I think one of the things that that we do to ourselves that sort of hampers actually addressing something is thinking that we need a single answer, that there is a best practice. And I I think we each our w in whatever role we're in, in whatever context or whatever goal we're we're attempting to work on, there's different strategies that will work. So I whether it's co co-designing, bringing the people that are going to be most impacted by the decisions we make in to help us determine whether in fact we should go with AI, whether AI is the the correct tool or whether we should question what it is that we're automating and rethink the the entire enterprise before we we automate it and and hand over control to the system which then reduces our choices and our flexibility or whether it is working determining and what are are the potential harms that that could happen and and having a monitor of harms so that we have indicators of where we sh we should redesign or shift things or whether it's creating benchmarks for strategies or processes that would result in more equitable AI. I mean there's a there's it there's a whole range. So I think I w what we try to do in the standard that we develop that we're hoping will move to regulation and is potentially moving to regulation is an a number of different things that organizations can do, including designers and developers, and especially people that are concerned about the human, because I think at the moment it's the human that is un that is under threat here. We are in in a sense almost creating a digital eugenics infrastructure because we're eliminating that difference that I think is is a vital characteristic of of humanity and something that will guarantee our surviv I mean it not guarantee, but it is much more likely to support our survival if we have that diversity of perspective, diversity of skills, diversity of understanding. we are removing what we d other other forms of truth and evidence and decision making and what we deem to be rigorous or quality or whatever, we're we're restricting ourselves quite a bit and ending in a monoculture, which of course is not going to survive. Hyun Song (43:54) I wanna thank you so much for this conversation today and for giving us a a different way of thinking about AI and and how it reflects and proliferates the values that we hold as a society and and the the need to kind of pause and rethink what we value as being important skills, important qualities, and to kind of expand that thinking to be more diverse and to, yeah, look out into the jagged edges of that starburst. So thank you so much for that. Jutta Treviranus (44:32) Thank you for the opportunity to talk about this.