As part of our AI For Growth executive education series, we interview top executives at leading global companies who have successfully applied AI to grow their enterprises. Today, we sit down with Dr. Rumman Chowdhury, Global Lead for Responsible AI at Accenture, a leading global professional services company providing a range of strategy, consulting, technology & operations services and solutions.
As a data scientist and social scientist, Rumman’s passion lies at the intersection of artificial intelligence and humanity. At Accenture, she drives the creation of responsible and ethical AI products for company clients. She has been honored as one of the BBC’s 100 Women, Silicon Valley’s 40 under 40, a TedX speaker, and is a fellow at the Royal Society for the Arts. Rumman will share how business leaders should define and enforce ethical behavior and ensure safety and transparency in AI and automated systems.
Watch the interview to learn:
- What it means to be ethical, and how to start thinking more responsibly about AI
- New algorithms that can assess models for bias and to explain their decisions in a transparent manner, as well as the challenges in implementing them.
- Steps toward setting up an ethical governance model for AI at your company
Mariya Yao: Hi everyone, thank you for tuning into our AI for Growth executive series. In this interview series, we learn from the top executives at global companies who have successfully applied AI to their companies.
Today, I’m sitting here with Rumman Chowdhury, who is the global lead for Responsible AI at Accenture.
Rumman, tell us a little bit about yourself and how you got into AI.
Rumman Chowdhury: Thank you for having me, Mariya, I’m really excited to meet your audience.
I am a rare bird in Silicon Valley. I come to data science with a social science background, so I’m one of the people who came over more from statistics and lots of programming, which has certainly informed my role as Responsible AI lead.
I came to Silicon Valley while finishing my dissertation, and I do not recommend anyone try to do what I did.
MY: Isn’t it that the classic Silicon Valley thing, go do something cool before you finish school?
RC: I was trying to finish my dissertation while working full time, which was not the easiest, and one does not have a social life when you do that.
I was actually teaching data science at a boot camp called Metis, and then Accenture found me. They had this interesting new role. No one’s ever held this job. The field of AI in ethics was brand new and they didn’t really know what to expect. They needed somebody who was not just a person who knew data science and AI but understood human beings and social behaviors and things like that, and there I was.
MY: What does your role as Global Lead for Responsible AI actually entail?
RC: A lot of flying around!
There’s a couple of things that we’re seeing happen. It’s not just artificial intelligence and AI being so pervasive in our society, though that’s certainly part of it.
As we start to use artificial intelligence for things like jobs, hiring, even who you date—which impact you might marry—what you see when you go online, the news you watch… In other words, what shapes your mind, people are really increasingly concerned about bias that might creep them to AI.
You’ve seen some pretty notable pieces other than the big ones from a few years ago, like Google facial recognition tagging black people as gorillas, natural language processing algorithms trained on Google News coming up with conclusions like “man is to programmer as woman is to homemaker”, or image recognition being trained on Getty stock photos that so strongly associate a woman with a kitchen, that when it sees a man in the kitchen, it tags the man as a woman, because it has not seen any photos of men in the kitchen.
The other part is this shift towards this idea of conscious capitalism. What that means is that it’s no longer sufficient for a company to say “I’m regulatory compliant and I’m fine”. We as consumers now consider the responsibility of an organization not just to be legally compliant but to do good, and that’s because regulation has had a hard time keeping up with technology.
“It is not illegal to tag black people as gorillas, but you should know better, Google.” That’s what consumers are saying. We’re seeing this shift towards more mindful and ethical capitalism, which is going hand-in-hand with the pervasiveness of this technology. What we’re seeing people say is, “I cannot see if an algorithm is being discriminatory towards me. I have no way of understanding because things are hyper-personalized, so then it is your company’s job to make sure that things are ethical and thoughtfully produced so that you know the outcomes that I see aren’t being biased in a way that I have no control over.”
MY: What does it actually mean to be “ethical”? Previously, we had discussed how difficult it is to have terms like “fairness” or “bias” and how loaded those terms are. How do you even tell that an algorithm is discriminatory? How can you tell it’s being fair or ethical? How do you get down to those technical details?
RC: There are the really obvious egregious cases, such as facial recognition tagging black people as gorillas. It’s obvious and there’s no ambiguity there.
But you’re right, there is some ambiguity in something like LinkedIn’s algorithmic recommendations. For me, personally, it actually gives me pretty terrible jobs. It still recommends entry level data science roles and I haven’t been an entry-level data scientist in years. I don’t know why, and I don’t think anybody at LinkedIn could untangle from the hyper-personalized algorithm, which has things like what I post and the things I’ve liked and the people’s pages I’ve read. It’s very difficult to disentangle whether it was my gender or my race that influenced the outcome.
The reason my title is Responsible AI Lead is that, as an employee of Accenture, it’s not my job to tell another company what it means to be ethical.
Here’s where one might fall into a philosophical morass. It’s almost like a Gartner hype cycle of responsible AI. You get started, and you’re like, “Oh, we need these Ten Commandments of AI.” Then you say, “AI should be fair.” Then you’re like, “How do I operationalize this?” And then you fall into the pit and you’re like, “Well, what does it need to be fair? Oh my god, Plato!” Then you start reading like Plato and Aristotle, then your head explodes.
At some point, you come out of it and you’re like, “All right, we have to do something, and it will not be an all-encompassing universal definition of fairness, but it will be something.”
The benefit of being a Responsible AI Lead is really about being responsible towards your consumers, toward society, and towards the community. For every company that I go to, I always joke that I have an easy out, and my easy out is that when I go to a company, I ask them, “First and foremost, what are your company’s core values? What is your mission? And is your artificial intelligence in line with that mission?”
Think about social media, which is everybody’s favorite punching bag right now when it comes to ethical behavior. If you were to ask Mark Zuckerberg what are the core values of Facebook and what is Facebook’s supposed to be doing for society? He would say, “It is a tool for communication and connection.” And then you ask, “Mr. Zuckerberg, is your artificial intelligence actually enabling people to connect and communicate?”
When I say it needs to be ethical or responsible, it is those core values that I’m helping companies aim toward.
MY: You mentioned one big challenge, which is it can be very difficult to define [what it means to be responsible.] What would you say are some of the other challenges that you’ve seen companies face when they go up this responsibility hype cycle and then fall into the pits? What are the dragons and demons waiting for them at the end of it all?
RC: It’s a very undefined space and we don’t have a regulatory environment yet. That is worrisome.
There is a common misconception that companies don’t want regulation. Companies actually do! People want to know what they should do so they don’t get in trouble.
There isn’t a regulatory space defined, but everybody is in this AI race at the moment. You can’t not be in the race, but at the same time how do you control for these unintended consequences [when] we don’t have these global guidelines or national guidelines or local guidelines yet?
What they’re trying to look at is, “How can I govern AI internally?” I thought that most people would ask me very technical questions like, “How do I unpack the black box in data?” Actually, most of the questions I get are about, “How do I create a system of governance for artificial intelligence internally at my company?”
It’s a fascinating conversation, because then it’s about breaking down silos between different parts of your organization. Data scientists now have to go talk to the Legal department to figure things out. Management has to understand what a data scientist says. The people who are your database administrators have to start talking to other parts of the organization as well.
It’s almost like a coming together moment of the company as a whole to help them on their AI journey.
MY: So how do I set up a governance model other than getting my people to talk to each other? Are there, for example, other very key action items or steps that an executive need to be thinking about other than what you just mentioned?
RC: A good place to start is always as an acronym we use in this space. FATE: fairness, accountability, transparency, and explainability. That encompasses things like data privacy and security, the algorithms you’re using, etc. The end goal would be something that encompasses all of FATE.
To move backwards from that beautiful world, one good way to think about as you get started is, “How can you enable your data scientists and your management to identify in which parts of your project development lifecycle ethics can fit in.
The challenge to someone like myself is, “How can aspects move at the pace of innovation?” Even at Accenture, we talk a lot about agile development, innovation, rapid prototyping, etc. I can’t come in and say, “Here’s a six-month process for validating your model.” That’s not fair and nobody would adopt it. It’s “how can we look at your product development lifecycle? How can we look at the biases that might occur and where you can address them?”
That’s when it starts to become difficult, where the data scientist say, “Well, I don’t know this stuff. I don’t know that the city of Chicago has a history of redlining, so that if I try to make an algorithm and I use it code to determine creditworthiness, it will be discriminatory towards black people.” That needed some very specific knowledge.
This is where we help form panels of experts for companies. We also provide training so that data scientists are enabled to ask the right questions. It’s not that you have to know everything, you have to start asking the right questions, and that’s what we’re enabling people to do.
We do things like having a best practices playbook, having a data science ethics training. Again, that’s about different types of biases.
The holy grail of all of this is the algorithmic impact assessment and a bunch of people have been talking about that. Kate Crawford at AI Now has a really good medium article talking about algorithmic impact assessment specifically for policy decisions. This is because New York City passed an algorithmic transparency law. I’m taking that and applying it in industry, so it’s building on concepts that already exists or papers that already exists called the “privacy impact assessment for data”.
How might that look for algorithms? We fill out this form, we have this series of protocols and practices we go through, and then something exists for internal consumption in the future, so that if something goes wrong or someone needs to go back and refer to a similar situation that happened in the past, they’re able to do that.
MY: Going back to what you were saying before about figuring out how ethics can go at the pace of innovation and figuring out where in your current process workflow or development you can fit in ethics, where have you noticed are some of the best places to open that ethical question? What are the questions that data scientists should be asking that they may not be right now?
RC: This is where my social science background comes in. When I was teaching at Metis, I noticed that sometimes with people with pure programming background, it’s hard to remember that data is about people. For social scientists that’s the only way I think—I cannot think of data any other way.
I realized that for my students that came from pure mathematical backgrounds or pure computer programming backgrounds, data was a grid. The grid was like a holy grail of truth. For a social scientist, it’s not like that. Data is not some objective truth; it’s inherently reflective of cultural and social biases and even human error or human biases.
If I ask you, “Hey Mariya, how are you doing today?” Your knee-jerk reaction would be to say, “I’m fine.” That’s just how we act with society.
If I’m going to go around doing a survey asking 100 people every day how they are and everyone said they’re fine. I’m like, “Look, the data says everyone’s great!” But we know that our social norms are such that if I ask you in casual conversation how you’re doing, unless we’re really good friends, you don’t start doing a brain dump on me. My boyfriend’s been doing this and my cat sick and blah blah blah. That’s not socially acceptable.
Another thing is that, when we think about measurement of our data, often we take things as ground truth.
I’ll give you a great example that’s often used. When I say I’m measuring crime, I am not actually truly measuring the amount of crime that occurs. It’s actually a proxy. What I’m actually measuring are the number of arrests and the number of reported crimes, so one can
imagine that really bad neighborhoods where crime is rampant, people probably don’t even report half the stuff that happens. On the other end, extremely affluent neighborhoods where there’s a bunch of really rich kids, people may not report crimes either because they’re afraid of people with wealth and power. There’s so many social considerations that may come in, so there’s a lot of interesting things to think about.
So it’s getting data scientists thinking about how their data looks in the real world, in the wild.
Also in Silicon Valley, we answer our own problems, so we are all quite similar to each othe. We come from even quite similar backgrounds. Most of us come from privileged enough backgrounds. Most of us have a good working knowledge of technology. It’s hard for us to think about how might a disabled person interact with this thing. How might this impact an elderly person, or a minority, or somebody who’s low income?
What’s interesting is that once you start asking the right question,
people’s brains really start to work. As human beings. we all have a bit of empathy in us. It’s just that we’ve never been asked to apply it to data. Then when we do, we’re actually able to start thinking of the interesting questions.
The second part was about where in the life cycle [to think about ethics]. The most helpful thing we do with companies are design thinking workshops and it’s to help them conceptualize the project.
I have run a few of them at conferences as a teaser. One of my favorite exercises is called “”break your toy”. I have people go through this exercise where they prototype a project or something, and then I tell them, “Imagine you want to game your own system. You want to lie, cheat, steal… You want to cheat the system. How would you do it, and what would you do?”
I think a lot of us like to think very positively. We like to think everyone’s gonna interact in very literal way with our tool, but that’s not always the case. Sometimes these edge cases are what we end up seeing in AI.
Then I think about how might you break this tool, then people said to come up with really creative solutions. “Well, if I were trying to hack my own system, here’s I would do it.”
And then the next exercise is, “All right, now think about how you would fix it.” Often, that’s when people start to see where in my development phase I might look for this thing or test for this thing. It might be in the very beginning when I’m thinking about “should this be a web-based platform or something on mobile?” Or it could be at the very end when I think about, “Okay, if elderly people were to interact with this, what would happen? How can I test for this?”
There’s a lot of really interesting exercises people walk through and we haven’t even started talking about algorithmic fairness yet…we’re still talking about data and design….
MY: Not everyone gets the luxury of having Global Lead of Responsible AI at Accenture to guide them on their responsible AI journey. You’ve mentioned a couple things people can do: one is to get more interdisciplinary cross-functional collaboration and discussion, another one is to do this “break your toy” exercise where they try to think through some of the design challenges that may occur at the edge outside of normal use.
What are some other things that executives can just start doing even if they don’t necessarily have your advisory, to start doing today to start thinking more responsibly about AI?
RC: A good thing to do is think about who your target population is, and whether your data is representative of that target population.
This was Google’s facial recognition didn’t identify African Americans. It simply didn’t train enough African Americans. There’s some amazing work called gender shades done about how these data sets are actually imbalanced. That is what’s leading to imbalanced ultimate outcomes of facial recognition working with near-perfect accuracy for white men and at about 68% accuracy for dark black women. That’s a massive discrepancy!
A really simple thing to think about is, “What is my data? What do I have? What does it represent? Where might it be lacking?” A lot of It is just thinking exercises and then—and this is more for executive, [because] there’s so many things that we can look at as data scientists—to think about where there might be a problem in the data that I’m getting.
For example, a classic example would be if I wanted to make a new flavor of ice cream. I have chocolate and vanilla or whatever, and I only asked the people who are consuming my ice cream what is the new flavor that they want. That actually is quite problematic because I haven’t asked the people who don’t eat my ice cream, so there’s a sample selection bias here. It might be that people don’t eat my ice cream because I only have chocolate and vanilla and they like strawberry, whereas if I’m appealing to the people who already buy my ice cream, I’m just cannibalizing my own market.
It seems sensible once I’ve laid it out like that for you, but it’s a really common mistake people make when they’re thinking about their data.
I’m using examples that are not related to things that are culturally sensitive like gender or race, but you can see how it can translate over into your market that way.
MY: You’ve covered data bias such having sample errors. You’ve also talked about product bias, such as not thinking through how your product might fit people who are disabled or who are elderly or who are low-income.
Now let’s talk about algorithmic bias! What are some of the key things that a business leader needs to know about potential algorithmic bias?
RC: This is where I go back to the acronym FATE: fairness, accountability, transparency, explanability.
GDPR has pretty much hit everyone, in the EU at least, and any company that impacts anybody in the EU like a sledgehammer. Because it asks for these things, specifically, accountability and explainability.
The first thing I would say is it is not impossible to identify algorithmic bias. There’s this common misconception—even among data scientists—that there is no way to identify bias, so we have no way of correcting for these things.
I did a talk at a data science conference a few months ago, and I was surprised at how many people in the audience hadn’t heard of the models I was talking about. I specifically, I focused on four of them two of them had to do with explainability, and two of them had to do with literally correcting bias.
First one is a pretty well-known one—it’s almost two years old at this point—called LIME, and the subtitle of it is “why should I trust you”, and it gives you explanations for classifiers. The example they like to use is, if you’re using a neural network to help the doctor diagnose a disease, a doctor can’t just go to a patient and say, “Oh by the way, you got cancer. and the patient’s like, “How do you know?” And the doctor’s like, “I don’t know, the AI told me.” It doesn’t quite work that way. It provides an explainable outcome to the doctor.
The second one, it creates good explanations of visual classifiers using natural language processing. For example, if I showed you two pictures of birds and one was classified as a sparrow and the other one a cardinal, and I asked why, you would say something that differentiated the two. You would say, “Oh the cardinal is red and the sparrow is brown,” but an AI doesn’t know to do that if you’re just constructing reasoning. What it would probably say is, “Bird X is a cardinal because it has wings,” and you’re like, “That’s not useful, all birds have wings.” It actually provides a unique description of the image, so again, helping with explainability.
The last two I looked at, one of them is a super popular paper going around right now called “counterfactual fairness”. There’s actually a few of them. One is published by Turing, one by Deep Mind. Counterfactual speaks to my heart because it’s about causal reasoning and causal relationships, which is a different branch of this work that really, it’s more social scientists that have done it. So going back to the LinkedIn example earlier, let’s say I think that this algorithms is being biased because of my gender. What this can do is switch my gender to male and it sees if I get a different outcome. If I do, then it knows that is biased and actually corrects for it.
The last one is called transparent model distillation, published by Microsoft Research. You have two models that are trained. Let’s say you have a complete black box with no idea what it’s doing. That’s a teacher model. You train the student model, which is much simpler, like a regression model on the outcomes of your neural net or whatever. Then you have a second model, again really simple and explainable, again with linear regression trained on the actual outcome. You take these two easy-to-explain models and look at the difference between them, and the difference between them would be bias. It’s so complex, it’s simple, and it’s brilliant. That’s one of my personal favorites right now.
Those are the four papers I presented, so super long answer to your question because I love nerding out over it.
MY: Let me summarize. The first one you mentioned was LIME, it’s the oldest one. The second one you didn’t give me a name for, that’s the one we use natural language processing to explain the output of visual models. Then the third one was counterfactual fairness, and the last one was transparent model distillation which came out of Microsoft Research. I’ll make sure that all of these links are available when we post this video.
Even knowing that some of these models exist, it still may be difficult to get your whole organization on board that there is bias. To your point that some data scientists are even like, “Oh there’s no such thing as bias, algorithms can’t be biased.” When you get these kinds of arguments from even very technical very mathematical people that bias is not a thing, what are some of your arguments against that?
RC: When I give my talk about data bias, I split it up into two parts. I talk about the measurement bias, the data bias, and then the silo bias. I always use hiring as a good example.
Let’s say you have the data set about hiring. I could have perfect data about the hiring salary and promotion practices at Goldman Sachs and train a model on it, and it would almost definitely be biased against women.
Is that the real world? Yes. Is that the world you want to perpetuate? That’s the question I ask. I agree with you, it is the real world, but is that really what you want to perpetuate?
I have what I call the three I’s about AI. AI is immediate, invisible, and impactful, and this is what lay people imagine AI to be. To them, AI is this magical thing. It’s invisible, so most of the algorithms that run, people love to anthropomorphize. When I put AI in a robot but that’s obviously not what it looks like. It’s just this algorithm and it’s running in this magical cloud right. Think about how this thing is running, and it’s kind of insane, so it’s invisible.
It’s immediate. If I launched this this new change to my algorithm, it happens right away, and it impacts all of my users. Facebook makes a change to their algorithm and it affects hundreds of millions of people throughout the world. If that is discriminatory, I literally cannot do anything about it. If there is some discriminatory outcome, I’m aware this is a discriminatory outcome, and I have no redress. Actually, most of the time I’m not able to know that a discriminatory outcome happened. So who holds the responsibility for that? Are we willing to say that we’re totally fine with building products that are prejudiced and biased and discriminatory, will deny people jobs and they’ll have no way of knowing, and they’ll have no way of fixing it?
It’s a thing I built and I just didn’t care enough. I think that there’s a bit of soul-searching that needs to happen if you really think that that’s okay
MY: What would you say are is still the unresolved barriers and challenges in AI ethics? It is so new, we’re still defining things, and data scientists don’t even know that some of these models you just mentioned existed.
What are the issues that keep you up at night?
RC: Unfettered capitalism!
To really answer your question, it is true that the research in this space is evolving. We have plenty of people and amazing research institutions looking into these things right now, but sometimes you know academics look at things in a very academic perspective.
For example. I love the counterfactual fairness paper, [but]I cannot think of a client that would have the resources and the time to actually implement it.
We have this tension…even if we created some really amazing models, we need to integrate it into the product lifecycle. That would be tension number one.
Tension number two, we’re not automating away ethical behavior. Ultimately, companies, organizations, and people have to be ethical. How might you have an ethical culture? This is where the governance comes in.
Let’s say I am a data scientist, and I’m just on some project. We all know that every project is over budget and under time. I’m working on this project and I feel like there may be a discriminatory outcome. If I go to my manager and they’re basically gonna yell at me because I’m gonna hold this project up because I think that there should be more black people represented in the facial recognition data set, that’s a problem. Any individual data scientist is working on a project where they may not have the power to say, “Hey, hold on. This isn’t working. Let’s fix this.” They may need to go to somebody else and they should, and it should get escalated. It’s not my job as a data scientist to go to my client and say, “Oh by the way, client, your data is discriminatory.” That’s probably the job of somebody else. So how do you enable a culture of ethics which comes from good governance, the transparency and accountability mechanisms, really clear guidelines, etc. But then we have to make all of those because they don’t exist yet.
MY: Speaking of challenges for executives to overcome, you’ve mentioned a lot of tips.
For the people who aren’t able to hire you and unable to hire Accenture to help them with their ethical journey to create all these guidelines and all these rules and all these accountability and measures, what are the top three things that they should do?
RC: Number one would be to keep an ear to the ground of the responsible AI space. I’ve started a crowd-sourced Google document. I was looking at it today and people are really adding to it. It’s really good place to get started to understand ethic from AI.
Number two, you can actually reach out to a lot of these people. The beautiful part about being in the responsible AI universe is we’re all trying to figure this out together. We’re super collaborative. I just got off a phone call with this group where I’m working with this professor at Princeton. We’re gonna look at algorithmic fairness in hiring. I don’t know if in another space where I would have ever connected with this guy, Professor Ed Felten. We’re all in this battle together. People want to help, to share and learn from each other because we’re all learning.
MY: There is a selection bias, right? We’re responsible people care about responsible AI.
RC: You’re probably right, but then that’s probably good for someone interested in learning, because this space is very welcoming.
The third, don’t feel the need to reinvent the wheel. There are so many really great people building things, and we’re trying to make things as publicly available as we can.
How might you implement this using a diverse set of voices? How might you within your own company start to include different kinds of voices in this development process?
At Accenture, I always say that we drink our own champagne. I’m here preaching responsible ethical AI, and we are ourselves going through some of these processes. We’ve paid a group to assess Accenture, because I’m not going to assess myself and pat myself on the back. It’s super helpful to get another perspective.
MY: I’ve never heard of ethical audits, but I feel like we should have ethical audits for our personal lives. To your point, ethics is not just a function of the AI or the technology. It starts with us being and choosing to be ethical people.
RC: In some of these models, when you correct for algorithmic fairness, you sacrifice model accuracy. If your goal is only to optimize your model, then you’re right, fairness doesn’t fit into that.
It’s a beautiful metaphor for what we’re talking about. If your goal is just to optimize business revenue or whatever overall, and you don’t really care, then you can do that. Sometimes, in the short term, fairness may not be immediately profitable. I do genuinely think that the long-term sustainability of businesses rely on being ethical and responsible because of how the market is shifting.
But sure, if we prioritize short-term gains over everything, then maybe it’s not worthwhile to be fair. But if we don’t, then we’re not gonna have the kind of world and society and culture and companies that we actually can create with all this amazing technology.
MY: Words of wisdom! Thank you so much. This was such a fascinating conversation, and I have this long laundry list of resources now that I would love to dig into myself, and I’m sure our audience will as well.
Thank you so much for joining us today.
RC: You’re welcome. It’s been a pleasure chatting with you.
Could you please
Post a link to the open source Google Doc she mentjoned? Thanks!