Automating Customer Support With Human-Like Virtual Agents (Interview with Rachael Rekart of Autodesk)

As part of our AI For Growth executive education series, we interview top executives at leading global companies who have successfully applied AI to grow their enterprises. Today, we sit down with Rachael Rekart, Director of Machine Assistance at Autodesk, a software firm that provides specialized software for the architecture, engineering, construction, manufacturing, and media industries.

Rachael is a leader in strategic planning and change management, having spearheaded numerous technical initiatives with enterprise-scale impact. She led the development and implementation of Autodesk’s first application of artificial intelligence for customer engagement. Their virtual assistant Ava reduced resolution times by 99% and cut costs from $15-$200 per ticket to under $1.

Today, she shares her insights on the process of developing a successful conversational agent for customer engagement. Watch the interview to learn:

1. The development process for AVA, Autodesk’s customer support assistant, and her advice for selecting a budget, software support, and design philosophy for startup teams looking to implement their own conversational agents.
2. Lessons gleaned from observing the differences in how customers behave with a human assistant versus an automated assistant.
3. Advice on the types of expertise that you’ll need when assembling a team to build and support an effective conversational agent for your company.

https://youtu.be/o7rhcrGSjaA

Marlene Jia: Hi everyone, thank you for joining our AI for Growth executive series. In this interview series, we learned from executives at leading global companies who’ve accepts successfully applied AI to their enterprise.

My name is Marlene Jia, you can call me MJ, and we’ll be chatting with Rachael, who leads machine assistance at Autodesk. She ran this initiative with this chatbot called AVA (Autodesk Virtual Assistant), and I think we’ll have really interesting things to learn, especially with all of the things that are going on with bots nowadays.

So, Rachael, thank you so much for being on our broadcast. We’re really excited. I know we’ve talked in the past and I really loved our conversations, so I’m really excited to share you with our audience.

Can you start with an introduction and tell us how you first became interested in bots and artificial intelligence, and maybe how you got into your role in your team at Autodesk?

Rachael Rekart: Thank you for having me. My name is Rachael Rekart. I am the director of machine assistance at Autodesk, and I essentially run our AI and ML teams for customer engagement. As MJ mentioned, our first product is AVA and she’s not our only product but she’s definitely our flagship product in the machine assistance or the machine learning world.

I got into AI by happenstance. My background is in process and system optimization. I actually have a supply chain background, and I did a lot of program management around optimizing and helping us scale our systems in our back office, and once I proved that I can kind of bring things from idea to implementation, they asked me to come help out with this proof of concept that they were working on with a chatbot. So they were piloting IBM Watson and they knew there was ROI there, but they couldn’t really figure out how to bring it to scale so they brought me in to help plan all of that, and then I just stuck.

MJ: That’s awesome. With AVA, I’m curious, was there a specific company need or was there like a mandate or initiative? How did AVA come to be, and what issues did AVA end up being created to solve?

RR: I think there was definitely an automation and kind of a scaling customer support without subsequently scaling human support at Autodesk that really prompted this initiative. About a year ago, maybe a little over a year ago, Autodesk announced our transition to a subscription-based platform. Historically, we sold perpetual licenses—you bought it one time, you owned it for life—that’s really the only time you had to interact with us, and we started to transition over to annual subscriptions, quarterly, and monthly.

Our support increased correspondingly. As you’re renewing your subscription, you’re contact us monthly or you’re interacting with us monthly, and we could not keep up. We already had really long wait times. Our average resolution time was one and a half to two days before we made this transition. Our CSAT was around 65 to 70, so not great customer satisfaction scores, really long resolution times, and we just could not keep up with the volume, so that’s what prompted this initiative. We also were reacting to a customer need of having 24 by 7 support, because they have 24 by 7 companies and are using our products 24 by 7, and we weren’t a 24 by 7 company.

So that is really what prompted this initiative, and we launched Ava in February of 2017 to really address our most common customer inquiries, but we found that customers want to talk to her about many more things, so at launch our first month, Ava had 20,000 conversations with customers, and she did so with a resolution time of five and a half minutes, and a satisfaction score in line with our agents, around 70 percent at launch.

MJ: Wow, okay, so already they were at the benchmark.

RR: Benchmark in terms of satisfaction, but far exceeded in terms of resolution time. She’s only gotten better from there. We launched her in February of last year. Today, AVA talks to over a hundred thousand customers a month, so she went from twenty to a hundred thousand and her CSAT this month hit 88, so she’s got an 88% customer satisfaction rate, and her resolution time is about 3.9 minutes.

So as she does more, she’s actually getting better. It’s really the benefit of this technology, because that just doesn’t happen with people. When you give people triple the quantity or five times the quantity of what they used to handle, their quality suffers. But when you give that to robots, they learn more.

MJ: Right, right, so you have to walk us through the process of how you guys developed AVA at the very beginning, because I think for a lot of people, the common question is, you know, how do you decide which use cases to focus on? How do you get to the first iteration of your bot? Now, obviously you mentioned that you tried to keep it at commonly asked questions and things like that, but Autodesk is also a very complicated product, so just curious, how did you guys even begin with creating and implementing AVA?

RR: We started with a use case that was what we called low-hanging fruit. It was one thing getting an activation code, and it constituted about thirty percent of our overall volume. So every time a customer activates their product, they need the activation code, and there are many number of reasons that they might get an error or contact us for a code. We started with that use case because there was high volume and it was something that we could automate and really help to disambiguate because there were what we thought about five big buckets of reasons that customers would contact us for an activation issue.

So that’s how we selected our first use case, and then we expanded to other use cases that we thought were easily automatable, easy to disambiguate. So we definitely started out by looking at all the ways our customers express a similar intent and then narrowed it down to those categories—okay, here’s the things we’re gonna do out the gate. And as customers talked to AVA, we use that to help us figure out where what we need to do next.

MJ: A lot of times people want to be able to answer all of these questions, and I think it’s actually a really good move that you guys chose just to start with that activation code and then go from there. Sometimes you bite off more than you can chew otherwise.

RR: I was just talking to someone from VMware today and I was telling her that if we waited until we thought we handled everything, we’ve still wouldn’t have launched. And honestly, you would have missed the mark anyways because customers interact with virtual agents so differently than they do with humans. So even training [bots] on everything you think they’re going to ask you based on historical human chats, you don’t hit everything that they’re gonna talk to a virtual agent about.

MJ: What are some examples of those differences that you saw?

RR: We have actually seen that…we call it shadow volume, of all these new areas that customers are trying to talk to us about, but they don’t really understand our terminology. An example is, we will say you need a license or a serial number or your license and a product key, and a customer will come in and say I need a license key, and that doesn’t exist in our world, so we need to help you understand this or this. Our agents hadn’t ever heard of it, so they couldn’t help us with it. So it was something that we weren’t seeing with our agents, because I think once you’re on the phone with someone, you kind of just stumble through “I need something that helps me to do this”. But when you’re talking to a virtual assistant, you’re actually trying to come up with the name of, like, “what do I think I’m asking for?” And so we started to pull out that terminology that customers are actually trying to communicate with us about, that we weren’t really getting in those one-to-one human interactions.

MJ: That makes sense. Now, going back, I think you mentioned earlier that you guys were working with IBM Watson. How did you choose between vendors, and why did you guys end up ultimately going with IBM Watson?

RR: I actually get asked this question a lot too. When we started this in the proof-of-concept phase, it was even before I joined, like, two years ago. There were really only a couple major players in this space. IBM and Microsoft were in the enterprise space, Amazon and Google were kind of in the consumer space. Those were really the big players. Looking at enterprise, Microsoft at the time—I don’t know if this is still the case—you really had to be plugged into their CRM back-end in order to get a lot of the benefits out of the box of their machine learning algorithms, whereas IBM was more of a you could plug-and-play with anything. I really liked the flexibility it afforded us.

We haven’t second-guessed it, honestly. We do a lot of market research within my team around other platforms, and IBM has the most intuitive interface, it gives us a lot of flexibility from both an engineering and dialogue design standpoint, and IBM as a company has really partnered with us from the get-go since we were an early adopter. We constantly meet with them and compare roadmaps and really figure out how we can solve problems together, so it’s not just the product, it’s also the partnership that we’re getting.

MJ: The ecosystem for IBM is just so large. I think anything you need there’s some partner, [or] some technical service that can help you wade through it. It is really interesting because the vendor evaluation question is a persisting question for a lot of people, and it’s sometimes very difficult to choose, because at the end of the day, a lot of vendors do the same thing, and then it really comes down to ecosystem and then service, things like that.

RR: A lot of it comes down to business goals. There are now a lot of startups that helping people just launch things quickly, and as long as you’re not trying to scale it at an enterprise level, there are a lot of really good solutions out there. But I think once you get into scaling it at the level that we’re starting to look at it, you need one of these big players. You need the computing capability and the algorithm and just the ecosystem, like you said, that is behind something like an IBM.

MJ: With Ava, [it] sounds like you guys have made so many improvements, and it started out being pretty bomb anyway but what were some of the challenges that you guys had as you rolled out new iterations or you try to incorporate new changes or new use cases?

RR: Some of the challenges that we came across were really [about] understanding the intricacies of our own problems. So like I said with that activation case, we really thought there were five major ways, but there’s actually literally two. So we have been able to automate a fraction of it but not the whole thing. Sp setting expectations on how much of the opportunity we’re actually going to be able to automate and capture with the solution is something that we’re getting way better at as we continue to do this.

We were, I think, just overzealous out the gate with like “yeah, once we put this out there, all volume can go through here.” Another challenge for me continually has been language expansion. Luckily for Autodesk,  about 60% of our volume is English, so you still get that ROI when we stay in the English space, but I can’t just localize everything I have in English and expect that to cover all the ways a customer’s going to express that same problem in Japanese. Localizing this at scale has definitely proven challenging. And then something that was a challenge early on was developing a persona in line with the solution.

So, making sure that we’re developing a cohesive experience through conversation was something that we really had to go back and kind of comb through after the fact, and I think in hindsight, I would have put a lot more emphasis on it up front.

MJ: What did you guys start with? Did you start with more of a neutral persona?

RR: We started with Otto. Otto was launched before AVA, and Otto was just a symbol and Otto did the activation proof-of-concept. We let the developers just code Otto’s responses, so he ended up…and early versions of AVA…having multiple personality disorder like however the developers express themselves was AVA’s response, so at some times she would say “shucks,” and other times, she would say something completely different. It just wasn’t consistent. I think that something to really…that I tell people to place a lot of emphasis on now is that you’re creating an experience through conversations, so persona is so important.

The feedback we got was, yeah, Otto for Autodesk, but Otto as a name sounded really decidedly rigid and he didn’t come off as super helpful out the gate, whereas females—and that’s why a lot of chat bots are female—come off as more collaborative and we’re gonna solve this together, and that is one of those things where you have to decide which bias you’re gonna buy into. Do you want to go against the grain and try to beat that bias, or do you want to buy into it because females are seen as more helpful in our society.

So really, we have put so much thought now into our persona, into our image, making sure that AVA is agnostic in terms of her cultural representation. She appears to be multinational, she has an accent, so that she’s not just American even though she’s mostly serving Americans. Ethically ambiguous is really what we call it but it was intentional to represent because Autodesk is multinational and it’s multicultural, and our support reps are the same. Our support reps are also 70% female, so that was one of the reasons we went with female as well was so that we’re really representing our Autodesk brand. I had to hire…or not have had to—I hired a creative writer to comb through all of AVA’s responses to really give her that persona and that personality and bring it through in all of her responses consistently.

MJ: Now that you mentioned you hired a creative writer, what all makes up your team for AVA? Because I think a lot of times when people think of bot solutions, it’s this technology solution, but oftentimes when you talk to people, the team is actually comprised of many other kinds of roles.

RR: Mostly when people think of these solutions, they think they need a data scientist and they’re good to go, and they’re so far from the truth! I have data scientists, I have computational linguists that are just focusing on the dialogue design and how to create or how to elicit a response through the way that you’re phrasing something. I have people that are creative writers, I have UX researchers I have business analysts, I have communications managers. It’s really a lot of people that understand the value of conversation and how to bridge technology and humanities, because it’s such a blend of the two.

MJ: You had said something at the very beginning about you know scaling bots, but also scaling human interactions. It’s not just about automating a lot of the workflows. You’re still creating a lot of roles for a lot of different people that composite this bot and you had mentioned in one of our earlier conversations that despite AVA being able to automate all of these interactions and all of these resolutions, the team hasn’t been reduced at all and I find that to be really incredible, because I think a lot of companies when they think about bots, they immediately think of that as like a cost-cutting measure.

RR: You have to decide from a corporate strategy standpoint what you’re going after when you invest in this, and what Autodesk decided was that we want to use that savings in two ways: one, so that our human agents can focus on reducing that resolution time that I talked about earlier, which is that one and a half to two days. So they handle more complex cases, [and] they’re getting to them faster, right? We’re not reducing workforce there, we’re just helping them to answer customer inquiries at a higher efficiency and a better rate. And the savings that we are seeing in things like resolution time and cost savings just from the advantages that we’re getting through AVA, we’re actually reinvesting in in my team. So my team has grown from two people when we started to now 13 people in the last year and a half.

So we’re continually expanding, and that’s really just…today, any given agent would help maybe 25 to 30 customers in a given day. AVA helps 1500 and so I’m hiring to that same ratio, so one of the things I tell people when they’re starting out is it’s so important to invest in the talent just as much as the technology. These aren’t set it and forget it solutions. You can’t just launched a chatbot and then let it sit, so you have to be willing to invest in the people and the talent, as much as you’re willing to invest in this technology.

MJ: Actually, that’s another interesting question that now I’m thinking is, what type of budget can people expect if you’re serious about investing in a bot for your company or for these types of use cases. What is a good starting budget for people?

RR: It honestly depends on your use case, so it’s really varies. A lot of times, what I recommend is before people go into budget, they really work with a company on a proof of concept and make sure they understand what their business goals are, and what they’re trying to accomplish, and how quickly they’ll want to scale it, because one of the things like I mentioned before is that I have two people and we were getting 20,000 conversations in the first month. I did not have a team there to go through 20,000 conversations to teach AVA fast enough on top of expanding the things that she could talk about. So,

really planning out initially where you’re trying to go, how quickly you want to scale, what cases you’re really going after, that will help you determine what type of investment you need to make in both technology and your team.

MJ: Wise words for sure. So where is AVA going from here? Right now, obviously, you guys are still working on the customer service side, do you guys have plans to use AVA in other capacities?

RR: So the first thing that we’re doing, or that we just did, is last week we launched a new functionality for AVA. So, Ava can now speak to you and video chat with you, so we launched voice and video capabilities for certain workflows within the AVA process. For our really complicated troubleshooting articles where we’re walking you through 10 steps, you probably don’t want to just be handed an FAQ or be sent this ten paged list from a chatbot, so we actually have AVA there to walk you through each step, and in each step you can diverge based on your understanding or based on what you see on your screen  because we’re asking you to do certain things.

So AVA can help walk you through those divergences.  To really improve in her functionality into voice and video is our next major thing, and then from there, we’re gonna look at getting her into more of the sales enablement. From a brand standpoint, we’ve said that AVA is there for customer engagement, so it’s not just support, it’s not limited to the platform she’s on today, so we’ll expand that as well, and we’re really just going to make her more of a brand touch point for Autodesk like an Alexa is for Amazon or Siri for Apple.

MJ: I think it all blends together, whether it’s about communicating for the brands or support questions, or just general questions about the company or the product, you really can’t get away from all of the different departments interacting with each other at some point, because it is about the customer, and the customer can touch a lot of different areas. For the sales enablement, do you guys have starting use case ideas already? Are you guys gonna do the same thing…were you just gonna do one thing in sales and then go from there?

RR: Right now we’re gonna start with the divergence between pre-sales and post-sales support, because what we’re finding is, like on our e-store sites, a lot of times customers will come back to the e-store site just because it’s what they know, and they’re actually looking for post-sales support. Putting [AVA] there to help with a lot of things she already knows is beneficial, but you don’t want to put her there until you actually can help with some of the pre-sales stuff as well.

So it’s helping to get customers to the right resources and answer questions about pricing models and customization and things like that, and then being able to help them if they’re coming back. It’s really that consistency of one touch point. It would be the same in a trial model, you would say, “Hey, I started with AVA, and now I converted.” And now you want to have that consistency across all fields of the customer lifecycle, so we’re just moving her up into the next logical phase.

MJ: I want to end this amazing conversation with this final pro tip question, which is “what are your top tips for business leaders who wants to implement a similar type of conversational AI solution?” Whether it’s in customer support or in sales, what are some of those things you have to consider, and what are the pitfalls that people really should avoid too?

RR: I talked about some of them already a little bit, so I’ll just reiterate a few of those. So one is that launch before you’re ready and iterate often. Don’t worry about getting it perfect, get it out there, get it learning, get it capturing customer inquiries, and then make sure you have the staff to iterate after you launch.

The second, like I said, is invest in talent not just technology. So build your team around your business goals to make sure that you’re not just using all of your budget for the best-in-class platform, and then you don’t have a team to support it because it will be limited. These solutions are still what I call “Wizard of Oz,” it looks like magic, but it’s a good team of people actually pulling levers and making things work. So that is how people need to start thinking about this stuff.

The third is that persona matters. Like I said before, I really wish we would have invested in that a little bit earlier, but I’m glad I’ll have such a strong persona for AVA that has been really intentional. We put a lot of thought into how we’re gonna represent our company, because if you don’t, your customers will. Whether you’ve been intentional about it or not, they will create a persona. And the last thing is that you should be prepared for trade-offs. I have really had to categorize things into new scope, improving current scope, and then adding new features, because there are so many directions you can go in. Of course you want to cover everything, so you’re always looking to expand scope, but then you always want to get better at what you currently launched, because you’ll find customers interact with you in all different ways. And then you want to add new functionality. This industry is moving so quickly, and you don’t want to just have, you know, just having text base will leave you behind. You need to be able to have image recognition and sentiment analysis and all these other bells and whistles that really enhance the overall experience—

MJ: —anything to do with communication, right?

RR: That’s right, and that’s on top of…you have to make those trade-offs between scope and improving.  That has really been my advice for people is to make sure you’re planning for all of that.

MJ: Thank you so much, Rachel. I look forward to seeing what other iterations and what else AVA will be doing. Hopefully we can continue our conversations another time too.

RR: Yes, thank you so much.

MJ: And thank you so much, and thank you everyone one else for tuning in, and we’ll come back the following week with another broadcast. Have a good day!