How do you sort of measure that? One of course is, did the user achieve their intended goal? And the goal is such a variety of things. It could be a one off like turn on the light, play a song. It could be a more complex back and forth about looking for a hotel room in New Orleans or something. It could even be Chitchat.
That could be a goal. Like I want to have a short little pleasant conversation with the system. So truly understanding the goal and not being too vague about the goals is very important. But of course in addition, did they achieve their goal? Was the process pleasant and efficient?
- Home and Small Business Guide to Protecting Your Computer Network, Electronic Assets, and Privacy?
- Turing test.
- Childhood's End!
So maybe I booked my hotel, but boy I was pulling my hair out. Another thing I want to caution people is not to be fooled by number of turns. But number of turns does not necessarily reflect a good or a bad experience. Sometimes breaking down a complicated question into two questions is actually a better experience.
And so again, just looking at number of turns is not necessarily the right way to go, but in terms of like really specific metrics, two things we absolutely look at are no matches and no input. I might say set a timer for AM tomorrow morning. But should be able to understand that. So really spending time looking at all those matches and figuring out is it grammar coverage, is the prompt confusing, is the flow confusing, all that kind of things.
You expect to be 15 questions and people are dropping out often at like question eight or nine. Why is there a dropout rate? And then retention, although again, I caution people retention. Is retention relevant to your particular thing? It could be something that someone only does once and that was a success or are you building something where you expect people to come back?
And so that could be another thing to look at. They sort of get invested and they move further down that flow. Cathy: Of course, this is just my personal speculation, but I just read this really great report that came out from AnswerLab. They did a diary study for people who have smart speakers.
The Network is the Computer: A Conversation with Ray Rothrock
And one of the things they mentioned is that users as they have these digital assistance over time, they want more fun and engaging conversation. So I think we started out very utilitarian, like set a timer and play music. So I think that speaks to a need that we want to have a little bit more fun and engaging experiences. Another thing I think will happen is conversations in more places. So I think technologies like that will come about that make it more natural.
When will we be able to have a conversation with a computer?
People will be more likely to use a speech in public, things like that happen. So for example, right now, if I asked to play the song Last Christmas, I want to know that I want the wham version and not the Carly Rae Jepsen version. It should know a few things with my permission, it should know a few things about me like that that will just make it a little even more frictionless and more enjoyable. Well, this has been super interesting. I really appreciate you coming on and taking some time, but before I let you go, I have a couple more questions.
What are some lessons that you learned sort of over the arc of your career and where you are right now that you think might be helpful to people who are just getting started in product? Cathy: Oh gosh, that is a big question. I started out as a software engineer and I kind of after a few years realized that I was really more interested, I love programming, but I was more interested in the design side of things and realizing that that could be a career.
Um, but I said yes anyway. Maggie: Awesome. Yeah, I love that advice. How do I know where I should be in the next however many years? And on top of that I would say, and again, totally, I know different people have different circumstances and not everyone can change jobs and things like that. And so sometimes being willing to maybe take the leap and try a different job, which may or may not be better than the job you have, but being willing to maybe try something because nothing is permanent these days.
Well, thank you Cathy. I really appreciate you coming on this episode. Obviously five stars only, and thanks for coming on the show. Every Sunday evening we'll send you a roundup of the best content and events from Drift and around the web. Make sure you're ready for the week! Subscribe now.
By entering your email below, you agree that we may send you emails about Drift's upcoming services and promotions. Is this okay with you? You may unsubscribe at any time.
- What’s Microsoft’s vision for conversational AI? Computers that understand you - The AI Blog;
- Creating Intelligent Computing: A Conversation with Rosalind Picard.
- Subscribe & Tune In!
- Google’s Cathy Pearl on Why Conversation Design is the Next Frontier.
No, not now Yes, I approve. Cathy Pearl: Thank you. Again- Maggie: Smart speakers being Amazon Alexa?
Cathy: That was the first one. Cathy: Yeah, that makes sense. Maggie: Oh, interesting. And then that was already magical the first ping as it went across the network.
And then, can I Telnet to one of these? So you know, getting the networking actually running was sort of the key thing. How important was networking for Sun in the early days? Was it always there? Rothrock: Yeah, it was there from the beginning, the idea of having a network capability. We sort of mimicked the mainframe world where we had green screens hooked into a Sun in a department for example.
And there was time sharing. But as soon as you got a Sun on your desk, which was rare because we were shipping as many as we could build, it was fantastic. I was sharing information with engineering and we were working back and forth on stuff. In those days, to send an all-Sun email around the world, modems spun up everywhere.
I asked him what he was doing. He was in a little computer room.
The Network is the Computer: A Conversation with Ray Rothrock
I was trying to typeset something. And I thought, oh my goodness, this is incredible. And then along comes HTTP and all the other protocols that followed.