Google's Liz Reid on Who Will Own Search in a World of AI | Odd Lots

We have seen, with AI overviews

meaningfully, longer queries. We see more natural language queries. But it's also not even something

as basic of that. It can also be, like you were searching for restaurants. We used to laugh about the

like before I worked on, search. I worked on maps and local. Some of the intersection with search,

and people would just be like restaurants in New York, and you're like,

what do you want me to do with that query? Right? Like, okay, the best

restaurants in New York are going to take three months, and 99.9% of the population

can't afford to go to them. Okay. But like,

are you picking ten random ones, etc.. But like part of why people will do

that is they had a much more company. I want a restaurant in this location

for five people. It can't be too pricey. I have a vegan member. I also have like I have kids. Like, that was the question

they had in their mind. And in the old world of keywords,

none of that information would be kind of spread

throughout the web. And so you wouldn't feel confident

you could just put in the question. Yep. And now with like ever using AI mode,

you can start to actually and you see people do this,

they tell you the real problems, right? They don't take their need and translate

it into what the computer understands. They try to give the computer

their actual need and expect us to do the translation. Hello and welcome to another episode of the Odd

Lots podcast. I'm Joe Weisenthal and I'm Tracy Alloway. Tracy, you know when we talk about AI,

we talk a lot about the, the big labs, the big independent labs,

particularly OpenAI. But of course, we know that the,

the legacy tech companies, the so-called hyperscalers, etc.,

they're not just going to give up this, this groundbreaking technology

without a fight. No. And we've certainly seen

various efforts to I don't want to say catch up,

but like to keep pace, like pace. Yeah. You know, one of the. So, like the major, of the major legacy

companies, you know, which I would say are like Microsoft,

Amazon, meta and Google. I would say that

like clearly or sorry, alphabet, I would say clearly, like alphabet

is the one that has the model that people, you know, talk about Gemini, right? And in a way, that's kind of surprising,

because one common intuition that people have is that companies

aren't very good at developing the thing

that disrupts their own legacy business. Right. This is just sort of like a phenom,

a famous sort of B-School thing

that people talk about all the time. Yeah. So the issue one would think, with AI and Google in particular is that Google,

you know, famous for its search. Yeah. No one believes me. By the way,

the first time I ever used Google in like I guess it was the other late

1990s, early 2000, I told my parents to invest in the company

and they didn't listen to. Oh, really? Yeah. They don't believe me either,

but I swear I did. I you know what I, the first time I used to, I was like,

oh my God, this is so much better. So I feel so dumb because I remember

the Google IPO very well, and I thought it was very smart because

it was like I was 20, it was 2000, 24. I was very I thought I was very smart back

in those days, I don't think. Now, now that I'm older,

I realize I don't know either, but I thought I was very smart. I was like, oh,

the sheep, they're just buying, you know? It's like it's overvalued. It's like,

should I short this IPO? It's like a bubble, etc. Thank God I didn't like put on some short

trade and go bankrupt. But like I said, it never occurred to me

to go long because I don't. And I think it's the journalist's

temperament. We're all like very cynical. That's right. You have to be an optimist

to be an investor. Yeah, you have to be an optimist. You have to be willing to be

part of the crowd. You have to ride the wave

little like riding waves, etc.. And so, like, I was like,

oh, should I short this? Anyway? I should have bought it. Okay. I should have held it. All right.

You know what? I should have bought it. Even though, you know, trade

like should have bought it in like 2023 when everyone was saying

that ChatGPT was going to, in its line. Should have bought it a year later when people were saying,

oh, Gemini is too woke, etc. all of these turns,

there are opportunities and but anyway, I'm happy to just talk about

all right, now that we've gone on a very long segue,

what we are getting away. It's an important segue. What we are getting at is that in theory,

I would seem to pose a threat to Google's core business,

which is search. So if you type in a query in Google Now, I think I have on open

from the last time we recorded a podcast. I don't know why I was looking this up. It says, can you see tankers

physically from the Strait of Malacca? Oh yeah, in the Strait of Malacca

from Singapore. So if you type that into Google. Yeah, it used to be you would just get

a bunch of search results. That's right. Now you get an AI overview,

which basically pulls in a bunch of results from other pages

and gives you a sort of summary. And so the question is if people are just

going to be looking at these summaries instead of actually going to the pages

that Google search used to turn out

as links at the top of the page, what does that mean for traffic via Google

search? Well, and then the other big element here,

which is that, you know, people will stop like clicking

as much on later seeing links. What does that mean

for the advertising business? The situation is that you like,

see the answer right there, etc.. And you know, one of the questions

we have about open AI in particular is I've ever going to be able to launch

like an advertising business. Can these two things combine, etc.? We know that to this day that, Google search ads are the greatest money

printer that's ever been invested

basically in the history of the world. And so how this is going to interact

and how Google is thinking about these questions, I just say

it's it's a little unclear to me because, yeah, it's nice. I could just

I put it on the same question. Can you see

tankers in the Strait of Malacca? I don't know if these are right. I don't know if the I, I don't know

the AI overview is right, but it's there. It's there. So lots of interesting questions here. Also I slop right

like are the search results that are being turned up

actually of any certain quality. Right. It's a great point. And one of the reasons perhaps, that

a lot of people and I would include myself in this use AI more and more is because, you have like, you know,

I have some issues, let's say, with us. So anyway, I'm

going to clip that I have some issues. Joe Weisenthal anyway, we really do

have the perfect guest to talk about this. Someone who is like, really, literally right in the middle of all of this

and can answer all of our questions. We're going to be speaking with Liz Reid. She is the VP of Search at Google,

and she's been at the company for over 20 years, has been in the current role, has

been on the search team for a few years. And so, we're going to answer, hopefully

get answers to all these questions. So Liz, thank you so much for coming on

the Overlords podcast. Thank you for having me. Delighted to be here today. Why don't you just start by telling

us, like, what's your role at Google? What does it mean? Okay, you're the VP of search at a company

that people have known for being a search company, but what is your

what is your title actually entail? I lead the the search team,

that you can think of as sort of covering the product, the engineering,

our user designers and data science, sort of the team that fundamentally builds

the search product that you use. So how much of your day to day

is taken up thinking about AI nowadays versus like,

let's say two years ago, two years ago, I would say

it was also still a fairly large about, I think AI is a deeply transformative

technology in what it opens up. And I think I has been in search,

for many years in different forms. It's much more in the forefront these days with things like AI overviews

and AI mode. But if you go back several years,

it was how we transformed a bunch of ranking

with efforts like Bert and, that were built on some of the early

transformer breakthroughs. But at the end of the day, you know,

and if you go back once upon a time, I didn't just refer to general,

I refer to, you know, general machine learning and other things like that. And, you know, in the early 2000,

Google had spell correction very felt sort of revolutionary

at the time that used AI. Right? No one calls spellcheck AI anymore. Yeah, nobody called spell checker,

but like it was at the time, right. And the thing, it just shows you

how far the world has come. But the opportunity to really transform,

search, and realize Google's mission at a new level is, is really exciting

and an amazing, opportunity and very humbling to, to do this,

for a product that so many people use. You mentioned Bert, my little software hobby project

of training a machine learning model to tell whether something

is more indicative of the written or spoken word is based on Bert. So thank you for developing that. And thank you for open sourcing it

so that someone like myself can, train it, but talk to us about like, just how you're thinking

about this core tension that for years Google had a business of you go to

and you put in a term in a search bar, and then people click out and some people

and some of the clicks were organic results, and some of the clicks

were, to pay the results. And now people are we're entering this world in which people expect

to get whatever they want right there from the query,

without that impulse for a click. And you have a business that is still dominated by that click out

one way or another. So just like, you know, and I want to dive

into the details, but big picture. Is that a real tension? I think what's interesting about it

is that, the space of search is very big, and

what people are trying to do is very big. And sometimes people really want quick answers

and they want it right in front of them. And, and sometimes they want to go deep or they want to hear from particular

individuals. Right. I think there's this sort of myth

that people want AI or the web, that I, that I actually think what we see is that

people want AI on the web together. Okay. There are certainly questions

for which, like, you just want the quick answer

and then you're done. And and that's been true

in many ways for years. Right. Like we'll talk about this with AI. But I bet most of the time you look up

the weather, you just want to know what the temperature

is and you're done. Right. And you don't once,

but then you're going to go on a trip and you're going to go and actually dig in

more because you're going surfing. On other pieces. I think if you think about this, people

like, oh, I have an answer, I don't. So why would I do an ad? Well, like the answer doesn't buy

the pair of shoes you actually have to buy the shoes, right? So you still have to

go pick a merchant for that. People care often to hear people's

perspectives, right? Like you'll talk about. Okay, well,

we want a bunch of answers. And yet this is like a golden age

for podcasts, right? So clearly people sometimes

want to spend a couple of seconds, and other times they'll spend a whole hour

listening to things. And so one of the things we see

with the shift with AI overviews is that, you get more of this pronouncement

with what's what's your goal? Okay. If all you were going to

do was go to the webpage, see the fact, and immediately

click back, you're going to spend like a half a second on the page. Okay. You see those things shift. But if what you were going to go in and do

is read an article for five minutes, you're still interested in reading

that article for five minutes, right? I might help you point to the right page

so we see fewer bounce clicks where a user would sort of go and immediately come back

because they weren't happy. You see people go though,

and they want to hear from other people. They want to hear their expertise,

their perspective, their unique take. You know,

I take, fashion is an interesting example. Sometimes, if you hate fashion, then like, you love using chat bots

to replace the need, right? Like, if you didn't really want to spend

any time, that's fine. But if you were someone who was spending

a lot of time reading influencers and what they're interested in, the fashion you have not decided

to replace that with the chat bot, right? You're still going to want to hear

from those fashion tastemakers. And so there's an opportunity

with AI overviews to, help you get started and then make it easy

for you to dig in, and connect. And I think people's interest

in connecting with other people is, just as strong these days in many ways,

just at a simplistic level. Can you tell me, like, how does Google determine

whether it shows the AI overview or not? So if I type in Corgi into Google Search AI, I'm biased because I have two corgis,

but it just gives me a bunch of links to like the American Kennel Club and like

the Corgi subreddit and things like that. If I type in what is a Corgi question

mark, it gives me the AI overview. Is it just everything with a question

mark returns an AI result, or how are you actually deciding

what to present to users? No. What are we trying like? An important premise of this is that we shouldn't give you

AI for the sake of giving AI, right? The point is for it

when we think it adds value to people. And so it's not really associate

with question marks. Question marks are often

when people are looking for more of a description

or they have a harder question, which maybe a single web page

doesn't answer, or whatever else. But what we're really using

is looking at signals from users to say, does the AI overview provide

additional value or not? And so, most people I don't like,

I haven't studied Corgi, the Corgi query in detail,

but for a query like that, probably what we're seeing is, that most people aren't

just trying to figure out what is a corgi. Maybe they want to see pictures about it. They want to click on, knowing more about the dog breed

because they're trying to engage in it. And so, we basically learn over time

based on user signals, the same way we learn about like,

when should you show the weather one box and when should you show local results,

and when should you have sports? That the AI overview provides

additional value. Great. We show it. It doesn't. Then we want to get out of the way. Right. You don't want your search for Wikipedia. You go and you type in Wikipedia,

which a lot of people do. They want to get to Wikipedia. They don't want to go and say,

let's give me the history of Wikipedia. That's

not why they search for that. Right? If they search for odd lots, right, they probably want to

quickly get to your podcast. And so we have a variety of signals that try and help us understand,

when is it adding value or not? And we get smarter over time

as, as people. Both, change how they ask questions. As the models get smarter. Right? Like, we don't want to put in a review if we think

it's not going to be high quality. So as the models have gotten

more powerful, we can cover more cases, and just continue to develop

really with the focus being what is the best response to give a user

for the question they've asked? I have a question

about what you see among user behavior. And my question is like,

do you see the same user or a cohort of people

who use both google.com and Gemini dot google.com? And do you see distinct

patterns of queries from the same user, but, different types of searches,

or do people just sort of throw the question mostly in the Google

search box and start from there? Or do you see people just use Gemini

and not move, and do everything there? Like, what do you see in terms of sort of, I guess, emergent patterns about how an individual chooses

which of the sites to enter into first and how they do

different queries in each one. So maybe just so we're

all talking about the same thing. There's sort of your main search page. Yes I mode. Yeah. That's part of search. And then there's the Gemini app. Right, right across. And I would say

like there's a lot of users. So their behavior varies

across all of them. But there are some patterns. Okay. There's plenty of people

who co use across them. Okay. There's plenty of people that are actually

using several AI products. Right now just in general. Right. Not even just within Google. Across Gemini and search, the more informational ones, like if it's an informational query

than the, than the probability that they're using search

or AI mode is going to be higher. If it's a creative query, it's a product

like a more of a productivity question. You know, I want to, like, please

rewrite this to make it sound more formal. Right? Right. Those type questions are going

to be more Gemini oriented, between AI mode and and search the main search page. Some people use AI mode mostly via

AI overviews. They start AI overviews

and they transition. For those who go direct to AI mode,

they tend to do that for queries that they consider,

sort of more complex, longer questions. Yeah. Questions where they expect

that they're going to do more follow ups. Versus

if you're doing a very drowsy query, you might choose to prefer all of the Serp

if you know that, like your goal

is to just get to a particular web page, you're more likely to start

with the search result page. But, you know, there's obviously overlap

in the use cases, but across search in AI mode, AI mode tends to be

sort of more longer complex, more conversational queries

versus more traditional queries. And between Gemini and Search,

there's more of a productivity and creativity

versus information slapped on. So since we're talking

about user behavior, one of the things that seems to be happening

now is people will use and, it doesn't

matter which one, they'll ask a question, and then they will go and fact

check the answer that they get on Google. And I'm really curious

if that's something that you're aware of as a sort of user behavior. And if the idea is that

maybe Google becomes maybe not an AI over viewer per se, but maybe the sort of fact

checker of last resort for other LMS. I think we're definitely aware that that, people use, Google as a fact checker for,

for some of their, use case. I think people have used Google as a place to fact check information

pre LMS for a number of things. A friend tells them something,

you know, and sort of come, but I, I think people use search

for a lot more than just fact checking. Right. Or even just looking up facts. You know, they want to go browse

what they're going to go by. They want to go check up the sports

score of, of the latest team. And we do see with, AI overviews

that with the presence of areas, people are asking more longer questions, they're asking

more conversational questions. And so some of these questions

they started bringing to an LM, as we brought AI overviews in, they, they, they took on those same types of questions

and have brought them to, to search. One of the complaints about search. And I would say I've complained about this

or complained about is how many of the, results in the Serp are like very like almost like two. Timely. Let's say someone enters the news,

there's a headline about so-and-so, and I'm like, I'm

really curious about this person. And so I, I like search their name

and I get like ten results or however many results, all from the last day

since they made the news. And it actually is sort of difficult

for me to find information that didn't have the context of the news. So it was like unbiased in some way. Start first in terms of there,

is there really like is that recognized as an issue? Because I certainly feel it is a user. But I'm curious from your perspective

at Google whether this is something that you think about as a way

in which search becomes less than ideal. One of the things that's, generally very challenging about search is that, people enter the same query with a

like that's often very short. Yeah, with a lot of different intents

in mind. Right. And so, you know, in a bunch of the examples

where you have probably our data says that, that most people

just want to see the recent articles and those are the ones that get

all of the clicks. But you didn't. Right. Yeah. And so, how do we figure out across all of the different intents and,

and match them across. And so I think there's

this question about both, how do you get the different facets

of, of a question? How can you personalize the results

more effectively for people? I do see, more this is anecdotal

as opposed to sort of complete data, but but to your example, people

using AI mode and AI overviews for some of the people queries

to like, understand more about the person. Yeah. Independent of the rest of the,

the news articles. Because, you know, if you don't know who most of the people searching for it

know who the person is. But some of the people

searching for it don't. And so you can you can see behavior like that. But I do think one of, sort of interesting things

about the evolution of AI is that that people stop talking

just in keywords as much and they start expressing

more of, of what they want. And then that becomes much easier

for us to give an answer. Right? So if you say, you know,

tell me about someone versus like, what's new with someone,

that's actually much easier for us to figure out how to give better results

than if all we just say is someone. Right? We used to talk about the the an example query

a different way is, is falafel. What do you want to know with falafel?

Some people don't know what falafel is. They want a definition.

Some people want recipes. Some people want to find where to eat. Some people want nutritional information. They all just use the word falafel. And that's just harder to figure out how across

all of them do that. We also see that with tensions on things

like, well, do you want video results or do you want

more, more text based results? People are

people are very opinionated about what the right answer is, but they are not

very opinionated in the same direction. Just, and so we try and meet, yeah,

you know, multiple billions of people's needs at once. Just to be clear,

the falafel example is great. Would you say that today

you see a greater diversity of falafel related queries, whereas maybe like 5

or 10 years ago you just get falafel and now people know that they could type

in, whereas where does falafel come from? What is a flavor falafel recipe, etc.? Have people gotten more sophisticated over time in their query

specifications about falafel? I don't know, falafel very specifically. But in general. But in general, yes. Okay. Okay. We have seen, with eye overviews,

meaningfully longer queries. We see more natural language queries. But it's also not even something

as basic of that. It can also be, like you were searching for restaurants. We used to laugh about the

like before I worked on, search. I worked on maps and local. Some of the intersection with search, and people would just be like restaurants

in New York. And you're like,

what do you want me to do with that query? Right? Like, okay, the best

restaurants in New York are going to take three months, and 99.9% of the population

can't afford to go to them. Okay. But like,

are you picking ten random ones, etc.. But like part of why people would do that

is they had a much more comp. I want a restaurant in this location

for five people. It can't be too pricey. I have a vegan member. I also have like I have kids. Like that was the question

they had in their mind. And in the old world of keywords,

none of that information would be kind of spread

throughout the web. And so you wouldn't feel confident

you could just put in the question. Yep. And now with like ever using AI mode,

you can start to actually and you see people do this,

they tell you the real problem, right? They don't take their need

and translated it to what the computer understands. They try to give the computer their actual need

and expect us to do the translation. And I think that's, really exciting to see

because when we can be more helpful, but also, like,

those are real problems people had. Let's actually,

if you go back to the mission, was organize the world's information and make it universally accessible

and useful like that useful part. Right. It's not just that it's organized. Is it useful to you? And I think

one of the most exciting things about AI, the transformation going on right now

is that you can actually make information much more useful to people. And that really opens up that makes it, so, so people just ask more questions

because we can actually do a better job meeting their needs. Does that come with any like complications in terms of privacy

or competition for Google? If people aren't using keywords

as much anymore, if they're doing basically like query brain dumps

into the prompt and saying like, you know, I am so-and-so, I have a kid, I live here,

I want to do the following things. This is my issue. Does that

is that like an added layer of complexity that you have to deal with as,

like a large search company? From a privacy perspective, we give people

sort of, a range of different things. They can be sort of incognito, they can be

signed out, they can be signed in, across. So I and

I think Google has a long tradition of really treating people's data, with, a great deal of care

and having cutting edge security and privacy by design. So I think people are seeing the value

and they have, continued their trust in Google. I think it means you have to do a it's

a harder job on quality, right? You you have to take this question. That's many parts and you have to

figure out how you break it apart. And you have to do, work

to think about things like, like latency because you can't just, you know,

if everyone uses the same keyword and it's not personalized,

then you can cache it all. If all of a sudden

the queries get much more diverse, you know, it has consequences there. But I think we we just see

that it's very empowering people, right? That that it takes some of the work

out of searching. You know,

I think sometimes people think, oh, a few years ago they said like, oh,

what more can you do with Google search? But like if you actually ask them, okay, when was the last time you spent

20 minutes searching when you would have preferred to spend two? It's actually not that hard for me. Oh, last time I was trying to like

go find a service prior. The last time I was trying to go like do

these bigger tasks in life. And so it's been kind of exciting to just, like, make people's lives easier

by helping them address their real need. I have so many different

theoretical questions I can ask. Here's one. Actually, you know, this is something that was inspired by our producer, Dash,

a conversation that I have with him ten minutes ago prior to this,

and really just something else, I imagine in your career at,

Google spanning, over two decades at this point,

you've been involved in quite a bit of recruiting and recruiting

software engineers in particular. And I imagine that's

a particularly important aspect in some way or another

for the VP of search, is the nature of, given what we've seen with AI coding

and so forth. When you're doing one of these, you know, LeetCode software developer interviews, etc.,

is it different today than five years ago? Do you have to think really differently

about the battery of technical questions that you would propose to a software

engineer today, given the just the restructuring of the nature of the job

in a world of AI generated code? I think the process

is definitely evolving. I wouldn't say that

we have, perfected the science yet. Okay. But there's there's sort of, two angles

in which you're thinking about it, right? One is,

you don't want to ask questions for which they just go and type in the answer

in the chat bot and recite it back to you. Okay? So you need to make sure, like, to the extent

that your goal is to understand, are they critically thinking

or are they able to think through a problem and do that? You want to make sure that that's actually what you're assessment quite

and so is in person. How are you doing

that on on on some basic way. But there's the other thing that I think, you know, the tools are powerful. You can use them in ways

that make you more effective, and you can use them in ways

that make you less productive. How do you think, as

the fluency is changing with, with AI? That sort of what, a software engineer,

the way a software engineer might approach the problem now is different

than they might have approached the problem five years ago

without some of these tools. And so I think we're all learning, how to change asking that question. Right. Are you building up that expertise?

Are you building up that fluency? And the the fluency isn't fixed

like what was possible with the tools six months ago, let alone two years ago,

is different than what's possible now. And in six months it will be different. So you

so you have to start thinking about how, part of your interview is thinking about, sort of fluency with the use of tools

in the same way that like when IDs became, important, or when people

stop using assembly language and they started doing your job, right,

you had to sort of evolve, the interviews. It's just that it's happening very fast. Okay. Right. And so we all have to be, on our toes, but exciting. Like, you play with this tool,

it doesn't work for something. And then. And then it's not like play with it

two years later. It's like, play with it

three months later. Maybe the tool will now

work for these things. Okay, well, speaking of tools, I mean, one of the things I, Joe,

I think you've said this, we've been playing around with Claude coat

this idea that actually when you start vibe coding everything

and telling your agent to do everything, it feels like you don't even necessarily

need, like a computer, much less a search engine, presumably. So I'm just curious if you gaze

like, 5 or 10 years into the future, what do you think the default entry point for interacting with

the web is actually going to be? Is it

going to be a search engine like Google? Is it going to be a specific limb? Is it going to be my personal agent that I

vibe coded for all of my preferences? Yeah.

Just can I just add on to this question? I feel like this is something I think about, like

if I want to send an email to Tracy today, then what I have to do,

like I find the tab in my browser, I scroll over there,

okay, that's my gmail tab, etc. whatever. I would like to just,

you know, be in my terminal. It would be so much easier to say here,

send an email to Tracy saying that there's all these steps that I currently do

because of the nature of graphical user, graphical user interfaces that once that now that I've gotten in like Claude coding

or whatever like feel a little clunky, it feels a little yesterday and so like,

yeah, I'm extremely curious about like, will the web with these series of boxes

that we drag and drop and etc., is that the future or will just be someone

talking in English to their computer? I don't think like ten years is

a long time right now where the tech is. That's right. As in one year. So we still have browsers pretty much no. And in the what I like is,

you know, it's people like, okay, we believe in ten years will achieve AGI. Will anyone be doing anything

the same. Okay. So with that but that aside,

I think there are some things I, I, I believe in and some things

I think we don't know. I think you already see, if you go back ten, you know, 20 years ago that the way you interact with

the tech has evolved a bunch, right? It used to be it was just the laptop.

Well, now it's the phone. Well, now it's also the watch. Okay. In some cases it's the glasses. The sense that it should feel like, that the information is sort of, at your fingertips in whatever medium is useful. Right. And but I don't know that this becomes a, we haven't so far replaced

all of the old ones, right? Like, you use the phone a lot more, but my guess is you're not doing

all your cloud code work on your phone, and you're doing some of it

on the desktop. That's true. Right? That the introduction of the watch

has supplemented it, but it hasn't

eliminated the desktop. Right. So what's been interesting actually,

is that it hasn't, got in the direction

of converging to the answer. It's actually increased the form factors

and so that you want to be able to access this information sort of wherever you are

in whatever form factor makes sense. And so, you know, will it be glasses,

will it be something else. Quite possibly. But but let's say, let's even say it's

glasses become a big deal. Glasses are very small screens even there. You're probably not going to do

your big productivity thing on desktop. So I think what you'll see is that the

access point is not confined to one thing, but that, the, the key is to eliminate the friction. Right. And the toil to your point. You had to do six steps. You didn't want to do the six steps. Why should you do the six steps? Okay. I think you see that, like,

some things are much easier to do with a chat interface

and then some things. Actually, a chat interface is a super slow

way to go. Do it right? Like if you have a list

and you have to go say, please remove this long title for the 10th item. That's actually much harder to do with chat than like an interface

that that does that. Right. So I don't think it

necessarily converges on a single thing. I do think it should feel much more

adaptive to your point about like, well, if this is the way you prefer to interact,

not just where you are, but how you interact,

and can you customize and can you create, sort of dislike? To what extent do the user interfaces

look designed for you versus designed for general,

and can you have influence in them? I think you'll see that. I do think we sometimes, like we're very aware

of what doesn't work. Well, we're not necessarily aware of

what does work well, like people spend, companies spend huge efforts

working on how do they do shopping carts. Really? Well, yeah. Okay. This belief that sort of like the

the chat bot will have a more optimized shopping cart

for every shopping cart place in the world than the one you go to every day. I don't know, not not clear. Right. For those things. But I do think it should feel much

more personal. It should feel much more dynamic. It should feel much more ambient

and available to you. And I don't think it will be one size

fits all, either per person or per form factor. First of all, that makes more sense. On a separate note,

I've been reading some articles there. I think I saw one. There was a big one

in the information recently. Let's talk about one of your competitors. Meta. It's kind of.

Yeah, it's a competitor. And there was like everyone's token maxing

there, and there's a token leaderboard, and people are competing to show

that they're using AI more than others. And from my perspective,

that boggles my mind because compute as a cost and just using compute

per se does not strike me as a particularly good way of measuring

who is productively contributing to the company. I mean, I could certainly find an easy,

quick, recursive way to burn tokens and generate a bunch of

AI images of corgis. Yeah, generate it

and then tell the tell it, create one that just keep selling it to improve

itself, etc.. The flip side,

which some people say is like, look, it doesn't matter at this point

because everyone has to figure out how they're going to use

AI productively in their work. So you know what?

Don't even worry about metering. I tell everyone to pedal to the metal

and I use and if someone is maxing out on tokens,

it means they're experimenting with something and then they'll find something

that really is a productivity enhancer. I'm curious if from your perspective,

it makes sense to like essentially see token consumption

or compute use as a proxy for like someone who's doing their job

aggressively. Well, I think the thing

with all of these proxy metrics is if you use them blindly,

you're going to run yourself into trouble. Okay. If you as if you as a leader,

sort of don't use judgment on them, then you get the example of like,

I will just create a job that runs in the background

and does dumb things to to do the tokens. Right. So, you know, as a leader, your, your job is to sort of like use good judgment

and not just think about the incentives. So if somebody isn't playing around at all

with the tools, when we know

that they can improve productivity, then we need to figure out like why

and how we help support. Like maybe there's some issue with with the part of the system

they're working on and we should go fix, or maybe we just need to help upskill them

or whatever else the case is. If I do think there is a level

of experimentation required, right? So I don't think it works. If your answer is like,

you need to ensure that all your token use is completely optimized,

like it's not going to work, right. People have to learn what's possible. They're doing different jobs. The tech is changing,

but it can either be like, don't use the tools or just max your tools

blindly. It's a it's a noisy signal,

but it's a signal, right? So go look at it and understand

as a as a place of where to look. Don't use it as a final judgment. So speaking of measures,

and not oversimplifying them, I want to go back

to that sort of core tension that we started the conversation out with,

which is, you know, the AI results versus people actually clicking

through to results and generating traffic. And I know you were talking about

AI being expansionary or complimentary for Google search, but I'm very curious

how you actually measure that. And the more granular

you can get on this, the better. Like what are you specifically looking at

to say that actually this is something that's good for our business versus

something that's detracting from the core. Google's guidance in North Star

has always been like focus on the user. So so that's our that's

our biggest question at the heart is how do we make a great

experience for for users. And then you know, then you want to then you want to be thoughtful

obviously about other consider okay. You like if you don't have a healthy ecosystem

you can't build a a service on going. So you need to make sure you're

nurturing a healthy ecosystem. You need to like if you have make no money

then you can't fund right. This wonderful service. So you have to be

thoughtful about those okay. But but the the place you start with is

try and build something amazing for users. One of the things we've seen, I get it again

with, with Google search is, if you're doing a really great job, like people will, not just do another query, they will

they will come back to you more often. Okay? They will take their phone

out of their pocket and extra time. Okay. That's a high bar, right? It's one thing to go and say,

I've showed you something. Can you do can you do one more thing

while I'm showing it to you? It's another thing to get you to decide. You're going to bother

to unlock your phone. You're going to boot up your desktop.

You're can navigate on the browser. And so one of the things

we really look for is when we're doing these changes, does it cause people,

to come to search more often, not just use search more often,

but become more often? We also do various like UX, research studies and trying to understand

what are people happy about or not? What are the things they find are

frustrating? Are more users, more users adopting it? Not just how much are they using it? So we look at a bunch

of different metrics. But one of the biggest is really like,

do you choose to come and ask Google, like,

do you essentially hire Google more often? Right. For things you need, and one of the things

that's very surprising to people at times is, they think they come somewhere for all the questions

they have already today. Right? Like maybe they think they come to Google

all the time, or they think they go to Google

Plus LMS or Google Plus, LMS plus TikTok plus whatever. Okay, okay. They think they ask

all the questions I have. But that's not true. And you actually make a calculation

when questions go through your mind of like, is it worth spending any time

to figure out the answer to this question? Okay. And if the answer is no,

then you just don't ask the question, right. And so when we talk about AI is an expense

sharing moment, all you really mean is there's

a whole bunch of questions people have. There's a whole

bunch of curiosity. That is, people are not exploring and

they're not exploring because they view it as too difficult or too much time

or not sure that it will be worth it. And AI lowers that barrier

and it can lower that barrier in, ways that are sort of, sometimes for us

English speaker are surprising, which is like actually

in a bunch of countries. Does not all the content in the web

in the language you speak. Okay. LMS can help unlock that content. You know, AI overviews,

because it's using an L, can be more multilingual

than just the web corpuses by default. Right? Okay. So suddenly information

that wasn't available to you as a Hindi speaker is now available. Okay. It can be visual. You had a question about this flower. You had a question about that cool purse you saw, like, where can you buy it,

but you don't know how to describe it. It's possible. You can also just be like my kid has a question. Do I say, like, I don't know, or

or do I go ask the question? Right. You see, with young kids,

they ask questions all the time, right? They go, why why why why why why why. And like at some point parents are like,

because stop bothering me. Go ask them alone. They go, okay, let me show you. I'm not gonna do that because, because,

because from a kid's perspective, they assume adults know everything, right? And it is no cost to them, right? They're not worried about their time

and other things as an adult. It's not that you're not curious. You just don't think everything is known

and you don't have the time, right? And so if you lower that barrier,

it allows you to be that that kid again, that just sort of explores

all of these things, or get started on those projects

that felt daunting or, you know, enables you to save

or learn a new skill or whatever else. And that's really exciting. But I don't want to dismiss the wonder of

being a kid and learning about the world. But I'm going to sound very callous in a second,

but how do you make money off of that? Like, how do you make money off of the

I overviews? Is it just customer retention? Is that

what we're basically boiling it down to? And then if it is customer retention

then like you could have a strong argument for saying that, like everything can

just go through Gemini instead of search. Search

only shows ads on the subset of queries, right? You're like

less than a quarter of queries, right? So there's a whole bunch of queries, pretty overviews, right,

that that you don't make money on because many of them are not of,

of commercial need. Right. And so, you know, you use your you asked a question about ten years

earlier, probably pre IO. That query wouldn't have shown ads anyway. It doesn't show in the book. Yeah, I checked those two.

It doesn't work. No ads on it doesn't show ads before. There are no ads in that. Correct. Right. Nobody's trying to advertise

something on that. Okay. So those queries every is like it doesn't it does disrupt. Right. Then there's a bunch of class of covers

where like your shopping, you still the, the presence of an eye

overview or general answer. It doesn't preclude

the need to still buy the item. So there's still this huge opportunity

with the ads because there's all of this,

this choice that's going, I think you also see that there's an

expansion of queries right to this point. So you get you get more queries. And so some of those queries

are more commercial. Some of them are not,

but some of them are more commercial. And so those become

new opportunities for ads. And there can also be things like when the query is under specified, or it's a single query,

you actually don't know as much. So you can't maybe target the ads as well if people start expressing

more of their need. If it's more of a conversation,

they're going more down funnel. You can actually create better ads, right. And so you can think about

new opportunities for ads formats. Right. You know, some number of years ago

people have said like, how can you make money from a fee? Okay. Well, Instagram ads are,

you know, very popular. Yeah. Right. And so there's new ads formats

as you recognize, new technology and new opportunities. But the commercial needs are still often

there. And the desire for user choice

is still often there. And so there's still a lot of possibility

going forward. And so it's worked out very well right now

with us in the balance. So I realized that,

like I am a user of both the Gemini app or Gemini Dog, google.com

and just google.com, and I have like a sort of an intuition

of which one I go to for which purpose. So if I want to look up

the capital of Moldova, I'm like, I'll just search capital of Moldova,

which I just go, do you know what it is? Tracing? Sorry,

I'm not trying to stump you. No, it's I feel like you're going to say

it, and I know it's kitchen now. I didn't oh no, I didn't. Yeah,

I didn't know that either. But anyway. But if I, I, you know,

if I want to understand a sort of like what are some academic papers

that have been written about why it is that high frequency trading firms

tend to not have outside capital, etc.? And what was the theory for this? I think at this point

I would use Gemini for that and hope to I do use Gemini or some other ones, but

within the context of this conversation, that's more of a Gemini query for me

than a Google one at this point. Well, there always be two boxes. Or do you foresee eventually there is just

one box and it will just know this is, you know, like, why do we

why do we how why do we need two boxes? I don't know what life will be like

in five years. Yeah. I think it's very, Sometimes

people want sort of an experience. Although the information need

seems just, like, similar. They actually want different experiences

across. Okay. And so if you take sort of a APL, example, people use YouTube for search. So in the US they use it

some in India they use it a huge amount. They bring a bunch of queries that you

would bring to Google search in the US. Right. You could say, okay, well we haven't,

why haven't we collapsed YouTube

and search search box into one search box. Right. And do that and and it hasn't been the case. We have the, the Google app

and we have Chrome. Yeah. They both allow you to search

and they both allow you to browse the web. You have a set of people

that love the Google app, and you have a set of people

that love Chrome, and you have a set of people

that use both, on a phone. But you can't necessarily convince

either population that they want to stop using one app

and just switch to the other app. So I don't think that the space is so huge and it's changing so quickly right now,

that to sort of, be able to know for sure

whether or not you can sort of create one sufficiently dynamic,

personalized experience, that one app,

one entry point can truly do it all. I don't think we know yet on that. People come for restaurant searches,

they come to maps and they come to Google search. We have not collapsed

the maps app in the search app. Yeah. Some point it becomes big. Do you you know,

you're putting all this directions, call it into the Google search app. Is that actually useful

even if you had a full maps view? So I think we're just going to have

to learn over time about what's good. But but the space is really giant. And they do have, different emphasis right now on what they sort of try to excel at. And you want to make sure that in the,

in the attempt to bring things together, you don't become sort of only okay

at everything. You know, and you want to make sure that,

that you can shine at all of the use cases people need. And, and that may mean to products

or that may not or may be a third product. Right. Like, I don't know, in five years

there may be a third product. The replaces all the products. You have your personal agent,

you don't talk to you. Yeah, I don't know. So I realized we we kind of promised to talk about

AI slop a little bit in this conversation. So one of the things that's happening with

AI is not just that I can ask a bunch of questions I might not otherwise

have had time or the inclination to ask. But also, AI is being used

to generate vast amounts of content that are aimed at potentially answering

any silly question. I or anyone else on earth might have. Yeah, just churning it out. And I'm very curious

how search is weighing, I guess, the quality of its results

in the new, slop era of the internet. I think there's a tendency at times,

sometimes think about AI slop as if it's, before AI slop, there was slop. Yes, human generated slop there. There was human generated slop. Now there's AI generated slop. So there, there has

there has always been slop on the web. And so what doesn't really matter at some level

is how much slop is on the web. So much as, is there great

content on the web and can you surface it? Right. And, and this is Google's bread and butter and ranking is and has a long history of looking for spam and trying to, drop it and make sure it doesn't show. And like, we crawl many, many more pages

than we even put in our index. There's pages we put in the index

that we never surface, right. So that we can keep that rate of spam

and slop, at a very low rate. And it is a

it is a constant effort, right. Like it's not a problem you solve because, some of the people

generating the spam, right? There's a lot of financial incentives

associated with it. But that is what like we, we what people have come to trust Google

is that it will show great information. And it's a thing that we will continue

to put a huge amount of effort in. And so that's the way I would think about

it, is not like how much I saw or non or human generated slop or,

you know, whatever automated slop

I post human generated there is but making sure that

the information you do see is trusted. Liz Reed,

thank you so much for, coming on on loss. That was a fascinating conversation. I like a billion more questions

I may have. We'll have you on in, three months,

when the entire world has changed. And we'll get an update from you. Thank you very much.

It was a pleasure to be on with you. I like that point about human

generated stuff. Do you remember? Yeah,

but the difference is the value. Like, I get it, but but, like,

do you remember, Jason Calacanis, this is startup. Mahalo. No. So Jason Calacanis,

who's been on the podcast. Yeah, he had this startup for a while. This is like the biggest piece of garbage

in the world. It was called no for real. Go. People need to go

look at the called Mahalo. And the idea was like

they were just going to hire a lot of people to, like,

write articles that were not very good, to, like, appear in Google, like, oh,

I see, yeah, like the swamp. Yeah. There was a famous one

that my old colleague at, I think, and, the business insider,

Nick Carlson, discovered. And it was like, if you search

like how to play the xylophone, there was a mahalo article for that,

and it was, I swear to God. Okay. It was step one. Decide if you want to play as well. That's an important step. Step two get a xylophone. Step three learn to read sheet music. Step four practice

reading sheet music and play this. So like this is actually like this

I just remembering like it is people have been trying to like stuff

complete garbage into the search result for a very long time. And I, I always get a chuckle thinking,

that's so good example and you should go look for it. And so

I'm glad that I'm looking for it now. I'm very distracted. It was so bad. Wait, we're just going to,

like, laugh about this article for look at this research. Mahalo. How to play the xylophone. Yeah, I see something from mahalo.com

on YouTube. Like can't see it. Yeah, yeah just search. So yeah. Business Insider February 21st. Hilariously useless. Mahalo as a guide to playing this, I did

you write that? No, no. Carlson

oh, I see what I see. Yeah. And I,

unfortunately now it's behind a paywall and itself is covered a sloppy ad,

so I guess. But anyway, sorry. Decide whether you want to buy

a used or new style of phone. Metal silo phones are less expensive

than wooden ones, but that's useful. That's a useful. Yeah, know,

but what do you see that it's like, please, I like,

save us from this human generated garbage. They were trying to clog search results

before. On a serious note,

yeah, I did think the point about, not customer retention, but, like, expanding the volume

of, like, user queries on the platform make a lot of sense, which I hadn't, like,

really considered that much before. So even if you do get a bunch of like,

no click users, yeah, they are more inclined to come back

to the platform in the future, and maybe some of that eventually lands

in clicks. The other thing I hadn't thought of, and I thought it was a great point,

which is that there are multiple Google

currently runs multiple search boxes. There is the YouTube,

you look at the Mahalo article and cracking up? sorry, I know it's step four is experiment

with it with different mallets. It's okay. I just really want to play the xylophone. Yeah. It's so it's so big. It's so good. Step five is practice regularly. Yeah. It's like I think I it's much better like if I, you know, it's

so good isn't it. It's like the biggest steaming pile of

garbage you've ever seen on the internet. 20, you know, 15 years or ten years

before anyone can we can we actually can we have Calacanis back on the podcast just to talk about

I should have Jason back on the podcast, just to like, grill him about

like what exactly he was thinking. And you know, and there's like his sins against the internet for having put this,

put this on there. Well he was an early adopter of non

AI slop. Anyway, that point about like

there are multiple search boxes already. Right there is the YouTube search box. You're still left. I'm sorry, I was trying to make eye contact with you

and not look at my computer. And so just think, should we just. Should we just leave? Shall we leave it

there? Let's leave it there. This has been another episode

of the Odd Lots podcast. I'm Tracy Alloway. You can follow me @tracyalloway And I’m Joe Weisenthal

You can follow me @thestalwart Follow our producers Carmen Rodriguez

@carmenarmen, Dashiel Bennett @dashbot

and Cale Brooks @calebrooks And if you want more Odd Lots content,

you should definitely check out our daily newsletter. You can find that@bloomberg.com/oddlots And you can chat about all of these

topics 24-7 in our discord, discord.gg/oddlots And if you enjoyed this conversation then please leave

a comment, like the video or better yet, subscribe Thanks for watching.