🚀 Add to Chrome – It’s Free - YouTube Summarizer
Category: AI Healthcare
Tags: AIHealthcareMedicineResearchTechnology
Entities: Carnegie Mellon UniversityCosmos databaseDARPAEpicGrockMicrosoft ResearchOpenAIPeter LeeProvidence Health SystemStanford
00:00
00:15
All right. Uh, welcome to another episode of The Pod.
Uh today's guest, if you're a regular uh listener of this of this podcast, uh really needs no introduction, but it's Peter Lee uh who's one of my uh colleagues here at Microsoft and he's the president of
00:31
Microsoft research and he's focused on steering the company's worldwide network of research labs incubating breakthroughs, research powered business on artificial intelligence, computer foundations, health and life sciences. covering the entire breadth.
Uh before joining Microsoft, uh Peter headed the
00:49
transformational convergence technology office at DARPA and also served as chair of Carnegie Melon University's computer science department. Uh he's a fellow of the ACM and actually recently elected member of the Mayo Clinic board of trustees.
Uh and of course you may know
01:04
him as well from uh his very influential book from 2023 called the AI revolution in medicine GPD4 and beyond. Uh, and then of course now the follow-up podcast series which we will get into today.
Uh, so welcome Peter. Thanks for thanks for joining us.
01:20
>> Oh, thanks Matt or should I say Dr. Lungren.
It's great to be here. >> Only only part-time now.
So that >> so Justin I I know that you know we and I have been uh kind of trying to keep up as we always do with the show of like
01:36
what's the most current uh event and and how are we following some of the trends and then how does it impact healthcare. I feel like the last few days has been uh even more frenzy than usual and I'm hearing that July overall will be crazy.
Um but I think we're kind of recording
01:52
this around the time of the Grock launch. Um so I think it's probably the first topic that um at least I'm interested in covering.
So I don't know if you have some of the latest info. >> So I've just pulled up now they just released their benchmarks.
You know this
02:09
this will be very similar to people who have been listening. New model comes out more money more compute better performance.
Uh I'll I'll state just a few other parts. It's uh amiss a few other things.
The CEO has stepped down. There's been some differences on
02:25
alignment and training and some controversy around the model. Uh we won't dive into that today.
But a few of the other things just to go fly through from kind of the data, you know, new model kind of jumping to the top of the benchmarks. Uh this is
02:41
starting to out compete, you know, 03 Pro, Gemini, 2.5 Pro, etc. on kind of a multimodal benchmark putting together here.
And then I'll show one more here talking about some of those specific exams, um, ARC, AGI, etc. And I know
02:57
Matt actually you had some extra comments you wanted to cover on this one. Well, yeah, I think I think this is this is a great uh opportunity um now that we have Peter with us too is just that, you know, I think we talk a lot about like bigger models, uh larger clusters, um maybe even more data.
And
03:15
that's sort of that pre-training world. And I feel like especially since O 01, if not 03, we've we've I've kind of, you know, DeepSeek and some others, we really have started to think more about um post uh post training, right, and test time.
I I don't know how you're
03:31
seeing this. I think there was that Gnome Brown quote from OpenAI who basically said that just a few seconds of letting the model quote unquote think is equivalent to you know uh scaling up by like a couple X.
Um how are you
03:47
looking at this? And then do you feel like this is uh yet another opportunity just to continue to infinitely scale at some level to the limits of physics I suppose uh and electricity right to get to you know that next big breakthrough.
Well, I think for sure at least for
04:04
people who are involved in the more research side of the development of these systems um that the the hot area and the focus is on post- training and and on inference time compute. Uh I I think that's where a lot of the thought
04:21
is going in u because it's getting harder and harder to get uh breakthroughs and real advances uh in the pre-training phase and and pre-training also is just high stakes. It's sort of like getting to the point where uh to have another pre-training
04:38
run at scale is sort of like making a commitment to I don't know a new silicon processor architecture. You can't do too many of those things.
And so uh there's just a lot more flexibility and a lot more opportunity if you are an innovator
04:55
uh to try out your ideas uh in the post- trainining phase. So there's just a huge flurry of activity there.
You know, one thing though I think does seem to be we don't know everything about why these reasoning models are working so well,
05:11
but one thing that seems to be true is that the the pre-trained model that base model has to be very good in order to reliably get good results. um uh in these reasoning uh paradigms.
05:28
>> I feel like the it seems like there's so many different paths on the on the test time compute, but also uh that we're I I worry that we're chasing benchmarks because I I feel like I in some cases like it shows super awesome performance on a lot of
05:44
these benchmarks. There's that old like is it like the law or whatever where it's you know uh when a measure becomes a target it ceases to be a good measure.
that whole idea, right? Like I feel like it's now almost we all are becoming more comfortable with various models maybe in our daily life and then we kind of get
06:00
the vibes I want to say or feel and we know our task pretty well and we get that Ethan Mollik's Jagged Edge understanding and then we can sort of say okay that one is better because I've used three or others for this exact same task and you know I feel like that is you'd almost have to wait for the
06:16
reaction as opposed to overreacting to the benchmarks. Um but this is pretty promising either way.
Oh yeah. Yeah.
I think it is part of a the evaluation problem of course is hard because it depends so much on what it is that you want out of these models. Um but yeah I
06:34
think um we're getting a better and better sense of this. you know out of Microsoft research there's a very experimental uh evaluation approach called Adele you know which actually use ideas from psychometrics uh to try to evaluate uh these things
06:50
and you know I think it that's different than the current benchmarks um that people are using which are which stress test you know problem solving ability logical reasoning and world knowledge much more directly um but in in the end
07:06
I I think what we want are collaborators. We want things that can work with us.
And so the question is how do you eval like when you hire an intern, how do you evaluate whether you have a good one or not? And and so I
07:22
think we're still trying to figure that out. >> By the way, you know Oh, go ahead, Justin.
Yeah. >> No, I was going to say but but you you you kind of know and there are things you expect your intern to do.
Do they retrieve the right data? they ask, do they follow the instructions well?
Are
07:37
they good at communicating back the results? Are they giving you leverage on your time, which is kind of the ultimate metric that that we're shooting for?
Um, and I do, you know, love, and I know you spoke with Ethan Mock as well, who talks a lot about just these these concepts.
07:53
So, there there are some ways uh that that we talk about it. >> Yeah, I think you want uh someone who's a good listener also uh knows when to ask the right questions.
it doesn't waste your time with those uh knows when to go somewhere else for help, you know.
08:09
So, there are uh things um there are some things that are still missing. Memory is a big one.
Um you want someone who kind of learns from experience uh and remembers what you like, what you don't like. Um there's something called entitlements,
08:25
uh where if you give permission to use certain kinds of tools that it knows how to go out, learn how to use those. um and and and use those kinds of things.
And so, you know, things are coming along and I think we're going to see
08:41
continued breakthroughs as these new capabilities get integrated into the models that we have available to us. >> I mean, I think this this is a great allegory for I mean, healthcare, right?
Because like I mean, I you know, I was a part of the you know, admissions committee for med school. We obviously
08:57
uh interviewed residents and yes, test scores played a role in that. Uh but everyone was always looking for that magic like combination of factors that led to phenomenal physicians and it and it it it's just so hard to do uh with
09:14
with humans. I mean for a while there was a period of time I think Justin in the was it maybe the early 2000s where was like liberal arts majors but also good MCAT scores.
That was the thing, you know, that folks were and they were kind of moving away from like your typical biology major. Then everyone was
09:30
kind of like where do we have a interesting like philosophy major we can grab, you know, and and I and I don't know if that actually led to any meaningful selection. I mean there was lots of theories behind it but there's a similar I think effect here where um it
09:45
really is feel and using it and all those things and to your point the things that are coming or the things that are being explored around um being able to complete a a complex task in the way that you want it to um without a lot of intervention. I think those are the
10:01
kinds of bars that I'm looking for personally because you know I I get more frustrated like you look at this kind of data like this is what folks are really looking to do and this may require additional post-training work in terms of like literally RL on a job category
10:17
like the the folks at thinking machines I think are you know famously trying to look towards which is take a given task and actually have folks you know sort of think their way through the task. Could that potentially be a a path towards a reinforcement learning?
And that agent
10:34
based approach is that's it ends up being really good at completing these long tasks without getting off into the into the ditch which you get super frustrated. You're like, "Oh, it's going to do it." And then like it comes back, you're like, "Man, I I waited for like 20 minutes and it, you know, uh it it
10:49
totally missed the boat and I want" and you almost want to pause it and stop it and you can't. Like there's moments like that.
You know, it's going to change, but um >> Right. Well, um, Matt, you were involved in that very interesting work on the healthc care agent orchestrator.
Um,
11:05
and, um, you know, for people who aren't familiar with that, it it's it's literally an agent that participates in an online tumor board meeting and can facilitate the use of other uh, AI models, but also helps facilitate uh, a
11:23
meeting in a really interesting way. And I think people who are getting to use that uh kind of early on particularly at Stanford um I think are finding it to be surprisingly useful and and there's a framework there that's interesting but
11:39
you know you and I have discussed this Matt it is still so limited for example as a participant in the meeting uh it doesn't have the ability to raise its hand and like butt into the conversation um there's a level of proactive
11:56
it it's a pure ser it's a secondass citizen still uh today and as we think about the future of these types of systems and the human machine collaboration that obviously where we're at today with
12:13
that orchestrator agent can't be the end point um there's something that I think is going to be much more of I think an eventual equal citizen, first class citizen um in
12:29
those kinds of settings. >> It it is really interesting that you know we all spend a ton of time looking at the newest research things that are coming out.
Very very little today is still at the bedside actually delivering care,
12:45
>> you know, and you know, Matt's been following, you know, the AI and radiology side of this for forever. we're just starting to kind of get more and more adoption.
And so, you know, what we can prove in a lab is still just so far away from from what we actually
13:01
see in in practice. >> And and it may turn out that, you know, to your to your point, we have we have been building like tons of narrow models for a long time.
The way I've my framework has shifted especially with like to your point Peter about having different agents is now it almost makes
13:17
it like I can think this one 1 plus one equals three to me because like those narrow models we spend a lot of time on those those are tools now you know for a given agent as long as the like to your point the super agent has at least enough contextual understanding knows what I need to get done understands the
13:33
intent and whatever protocol ends up winning today looks like MCP or whatever that's going to end up being can figure out, okay, you're a you're a model that does X. You measure lung nodules.
Great. That was part of the thing that that, you know, I was asked to to do in my task.
I'm going to check that box off
13:49
and I'm going to use that model to run that. I mean, it's I I see it kind of coming into view.
Um, and then I think we're starting to to feel like in terms of the scales where there's the camp that thinks we just need one really super smart model to do it all. And we can uh debate that.
But I also feel like
14:05
the the narrative today to me is um multiple agents that are more specialized um and overall coming up with a better way to complete a given task. And that I think you you flashed that paper up Justin, but um similar to that MAI work,
14:20
can we actually start to look at the the comparison between like here's the here's the various tasks the you know walk me through a diagnosis and then base model versus multiple models. And it's an interest especially when you start to put other lenses on it like like you know this paper shows which is
14:37
like you could order every test in the world and you'll get the diagnosis but like is that economically feasible? No.
So, like can we start to look at this more practically? And I think they did a great job with this.
>> Oh, I I think um you know this uh latest work and I I'm aware of three or four
14:54
labs at least around the world that are pursuing the same thing, but I think um you know the MAI team here at Microsoft is the first out on this and internally we refer to this as sequential diagnosis. What's so interesting uh for
15:09
people that haven't read the paper um is you know the model starts uh with the simplest of all prompts. Uh you get the presentation of a patient that's literally a oneliner like you know you know 18-year-old woman presents with a
15:26
cough and sore throat some something as very minimal and simple as that. Um and at that point the model has to be able to ask questions.
Uh has to be smart and
15:42
economic about doing an exam, ordering labs, uh perhaps making referrals to uh other agents or other medical specialists. Um and there's a penalty uh for the costs of
15:58
those things. And so as you start to really delve into this, yes, the AI model itself is interesting, but I think to your point, uh, Matt and, uh, Justin, to my mind, what's even more interesting
16:14
is the the evaluation setup for this thing because it it's really having to be a collaborator. It's having to understand the context of medical care uh work with other agents both human and AI uh in order to achieve
16:33
an economically reasonable outcome uh for for the case. And of course the headline is this thing does four times better four and a half times better than human doctors.
But that's really not the point. the the real point is that
16:48
starting from that just simple prompt uh it's able to proceed with the diagnosis. Now there's still huge questions you know like there's the question of what happens if a totally healthy patient that just needs a cup of tea and needs to go have a good you know couple days
17:04
of rest. What happens in those situations?
And so there's still a lot of work to do, but the evaluation framework, at least right now, appears to be able to accommodate the study of that kind of thing. So I think that's super exciting.
17:19
>> Yeah. Yeah.
And I think this paper's gotten a ton of attention from the media and other places, especially with those headlines. And it's something I think people who don't spend a ton of time focused on AI and healthcare, it's like, oh, but the the models won't be able to ask the questions.
models won't be able to
17:36
follow. Actually, no, no, no, they can.
To your point of starting with a very simple prompt. Interestingly though, what you mentioned on the evaluation framework, um there's been some discussion too of well, wait a second, a hospital actually doesn't necessarily want the lowest cost path.
Is it an
17:53
issue if we're not ordering the expensive tests? And so it just shows the really interesting point you bring up, which is what we choose for these models to evaluate is just so important to make sure we're kind of getting towards the answers we want.
And we
18:09
won't get into the whole, you know, healthcare economics debate for what happens and and you know, the incentives to use AI properly. Um, but the the evaluation is just it's just so important.
>> It's such a good point, Justin. Um, and maybe this is a question for Matt since
18:26
he's connected with the nuance business at Microsoft, but there's always been a question, the story that I've told on a podcast um, not too long ago is um, during COVID uh, I, uh, had to go see a dermatologist because I had a growth on
18:43
my left cheek. So, I go see the dermatologist and this clinic happened to be wealthy enough that they had a human scribe uh in the exam room.
And so, I get treated. you know, they freeze off the uh you know, this growth.
And um
19:01
a few days later, I go online and look at the submitted uh note, clinical note. And the note basically says that um the treatment um was necessary because I was unable to
19:18
wear a COVID mask, which was not really true, but it did allow a code that then uh achieved it was up coding basically. and and and so I was thinking about this
19:33
because you know at that time Matt and I and a bunch of others were really hard at work on uh our uh medical uh you know clinical note-taking applications um and actually it's something we've been working on since 2018. We started a
19:50
project called empower MD um which actually resulted in both dragon copilot and the bridge um and I I tried to wonder would our product or bridge or ambiance or any of these products do similar kinds of
20:06
coding and what is a revenue impact on clinics uh like that one. I mean it's funny that you mentioned that because I think uh revenue cycle management is is a massive space and I think you know where where does the ambient note move into that like you know the proper words
20:23
and you know my field in in medical imaging there are certain things you have to mention right in order to is it a complete is it a limited those kinds of things that come up. Um, I think that there are both really optimistic ways to to look at this and then also
20:38
pessimistic ways and again this goes into kind of just the health economic side, but you could go as far as to say into some of the darker areas like you know literally fraud and other things that that we know occur that cause a lot of waste and expense to the health system on top of the administrative
20:55
costs. And so maybe this is another way to say is the automated approach going to give us a better handle on it or a better way to audit or monitor that uh at a at a you know at a system level as opposed to like these one-offs that are happening or folks
21:11
frankly even having to potentially go back and then add these things to the notes uh post you know post treatment which again doesn't doesn't necessarily reflect reality um in in some cases. >> Yeah.
By the way, this touches on
21:26
something you brought up earlier about, you know, whether there's one super intelligence, you know, that does it all or whether there's lots of specialized AI models. Um, you know, the like one obvious next step for something for this
21:42
sort of ambient listening AI, I think, is uh some connection to quality measures. Um, and the reason I think that that makes sense is there's a business logic there because quality measures are directly tied to improved
21:58
revenues for most healthcare delivery organizations. And and you know, right now that's a problem for these ambient tools because they are pure costs.
They're costs that doctors and nurses
22:13
seem to like and so I think that's why they're doing surprisingly well in the market right now. But eventually you would want these things also to improve um the kind of cost structures at these places and so quality measures seems you
22:29
know there's a lot of other interest in prior authorization and referrals and and so on um as as well as reducing errors like medication errors or other kind of diagnostic errors. Um, but quality measure seems like the one that just
22:45
it it would surprise me if there aren't real advances there. Um, and particularly with the receptiveness of organizations like Medicare and Medicaid to these kinds of ideas.
And so the
23:00
question to my mind is whether that ends up being a system of AI agents specialized in different ways or whether whether you need some more holistic integrated uh AI and I I don't know the answer to that question right now but my
23:16
instinct has always been the more integrated approach even though agents and more specialized models seem to be uh meet the popular approach at the at the moment. Yeah, I won't have Matt comment on this because I know he's but yes, you know, I
23:32
spend my days speaking with health systems across the country and it's nuance and a bridge that come up on the ambient documentation space. Although I'll say interestingly uh ambience I'd say yes >> another company in the space they are
23:48
trying to publish now by the way we let you build more. >> Right.
>> Right. and they actually so so you know is that the macro healthcare outcome you know we want to get to no of course not however they do think that is the kind
24:05
of early proof point for where we're running with these tools you know I agree with you you know Peter I would love if quality were the ultimate metric um it's not really measured you know yet especially with these tools >> actually you know Justin I I don't want
24:21
to let go of that point that you're making because The one thing that's bothered me the most over the last five or eight years in medicine is that both technologists and policy
24:37
makers in in an in a good-hearted attempt to improve matters have tended more often than not to uh put more burden on providers. more cognitive burden, more effort
24:53
burden and more cost burden. uh and you know so like we pushed very hard uh with the government uh on the fire mandates uh the fast healthcare interoperability resources you know data standard mandates but so much of the burden of
25:09
actually making that happen has fallen on the shoulders of providers and this is a repeating pattern and techies like me have just blindly done that over and over again. We just think that doctors
25:24
can just do more and more, nurses can just do more and more. We have to get way smarter about that uh somehow and and spread the burden uh in in smarter ways.
So that's a little bit of a rant, but if there's one thing I've learned uh
25:40
in my role here at Microsoft Research over say let's say the last eight years is is that we've got to stop doing that. >> Peter, we encourage rants on this show.
That's that's of the show. But no, but I mean honestly though I think uh to just to touch on I think the ambient space uh
25:57
is absolutely you know on fire right now in terms of in a good way right that you know I think that clinicians are saying and it actually is almost a comment that to reflect what you just said about the burden on providers if just saving that time >> is a revolution like I mean physicians
26:14
are saying they were going to retire and then they're not like if that's how sort of low the bar kind of is which means our healthcare workforce is suffering, right? And and there's no there's no question about in fact to the point where and we've talked about this on the show multiple times.
You know, the some
26:31
of the patterns of behavior in the market are are sort of like the consumer patterns of behavior. In other words, I'm going to use these models on my on my phone if I have to because I know they can save me time.
They're saving me time in my personal life. I'm going to find ways that they help me.
And then
26:47
then there's this tension right because of the safety and and all the issues that we worry about privacy in the health system and and a lot of folks are finding that just a shortcut directly can I connect the models compliantly
27:02
either with fire or other interop standards directly to the HR. Yeah.
>> And you know there've been several versions of this. I mean this is the this is just an article from uh one of the Stanford efforts chatty HR which is intimately tied to the work that you know sort of we're doing in the multiplayer version of this which is the
27:19
the work with the healthcare agent orchestrator and more complex workflows but if you look at the work that they've done here is essentially hey we know it's useful we're going to put you through training so you at least have a you know an understanding of where things could go wrong and you're aware
27:35
of that but nonetheless we want to make the best available technology directly attached to the the clinical systems of record. So because the other cool thing though that's happening by the way is in in doing this now they're seeing like the most common use they're
27:52
able to say okay I used chatty HR yesterday man my this prompt killed it like I really got a really great fire off note or whatever it is and that shared learning is really accelerating too so I I'm extremely bullish on this I don't know where >> this starts to fit in with more specific
28:09
purpose-built models like how high up into the product land you need to go versus just being able to provide the latest intelligence directly into the context like I don't know the answer. >> Yeah.
Well, first off, the chat ehr work at Stanford has really impressed me um
28:26
because um as you know Matt and within Microsoft research, we've worked for years on uh machine learning uh in order to understand clinical data and largely unstructured clinical data.
28:41
And it's harder than it looks uh to do that. We've had some big wins.
You know, I think you know the first really big win was working with the Providence Health System on their reporting to seven different state cancer registries, uh which used to involve about three
28:57
dozen nurses all day, gathering the clinical notes and then figuring out how to extract the right information for these seven different registry uh input forms. you know it's just a a big mess and across 51 hospitals you know that
29:13
they have in their system you know it the you know it's a huge huge mess and you know this was pre-generative AI um Hoyong Pun uh led an effort that actually created an operational capability that's actually currently in
29:30
operation at Providence um but now uh you know I had this funny episode I was uh working with what we now know as GPT4 but it wasn't released to the public yet and I wanted to test some things out and
29:47
so I was interacting with Hoyong and I told him that I had just met this new posttock you know at Harvard Medical School and you know just wanted to test his abilities to to do certain things. Um, and so we were going back and forth
30:04
uh with some ability to read some oncology research papers and uh answer questions and you I think Hyong was really impressed. In fact, there was a funny moment when they disagreed on a
30:20
certain point in the paper and they had to agree to disagree and at some point Hung realized this is not a person. This is an AI.
Well, I tell that story because after Hoyong finally got his hands on that experimental GT GPT4, he managed to recreate
30:36
that Providence work in one week. >> Yeah.
>> And you know, it's it it's sort of a watershed moment. Um, and it didn't solve all the problems, but I think it's I agree with you, Matt.
The potential is
30:53
so huge. As you know, we're doing a lot um with the Cosmos database at Epic.
Um and that's where we're learning a lot of our lessons about what's hard and what's easy, what works and what doesn't work. I think the dream of
31:09
having a new modality in diagnosis and treatment that is very data driven, you know, where you have a you have the presentation of a patient, some labs, physical exam, other data and and you have an one component
31:28
of your doctoring being can you summarize 50 patients just like this. >> Yeah.
>> Tell me, you know, how they were diagnosed, treated, their outcomes
31:43
and let's have a conversation about that. I think, you know, that can that can be done and tools like Chad Ehr at Stanford are sort of step one uh you know on the 12step path to get there.
>> Absolutely. I well and this and this
32:00
really starts to lead into to some of the comments that you know because you know most people know uh we've done just you know tons of tons of work on NLP and healthcare like as a as a field but MSR you know fun and and MSR in general
32:15
really have led a lot of those techniques and then now sort of saying because we have all this experience we know where some of the biggest problems to tackle are and how do we apply the newest intelligence and where does it you know where does it work where does it fall short I think as you start to to look at connecting the intelligence
32:30
directly in the HR. Um, this comes up a lot and I think in as I listen to your podcast and and sort of how you've revisited some of the predictions in the book around this, I think some themes are starting to emerge.
I don't know if you're I mean you're obviously having a
32:46
lot of these interviews with luminaries across the the spectrum, but I think the folks are kind of making the assumption at this point that uh if you almost are obligated to at least be double-checking with some of these models on various tasks like it's almost like it's not
33:02
just like a nice to have, it's almost mandatory to some extent. And I don't know when that when that will flip entirely, but um I'm curious like have you started >> Matt when when do you think >> what's your what's your bet?
Well, we can all we can all go around. So
33:19
>> I'm the most biased guy here. So like in terms of like I am I'm such a fanboy of some of this stuff, but also realize where the pitfalls are.
But I will I will say like I'm willing to look past some of the faults because I feel as though I I have this mantra in my head that you know we this is the worst it'll
33:35
ever be like like so I I'm just sort of like you know okay with that and so uh I think it's almost now like I mean honestly I think we really at the point where especially just for for literacy uh to get to get a feel of it if you
33:52
haven't I think that that's been my as you know my soap box for a while now. >> Yeah.
You know, I think um I I think
34:07
you there's a difference between uh what is medically best and what that both doctors and patients are ready for. And um and so I think the now I agree with you that I I I I think that
34:25
there would be a benefit for example in the reduction of medical errors um or misdiagnosis if AI were used much more routinely as a second opinion or second set of eyes. um
34:42
you know it's just more more data more intelligence applied to things that would happen right today um you know whether you know doctors and patients would trust that or and would go the extra
34:57
mile I don't know but I I do agree that I think in just a single-digit number of years at least patients would be alarmed if they found out that their doctors weren't getting the assistance of AI
35:13
and and so that that flip I think is much less than 10 years. Yeah, I think I think uh I agree I agree with that, Peter.
And as I and we've talked about this some before, you know,
35:30
five to 10% of open AAI buries are medically related. Some of those doctors, some of those patients, it's it's happening now.
>> Yeah. >> You know, I I'll give a number to I'll say two years, right?
And at least in
35:45
certain cities, right? Take San Francisco, Silicon Valley, you know, where where Matt and I are based.
It's a very techforward population. You know, a lot of early adopters in general.
If you the patients are going to come in with significant turns on AI to their
36:02
doctors, and if the doctors aren't at least capable to have a discussion with those patients about those results and be literate on what's happening, they're going to start to lose trust of their patients. And so, you know, it's it's going I think it's going to be forced
36:18
and then it'll, you know, you know, the future is here. It's just not evenly distributed, right?
I mean, if you think of Stanford medicine, uh I I don't know if Stanford Medicine has its own patient portal or if they use my chart um or some blend, but it's inconceivable, say,
36:35
in that 2-year time frame, Justin, that patients wouldn't be able to have a normal conversation uh with their chart uh through the patient portal, whatever it is. It could be uh an epic supplied my chart thing
36:50
where you have a conversation about things or it could be a Stanford thing or some blend. But you know I think that the patient demand will be there and it's just such a natural thing because you know everyone is motivated to have patients engage with that portal more.
Again it's an economic driver. Um, and
37:08
right now, let's face it, uh, if you've just had a surgical procedure, you look at your my chart, let's say, it's inscrutable to any normal patient. You have no idea what these pathology, you know, results are and and so on.
Um, and so to be able to ask questions, to have
37:26
a conversation, explain it to me like, you know, like a person who has no medical knowledge or training or like I'm six years old or whatever it is. and to have the conversation like that I think um is absolutely
37:42
uh you know it's absolutely inevitable and I cannot imagine Stanford wouldn't be providing its patients with that capability in in that two-year time frame. I I think I mean what what you're sort of saying is like what's the chatty HR equivalent for for patients and
37:59
they're because right because like the the current problem was that doctors were sort of having to go cut and paste into this other place patients are doing that right they're taking the they're cutting and pasting how do we connect that and I I think it's comp to your point it's complex and and right now the
38:14
behavior as Justin's pointing out is patients are are do are using it and you know you you have plenty of anecdotes where they're catching things or they're able to, you know, understand things. We've talked about the information asymmetry that has plagued medicine since the dawn of the field, right?
Just
38:32
how much how how well can you can explain things to your patient um and and be really on the a peer level journey together as opposed to this dictating, you know, uh at them and expecting them just to to follow along. I I um I I I think that I think this is
38:49
going to be inevitable in in the very short term. >> Yeah.
Since you mentioned uh Matt the podcast series uh maybe I can plug that a little bit. Um you know I Carrie Goldberg Zakohani and I wrote a book you know and
39:05
published in March of 2023. Um and you know we made a whole bunch of guesses about what might happen with generative AI in medicine and um but since no one had access to GPD4 at that
39:21
time that we published the book uh you know it was all a work of pure speculation informed speculation but still speculation and so two years on the question is what have we learned when what's really going on for real and the threat was we'd have to write
39:38
another book. Uh, which is the last thing I wanted to do.
Um, and so agreed instead to do a a series of 12 podcasts to talk to people, you know, who've been hands-on out in the field or observing
39:55
uh, the business aspects of this uh, both clinical and business aspects as well as technological aspects. So, it's part of the Microsoft Research podcast series if people are interested.
And Matt, you were of course a great guest on this. >> I like we we had this conversation
40:10
before we came on, but it's it's the our production value is is it's is very much uh not at your level, but no, it was it's a phenomenal it's a phenomenal listen and I and like I was saying before, I think some of the some of the comments are echoed by different you have folks from all these different
40:26
backgrounds, all these different areas of expertise, but the themes are really uh pretty clear. I I think so.
almost like a reinforcement of some things. Uh, a lot of open questions are raised that I think are are valid.
Um, I think the most one of the most provocative things and we can probably wrap with with this
40:42
last topic, but I think I think it was Zach that talked about how does the field of medicine change with this? not just that we're going to be using the models and doing our making our lives easier, but does the subsp specialist,
40:58
you know, kind of start to fade and we go back to having a generalist that has these superpowers with these. I thought that was a very interesting topic because right for the last what 30 plus years even in my I mean I'm I'm a sub sub specialist right I and part of that's the information doubles every 90
41:13
days all the things we know that make it important to know in your field that the the amount of knowledge that you can possibly stuff in your head is limited but the generalist with these tools I think that was a very interesting comment and that would literally shift the culture of medicine even just what
41:29
folks choose to go into and all kinds of things I thought I don't I don't know if you had a >> Yeah, I I thought about this and Zach and I obviously have uh kind of discussed and debated this a little bit.
41:45
You know, for me, I always have the problem. I see you two opposing arguments here.
You know, historically, technology has only increased the number of medical specialties, not decreased
42:00
it. And um uh you know I've said publicly uh there have been two times where technology has eliminated specialties.
You know we don't have phronology anymore um um because of all the technologies that have led to the
42:15
rise of neurology and and so on. And we also don't have barbers doing bloodletings anymore.
And you know there's lots of specialties that uh uh are technology part at least in part technology powered. Um but beyond that
42:34
it's you can't find examples of medical specialties that have disappeared because of advances technology. In fact just the opposite.
So that's one argument. >> Yeah.
The other argument though is um and in I think Sebastian Bubck in the
42:49
podcast you know made this point humans have a hard time coping with a huge amount of knowledge and so that's one of the likely reasons why we have medical
43:04
specialties to begin with. you know, you can become an endocrinologist, but it's hard to become an endocrinologist and a nefologist and a cardiologist and and so on.
Um, it's just too much for a single human being to cope with. But an AI could and at least in other fields um
43:23
we've seen this most uh vividly in agronomics. Um, you can actually see a benefit to an AI model, understanding agriculture in Brazil versus Europe versus the US and
43:39
being able to essentially triangulate uh across those different agricultural practices to become a a superhuman aronomist. And so the counterargument is is it possible that an single AI that can be
43:58
as good as any human being in you know 50 medical specialties it might then enable general practitioners human beings to be general practitioners and and I don't know uh I
44:16
don't know which way it would in fact I I think I'm less equipped than both you, Justin, and you, Matt, in predicting which way it'll it'll go.
44:31
Well, let's my my own version is I also need more time. I don't have a strong I don't have a strong opinion yet, and I know I'm coping out on the answer right now, but uh I guess maybe it's a push to really think through it and get there soon.
But I agree with you. I see good
44:48
arguments both both ways. Does primary care shift solely AI first and care move towards specialties and interventions and doing?
Does it shift the opposite and everything gets done by a superpowered primary care physician? I
45:04
think I think in some cases we may see both right of like I think early on actually we see both. We see some primary care physicians who take on so so so much more and really lean into the tools and cover many more patients much more in
45:20
depth. And I think we see certain routes in primary care where patients are going to self diagnose go to Amazon or something kind of get what used to be a primary care task done almost in a fully automated way.
And so
45:37
I see both of those things happening, you know, almost immediately. Then the question is kind of how does that perturb the system which I I haven't come to to my own conclusions yet.
You know, if I could just say, I think it's so important for us to be thinking about these things now. um and
45:53
to be really grappling and I have tremendous optimism because what I see in the medical world from as a techie is the medical world and especially leading institutions like Stanford and others really confronting these things headon because another question that's sort of
46:09
similar is if we empower every person with this kind of medical super intelligence will we finally realize the true benefits of early diagnosis and better
46:25
health and therefore reduce cost and burden uh on the system or will just the opposite happen where you know medicine is just always going to be inexact enough that your AI is always going to find things that are wrong with you and
46:40
is going to now motivate people to be an even bigger burden on an overstressed healthcare system and again I think it's very hard to know which way it will go but What I think I feel good about is that really really smart people uh in
46:56
the field are thinking really hard about these things right now and and we just have to keep pressing on that. >> Well, I I guess we we pulled this up but didn't go through it before.
So this is a friend uh from Morgan Cheetum and he
47:13
wrote and kind of talked about kind of you know what you mentioned the shifting of value where you know it's been focused on diagnosis here in the middle and maybe it'll shift towards diagnosis versus intervention. Um I I think this may be someday someday
47:28
someday uh where it goes but I actually think value is going to shift far to the right. I don't think we're near I don't think we're close at all to shifting value upstream towards diagnostic and payment models.
Even though in theory
47:44
that's possible from the technology, I think we're really going to shift to the right, which is what we're talking about with changes in coding practices, changes in finding high value patients and procedures, hospitals doing a lot more. So, um, maybe maybe Morgan was just, you know, optimistic from his
48:00
timeline, but I think we're going to shift a lot to the right at least as a first step for for where this goes. >> One thing I like about this chart though is um I I would let me just stick to Microsoft research, but I think every
48:16
technology R&D organization uh is the same. 10 years ago, if a techie uh researcher thought about medicine, they immediately gravitated to diagnosis.
>> Yeah. >> Um and that's good and that's important,
48:34
but that is not healthcare and medicine. And so, one thing I think that we've gotten smarter about Microsoft Research, but we're far from alone.
I think lots and lots of places have gotten smarter, is we're seeing the bigger picture better than we used to.
48:50
And so, you know, when I see Morgan's chart here, that's what I see is that we actually understand that there are things like prevention and intervention that that could also be helped through technology. And that's actually a major
49:06
advance for the tech world. >> It's the start of the journey.
I mean, it really is, right, the diagnostic step. And um oftentimes uh Justin I mean it most of the patients I've seen in my career even come with essentially having that diagnosis and and then it's about
49:22
the decision- making the the judgment um and then you know the knowledge to to to make the right decisions later um over time. So anyway I this more to come here but um but but yeah this has been absolutely phenomenal.
Thank you, Peter,
49:38
for sharing your time with us and your insights. Um, and you know, as you know, we we we try to keep this as a ongoing thread through through multiple discussions and um I think this really added a lot to to the prior the prior
49:53
ones. So, thank you so much for joining us.
>> That was really fun, Matt. Thanks.
Uh, and thanks to you, Justin, for having me.