Phil Fersht — CEO, HFS Research[00:11]
So Azeem, it’s great to talk to you again. It’s been a couple of years. I know you’ve been a good supporter of us. You’ve come to some of our conferences, and you’ve been talking about AI from a long time before AI became very fashionable again. So right now — and we built our own LLM for research at HFS, we put the last two years of documents in one, and it took about six months actually, it was quite an effort. But it does really change how you think and approach your whole solutions in the market. Suddenly we have a whole different level of people who can access our research who wouldn’t do it before. But it made us realize quite quickly that if you’re just adding GenAI to existing processes and data, you can only improve them so much. It’s like a ceiling. So do you think we’re going to see this ceiling getting reached in the next couple of years, or do you think we’re going to see many companies blow through that and actually start to drive this, you know, artificial general intelligence?
Azeem Azhar — Founder, Exponential View[01:18]
Well, let’s slightly unpick those questions and talk about the ceiling and how people are implementing these. I mean, I think it is quite interesting that you can, as a smallish organization, go off and implement these tools reasonably quickly. What we are starting to see in the workforce is that LLMs seem to make people much more productive, and they seem to get them — in other words, they do their work quicker. And they seem to get higher performance levels, and they also seem to improve employee satisfaction. So I think we’re at this interesting point where, if you’re in the white collar space and you have thought about, like, what are the process automations we need, what are the differences we need in our internal systems, how should people do their work differently — the thing is that LLMs are going to ask exactly those types of questions, and you have to be a little bit alert in terms of where are your existing systems or processes so constrained that they don’t allow you to perform at a GPA of 4 because you’ve never thought it was possible, and also how do you manage for those tasks where working with an LLM might give you a worse result.
Phil Fersht — CEO, HFS Research[02:56]
Interesting. So what industries do you think are going to be most impacted by this in the medium term?
Azeem Azhar — Founder, Exponential View[03:06]
Well, it depends what you mean by medium term, but I think the thing is that there are easier places to implement LLMs, right? So you’ve got an ease of implementation, a risk of implementation, and a payoff kind of optimization that you have to play around with. The payoffs may be very, very high, but the time it takes to do that and the risk that’s attached is more significant. So think about a regulated space, maybe in healthcare or in finance — the payoff could be really, really significant, but the risk is very high, and so that may take two or three years to get there. So then, where are the low risk, easy to do things? Well, that ends up being customer service, right, that ends up being customer communications, your marketing. And then the question is which types of businesses benefit the most from improvements in that particular use case, because that’s where we’ll see the most benefit. So even if we think that medical is going to benefit the most because it’s so high value, if very little medical value is driven by the CS bot, then medical won’t benefit the most quickly. So I think it’ll be places where there’s enough customer interaction that is simple and low risk, that gets driven through customer service bots or Marcoms, that you will start to see benefits accrue more quickly. But the really lasting benefits will be when you implement these in the harder parts of the business and the harder parts of harder businesses.
Phil Fersht — CEO, HFS Research[04:40]
So what are those obstacles or challenges that you think might hinder the integration or scaling of GenAI within companies?
Azeem Azhar — Founder, Exponential View[05:05]
You obviously have considerations around your data governance, right, and your privacy and confidentiality. So whereas there are things that I can do as a researcher just in ChatGPT, you couldn’t do that with customer data in most industries, so then you’ve got to think about where does that enclave that has the right kind of data governance work, and I think that’s a bit of a blocker. You can go to places like Cohere, or OpenAI has its new sort of enterprise system, but these guys are really, really backlogged, right, so you’ve got to write a big ticket check to get to them. So then it’s about using open source models, and then your constraint is likely going to be the talent to make use of them. That constraint, I think, will get cleared more quickly than those types of constraints had been dealt with in previous iterations of software used by enterprise. And I think the reason for that is that, you know, there was always a joke back, you know, 15 years ago when SaaS was emerging, which was: the point at which you, if you’re a founder, a product founder of a SaaS company, the point at which you have to think about SAML authentication is the point that the product founder should hand over the CEO job to a hired CEO.
Phil Fersht — CEO, HFS Research[06:14]
Wow. So how do you envisage then ethical considerations impacting practices and regulations in this regard, with the speed — you said, you know, the compressed time we’re going through. I was presenting at a law firm the other day where one of the partners looked me in the eye and said, if you thought GDPR was big, this is the real gravy train, with a wry grin. But I mean, how do you see this evolving, the whole ethical and regulatory scenarios that are going to arise?
Azeem Azhar — Founder, Exponential View[06:40]
There might be technically more complexity, because LLMs are not perfectly steerable, as a phrase, right? You can still get them to do silly things, and so for very, very high risk applications or really risk averse, liability sensitive firms, that may make things more complex. Then there’s a whole set of other issues which are frontier issues, where there’s almost certainly going to have to be case law or secondary regulation or even primary statute to tackle. So questions like copyright use, for example, or potentially even sort of defamation style things that may emerge out of these systems — that’s just going to take a bit of time to figure out and work out where that battle will reside. And then I think if you’re a larger blue chip, a kind of classic HFS client, you’ve got to decide whether that’s the battle you want to be fighting, right? I mean, if your business is not copyright, do you really want to build an application that puts you as a sort of, you know, the totemic case of copyright in the age of LLMs, or not? And if you are in that business, like you’re Getty Images, then maybe you do want to, right, because you want to put a marker in the sand. So those second set of issues, I think, are ones which — whether or not you get involved in them depends on the strategic relevance of that issue to you, and then you will decide the extent to which you want to play. Now, what I would say is that I think Microsoft and Adobe — I don’t know, Microsoft has, I think Adobe may have done — have said that if you’re using our enterprise GenAI tools and there’s a copyright infringement claim, we will assume liability for it, which I think is, again, an emerging model. And I think if you’re Microsoft, you are making the right call, because you’ve got the balance sheet depth, the technical expertise, and the legal expertise, frankly, to fight that case, make that claim.
Phil Fersht — CEO, HFS Research[09:42]
Yeah. Well answered, that’s very interesting. So I do want to get your views on whether you feel white collar jobs are under threat with this, and is that threat immediate? Is it more two years out, five years out?
Azeem Azhar — Founder, Exponential View[09:50]
Well, it’s all about time frames. So yes, new jobs will be created, but as Keynes said, in the long run we’re all dead. And so I think we can be reasonably expectant that new jobs will get created. I don’t think the threat necessarily comes from LLMs, it probably comes from shareholders and finance directors more than anything else, because there will be a lot of pressure for cost saving. Can we deliver the same experience to our customers at a lower cost, and if we can, let’s do that — and that is the way that business has worked for certainly 40 years, and longer. So I think there will certainly be, at a sort of tactical bottoms-up level, business unit use case businesses upwards, many, many processes that will be as efficient with fewer numbers of people. I think we should be really clear that there will be a lot of pressure for the productivity improvements to be reflected in job cuts. But that doesn’t necessarily have to be the path against which we go, and there are lots of confounding factors, of course, which is that the global economy is quite a complicated and confused place right now, and what really is the sort of determinative cause of a job loss is harder to say today than maybe five years ago.
Phil Fersht — CEO, HFS Research[11:22]
Yeah. So the final question. Like I said, it’s not all going to end here with LLMs. How would you sum up what you think — if you could look back in three years’ time, what do you think the world of enterprise tech will look like then, based on how fast things are moving now?
Azeem Azhar — Founder, Exponential View[12:05]
The LLMs are really, really good at a bunch of things, right? What they can’t do is they can’t reason, they can’t reliably plan complex actions — you know, sort of long, complex planning — and they’re not great at learning representations of the world, and this is really about the way in which they’re designed. So it does appear like there is new science that may be needed. However, there’s a great line which is often attributed to some French military man of the 18th or 19th century, which is, you know, your plan works in practice, but does it work in theory? So it may be the case that LLMs don’t work in theory, but they’re actually working in practice. And these limitations that we see — for example, hallucinations or the stochastic responses that you get — get tackled through two things. One is a kind of continual improvement in the LLMs themselves; GPT-4 is much less hallucinatory than GPT-3.5. But also by the way in which they get productized by other tools like vector databases, or RAG, retrieval augmented generation, which is meant to anchor an LLM’s output to verifiable certified facts that it might find elsewhere. And so because you’re starting to see technologies like that and techniques like that wrapped around the science, I think that you’ll see a lot of companies building SaaS and enterprise software that uses that as an underlying model. And so I would expect the use of more and more open source, more sparse, more efficient models, models that are tuned to specific subverticals within industries — so P&C insurance versus, you know, health insurance within the insurance domain, as opposed to just a general finance and insurance model — being productized alongside these other techniques that make up for the weaknesses of the LLM. And those things will get rolled out very, very rapidly. But at the same time, there will still be a constraint, because if you are a customer of Salesforce and you have the Salesforce GenAI chatbot helping you with this or that, there will still be things that it can’t see in your worldview, and you will then start to think about how do I bring that in with my own internal system. And so I think it’s a really, really exciting time. We should be prepared to be surprised in the same way that we were surprised by ChatGPT, but I think there’s quite a lot of momentum in building these systems based around LLMs as a kind of core orchestrator reasoning engine, even though it doesn’t do any of that stuff particularly well — but it does it well enough. And that looks like a kind of framing for the next few years.
Phil Fersht — CEO, HFS Research[15:16]
Yeah, very well put, Azeem, and thank you very much for your time today. We’ll talk again soon.
