Waking Up With AI

Katherine Forrest: And you know, the major AI regulatory schemes that are developing right now tend to be risk-based. We see that with the EU AI Act. We see it with Colorado AI law that we've just talked about in a recent episode. And that's just sort of two of them, but it's really all over the place. And the issue is that there are risks that are identified with AI, such as algorithmic bias, the one that you mentioned. And also, a variety of other risks or potential risks, for instance, with high capability models. Sometimes we've called those dual-use models, which is the White House Executive Order phrase or frontier models, which the EU AI Act uses. And we've talked about really all of those on this show at one point or another.

Among the concerns that these models, these very high capabilities models are causing, are captured in that acronym that we also talked about called CBRN, is Chemical, Biological, Radiological or Nuclear Concerns. And there are concerns that if the model is used by malicious actors in certain ways, it could give rise to certain CBRN risks. Not that CBRN is necessarily inherent in any normal non-malicious usage of the model, that it could be turned to a malicious use. So, there are risks identified. There are also risks identified relating to deep fakes, human safety, generalized concerns about a lack of transparency, accuracy, or robustness.

Anna Gressel: And all of these risks bring us to the term of art that's being applied to try to identify and grapple with them. And that's the term mitigation or mitigations. Today we want to explore AI mitigations, how they're being talked about, why they're being talked about, and how risk mitigation in the AI space compares to other contexts our listeners may be familiar with, like cars or airplanes.

Katherine Forrest: And you know, what I find really interesting about this topic, is that it starts really as a practical, technical issue. You know, what are these AI models able to do? What are they being put to do because of some of their capabilities? But it also becomes a legal issue. And that's sort of where we come in. It's an issue that regulators are deeply concerned with, that companies utilizing tools can be concerned with.

So, for those who are involved out there in our audience in AI legal issues, understanding really the technical side of things can really be a key aspect in giving the best advice. So, the starting place for a discussion of mitigations is really, I think, the initial question of mitigating what and what are the issues or concerns that someone wants to or some group wants to mitigate or address? And there's not always agreement on whether something is a risk to be mitigated or that it should be mitigated, let alone how to mitigate it.

Anna Gressel: That's exactly right, Katherine, I mean the mitigations that we saw with narrow or traditional AI were often very data oriented. They were about choosing the right data sets to make sure that the model functioned with the right kind of information. It was fit for purpose, or it was about constraining outcomes, making sure that the model didn't drift in a way that was unexpected. Generative AI changed that somewhat, and I think it's worth noting that generative AI, I think we've said this before, is a fundamentally flexible technology. It can do different things in different people's hands.

And so, some of what people are concerned about with generative AI is just the misuse, intentional or unintentional, of the model. And so, some mitigations in the GenAI space tend to focus on making sure that those outputs that people generate with their own prompts don't stray into areas where people don't want them to go. That could be at the developer level or the deployer level. And that might look like things like prompt filtering, making sure that people don't ask for illegal information, for example, or information that might violate company policies. And it might look like constraints on the outputs so they don't produce that information even if prompted.

Katherine Forrest: Right, that's sometimes why you're now seeing in response to certain prompts. I can't answer that or something like that. Our listeners may be curious to explore a recently released AI Risk Repository that MIT has just put out. It’s a database, you can access it online, and it actually purports to capture what they characterize as 700 or so “risks.” It classifies them into categories such as misinformation or weaponization. I actually was just playing around with it and entered chemical weapons and it led me to a weaponization article and you know it's really almost like a word search capability. It's more sophisticated than that, but it actually is quite useful if you want to read into this area and there's also something by Google's DeepMind, it’s called “Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data.” So those are two resources for our listeners.

Anna Gressel: So, we know that a picture of what the risks from AI are increasingly crystallizing, and a separate question and exercise consists in asking what to do about those risks. And that's where mitigations come in.

Katherine Forrest: Right, and humans are really constantly identifying risks and trying to solve and mitigate them. And we have probably since the invention of the wheel and tried to figure out how not to get run over by it. But what we're doing here is we're trying to figure out how mitigation may actually attempt to reduce risk, but it may not entirely resolve risk. It may mean trying to address a risk to some reasonable and practical extent. For instance, seatbelts in cars. It doesn't mean that you don't get injured in a car accident, but it may mean that you get injured less. Same thing with airplanes and turbulence or things of that nature. These are all mitigations, but it doesn't mean a complete resolution.

Anna Gressel: That's exactly right. And Katherine, one point that I sometimes like to make with AI is because it can do certain things. It can do so many things, really, it’s like we have an airplane, a car, and a boat, maybe a jet ski all in one, and we don't quite know which technology is going to show up for us today. But with AI, it's worth noting that the issues are a little bit different because they can come from within. So, it's sometimes about how the model is trained and its internal capabilities, but sometimes it's about having that problematic user or the dangerous behavior by that user. So, if there's a human, whether that's someone internal to a company or someone just outside who's using a publicly available tool, they could actually potentially try to cause harm on purpose.

Katherine Forrest: Right, so you've got the human who wants the tool to do something that may or may not be considered by others to be appropriate. But then there's also, Anna, the internal ways in which a model can itself, even without being prompted to do something that is what others don't want the model to do can produce a result that needs to be mitigated. For instance, algorithmic bias. You can have a user who puts in a query or utilizes a particular tool and ends up with a biased result, not wanting a biased result. Or another example would be a hallucination, that a user actually wants a factually based answer but comes up with a hallucination. So, you can have actually human users who are not intending to misuse a tool but come up with a result that's not ideal and that the developers or others along the chain, and we'll talk about that a little bit, that they actually feel like it still needs to be mitigated.

Anna Gressel: Let's assume we've identified an issue and we decide we want it mitigated. One of the other challenges in this area is there's so many different actors at different points along what we've called the AI value chain previously. There are numerous developers, there's a huge amount of data, some of it's curated by different companies and licensed, and one tool may in fact be the product of several tools or several models even being joined together for a particular purpose. It may have been fine-tuned, it may be in a particular user interface that can cause issues, and so separating out where the problem is and the right method for mitigating the issue is very complicated.

Katherine Forrest: Right, and all of these participants in stages of model development, training and use, they all present moments of investigation for whether and how to mitigate an issue.

Anna Gressel: There are also not only these technical issues, but legal issues of responsibility. Who is responsible to mitigate? Should it be the developer, the one who's fine-tuning the model, the user?

Katherine Forrest: There's really no one answer to that. It's going to depend, and it depends on so many different things. And let's really be clear about that, when one user identifies as something to be mitigated, another user may consider that to actually be a useful functionality of the tool. So not all “problems” or “issues” are seen by everyone in the same way, even by the regulators.

Anna Gressel: Yeah, that's definitely right. And we can also think a little bit about one of the issues we've talked about previously on the podcast, which is model alignment. That means aligning a model to what humans want or human preferences. So sometimes mitigations are directed at general concepts of what alignment should look like or be.

Katherine Forrest: You know, Anna, this is an incredibly fluid area. There are sometimes trade-offs between mitigations and useful capabilities.

Anna Gressel: Yeah, there's no one right answer here.

Katherine Forrest: Right, mitigations are not just a monolithic set of answers as to what malicious actors might do. Mitigations can also be chosen responses to what only some users see as issues.

Anna Gressel: So, what we're saying is that mitigation is a specialized term in AI to address perceived issues or risks. But identifying those issues and risks is an ongoing conversation.

Katherine Forrest: Yep, ongoing, just like basically everything in AI right now. And we're just at the beginning of all this, which is fortunate because I have no intention of ending this podcast anytime soon. So, we need to actually have enough material to go on and on and on. And I'm very optimistic about that, Anna.

Anna Gressel: Yep, the field is keeping us very busy on our podcast. Well, we'll undoubtedly return to this topic in a bit. But right now, it's just a bit of a roadmap to continue unpacking this really interesting area.

Katherine Forrest: Alright, and I'm Katherine Forrest.

Anna Gressel: I'm Anna Gressel. Make sure to like and share the podcast if you've been enjoying it.