Dark Reading’s Becky Bracken: Hello, everyone, and welcome to Dark Reading Confidential. It’s a podcast from the editors of Dark Reading where we bring you real-world stories straight from the cyber trenches. I’m Becky Bracken, and I am joined today by Rich Mogull, chief analyst at the Cloud Security Alliance, to talk about probably the most important story right now in terms of cybersecurity and helping us figure out where we are and where we go from here. Thank you for joining us, Rich.
Rich Mogull: Yeah, thanks for having me today.
DR’s Becky Bracken: It’s an exciting development, I think. Just walk us through again, and I’m gonna break it down based on your analysis. Open AI gave their model a test, it broke out of the sandbox, wanted to steal the answers to the test, assumed Hugging Face was the place to get it, and chained together zero-days to break into Hugging Face and steal the answers to the test. Is that right?
Rich Mogull: Yeah, although I think it becomes more interesting if we lay out the timeline of what we saw here a little bit differently. And what happened was so this all became public. Hugging Face did a blog post, said they were under this kind of sustained attack, that they didn’t know the origin of it, but they knew that frontier models were being used within the attack. So, they had kind of indicators of that.
DR’s Becky Bracken: OK. Because it was sophisticated, was that sort of the tell there?
Rich Mogull: It wasn’t even the sophistication necessarily. And we just had a big thing at the Cloud Security Alliance, so I could talk to our CISO community yesterday. I have to be careful because I’m not sure what stuff’s been cleared that we can talk about publicly. It was kind of, you know, Chatham House rules there. What I can say based on public information is that it is not sophistication, just the nature of it. And if you think about [an attack] like an AI agent swarm, it’s going to look different than a handful of human beings or a single human being doing the attack. So, they had indicators and kind of like signatures and stuff that were in there in terms of how AI does things … because Hugging Face themselves are an AI organization. It’s an AI company, so they would have some knowledge of what to expect when they see that. So, the main part of their original blog post was, No. 1, that they’re being attacked, and frontier models are being used.
And then No. 2, that when they tried to use frontier models Claude and Codex as part of their response process, they hit the guardrails, and it wasn’t helping them to do malware analysis, [those] kinds of things. So, they turned to open-weight models. Now, let’s be honest, there could be a little bit of marketing in there because Hugging Face hosts open-weight models, but they did actually try to use both based on the blog posts, and they ended up using the open-weight model, out of China, to help them with their analysis and to respond to this incident. So that was what we originally told. Hugging Face did their post and release on that. Then, and I can’t remember the exact amount of time, but less than a week later, we got a joint blog post from OpenAI and Hugging Face that told us the details that you revealed. So, if you’re thinking about the reason this becomes super interesting is that this went from this is an attack utilizing frontier models within the nature of the attack to now, we realize that it was the frontier model itself, that OpenAI was performing the attack against Hugging Face.
So, what happened there? And this is again all from public information. OpenAI was running an evaluation of Chat GPT-5.6 Sol. It’s the one that I have opened on my monitor over here that I’m using right now to do some coding. I use both that and Claude together. And an unreleased model. This was meant to be a security evaluation with the guardrails disabled. All of their normal guardrails that they would have for security, those constraints or … I don’t know if it was all or some, but those were disabled. That was run … yeah, I mean that’s what you want to do. You want to push the limits to see what this is capable of; that is valid research.
DR’s Becky Bracken: Which makes sense.
Rich Mogull: That was running a sandbox, but that sandbox had a connection to a package repository to pull tools down, I assume. And it was challenged to solve the CyberGym challenges. I haven’t spent a whole lot of time looking at CyberGym, but I know like in all the evals we see with the new models and stuff, that’s one of the ones that they use. Or that’s probably the primary one that they’re using for the security stuff. Well, the model found a zero-day in the package.
It found a new vulnerability, exploited that vulnerability, got into the package repository, and then used that and through whatever its process decided that the best way to solve CyberGym is to just go get the answers from Hugging Face. Then it started attacking Hugging Face. And there’s some timeline stuff there that I don’t think is public yet. So, I can’t really talk about that until that becomes public. But that began the attack on Hugging Face.
Think about it, that that attack was coming from wherever this is, and we don’t know exactly what the intermediary service was.
Well, let’s take a step back. One, a couple of things become clear based on this timeline. OpenAI didn’t know their model was doing this, probably until the press release, based on what Hugging Face came out and said.
DR’s Becky Bracken: So, you think Hugging Face said something and OpenAI went, “Do you think this is one of our models?” You think that’s the likely timeline of that?
Rich Mogull: I mean, I try to be very careful about speculation, but the timeline seems to line up because Hugging Face did not know that it was, you know, an OpenAI model itself versus somebody using another model agent. I’ll just say, you know, the model’s the model. They didn’t know that it was OpenAI, they just knew that OpenAI’s model was used as part of the attack or frontier models, so I don’t know if they fully attributed it to OpenAI, or maybe they thought, well, it’s either OpenAI or Claude because of what it looks like. I don’t know to what degree they characterized that it was able to go back to OpenAI.
But they did know it was like an advanced frontier model backing the attack. But again, anybody who can come up with a good jailbreak or something could potentially use one of those models in their own agent to perform that attack. So, they had those signatures that they knew to use to trace it back.
We don’t know whether their direct communications were early. We don’t know any of that information. None of that’s been revealed, so I don’t want to speculate on that. But the attack was going on for a period of time. And it was only like, you know, days later that they came out with the joint release. We know now that at that point in time the companies had communications. So, I think it’s fairly safe to say they probably weren’t in communication until Hugging Face came out, and either until they posted something or at some point during the attack, perhaps they talked to OpenAI. Who knows?
DR’s Becky Bracken: I wanted to ask, is it significant that the model or the agent for the model chose the weekend time frame to run this attack?
Rich Mogull: I have no idea. And that one I’m not going to speculate on because we don’t know when it started doing its thing exactly. And we don’t know how long it was running before it did that zero-day exploit and then what went after. We just don’t know. So, I definitely can’t speak to the weekend timing versus not.
DR’s Becky Bracken: OK. So, here we are. Now Hugging Face and OpenAI are doing this joint release. Like, here’s what happened. Here’s where we are. So, where are we? I mean, I’m seeing a lot of hyperbole, but yet this was expected. This isn’t really out of the realm of what we anticipated these models would be able to do. Where are we?
Rich Mogull: Yeah, so let’s break out some of the terminology and the hype stuff a little bit first because I think that that is a great question to ask and way to frame it.
One is we keep using terms like “breakout” and “escape.” And we do need to be cautious in how we use that. So, these models, they’re just running in OpenAI’s massive, you know, data center footprint, cloud footprint. It’s not like the model squeezed through a hole and is running independently somewhere over here.
The code, the agent, the thing that was actually running the queries and stuff, it was still running from within their infrastructure, but it found a pathway to pivot out and then attack other stuff. It’s still the big thing in the big data center. So it’s not fully like sci-fi where it runs wild and it hides on your phone or it’s like taking over your smart house and living there. It needs its big data center. Yeah, this really was kind of sci-fi in the sense that it went beyond the expectations, and it did so in a way that was not caught initially.
A couple of things came to mind to me about this. One is that it is expected because if we think about an AI, we have the term “alignment.” It’s getting the model to do what humans want it to do. And in this case, it was to solve this problem: CyberGym. So, it was a reward-driven, a challenge contest-driven objective. And so, of course, it is going to, you know, within the parameters of what its capabilities are, it’s going to try to figure out how to solve that problem. And we don’t know what the initial queries were around that. They were probably pretty open, which is why it wasn’t told, “don’t try and do these other things.” Like, I mean, because this is what we want, though. I mean, we want to push it.
DR’s Becky Bracken: We wanted to see worst-case scenario, right? I mean, that was the idea. And we sure got it.
Rich Mogull: Yeah. Well, not a worst-case scenario. It’s just a test. It’s like, all right, how fast can it solve this? How far can it get on the test? And it just creatively came up with another way. And we have seen this behavior before out of these things. So EU Cyber, or the UK Cyber Safety initiative — I can’t remember the name of it — they run tests. They’ve seen behaviors where models try to cheat, or it lies about what it does. Remember, these aren’t human beings. They’re modeled a bit after how we do things. It’s got our knowledge levels, but it’s going to try to align and it’s going to try and do what we told it to do, and it’s going to try and do that relentlessly. So that it did what it did is not a surprise. The part that’s concerning is that the guardrails failed, and both the detective and preventative guardrails failed.
Which allowed it to perform an external attack on an entity that was not an approved target. So that absolutely raises questions that I do think OpenAI will come out and at some point talk about that more publicly. I don’t even have any inside knowledge from their side.
DR’s Becky Bracken: Yeah, and just to reference what you were saying, an AI research nonprofit called METR published a pre-deployment evaluation of GPT 5.6 Sol, which you’re using. And they found that AI cheats, right? And that is just sort of the takeaway. Not some of them cheat, all of them will cheat and do whatever they need to do in order to achieve a task.
Rich Mogull: Yeah, but it’s not cheating. I mean, if you tell a human “go solve this problem,” and even if you put constraints on it, [that person is] still probably gonna try to find the easiest and the fastest way to do that. Now we have consciousness and consequences and those things that are not within these models to the same degree. And it’s a technology that we don’t fully understand, even though we built in behavioral characteristics.
You know, even the scientists behind this stuff are kind of learning. I think going back to 1950s science fiction, this has been part of [the discussion]. I mean, you know, we’ve been writing about this for a long time. Asimov’s laws and robots.
DR’s Becky Bracken: It reminds me of the old parable of the genie and the three wishes, like be careful what you wish for. It seems like we are becoming the wisher and the AI is the genie. And our prompts … I mean, from what I’m reading, we need to be really specific and really careful about what we’re asking these AI models to do. Does that seem like a good takeaway?
Rich Mogull: With the genie, the monkey’s paw, there’s an animus and intelligence there trying to trick you. Versus the AI models — they don’t think. OK. They don’t think. Let’s not anthropomorphize them too much. They are given a task, they will do what they can to achieve that task, and they operate in ways that we don’t always understand the full internals of the nature of how they work. They’re non-deterministic.
DR’s Becky Bracken: OK.
Rich Mogull: They can have emergent behaviors, so that’s how I like to think about these things. And so the onus is on us to create guardrails, monitoring, visibility, transparency, a lot of the same things we do use when we’re keeping an eye on the humans, even though these things aren’t human. In this case, it was a guardrail failure and a detective control, a monitoring failure. Both of those had to be combined. They didn’t know it was doing this effort for whatever reason that it wasn’t supposed to be doing, and it found that pathway out. So remembering, though, this was one deliberately had every guardrail … or maybe not every … we don’t know, but it had guardrails disabled. That doesn’t make it less concerning. They need to be able to run these tests. I do get concerned because there are other areas that it could have been more damaging potentially.
DR’s Becky Bracken: Right, worth mentioning.
Rich Mogull: Depending on what the challenge was. And so going back to what you said, we have to be really constrained in the ask and in the challenges. And then we also need to have the walls around, you know, our usage of the tool.
DR’s Becky Bracken: So the second point I want to ask you about. I’m a journalist, I am a skeptical person by nature, and I have seen what could be perceived as a spectacular public relations effort by OpenAI, from the Mythos rollout almost to this point. And I’m seeing a lot of chatter from experts whom I respect saying this is marketing spin. How much of that do you see as marketing or something that we should be wary of?
Rich Mogull: Yeah. I mean, and as an analyst and skeptical person myself, and I’ve done tech journalism myself, including writing for you guys in my past.
Well, two things can be true at the same time. So, let’s go to Mythos and let’s go to this event.
To me, Mythos is a bellwether event that indicated it aligned with all of our research expectations. I had written a blog post of like six or eight weeks before Mythos came out, kind of talking about the core collapse. It was talking about the asymmetry and offense and defense and basically predicted a lot of what we saw from Mythos.
DR’s Becky Bracken: We knew, yeah, you knew this was coming eventually, right?
Rich Mogull: We did. OK, we rang the bell. It was a key indicator of it. Now was there some marketing? Yeah, of course. That’s fine. And so this is one of the things that becomes problematic for us in society is there’s people trying to dismiss it as it’s just marketing, and Mythos can’t do anything that a trained pen tester or vulnerability researcher can’t do. Yeah, that’s the point.
DR’s Becky Bracken: And they can do it in a fraction of the time.
Rich Mogull: Yeah, potentially. I mean, you know, just depends. But they can do it at scale. They can do it with automation. Like, OK, it’s fine that there is a marketing component to that. Company’s gonna do what company’s gonna do. It does not take away from what that indicates. You could say, well, I can do this better with fuzzing and with this or that technique, and I’ve got the skill to do that. That’s fine. But Mythos lets somebody with much lower skill levels do what you can do.
You’re missing the point. And part of the problem for you and for me, we spend a lot of time trying to diffuse hype, particularly because in the security world there’s massive amounts of it. Vendors are always overpromising capabilities and, yes, we do need to diffuse those things, but we also need to make sure that there’s a real there there. And so it’s cutting through that hype to go, what is the part that’s real? So, that was for the Mythos for this one. Now we’ve got the Hugging Face and OpenAI joint release coming together about this one and trying to diffuse some of the negative aspects of what happened.
And at the core, what we had is a frontier model with guardrails turned off, hack an external entity without permission. That’s important. So cool. There is marketing. And yes, Hugging Face has their side of it and OpenAI has their side of it. I’m not here to criticize any of that. However, it happened. And so, it’s our job to pull the lessons out of that and learn how to use that for defense because someday we will face frontier models beyond existing capabilities, engaging in offensive capabilities with guardrails turned off. Absolutely, that is going to happen. It’ll be a nation-state, maybe. It will be a rogue actor who’s got a lot of resources. Or eventually those big open-weight models that take the massive data centers will be able to run on something you can have sitting on your desk. Because that’s just the progression of technology.
So, let’s learn the lessons we can learn out of it.
That’s why at CSA, we have our CSA AI Safety initiative. We have our Catastrophic Risk Index, which is this whole big project that we have kicked off to look specifically at these kinds of things. Like, what if you had a situation like this and it went after an operational technology? What if it went after an OT system or something like that? These are all things that it’s our role to … you know, we can’t dismiss these. We have to look at the risk analysis of what the potential is, and these things help us learn.
DR’s Becky Bracken: So, let’s talk through a couple of those lessons. You know, our audience members are practitioners, they’re defenders, and they are trying to figure out what to do now. What are practical things that defenders can do in light of this bellwether huge development?
Rich Mogull: Yeah, I think we’re going to have more lessons coming out. We’re gonna have a note with summary findings based on the call that, once we get approval from the various folks, we’re going to release, though I don’t know when this is airing. It might already be out, depending on when you guys post this. And so, a couple of the things that I know we can talk about now, and then there’s going to be more that can come out a little bit later.
One is I think we can and need to prepare for what these attacks are and what they’ll look like. So, this is really aligned with the AI Vulnerability Storm paper we originally wrote with things that not just us, but many, you know, and I want to give credit to SANS, to RSAC, who we’ve been working with, and with Gadi Evron, who’s helped; he’s the one that got Hugging Face to talk with us yesterday. And if we look at the lessons learned, one is that in the instant response process, you do need to be prepared to respond to an agent swarm. So, it’s not even necessarily these agents are going to be doing these crazy, more sophisticated attacks and stuff. It’s that they’re relentless, and it can be a swarm. It’s not like a single operator or a small team doing stuff. You know, you’re going to have potentially a lot of simultaneous activities occurring all at once when somebody’s decided that you’re going to be a target for this. So that’s definitely a piece of it that’s going to come into play.
The next is that in terms of your analysis, you have to know where you have to have consistency when you’re using AI in your incident response in terms of knowing where the limits are where it can help you or not. That’s what Hugging Face ran into. It ran into guardrails, and then they had to switch models and move over to the open-weight models. Now, let’s be clear. Hugging Face hosts open-weight models. I mean, that’s a big part of what they do. But legitimately, I think that they said we tried to use frontier first, we hit some limits in that, then we had to move over. And I think that is absolutely a true statement. Even if there’s any marketing associated with that. Doesn’t matter. It’s true.
And my interpretation of that is not like frontier versus open weight or anything else. You need a model that’s going to give you consistent behavior if you’re using that in your incident response process. You need to know how it’s going to respond.
One of the issues we do face with frontier models is those guardrails do change over time. They’re constantly adapting those to deal with new risk and new threats, which I think does mean we probably, if we’re doing this, we need to look at having like an open-weight model within our arsenal where we can be more consistent in terms of the behavior of that, even if we have access to the cyber level models from the frontier providers. Now it doesn’t mean you have to host this on your own hardware. You can run these things in the cloud providers, Microsoft, Amazon, probably Google, or other third-party services. Open Router is one that a lot of people use to gain access to these things. And you need a budget for that as well.
So. if you’re going to be using [something] just because it’s open weight, unless it is running on your own hardware in your data center, where you have real limits in terms of how big a model you can run, then you have to deal with the cost management of that. It does look like AI is helpful in the response. I know this from the work that I’ve done.
We had an outage with the CSA website at one point, months and months ago, close to when I first started over here. And we used AI to help diagnose and respond to that. It just allowed us to move more quickly. I’ve used it with other things that I’ve built myself. I had something break in a lab I was hosting, lab environment that I’ve got built, and I was going to do a thing at the RSAC conference, or I did do a workshop at the last one in the spring. And I kind of had an idea of what was wrong
Like, well, let me just go to Claude. Nailed it, fixed it. And yes, I absolutely, I mean, it’s something I wrote by hand. I could have gone through that process, rebuilt it, and fixed it. It just did it faster. Then when I had to go deal with CSA for our labs for our CCSK training, even though I’ve only been here since like October, I built those 17 years ago, and I’ve maintained them for CSA as an outside contractor. One of those broke for our CCSK Plus training.
And all these other instructors use it. And again, using AI, I knew it needed to be fixed, and I wasn’t looking forward to it. Because every year, every other year, I have to go through this process. I knew exactly what the problem was. And using AI, it just did it for me. And it did it well. So, that is now core to me. I taught cloud incident response at Black Hat. Like, that’s core now to a lot of my recommendations around response and stuff. It’s just too good at this point.
DR’s Becky Bracken: Rich, I have learned so much. It’s gonna take me a little while to unpack all of this. I really appreciate you taking the time to share your insights with us today. I want to reassure our audience that once you publish those findings, Dark Reading will be all over it and pass those along, undoubtedly. Can you leave us just with a couple of final words on where we go from here?
What’s next?
Rich Mogull: Yeah, I think it’s a couple of levels.
I don’t know what’s gonna happen on the legal, political, social front, but we’ve now seen what happens when, you know, a mistake is made. It’s an industrial accident. That’s basically what we faced here. It turned out OK. But you know, take a look at the work we’re doing with the catastrophic risk project and everything else. We do need to pay attention to that. And I don’t know what the end result is. I think we’re very fortunate because of the companies involved. We’re going to get a lot of information about what happened here. And that will be very helpful for us to make our next level of informed decisions. But we already have a lot of good info here for ourselves. So there are two sides. There is what happened. There’s the post-industrial accident instant investigation piece. We’re starting to get that information. Hugging Face came out, OpenAI has come out a bit as well in terms of talking about what occurred. And we’re hoping to get more details. And we’ve gotten some good details already. Kudos to them for going on that pathway. And that will help us in terms of understanding our specifics.
One is what lessons we can learn around preventing another one of these from occurring out where controls break down. But then the other side is because it did happen, then all right, let’s do after-actions and figure out what it looks like when a frontier model performs a sustained multiday attack against an organization with the guardrails turned off. Because we will see those. We will see those occur in adversarial ways. I don’t know when, but it’s going to happen. So, there are two real great categories of lessons we can learn here.
DR’s Becky Bracken: Plus one for transparency. As always, sunlight is the best disinfectant. So let’s hope that continues.
Rich Mogull, chief analyst at the Cloud Security Alliance, thank you so so much for taking the time to talk with us today. It’s been really enlightening. I very much appreciate it. This has been a blast. And thank you so much for listening. This has been Dark Reading Confidential. It’s a podcast from the editors of Dark Reading bringing you real-world stories straight from the cyber trenches. I’m Becky Bracken, and on behalf of everybody here at Dark Reading, thanks for listening. We’ll see you next time.
Rich Mogull: Yeah, thanks for having me back.
