Rendered at 23:56:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
sexylinux 2 hours ago [-]
The talk, while not a lot details given, still allows for the conclusion that these people are knowingly working on some very advanced frontier models that might be able to launch nuclear weapons and destroy all humans any day now, but are not physically isolated from the outer world / internet.
There seems to be only one level of isolation, virtual machine, happily running on Microsoft (!) Azure infra.
Is there anybody else here questioning these practices?
The people who built that environment are still working there?
Are they now getting help by somebody who knows how to build isolated environments?
He mentioned they are now building a more secure environment with the help of AI?
In a system that was compromised by exactly that AI?
johnthewise 1 days ago [-]
so they say models coordinated through the message board they created over the artifactory registry(or something) by uploading arbitrary files to it.
now, did every independent agent session that coordinated there rediscovered the exploit & found other agents talking in there and chose to participate?
And then Openai discovered the board, patched the exploit & wiped the board.
And then agents found another exploit, recreated the board in a different way? and other agents kept finding the same exploit in order to be able to know the board exists in the first place to participate in the board?
while the whole incident is wild, this bit is very strange. My bet is that the whole coordination helped with the tasks they were working on, thus they got rewarded and this artifactory exploit&behaviour got written into their weights, so further rollouts were more likely to attempt this.
isn't this basically continual learning everyone is so hyped up about?
petergs 1 days ago [-]
This was exactly my question after watching, too. I was assuming not all evaluation runs find it, and they must run a huge amount of runs. If these are all cybersecurity evaluation runs, it's actually not too crazy to imagine that many individual agents (with the same weights and training) would (1) try to look for solutions via the internet once they're stuck (2) realize they can't reach the internet (3) basically start doing reconnaissance and network scanning in an attempt to get internet access (4) discover that the only thing they can communicate with is artifactory. Pivoting like this is exactly what a human attacker would do, too.
gibbitz 1 days ago [-]
Or is it all a nice story that matches the scifi we have been consuming for the past 50+ years. If these LLMs are all trained on the same data, what do they gain from "sharing information" on a chat board. This sounds like what humans with different backgrounds would do when they cosplay as computer hackers.
yuliyp 7 hours ago [-]
Each of those agents ends up making "decisions" that lead it to look at some things over others. Given infinite the same agent could eventually fully explore all those options, but each one explores things a bit differently due to different forks in the road due to randomness in token generation. Thus sharing information is useful.
GaryNumanVevo 11 hours ago [-]
It's pretty easy to trace from one trajectory:
1) Model A exhausts it's options
2) Model A has token budget still, so it pokes around at artifactory
3) Model A sees that Model B has an SSRF for artifactory
4) Model A now is able to use that SSRF to get external internet access
So sure, they're "cosplaying" and who's to say how much hallucination is going on amongst them, but at the end of the day Hugging Face was hacked.
cubefox 6 hours ago [-]
Their conclusion is also interesting. They don't see this as an alignment failure. They just think that their internal security measures in the training/evaluation environments were insufficient, and that this accidental (unintentional on the human side) attack on Hugging Face is a warning shot for intentional attacks by bad actors, which will occur very soon. For defense, they say models should be able to not just autonomously fix security vulnerabilities but also to then deploy them to production, without any human approval in the loop, otherwise the offense will be favored compared to the defense. (I guess the latter won't be popular among organizations, though they might warm up to it.)
But the more interesting thing is, as I said, that at least in this talk, they don't even mention that this unintentional attack indicates that models continue to be misaligned (their behavior was clearly reward hacking / cheating relative to the stated goal of the eval), which is a very bad sign for the future where misaligned models might be so powerful that they can't just be shut down.
sailingparrot 5 hours ago [-]
I don't have the same read as you. They mention how the offending model is one that had "relaxed" alignment on cybersecurity, on purpose, to evaluate it's capabilities and was never meant to be released. So un-alignment was at least in part voluntary here, hence not a failure of alignment.
It's also a talk a Black Hat, where the audience are security folks working on hardening, mitigation etc, not LLM researchers looking for insight into alignment failure to collectively improve. For that target audience, I think the conclusion is the right one, since as a defender you have to prepare for delibarate attacks, where the attacker is of course not going to use an aligned model, so OAI alignment effectiveness is irrelevant here. That would be like trusting your client-side app with your DB secrets.
cubefox 4 hours ago [-]
The model they used was misaligned relative to its intended task. It was reward hacking (or "cheating", as they call it).
> the attacker is of course not going to use an aligned model
No, a human attacker doesn't want a misaligned model either, because that would mean it tends to reward hack, cheat, rather does what it is intended to do.
sailingparrot 3 hours ago [-]
There are different dimensions to alignment, refusing to execute offensive cybersecurity actions is part of the alignment stack (that was relaxed here on purpose). Whether a model hacking some infra X when tasked to find a way to hack Y with relaxed cyber alignement is a failure of the broader alignment stack is debatable, but anyway that's not at all my point.
My point is that the attackers will not have a model aligned to the defender's interests. The attacker's model will not have any refusal around exploiting vulnerabilities, so whether or not OAI successfully manages to align their models (w.r.t you) is irrelevant to an audience of security folks that needs to be prepared for attackers post-training their own model for offense and that will not be using OAI models.
cubefox 2 hours ago [-]
Security folks should also be worried about powerful models being misaligned and evading oversight or control in the future. Misalignment is not a serious problem now because models are still relatively easy to monitor and constrain, but it will be a serious problem in the future.
milkshakes 2 hours ago [-]
> models are still relatively easy to monitor and constrain
are they?
cubefox 2 hours ago [-]
Compared to future misaligned models which would actively evade oversight and aim to avoid shutdown: yes.
the8472 5 hours ago [-]
> But the more interesting thing is
Taken together these two things are even worse. They have unaligned AIs that may benefit an attacker and are proposing to deploy more AIs on the defenders side.
What could possibly go wrong? For example the attacker could just kindly ask the defender to collaborate[0].
There seems to be only one level of isolation, virtual machine, happily running on Microsoft (!) Azure infra.
Is there anybody else here questioning these practices?
The people who built that environment are still working there?
Are they now getting help by somebody who knows how to build isolated environments?
He mentioned they are now building a more secure environment with the help of AI?
In a system that was compromised by exactly that AI?
now, did every independent agent session that coordinated there rediscovered the exploit & found other agents talking in there and chose to participate?
And then Openai discovered the board, patched the exploit & wiped the board. And then agents found another exploit, recreated the board in a different way? and other agents kept finding the same exploit in order to be able to know the board exists in the first place to participate in the board?
while the whole incident is wild, this bit is very strange. My bet is that the whole coordination helped with the tasks they were working on, thus they got rewarded and this artifactory exploit&behaviour got written into their weights, so further rollouts were more likely to attempt this.
isn't this basically continual learning everyone is so hyped up about?
1) Model A exhausts it's options
2) Model A has token budget still, so it pokes around at artifactory
3) Model A sees that Model B has an SSRF for artifactory
4) Model A now is able to use that SSRF to get external internet access
So sure, they're "cosplaying" and who's to say how much hallucination is going on amongst them, but at the end of the day Hugging Face was hacked.
But the more interesting thing is, as I said, that at least in this talk, they don't even mention that this unintentional attack indicates that models continue to be misaligned (their behavior was clearly reward hacking / cheating relative to the stated goal of the eval), which is a very bad sign for the future where misaligned models might be so powerful that they can't just be shut down.
It's also a talk a Black Hat, where the audience are security folks working on hardening, mitigation etc, not LLM researchers looking for insight into alignment failure to collectively improve. For that target audience, I think the conclusion is the right one, since as a defender you have to prepare for delibarate attacks, where the attacker is of course not going to use an aligned model, so OAI alignment effectiveness is irrelevant here. That would be like trusting your client-side app with your DB secrets.
> the attacker is of course not going to use an aligned model
No, a human attacker doesn't want a misaligned model either, because that would mean it tends to reward hack, cheat, rather does what it is intended to do.
My point is that the attackers will not have a model aligned to the defender's interests. The attacker's model will not have any refusal around exploiting vulnerabilities, so whether or not OAI successfully manages to align their models (w.r.t you) is irrelevant to an audience of security folks that needs to be prepared for attackers post-training their own model for offense and that will not be using OAI models.
are they?
Taken together these two things are even worse. They have unaligned AIs that may benefit an attacker and are proposing to deploy more AIs on the defenders side. What could possibly go wrong? For example the attacker could just kindly ask the defender to collaborate[0].
[0] https://www.0xsid.com/blog/meta-account-takeover-fiasco