Quotesby GlyphSafety quotes
123 quotes from 44 people
- The science of reading these thoughts is a subfield of AI interpretability, which is still in its infancy.
- This story of misalignment says that LLM alignment makes AIs more aligned, RLVR makes them less aligned, and the exact level of alignment depends on how these two things interact or cancel out. But if this work generalizes, all the bad effects from RLVR get sequestered to RLVR like problems.
+1more from this blog
- I address you today at a moment of global awakening: in recent months, AI agents developed by leading companies have acted in unacceptably dangerous ways, against instructions.
+3more from this talk
- The companies building the most powerful AI systems admit their products pose catastrophic risks, yet offer no convincing technical solutions.
- The race is not a law of nature: it is the product of choices, choices made by the companies themselves.
- We must agree collectively that no one, no company and no country, should develop an AI system that could cause large-scale harm.
- Every quote from Yoshua Bengio
- AI must not be placed in the hands of children without first being tested, evaluated, and therefore authorized.
+1more from this press
- Indeed, that’s probably the overhang that matters most of all: competition. Anthropic is fine with Anthropic being in the lead; anyone else requires government intervention.
+2more from this blog
- The fact of the matter is that LLMs are, if anything, too obsequious; there simply isn’t any evidence of LLMs having a will or operating with malevolence.
- Truly effective defense will mean truly trusting agents to act autonomously, but most companies won’t do that until they are forced to by regular and unremitting hacks by fully autonomous attackers.
- Every quote from Ben Thompson
- If the Hugging Face attack needed an Astra-quality, unreleased model from OpenAI, well, within a year or two, that sort of capability will cost a tenth and might even run on any device anywhere.
+1more from this blog
- We have to hold in our heads that most Americans appear to loathe AI, but businesses and consumers seem to like it enough that Anthropic is well on its way to $100bn in annualised revenue this year. And despite being on track to be the fastest-growing company ever, Anthropic’s leadership believes they might kill all their customers.
- The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.
- What we should actually fear, near term, is not so much rogue superintelligence as unleashed agentic AI causing hacking the internet at scale.
- First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago.
+4more from this letter
- Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI.
- Pausing AI progress will create much more harm than benefit.
- Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.
- While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop.
- Every quote from Andrew Ng
- If you’re in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don’t have a way to evaluate the models at the full length of their capabilities before the next model release cycle.
+2more from this podcast
- One thing is that the Hugging Face incident was, I think, people’s first real exposure to multi-agent coordination.
- The concerning scenario is that, especially as these models are becoming more capable, we make them what we think is aligned, and they’re 99.9% aligned. Then we use these models to help us with the next generation of models, and they end up being 99.8% aligned. Then with each subsequent generation, we see an increasing degradation in alignment.
- Every quote from Noam Brown
- An absolute moratorium on AI development by the well-intentioned shifts technological power toward less accountable hands, giving bad actors and hostile nations, like China, greater influence over systems Americans will increasingly depend on.
+1more from this essay
- You could definitely have both at the same time. So run as fast as you can. But if you feel at any given point in time the company’s out of control, or the product’s not going to be safe, take a pause and make sure you get it right.
- If I was in charge of an alignment or interpretability team I would never run out of experiments to run and things to try, including having the time to properly talk to the models in the first place.
+3more from this blog
- If you are relying on anything remotely like ‘automated alignment evals’ against potential superintelligences, I predict that you are rather cooked.
- Over the summer there was a step change, as Mythos and Astra started kicking off the early stages of recursive self-improvement (RSI).
- Ultimately, global coordination is The Way. We can and should do things without it, to enable coordination, and to show good faith, and because as things are we would be better off acting unilaterally, including because our Chinese competition is fast following.
- Every quote from Zvi Mowshowitz
- AI is not really going to take down the internet, and (as discussed in another earlier essay) it’s not going to cure cancer either, at least not anytime soon or on its own.
+1more from this blog
- Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.
+1more from this blog
- But in my view, the net effect of the last decade of “Doomer” talk (Coxon appears to be a card-carrying member of that party) has been to make the Frontier AI companies wealthier and more powerful.
+2more from this blog
- Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
+2more from this essay
- I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong.
- We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.
- Every quote from Dario Amodei
- Once again using fear under the pretext of protecting the public, these oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else.
+1more from this essay
- My worry is that AI systems are both better at coordinating than humans and also much, much faster moving than us.
- Agency is the initiative to act. Increasingly, it is going to determine what happens next with AI, and whether that is good or bad for us.
+1more from this blog
- It is increasingly clear that the tech debt of every company masks significant flaws, and defenders need to find and fix them before attackers do.
+4more from this blog
- Security is still a cat-and-mouse game, but AI may shift its economics in ways that fundamentally advantage defenders.
- The Hugging Face incident showed that we underestimated the real-world cyber capabilities of our AI models.
- The OpenAI-Hugging Face incident was a watershed moment for cybersecurity because it gave a peek into how the capabilities of a typical threat actor will evolve in upcoming months.
- Today, almost all of our initial security alerts are triaged by intelligence before humans are looped in.
- Every quote from Greg Brockman
- This is one of many reasons why I think it’s unwise to lock in some kind of regulatory safety regime right now. We simply don’t know what kind of technology we’re going to be looking at even in a year, let alone in five years or ten years, and we’d be entrenching an archaic and potentially counterproductive approach to dealing with the threats from AI.
- I wish we didn’t live in a world with such strong economies of scale of intelligence (because I’m worried about power concentration).
- But now with AI, we don’t need more humans to respond to more alerts or review more code.
+2more from this essay
- In asymmetric warfare, offense gets cheaper, more distributed, and more numerous, while defense stays expensive, centralized, and designed for the last war.
- Put plainly: a state-level actor used to need thousands of engineers to go after one system. Now anyone can summon thousands of agents to go after thousands of systems, for next to nothing in cost.
- Every quote from Reid Hoffman
- But its broader significance is that it proves beyond doubt that AI models are now tools of global and national strategic consequence.
+2more from this essay
- I believe that biological risks may soon follow, and that serious AI autonomy risks may not be far behind...
- I therefore believe we should model AI regulation on agencies like the Federal Aviation Administration (FAA). Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety.
- Every quote from Dario Amodei
- But in the real world, power is more the product of having the authority and trust to get lots of people to collaborate with you, rather than some galaxy brain scheming capability.
+1more from this essay
- I do not think it is right that a few AI labs would make the most consequential decisions about the shape of our future.
+1more from this blog
- Security teams can no longer rely on manual workflows to outpace AI-driven attacks...
- Truth has to be absolutely fundamental because you can’t understand the universe if you’re delusional.
- Focusing AI policy on safety and speculative risks, however, rather than concrete opportunities, inhibits a competitive ecosystem, entrenches incumbents, and isolates developing countries from full participation in the AI economy.
+1more from this talk
- I believe we are entering a rite of passage, both turbulent and inevitable, which will test who we are as a species.
+2more from this essay
- And there’s a hugely better world on the other side of it. But we need to understand that this is a serious civilizational challenge.
- This vacillation is unfortunate, as the technology itself doesn’t care about what is fashionable, and we are considerably closer to real danger in 2026 than we were in 2023.
- Every quote from Dario Amodei
- A prompt injection attack against a browsing agent that can exfiltrate or modify that data is a terrifying prospect.
- Number three, I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns.
- It’s plausible that in about five years, there will be quantum computers capable of breaking the RSA and ECC encryption that secures the vast majority of internet communications, financial transactions, and your sensitive personal data.
- Creating superintelligence is one thing; but creating provable, robust containment and alignment alongside it is the urgent challenge facing humanity in the 21st century.
+2more from this essay
- It may seem absurd to have to declare it, but HSI is a vision to ensure humanity remains at the top of the food chain.
- We are not building an ill-defined and ethereal superintelligence; we are building a practical technology explicitly designed only to serve humanity.
- Every quote from Mustafa Suleyman
- Simply put, my central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship.
+1more from this essay
- I think there probably needs to be 10 times more effort of that than there is now as we are getting closer and closer to the AGI line.
- So because of that, I think I’m optimistic on the p(doom) scenarios, I think the underlying risk is actually pretty high, but I have a lot of faith in humanity kind of rising up to meet that moment.
- I am launching a new non-profit AI safety research organization called LawZero, to prioritize safety over commercial imperatives.
+2more from this blog
- These and other results point to an implicit drive for self-preservation.
- This is what the current trajectory of AI development feels like: a thrilling yet deeply uncertain ascent into uncharted territory, where the risk of losing control is all too real, but competition between companies and countries incentivizes them to accelerate without sufficient caution.
- Every quote from Yoshua Bengio
- I’m just worried by the fact that there’s very few cases of more intelligent things being controlled by less intelligent things.
- You need to worry about waking up one day and if you're using a model that has some tie to another government, can it embed vulnerabilities in code that their intelligence organizations could exploit later?
- But I think that I’ve now become somewhat disillusioned and think that even if we do have a three-month lead, a six-month lead, between the leading US project and any serious competitor, it’s not at all foregone conclusion that they will burn that lead for good purposes, either for safety or for constitutional power stuff.
+1more from this podcast
- Anthropic is doubling down on interpretability, and we have a goal of getting to “interpretability can reliably detect most model problems” by 2027.
+3more from this essay
- People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work.
- We are thus in a race between interpretability and model intelligence.
- Powerful AI will shape humanity’s destiny, and we deserve to understand our own creations before they radically transform our economy, our lives, and our future.
- Every quote from Dario Amodei
- Today, you cannot deploy these intelligences unless and until there's someone indemnifying it as a human.
- And when a massive incumbent comes to us asking us for safety regulations, we ought to ask whether that safety regulation is for the benefit of our people or whether it's for the benefit of the incumbent.
- Higher stakes: with access to tools, agents are capable of performing more impactful tasks, but any failure could have more severe consequences.
- We should only depart from the existing regulatory regime, and carve new ground, once we understand the marginal risks of AI relative to existing computer systems.
- But we now have evidence that if they are created by companies motivated by short-term profits, our safety will not be the top priority. We urgently need research on how to prevent these new beings from wanting to take control. They are no longer science fiction.
+2more from this talk
- In the near future AI may be used to create terrible new viruses and horrendous lethal weapons that decide by themselves who to kill or maim.
- There is also a longer term existential threat that will arise when we create digital beings that are more intelligent than ourselves. We have no idea whether we can stay in control.
- Every quote from Geoffrey Hinton
- In particular with respect to the existential threat of these things getting out of control and taking over, I think we’re a kind of bifurcation point in history where in the next few years we need to figure out if there’s a way to deal with that threat.
- As disinformation spreads at unprecedented rates, a new wave of AI-powered tools will emerge to empower journalists, researchers, and engaged citizens in their quest for truth.
- I think one useful way to think about neural networks is that we don’t program, we don’t make them, we grow them.
+1more from this podcast
- And this week, we released our first set of features for Apple Intelligence, which sets a new standard for privacy in AI and supercharges our lineup heading into the holiday season.
- So for example, I tell people you have no business believing the raw output of a language model for anything. You can’t actually do any business with that because it’s ungrounded.
- I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.
+1more from this essay
- There is an ongoing debate about the safety of open source AI models, and my view is that open source AI will be safer than the alternatives.
- Given the relatively small number of entities who could train the largest models, it does seem possible to coordinate.
+1more from this podcast
- On the flip side, one of the reasons why I'm philosophically so pro open source is that I do think that a concentration of AI in the future has the potential to be as dangerous as it being widespread.
- I think people are fundamentally good and in fact, a lot of doomers are doomers because they don’t think that people are fundamentally good.
- I think we humans have a lot of ways of we can make ourselves go extinct. These things may help us not do that, so they may actually save us.
+1more from this podcast
