Episodes

  • LW - Secondary forces of debt by KatjaGrace

    LW - Secondary forces of debt by KatjaGrace
    Jun 28 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Secondary forces of debt, published by KatjaGrace on June 28, 2024 on LessWrong. A general thing I hadn't noticed about debts until lately: Whenever Bob owes Alice, then Alice has reason to look after Bob, to the extent that increases the chance he satisfies the debt. Yet at the same time, Bob has an incentive for Alice to disappear, insofar as it would relieve him. These might be tiny incentives, and not overwhelm for instance Bob's many reasons for not wanting Alice to disappear. But the bigger the owing, the more relevant the incentives. When big enough, the former comes up as entities being "too big to fail", and potentially rescued from destruction by those who would like them to repay or provide something expected of them in future. But the opposite must exist also: too big to succeed - where the abundance owed to you is so off-putting to provide that those responsible for it would rather disempower you. And if both kinds of incentive are around in whisps whenever there is a debt, surely they often get big enough to matter, even before they become the main game. For instance, if everyone around owes you a bit of money, I doubt anyone will murder you over it. But I wouldn't be surprised if it motivated a bit more political disempowerment for you on the margin. There is a lot of owing that doesn't arise from formal debt, where these things also apply. If we both agree that I - as your friend - am obliged to help you get to the airport, you may hope that I have energy and fuel and am in a good mood. Whereas I may (regretfully) be relieved when your flight is canceled. Money is an IOU from society for some stuff later, so having money is another kind of being owed. Perhaps this is part of the common resentment of wealth. I tentatively take this as reason to avoid debt in all its forms more: it's not clear that the incentives of alliance in one direction make up for the trouble of the incentives for enmity in the other. And especially so when they are considered together - if you are going to become more aligned with someone, better it be someone who is not simultaneously becoming misaligned with you. Even if such incentives never change your behavior, every person you are obligated to help for an hour on their project is a person for whom you might feel a dash of relief if their project falls apart. And that is not fun to have sitting around in relationships. (Inpsired by reading The Debtor's Revolt by Ben Hoffman lately, which may explicitly say this, but it's hard to be sure because I didn't follow it very well. Also perhaps inspired by a recent murder mystery spree, in which my intuitions have absorbed the heuristic that having something owed to you is a solid way to get murdered.) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    Show more Show less
    3 mins
  • LW - AI #70: A Beautiful Sonnet by Zvi

    LW - AI #70: A Beautiful Sonnet by Zvi
    Jun 28 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #70: A Beautiful Sonnet, published by Zvi on June 28, 2024 on LessWrong. They said it couldn't be done. No, not Claude Sonnet 3.5 becoming the clear best model. No, not the Claude-Sonnet-empowered automatic meme generators. Those were whipped together in five minutes. They said I would never get quiet time and catch up. Well, I showed them! That's right. Yes, there is a new best model, but otherwise it was a quiet week. I got a chance to incorporate the remaining biggest backlog topics. The RAND report is covered under Thirty Eight Ways to Steal Your Model Weights. Last month's conference in Seoul is covered in You've Got Seoul. I got to publish my thoughts on OpenAI's Model Spec last Friday. Table of Contents Be sure to read about Claude 3.5 Sonnet here. That is by far the biggest story. 1. Introduction. 2. Table of Contents. 3. Language Models Offer Mundane Utility. I am increasingly persuaded. 4. Language Models Don't Offer Mundane Utility. EU's DMA versus the AiPhone. 5. Clauding Along. More people, mostly impressed. 6. Fun With Image Generation. They are coming for our memes. Then Hollywood. 7. Copyright Confrontation. The RIAA does the most RIAA thing. 8. Deepfaketown and Botpocalypse Soon. Character.ai addiction. Am I out of touch? 9. They Took Our Jobs. More arguments that the issues lie in the future. 10. The Art of the Jailbreak. We need to work together as a team. 11. Get Involved. AISI, Apollo, Astra, Accra, BlueDot, Cybersecurity and DOE. 12. Introducing. Forecasting, OpenAI Mac App, Otto, Dot, Butterflies, Decagon. 13. In Other AI News. OpenAI equity takes steps forward. You can sell it. 14. Quiet Speculations. A distinct lack of mojo. 15. You've Got Seoul. Delayed coverage of the Seoul summit from last month. 16. Thirty Eight Ways to Steal Your Model Weights. Right now they would all work. 17. The Quest for Sane Regulations. Steelmanning restraint. 18. SB 1047. In Brief. 19. The Week in Audio. Dwarkesh interviews Tony Blair, and many more. 20. Rhetorical Innovation. A demolition, and also a disputed correction. 21. People Are Worried About AI Killing Everyone. Don't give up. Invest wisely. 22. Other People Are Not As Worried About AI Killing Everyone. What even is ASI? 23. The Lighter Side. Eventually the AI will learn. Language Models Offer Mundane Utility Training only on (x,y) pairs, define the function f(x), compose and invert it without in-context examples or chain of thought. AI Dungeon will let you be the DM and take the role of the party, if you prefer. Lindy 'went rogue' and closed a customer on its own. They seem cool with it? Persuasive capability of the model is proportional to the log of the model size, says paper. Author Kobi Hackenburg paints this as reassuring, but the baseline is that everything scales with the log of the model size. He says this is mostly based on 'task completion' and staying on topic improving, and current frontier models are already near perfect at that, so he is skeptical we will see further improvement. I am not. I do believe the result that none of the models was 'more persuasive than human baseline' in the test, but that is based on uncustomized messages on generic political topics. Of course we should not expect above human performance there for current models. 75% of knowledge workers are using AI, but 78% of the 75% are not telling the boss. Build a team of AI employees to write the first half of your Shopify CEO speech from within a virtual office, then spend the second half of the speech explaining how you built the team. It is so weird to think 'the best way to get results from AI employees I can come up with is to make them virtually thirsty so they will have spontaneous water cooler conversations.' That is the definition of scratching the (virtual) surface. Do a bunch of agent-based analysis off a si...
    Show more Show less
    1 hr and 12 mins
  • EA - My Current Claims and Cruxes on LLM Forecasting & Epistemics by Ozzie Gooen

    EA - My Current Claims and Cruxes on LLM Forecasting & Epistemics by Ozzie Gooen
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Current Claims and Cruxes on LLM Forecasting & Epistemics, published by Ozzie Gooen on June 27, 2024 on The Effective Altruism Forum. I think that recent improvements in LLMs have brought us to the point where LLM epistemic systems are starting to be useful. After spending some time thinking about it, I've realized that such systems, broadly, seem very promising to me as an effective altruist intervention area. However, I think that our community has yet to do a solid job outlining what this area could look like or figuring out key uncertainties. This document presents a rough brainstorm on these topics. While I could dedicate months to refining these ideas, I've chosen to share these preliminary notes now to spark discussion. If you find the style too terse, feel free to use an LLM to rewrite it in a format you prefer. I believe my vision for this area is more ambitious and far-reaching (i.e. not narrow to a certain kind of forecasting) than what I've observed in other discussions. I'm particularly excited about AI-heavy epistemic improvements, which I believe have greater potential than traditional forecasting innovations. I'm trying to figure out what to make of this regarding our future plans at QURI, and I recommend that other organizations in the space consider similar updates. Key Definitions: Epistemic process: A set of procedures to do analysis work, often about topics with a lot of uncertainty. This could be "have one journalist do everything themselves", to a complex (but repeatable) ecosystem of multiple humans and software systems. LLM-based Epistemic Process (LEP): A system that relies on LLMs to carry out most or all of an epistemic process. This might start at ~10% LLM-labor, but can gradually ramp up. I imagine that such a process is likely to feature some kinds of estimates or forecasts. Scaffolding: Software used around an LLM system, often to make it valuable for specific use-cases. In the case of an LEP, a lot of scaffolding might be needed. 1. High-Level Benefits & Uses Claim 1: If humans could forecast much better, these humans should make few foreseeable mistakes. This covers many mistakes, particularly ones we might be worried about now. Someone deciding about talking to a chatbot that can be predicted to be net-negative (perhaps it would create an unhealthy relationship) could see this forecast and simply decide not to start the chat. Say that a person's epistemic state could follow one of four trajectories, depending on some set of reading materials. For example, one set is conspiratorial, one is religious, etc. Good forecasting could help forecast this and inform the person ahead of time. Note that this can be radical and perhaps dangerous. For example, a religious family knowing how to keep their children religious with a great deal of certainty. Say that one of two political candidates is predictably terrible. This could be made clear to voters who trust said prediction. If an AI actor is doing something likely to be monopolistic or dangerous, this would be made more obvious to itself and those around it. Note: There will also be unforeseeable mistakes, but any actions that we could do that are foreseeably-high-value for them, could be predicted. For example, general-purpose risk mitigation measures. Claim 2: Highly intelligent / epistemically capable organizations are likely to be better at coordination. This might well mean fewer wars and conflict, along with corresponding military spending. If highly capable actors were in a prisoner's dilemma, the results could be ugly. But very often, there's a lot of potential and value in not getting into one in the first place. Evidence: From The Better Angels of Our Nature, there's significant evidence that humanity has become significantly less violent over time. One potential exception is t...
    Show more Show less
    40 mins
  • EA - How can we get the world to talk about factory farming? by LewisBollard

    EA - How can we get the world to talk about factory farming? by LewisBollard
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How can we get the world to talk about factory farming?, published by LewisBollard on June 27, 2024 on The Effective Altruism Forum. Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post. Silence favors the status quo, which doesn't favor animals It's easy to ignore factory farming. Inflation sparks public debates about the economy. Natural disasters spur news stories on climate change. Advances in artificial intelligence prompt discussion of its risks. But abuses on factory farms go ignored. An analysis by my colleague Emma Buckland found that, since 2010, global English-language print and online news coverage of factory farming has only grown in line with other reporting on agriculture (see graph below). By contrast, coverage of climate change has grown two to three times faster. Google News lists 0.02 - 0.4% as many articles in the last week on factory farming as on climate change. Undercover investigations once broke this media silence. In the decade up to 2018, top US media outlets, like CBS, CNN, and NBC, routinely covered their findings. Since then, they seldom have. Before 2018, 27 undercover investigations from the top three investigative groups surpassed 500,000 views on YouTube. Since then, none have. This matters because factory farming thrives in the dark. Many industry practices are publicly indefensible, so the industry prefer to not publicly discuss them at all. And when the media ignores factory farming, politicians and corporate leaders can too. A 2022 Faunalytics study tested the impact of various advocacy tactics on 2,405 people. News articles and social media posts most reduced self-reported animal product consumption and improved attitudes toward farm animal welfare. (Though the impact of all tactics was small.) They also didn't trigger a backlash, as more confrontational tactics like disruptive protests did. Why is factory farming so rarely publicly discussed? Some blame industry capture of the media. But the industry struggles to get news coverage too. The US chicken industry's main communications initiative, Chicken Check-In, appears to have never secured a story in a mainstream news outlet or many "likes" on its social media posts. The problem is not media bias, but media indifference. That indifference likely has many causes. Factory farming's horrors aren't new, so they're not "news." The topic is too obscure for most newspapers, too gruesome for most television shows, and too mundane for most online culture warriors. It doesn't help that animals can't speak, so they can't squawk about their plight online. The decline in coverage of undercover investigations is more mysterious. It may have to do with ag-gag laws, the collapse of investigative journalism, or the media's obsession with US politics. But it may also be thanks to pink slime. ABC News' reporting on that meat-derived goo ensnared it in a lawsuit, which led to a record defamation settlement of $177M in 2017. Soon afterward, media coverage of factory farm investigations began to decline. The story on social media is even less clear. The algorithms likely changed, but we don't know how or why. We may be victims of the social media giants' 2016 post-election crack-down on distressing videos. Or the algorithms may just have mastered what people really want to watch - and the answer is kittens in a maze, not tortured chickens. What can we do about this? I'm no PR expert, so I asked some movement leaders who are, plus a few friends in the media. They had lots of ideas - far too many to list here. So I focus below on some broad points of agreement across three areas: media, influencers, and narrative. (A disclaimer: this is a list of int...
    Show more Show less
    11 mins
  • EA - Detecting Genetically Engineered Viruses With Metagenomic Sequencing by Jeff Kaufman

    EA - Detecting Genetically Engineered Viruses With Metagenomic Sequencing by Jeff Kaufman
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Detecting Genetically Engineered Viruses With Metagenomic Sequencing, published by Jeff Kaufman on June 27, 2024 on The Effective Altruism Forum. This represents work from several people at the NAO. Thanks especially to Dan Rice for implementing the duplicate junction detection, and to @Will Bradshaw and @mike_mclaren for editorial feedback. Summary If someone were to intentionally cause a stealth pandemic today, one of the ways they might do it is by modifying an existing virus. Over the past few months we've been working on building a computational pipeline that could flag evidence of this kind of genetic engineering, and we now have an initial pipeline working end to end. When given 35B read pairs of wastewater sequencing data it raises 14 alerts for manual review, 13 of which are quickly dismissible false positives and one is a known genetically engineered sequence derived from HIV. While it's hard to get a good estimate before actually going and doing it, our best guess is that if this system were deployed at the scale of approximately $1.5M/y it could detect something genetically engineered that shed like SARS-CoV-2 before 0.2% of people had been infected. System Design The core of the system is based on two observations: If someone has made substantial modifications to an existing virus then somewhere in the engineered genome there will be a series of bases that are a good match for the original genome followed by a series of bases that are a poor match for the original genome. We can look for sequencing reads that have this property and raise them for human review. Chimeric reads can occur as an artifact of sequencing, which can lead to false positives. The chance that you would see multiple chimeras involving exactly the same junction by chance, however, is relatively low. By requiring 2x coverage of the junction we can remove almost all false positives, at the cost of requiring approximately twice as much sequencing. Translating these observations into sufficiently performant code that does not trigger alerts on common sequencing artifacts has taken some work, but we now have this running. While it would be valuable to release our detector so that others can evaluate it or apply it to their own sequencing reads, knowing the details of how we have applied this algorithm could allow someone to engineer sequences that it would not be able to detect. While we would like to build a detection system that can't be more readily bypassed once you know how it works, we're unfortunately not there yet. Evaluation We have evaluated the system in two ways: by measuring its performance on simulated genetic engineered genomes and by applying it to a real-world dataset collected by a partner lab. Simulation We chose a selection of 35 viruses that Virus Host DB categorizes as human-infecting viruses, with special attention to respiratory viruses: Disease Virus Genome Length AIDS HIV 9,000 Chickenpox and Shingles Human alphaherpesvirus 3 100,000 Chikungunya Chikungunya virus 10,000 Common cold Human coronavirus 229E 30,000 Common cold Human coronavirus NL63 30,000 Common cold Human coronavirus OC43 30,000 Common cold Human rhinovirus NAT001 7,000 Common cold Rhinovirus A1 7,000 Common cold Rhinovirus B3 7,000 Conjunctivitis Human adenovirus 54 30,000 COVID-19 SARS-CoV-2 30,000 Ebola Ebola 20,000 Gastroenteritis Astrovirus MLB1 6,000 Influenza Influenza A Virus, H1N1 10,000 Influenza Influenza A Virus, H2N2 10,000 Influenza Influenza A Virus, H3N2 10,000 Influenza Influenza A Virus, H7N9 10,000 Influenza Influenza A Virus, H9N2 10,000 Influenza Influenza C Virus 10,000 Measles Measles morbillivirus 20,000 MERS MERS Virus 30,000 Metapneumovirus infection Human metapneumovirus 10,000 Mononucleosis Human herpesvirus 4 type 2 200,000 MPox Monkeypox virus 200,000 Mumps Mumps orthor...
    Show more Show less
    15 mins
  • AF - An issue with training schemers with supervised fine-tuning by Fabien Roger

    AF - An issue with training schemers with supervised fine-tuning by Fabien Roger
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An issue with training schemers with supervised fine-tuning, published by Fabien Roger on June 27, 2024 on The AI Alignment Forum. This is a rough write-up of ideas by Ryan Greenblatt. The core ideas are his and mistakes are mine. Thanks to him, Buck Shlegeris and to Aghyad Deeb for feedback on the draft of this post. One way to use scheming AIs is to train them to imitate humans. When does imitation robustly avoid frequent and intentional failures? On single-token classification tasks, imitation works great and can allow you to get a lot of high-quality labor with a tiny rate of intentional failures; On tasks that involve generating many tokens, imitation potentially fails by default: schemers could notice when an output was generated by themselves, which enables them to consistently behave badly in deployment while doing best-effort imitation during training. We describe two techniques that avoid imitation failures on generation tasks: A technique that works perfectly but is too expensive A technique that we think works in practice While pure imitation may not be sufficiently competitive, we think it could prove useful for tasks such as: (No-CoT) single-token classification; KL-regularization for RL: we may want to penalize models trained with RL for straying too far from a human imitation prior (and we would want this human prior to not be malign!) (see Appendix); Tasks for which it is fine to take the competitiveness hit from not using RL, but which weaker trusted models are not good enough at.[1] Assumptions In this post, we focus on scenarios where: AIs are smart (around human-level or above); AIs are scheming and know how we are training and deploying them; We want to use these AIs productively (e.g. to do safety research)[2]; Training and deployment tasks are indistinguishable (e.g. because we randomly sampled training tasks from the deployment distribution); SGD works: we ignore gradient hacking and AI architectures where SGD is very sample inefficient; AIs can't cause catastrophes in very few actions (e.g. because they are too weak to break safety measures that address these sorts of concerns); Even when AIs try their best, they can't perfectly predict the probability distribution over the actions that humans[3] would take. Easy-mode: imitating human judgment on classification tasks Here is how you can use an AI to 100x the speed at which you can complete a billion single-token (IID) classification tasks (e.g. detecting software bugs, scoring project ideas, …): 1. Sample 1% of the tasks, and ask humans to label them (using their judgment); 2. Train the AI on that 1% of tasks you have solved yourself, using normal training (e.g. SGD with a cross-entropy loss); 3. Use the AI to solve the remaining 99% of tasks. This will get you an AI which failed intentionally at most a few times in the whole training set because intentional failures would have quickly been trained away[4]. Therefore, you got at most a few hundred intentional failures in the whole set of tasks. That is an intentional failure rate at most around 1/10M, not bad![5] In other words, human imitation avoids intentional low-stakes failures: the model won't often be worse than a best-effort imitation of a human. (But best-effort imitation of a human could be catastrophic if that isn't sufficiently capable.) The basic problem with imitating humans on generation tasks But most tasks we care about involve generating many tokens (e.g. coding a complex experiment, writing a detailed project proposal, …). Does imitation work for such generation tasks? Not quite, here is a problem that could arise if you train an AI to imitate human answers on 1% of tasks and use it on the remaining 99% without additional supervision: On the 1% of training tasks, the AI tries its best to imitate human answers when the star...
    Show more Show less
    17 mins
  • LW - Schelling points in the AGI policy space by mesaoptimizer

    LW - Schelling points in the AGI policy space by mesaoptimizer
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Schelling points in the AGI policy space, published by mesaoptimizer on June 27, 2024 on LessWrong. I've been thinking about memetically fit Schelling points in the AGI policy space. I'll describe four such "Schelling policies", and use them as pedagogical examples. Shut it all down MIRI's new stated objective is the clearest example of a Schelling policy: "Shut it all down". MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies. If you want to coordinate a large number of people coherently towards furthering a particular policy, "you get about five words" that you can make 'common knowledge' such that people can coordinate in a specific direction. The ease of communicating the policy makes a big difference in such conditions. When you attempt to communicate an idea widely, you'll notice that people usually end up with multiple slightly (or sometimes wildly) differing copies of the original idea. If you've played the Telephone game, you've experienced just how much information can be lost as an idea spreads from one person to another. In the context of policies, individual people's beliefs and incentives will warp the instantiation of the policy they will communicate and support. (For example, you'll find companies lobbying regulators to carve out exceptions that benefit them.) Here's where Schelling points are invaluable: they serve as natural attractors in the space of ideas, and therefore enable people to 'error-correct' the idea they encounter and figure out the policy that everyone is coordinating around. "Shut it all down" is a Schelling point. "Shut it all down if we see evidence of unprompted deception and power-seeking in AGI models" is not a Schelling point, you have multiple free variables that can and will be optimized to benefit the people spreading the idea -- which can result in a lack of coordination and the idea being outcompeted by memetically fitter ideas. "Prevent the training of models using compute greater than 1025 floating point operations" also has a free variable: why exactly 1025 floating point operations? Why not 1024 or 1026? Until 1025 floating point operations becomes a Schelling number, the policy containing it is not a Schelling point. Effective Accelerationism (e/acc) The biggest difference between e/acc and the PauseAI memeplexes is that e/acc doesn't seem to have a coherent set of goals and beliefs. Here are a bunch of memes that e/acc people tend to espouse: "It's time to build." (also the last line of The Techno-Optimist Manifesto) "Come and take it." (where "it" refers to GPUs here) "Accelerate or die." At a first glance, one might say that e/acc isn't a Schelling policy -- it seems less like a coherent policy, and more like a set of 'vibes', verbal and non-verbal statements designed to create a desired emotional impact, regardless of the actual content. I disagree. A policy (or well, a memeplex) does not need to have an explicitly coherent set of beliefs and goals for it to result in coordinating people towards particular consequences. You might expect this to reduce the spread rate of this particular policy, but e/acc specifically compensates for it by being significantly more fun and socially, financially, and professionally profitable to coordinate around. For example, venture capital firms such as a16z want the opportunity to make a lot of money from the gold rush that is the race to AGI, and a lot of software developers want a shot at making billions of dollars if their startup succeeds. The possibility of regulations would cause the music to stop, and they don't want that. In fact, you don...
    Show more Show less
    9 mins
  • LW - Live Theory Part 0: Taking Intelligence Seriously by Sahil

    LW - Live Theory Part 0: Taking Intelligence Seriously by Sahil
    Jun 27 2024
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Live Theory Part 0: Taking Intelligence Seriously, published by Sahil on June 27, 2024 on LessWrong. Acknowledgements The vision here was midwifed originally in the wild and gentle radiance that is Abram's company (though essentially none of the content is explicitly his). The PIBBSS-spirit has been infused in this work from before it began (may it infuse us all), as have meetings with the Agent Foundations team at MIRI over the past ~2 years. More recently, everyone who has been loving the High Actuation project into form (very often spontaneously and without being encumbered by self-consciousness of this fact):[1] individuals include Steve Petersen, Mateusz Baginski, Aditya Prasad, Harmony, TJ, Chris Lakin; the AISC 2024 team, Murray Buchanan, Matt Farr, Arpan Agrawal, Adam, Ryan, Quinn; various people from Topos, ALIFE, MAPLE, EA Bangalore. Published while at CEEALAR. Disclaimers Very occasionally there are small remarks/questions from a remarkable human named Steve, since this and the next two posts are an edited transcript of me giving him a talk. I left them in to retain the conversational tone. Steve has also consistently been a fantastic ground for this channeling. I use the term "artefact" a fair amount in this sequence. Unfortunately for you and me, Anthropic also recently started using "artifact" in a different way. I'm using "artefact" in the common sense of the word. The British spelling should help remind of the distinction. Taking Intelligence Seriously Sahil: I gave a talk recently, at an EA event just two days ago, where I made some quick slides (on the day of the talk, so not nearly as tidy as I'd like) and attempted to walk through this so-called "live theory". (Alternative terms include "adaptive theory" or "fluid theory"; where the theories themselves are imbued with some intelligence.) Maybe I can give you that talk. I'm not sure how much of what I was saying there will be present now, but I can try. What do you think? I think it'll take about 15 minutes. Yeah? Steve: Cool. Sahil: Okay, let me give you a version of this talk that's very abbreviated. So, the title I'm sure already makes sense to you, Steve. I don't know if this is something that you know, but I prefer the word "adaptivity" over intelligence. I'm fine with using "intelligence" for this talk, but really, when I'm thinking of AI and LLMs and "live" (as you'll see later), I'm thinking, in part, of adaptive. And I think that connotes much more of the relevant phenomena, and much less controversially. It's also less distractingly "foundational", in the sense of endless questions on "what intelligence means". Failing to Take Intelligence Seriously Right. So, I want to say there are two ways to fail to take intelligence, or adaptivity, seriously. One is, you know, the classic case, of people ignoring existential risk from artificial intelligence. The old "well, it's just a computer, just software. What's the big deal? We can turn it off." We all know the story there. In many ways, this particular failure-of-imagination is much less pronounced today. But, I say, a dual failure-of-imagination is true today even among the "cognoscenti", where we ignore intelligence by ignoring opportunities from moderately capable mindlike entities at scale. I'll go over this sentence slower in the next slide. For now: there are two ways to not meet reality. On the left of the slide is "nothing will change". The same "classic" case of "yeah, what's the big deal? It's just software." On the right, it's the total singularity, of extreme unknowable super-intelligence. In fact, the phrase "technological singularity", IIRC, was coined by Vernor Vinge to mark the point that we can't predict beyond. So, it's also a way to be mind-killed. Even with whatever in-the-limit proxies we have for this, we make various sim...
    Show more Show less
    20 mins