Nick Bostrom - From Superintelligence to Deep Utopia - Can We Create a Perfect Society?

Since Nick Bostrom wrote Superintelligence, AI has surged from theoretical speculation to powerful, world-shaping reality. Progress is undeniable, yet there is an ongoing debate in the AI safety community – caught between mathematical rigour and swiss-cheese security. P(doom) debates rage on, but equally concerning is the risk of locking in negative-value futures for a very long time. Zooming in: motivation selection-especially indirect normativity-raises the question: is there a structured landscape of possible value configurations, or just a chaotic search for alignment? From Superintelligence to Deep Utopia: not just avoiding catastrophe but ensuring resilience, meaning, and flourishing in a 'solved' world; a post instrumental, plastic utopia – where humans are 'deeply redundant', can we find enduring meaning and purpose? This is our moment to shape the future. What values will we encode? What futures will we entrench? 0:00 Highlights 1:32 Intro 2:40 Interview - shifts in discourse 6:04 p(doom)? Likelihood that AI will doom us? What counts as doom? 7:34 From dismissal to acknowledgement - geopolitical dynamics surrounding AI advancements and safety measures 9:10 How soon superintelligence? 12:34 AI risk challenges: alignment, political challenges, ethical considerations, and fitting into broader cosmic context (i.e. a society of mature civs/cosmic hosts) 15:06 Cosmic hosts, trust, game theory & utility maximisation: the need for humility in deciding which norms to load into superintelligence 17:55 The dynamics of the landscape of value & initial trajectories effecting where superintelligence ends up 22:19 Maximisation tempered with humility, and distinguishing good from bad values - dichotomy between moral realism and anti-realism 23:55 Satiability vs insatiability 26:43 Metacognition and it's location in the landscape of value 29:12 AI alignment - control vs motivation 33:34 The Indirect normativity approach to AI alignment (alternative to capability control) 34:01 Elios AI 35:00 Indirect normativity to debug metaphysical, epistemic, metaethical assumptions 40:37 Indirect normativity in practise given the kind of AI we see today 42:58 AI safety: Swiss cheese vs mathematical precision 45:57 Do we need to provide AI with metaphysical/epistemic/metaethical axioms? 50:54 Epistemic deference to AI on concepts too complex for the human mind to grasp 53:29 Does AI understand stuff? 54:56 Human-like AI now b/c training data - perhaps alien-like AI soon based on new training regimes 57:30 Base Camp at Mt Ethics, cosmic norms as metaethics developed by a cosmic host 59:39 AI Creation & the Cosmic Host - train AI to defer to cosmic norms 1:03:20 Values as causal influencers of goals, so better get the values right 1:06:36 Deep Utopia if we get superintelligence right - what is Utopia 1:13:16 Shallow redundancy vs deep redundancy - subtle values coming into focus 1:14:12 Deep utopia may involve giving up some things. Workng through repulsiveness of utopain visions 1:14:39 Going back to basic constituents of values, from there rebuilding value 1:15:36 Philosophical particle accelerator 1:19:12 Taking the scenic route to posthuman forms instead of rushing 1:28:01 Hedonics and the 5 Ringed Defence of Meaning in a plastic utopia 1:40:20 The space of subtle values is enormous, Ikagai and Japanese culture 1:49:45 Maximum novelty experienced early in life? 1:52:04 Putting the picture together 1:55:34 Outriders, and Finding Purpose Now 1:57:04 Reflections on the old era of futurology - Conclusion Elios AI https://eleosai.org/ #ai #utopia #superintelligence #agi The music at the end is 'Hedonium 1' which is guitar saturated with Strymon reverbs, delays and modulation: https://soundcloud.com/limping-fiddler/hedonium-1 Many thanks for tuning in! Please support SciFuture by subscribing and sharing! Buy me a coffee? https://buymeacoffee.com/tech101z Have any ideas about people to interview? Want to be notified about future events? Any comments about the STF series? Please fill out this form: https://docs.google.com/forms/d/1mr9PIfq2ZYlQsXRIn5BcLH2onbiSI7g79mOH_AFCdIk/ Kind regards, Adam Ford - Science, Technology & the Future - #SciFuture - http://scifuture.org