> To the extent that the reason we're worried about motivations in the first place, it’s because we think a balance of power which includes at least one thing with human-descended motivations is difficult. To the extent that we think that's the case, this seems like a big crux that I often don't hear people talk about. I don't know how you get the balance of power. Maybe it’s just a matter of reconciling yourself with the models of the intelligence explosion.
There's definitely an inherent contradiction here, between inclusiveness and wanting to personally (or at least for some segment of humanity to) be included in an intelligence explosion.
Sure, in an ideal world we're rich enough in resources we can give non-FOOMed base humans a Homo Sap habitat to putter around in, being idiots, and protect the resources and energy it uses from more greedy optimization entities.
But I can't see it going any other way than turning into a situation of anyone who wants to participate in Economics 2.0 or "real relevance" in the future light cone is going to have to be upgraded to above a certain intelligence and capability floor (whether their initial origins were biological or technological), which is going to be far, far above base humans.
Relevance in the future light cone isn't going to be accessible to the big tent that you want to create, because definitionally, many in the big tent will opt out of relevance by choosing to remain dumb and limited, and even in a completely benign and power-balanced world, progress and machinations among higher intelligence entities will proceed at a pace that leaves them so far behind so quickly that they can never catch up with the resources they have, even if they wanted to. They'll be the equivalent of an endangered and protected worm or mollusc in a backwater swamp somewhere, forever.
And sure, conservation is nice, especially if it costs you little. But I'm not sure it would count as success in terms of "human motivations" or even "human descended motivations" influencing the future light cone.
After all, how much more powerful and capable are you now than when you were 8 years old? Yet you don't go around eating nothing but ice cream, never brushing your teeth, playing video games all day, etc. The things you dearly wanted as an 8 year old aren't relevant or interesting to you any more, even though you could easily do one or all of them now. I think a similar effect is almost certain to happen to human-origin FOOM-ed intelligences too.
> To the extent that the reason we're worried about motivations in the first place, it’s because we think a balance of power which includes at least one thing with human-descended motivations is difficult. To the extent that we think that's the case, this seems like a big crux that I often don't hear people talk about. I don't know how you get the balance of power. Maybe it’s just a matter of reconciling yourself with the models of the intelligence explosion.
There's definitely an inherent contradiction here, between inclusiveness and wanting to personally (or at least for some segment of humanity to) be included in an intelligence explosion.
Sure, in an ideal world we're rich enough in resources we can give non-FOOMed base humans a Homo Sap habitat to putter around in, being idiots, and protect the resources and energy it uses from more greedy optimization entities.
But I can't see it going any other way than turning into a situation of anyone who wants to participate in Economics 2.0 or "real relevance" in the future light cone is going to have to be upgraded to above a certain intelligence and capability floor (whether their initial origins were biological or technological), which is going to be far, far above base humans.
Relevance in the future light cone isn't going to be accessible to the big tent that you want to create, because definitionally, many in the big tent will opt out of relevance by choosing to remain dumb and limited, and even in a completely benign and power-balanced world, progress and machinations among higher intelligence entities will proceed at a pace that leaves them so far behind so quickly that they can never catch up with the resources they have, even if they wanted to. They'll be the equivalent of an endangered and protected worm or mollusc in a backwater swamp somewhere, forever.
And sure, conservation is nice, especially if it costs you little. But I'm not sure it would count as success in terms of "human motivations" or even "human descended motivations" influencing the future light cone.
After all, how much more powerful and capable are you now than when you were 8 years old? Yet you don't go around eating nothing but ice cream, never brushing your teeth, playing video games all day, etc. The things you dearly wanted as an 8 year old aren't relevant or interesting to you any more, even though you could easily do one or all of them now. I think a similar effect is almost certain to happen to human-origin FOOM-ed intelligences too.
Hell yea!