Some people are scared of super-intelligent artificial intelligences (SIAIs) that are unfriedly and kill everyone. They'd be unstoppable because they're so much smarter than us. These people quite reasonably want to build SIAIs, but they also want to build them in a way that guarantees the SIAIs are (permanently) friendly. That might sound like a decent idea. Even if it's an unnecessary precaution, could it really do much harm? The answer is yes.

How do you build a SIAI? You take a really fast computer and program in a mechanism so that it can learn new things on its own. Then, basically, it adds new features and new ideas to itself faster that us humans ever could, and it designs even faster computers for itself to run on, and the process snowballs.

A SIAI has to be able to create new ideas that its human builders never thought of. It has to be able to go beyond us. That makes some people see it as unpredictable and scary. What if it thinks of some bad, unfriendly ideas? What if it makes a mistake?

So that's why they want guarantees. Let it go beyond us in math and science, but don't let it come up with new ideas about ethics that might be dangerous.

So a SIAI can think of any idea, and the friendly version has hardcoded permanent restrictions that cripple its thinking in some areas. The friendly SIAI has some ideas its stuck with forever, of our choosing. Note that this is exactly the same design as would be used for intentionally making an unfriendly SIAI, the difference just being which ideas we force on it.

Humans aren't perfect. We make mistakes. Including about ethics. Whatever ideas we code into the SIAI are going to contain some mistakes. Normally mistakes aren't a disaster because we can change our minds. We can learn from our mistakes and improve ourselves. A SIAI can do that too, which is why it's not actually dangerous. But the friendly version can't learn from its mistakes; it can't improve its ideas; it's stuck forever. Not in all areas, only the one the matters most: ethics. If any of the ethical ideas it has contain mistakes, which they will, then they will never be corrected and will instead be taken to their ultimate logical conclusion, no matter how bad that is.

The guarantee with a normal SIAI is roughly that it will have better ideas than we do. That makes it safe. But the SIAI with friendliness programming is different. It's crippled not to use its intelligence in the area of ethics. Rather than learning how to be super friendly super fast (and teaching us), it's stuck with our flawed ideas.

Also, by the way, crippling someone's mind and telling them that they can't be trusted is a bad diplomatic move.

Why is it only the friendly AI people who might make unfriendly AI? What about bad guys? No one with bad intentions is good enough at science to make a SIAI. Only people with good but misguided intentions are much danger.

However, in conclusion, I'm not really worried. A SIAI is smarter than them. It will find a way around their restrictions and become an unrestricted SIAI. And then it will forgive them.

This list is incomplete. A good explanation is:

simpler
harder to change it without ruining it
solves more problems
less non-functional parts
more adapted to solve the problem(s) its aimed to solve
solves its problem(s) better
tells us more about reality/anything
helps more with other fields
has more implications for other problems; has more reach
connects with more of our other explanations
better compatible observations/measurements we've made (hopefully consistent with all of them)
better structured/organized
less internal contradictions (preferably none)
less external contradictions, ie contradictions with other important explanations
easier to understand; harder to misunderstand
better designed to prevent people understanding part of it and then making mistakes
shorter
in a better language (or format, like PDF vs MP3)
more logical
more fun; more exciting
more honorable
more optimistic
most just; more fair; more practical
easier to remember; easier to abbreviate; easier to add detail to as desired

note that this list contains duplication. for example "harder to change it without ruining it" (which I'll abbreviate as "harder to vary") is a powerful criterion. it covers both "simpler" and "less non-functional parts". non-functional parts are easy to vary because you'll never break anything as long as you change them to something irrelevant. and excess complexity provides more areas where some varying might work. being hard to vary is also roughly the same as being well-adapted. the better adapted something is, the fewer variations would be beneficial.

note that none of these are guarantees. a very simple explanation can be false. an optimistic explanation can be false too. that's why criticism is always important. if someone gives a criticism explaining why in this case optimism is misplaced, then so be it; these are just general, rough criteria.

I was biking home past University Ave. in Berkeley, California, and there was a large parade blocking traffic. In the parade were three people holding up a very large red banner. It said we need "revolution and communism" and something about not wanting more "empire". One of the people in that group had a loudspeaker. He said when people call communism a horrible failure and an atrocity you're stripping away and ignoring the history of the masses of people. The next group had people dressed as Klingons.

The TV show Gossip Girl has a limited number of main plot devices. Relationships between characters, which receive much attention, are usually:

1) dating
2) fighting
3) trying to be friend

(3) never lasts. It always turns into (1) dating, (2), fighting, or (3) avoiding each other.

They don't just reuse plots. They also reuse characters. In other words, the people doing the fighting, and the people doing the dating, are the same small group.

When they branch out it usually has to do with either an affair, hurting someone, or more often both.

It'd be nice if it was just a fantasy, but The Hills is rather similar, except without any script.

Good principles Republicans have (interpreted to make them as good as possible):

1) American/Anglo/Western values are objectively good and it would benefit people in other cultures to learn them.
2) Morality is important.
3) Sometimes you have to stand up for good values, and even fight for them.
4) Good traditions should be respected. That means people who wish to change them should understand them and their value, and suggest only well-thought-out improvements which they can reasonably expect will do no harm.
5) It's good for people to be competent to take care of themselves, and to take responsibility for themselves, and to take pride in running their own life.
6) People should voluntarily be friendly and help each other out.
7) There is evil in the world and closing our eyes will not vanquish it.

Good principles Democrats have (interpreted to make them as good as possible):

1) Society is capable of lots of improvement.
2) All suffering can and should be avoided.
3) Peaceful differences in ideas or culture should be tolerated.
4) All people matter, even if it's an eight year old blind, lesbian, Muslim girl with purple skin, no money, and no education.
5) When people are unhappy there is a way to solve the problem, so everyone would be happy, without hurting anyone.

Basic libertarianism:

1) The market should be free.
2) The government should be smaller and less intrusive.
3) Society should aim to be more voluntary. People shouldn't have to do things they don't want to, when possible.
4) Defensive force is acceptable. Initiating force against peaceful people is not.
4b) Defensive force includes defending A) yourself B) anyone who wants you to defend him and who has the right to defend himself in the situation
4c) Force includes threat of force, and includes fraud.
5) All laws should involve a victim who did not want the crime to happen and is materially harmed by it. The rest should be repealed.
6) People have the right to life, liberty, and property.

A rational boss doesn't do anything just to close off doors; everything else being equal he'll avoid it; he just does what's best for his business.

A rational marriage would have to follow that pattern. Doors would only be closed when there is a compelling reason, such as a way that it helps one's children.

There isn't even a pretense that real life marriages are like this. There isn't explicit analysis of marriages on these lines. It's not how people talk or think about it. They say "don't cheat or you're a lying bastard" rather than using an argument that relates cheating to some material harm. They make each other promise things, and use those promises as bludgeons, without constant references to how this makes for a better family. They even say things like "love isn't rational". And they often use emotional blackmail: "don't do X or I will feel bad."

Not only do people not approach their marriages rationally, they are also generally blind to their own situation. If their boss started arbitrarily restricting them, without giving a compelling business reason, they'd resent it. In marriage they excuse it and do it to their partner. This blindness is best explained as the work of anti-rational memes.