Thread by @JonAskonas

Google's motto is "Don't be evil," but human society make this impossible - you can't contain ideas.

And so you get left-liberal utopian ideas ("the selfish ledger") that can and will be easily, naturally used to authoritarian and dystopian ends.

theverge.com/2018/5/17/1734…

The English School of International Relations has recently revived Trotsky's notion of "Uneven and Combined Development" - global interconnections mean that we cannot isolate developments in one social context from their repercussions into others.

…nandcombineddevelopment.wordpress.com

So it matters, for instance, that Western tech companies are intensively developing the manipulation of human social interaction and learning systems (for their own internal reasons) at the same time that post-Communist semi-authoritarian states are seeking soft social controls.

I don't know how Westerners should think about this. But it's disturbing to think that the Big Brains at Google put together a concept video about manipulating behavior through our knowledge of epigenetics and memetics while thinking this would be used towards good ends.

"Here's a sophisticated system using machine learning, cybernetics, and communications theory designed to subtly manipulate the user into believing certain things or performing certain roles. We think people will use this in order to reduce their carbon footprint."

In reality, uneven and combined development means that the same algos that turn American Instagram users into #aesthetic zombies turns Sri Lankans into genocidal extremists. Anything to boost engagement. Literally anything. nytimes.com/2018/04/25/wor…

Ironically, evolutionary theories of memetics ("The Selfish Ledger") PERFECTLY explain this behavior, as the algorithms create environmental pressures leading users to adapt through learning (what the algo wants) or selection (newsfeeds picking users who boost engagement).

TLDR: We have built a memetic, cybernetic, consciousness-hacking engine and there is no off-switch. The people who built it were incredibly naive about how powerful actors would use it, even though there were lots of people warning them. Hold on to your butts.

cc: @Aelkus @rejectionking @IRHotTakes @profmusgrave @SevaUT @DouthatNYT