happy chaos freedom machine

 Virtualization is hot (or was, until VMWare’s stock took a beating — nevermind, it’s still hot in IT departments, where it counts), Web 2.0 is hot, and “Cloud computing” is hot. Together these trends must be sending cold chills down the collective spines of the giants of the IT industry.  This article on Forbes.com captures the challenge to the dominant business model.

Faced with a massive computational job, Zillow.com turns to Amazon’s EC2 service instead of buying their own servers and deploying commercial software. Instead of spending millions and taking 6 months to deploy the system and complete the job, they took 3 months and spent $50,000. They turned on 500 virtual servers in Amazon’s cloud, presumably installed their favorite LAMP stack on them, and cranked it up. The economic implication is a giant below-the-waterline hole in the S.S. Enterprise IT.

 The work of Duncan Watts, described in this article at Fast Company, has truly profound ramifications.

In the past few years, Watts–a network-theory scientist who recently took a sabbatical from Columbia University and is now working for Yahoo NASDAQ:YHOO –has performed a series of controversial, barn-burning experiments challenging the whole Influentials thesis. He has analyzed email patterns and found that highly connected people are not, in fact, crucial social hubs. He has written computer models of rumor spreading and found that your average slob is just as likely as a well-connected person to start a huge new trend. And last year, Watts demonstrated that even the breakout success of a hot new pop band might be nearly random. Any attempt to engineer success through Influentials, he argues, is almost certainly doomed to failure.

In essence,  what Watts is proposing is that trends, fads, adoption of any idea or product or behavior or other socially-transmitted trope depends not on the vector finding a highly connected and influential individual to adopt and promote it, but rather random “luck.” If this is true, the “Tipping Point” type theories of social trends and marketing are complete fabrications, the product of the human proclivity to retroactively create rationale for events.

If this is true, this not only means that the assumptions behind viral marketing and influencer-based marketing are false, but that our entire approach to complex systems are also flawed. It may in fact be that scientific predictability is limited to very simple systems. Drop an apple, it falls. Systems that cross some threshold of complexity in terms of the number of nodes, of actors, of variables become resolutely uncertain. Drop a leaf in a creek, try to predict what shore on which it will land. You can’t. The stock market, despite  all our various attempts to predict its behavior, is random. The economy — random.

Perhaps more psychologically important, especially in our capitalist society, is that fame and fortune are random. In one of Watts’ experiments, he created mini-societies of musical taste. Rankings were socially influenced and songs rose to the top, resulting in Pareto power distributions of rankings. All as one would expect, but what was not expected was the eight parallel taste societies he created all chose different songs to make famous. The implication? If one could rewind time to before an individual met with success and then play the events all over again, there is no guarantee the same individual would be successful again. Another individual could just as easily step into the shoes of any success story.  This is fundamentally at odds with the mythos of the capitalist system - that talent and hard word equal success. They may improve your odds, but in the end, it’s all just a roll of the dice.

So why the insistence that there is some reason why a product or web site or person becomes successful while another does not? To me this seems like an example of a primate bias - something in our basic psychological building blocks abhors randomness — if something happens there must be a reason for it. It makes sense, in evolutionary terms — how else would we have begun making tools, harnessing natural phenomena,  and mastering our environment without the nagging suspicion that things happened for some reason? Is there a term for when a beneficial evolutionary trait is taken to a point that is detrimental?

 Wired has this great story about the slow percolating return of vinyl records:

Vinyl May Be Final Nail in CD’s Coffin

As counterintuitive as it may seem in this age of iPods and digital downloads, vinyl — the favorite physical format of indie music collectors and audiophiles — is poised to re-enter the mainstream, or at least become a major tributary.

Interestingly, it is the “download and vinyl” release format that Radiohead just used that might actually kill CDs. Also interesting is the mention of the “loudness wars” — the urge to compress the audio when mastering to CD so that the volume can be increased — degrading the fidelity of the sound on CD. You can’t twiddle that knob when mastering to vinyl, so the sound ends up better. Amazing example of how it takes limitations to get a human to use creativity and produce quality.

Cory Doctorow nails my skepticism about the current crop of social networking site: How Your Creepy Ex-Co-Workers Will Kill Facebook

This is somewhat off-topic (as if I actually have a topic to stay on) but Stephen Colbert is a hero of mine, and this article is really quite fascinating. I agree with Jon Stewart’s comment in the article, about Colbert interviewing subjects while in character:

“[The whole show] depends on Stephen’s ability to process information as this other person,” says Stewart. “I’ve seen talk-show hosts who can’t do that for real.… And then you watch Colbert and it’s like the first time you use broadband: ‘How the fuck did that happen?’ He’s rendering in real time. He’s basically doing his show in a second language.”

The Man in the Irony Mask: Fame & Scandal: vanityfair.com

 Over at Mutating Pictures, Philipp Lensman started wit a pool of 1000 randomly created images, and is asking people to rate how much each image looks like a face. He put the site up yesterday, and look how far it’s come.

There’s also a forum at Google Blogoscoped, where technical details and the implementation  are being discussed.

 Fascinating article in  New Scientist Tech  about the increasing utility of “evolutionary algorithms” across a wide range of engineering disciplines. Engineers create a simulation of the context of the problem for which they are trying to create a solution, then allow the components of the solution to be rearranged randomly and tested by computer software.

Thanks to Moore’s Law, we now have sufficient computational power to create very complex simulated environments and then to test very large numbers of design variations. The randomness of the designs ensures novelty that might not occur to a human engineer due to lack of time, or, even more interestingly, human bias.

Unsurprisingly, the use of evolution in engineering is causing controversy. The main criticism seems to be that:

Critics argue that the technique may lead to designs that can’t be properly evaluated since no human understands which trade-offs were made and therefore where failure is likely.

This seems like a specious argument — by definition points of failure are difficult to predict. If they weren’t, we would be able to avoid them, and they would no longer be points of failure. The real discomfort caused by “EA” (the unattractive acronym that the unwieldy “evolutionary algorithms” is shortened to) seems rooted in anthropocentric grieving. The article quotes  the inventor, James Dyson, as saying,

“Evolutionary algorithms will mean the end of those exciting stories abut how people made great inventions by accident,” he says. “Human ingenuity and intuition should remain crucial in making a success of any product.”

In short, the controversy is the same as that caused by biological evolution — a defensive, emotional reaction to the sense that man is being usurped as the pinnacle of progress and the ultimate product of a universe created for his benefit. Get over it.

I suppose this was inevitable, but it’s still fascinating to see tha, as the New York Times suggests, outsourcing works, so India is exporting jobs.

Think I’ll buy this article: IngentaConnect Disagreement and the Stock Market, which attempts to catalog the types of “disagreement models” that describe patterns like

…the tendency for stocks that have had unusually high past returns or good earnings news to continue to deliver relatively strong returns over the subsequent six to twelve months (and vice-versa for stocks with low past returns or bad earnings news); [and] longer-run fundamental reversion—the tendency for “glamour” stocks with high ratios of market value to earnings, cashflows, or book value to deliver weak returns over the subsequent several years (and vice-versa for “value” stocks with low ratios of market value to fundamentals)

This sounds like the marriage of behavioral economics with investing - an ambitious and intriguing project. Accurate models for how investors’ beliefs about firms and their stocks disagree and interact in the market might actually lead to accurate predictive abilities.

Goldman Takes ‘Private’ Equity To a New Level - WSJ.com: “It represents the latest step in the creeping exclusion of individual investors from a growing proportion of financial-market activity. For instance, giant private-equity firms are busy buying public companies and delisting them from stock exchanges. The growing importance of hedge funds — which are generally limited to wealthy investors, institutions and endowments — also excludes individuals.”

I see this trend as a kind of counter-insurgency action, a move by the big money against the retail investor and the increasingly level playing field engendered by the Internet and ETFs, among other things.

http://today.reuters.com/misc/PrinterFriendlyPopup.aspx?type=topNews&storyID=2007-05-31T200325Z_01_N29313303_RTRUKOC_0_US-USA-ALGORITHM-STOCKS.xml

cult of the amateur
http://www.nytimes.com/2007/06/29/books/29book.html?ex=1340769600&en=7667d825909fca51&ei=5090&partner=rssuserland&emc=rss