Diaspora Follow-Up: How I'd Do Social Networking...

So... after having spent so much time bashing the Diaspora project in my last post, I'd promised to provide some constructive input.

Here it is!

Rather than address Diaspora's design directly, what I'll do instead is outline how I'd go about starting a project like Diaspora myself, with the same general goals in mind.

Name:

Any good project has to have a name, and preferably one that regular Americans, as the lowest common denominator, might pronounce correctly. It also helps not to include random special characters for no apparent reason (other than to confuse search engines perhaps).

Since I'm not feeling inspired today, I'll just steal the fictional name "Gink" (from the college humor parody video on social networking --very funny stuff!). Gink would have been a pretty good name for a real social-network; easy to remember, catchy, fun, and unlikely to be mispronounced... it also makes a good verb (which is one of Google's marketing strengths too). As a word, Gink does have some definitions already, but none seem so well entrenched yet that we couldn't usurp it for ourselves.  

Defining the Mission:

There are two ways to go about Gink.

Creating fantastic social-networking software can be the mission. Here the software is your focus, and the network a by-product of success. In the wild, it would (hopefully) evolve organically and displace entrenched social-networks through just sheer awesomeness. This seems to be the angle the Diaspora is taking, and it is a viable one --Linux started out like this.

Me though, I'd set a more explicit mission. My primary goal would be to create the social-network everyone wants to use; the one that empowers users. The software would exist only as a means to achieve that mission.

Security, Privacy and Control - Organization and Politics:

Giving the users a sense of ownership of their data and control over their privacy is the driving factor for Diaspora's public interest, and is central to its design. Even with Gink's slightly modified mission, this still remains the primary requirement.

So let's start by solving this problem with Gink...

As I mentioned in my last post, these requirements have been misunderstood by the Diaspora team. Users want control over how their data is used, and a sense of ownership. They want a binding promise that the system will not abuse their privacy. The part Diaspora misunderstands is that users have no interest in taking responsibility for the technical details themselves. They don't want to manage infrastructure, nor keep up with the data themselves.

At heart, this is a just a trust issue. The requirement is political, and political requirements are best solved through political architectures, not technical ones.

Since our team has the mission of providing a social-network, not just to develop software; the easiest way to handle this whole trust thing is by establishing a formal non-profit organization to oversee the network itself.

It is this organization to which I'd give the name "Gink".

The org can make enough promises in its charter, sign-up policies, etc. to solve the trust issues. The promises would take lawyer time to word up, but the highlights would be:

• The system would not be a vehicle for advertisers; an ad-free network.


• Users would have full control of their own personal data and how it used


• Users would have the ability to examine all data the system collected about them (down to the logs) at any time.


• The network would not censor user generated content.


• The network would not be able to sell or disclose personal data (individually, or in aggregate) to 3rd parties.


• Any non-personal data that may be shared would be public to all.


• A prohibition against the transfer of ownership of the data --the network cannot be bought out by someone else who then changes the rules.


• These basic promises would be unalterable after the fact; none of that "we reserve the right to screw you with 30 days written notice" trickery.

There are probably several other promises to make too, but being a non-profit organization and having made our policy plain, unalterable and user-centric, we've pretty much solved the critical trust issues; and without any code at all.

All the code has to do is avoid leaking data all over the place.

The Gink organization would be chartered to oversee the network itself and steward the direction of the "official" branch of the software. It would also completely control the infrastructure of the network, though it need not necessarily own that infrastructure directly.

System Architecture:

In my previous post, I also went into detail about my misgivings on Diaspora's technical design. Having solved Gink's trust issues, a fully distributed system of user operated peer-server nodes is unnecessary. Such a design would not advance Gink's mission; which is just to provide a network of pure amazing!

Still though, a system like this will be very large, and a distributed architecture of some type is absolutely essential. We control the infrastructure ourselves though, so we can manage a lot more architectural diversity on the back-end.

I would not design a "do-everything" kind of distributed server, such as what Diaspora proposes. Instead, I'd break the system down into separate service roles; each having its own server design using whatever architecture best supports that particular role.

Why should my aggregation services be limited by constraints imposed by the data storage architecture?

I'm not going to go into specifics about the design here. We have not fleshed out the requirements enough to even begin thinking about more specific details yet.

Software Project:

While the Gink network is operated as a non-profit organization, there is no reason to develop the software through the same organization. There several good reasons not too though. As non-profit, employees of the org would be limited in any income potential... Not that the org can't pay them well, but there may be opportunities related to the software beyond just the Gink network itself. Separating the development from the non-profit would allow the development team to capitalize on other opportunities without unnecessary restrictions.

So I'd develop the software though a different organization, which we'll call "Ginkworks" for lack of a better name. Ginkworks would produce open, royalty-free specs for the protocols, APIs, etc. of the Gink network. It would then develop an open source reference implementation of those specs.

The version of the software in use on the Gink network itself doesn't have to be the same internally as the reference implementation. There are cases where the network, due to practical realities, might need different or extended internal implementations... as long as it conforms to the open specs, this should be allowed where needed.

This is how Google runs projects like Wave, and Microsoft did much the same with the .NET CLR and C# languages. This strategy seems to work quite well, allowing each organization to use the best political architecture to meet their different requirements in much the same way as our system architecture does.

Driving User Adoption:

In order to be successful, we need to get users to switch to using our network as their primary social networking tool. This is the hardest part of the entire plan by far. It isn't enough to just enable users to do what they can do on other networks. You have to give them reasons beyond that, or they will just stay where they are comfortable.

This is the make or break requirement, and will decide the priorities and focus for the rest of the project's design. The focus would be on user features, and all back-end considerations must align to these.

I can identify four major areas of user feature enhancement that could generate the needed attractiveness for user switch-over. There are probably others, but these are the ones that stand out to me.

Interoperability:

Here Diaspora has the right idea. To get users on the system, you have will have let them bring their established networks with them. They aren't going to give up existing communities that they've spent years building on other sites. So the system must aggregate the user's other online identities, and allow them to participate across multiple networks simultaneously.

You can see the general shape of this already, especially in the smart-phone market, with social aggregator apps, but there is a lot more that needs to be done in this area.

The interoperation needs to be dumb-ass simple; so intuitive, seamless and effortless that cross participation appears to just happen by pure-fucking-magic (for you non-programmer types, this is a real technical term).

This interoperation needs to be bi-directional and highly cooperative, not just a read-only aggregation. For example, if users like flickr better than gink's own photo systems, let them use flickr through gink as if it were a native part of our system (as much as it is possible to do so, of course).

This will probably be the hardest of the technical challenges to meet since interoperation depends on the APIs of other networks, which are beyond our control and highly variable in their capabilities.

Mobile:

So far, the other social networks are providing mobile support as a secondary add-on service. Their mobile experience is limited to a sub-set of the network's features. But with social networks in particular, mobile devices are fast becoming the primary means by which people interact.

I'd develop the entire system as if the primary clients were mobile devices, treating full desktop browsers as a secondary design consideration. There will still be a small set of features that just can't be adapted to mobile (yet) of course, but these should be limited to admin and management functions where possible.

There are also some opportunities to provide features that leverage the unique strengths of mobile devices too. The most obvious of these are location services, and this is an area of opportunity that the competition seems to largely ignore (except Google).

Neglected Social Networks:

One of the bigger failing of the major social networks is that they all fail to recognize older social network technologies, many of which are quite well established.

Email is the most obvious one. Email is very much a form of social networking; and is the oldest such mainstream service still in existence. The existing social nets all treat email as a just an out-of-band notification channel for their own convenience. At best, they provide a poor-man's "message" feature, but otherwise treat actual email like it were a plague (which it probably is).

But why duplicate, badly, email-like functionality in Gink when we can try to aggregate real email into our system? So I'd try to treat this as if it were just another social network users participate in.

Online forums are another neglected social network that needs to be pulled in, along with users' blogs, if they have one.

I might not put much effort into real-time chat networks though. I'd have to investigate this area more myself to be sure how to handle it.

Non-Social Information Streams:

Social Networking, from the user's point of view, is largely just a personalized information stream. But on most networks, one of the most popular sets of 3rd party add-on features are ones that bring in read-only subscriptions and news. Since we're aggregating so much already, I'd want to subscription service aggregation to be a core, and first-class, feature set.

These four broad swaths of user-centric feature sets are things that might generate enough user acceptances to get them to switch. Of course, you also have to do a bang-up job on the expected feature sets as well.

So before I started on the deep technical nitty-gritty of the Gink back-end architecture, we'd flesh out as much about these user features as we can. Identify how these features might work, so we can ensure that we build a framework that directly addresses the technical needs of our feature set.

And that's how I'd go about starting a social networking project.

Diaspora - Good Idea, But...

I've been watching the Diaspora project carefully for little while now. If you aren't familiar with it, the simplest explanation is that it is an open-source version of facebook where users owns their own server.


Diaspora's own explanation is not so concise, but here it is:

Diaspora aims to be a distributed network, where totally separate computers connect to each other directly, will let us connect without surrendering our privacy. We call these computers ‘seeds’. A seed is owned by you, hosted by you, or on a rented server. Once it has been set up, the seed will aggregate all of your information: your facebook profile, tweets, anything. We are designing an easily extendable plugin framework for Diaspora, so that whenever newfangled content gets invented, it will be automagically integrated into every seed.

[read the rest here]: 

I've been looking for something like this to come along for a while. There have been a few feeble attempts so far, but none have gained any traction in the wild yet... and I'm not expecting this one to either.

Here's my take on it...

The Diaspora project has gotten a lot of media attention recently including a feature story in the New York Times. The reason it has gotten so much coverage is mostly just lucky timing. They happened to launch a $10k request for donations on kickstarter.com right as privacy and security concerns with Facebook became a mainstream press issue. Being lazy, the media just latched onto Diaspora as the underdog counter-story to Facebook, rather than do something rash like actual journalism.

Diaspora, thanks to the coverage spike, has accumulated over $175,000; much more than the mere $10k they'd originally asked for. If the donations keep rolling in, they can just skip code and go straight to being famous, rich, and retired.

I'm always pretty excited by these kinds of projects though, and I love the general idea of an open source social network. And, really, who doesn't like a good under-dog?

But, despite my enthusiasm, I am not all that impressed by what I've seen and read about Diaspora so far. I don't have a good feeling about the technical approach, nor the fundamental concepts.

Fundamental Concepts:

The project's fundamentals are subtly flawed in at least two very important ways.

The first is related to the design architecture; that Diaspora will be a distributed multi-node system.

The idea of distributed systems itself is quite solid of course. Spread storage and processing out to as many machines as possible, each fully responsible for their own information domains. This is the principle upon which the web itself operates, so it is well proven.

Diaspora's flaw is the assumption that every user will "own their own server".

This isn't 1994!

Social networking users are regular people, and regular people do NOT want to manage a server node. It doesn't matter how easy you make node management!

Users can barely manage copy and paste, and you want to put them in charge of a distributed information server?

I think the team has completely misunderstood the user's requirements here. Users "want control of their data". They never said that they "want to control their data". This isn't just semantics; it is the critical distinction between what they want, and what Diaspora is proposing to build.

For several years, the trend among users, even the technical ones, has been away from user owned software. Today's users don't want to install or configure anything themselves. They use pure internet apps almost exclusively. The same is true with user data. They don't want to have to deal with keeping track of it, or backing it up, or moving it around. In short, they don't want the responsibility for all that technical crap. They just want to use the software, have the data, and have someone else to blame when things break or data gets lost.

You can see this user trend clearly in the rise of gmail, youtube, flickr, google apps, etc. Wherever possible, people are steadily switching to web supplied software, and are moving data off of their own vulnerable machines into the cloud. All the users have been asking for is a sense of control over what the cloud does with the data once it gets there.

The second conceptual flaw is more subtle. It is the simple lack of focus on end-user features. Their explanations of the user features are, at best, vague; when they bother to mention them at all.

Their focus seems confined to the technical back-end details. Even worse, they use the term "feature" to label the various plumbing aspects of the system. These are not features, these are mechanics. Mechanics are just the means to the ends; it is in the ends that the features hide.

My concern here is really that the team just doesn't seem to understand their true goal well enough to build towards it reliably. They seem focused too narrowly on "how", but not enough on "what".

The plan appears to be to design and build a framework first, and deal with user features later. They say so on their blog in some detail actually, but frameworks are not applications.

I like the iterative and pragmatic approach they describe, but they are already building a framework and still they don't provide coherent explanations of actual user facing features. They only describe them in abstract vagaries; and this is what concerns me.

This kind of thinking is tragically common in application design, and it rarely turns out well. If you don't clearly define the user features you are aiming for, in quite some detail, then you cannot know if you are building the *right* framework; one that will usefully support the user features you'll layer on top later. The framework is likely to be too generic to be very useful during the real feature development stages. When developing front-end code, nothing is harder than having to code against an abstract framework's limitations and constraints, but not getting much help with the very concrete requirements you are faced with.

There is nothing wrong with building a great framework of course, but the team is getting donations based on the promise of a better social-networking application; not just a social networking framework.

Perhaps the fault just lies in the public's misconception about what Diaspora is proposing to build.

The Technical Approach:

Distributed systems are difficult, but distributed systems with a variable number of member nodes, coming and going at will, are many orders of magnitude more so.

Then there are the problems of distributed document and NoSQL database systems, which are still rather new and immature technologies. Even the best of these still have serious limitations.

Performance is a significant concern in such a system. Nodes have to be able to locate and query other nodes for data very quickly, and a social network will need to hit a large number of peer-nodes for even the simplest user features.

And then there is the problem of fault-tolerance in such a highly distributed system. When a member node is down, slow, or unreachable, the system has to gracefully degrade, without error, and without significant disruption in the user's experience.

Any of these areas present the kind of technical challenge that even companies the size of Google struggle with, but few systems combine so many such elements as Diaspora would. Their back-end architecture is up against some of the nastiest technical challenges the field has to offer, plus competing with established players whose centralized systems do not suffer the same.

To me, the worst part is that, after having overcome the design problems, the system itself will have gained almost nothing that would add any value for the users. I can't think of anything this design allows Diaspora's users to do, that a centralized social network doesn't also provide. Even the promise of giving users control over their data is not in any way unique to this design. Facebook can give users the same control, and with little technical effort (though, it would have a severe impact to their business model).

Despite my misgivings about the architecture, I still think Diaspora should try this approach. Just because I shat all over it, doesn't mean it's wrong to give it a try. If nothing else, they might advance the applied sciences of distributed system design. Even if Diaspora doesn't succeed with a distributed social-network, someone eventually will.

Whether real users will care or not is an entirely different matter. With the team putting so much effort into the complex back-end architecture, they will likely not have the resources left to do a bang-up job on the front-end features; at least not quickly enough to still be relevant.

The front-end features have to be amazing, or the system will not have much user interest. They are up against well-established competition that has had a lot of time to evolve and design their UIs. Creating a compelling experience to compete with them will be quite a daunting, and non-trivial, technical challenge all by itself.

Next...

It isn't fair to spend this much text bashing someone else's project without at least doing them the courtesy of providing constructive suggestions or alternative ideas. But, I'm going to save the constructive ideas for another post.

How Apple Will Lose the Mobile Computing Race

I've heard several people say that "Apple is the new Microsoft". These people are wrong; dead wrong.

Apple is the new Apple. They've not changed in the last 20 years, not even a little bit.

Google is the new Microsoft, and Apple is going to lose the mobile device market to Google the exact same way it lost the desktop market to Microsoft 20 years ago.

How is this going to happen? First you have to know the history.

Let me tell you a little story...

Back in the 80's there were two companies; Apple and Microsoft. Suddenly Apple comes out with a magic new product way ahead of the competition. Enter the Macintosh. The Mac was guaranteed to change personal computing forever, and everyone knew it before the Mac even hit the shelves. There was media hype and gobs of nerd excitement well in advance of the Mac's debut. And the Mac delivered spectacularly on all of its promises.

But Apple insisted on near complete control of the entire platform; hardware and soft. They were terrified that someone else might come along and spoil the pure awesome.

Apple used deliberately unusual hardware, and controlled it with an iron fist. Only Apple and a few licensees were allowed to make Mac hardware. The 3rd party accessory market was as restricted as Apple could make it too.

Programmers could make applications for the Mac of course, but Apple maintained tight control over how applications used the hardware and interacted with the OS. Developers played in the sandbox, but it was Apple and select partners that created the majority of useful Mac software.

Apple was also territorial about who could sell their precious. Macs were distributed mostly through select retail partners. Anyone else that wanted to sell a Mac paid full retail price --which made it very hard to compete with Apple's preferred retailers.

Because Apple had tight control end-to-end, the Mac was a very stable and reliable computer. Most of the software was also high quality, which made the Mac a joy to use.

On the other side, you had Microsoft. They weren't really a hardware company though. They concentrated on making a decent operating system and left the hardware to other companies that specialized in that sort of thing.

There was a lot of compromise and cooperation between Microsoft and the hardware makers though, of which there were many. They created a large market with a wide range of different systems, each with different capabilities, quality, and prices. Microsoft just did what they could to make sure that Windows worked reasonably well with whatever crazy-ass hardware the other guys came up with.

Microsoft made a decent enough OS, but what they did really well was make programming languages, compilers, and development tools --all that stuff you need in order to make software for the regular folks. Microsoft made it easy to develop applications for Windows, and they stayed out of the way as much as possible. They didn't put up a fight when developers released software to compete with Microsoft's own, and they didn't stop developers from extending the operating system itself in new and unusual directions.

Microsoft was later than Apple in coming out with a good GUI based OS. Windows wasn't too pretty nor much fun compared to the Mac; especially at first. But PCs were cheaper, and came in a variety that fit different people's needs and budgets. The most important thing was that a Windows PC could do everything that a Mac could, though not always with the elegance of the Mac. People often preferred a Mac, but they could (and did) settle for PCs instead.

In the end, what killed Apple the first time around was Apple's own paranoia. All by itself, Apple couldn't evolve the Mac hardware nor the OS fast enough to keep up with a young and rapidly changing personal computer market.

All Microsoft had to do was try and keep up while other companies raced forward with new ideas.

Now jump forward 20 years.

There are two companies; Apple and Google. Suddenly Apple comes out with a magic new product way ahead of the competition. Enter the iPhone. The iPhone was guaranteed to change mobile computing forever, and everyone knew it before the iPhone even hit the shelves. There was media hype and gobs of nerd excitement well in advance of the iPhone's debut; and the iPhone delivered spectacularly on all of its promises.

But Apple insists on near complete control of the entire platform; hardware and soft. They are terrified that someone else might come along and spoil the pure awesome.

Apple uses deliberately unusual hardware, and they control it with an iron fist. Only apple is allowed to make iPhones, and the 3rd party accessory market is as restricted as Apple can make it.

Programmers can (and do) make applications for the iPhone of course, but Apple maintains near complete control over how those applications use the hardware and interact with the OS. Developers are allowed to play in the sandbox only if they use the specific tools that apple permits. Apple and a few select partners make the majority of the useful iPhone apps. Most developers that make apps to competes with Apple's own are denied access to the app store; which is the only way users can obtain apps for the iPhone. Apple will even deny an app if they just don't like what it does.

Apple is territorial about who sells their precious. The iPhone is (currently) only distributed through one network in the U.S., and apple exerts enormous control over how that network does business. AT&T had to wait to roll out 3G services until there was an iPhone that also supported it. There were other mobile phones sold by AT&T that had  3G support as much as a full year before, and AT&T had deployed the network hardware long before Apple's 3G iPhone came out.

Because Apple is in such tight control, the iPhone is a very stable and reliable smartphone. Most of the software is also of high quality, which makes the iPhone a joy to use.

On the other side, you have Google. They aren't really a hardware company though. They make a good mobile operating system, but have left most of the hardware in the hands of companies that specialized in that sort of thing.

There is a lot of compromise and cooperation between Google and the many hardware makers. They have created large market with a wide range of different Android phones with different capabilities, quality, and price. Google just does what it can to make sure the Android OS works reasonably well with whatever crazy-ass hardware the other guys come up with.

Google makes a good mobile OS, but what they do really well is provide cloud services --services developers can use to make useful mobile software for the regular folks. Google makes it easy to build applications for Android, and supply much of the cloud services free of charge. They stay out of the way as much as possible, but give developers a way to distribute software through a centralized app store, with very few restrictions. Apps can also be obtained through independent channels too. Google doesn't put up a fight when developers release software to compete with Google's own products, and they don't stop developers from extending the Android OS in new and unusual directions.

Google was later than Apple with a smartphone. Android isn't quite as pretty compared to the iPhone; though it is rapidly getting there. Android phones are cheaper, available for nearly any network, and come in a wide variety of forms to fit different people's needs and budgets. And the important thing is that an Android Phone can do anything that an iPhone can; sometimes even with the same elegance as the iPhone. People often prefer the iPhone, but they can (and do) settle for Android phones instead.

In the end, what will kill Apple the second time around is Apple's own paranoia. All by itself, Apple cannot evolve the iPhone fast enough to keep up a young and rapidly changing mobile computing market. Their treatment of developers is pushing them to other platforms fast.

All Google has to do is try and keep up while other hardware and software companies race forward with new ideas.

This same exact pattern of behavior is repeated with the new iPad.

The situations are so similar it is almost insane that Apple can't see their own doom coming.

There are some major differences between now and 20 years ago, but none of them bode well for Apple.

Google is in a much more competitive position than Microsoft was 20 years ago, and Android is a much better product to pit against the iPhone than Windows was against the Mac.

Perhaps the biggest difference between now and then though is that Apple wasn't crapping all over their software developers back then. Apple did limit developers in some ways, but never in the arrogant and insulting way they do now... and especially not through any legal or contract trickery like the iPhone developer agreement.

Apple is also much more blatantly motivated by pure monetary greed. Twenty years ago, it felt like the Mac more about a vision of excellence, making a better tomorrow, being different, and enabling new and wonderful possibilities. The iPhone today though is clearly just a cash cow. Apple's every legal and marketing move is so transparently profit motivated that it doesn't even fool kindergarteners.

There is no greater vision behind the iPhone like there was the Mac; behind the iPhone it's just dollar signs, abused developers, and shackled customers.