A simple, distributed micro-blogging platform

In the following I describe a micro-blogging platform built on the well-known, widely used, free and open Atom Syndication Format. The format is already used by most blogging platforms for syndication of blog contents to a multitude of aggregators and feed readers.

Introduction

The platform I describe involves a particular use of the well known Web content and metadata syndication format called Atom, combined with the PubSubHubbub protocol for realtime content syndication, and feed aggregators in the form of custom client apps for creating and consuming content.

The concept presented here is neither new or original, as it has been explored many times before, see for example this video for a discussion about the technology and its potential implications.

Feeds, hubs and aggregators

The central parts of this micro-blogging platform are feeds, hubs and aggregators: Feeds have the content. Hubs distribute it. Aggregators collects it and present it to users.

Feeds are the lists of entries produced by people or machines. Feeds are stored and served as Atom XML text files linked in a chain to give the full list of entries, as described below. Anyone wanting to publish can either self-host, or use a shared host (e.g., a web app that incorporates hosting of user feeds and feed aggregation).

A feed subscriber initially pulls the first part of the feed in the conventional way, i.e., by requesting it from the feed server. The subscriber then inspects the feed, and if it references a hub, the subscriber can subscribe to the feed URL on that hub.

If the subscriber runs a server (or access the feed through a web app which acts as the feed subscriber), the hub can directly notify it when any of its subscribed URLs have updated.

Publishers expose their contents as feeds, in the format described below, with the (optional) inclusion of a hub reference. A publisher may post notifications to the referenced hub whenever they publish something. Thus, when a publication event occurs, the publisher calls its hub and the hub call its subscribers.

It is also possible to publish without a hub, in which case hubs like Superfeedr may start to frequently ping the feed server to check for updates to the feed.

Hubs may distribute feeds from several publishers, and publishers can have several hubs distributing their feed.

Feed aggregators serve as entry points for users, for consuming the micro-blog content. This may be web apps, like Google Reader, or native apps on mobile devices or computers. Aggregators should preferably have an integrated publisher feature, for creating new entries in user's feeds, and notifying hubs distributing the feeds.

Feed format

The feed format described in the following is a particular use of the Atom feed syndication format, making it possible to view the feed in just about any feed viewer, still allowing for the micro-blogging characteristic features we have come to expect from commercial micro-blogging services.

A feed expose entries stored in batches of optional size (but for efficiency they should preferably be limited to, say, a max. feed file size of 10kb). Each batch of entries contains a link to the previous batch of entries (if such exists).

The feed files are served as Atom XML encoded text, with the following minimal set of elements (described in more detail below):

  <?xml version="1.0" encoding="utf-8"?>
  <feed xmlns="http://www.w3.org/2005/Atom">
    <id>http://example.com/example.atom</id>
    <link rel="self" href="http://example.com/example.atom" />
    <link rel="hub" href="http://example.com/hub/example" />
    <title>example</title>
    <author>
      <name>John Appleseed</name>
    </author>
    <subtitle>A short bio goes here.</subtitle>
    <icon>http://example.com/images/john.jpg</icon>
    <updated>2012-07-02T08:05:36Z</updated>

    <entry>
      <id>http://example.com/example/3</id>
      <title>There is a need for an open
        micro-blogging platform. Not closed, as
        twitter, facebook and google+.
      </title>
      <content>The platform doesn't, however, have to be
        limited to micro-blogging, but may allow for
        longer posts as well.
      </content>
      <updated>2012-07-02T08:05:35Z</updated>
    </entry>

    <entry>
      <id>http://example.com/example/2</id>
      <title>@example and also follow Gruber's twitter feed
        @daringfireball
      </title>
      <content></content>
      <updated>2012-07-02T08:03:46Z</updated>
      <link rel="via" title="reply">
        http://example.com/example/1
      </link>
      <link rel="related" title="example">
        http://example.com/example.atom
      </link>
      <link rel="related" title="daringfireball">
        https://twitter.com/daringfireball
      </link>
    </entry>

    <entry>
      <id>http://example.com/example/1</id>
      <title>More people should listen to @thetalkshow.
      </title>
      <content></content>
      <updated>2012-07-02T07:59:32Z</updated>
      <link rel="related" title="thetalkshow">
        https://twitter.com/thetalkshow
      </link>
    </entry>

  </feed>

Feed elements

The feed id element is a required unique URL for the feed. The feed also has to include a link to rel="self", and can also include rel="hub" links referencing hubs syndicating the feed.

The feed title element shall be the display name of the micro-blog, e.g., the handle of the blogger (like a Twitter handle). This name (or the first part of it, if the name contains spaces or non-alphanumeric characters) should be used for @-replies and @-mentions, accompanied by the appropriate rel="related" link to the feed.

The optional subtitle element is for a short bio of the blogger.

The feed is required to have the author element, which have to provide the author name, and optionally the email and/or a personal link. The author name can be the full name or just repeat the display name given in the title.

The optional icon element is for linking to an avatar image, representing the micro-blog. The optional logo element can link to a full size image.

Entry elements

Each entry is required to have an id element, which is a unique URL identifying the entry within all entries contained in the particular feed.

The title element of the entry contains the micro-blog content. Links to other feeds may be referenced in the title text by an "@" sign followed by a string of alphanumeric characters, ending with either a space or another non-alphanumeric character. The "@" referenced feed URL should be given as a "related" link in the message entry block.

If the entry is a reply, i.e., a part of a micro-blog discussion, the entry being replied to can be referenced with a ref="via" link having title="reply".

Another use of ref="via" links, similarly indicated by the link title, is title="repost" for (full or partial) copying an entry from another micro-blog.

The content element may contain a longer post, supplementing the title (micro-blog contents) with a more traditional weblog-like content. But, if only title and no content element is included in the entry, the Atom syndication spec requires an ref="alternate" link to contents of the entry. The spec. has other guidelines, like the type attribute on the content element, suggested other common elements to include in the entry and feed elements, etc.

Getting to the micro-blog experience

For the user to get an experience like that available on other micro-blogging platforms, developers of feed aggregators have to make delightful looking apps, which allow the user to subscribe to any feed and present a coherent stream of messages sorted by date and time. Much like Google Reader, but with a layout focussed on displaying micro-blog content, and with an integrated publishing feature.

A significant part of the success of micro-blogging is the limit on post length, making it possible for people to consume a combined stream of many micro-blogs. To allow for a similar experience, publishers should self-impose a character limit on the entry title lengths.

Other liked micro-blogging features like "starring", syncing and feed discovery may be implemented in each feed aggregator app, or be external services. This opens for a whole ecosystem of web services interacting with each other, like in the early days of Twitter, before their platform was severely limited for third party developers.

It will be up to feed aggregators, or third party services, to parse feeds and collect information like @-mentions and @-replies. Here, feed aggregators/third party services become responsible for performing spam filtering and blocking of malicious content. Users consuming the content will naturally prefer aggregators, which have a balanced level of filtering, so widespread censorship should not prevail.

The feeds that a particular user subscribes to may be published as an outline in the OPML format, from which the "following" and "followers" counts can be determined.