Dancing About Architecture: Microfeed for "Using P2P for RSS Distribution"

Using P2P for RSS Distribution

Tue, 16 Dec 2003 10:20:55 CDT

As more and more people wake up to the idea that a fundamentally polling-based RSS distribution model might have some "scaling challenges", different ideas to handle this come to the fore. Here's an interesting model: using the BitTorrent peer-to-peer infrastructure to distribute RSS feeds. BitTorrent is a cool bit of engineering that solves the "Tragedy of the Commons" problem that plagues many p2p protocols: you gotta pay (i.e., offer some upstream bandwidth) to play (download). The Java-based client Azureus is my client of choice, but it's really the protocol that's the smart stuff here. Getting back to how this works with RSS, I personally think this is overkill for the scaling issues at present: I think getting the clients and server-side aggregators/publishers to agree on a push protocol and transport is a more pressing concern. I think the P2P idea is a good thing to keep in our pocket when RSS starts getting bulkier. BitTorrent and RSS Create Disruptive Revolution...

Comment from Peter

Wed, 20 Oct 2004 14:51:40 CDT

I also think the BitTorrent angle for feeds is not (yet) necessary. But what you guys could do: change the syndication frequency dynamically.
E.g. if your polling of a source feed shows it only changes twice a week on average, you could add an updatePeriod/updateFrequency to the burned feed that tells RSS aggregators to update daily instead of hourly. http://web.resource.org/rss/1.0/modules/syndication/
This would drop the traffic you guys are getting (to 1/24). You could also decide to only poll the source feed once per X hours instead of every 60 minutes.

Comment from Randy Charles Morin

Thu, 10 Mar 2005 07:24:56 CDT

Oh! Oh! RSS is broken again!