Published on One Big Library. (http://onebiglibrary.net)
A simple, old design for widespread blog mirroring
By dchud
Created 2007-02-19 14:36

(Remembered this after last week's interview, and thought I would re-post it in case it makes any sense to anyone, assuming a few more interested people might be watching just now.)

Back at the Access 2004 Hackfest I worked with a few folks to design up a design for widespread copying and mirroring of blog content [1] for distributed-copy and "preservation" purposes. I think we came up with something that could definitely work, at low cost, on a wide scale. Rethinking it today I'd substitute Atom for RSS, and maybe rethink using METS [2] (perhaps instead just using Atom for both purposes).

It's a pretty simple idea: you extend an aggregator system to "archive" entries posted each day into bittorrent files, and then build a secondary system to turn the data distributed over bittorrents back into browseable "blog" mirrors if/when you need to. The best part is that you don't really need any new technology to do it.

Weblog mirroring system diagram [3]

The main drawback is that you're dependent on the quality and completeness of what you get in the source feeds to begin with, which isn't always good enough. But, I still think it could work.

Trackback URL for this post:

http://onebiglibrary.net/trackback/166
DataPortability.org and the Dream of a Web 2.0 Backup System [4]
from thesecretmirror.com on Tue, 2008-01-29 19:13

I just discovered DataPortability.org through Peter Van Garderen’s blog post about it. I was entirely surprised that I’d heard nary a peep about it. Some basic examination (running a WHOIS query on the domain) shows that it’s still a ...


Source URL (retrieved on 2008-09-07 01:50): http://onebiglibrary.net/story/simple-old-design-for-widespread-blog-mirroring

Links:
[1] http://curtis.med.yale.edu/dchud/log/2004/Oct/20
[2] http://www.loc.gov/standards/mets/
[3] http://www.flickr.com/photos/dchud/395671866/
[4] http://thesecretmirror.com/archives/dataportabilityorg-and-the-dream-of-a-web-20-backup-system