ZeroConfOpenMetaSearch: the code4lib talk wrap-up

(Last in a series.)

Last week I gave the talk I'd been leading up to. Here it is in a nutshell (photo by Nicole Engard):

Photo by Nicole Engard

The premise of the talk is: people are already starting to ask us "why doesn't the library work like iTunes?" By this they really mean (I think) "how come when I walk into the library everything in the library doesn't just show up on my machine like when I open iTunes and my friends' music just shows up on my machine?"

I think we need to take this question very seriously. Especially because there's no good reason we can't do just what that not-so-hypothetical user wants. A great candidate networking protocol (ZeroConf) is already out there, flowing in the network. We could layer OpenSearch over our metasearch interfaces, and intermingle search and resolver services fairly easily, and along the way *every* one of those steps would have side benefits: merging OpenURL resolution and metasearch UIs into a single view simplifies things for users; adding OpenSearch means you can put your library's catalog right into your users' browsers; using ZeroConf to announce your service makes it easy for everybody to find your interface from anywhere.

During the talk I demonstrated just how much ZeroConf is already flowing all around us - we're swimming in it!. Apparently the video didn't video, so to see how it works for yourself just try the follow the next time you're on a network with a lot of other users (like, a public wifi spot would be ideal, but it should also work at your workplace).

  • If you haven't seen it before, turn on iTunes music sharing and look for people sharing their music. It's fun and incredibly easy.
  • Install iStumbler and look at the bonjour services tab (note that this is also a useful tool for browsing wireless nets.
  • Install Bonjour Browser to get a closer look at the network traffic underlying the more user-friendly screens in iStumbler.

The conference wireless AP was unreliable, so I bought my own and turned it on as an open wireless net (named "librarywebclique", in honor of librarywebchic). In the middle of the talk I invited participants to join the net so I could show the ZeroConf traffic in iStumbler and Bonjour Browser. In a matter of seconds, about 20 people had done just that and all of us could see who was sharing iTunes, who had visible web servers, SMB shares, or ssh ports open, among other things. From where I was standing, it was a pretty convincing demonstration, and all it entailed was showing everybody what their machines were *already* doing.

I can't stress enough how important it is to keep a close eye on what's already happening out there on the net at a global scale. The project is frustrating in many ways but they're dead right in their insistence on recognizing what's already out there as primary (though I'd quibble over their self-consistency in enforcing that, but that's another story). This traffic is flowing on the network - lots of widely-used apps already use it - there's no reason we can't just hop aboard.

I'm attaching slides and audio as best I can, but imho it was the kind of thing you had to see - it's very easy to make a point about what's possible by showing that it's actually already out there. Still, try those apps above (they're OSX only, but the traffic isn't, so you can probably find similar apps on other platforms) and you'll get the point.

Near the end I took a few questions, but didn't think to repeat them. One was "what about remote access?", to which I responded that this model doesn't really improve much on that set of issues or make it much worse, though in theory by adding desktop-level discoverability it's possible it could improve things slightly. Another was "what do you think the interface to this would look like", which is a bit tougher to answer. There are examples like iTunes and Delicious Library to follow for what it should look like when a library just "shows up" on your machine. But I think my immediate answer was the right one - I don't know, but it could show up in a lot of places, like the browser, or a desktop app, or on server apps, because this is a platform/infrastructure level service opportunity. In one of those answers I stated that "if we don't provide this level of ease of use, then we're failing", and because we don't, I think we are. Failing, that is.

There is a lot of talk out there in biblioblogoville about "getting into the flow", but I can't imagine a better way to do that than to radically integrate our most flow-oriented services (metasearch and openurl service resolution) and advertise its availability on our networks. And it can be accomplished quickly, cheaply, uniformly, and with a variety of side benefits.

Which, of course, probably means that it'll never happen, because our community isn't really capable of this kind of realization, let alone acting quickly upon it at scale.

That doesn't mean that I don't think you should bother with it - of course not, I think this is all very doable, and I hope that I met my objective of trying to frame all of this in a positive light: see the last few slides, where I suggest how to respond to "can't we make the library work like iTunes?": we should respond by saying "yes, let's."

The slides are attached as pdf, and here's the audio to lead you through 'em. You can hear my clicky-taps to advance the slide most of the time.

20070301-code4lib-zeroconfmetaopensearch.pdf2.11 MB


Good. Very Good.

There are folks working on "Search across everything" - at least on the iTunes front. Goombah does this by sending local iTunes metadata to a central server where they slice and dice and do some Pandora-meets-LibraryThing magic. The result is music recommendations, with adjustable fuzziness.

Is there a way to do this without the one-off metadata harvesting that traditional itunes does, and/or without the privacy implications of a centrally hosted server? Or do we just give users the choice - I'd be happy, e.g., for my Zotero database to be shared, perhaps without my personal info, perhaps with... not sure others would agree.

(Unrelated ote: there's a little program called ProfCast that lets you point it to your Keynote file, then present while it records audio and captures slide timing and, optionally, screen or window shots (for live browser stuff). Then, export to enhanced podcast for slides+audio. No SWF or direct QT export for now - only AAC or GarageBand, but still a nice capture for iTunes users. (I'm not the developer, don't know him, just like the product.))

Thanks for the comments. You

Thanks for the comments.

You asked: "Is there a way to do this without the one-off metadata harvesting that traditional itunes does, and/or without the privacy implications of a centrally hosted server?"

I think I mentioned in the talk that this likely comes down to efficient search and network traffic. Doing any amount of harvesting among diverse networked resources is likely to allow more efficient search across the network than if everybody is querying everybody else all the time. I don't have hard numbers to back that up, but I've done a basic study with simple assumptions and that always seems to be the case with any number of users greater than a handful.

There's lots of good, recent distributed search research by skilled scientists to find if you want to know what somebody who really knows what they're doing has to say about it. :)

I've peaked at ProfCast before but haven't tried it. Thanks for the reminder!