One Big Library.

Has anybody out there thought about advertising an OpenSearch service via ZeroConf-style wide-area DNS Service Discovery (DNS-SD)? A quick look at the DNS-SD SRV Service Types doesn't seem to indicate as such, and a search for "opensearch and dns-sd" or a search for "opensearch and zeroconf" mostly shows hits back to this blog.

I see three reasons for considering this:

• Lets the discovery happen even when users are looking at one of your webapps or subdomains where you can't twiddle the HTML HEAD element (there are still plenty of these)
• Lets a remote service find out more about the institution running your network (modulo "but I'm not on that network now" problem, which plain ol' zeroconf itself could help)
• The more avenues for discovery the better

Er, those being "general reasons for anybody using OpenSearch to consider this". My own reasons extend these with more specifics about library-specific uses like metasearch and link resolvers, some details of which I've written up before.

One main drawback to all this is:

• Browsers and javascript-in-browsers don't do DNS

...or, well, Safari and Konq do, but not IE (not without a plugin, and "First Install This Thingy" doesn't scale) and not Mozilla/Firefox. But then again we could always proxy that with a web service, yadda, yadda. See also mozilla bug 14328 and discussion.

Defining an SRV type would be pretty straightforward - it probably only needs a key pointing to the path of the OpenSearch description document, since that's what that's for.

(Last in a series.)

Last week I gave the talk I'd been leading up to. Here it is in a nutshell (photo by Nicole Engard):

The premise of the talk is: people are already starting to ask us "why doesn't the library work like iTunes?" By this they really mean (I think) "how come when I walk into the library everything in the library doesn't just show up on my machine like when I open iTunes and my friends' music just shows up on my machine?"

I think we need to take this question very seriously. Especially because there's no good reason we can't do just what that not-so-hypothetical user wants. A great candidate networking protocol (ZeroConf) is already out there, flowing in the network. We could layer OpenSearch over our metasearch interfaces, and intermingle search and resolver services fairly easily, and along the way *every* one of those steps would have side benefits: merging OpenURL resolution and metasearch UIs into a single view simplifies things for users; adding OpenSearch means you can put your library's catalog right into your users' browsers; using ZeroConf to announce your service makes it easy for everybody to find your interface from anywhere.

During the talk I demonstrated just how much ZeroConf is already flowing all around us - we're swimming in it!. Apparently the video didn't video, so to see how it works for yourself just try the follow the next time you're on a network with a lot of other users (like, a public wifi spot would be ideal, but it should also work at your workplace).

• If you haven't seen it before, turn on iTunes music sharing and look for people sharing their music. It's fun and incredibly easy.
• Install iStumbler and look at the bonjour services tab (note that this is also a useful tool for browsing wireless nets.
• Install Bonjour Browser to get a closer look at the network traffic underlying the more user-friendly screens in iStumbler.

The conference wireless AP was unreliable, so I bought my own and turned it on as an open wireless net (named "librarywebclique", in honor of librarywebchic). In the middle of the talk I invited participants to join the net so I could show the ZeroConf traffic in iStumbler and Bonjour Browser. In a matter of seconds, about 20 people had done just that and all of us could see who was sharing iTunes, who had visible web servers, SMB shares, or ssh ports open, among other things. From where I was standing, it was a pretty convincing demonstration, and all it entailed was showing everybody what their machines were *already* doing.

I can't stress enough how important it is to keep a close eye on what's already happening out there on the net at a global scale. The microformats.org project is frustrating in many ways but they're dead right in their insistence on recognizing what's already out there as primary (though I'd quibble over their self-consistency in enforcing that, but that's another story). This traffic is flowing on the network - lots of widely-used apps already use it - there's no reason we can't just hop aboard.

I'm attaching slides and audio as best I can, but imho it was the kind of thing you had to see - it's very easy to make a point about what's possible by showing that it's actually already out there. Still, try those apps above (they're OSX only, but the traffic isn't, so you can probably find similar apps on other platforms) and you'll get the point.

Near the end I took a few questions, but didn't think to repeat them. One was "what about remote access?", to which I responded that this model doesn't really improve much on that set of issues or make it much worse, though in theory by adding desktop-level discoverability it's possible it could improve things slightly. Another was "what do you think the interface to this would look like", which is a bit tougher to answer. There are examples like iTunes and Delicious Library to follow for what it should look like when a library just "shows up" on your machine. But I think my immediate answer was the right one - I don't know, but it could show up in a lot of places, like the browser, or a desktop app, or on server apps, because this is a platform/infrastructure level service opportunity. In one of those answers I stated that "if we don't provide this level of ease of use, then we're failing", and because we don't, I think we are. Failing, that is.

There is a lot of talk out there in biblioblogoville about "getting into the flow", but I can't imagine a better way to do that than to radically integrate our most flow-oriented services (metasearch and openurl service resolution) and advertise its availability on our networks. And it can be accomplished quickly, cheaply, uniformly, and with a variety of side benefits.

Which, of course, probably means that it'll never happen, because our community isn't really capable of this kind of realization, let alone acting quickly upon it at scale.

That doesn't mean that I don't think you should bother with it - of course not, I think this is all very doable, and I hope that I met my objective of trying to frame all of this in a positive light: see the last few slides, where I suggest how to respond to "can't we make the library work like iTunes?": we should respond by saying "yes, let's."

The slides are attached as pdf, and here's the audio to lead you through 'em. You can hear my clicky-taps to advance the slide most of the time.

I spent an unexpected amount of time during a meeting today trying to get a little search interface I wrote to properly support opensearch autodiscovery and to get the "you can install this search in firefox 2 and ie7" thingy working. Then I just did the same for this blog's opensearch interface. In case you want to do the same (get browsers to autodiscover and two-click install search plugins for your opensearch targets), here are some hints. Note, though, that they might be firefox2-specific.

• Follow the Firefox troubleshooting tips. All of them, they all matter, it seems.
• I might have just started from an old version, but the drupal opensearch module wasn't emitting the opensearch description element Url with text/html info as indicated as required in the troubleshooting tips.
• The "Unable to install your search plugin thingy" error message is confoundingly unhelpful. To get useful debugging info in firefox2, go to "about:config" and enable browser.search.log (just double-click it). Then open Tools > Error Console and clear it before each time you attempt to install the search plugin. Read the output closely and go from there until the error message goes away and it works. Then you probably want to go back to about:config and disable browser.search.log again.

Here's the thing: getting opensearch-based search plugins for your browser can be a huge win for the visibility and usefulness of your resources to your user community. It's easy and it's great to have your stuff right there, an easy query away all the time. But if you want to make it easily auto-discovered, you have to get all the details exactly right *for* them.

That said, I'm not 100% certain it works in ie7. If anybody has it handy, tell me if it works for you?

I will get back to the ZeroConfMetaOpenSearch series soon. But before I could go much further, I had to try this first.

Here's what the OpenURL KEV format for journal and journal article reference queries might look like if it were to follow OpenSearch 1.1 Draft 3. More to the point, this is what the block specifying the journal KEV params might look like inside the OpenSearch description response if it were to use the Parameter extension.

<Url xmlns:parameters="http://a9.com/-/spec/opensearch/extensions/parameters/1.0/"
     type="text/html"
     template="http://example.com/search">
  <parameters:Parameter name="q" value="{searchTerms}"/>
  <parameters:Parameter name="count" value="{itemsPerPage?}" minimum="0"/>
  <parameters:Parameter name="start" value="{startIndex?}" minimum="0"/>
  <parameters:Parameter name="aulast" value="{aulast?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="aufirst" value="{aufirst?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinit" value="{auinit?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinit1" value="{auinit1?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinitm" value="{auinitm?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="ausuffix" value="{ausuffix?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="au" value="{au?}" minimum="0" maximum="*"/>
  <parameters:Parameter name="aucorp" value="{aucorp?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="atitle" value="{atitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="title" value="{title?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="jtitle" value="{jtitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="stitle" value="{stitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="date" value="{date?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="chron" value="{chron?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="ssn" value="{ssn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="quarter" value="{quarter?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="volume" value="{volume?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="part" value="{part?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="issue" value="{issue?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="spage" value="{spage?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="epage" value="{epage?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="pages" value="{pages?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="artnum" value="{artnum?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="issn" value="{issn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="eissn" value="{eissn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="isbn" value="{isbn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="coden" value="{coden?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="sici" value="{sici?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="genre" value="{genre?}" minimum="0" maximum="1"/>
</Url>

I *think* that's right. It isn't clear to me whether the q/searchTerms param is required in OpenSearch or not. If it is, then we could use it to paste through an url-encoded text representation of the reference, I suppose. If not, well, it could just be optional, or dropped entirely.

We'd still need to do work on the response record format - perhaps something might be borrowed from the IESR Metadata specs and inserted into RSS2.0 or Atom responses.

What's amazing to me about OpenURL is that it's been implemented so successfully (in the sense that we had nothing before, and now we have something). That said, I repeatedly hear arguments that by defining and using more OpenURL application profiles you gain benefits through the shared OpenURL infrastructure. I don't buy that because I've never seen such benefits. If you swap out OpenURL and swap in OpenSearch and use the same application profiles, like in this example, then I could see the benefits of shared OpenSearch infrastructure.

I worked up and have been giving a new talk recently about the directions I think library services should go. As far as I could tell, they were quite well received by both technical and non-technical audiences alike, and I've been exchanging messages with several people offline about it in one way or another.

The more I think about it, the more I'm convinced that this direction, stumbled into with gentle nudges and nudzhes alike from people like Ed and Jeremy and Terry, is the right next set of steps to take. I'm not sure how the outcome of it all will turn out, but I don't think we can afford not to try, and soon. The best thing about it all is that it's all inexpensively accomplished, or at least prototyped, and we should be able to find out quickly whether it's Good (tm) or not.

The thing is, it would be better to engage a *lot* of people on these topics, and this kind of project, because we'll only see if it works if it can be taken up quickly and broadly. So this is the first of a series on ZeroConfMetaOpenSearch, introducing the plan one piece at a time, spinning out what's more compactly worded in those slides.

Where to start?

Imho the best place to start is with a challenge. Have you ever used iTunes? More specifically, have you ever used its music sharing function? It's awesome. You can see what other people near you are listening to, and you can listen to their stuff, and they can find and listen to yours. The best part about it is that once you enable the feature in your iTunes preferences, it Just Works (tm). There's no configuration needed - the protocol works on top of a local-net service discovery protocol that is the most magical protocol this side of HTTP itself.

So the challenge is: what are we going to say when our library patrons start asking us: "why can't we share our personal libraries with your big library as easily as we can share music with iTunes?"

Got an answer?

There is no good answer, because there's no reason we can't apply a glossy sheen to our sloppy mess-o-OPACs and metasearches to play very nicely with others, just like iTunes does. (Actually, afaik, delicious library already does use the same networking protocol for this exact purpose, but, we don't all have that, do we.)

When I say "there's no reason why we can't", what I really mean is that everything we need to do it is already sitting around waiting for us to plug in. Just like on the front page of today's WSJ, featuring how the Zimbra folks assembled their Exchange-competitor largely out of FLOSS components, nearly all the components we need are already available. But the key thing is that they have to plug in easily, cheaply, and *universally*, with no "first we need a new standard to do X" step. And they cannot require end-users to have to plug anything in or install or configure anything. To accomplish this, what we really should be doing is using the best existing standards, even if they're (gasp!) not from the library community.

In the talk I use these three tests to judge whether a new approach adequately meets these objectives:

1. Is it *simple*? i.e., can any competent coder grok it in a quick sitting, and churn out compliant apps in a few hours?
2. Does it require a new specification? We don't have time form more committees, I fear... we have to move forward with what's already here.
3. Does it Just Work? Do normal people have to actually follow any kind of elaborate instructions to make it happen, or stand on their head, or sign something, or offer sacrifices and pay for a warranty with a mail-in rebate? 'Cuz those are right out.

iTunes meets all the needs above (we could quibble about DAAP and #2, but, it's zeroconf and HTTP that make DAAP possible as much as DAAP itself). We have to, too.

Where is this all leading? Essentially, to the three points listed here. In the next few posts, I'll spin out toward each one with a little more detail, and will hopefully draw some of you into the discussion along the way.

Next time, I'll evaluate how COinS and unAPI stack up against the three tests listed above. (Then we'll get into the collapsing of metasearch and resolver interfaces into a single interface, and after that how OpenSearch could layer usefully over all of that, and then I'll close with the ZeroConf piece and why it ties everything together so neatly.)

But first things first: shouldn't our whole libraries be as easy to connect and share as iTunes?