5 ms·

One thing that this article fails to point out is how often librarians were wrong before Google. In 1986 there was a study[0] that showed that across the board

One thing that this article fails to point out is how often librarians were wrong before Google. In 1986 there was a study[0] that showed that across the board reference librarians were only correct about 55% of the time. I think most people today would consider a query answering system that had an accuracy of 55% to be an interesting curiosity, but certainly not ready for real-world application. It's funny how frequently we measure machine learning performance of an "easy for humans" task and fail to compare it to human accuracy on the same data. We just assume humans would do perfectly on it. I'm sure there are a few MNIST digits that I would get wrong. [0] P.Hernon, C McClure "Unobtrusive Reference Testing: The 55 Percent Rule," Library Journal, 111 April 15,1986

brudgers 12y ago
How often are less than 55% of the results of a Google search relevant? Count ads.
- corin_ 12y ago
  That' s not comparable at all. 55% of answers were correct, not 50% of the possible material in the library was relevant.
  - brudgers 12y ago
    If a search for something like "crane hook design principles" returns a page with "buy crane hook design principles for less" at the top, it is wrong. And wronger than the wrong of a librarian because it's not even trying to be correct, just thrown out there for the edge case. We just happen to write it off as "that's just Google".
    - estel 12y ago
      I don't think that's entirely fair. Surely the most useful measure of "wrong" is whether the user comes away with the correct answer? Adverts on a webpage might definitely make it less likely that a user comes away with that answer, but I don't think they'd invalidate a user's successful result any more than gossiping with the librarian would invalidate their help if they successfully find an answer.
      - brudgers 12y ago
        The advertisements are noise. SEO also produces noise. At some point the signal is lost. Google's business model is built on maximizing the amount of noise a user will tolerate whenever the user in hope that the user stops searching and starts shopping. If you search for a book, Amazon will appear before the Wikipedia page.
        musername 12y ago
        Nobody contested your argument, specific knowledge of niches s still fairly scarce on the net, at least if looking for gratis information. OTOh a well sorted library supplies books that cost a fortune, so google couldn't be replace that, at least not yet.
    - ricree 12y ago
      Is that what it shows you? I just searched for that exact thing. The first two were ISO standards, the third wikipedia, and the fourth a paper on crane hook stress analysis. Not as good as a directed search by a professional, sure (and I haven't actually looked at any of those papers, so who knows how relevant they actually are), but my experience seems a lot better than what you're implying.
      - brudgers 12y ago
        If you followed either of the links to ISO, you would have found that they are offerings to sell the standards. The technical information is behind a paywall. That's ISO's business model. The link to Wikipedia is about cranes in general. If a librarian said "the answer will cost you $181 (CHF 178,00)" or "here is a book about cranes" we would probably agree these high quality agree even if we might disagree over their being wrong. Nevermind that ISO standards are not necessarily a good set of design principles because minimum requirements are not the same as design tradeoffs.
saraid216 12y ago
Suggesting that reference librarianship is "easy for humans" is sort of... ridiculously wrong. There's a reason it requires a master's degree.
baldfat 12y ago
It's the pre-internet sources and Government employees fault and not the Reference Librarian as a profession. This is out of the US Library system only. Your "Across the Board" = "Government documents and central reference areas of 26 U.S. libraries" Not your academic or local libraries that most of us think of as Librarians. I was a System's Librarian and contend that Librarians are needed more now more then ever due to the Internet but that is a different tale. One-Third the Librarians were let go in my state during the economic down turn). Reference Librarians should be quoting sources and I bet you a million dollars that a Reference Librarian gives you the source and it is the source that is wrong.
crististm 12y ago
You can get pretty far with being right only 51% of the time. With 55% you have an edge.
greggarious 12y ago
But why were people asking reference librarians questions? When I spoke to librarians in the mid-90s, they'd just point me towards sources that could have an answer. It was up to me, the reader, to ultimately find the answer.
- dredmorbius 12y ago
  Depends on where you were and who you were asking. At several large research university libraries, I found the reference librarians could be downright obsessive in tracking down some questions. It would depend on workloads and other factors, I suppose.
  - greggarious 12y ago
    This was mostly for pleasure or K-12 assignments. The sort of questions that were very fact based. "Do snakes poop?" "How far away is Mars?" That sort of thing.
    - dredmorbius 12y ago
      I was specifically thinking of uni-level questions.
      - greggarious 12y ago
        At the uni level, aren't they more supposed to help you find the appropriate database rather than find the answer for you? I might ask a librarian how to access back copies of a journal, but I wouldn't expect him or her to do my lit review for me...

brudgers 12y ago

How often are less than 55% of the results of a Google search relevant? Count ads.

corin_ 12y ago
That' s not comparable at all. 55% of answers were correct, not 50% of the possible material in the library was relevant.
- brudgers 12y ago
  If a search for something like "crane hook design principles" returns a page with "buy crane hook design principles for less" at the top, it is wrong. And wronger than the wrong of a librarian because it's not even trying to be correct, just thrown out there for the edge case. We just happen to write it off as "that's just Google".
  - estel 12y ago
    I don't think that's entirely fair. Surely the most useful measure of "wrong" is whether the user comes away with the correct answer? Adverts on a webpage might definitely make it less likely that a user comes away with that answer, but I don't think they'd invalidate a user's successful result any more than gossiping with the librarian would invalidate their help if they successfully find an answer.
    - brudgers 12y ago
      The advertisements are noise. SEO also produces noise. At some point the signal is lost. Google's business model is built on maximizing the amount of noise a user will tolerate whenever the user in hope that the user stops searching and starts shopping. If you search for a book, Amazon will appear before the Wikipedia page.
      - musername 12y ago
        Nobody contested your argument, specific knowledge of niches s still fairly scarce on the net, at least if looking for gratis information. OTOh a well sorted library supplies books that cost a fortune, so google couldn't be replace that, at least not yet.
  - ricree 12y ago
    Is that what it shows you? I just searched for that exact thing. The first two were ISO standards, the third wikipedia, and the fourth a paper on crane hook stress analysis. Not as good as a directed search by a professional, sure (and I haven't actually looked at any of those papers, so who knows how relevant they actually are), but my experience seems a lot better than what you're implying.
    - brudgers 12y ago
      If you followed either of the links to ISO, you would have found that they are offerings to sell the standards. The technical information is behind a paywall. That's ISO's business model. The link to Wikipedia is about cranes in general. If a librarian said "the answer will cost you $181 (CHF 178,00)" or "here is a book about cranes" we would probably agree these high quality agree even if we might disagree over their being wrong. Nevermind that ISO standards are not necessarily a good set of design principles because minimum requirements are not the same as design tradeoffs.

saraid216 12y ago

Suggesting that reference librarianship is "easy for humans" is sort of... ridiculously wrong. There's a reason it requires a master's degree.

baldfat 12y ago

It's the pre-internet sources and Government employees fault and not the Reference Librarian as a profession. This is out of the US Library system only. Your "Across the Board" = "Government documents and central reference areas of 26 U.S. libraries" Not your academic or local libraries that most of us think of as Librarians. I was a System's Librarian and contend that Librarians are needed more now more then ever due to the Internet but that is a different tale. One-Third the Librarians were let go in my state during the economic down turn). Reference Librarians should be quoting sources and I bet you a million dollars that a Reference Librarian gives you the source and it is the source that is wrong.

crististm 12y ago

You can get pretty far with being right only 51% of the time. With 55% you have an edge.

greggarious 12y ago

But why were people asking reference librarians questions? When I spoke to librarians in the mid-90s, they'd just point me towards sources that could have an answer. It was up to me, the reader, to ultimately find the answer.

dredmorbius 12y ago
Depends on where you were and who you were asking. At several large research university libraries, I found the reference librarians could be downright obsessive in tracking down some questions. It would depend on workloads and other factors, I suppose.
- greggarious 12y ago
  This was mostly for pleasure or K-12 assignments. The sort of questions that were very fact based. "Do snakes poop?" "How far away is Mars?" That sort of thing.
  - dredmorbius 12y ago
    I was specifically thinking of uni-level questions.
    - greggarious 12y ago
      At the uni level, aren't they more supposed to help you find the appropriate database rather than find the answer for you? I might ask a librarian how to access back copies of a journal, but I wouldn't expect him or her to do my lit review for me...