On Sat, Jul 26, 2003 at 12:58:44PM -0700, blinc wrote: (01)
> My strong view of the rating issue is that most ratings without comment are
>of
> nearly no value.
>
> There are places for purely numerical ratings, but when discussing factual
> information, it is almost never enough. (02)
I'm not sure if I agree with "no value," but I do agree that ratings
augmented with comments are useful. I also think this has some
implications on design of ratings systems. (03)
When I was playing with Kathryn and Chris's faceted classification
tool for the Bootstrap Alliance archives, one of my comments was that
the rating system was meaningless to me. It used a 10 point numerical
scale to rate posts to the BA lists. The problem was that other
people were not rating articles the way I was, and I wasn't even
rating articles consistently. For example, I would rate an article as
an 8 because it was interesting, but not relevant, and then would rate
a later article a 7 because it was relevant, but not interesting.
When I reviewed my rated articles, I would find that I didn't like
having the latter rated below the former, because relevance was more
important to me than interest (in this particular case). (04)
I resolved this problem by coming up with a more well-defined system
for myself, and applying that system consistently. This still didn't
resolve the problem of other people rating articles their own way. (05)
As an aside, I read a great book by Gregory Treverton about the
problems of the American intelligence system that was published a few
months before 9/11. There was an anecdote about one of the
intelligence agencies working out a consistent rating system
internally. I'll try to pull out the reference next week; I'm away
from my archive of notes this week. (*sigh* If only I had a DKR with
intelligent mobile capabilities.) (06)
What implications does this anecdote and Brian's point have on ratings
systems? Don't go overboard with numbers, first of all. Is a 10
point scale really better than a five point scale, or even a three
point scale? (07)
Second, what people are supposed to be rating matters, except when it
doesn't. :-) If people are supposed to be rating relevance, then say
that. In the case of a trust-network rating system, defining this
stuff probably matters less, because you're probably interested in
other people's interests on a variety of levels. I might include Bill
in my trust network not just because of his discriminating eye for
factual correctness, but also because his sense of humor agrees with
me. (08)
-Eugene (09)
--
Was I helpful? Let others know. (010)
http://svcs.affero.net/rm.php?r=eekim&p=EnablingOnlineCommunities (011)
--
This message is archived at: (012)
http://collab.blueoxen.net/forums/cgi-bin/mesg.cgi?a=tools-yak&i=20030728154444.GJ25175@douge.blueoxen.net (013)
|