I have a form where users can rate presentations and they can say how knowledgeable they are on the subject. The range for both sets is 1-5. (1 being lousy and 5 being great) So for example, a user can rate a presentation with score 1 and their knowledge 5, which means they are very sure that the presentation was bad.
A presentation can be rated by two distinct people who don't know what the other person rated. If these scores are far apart, a third rater should come into play who acts as a tiebreaker.
What I need is a way to calculate the difference between the two distinct ratings on which I can decide whether or not I should ask the tiebreaker to rate. Obviously it should be some sort of weighted difference. If we go down this path, it could be implemented as follows:
(score person A)(knowledge person A) - (score person B)(knowledge person B)
However this doesn't have the desired result, because for example 3*2 - 1*5 = 1 is a very small difference whereas person B is really sure about his rating so a tiebreaker should probably come into play here. On the other hand 5*5 - 4*5 = 5 is a big difference but both raters are very confident that they know what they are talking about so a tiebreaker should NOT come into play.
What I think would be of help is if somehow the knowledge factor is not linear but progresses along a sort of bell curve. Any ideas on how to come with a better algorithm would be appreciated.