Thread:A few thoughts on the peer evaluation proposal. (1)

First off -- Akash belated congratulations on being accepted for this years GSoC on WikiEducator. I was travelling offshore at the time the announcement was made.

Reading your document a few thoughts come to mind;


 * 1) I think its appropriate to have each learner "opt in" at the individual task level for peer review rather than the course level. In the MOOC environment some learners may only be interested in parts of a course. Opting in at the task level will avoid problems of folk who leave the course and do not complete their peer-evaluation tasks.
 * 2) Mapping to rubrics is an excellent idea - perhaps a link to a wiki page. Here is an example of a rubric, which is too detailed for a shorter task - but illustrates how grades can be  associated with different criteria.  The model of specifying a rubric will create flexibility without overcomplicating the coding interface.
 * 3) I would suggest that the course designer specifies the weightings for grade versus participation rather than the system pre-determining this weighting.
 * 4) * If learners opt in at the task level - we will need to think of a mechanism to aggregate the participation for all tasks within a course.
 * 5) Should we decide on a maximum number of categories - or is this indefinite?
 * 6) If we have catagories, we should also have the ability to apply weightings to the categories where the sum of the weightings of all categories = 100%.
 * 7) What do you mean by 3 grades from a scale of 1-10? Do you mean, for example A, B and C?
 * 8) Your point is well made about the problems of using the average score where there our outliers.  My gut feel is that an absolute deviation, for example 1.5 points is going to be problematic for a small number of responses. Perhaps the standard deviation or a z-score might be a better measure for reliability. I think we should consult with a statistician for advice on the best measure.
 * 9) The other idea I have is that there is research to support the validity and reliability of self-evaluations when compared with the teacher evaluations. There are also sound pedagogical reasons for implementing self evaluations. It would be a good idea to implement a self-evaluation option and then decisions on "counting" or "flagging" problem evaluations where the rater is not being fair could be based on deviations from the self-evaluation.
 * 10) I think random assignment will mitigate against "gaming" the system. If only 3 people are evaluating discarding outliers may not generate reliable results - -the sample is too small.  Perhaps using a broad scale parameters like "Poor" "Satisfactory", and "Excellent" could help or mitigate the problem. Ultimately, if there is a major difference in the grades, it would be better to flag the grade as problematic requiring review by the instructor or some other mechanism.
 * 11) Deadlines are important.  I suggest we have a mechanism to email reminders that a peer evaluation is due. Open learners are notorious in forgetting deadlines ;-)
 * 12) For the prototype phases - I suggest that we apply peer grading for formative assessment (i.e. feedback on the learning process) rather than contribution to final grades. Most institutions would be uncomfortable with using peer evaluation at this time as a contribution to final grades, however many institutions would recognise the participation metrics as part of the final grade.

This is a great start! Looking forward to this development.