Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted a 2x2 experiment in which student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We find that rubrics help novice workers provide feedback that is rated just as valuable as expert feedback. A follow-up analysis on writing style showed that student designers find feedback most helpful when it is emotional, positive, and specific, and that providing a rubric improves the application of these characteristics in feedback. The analysis also finds that expertise only affects style by increasing critique length, but an informal evaluation suggests that experts may instead produce value through providing clearer justifications.




Download Full History