Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted a 2x2 experiment in which student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We find that rubrics help novice workers provide feedback that is rated just as valuable as expert feedback. A follow-up analysis on writing style showed that student designers find feedback most helpful when it is emotional, positive, and specific, and that providing a rubric improves the application of these characteristics in feedback. The analysis also finds that expertise only affects style by increasing critique length, but an informal evaluation suggests that experts may instead produce value through providing clearer justifications.
Title
Worker Expertise and Expert Rubrics in Crowdsourced Design Critique
Published
2015-12-01
Full Collection Name
Electrical Engineering & Computer Sciences Technical Reports
Other Identifiers
EECS-2015-223
Type
Text
Extent
14 p
Archive
The Engineering Library
Usage Statement
Researchers may make free and open use of the UC Berkeley Library’s digitized public domain materials. However, some materials in our online collections may be protected by U.S. copyright law (Title 17, U.S.C.). Use or reproduction of materials protected by copyright beyond that allowed by fair use (Title 17, U.S.C. § 107) requires permission from the copyright owners. The use or reproduction of some materials may also be restricted by terms of University of California gift or purchase agreements, privacy and publicity rights, or trademark law. Responsibility for determining rights status and permissibility of any use or reproduction rests exclusively with the researcher. To learn more or make inquiries, please see our permissions policies (https://www.lib.berkeley.edu/about/permissions-policies).
