Crowdsourced labor has successfully tackled a variety of rote tasks like image tagging and transcription, but has difficulty accomplishing complex tasks. Visual design critique is a crucial part of the design process for improving upon existing work, but it is a complex task that requires some implicit knowledge of design concepts on the part of the critiquer. In this report, we propose a model for crowdsourcing visual design critique to enable online workers who lack background knowledge to provide useful feedback on designs. Our model organizes the "crit" around a set of design principles, and asks workers to select relevant statements about design principles and provide freeform comments within the scope of the selected statements. We evaluate two variants of a critiquing system we built based on this model (CrowdCritter) through a study with 73 participants. We found that the integration of a set of design principles allowed novice workers to provide more critiques than freeform feedback alone. We were unable to identify significant differences between the two variants of CrowdCritter in the quality and level of agreement of critiques produced through each interface.




Download Full History