Report for Turkopticon Community Survey

Thank you all for your patience with us! 1068 of you filled out the 2016 Turkopticon Community Survey.

First, why the delay? We had some trouble with SurveyGizmo, the service we used to host the survey. It took a few weeks to unbreak our account. Sorry about that!

Our hope was that the survey would have a couple of purposes.

First, it is a chance to communicate and reflect as a community. We compiled a summary report of responses so you can see how others feel about how Turkopticon is going. We also selected anonymous quotes from the open-ended questions that represent a range of insights, appreciations, and concerns expressed by the community. Some people love apples, and other people hate apples. But lots of people agree that fruit is important.

Second, the survey helps us understand the impact and intensity of issues. Some problems affect lots of people a bit. Other problems affect a few people a lot. One is not necessarily more important than the other. But it is helpful to know the difference. The survey helped us understand the difference.

We invite you to join us for a community call to talk about the results, ask questions, or discuss what is next. The call will be by voice and webchat. (Details to follow.) 150 of you told us yo are willing to help out. This call can be a place to discuss, suggest, and collaborate on small projects to improve Turkopticon.

Some of the top things we learned:
  • 78.8% of you are satisfied with how Turkopticon is going. There are many things that could be improved, but there seems to be a lot of good here despite conflicts and drama that can flare up.
  • A range of things people said points to the issue of what counts as a good review and who decides. Numerical ratings like fairness ask reviewers for their opinion, but some people want fairness standardized so they know how to interpret the rating. Some people want to explain the details of their experiences. But newbies get confused about which details are okay to reveal. Completely well-intentioned contributions and ideas about what makes a good review can cause lots of conflict.
  • We take this as a mandate to make a new review form that separates objectively reportable data about a HIT from subjective assessments of a HIT. We hear you that we need both. One worker even suggested a brilliant idea for an unmoderated question: "Would you recommend this requester to others?"
  • Another issue that came up across many responses was that of how to spread the burden of moderation more widely. The new review form should make moderation a lot easier. Once the form is in place, we can talk as a community about what moderation, community management, and newbie education should look like in the future.
  • We would like to make the survey an annual practice. A lot of people said they were happy we were doing it. It is one step in making Turkopticon something more decentralized, more worker governed, and something more people can actively have a hand in shaping.

Lilly and Six
Turkopticon Admins

    • Map

      Powered by Leaflet — © OpenStreetMap contributors, Tiles Courtesy of MapQuest
    • Response Counts

      Completion Rate: 68.8%
      Complete
      736
      Partial
      334
      Disqualified
      0
      Total 1,070
    • Please indicate your relationship to Amazon Mechanical Turk:

      Created with Highcharts 4.2.389.4% : Only a worker2.3% : Primarily a worker, butsometimes a requester2.3% : Primarily a worker, but sometimes a requester 0.3% : Both a worker and a requesterequally0.3% : Both a worker and a requester equally 3.0% : Primarily a requester, butsometimes a worker3.0% : Primarily a requester, but sometimes a worker 4.1% : Only a requester0.8% : Other - Write In
      Value Percent   Count
      Only a worker 89.4%
      955
      Primarily a worker, but sometimes a requester 2.3%
      25
      Both a worker and a requester equally 0.3%
      3
      Primarily a requester, but sometimes a worker 3.0%
      32
      Only a requester 4.1%
      44
      Other - Write In 0.8%
      9
          Total 1,068
    • How satisfied are you with Turkopticon?

      Created with Highcharts 4.2.338.4% : Very Satisfied40.4% : Satisfied13.6% : Neutral4.7% : Dissatisfied2.9% : Very Dissatisfied
      Value Percent   Count
      Very Satisfied 38.4%
      409
      Satisfied 40.4%
      431
      Neutral 13.6%
      145
      Dissatisfied 4.7%
      50
      Very Dissatisfied 2.9%
      31
          Total 1,066
    • Do you currently use Turkopticon?

      Created with Highcharts 4.2.390.9% : Yes9.1% : No
      Value Percent   Count
      Yes 90.9%
      968
      No 9.1%
      97
          Total 1,065
    • How many hours did you spend working on AMT over the last seven days?

      Created with Highcharts 4.2.3Count0135791113151820222528323538414448535664708085951100255075100125150175
    • How important is your income from AMT?

      Created with Highcharts 4.2.338.6% : I do not rely on the income tomake ends meet38.6% : I do not rely on the income to make ends meet 28.5% : Always rely on it to makeends meet28.5% : Always rely on it to make ends meet 26.6% : Sometimes rely on it to makeends meet26.6% : Sometimes rely on it to make ends meet 6.3% : Other - Write In
      Value Percent   Count
      I do not rely on the income to make ends meet 38.6%
      388
      Always rely on it to make ends meet 28.5%
      286
      Sometimes rely on it to make ends meet 26.6%
      267
      Other - Write In 6.3%
      63
          Total 1,004
    • How many months did you use Turkopticon?

      Created with Highcharts 4.2.3Count01234689101112182428303236406102.557.51012.515
    • How fairly do you feel your reviews were treated by moderators?

      Created with Highcharts 4.2.367.9% : Very fairly22.7% : Somewhat fairly5.4% : Somewhat unfairly4.0% : Very unfairly
      Value Percent   Count
      Very fairly 67.9%
      505
      Somewhat fairly 22.7%
      169
      Somewhat unfairly 5.4%
      40
      Very unfairly 4.0%
      30
          Total 744
    • How fairly do you feel others' reviews have been treated by moderators?

      Created with Highcharts 4.2.355.1% : Very fairly31.3% : Somewhat fairly8.5% : Somewhat unfairly5.2% : Very unfairly
      Value Percent   Count
      Very fairly 55.1%
      414
      Somewhat fairly 31.3%
      235
      Somewhat unfairly 8.5%
      64
      Very unfairly 5.2%
      39
          Total 752
  • Please explain your answer above:

    • Which do you consider more important?

      Created with Highcharts 4.2.311.0% : Numerical ratings25.1% : Description of experiencewith requester25.1% : Description of experience with requester 63.0% : I consider them equallyimportant63.0% : I consider them equally important 0.9% : Neither is important
      Value Percent   Count
      Numerical ratings 11.0%
      84
      Description of experience with requester 25.1%
      191
      I consider them equally important 63.0%
      480
      Neither is important 0.9%
      7
          Total 762
    • Please explain your answer above:

    • Much Turkopticon communication has happened through ratings. For each rating, select whether you feel it is moderated too strictly, not strictly enough, or fine as it is.

        too strictly fine not strictly enough
      Fair 123
      16.5%
      566
      75.8%
      58
      7.8%
      Fast 40
      5.4%
      667
      89.3%
      40
      5.4%
      Pay 65
      8.7%
      549
      73.7%
      131
      17.6%
      Communicativity 51
      6.9%
      639
      86.1%
      52
      7.0%
    • Do you have other suggestions for how to improve the governance of Turkopticon?


      See the responses in a new page
    • Would you be interested in receiving monthly email newsletters from Turkopticon about policies, community, and issues?

      Created with Highcharts 4.2.351.5% : Yes40.3% : No8.2% : Depends - Write In (Required)
      Value Percent   Count
      Yes 51.5%
      372
      No 40.3%
      291
      Depends - Write In 8.2%
      59
          Total 722
    • Would you be interested in being contacted to volunteer to make Turkopticon better? If so, please check off relevant skills.

      Created with Highcharts 4.2.3Percentwriting (e.g. helptext,newsletters)writing (e.g. help text, newsletters)conflictresolutionconflict resolutioncommunitymoderationcommunity moderationJavascriptRuby on railsOtherprogrammingOther programmingOther - Write In(Required)Other - Write In (Required)01020304050
      Value Percent   Count
      writing (e.g. help text, newsletters) 48.7%
      150
      conflict resolution 39.0%
      120
      community moderation 36.4%
      112
      Javascript 7.8%
      24
      Ruby on rails 2.6%
      8
      Other programming 9.7%
      30
      Other - Write In (Required) 33.1%
      102