This document contains information for reviewers, requesters, and other users of Turkopticon. It addresses:
This document is a rewrite of the 2013 FAQ, which it replaces.
This document was last updated September 28, 2014.
1. Reviews, flags, and comments
1.1. What are the requester "attributes" ("comm", "pay", "fair", and "fast")?
1.2. How do I post a review?
1.3. I am not getting the verification email.
1.4. How can I write a good review?
1.5. Can I delete my review?
1.6. What is a comment and how can I post one?
1.7. I asked for commenting over a week ago, but I still can't comment and I haven't heard back.
1.8. Can I delete my comment?
1.9. What is a flag and how can I post one?
1.10. Can I delete my flag?
1.11. What are appropriate uses of flags and comments?
1.12. I am a requester. Can you delete or censor a review of me?
2. Miscellaneous technical issues
2.1. My display name is based on my email address (e.g., s...@g...). How do I change it?
2.2. Can I delete my account?
2.3. Does Turkopticon collect personalized data about me?
2.4. I am a requester and I changed my name on Mechanical Turk. Can you change my name on Turkopticon?
3. The Turkopticon application programming interface (API)
3.1. What is the Turkopticon API?
3.2. Why does it exist?
3.3. What else is it used for?
3.4. How do I use it?
3.5. Do I need permission to use it?
3.6. Can I see the code?
4. Known issues and future plans
4.1. Known issues
4.1.1. I don't get automated emails from Turkopticon (verification, password change, etc.).
4.1.2. Requesters whose names have apostrophes in them are messed up in Turkopticon.
4.1.3. The official browser add-ons point to an old, less reliable server.
4.1.4. There is a 20-minute delay in seeing new ratings included in the averages.
4.2. Small or medium changes suggested by users in the past that we would like to make
4.2.1. Can you send me email alerts when someone flags or comments on my review?
4.2.2. I am a requester. Can you send me email alerts when someone reviews me?
4.2.3. Can you make a way to save a "draft" review?
4.2.4. Can you show me my review after I post it, instead of sending me to Mechanical Turk?
4.2.5. Can you make profile pages for requesters, with contact and other useful information?
4.2.6. Can the user script/browser add-on automatically populate the HIT name field?
4.3. Other small or medium changes we would like to make
4.3.1. Add "Did you contact the requester?" to review form
4.3.2. Add "If you contacted the requester, did they reply to your satisfaction?" to review form
4.3.3. Speed up the site by using page and fragment caching
4.3.4. Speed up site by optimizing slow database queries or reducing queries per page
4.4. Big changes we are considering
4.4.1. Let people review specific HITs, not just requesters
4.4.2. A new approach to moderation
5.1. Are messages sent to email@example.com private?
5.2. I am a requester. A reviewer posted an email I sent them in a review. Can you remove it?
6. Information for researchers
6.1. I am a researcher. Can I use average requester data from Turkopticon?
6.2. I am a researcher. Can you give me anonymized data from the Turkopticon database?
6.3. I am a researcher. Can I run a field experiment on Turkopticon?
7. Turkopticon people and background
7.1. Who manages Turkopticon?
7.2. Who has been involved with Turkopticon in the past?
7.3. Is Turkopticon associated with Amazon?
7.4. Why was Turkopticon built?
7.5. Do you make money from Turkopticon?
7.6. Can I give a donation?
The "attributes" used to describe requesters answer the following questions:
The easiest way to post a review is to click the link "Report your experience with this requester" in the add-on drop-down.
However, you can also review any requester by going to turkopticon.ucsd.edu/report (Turkopticon login required). You'll need to know their Mechanical Turk requester ID, though (e.g., A1BC2DE34FGHJ5). If you don't include a Mechanical Turk requester ID, your ratings won't be included in the average ratings shown by the add-on.
This is probably because of a problem with our mail sending system. Fixing this is on our list of things to do. In the meantime, email us at firstname.lastname@example.org from the email address associated with your Turkopticon account and we'll verify it manually.
Give the ratings you feel best describe your experience.
You may wish to put the name(s) of the HIT(s) you are reviewing in the "HIT Name(s)" field.
If one of the fields is not relevant, leave it blank.
For example, if you didn't need to communicate with the requester, leave that field blank.
Don't include or link to profanity, racial slurs, or ethnic slurs.
Don't personally insult or attack requesters or other workers. Calling a requester "cheap" is okay, but anything stronger is subject to being hidden.
Don't include personal information about the requester unless it is part of the HIT.
Don't write your review in ALL CAPS.
If you're a requester, don't review yourself. Comment instead.
Reviews that are profane or insulting or appear to be requester self-reviews may be hidden.
The most up-to-date version of these guidelines will always be posted on the review page (Turkopticon login required).
For now, no.
If you managed to post a review of the same requester twice (i.e., there was a technical error), email us and we will delete the duplicate, as long as nobody else has flagged or commented on it yet.
The main reason we don't allow deletion of reviews is accountability. The world around Mechanical Turk, including Turkopticon, is generally one of mistrust and suspicion. If we allowed untraceable deletion of reviews, people might worry that we were deleting reviews at the request of requesters. We don't do that, and we don't want to jeopardize the usefulness of Turkopticon by giving anyone any reason to imagine that we might. So we don't delete any reviews, even at the request of the author. (Duplicates are the only exception.)
A second reason is that if we were to delete a review that had comments or flags on it, those comments or flags would disappear. This could be confusing, and confusion causes frustration and suspicion, which we try to avoid.
Sometimes people post a review they later regret posting. This often happens when other people flag the review or leave mean comments. The review author often wants to delete the review in order to delete the comments or flags. In our view this is unfair to the comment authors or flaggers, even if they said mean things. We do not have a policy of censoring something if it is merely mean—only if it is profane. In the long run, usually the person who said the mean thing is more embarrassed than the person they said it to. If you are in this situation, you can always edit your review!
All that said, in the future, we may allow review authors to delete their own reviews. If we do this, we will find a way to keep a record that the review existed, and keep comments and flags accessible. For now, if you have the urge to delete a review you posted, the best thing to do is edit it instead, or leave a comment under it.
A comment is a piece of text that responds to a review. (See figure at right.) You post a comment "on" a review.
New Turkopticon accounts cannot post comments. Once you have posted some reviews, you can request commenting (Turkopticon login required) by clicking on the previous link or on the link that says "request commenting ability" that appears between the "settings" and "logout" links at the top of every Turkopticon page while you are logged in.
Commenting requests are reviewed manually, usually within a few days, by our administrators. If you have only posted a few reviews, please expect to be told that you will need to wait to be able to comment.
If you are a requester and you want to post comments in reply to reviews, please email us.
If your comments and/or reviews have been the subject of heated community discussion in the past (for example, you have been involved in flame wars with other people on the site, or have repeatedly posted comments that were hidden for profanity or racism), your commenting request may have been ignored. If you think this may have happened, but you really want to be able to comment and you are willing to make a serious effort to change your ways, please post to turkopticon-discuss.
For now, no. You can edit it, though.
A flag is a sign for moderators to consider hiding a review that may violate Turkopticon's civility guidelines (see above, Section 1.4, "How can I write a good review?"). If a moderator agrees that the review violates civility guidelines or is otherwise inappropriate, the moderator will also flag the review. If a review has at least two flags, and at least one is from a moderator, the review is hidden. Ratings from hidden reviews are not included in the average ratings shown by the add-on.
Only people who can comment can flag reviews. (See above, Section 1.6, "What is a comment and how can I post one?" for information on commenting and getting access to commenting.)
Also, if a comment on a review violates civility guidelines, please don't flag the review. Post to turkopticon-discuss instead.
For now, no. You can edit it or convert it to a comment, though.
Flag a review if it:
In general, flags are for reviews that violate our civility guidelines or threaten the integrity of the review database. If you simply disagree with a reviewer, but believe they are reviewing with good intent, even if you think they are reviewing "wrongly", comment instead. Flags, generally, are for reviews which appear to have some sort of malicious intent.
Moderators and administrators may ignore flags at their discretion. For example, even if a review incorrectly claims that the requester has violated the Mechanical Turk Terms of Service, and is flagged, a moderator may not choose to hide the review.
If you're not sure if you should flag or comment, comment.
As in reviewing (see above, Section 1.4, How can I write a good review?), when commenting, please:
Comments violating these guidelines may be edited by administrators or moderators.
Delete? No. (See above, Section 1.5, "Can I delete my review?")
Censor? Probably not.
With the exception of technical errors (e.g., a review posted twice), we do not delete reviews, even at the request of the author.
In general, requesters are encouraged to comment on reviews rather than flag. But if a review clearly falls within the flagging guidelines (see above, Section 1.11, "What are appropriate uses of flags and comments?"), feel free to flag it. If a moderator agrees that the review violates Turkopticon's civility guidelines (see above, Section 1.4, "How can I write a good review?"), they will also flag it, and the review will be hidden. The ratings included in hidden reviews are not included in the averages displayed by the add-on. (For more on flagging, see above, Sections 1.9–1.11.)
We have had several cases of particularly angry workers posting personal contact information about requesters. In the past it was our policy not to censor this information if it was discoverable via a simple Google search. Following the advice of expert Turkers, however, we have as of August 8, 2014 chosen to discourage reviewers from posting personal requester information in reviews, flags, comments, or posts to turkopticon-discuss, even if the information can be found through Google or is otherwise publicly available. This includes a requester's personal email address (e.g., an address other than that associated with a requester's place of employment), home or cell phone number, or home address, unless it is included in the HIT (e.g., in a consent form workers are expected to read before completing a HIT associated with an academic study). If such information is posted, we reserve the right to censor this information at a requester's request, or at our discretion. However, we do not guarantee that such information will be censored, even at a requester's request.
In general, once your display name is set, it is supposed to stay as it is. But if there was a mistake and somehow your display name was set to the default display name (which is based on the email address you used to sign up for Turkopticon), email us and we will "unset" it, after which you can pick a different display name.
If you already have a "real" display name, we will not unset it. The point of this is to create a little bit of accountability.
For now, the best we can do is "close" your account.
We realize this can be frustrating. "I own all of my reviews!", you might say, "I have a right to delete them!" In principle, we agree. Unfortunately there are technical issues. The main one is that if we actually deleted your account—deleted the entry in the Turkopticon database that represents your account—we would have a bunch of reviews left over associated with a nonexistent account. When the web site tried to display those reviews, it would choke. "So delete them!", you might say. If nobody else had commented on or flagged any of your reviews, we could do that without any technical issues—although we would still have the accountability issues discussed in Section 1.5, "Can I delete my review?". But this is not usually the case. And if we delete a review that someone else has commented on, we are effectively disappearing their comment, which is not fair to them (see above, Section 2.2, "Can I delete my review?", for more on this topic).
So, for now, the best we can do is "close" your account. A closed account cannot be logged into. In the future it is possible we will hide reviews from closed accounts in the same way we hide reviews that violate civility guidelines, although for the moment we see no real need to do this.
As of 5 August 2013, we have not closed many accounts (in fact we have closed exactly one), so our policy on this topic is not very well developed. We don't understand fully why people are motivated to delete their accounts, so we don't know exactly what else we should do when an account is closed. We are open to discussing ways to improve how we handle it.
To close your account, email us. In the future, we may set up a way to do it automatically.
A version of Turkopticon posted in 2010–2011 did collect anonymized data automatically. This code was written by an economist who wanted to make actual wage data available to Turkers, but it was removed because of privacy concerns.
Yes. Email us.
The Turkopticon API is a PHP file that accepts Mechanical Turk requester IDs and returns the information about those requesters available in the Turkopticon database. Specifically, for each requester ID, it returns the requester's name, average ratings for the four requester attributes ("comm", "pay", "fair", and "fast"; see Section 1.1, "What are the requester 'attributes'?"), the number of reviews for the requester in the Turkopticon database (excluding hidden reviews), and the number of those reviews that reported that the requester violated the Mechanical Turk Terms of Service.
This information is returned as a JSON object [wikipedia entry; example JSON object; example from Turkopticon API].
It is used by the Turkopticon user script and browser add-ons to quickly retrieve data about requesters from the Turkopticon database.
It is used by at least one Turker-managed service, mTurk List.
bat are the Mechanical Turk IDs of the requesters you want information for. For example, calling
returns information about the requesters
IMP AMT and
You may pass as many IDs as you like to
multi-attrs.php. It has not been tested with more than a hundred or so, however. Calls with hundreds of IDs or more may be slow, or fail entirely.
No. However, if you plan to use it on an ongoing basis, please also tell us, so we can warn you if we plan to make any changes that may break your application. Also, if you use it a lot over a short period of time, the server may crash. This will make your application fail and cause problems for other users. Please ask us before doing this so we can keep an eye on server load.
Yes. It is on GitHub here. It is licensed under the GNU Affero General Public License.
This section lists known technical problems.
This is a problem with the way we send mail. We are aware of it but are not exactly sure how to fix it yet. Email has become a very complex technology because of all the spammers, and email that does not follow very specific rules is often bounced by mail servers. We haven't figured out all the rules yet. Fixing this is on our list of things to do, however. If you do not get the verification email, or any other email that has information you need to use Turkopticon, please feel free to email us; we will be happy to verify your account or get you the information you need "manually".
This is an issue with the user script/browser add-ons. We are aware of it but have not yet isolated the cause. We aim to fix it in the next release of the user script and browser add-ons.
This is fixed in the most recent version of the user script, and will be updated in the next release of the browser add-ons.
This is due to our cache, which makes everything else much faster. It's possible this can be fixed, but is a low priority for now because nobody seems to mind it too much.
While we will generally refrain from making emails sent to email@example.com public, we reserve the right to ask you to post emails sent to firstname.lastname@example.org to turkopticon-discuss for public discussion, or to do so ourselves.
We are more likely to make an email public if it is rude or threatening, especially if you are a requester demanding to have unfavorable reviews censored.
Almost certainly no.
While you may feel that emails you send to workers are implicitly private, such emails are in our view not 'personal communication' but 'business communication'. While requesters are not employers in the legal sense, requesters are in a position of power over workers by virtue of making payment decisions that may have significant practical consequences. Therefore if a worker feels threatened, intimidated, condescended to, or even just frustrated by an email sent by a requester, we feel it is reasonable for them to be able to post it publicly in a place where other workers or potential workers can see it. We may not always agree that it is a good idea for a worker to do so, but in general we will not censor reviews or comments that include emails sent to workers by requesters.
There are three main exceptions:
Yes. You may use the publicly-accessible Turkopticon API (see above, Section 3).
If you publish or present anything using this data, or if you discover something interesting about it that we might not know, please tell us!
If you plan to use the API on an ongoing basis, please also tell us, so we can warn you if we plan to make any changes that may break your application.
We have not done this so far, although we might be willing to consider it. We would consult publicly with workers first, so if you want to make a request like this, please ask well in advance of any deadlines you may have.
We recommend strongly that you do not do this, even if you think you can do it undetected. A previous field experiment that we were unaware of was discovered by workers while in progress and resulted in formal IRB complaints to the researchers' home institution and severe stress and large amounts of lost time, money, sleep, and effort for the researchers.
While Turkopticon was built by, and continues to be operated partly by, researchers, we do not use Turkopticon data to produce research. We have published papers about Turkopticon, but we do not publish papers using Turkopticon data. Use of Turkopticon is not implicit consent to being a research subject. Turkopticon's primary research contribution is to help imagine more responsible approaches to the design and operation of computing systems. Thus Turkopticon's first responsibility is to the community of workers that relies on it.
With the help of moderators (in alphabetical order) Anne M, Honuagal (now moderator emeritus), Nurse Rachet, rubyredtan, taintturk, and Tribune, Turkopticon is maintained by Lilly Irani and Six Silberman. Lilly is an assistant professor at the University of California, San Diego. Six is a graduate student at the University of California, Irvine.
Note: Answering this question involves two things. First, acknowledging people who have helped us and tried to help us. Second, acknowledging that some of those efforts have ended badly or have raised suspicions among Turkers. It's all here, with the exception of one omitted name, because he's paid enough for the mistakes we made together. If you have questions or concerns about any of this, please feel free to email us.
Turkopticon was motivated by the responses to our 2008 Workers' Bill of Rights survey; the first thanks have to go to the workers who responded to that survey.
Second, in late 2008 and early 2009, Dolores Labs helped us collect initial reviews and spread the word about Turkopticon by linking to us at the bottom of some of their HITs. They wanted to support Turkopticon because they liked the idea of a website that let workers review requesters. Awkwardly for us, Dolores Labs eventually changed its name to CrowdFlower, and is now famous for being a huge requester of dubious quality and low pay—and for being sued for not paying workers minimum wage.
In 2009 and 2010, Dahn Tamir of Techlist helped us with some code maintenance.
In 2010 and 2011, we worked with an economics professor who wanted to give Turkers exact wage data about HITs. We added code to collect this data to the extension. This freaked people out and was taken out.
In early 2014, Jay Tolentino, then an undergraduate at the University of California, Irvine, rewrote the Turkopticon API and our search feature, making both parts of Turkopticon much faster, more reliable, and easier to maintain. As of August 8, 2014, Jay is available for hire.
No. It is not and it has never been.
Turkopticon started in 2008 as a class project to raise awareness about the unaccountability of Mechanical Turk requesters. We saw requesters and researchers celebrating how cheaply they could get work done on Mechanical Turk and we felt this was wrong. We didn't understand why Amazon didn't build in a requester reputation system, like Amazon Marketplace and eBay have. Building Turkopticon was our way of making this point. We never expected it to be used by so many people, or to become a fixture in the Turker community.
No. We do not, and never have.
We do not charge for access to Turkopticon, sell ads or information, or have any other income from Turkopticon. We do not ever plan to charge for access or sell ads, although we may accept donations in the future.
This could change, but at present, we have no way to accept donations.