iamelgringo

Tuesday, September 09, 2008

Mechanical Turk: Now with 25 percent more Awesome.

I just finished my first big project using Amazon's Mechanical Turk service. I'm in love.

I had a list of over 6000 business names, addresses and url's of dubious quality that I needed to make sure was accurate. For a brief moment, I thought about checking them myself. But after doing several dozen by hand, I realized that I was violating the principle of Don't Be Silly(TM). So I was stuck. Either I could use half baked data as is or shelve the project entirely.

Enter Mechanical Turk, and the hordes of awesomeness to save the day. I'd heard about Mechanical Turk several years ago, and I had been dying to use it. But, I never quite found 12,000 itches that I needed to scratch.


What it took to do the project

I needed the Turkers (preferred nomenclature for those who complete tasks on Mechanical Turk) to look up the business name and URL, correct any errors they found and submit any comments. To increase accuracy, I wanted each task completed twice.

Here are the basic steps I did to create a batch job.

  • I clicked a few buttons on the Turk's interface to tell it how many times you need each task to be done, any requirements on the Turkers that you wish to enforce, and how much you're willing to pay a Turker for the task.
  • I then created a HIT, or Human Intelligence Task. "Human Intelligence Task" sounds a bit complicated, but it's really simple and straight forward. You design a specialized HTML template with a form in it based off of sample ones that Mechanical Turk give you.
  • Finally, I reviewed everything, uploaded a CSV file filled with the name, addresss and URL data that I need to complete the template, and I hit submit.
  • Approve each task as it's completed, or wait until it's finished and you are able to use the "Approve All" button.
  • 5 days later, download my CSV file full of awesome.

I did run a test of 2-300 addresses through the system to be sure the process worked. A few tweaks later, I loaded up the 6,204 line CSV file, sat back, and within minutes, hordes of Turkers were chipping away at my 12,408 tasks.


The Awesomeness

Now, before I started, I was concerned about the quality of work that I was going to get for $.02 a task. I was half expecting a CSV file filled with unusable crap at the end of this, and I was ready to chalk it up to a lesson learned, and $300 wasted.

Instead, I was blown away at how people handled the task. People really took pride in their work, and were interested in helping me out. Several Turkers emailed asking for clarification on how I wanted the individual fields completed. One or two emailed me wishing me luck with my project.

One user pointed me to Turker Nation, a bulletin board where Turkers can bounce questions off of each other. She had created a thread devoted to my task. So, I used the chance to the people working on it and give them a bit of guidance as to exactly what I wanted.

Further examples of awesomeness were in the comments people submitted along with the information:

  • Can't find the exact web address for the hospital. Apologies for using a directory listing, in spite of it being the less desirable method.
  • The business you're looking for is a famous cancer institute setup in 1898 by Dr. Roswell park. And it happens to be America's first cancer institute.
  • I was unable to find a website, but went through the county government site to get the information. I checked multiple sites, since the address I found was different. They all listed the Airport Road address.

One person couldn't find a URL for the business for me, so they emailed me to let me know that they called the business long distance and verified the address for me.

Holy crap. People did all that for two cents a task.

Tips

Several Turkers mentioned that they appreciated working for a decent requester (me). Apparently, Turkers tend to get upset when requesters do things like treat them like crap, talk down to them in their form, or don't approve their work on a task that pays a measly two cents.

So, from my experience, I'd suggest a couple of tips when using the Mechanical Turk service.
  • Don't be a jerk. Approve people's work fer crying out loud. You're paying two cents to a nickels for people's time and effort. Reserve not approving people for users that fill your forms with spam or other hostile acts.
  • Appeal to people's kindness. Most people really aren't doing this for the cash, although I'm sure some do. If people feel like they're helping someone out, or doing something for a good cause, they're going to feel good about themselves as well as earn two cents a task. Turkers really want the requesters to succeed.
  • Be clear about what you need. I read a number of threads on Turker Nation where people were frustrated that they couldn't figure out what the requester wanted. Decreaseing people's pain level in completing your task means that more people will be interested in working for you.
  • Be thankful. I was really truly humbled at how many people helped me complete my task. I was also overwhelmed at how many people wished me luck on my new business venture. So, I made a point to express my gratitude in the HIT as well as on Turker Nation. People really responded to that.

Finally

The whole project of verifying a list of 6,204 business names, addresses and URLs was completed in under 5 days at a cost of $300. I'm thinking of all the different ways in which I can make use of the service.

Seriously, it's cool. You should check it out.

30 Comments:

At 8:06 PM , OpenID igrigorik said...

Awesome, thanks for sharing your experience. I've been eyeing turk for a long time now.. but have been largely skeptical. I'll definitely have to give it a try.

 
At 2:59 AM , Anonymous Anonymous said...

A digital sweatshop if you ask me. I especially like this line line : I was expecting crap for the 0.02 cent they got paid, but I was surprised how they took pride in their work...

 
At 5:02 AM , Anonymous Anonymous said...

Digital sweatshop? What are you even on about? Are they forced to do that work?

Obviously from their perspective its worth it - otherwise they wouldn't do it.

I imagine that each Turker has a few projects going on at the same time. So if they can complete 2 such HITs per week that $600 per week - which is not too shabby.

 
At 5:11 AM , Anonymous Anonymous said...

I've been watching MT on the sidelines for a couple of years (I was involved in a Crowdsourcing startup, Cambrian House) and always wondered how MT would stand up.

It's great to hear it does and see an actual example of what it could be used for. Nicely done and THANKS!

 
At 5:34 AM , Blogger Daniel Furrer said...

I don't think your assumptions based on 600usd per week are accurate.

A yearly salary in india could be 1000usd (I was not able to find reliable numbers).

Assuming 2 minutes per address check, that would make a competitive salary of 1100usd a year. Assuming 4 weeks of holidays and 40hrs a week worktime. (0.02 * 60/2 * 155 * 12)

Daniel

 
At 6:47 AM , Blogger asdfdsfdsf said...

Some of you guys are over estimating the time it takes to verify this type of information. That said, I'm not sure you could make $1000 a month off this type of activity.

Doing a quick WAG, I made up a name of a hospital, googled it, and 'verified' the URL and phone number in under 15 seconds.

I think, given you were doing a more few dozen of these at a time, you could average a look up time of under a minute per.

Given a 30 second lookup, and 2 cents a lookup, that would be roughly a $2.40 payout per hour. That would translate into $96 a week, or $400 a month.

 
At 8:19 AM , Blogger Alex said...

I also recently started using MTruk and was very much impressed with it. The quality and time it took to complete was just great.

Is it a sweatshop or not? Money is not the primary thing motivating turkers as shown by this study.

 
At 8:22 AM , Anonymous Anonymous said...

don't forget, niggaz r broke these dayz

 
At 10:55 AM , Blogger dasil003 said...

My theory is that the reason so many people do this is because they have desk jobs where they're just sitting around anyway. They can do this instead of play solitaire, and the extra $1-$2 dollars an hour is just a nice little bonus to their regular paycheck. Capturing lost productivity if you will.

 
At 11:54 AM , Anonymous Anonymous said...

As far as you can tell, where were most of your turkers located?

 
At 12:40 PM , Anonymous Anonymous said...

Couple hundred dollars a month is a college educated person's salary in china...

 
At 12:51 PM , Anonymous Anonymous said...

> Obviously from their perspective its worth it -
> otherwise they wouldn't do it.

Same can be said of real sweatshops, nobody forces people to work there, does it make it right ?

Is it worse to offer this kind of job or to create the conditions that make it "worth doing it" ?

 
At 8:46 PM , Anonymous Anonymous said...

I dont care

 
At 10:13 AM , Anonymous Eric said...

Mechanical Turk is strange. I signed up as a turker once to see what it was like, and quickly decided it wasn't worth my time. It pays really badly.

 
At 11:41 AM , Anonymous Anonymous said...

How long does it generally to take for tasks to be completed? I started with 291 4 hours ago and still sitting at 291? Is this normal?

 
At 1:11 AM , Anonymous nihat hatipoglu said...

Awesome, thanks for sharing your experience. I've been eyeing turk for a long time now.. but have been largely skeptical. I'll definitely have to give it a try...

 
At 10:49 PM , Anonymous Peregrine said...

You write very well.

 
At 6:01 PM , Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 10:51 PM , Anonymous electric actuators said...

this idea is quite new! very interesting.

 
At 2:35 AM , Blogger kim said...

I have recently started working as a turker.
It certainly doesn't pay me anything like my amazon.com sales but it has taught me more about internet structure and how to find the money than 8yrs of surfing the "money making opportunities" did.

I might be rich.

 
At 1:00 PM , Anonymous Anonymous said...

It was extremely interesting for me to read that post. Thanks for it. I like such themes and anything connected to this matter. I would like to read a bit more on that blog soon.

 
At 4:40 PM , Anonymous Turking4aLiving said...

Great article. Coming from a worker its nice to hear the other side of the Requester. Mechanical Turk certainly has a learning curve for workers. However, once you start thinking in terms of an hourly rate, you’ll start finding those gem HITs that can make some real money. Check out my website for real numbers, real money, and real analysis of work with Mechanical Turk. http://www.turking4aliving.com

 
At 6:47 AM , Anonymous Anonymous said...

Nice post and this enter helped me alot in my college assignement. Gratefulness you as your information.

 
At 10:26 PM , Anonymous Anonymous said...

qiut smoking hypnosis [url=http://usadrugstoretoday.com/categories/antiviral.htm]antiviral[/url] the way to a mans heart is through his stomach http://usadrugstoretoday.com/products/accutane.htm atkins diabetes revolution book http://usadrugstoretoday.com/products/floxin.htm
beautify your heart [url=http://usadrugstoretoday.com/categories/skin-care.htm]skin care[/url] red blood cell with dark edge [url=http://usadrugstoretoday.com/categories/anti-allergique-et-l-asthme.htm]sphincter muscle[/url]

 
At 10:45 PM , Anonymous Anonymous said...

sildenafil traditional chinese medicine [url=http://usadrugstoretoday.com/products/leukeran.htm]leukeran[/url] watch streaming breast reduction procedure http://usadrugstoretoday.com/products/plavix.htm binge eating disorder heal naturally http://usadrugstoretoday.com/categories/female-enhancement.htm
whats going on if my chinese water dragon is sleeping alot [url=http://usadrugstoretoday.com/products/trental.htm]trental[/url] place in my heart [url=http://usadrugstoretoday.com/catalogue/n.htm]quitting smoking and sunflower seeds[/url]

 
At 9:21 PM , Anonymous Anonymous said...

http://rxonline.in/retin/murad-acne-medication-retin-a-acne-scars-cures-for-cystic
[url=http://rxonline.in/haldol]viagra cialas levitra compare[/url] drug addict behavior [url=http://rxonline.in/orlistat/non-prescription-orlistat]non prescription orlistat[/url]
drugs when pregnant http://rxonline.in/requip/requip-sex-drive
[url=http://rxonline.in/zithromax/zithromax-perscription]cardinal health rmote order entry illinois[/url] larkin medicine cabinet [url=http://rxonline.in/omnicef/what-is-the-difference-in-rocephin-and-omnicef]what is the difference in rocephin and omnicef[/url]
date rape drug symptoms http://rxonline.in/ribavirin/ribavirin-and-tinitus
[url=http://rxonline.in/promethazine/promethazine-and-constipation]drug poems for elementary students[/url] tapeworm treatment drugstore [url=http://rxonline.in/mircette/mircette-vs-ortho-tricyclen]mircette vs ortho tricyclen[/url] anti inflamatory drug [url=http://rxonline.in/zestril/vasotec-versus-zestril-for-heart-failure]vasotec versus zestril for heart failure[/url]

 
At 10:30 PM , Anonymous Anonymous said...

http://rxmeds.in/cymbalta/does-cymbalta-have-a-generic-pill
[url=http://rxmeds.in/allergic/can-dogs-be-allergic-to-rice-in-their-food]ct board of pharmacy[/url] discount viagra [url=http://rxmeds.in/advair/advair-and-weird-feeling-in-head]advair and weird feeling in head[/url]
drug abuse ceu intervention on site programs http://rxmeds.in/depressive/quick-inventory-of-depressive-symptomatology
[url=http://rxmeds.in/xeloda/acute-onset-headache-with-xeloda]list of commonly used drugs[/url] drugs work [url=http://rxmeds.in/aceon/acetazolamide-and-dexamethasone]acetazolamide and dexamethasone[/url]
felton burke drug charges http://rxmeds.in/acidity/oranges-acidity
[url=http://rxmeds.in/acai/harmful-effects-of-the-acai-berry]how could cloning advance drug therapies[/url] fun fun pharmacy manga [url=http://rxmeds.in/famciclovir/famciclovir-and-birth-control]famciclovir and birth control[/url] pharmacy certification preparatory courses [url=http://rxmeds.in/premarin/hrt-premarin]hrt premarin[/url]

 
At 10:35 PM , Anonymous Anonymous said...

http://rxonline.in/levlen/levlen-and-weight-gain
[url=http://rxonline.in/rhinocort/rhinocort-side-effects]drugs in badminton[/url] drugs in iraq [url=http://rxonline.in/requip/know-the-side-effects-of-taking-requip]know the side effects of taking requip[/url]
drug names and hcl http://rxonline.in/rhinocort/rhinocort-aqua-steriod
[url=http://rxonline.in/revia]viagra versus cialis[/url] drug travel in fort myers [url=http://rxonline.in/rimonabant/measurement-of-efficacy-of-rimonabant]measurement of efficacy of rimonabant[/url]
the importance of not using drugs http://rxonline.in/lipitor/vitamin-e-lipitor
[url=http://rxonline.in/linezolid/linezolid-resistant-strain-atcc]napa valley pharmacy[/url] purchase prescription drugs online [url=http://rxonline.in/hypnotherapy/tinnitus-hypnotherapy-script]tinnitus hypnotherapy script[/url] free practice pharmacy exam [url=http://rxonline.in/proscar/generic-proscar]generic proscar[/url]

 
At 9:01 PM , Anonymous Anonymous said...

http://newrx.in/carbohydrate/carbohydrate-cereal
[url=http://newrx.in/imuran/how-to-decrease-imuran-dosage]metal affects caused by drugs[/url] why floatable drug are produced floatable drugs [url=http://newrx.in/imuran/low-lymphocyte-and-imuran]low lymphocyte and imuran[/url]
drug abuse affect on economy http://newrx.in/weight-loss/extreme-weight-loss-success-stories
[url=http://newrx.in/ditropan/ditropan-xl-ter]self diagnosis for drug intoxication[/url] generic levitra [url=http://newrx.in/fluconazole/buy-cheap-fluconazole-online]buy cheap fluconazole online[/url]
altace rampiril drugs http://newrx.in/methylprednisolone/methylprednisolone-diabetes
[url=http://newrx.in/carbohydrate/calorie-king-fat-and-carbohydrate-counter]what drugs to use for high blood pressure veterinary medicine[/url] idebenol mexican pharmacy [url=http://newrx.in/angina]angina[/url] study medicine in malaysia [url=http://newrx.in/remeron/remeron-elevated-thyroid-levels]remeron elevated thyroid levels[/url]

 
At 10:41 PM , Anonymous Anonymous said...

s 86 67 pill [url=http://usadrugstoretoday.com/products/azulfidine.htm]azulfidine[/url] trend in medical tourism http://usadrugstoretoday.com/products/cephalexin.htm itt health insurance for retirees http://usadrugstoretoday.com/products/hydrea.htm
female orgasm during sleep [url=http://usadrugstoretoday.com/contact.htm]medications without a prescription[/url] south carolina medical board [url=http://usadrugstoretoday.com/products/dostinex.htm]health affects of carotenoid[/url]

 

Post a Comment

Subscribe to Post Comments [Atom]

Links to this post:

Create a Link

<< Home