During my regular perusal of Hacker News a little while back, I ran across what appeared to be an interesting blog post, specifically Peter Thiel’s CS183: Startup - Class 4 Notes Essay by Blake Masters. Not knowing much at the time, I just saw that it was notes for a lecture given on startups and thought it would be worth checking out. To bring you up to speed now, though, they are notes for an actual course at Stanford University with Peter Thiel as one of the lecturers, a PayPal co-founder.
Now, there are tons of interesting blog posts and articles that show up on HN. Most of which I end up browsing and often discard a few paragraphs in. Though there are still more I would like to read through completely but I just don’t have the time. Particularly as I am a rather slow reader. Thus when I came across these notes at around 6200 words, I was a bit saddened as I figured I would never get around to reading through it.
I just so happened to have recently read through this great article on taking back your commute. I drive at least 1 hour every day for my commute to and from work. I often just perceived it as a waste of 5 hours every week that I couldn’t spend doing anything worthwhile other than getting to work or home. Though the article above is about orating blog posts during one’s commute, it got me to start doing the opposite in hunting down some good podcasts. Though, how do I ‘listen’ to all these other great blog posts and articles for which there is no MP3 to download?
There are numerous text-to-speech tools out there. I cycled through a few online and tried whatever was on my Andriod phone by default but was not impressed. The text is read so poorly that it is nearly impossible to follow. Poor pauses between words, bad pronunciation at times, overall just really bad flow. Could I just get someone to read this for me to an audio file so I could play it back in the car?
The solution that then came to mind was to post a job on Amazon’s Mechanical Turk service requesting an oration of the blog post. I had never used the service before but felt I had a decent idea of how it worked. You post simple work that is ideal for a human to do versus a computer. Usually simple stuff like locating an object in a picture. I saw that jobs for voicemail transcription were available so I figured my job couldn’t be much, if any, harder. So I signed up as a requestor using my Amazon account.
I wasn’t sure what to offer in terms of price. Jobs again are usually small and only reward a few cents. Looking at the transcription jobs, it seemed they were paying about $24/hr. This is probably the most similar job to what I’m looking to post so I’ll base it off that. The text being 6200+ words and a Google search telling me people often read around 200 words per minute, this sounds like a half hour job. I settled at $13 with Amazon taking a $1.30 fee for the post.
My detailed posting:
First impressions. I was a little weary just having my full name out there on the post, not having used this service before. There’s no option to have a separate name that I could find. I ended up changing my name in my Amazon account. Thankfully it doesn’t change on my billing or shipping information, though my Amazon orders now come through with the change. I’ll just have to switch it back.
I wasn’t particularly fond of the interface to manage the HITs (Human Intelligence Task, a job essentially). I could not find a way to “go to” my HIT. In the management tab, I can find basic information on it but I can’t go to the same page the workers see. Clicking the title displays my description text but it is heavily elongated vertically and looks terrible. Is this my fault? Is this what the workers see? I don’t know. I search for my post in the HIT index but cannot find it no matter how I search.
Come the next day, searching still yields nothing and my management panel has not changed to indicate any completed work. I decide to contact Amazon support. Some hours later I received an e-mail of moderate length written by an actual person. This part I was impressed with, I half expected just some “hey, make sure you read the FAQs” message with a link. While I feel the person read and understood my e-mail, I however ended up confused at the end.
First paragraph:
“It is very likely that someone grabbed your HIT as soon as you published it due to the reward amount, which should explain why you weren't able to find it on the Mechanical Turk website.“
Fair enough, but there’s no way to inform me of this on the “Manage” tab for individual HITs. There is no indicator for “someone claimed your job but hasn’t submitted yet.”
Next paragraph:
“I reviewed the HIT and found that it only had one assignment which was completed and submitted. It is currently under "Pending Review". You can view the submitted result under the Manage tab, when you click on the results. If the work wasn't done according to the description, you have a right to reject the submission.“
Problem is, that’s not what my Manage tab was saying. “Assignments Pending Review” was 0 and no amount of refreshes or log-in’s and outs would change it. Another link was provided in the e-mail for feedback if the Amazon person could not answer my question. I submitted appropriate feedback stating my problem and also sent an additional help request through MTurk.
Hours later, my “Assignments Pending Review” goes to 1 and I can finally download the submitted results. I never received a follow up message regarding either the feedback or the additional message through MTurk. I don’t think this was a data “freshness” problem on my management panel as this all happened over the course of several hours, not minutes.
Now, on to the work itself. While I requested a good quality MP3, what I actually received was a WMA2 stereo file with audio in only the left channel. Not a big deal. The person speaking sounds to be maybe around 12, female I believe. (If you’re the worker who did this job, I apologize for any inaccurate guessing!) Overall though everything seems good, I can understand them and minus some pronunciation mistakes, the audio is of overall good quality. I later accept the work so that the worker can receive payment.
Now, later, a friend of mine whom I told this story to informs me that Mac’s have an awesome text-to-speech program. I don’t have a modern Mac but in doing some digging on Youtube, I can confirm it is much better sounding than anything else I tried previously. Good enough for listening to on the road? Perhaps. I was a bit dismayed, feeling like my MTurk oration was a wasted pursuit. Though in hindsight I still don’t regret it, I got to learn a bit as well as “read” the lengthy blog post. Even if it was a bit expensive. (During writing this post, I ran across this longer demo of Alex on Mac, I’m not sure it would work for me, though still very impressive).
My overall impression of Amazon’s Mechanical Turk is mixed. The management system and information presentation definitely needs some work. The service has been out since 2005, it’s technically still in beta but with 100,000 available HITs at any given time, it’s hard to see this as not being a prime time service. I just can’t help but feel that the user interface is still rather immature with regards to enabling me to see what’s going on with my work.
What most impressed me was the results. Maybe I got lucky but I just thought it was really neat to submit a work item for a person to perform and then bam, there it is. I didn’t discuss anything with the worker ahead of time or have to choose through a list of possible workers. It just works. Now, there are options for only selecting workers of a certain quality. I skipped all that and can’t really say off hand how it all works. Despite Amazon’s issues, I still look forward to the day I have another random human intelligence task to post.
If you’re interested, here is a download link to the audio file. I cleaned up the noise a bit, moved it to single channel joint stereo and converted to MP3. Before posting this, I was a bit concerned about who technically owns it. From what I understand (I’m not a lawyer FYI), anything you do in the US is copyrighted by default and owned by you unless you give it away. Kind of like the photo studios who own your family photos even though you paid to have them done. Does the worker technically own this audio production? Nothing came up searching on Google so I checked out the terms on MTurk’s site. Under 3(b) it states:
So it looks like I own the recording. Though the work being read, I do not own. I tried to contact Blake through Facebook but didn’t receive a response. I’m posting it anyways though as I’m pretty sure it qualifies as fair use since I’m not selling it and Blake doesn’t appear to get any revenue from the posting of his notes.
On a side note, it would be interesting to see what this service could grow into. What out of the box things could people think of? Could someone in a big city post a “Make and deliver me a sandwich by 3pm tomorrow” and have it filled? Anyone want to try? :D
Now, there are tons of interesting blog posts and articles that show up on HN. Most of which I end up browsing and often discard a few paragraphs in. Though there are still more I would like to read through completely but I just don’t have the time. Particularly as I am a rather slow reader. Thus when I came across these notes at around 6200 words, I was a bit saddened as I figured I would never get around to reading through it.
I just so happened to have recently read through this great article on taking back your commute. I drive at least 1 hour every day for my commute to and from work. I often just perceived it as a waste of 5 hours every week that I couldn’t spend doing anything worthwhile other than getting to work or home. Though the article above is about orating blog posts during one’s commute, it got me to start doing the opposite in hunting down some good podcasts. Though, how do I ‘listen’ to all these other great blog posts and articles for which there is no MP3 to download?
There are numerous text-to-speech tools out there. I cycled through a few online and tried whatever was on my Andriod phone by default but was not impressed. The text is read so poorly that it is nearly impossible to follow. Poor pauses between words, bad pronunciation at times, overall just really bad flow. Could I just get someone to read this for me to an audio file so I could play it back in the car?
The solution that then came to mind was to post a job on Amazon’s Mechanical Turk service requesting an oration of the blog post. I had never used the service before but felt I had a decent idea of how it worked. You post simple work that is ideal for a human to do versus a computer. Usually simple stuff like locating an object in a picture. I saw that jobs for voicemail transcription were available so I figured my job couldn’t be much, if any, harder. So I signed up as a requestor using my Amazon account.
I wasn’t sure what to offer in terms of price. Jobs again are usually small and only reward a few cents. Looking at the transcription jobs, it seemed they were paying about $24/hr. This is probably the most similar job to what I’m looking to post so I’ll base it off that. The text being 6200+ words and a Google search telling me people often read around 200 words per minute, this sounds like a half hour job. I settled at $13 with Amazon taking a $1.30 fee for the post.
My detailed posting:
Read the blog post linked below. Reader must be a fluent English speaker. Accent is OK but not too heavy (relative to American Midwest). Record and upload a good quality MP3 file for the entire post speaking with natural speed, paragraph and sentence structure such that the average listener can keep up with and understand.
First impressions. I was a little weary just having my full name out there on the post, not having used this service before. There’s no option to have a separate name that I could find. I ended up changing my name in my Amazon account. Thankfully it doesn’t change on my billing or shipping information, though my Amazon orders now come through with the change. I’ll just have to switch it back.
I wasn’t particularly fond of the interface to manage the HITs (Human Intelligence Task, a job essentially). I could not find a way to “go to” my HIT. In the management tab, I can find basic information on it but I can’t go to the same page the workers see. Clicking the title displays my description text but it is heavily elongated vertically and looks terrible. Is this my fault? Is this what the workers see? I don’t know. I search for my post in the HIT index but cannot find it no matter how I search.
Come the next day, searching still yields nothing and my management panel has not changed to indicate any completed work. I decide to contact Amazon support. Some hours later I received an e-mail of moderate length written by an actual person. This part I was impressed with, I half expected just some “hey, make sure you read the FAQs” message with a link. While I feel the person read and understood my e-mail, I however ended up confused at the end.
First paragraph:
“It is very likely that someone grabbed your HIT as soon as you published it due to the reward amount, which should explain why you weren't able to find it on the Mechanical Turk website.“
Fair enough, but there’s no way to inform me of this on the “Manage” tab for individual HITs. There is no indicator for “someone claimed your job but hasn’t submitted yet.”
Next paragraph:
“I reviewed the HIT and found that it only had one assignment which was completed and submitted. It is currently under "Pending Review". You can view the submitted result under the Manage tab, when you click on the results. If the work wasn't done according to the description, you have a right to reject the submission.“
Problem is, that’s not what my Manage tab was saying. “Assignments Pending Review” was 0 and no amount of refreshes or log-in’s and outs would change it. Another link was provided in the e-mail for feedback if the Amazon person could not answer my question. I submitted appropriate feedback stating my problem and also sent an additional help request through MTurk.
Hours later, my “Assignments Pending Review” goes to 1 and I can finally download the submitted results. I never received a follow up message regarding either the feedback or the additional message through MTurk. I don’t think this was a data “freshness” problem on my management panel as this all happened over the course of several hours, not minutes.
Now, on to the work itself. While I requested a good quality MP3, what I actually received was a WMA2 stereo file with audio in only the left channel. Not a big deal. The person speaking sounds to be maybe around 12, female I believe. (If you’re the worker who did this job, I apologize for any inaccurate guessing!) Overall though everything seems good, I can understand them and minus some pronunciation mistakes, the audio is of overall good quality. I later accept the work so that the worker can receive payment.
Now, later, a friend of mine whom I told this story to informs me that Mac’s have an awesome text-to-speech program. I don’t have a modern Mac but in doing some digging on Youtube, I can confirm it is much better sounding than anything else I tried previously. Good enough for listening to on the road? Perhaps. I was a bit dismayed, feeling like my MTurk oration was a wasted pursuit. Though in hindsight I still don’t regret it, I got to learn a bit as well as “read” the lengthy blog post. Even if it was a bit expensive. (During writing this post, I ran across this longer demo of Alex on Mac, I’m not sure it would work for me, though still very impressive).
My overall impression of Amazon’s Mechanical Turk is mixed. The management system and information presentation definitely needs some work. The service has been out since 2005, it’s technically still in beta but with 100,000 available HITs at any given time, it’s hard to see this as not being a prime time service. I just can’t help but feel that the user interface is still rather immature with regards to enabling me to see what’s going on with my work.
What most impressed me was the results. Maybe I got lucky but I just thought it was really neat to submit a work item for a person to perform and then bam, there it is. I didn’t discuss anything with the worker ahead of time or have to choose through a list of possible workers. It just works. Now, there are options for only selecting workers of a certain quality. I skipped all that and can’t really say off hand how it all works. Despite Amazon’s issues, I still look forward to the day I have another random human intelligence task to post.
If you’re interested, here is a download link to the audio file. I cleaned up the noise a bit, moved it to single channel joint stereo and converted to MP3. Before posting this, I was a bit concerned about who technically owns it. From what I understand (I’m not a lawyer FYI), anything you do in the US is copyrighted by default and owned by you unless you give it away. Kind of like the photo studios who own your family photos even though you paid to have them done. Does the worker technically own this audio production? Nothing came up searching on Google so I checked out the terms on MTurk’s site. Under 3(b) it states:
As a Provider, the Requester for whom you provide Services is your client, and as such, you agree that the work product of any Services you perform is deemed a "work made for hire" for the benefit of the Requester, and all ownership rights, including worldwide intellectual property rights, will vest with the Requester immediately upon your performance of the Service. To the extent any such rights do not vest in Requester under applicable law, you hereby assign or exclusively grant (without the right to any compensation) all right, title and interest, including all intellectual property rights, to such work product to Requester.
So it looks like I own the recording. Though the work being read, I do not own. I tried to contact Blake through Facebook but didn’t receive a response. I’m posting it anyways though as I’m pretty sure it qualifies as fair use since I’m not selling it and Blake doesn’t appear to get any revenue from the posting of his notes.
On a side note, it would be interesting to see what this service could grow into. What out of the box things could people think of? Could someone in a big city post a “Make and deliver me a sandwich by 3pm tomorrow” and have it filled? Anyone want to try? :D
Comments