Feedback Army – usability testing for the rest of us

Posted on | November 26, 2008 | 5 Comments

just discovered Feedback Army, a service offering dirt-cheap website-usability-testing based on Amazon’s Mechanical Turk. how does it work? you post the URL of the site tested alongside several open questions (f.e. “Describe in your own words: what’s the main purpose/functionality of the website?”, “Are there any browser-related issues, like f.e. slow responses?”) – Feedback Army automagically creates a HIT (aka “task”) on Mechanical Turk, and soon the answers start trickling in (literally, via RSS)… since it’s only 7,- USD for 10 responses I gave it a try and requested some feedback on Charts.fm, a yet unfinished mashup I’ve been toying around with recently. Here’s my post:

after less than 2 hours I’ve got 4 replies of varying quality. here are some highlights (full responses can be found here):

Comming from the UK the main page is slow to load. i like how your instantly. I also like how you are instantly greeted with top songs and the option to play then and buy on iTunes and amazon. The domain name itself is a TLD which makes it more searchable. I most liked the bio of the artist that appears when you click on the song of your choice, this gives good facts for research projects. I would however like to see lyrics added to the website and also more colour added because it does look a bit plain. The signup button is hard to find, it is just tiny text in the top right; usually websites like these have big flash enterances for new registrations. When building my charts i fount it very easy to do, and the resuts were spot on really. Overall a good site, just needs more colour and ease of registration.

there’s some nice feedback & suggestions included here. although language is poor at times, I’ld say it’s a quality response. others have been sticking more tightly to the structure of my request:

1. Chart.fm builds radio charts for the user’s selection of channels and stations. 

2. The results are fine but I don’t see what you mean by “remixing”. I think there are only options to narrow down the selection. 

3. I’m using Firefox 3 on Vista and had no issues. 

4. The site looks as it could be used without sign-up. If it stores charts for registered users, I wouldn’t provide any more information than my email adress. 

5. The radio stations should be grouped in some way. No one will go through a list of 2644. 

6. chart.fm is a nice way to get free music. You could make this clearer by putting a play button next to the song titles to signal that one can listen to them right away. At the moment the initial impression with the amazon/iTunes buttons is that you are only trying to sell.

some thoughts:

• overall I was positively surprised by the quality of responses – I’ld have expected worse of Mechanical Turk (clearly a prefjudice as I’ve never used the service before!)
• there is no hard limit on how many questions you get to ask, however I don’t think it would make sense to ask more than 10 questions per HIT
• phrase your questions as open as possible – try to assign certain tasks you want being tested (like signup – although only 2/4 responses actually signed up in my case)
• no idea if it would make sense to test a german (non-english) website – how international is Mechanical Turk’s audience?
• the genereated feedback is publicly available, if you can guess the URL of a tested site. I tried out http://www.feedbackarmy.com/get_feedback.slp?url=http://www.feedbackarmy.com, and – voila – got the feedback about the service itself …no big deal, I guess
• of course, you can always create a HIT in Mechanical Turk yourself, but this way faster / easier. haven’t checked on the pricing though

 

Comments