First Prototype Trial

I did a pre-trial test study today, where five users (university staff and students) were asked to try the prototype and answer the questionnaire that will be used in the proper study. As expected the Povey version of the display had some issues that affected the way it was used. The initial prototype (finding points) got a mixed response, with some positive feedback and some less so. I will conduct a more detailed look at the feedback over the next few days, but I wanted to get down some initial observations while they are still fresh.

• In discussion after the participant had tried the display, it became evident that there focus was more on testing the technology, rather than looking at and learning about the painting, whilst this is to be expected (the novelty factor) it poses a slight problem for the study. I think that a partial solution is to be careful about the wording about what is expected of them (letting them know that 1) the display is designed to help them understand the painting 2) they should use the display for as long as they want 3) they will be asked questions about their experience of the display and, 4) that they will be asked questions about how well they understand the painting after using the display.
• One user expressed that they were less impressed with the display, as they were expecting something more advanced (a more visually stimulating experience with more of a technology wow factor, like the advanced depictions in films), However they did mention that they are more into there gadgets, and less interested in the content side of the display. I think part of the issue is that I have specifically chosen quite a simple aesthetic as the study is more focused on the impact of different underlying structures (through comparison), than the visual style. They also mentioned that they wanted to see the painting and content in more of a merged experience on the screen, with more interactivity with the displayed content.
•  I think most of the users were unsure how to use the display at first, I think that a better on device introductions (with visual material - illustration, photo or video) may work better, especially for the less technology competent user. 
• As expected most of the visitors complained of their arms getting tired as they needed to hold the tablet in such a high position in order to get it to track. 
• Some users seemed uncertain about where to stand and the distance they should be from the painting. Whilst some moved around to find the best position others seemed less willing to do so. The problem was that with the Povey trial version some points required you to be close and some further away, however the users seemed to find a mid position and only looked at the points that tracked from that distance. In the proper version it will be better to have all points track from about the same distance, and tell the users the optimum distance at which they should hold the device (e.g. 1 meter) from the artefact.
• Some users said that they gave up as they were not sure how many content points there were to find, and as such were not sure if they had finished.
• Glare was a bit of an issue with some of the tracking points, where the painting was by the bright light source of the window. This resulted in the user pointing the device to where a tracking point is and getting no response. 
• I found the observation scale quite hard to use as it is quite subjective (it is impossible to tell if the user is enjoying the display). I think most of the category's are OK, and a bit more practice at using the scale will be useful. It is also essential to have the 'cant tell' option and use it where no observation judgement can be identified.
• For the proper study I need to be more consistent in how the questions are asked. In the pre-trial I took the initial standpoint of holding the questionnaire and reading out the question but allowing the participant to scan the available options (multiple choice) and say which one they wanted. One participant expressed a preference for being given the questionnaire and filling it in on there own. This may have the benefit that they will be more willing to put down their true opinion, by there is more possibility that they will not answer all the questions.