The use of images in multiple-choice formats for comprehension testing is common. The validity of responses of those tests may be confounded by (a) physical stimulus features, such as size, luminance or clarity of images or (b) semantic content conveyed by multiple-choice images such as image familiarity and frequency. An instrument, based on the literature and validated by experts, was developed to assess such influences within multiple-choice images. Twenty-three adults rated 20 sets of images from five published aphasia batteries. Eye movements were recorded for a separate group of 20 adults viewing the same image sets. All sets prompted disproportionate visual attention, suggesting a need for consideration of these influences when using current tests and better control in designing test images.