Semantic descriptor ranking: A quantitative method for evaluating qualitative verbal reports of visual cognition in the laboratory or the clinic

Matthew Maestri, Jeffrey Odel, Jay Hegdé

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

For scientific, clinical, and machine learning purposes alike, it is desirable to quantify the verbal reports of high-level visual percepts. Methods to do this simply do not exist at present. Here we propose a novel methodological principle to help fill this gap, and provide empirical evidence designed to serve as the initial "proof" of this principle. In the proposed method, subjects view images of real-world scenes and describe, in their own words, what they saw. The verbal description is independently evaluated by several evaluators. Each evaluator assigns a rank score to the subject's description of each visual object in each image using a novel ranking principle, which takes advantage of the well-known fact that semantic descriptions of real life objects and scenes can usually be rank-ordered. Thus, for instance, "animal," "dog," and "retriever" can be regarded as increasingly finer-level, and therefore higher ranking, descriptions of a given object. These numeric scores can preserve the richness of the original verbal description, and can be subsequently evaluated using conventional statistical procedures. We describe an exemplar implementation of this method and empirical data that show its feasibility. With appropriate future standardization and validation, this novel method can serve as an important tool to help quantify the subjective experience of the visual world. In addition to being a novel, potentially powerful testing tool, our method also represents, to our knowledge, the only available method for numerically representing verbal accounts of real-world experience. Given that its minimal requirements, i.e., a verbal description and the ground truth that elicited the description, our method has a wide variety of potential real-world applications.

Original languageEnglish (US)
Article numberArticle 160
JournalFrontiers in Psychology
Volume5
Issue numberMAR
DOIs
StatePublished - Jan 1 2014

Fingerprint

Semantics
Cognition
Dogs

Keywords

  • Natural language processing
  • Neuropsychological tests
  • Qualitative research
  • Semantic processing
  • Visual cognition

ASJC Scopus subject areas

  • Psychology(all)

Cite this

Semantic descriptor ranking : A quantitative method for evaluating qualitative verbal reports of visual cognition in the laboratory or the clinic. / Maestri, Matthew; Odel, Jeffrey; Hegdé, Jay.

In: Frontiers in Psychology, Vol. 5, No. MAR, Article 160, 01.01.2014.

Research output: Contribution to journalArticle

@article{1dc0882fc7b34a018940c02ea2353519,
title = "Semantic descriptor ranking: A quantitative method for evaluating qualitative verbal reports of visual cognition in the laboratory or the clinic",
abstract = "For scientific, clinical, and machine learning purposes alike, it is desirable to quantify the verbal reports of high-level visual percepts. Methods to do this simply do not exist at present. Here we propose a novel methodological principle to help fill this gap, and provide empirical evidence designed to serve as the initial {"}proof{"} of this principle. In the proposed method, subjects view images of real-world scenes and describe, in their own words, what they saw. The verbal description is independently evaluated by several evaluators. Each evaluator assigns a rank score to the subject's description of each visual object in each image using a novel ranking principle, which takes advantage of the well-known fact that semantic descriptions of real life objects and scenes can usually be rank-ordered. Thus, for instance, {"}animal,{"} {"}dog,{"} and {"}retriever{"} can be regarded as increasingly finer-level, and therefore higher ranking, descriptions of a given object. These numeric scores can preserve the richness of the original verbal description, and can be subsequently evaluated using conventional statistical procedures. We describe an exemplar implementation of this method and empirical data that show its feasibility. With appropriate future standardization and validation, this novel method can serve as an important tool to help quantify the subjective experience of the visual world. In addition to being a novel, potentially powerful testing tool, our method also represents, to our knowledge, the only available method for numerically representing verbal accounts of real-world experience. Given that its minimal requirements, i.e., a verbal description and the ground truth that elicited the description, our method has a wide variety of potential real-world applications.",
keywords = "Natural language processing, Neuropsychological tests, Qualitative research, Semantic processing, Visual cognition",
author = "Matthew Maestri and Jeffrey Odel and Jay Hegd{\'e}",
year = "2014",
month = "1",
day = "1",
doi = "10.3389/fpsyg.2014.00160",
language = "English (US)",
volume = "5",
journal = "Frontiers in Physiology",
issn = "1664-042X",
publisher = "Frontiers Research Foundation",
number = "MAR",

}

TY - JOUR

T1 - Semantic descriptor ranking

T2 - A quantitative method for evaluating qualitative verbal reports of visual cognition in the laboratory or the clinic

AU - Maestri, Matthew

AU - Odel, Jeffrey

AU - Hegdé, Jay

PY - 2014/1/1

Y1 - 2014/1/1

N2 - For scientific, clinical, and machine learning purposes alike, it is desirable to quantify the verbal reports of high-level visual percepts. Methods to do this simply do not exist at present. Here we propose a novel methodological principle to help fill this gap, and provide empirical evidence designed to serve as the initial "proof" of this principle. In the proposed method, subjects view images of real-world scenes and describe, in their own words, what they saw. The verbal description is independently evaluated by several evaluators. Each evaluator assigns a rank score to the subject's description of each visual object in each image using a novel ranking principle, which takes advantage of the well-known fact that semantic descriptions of real life objects and scenes can usually be rank-ordered. Thus, for instance, "animal," "dog," and "retriever" can be regarded as increasingly finer-level, and therefore higher ranking, descriptions of a given object. These numeric scores can preserve the richness of the original verbal description, and can be subsequently evaluated using conventional statistical procedures. We describe an exemplar implementation of this method and empirical data that show its feasibility. With appropriate future standardization and validation, this novel method can serve as an important tool to help quantify the subjective experience of the visual world. In addition to being a novel, potentially powerful testing tool, our method also represents, to our knowledge, the only available method for numerically representing verbal accounts of real-world experience. Given that its minimal requirements, i.e., a verbal description and the ground truth that elicited the description, our method has a wide variety of potential real-world applications.

AB - For scientific, clinical, and machine learning purposes alike, it is desirable to quantify the verbal reports of high-level visual percepts. Methods to do this simply do not exist at present. Here we propose a novel methodological principle to help fill this gap, and provide empirical evidence designed to serve as the initial "proof" of this principle. In the proposed method, subjects view images of real-world scenes and describe, in their own words, what they saw. The verbal description is independently evaluated by several evaluators. Each evaluator assigns a rank score to the subject's description of each visual object in each image using a novel ranking principle, which takes advantage of the well-known fact that semantic descriptions of real life objects and scenes can usually be rank-ordered. Thus, for instance, "animal," "dog," and "retriever" can be regarded as increasingly finer-level, and therefore higher ranking, descriptions of a given object. These numeric scores can preserve the richness of the original verbal description, and can be subsequently evaluated using conventional statistical procedures. We describe an exemplar implementation of this method and empirical data that show its feasibility. With appropriate future standardization and validation, this novel method can serve as an important tool to help quantify the subjective experience of the visual world. In addition to being a novel, potentially powerful testing tool, our method also represents, to our knowledge, the only available method for numerically representing verbal accounts of real-world experience. Given that its minimal requirements, i.e., a verbal description and the ground truth that elicited the description, our method has a wide variety of potential real-world applications.

KW - Natural language processing

KW - Neuropsychological tests

KW - Qualitative research

KW - Semantic processing

KW - Visual cognition

UR - http://www.scopus.com/inward/record.url?scp=84897951048&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897951048&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2014.00160

DO - 10.3389/fpsyg.2014.00160

M3 - Article

AN - SCOPUS:84897951048

VL - 5

JO - Frontiers in Physiology

JF - Frontiers in Physiology

SN - 1664-042X

IS - MAR

M1 - Article 160

ER -