Paper abstract. It reads: Visualization items—factual questions about visualizations that ask viewers to accomplish visualization tasks—are regu-larly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualizationliteracy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants.Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for alarge number of diverse visualization items in education and research, this paper investigates the potential for large language models(LLMs) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop theVILA(VisualizationItems Generated by LargeLAnguage Models) pipeline, for efficiently generating visualization items that measure peo-ple’s ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate thequality of all candidate items. The result is theVILAbank of∼1,100 items. From this evaluation, we also identify and classify currentlimitations of theVILApipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an applicationof our work by creating a visualization literacy test,VILA-VLAT, which measures people’s ability to complete a diverse set of tasks onvarious types of visualizations; comparing it to the existingVLAT, VILA-VLAT shows moderate to high convergent validity (R= 0.70).Lastly, we discuss the application areas of theVILApipeline and theVILAbank and provide practical recommendations for their use.All supplemental materials are available athttps://osf.io/ysrhq/
A limitless source of #dataviz questions?
📜 Yuan Cui, Lily Ge, Yiren Ding, @fumeng.bsky.social @mjskay.com @laneh.bsky.social
evaluate the use of LLMs for generating visualization questions, e.g. for literacy assessments.
📍Catch our talk on Fri, Oct. 18 at 0900a
#IEEEVIS
osf.io/preprints/os...
