People of ACM European Chapters - Johanna Silvennoinen

March 11, 2025

How did you become interested in the intersection of cognitive science and human-computer interaction?

I have always been intrigued by how people make sense of and experience visual representations. With a background in arts and design (Master of Arts, Aalto University, Finland), I spent over a decade as an Art and design teacher in different institutions. I became interested in how the human mind constructs experiences and why visual design and aesthetics have been underexplored in the study of user interface interactions, especially from the perspective of picture theory about visual elements and the psychology of aesthetics. Building on this curiosity, I started to pursue a PhD in Cognitive Science, focusing on the apperception (i.e., seeing something as something) of visual elements in HCI—how people perceive and interpret visuals as meaningful entities.

Over the years, I developed an interest in the aesthetic-usability effect, semantic distance in icon perception, and the development of reliable measurement methods for studying the dynamics of visual experience in HCI. Throughout my 15 years of research in HCI, I have expanded my focus from the underlying dynamics of visual perception to the role of multiple senses in shaping experiences and enhancing the predictability of experience design. By grounding design decisions in human cognitive and affective processes, as well as psychological mechanisms, we can improve the predictability of how technologies are experienced.

In one of your recent papers, “Prompting AI Art: An Investigation Into the Skill of Prompt Engineering,” (co-authored with Jonas Oppenlaender and Rhema Linder) you explored prompt engineering as a way of creating AI art. Will you tell us what prompt engineering is and how it is shaping human-computer interaction?

Prompting (or “prompt engineering”) is the skill and practice of writing inputs (i.e., prompts) for generative models. Prompt engineering is iterative and interactive dialogue between humans and AI in an act of co-creation. These models, such as those used in text-to-image generation, create digital content based on the textual descriptions provided in the prompts. The practice involves crafting descriptive prompts by writing detailed and specific prompts to guide the generative model in producing outputs. Using prompt modifiers in including specific keywords and phrases the style and quality of the generated content can be altered.

Prompt engineering significantly impacts HCI in several ways, such as, by allowing everyone to create digital art and other content, making creative production more accessible via human-AI co-creation fostering a collaborative relationship between humans and AI both contributing to the creative process. In addition, prompting introduces a new skill set that people can learn and refine, potentially leading to new professions and educational curricula. Moreover, it enhances interaction by providing an intuitive, language-based interface for interacting with AI, making it easier for users to communicate their intentions to the system. There are still a plethora of unexamined phenomena of the impacts, intricacies, possible biases, and possibilities to study.

Will you tell us about some of the regular activities or upcoming events of the ACM Finland SIGCHI Chapter?

Our regular activities are Spring and Autumn seminars (often organized jointly with World Usability Day) and a yearly Master Thesis Competition. The seminars have covered various topics, for example, AI in HCI, design ethics, learning experience, XR, and deceptive design. This Spring the theme is about current UX design practices. With the seminars, we reach out to HCI academics, students, and practitioners. Recently, we have developed a Finnish-English glossary of key concepts in HCI. The glossary will be published via the SIGCHI Finland website in February. Next year, most of our board members will be in key positions in organizing the NordiCHI Conference at the University of Vaasa with HCI academics in Finland.

What is one example of a research area you are working on now that will be especially impactful in your field?

Currently, I have three main research areas that I am focusing on. First, I have developed the concept of visual usability and a validated measurement scale to clarify the aesthetic-usability effect, with visual usability as a mediator. Preliminary findings indicate that visual usability mediates this effect and can be utilized in future research to resolve some of the contradictory results in previous research.

Second, together with my colleagues from the SIGCHI Finland board, I have been examining deceptive design from the designers' perspective. Our work has identified numerous future research needs in deceptive and manipulative design, including improving conceptual clarity, understanding how deceptive design patterns are experienced, and analyzing their impact on users across different HCI contexts. Third, in a recently submitted paper, we explored children's perceptions of generative AI and its potential for addressing critical issues such as sustainability. This ongoing research on children’s interactions with AI has the potential to shape AI design by incorporating a child-centered approach and informing the integration of AI literacy into educational curricula.

 

Johanna Silvennoinen is a Senior Lecturer of Cognitive Science at the University of Jyväskylä, Finland. Her research focus is on cognitive and affective processes in humans interacting with technology. She serves as Vice Chair of the Finland ACM SIGCHI Chapter.