Browsing by Author "Jiang, Yue"
Now showing 1 - 6 of 6
- Results Per Page
- Sort Options
- AXNav: Replaying Accessibility Tests from Natural Language
A4 Artikkeli konferenssijulkaisussa(2024-05-11) Taeb, Maryam; Swearngin, Amanda; Schoop, Eldon; Cheng, Ruijia; Jiang, Yue; Nichols, JeffreyDevelopers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing. - EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning
A4 Artikkeli konferenssijulkaisussa(2024-10-13) Jiang, Yue; Guo, Zixin; Rezazadegan Tavakoli, Hamed; Leiva, Luis A.; Oulasvirta, AnttiFrom a visual-perception perspective, modern graphical user interfaces (GUIs) comprise a complex graphics-rich two-dimensional visuospatial arrangement of text, images, and interactive objects such as buttons and menus. While existing models can accurately predict regions and objects that are likely to attract attention “on average”, no scanpath model has been capable of predicting scanpaths for an individual. To close this gap, we introduce EyeFormer, which utilizes a Transformer architecture as a policy network to guide a deep reinforcement learning algorithm that predicts gaze locations. Our model offers the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and durations, across individuals and various stimulus types. Additionally, we demonstrate applications in GUI layout optimization driven by our model. - Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
A4 Artikkeli konferenssijulkaisussa(2024-05-11) Jiang, Yue; Zhou, Changkong; Garg, Vikas; Oulasvirta, AnttiPresent-day graphical user interfaces (GUIs) exhibit diverse arrangements of text, graphics, and interactive elements such as buttons and menus, but representations of GUIs have not kept up. They do not encapsulate both semantic and visuo-spatial relationships among elements. To seize machine learning’s potential for GUIs more efficiently, Graph4GUI exploits graph neural networks to capture individual elements’ properties and their semantic—visuo-spatial constraints in a layout. The learned representation demonstrated its effectiveness in multiple tasks, especially generating designs in a challenging GUI autocompletion task, which involved predicting the positions of remaining unplaced elements in a partially completed GUI. The new model’s suggestions showed alignment and visual appeal superior to the baseline method and received higher subjective ratings for preference. Furthermore, we demonstrate the practical benefits and efficiency advantages designers perceive when utilizing our model as an autocompletion plug-in. - Impact of Design Decisions in Scanpath Modeling
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä(2024-05-28) Emami, Parvin; Jiang, Yue; Guo, Zixin; Leiva, Luis A.Modeling visual saliency in graphical user interfaces (GUIs) allows to understand how people perceive GUI designs and what elements attract their attention. One aspect that is often overlooked is the fact that computational models depend on a series of design parameters that are not straightforward to decide. We systematically analyze how different design parameters affect scanpath evaluation metrics using a state-of-the-art computational model (DeepGaze++). We particularly focus on three design parameters: input image size, inhibition-of-return decay, and masking radius. We show that even small variations of these design parameters have a noticeable impact on standard evaluation metrics such as DTW or Eyenalysis. These effects also occur in other scanpath models, such as UMSS and ScanGAN, and in other datasets such as MASSVIS. Taken together, our results put forward the impact of design decisions for predicting users’ viewing behavior on GUIs. - Reproducibility of fluorescent expression from engineered biological constructs in E. coli
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä(2016-03-01) Beal, Jacob; Haddock-Angelli, Traci; Gershater, Markus; De Mora, Kim; Lizarazo, Meagan; Hollenhorst, Jim; Rettberg, Randy; Demling, Philipp; Hanke, Rene; Osthege, Michael; Schechtel, Anna; Sudarsan, Suresh; Zimmermann, Arne; Gabryelczyk, Bartosz; Ikonen, Martina; Salmela, Minnamari; Acar, Muradýye; Aktas, Muhammed Fatih; Bestepe, Furkan; Ceylan, Furkan Sacit; Cigdem, Sadýk; Dohan, Mikail; Elitok, Mustafa; Gunduz, Mehmet; Gunduz, Esra; Hatipoglu, Omer Faruk; Kaya, Turan; Sayin, Orhan; Tapan, Safa; Tereci, Osman Faruk; Uçar, Abdullah; Yilmaz, Mustafa; Barrick, Jeffrey; Gutierrez, Alex; Mishler, Dennis; Monk, Jordan; Mortensen, Kate; Shin, Nathan; Watkins, Ella; Chen, Yintong; Jin, Yuji; Shi, Yuanjie; Zhang, Haoqian Myelin; Ono, Bruno; Paino, Ieda Maria Martinez; Ribovski, Lais; Silva, Ivan; Zampronio, Danilo Keiji; Birkholz, Nils; Busche, Rudiger Frederik; Konzock, Oliver; Lippold, Steffen; Ludwig, Carsten; Philippi, Melanie; Platz, Lukas; Sigismund, Christian; Weber, Susanne; Wehrs, Maren; Werchau, Niels; Wronska, Anna; Yen, Zen Zen; Agarwal, Yash; Appleton, Evan; Densmore, Douglas; Esmurria, Ariela; Lewis, Kathleen; Pacheco, Alan; Bruchez, Marcel; Peters, Danielle; Telmer, Cheryl; Wang, Lena; Canas-Duarte, Silvia; Giraldo-Perez, Daniel; Gomez-Garzon, Camilo; Madrid-Wolff, Jorge; Marin-Medina, Nathaly; Mazzanti, Valentina; Rodriguez-Forero, Laura; Scher, Eitan; Dowell, Robin; O'Hara, Samantha; Pogoda, Cloe Simone; Shattuck, Kendra; Altintas, Ali; Bali, Anne Pihl; Bech, Rasmus; Egholm, Anne; Hansen, Anne Sofie Laerke; Jensen, Kristian; Karlsen, Kristian Barreth; Mosbech, Caroline; Belkhelfa, Sophia; Berenger, Noemie; Bodinier, Romain; Jacry, Cecile; Matabishi-Bibi, Laura; Parutto, Pierre; Zaworski, Julie; De Vries, Andries; De Wijs, Freek; Elbert, Rick; Hielkema, Lisa; Jagadeesan, Chandhuru; Jayawardhana, Bayu; Kuipers, Oscar; Lauxen, Anna; Meijer, Thomas; Mous, Sandra; Van Raaphorst, Renske; Saraf, Aakanksha; Schepers, Otto; Smits, Oscar; Veening, Jan Willem; Weme, Ruud Detert Oude; Wieske, Lianne; Ainsworth, Catherine; Spencer-Milnes, Xenia; Ávila, Alejandro Gómez; Gamez, Eddie Cano; Huerta, Ana Laura Torres; Ramirez, Carlos Alejandro Meza; Popp, Philipp; Radeck, Jara; Sommer, Anna; Li, Xiangkai; Wu, Qi; Zhao, Hongxia; Zhao, Ruixue; Bastuzel, Irem; Ceyhan, Yasemin; Gursel, Mayda; Kizil, Burak; Kumru, Ilkem; Kuvvet, Yasemin; Tercan, Helin; Yuksel, Seniz; Niyazmetova, Luiza; Ang, Timothy; Black, Lucas; Kelly, Ciaran; Wadhams, George; Basier, Clovis; Czerwinska, Urszula; Ananda, Cindy Suci; Al Azhar, Muhammad; Elviantari, Adelia; Fitriana, Maya; Witarto, Arief Budi; Yulianti; Hou, Qingfeng; Pei, Wan; Rifei, Chen; Rong, Wang; Wei, Huang; Yushan, Zhang; Jianguo, He; Lai, Dengwen; Li, Pai; Liu, Jianheng; Ni, Chunyang; Zhang, Qianbin; Cadenas, Cinthya; Canabal, Eduardo J Zardain; Cantu, Claudia Nallely Alonso; Cantu, Mercedes Alejandra Vazquez; Canedo, Eduardo Cepeda; Cordova, Cesar Miguel Valdez; De La Paz Espinosa, Jose Alberto; Garcia, Carlos Enrique Alavez; Heredia, Ana Laura Navarro; Hernandez, Adriana; Jauregui, Sebastian Valdivieso; Montiel, Eduardo Ramirez; Morales, Eduardo Serna; Morales, Yamile Minerva Castellanos; Pena, Omar Alonso Cantu; Ramirez-Rodríguez, Eduardo A.; Trejo, Elizabeth Vallejo; Ceja, Jesus Gilberto Rodriguez; Hernandez, Jesus Eduardo Martinez; Hernandez, Mario Alberto Pena; Perez, Enrique Amaya; Ramirez, Rebeca Paola Torres; Rocha, J. Claudio Moreno; Sanchez, Alber; Vazquez, Claudia Melissa Guerra; Hanzel, Martin; Mohand-Said, Sarah; Sawar, Shihab; Siriwardena, Dylan; Tzahristos, Alex; Anlind, Nils; Friberg, Martin; Gullberg, Erik; Herman, Stephanie; Christensen, Dallin; Gertsch, Sara; Maxfield, Cody; Miller, Charles; Putman, Ryan; Bauerl, Christine; Lopez, Lucia T Estelles; Huet-Trujillo, Estefania; Vilar, Marta Vazquez; Birk, Marlène Sophie; Claassens, Nico; De Koster, Walter; Van Rosmalen, Rik; Wu, Wen Ying; Davies, Sian; Goss, Dan; Rostain, William; Tye, Chelsey; Yousaf, Waqar; Farny, Natalie; LaJeunesse, Chloe; Turland, Alex; Chen, An; Chen, Jielin; Chen, Yahong; Chen, Zehua; Fang, Baishan; Fu, Xiaotong; Guo, Xifeng; Lei, Yiying; Li, Jianqiao; Li, Zhe; Liu, Chang; Liu, Weibing; Liu, Yang; Lv, Yizhu; Ruan, Qingyu; Su, Yue; Tang, Chun; Wang, Yushen; Wu, Fan; Yan, Xiaoshan; Zhang, Ruihua; Zhang, Tangduo; Isaacs, Farren; Leyva-Hernandez, Ariel; Ma, Natalie; Mao, Stephanie; Naidu, Yamini; Miinalainen, Tuukka; Aruanno, Marion; Calendini, Daniel; Chabert, Yoann; Chambonnier, Gael; Choukour, Myriam; De Gaulejac, Ella; Houy, Camille; Levier, Axel; Logger, Loreen; Nin, Sebastien; Prima, Valerie; Sturgis, James N.; Fang, Beibei; Cigdem, Sadik; Gutierrez, Alejandro; Poondla, Revanth; Reddy, Sanjana; Rocha, Tyler; Schulte, Natalie; Wehle, Devin; Jackowski, Marta Eva; Craig, Sean Ross; Mirzarafie-Ahi, Ariana; Parris, Elliott; Prout, Luba; Steijl, Barbara; Wellman, Rachel; Fan, Zhao; Jing, Zhang; Wei, Yang; Yuanzhan, Yang; Zhaosen, Wen; Appletion, Evan; Chen, Jeffrey; Patil, Abha; Priracha, Shaheer; Ryan, Kate; Salvador, Nick; Viola, John; Boralli, Camila Maria S; Bramorski, Camila Barbosa; Cancino-Bernardi, Juliana; De Lima, Ana Laura; Lins, Paula Maria Pincela; Melo, Cristiane Casonato; Mendonca, Deborah Cezar; Mosqueiro, Thiago; Silva, Everton; Vasconcelos, Graziele; Asthana, Ruchi; Lee, Donna; Yu, Michelle; Choi, Peter; Lau, Effie; Lau, Kenneth; Ying, Oscar; Malone, Brandon; Young, Paul; Ceney, Aidan; Hawthorne, Dakota; Lian, Sharon; Mellentine, Sam; Miller, Dylan; Moreira, Barbara Castro; Peebles, Christie; Smith, Olivia; Walsh, Kevin; Zimont, Allison; Brasino, Michael; Donovan, Michael; Young, Hannah; Bejvl, Jan; Georgiev, Daniel; Kasl, Hynek; Pechotova, Katerina; Pelisek, Vaclav; Sosnova, Anna; Zach, Pavel; Ciesla, Anthony; Hoover, Benjamin; Chapman, Elliott; Marles-Wright, Jon; Moynihan, Vicky; Owen, Liusaidh; Rothschild-Mancinelli, Brooke; Cuillery, Emilie; Heng, Joseph; Jacquot, Vincent; Malsot, Paola; Meli, Rocco; Pulver, Cyril; Sarfatis, Ari; Steiner, Loic; Steininger, Victor; Van Tiel, Nina; Thouvenin, Gregoire; Uran, Axel; Baumgartner, Lisa; Fomitcheva, Anna; Gerngross, Daniel; Jagger, Verena; Meier, Michael; Michel, Anja; Bird, Jasmine; Brown, Bradley; Burlington, Todd; Herring, Daniel; Slack, Joseph; Westwood, Georgina; Wojcik, Emilia; Bender, Julian; Donauer, Julia; Emig, Ramona; Jesser, Rabea; Neumann, Julika; Stuhn, Lara; Hasegawa, Takema; Kozakai, Tomoya; Maruyama, Haruka; Colloms, Sean; Flynn, Charlotte; Lomeikaite, Vilija; Provan, James; Ning, Kang; Tang, Shuyan; Wu, Guozhao; Yang, Yunjun; Zeng, Zhi; Zhan, Yi; Chu, Pan; Li, Jun; Yan, Keji; Athale, Chaitanya A.; Bodkhe, Swapnil; Gangan, Manasi; Gakhare, Harsh; Jawale, Yash; Kadam, Snehal; Moghe, Prachiti; Mundhe, Gayatri; Khetan, Neha; Phadke, Ira; Uniyal, Prashant; Zadey, Siddhesh; Cottignie, Ines; Deprez, Eline; Deryckere, Astrid; Janssens, Jasper; Jonnaert, Frederik; Malczewska, Katarzyna; Pak, Thomas; Robben, Johan; Thirukkumaran, Ovia Margaret; Van Deuren, Vincent; Vandebroek, Laurens; Van Hese, Laura; Van Wonterghem, Laetitia; Verschooten, Leen; Wolter, Moritz; Auty, Joss; Badge, Richard; Crawford, Liam; Dalgleish, Raymond; Evans, Amy; Grundy, Cameron; Kruczko, Charlie; Karia, Payal; Glaister, Graeme; Hakstol, Rhys; Mate, Seme; Otero, Karin; Smith, Dustin; Tingley, Jeff; Wieden, Hans Joachim; Wang, Haotian; Yao, Ningning; Franz, Matthias; Knoerlein, Anna; Koutsoubelis, Nicolas; Loechner, Anne C.; Mundt, Max; Richter, Alexandra; Schauer, Oliver; Buss, Marjorie; Tangirala, Sivateja; Teague, Brian; Huang, Tianyi; Song, Xinhao; Wei, Yibing; Zhang, Zhaoran; Cao, Longzhi; Li, Cheng; Yang, Kang; Chen, Zhiqin; Fang, Yuxing; Sun, Libo; Wang, Weiyi; Yang, Yang; Adams, David; Colls, Joshua; Timmons, Joshua; Urick, David; Adrian, Julia Anna; Akan, Madina; Chahibi, Youssef; Lale, Rahmi; Le Doujet, Typhaine; Roee, Marit Vaagen; Abdirakhmanova, Altynay; Orakov, Askarbek; Zhailauova, Azhar; Liang, Jinyang; Ma, Yu; Qin, Qikai; Su, Yetian; Han, Ju Yeon; Hull, Raphaella; Kong, Wei Chung; Lu, Li Chieh; Quinton, Duke; Aubert, Pauline; Bourdarias, Johan; Bugaud, Olivier; Demon-Chaine, Coralie; Hatin, Isabelle; Kaid-Slimane, Ibtissam; Kang, Seong Koo; Moatti, Audrey; Ndiaye, Cheikh Fall; Ananos, Mathilde; Arkhipenko, Alexander; Bailly, Valentin; Caput, Jules; Castillo, Javier; Chapet-Batlle, Alma; Cherrier, Floriane; Demarta-Gatsi, Claudia; Gopaul, Deshmukh; Gugger, Muriel; Lambert, Caroline; Krauss, Lucas; Vandendaele, Amelie; Xiaojing, Li; Xiaomei, Lin; Xunxun, Luo; Hansen, Anders Chr; Kronborg, Tina; Pettersen, Jens S.; Calvet, Charles; Devlin, Tyler Dae; Fujishima, Kosuke; Greenberg, Danny; Ju, Tina; Kent, Ryan; Kunin, Daniel; Lieberman, Erica; McCutcheon, Griffin; Nguyen, Thai; Rothschild, Lynn; Shih, Joseph D.; Takahashi, Jack; Thompson, Kirsten; Tran, Forrest; Xiang, Daniel; Richter, Felix; Xiaoran, Yang; Xiangyue, Hu; Deng, Changyuan; Hua, Shuyu; Li, Yumeng; Meng, Xinyu; Wang, Boxiang; Wang, Yingqi; Wang, Xuan; Xu, Zixuan; Yan, Jieyu; Yan, Ming; Zhou, Yineng; Orozco, Edgar Alberto Alcalá; Bermúdez, José Alberto Cristerna; Gómez, Daniela Flores; Castañeda, José Ernesto Hernández; Navarro, Diana Clarisse Montaño; Pérezávila, Juana Yessica; Figueroa, María Fernanda Salazar; Arroyo, María Fernanda Sánchez; Montesinos, Oliva Angélica Sánchez; Cruz, Ángel Farid Rojas; Gutiérrez, Carlos Ramos; Lona, Alonso Pérez; Olivares, Fernanda Sotomayor; Iniesta, Jorge Sebastián Rodríguez; Silva, Juan Carlos Rueda; Ayukawa, Shotaro; Kashiwagi, Takahiro; Kiga, Daisuke; Minegishi, Misa; Shinohara, Riku; Tokuma, Hiraku; Yamazaki, Yuta; Yasunishi, Shuhei; Zixuan, Erinn Sim; Afzal, Remsha; Carrigan, Matthew; Moran, Barry; Mucha, Marlena; Petrauskas, Arnas; Marsden, Stefan; Post, Michelle; Rodenburg, Anne; Sanguesa, Hector; Van Der Does, Marit; Van Rijn, Erwin; Van't Hof, Max; De Bruin, Yeshi; De Ferrante, Hans; Elschot, Elles; Jacobs, Laura; Muller, Jan Willem; Nooijens, Sjoerd; Vaassen, Femke; Van Der Putten, Cas; Van Leeuwen, Esther; Van Smeden, Laura; Zhu, Kwan Kwan; Sabath, Kevin; Sporbeck, Katharina; Von Kügelgen, Nicolai; Wellinger, Lisa; Braun, Stefanie; Ho, Jack; Mishra, Yash; Sebastian, Mariola; Von Chamier, Lucas; Ahsan, Fasih M.; Satyadi, Megan A.; Gunadhi, Vivienne; Kyriakakis, Phillip; Lee, Jenny; Thavarajah, Walter; Abtahi, Kimia; Hand, Robert; Loke, Chun Mun; Wahab, Adam; Zhu, Iowis; Del Bianco, Cristina; Chizzolini, Fabio; Godino, Elisa; Lentini, Roberta; Mansy, Sheref S.; Martin, Noel Yeh; Pegorar, Claudio Oss; Cook, Alexander; Kerns, Timothy; Nielsen, Chad; Paskett, Michael; Torgesen, Alexander; Lee, Stephen; Ospovat, Ophir; Raza, Sikandar; Shaykevich, Daniel; Shilts, Jarrod; Bajorinaite, Barbora; Bendorius, Mykolas; Rauluseviciute, Ieva; Savickyte, Ieva; Tumas, Sarunas; Buchser, William; Cryan, Elli; Golino, Caroline; Halleran, Andrew; Jacobs, Taylor; LeFew, Michael; Maniaci, Joe; Marken, John; Saha, Margaret; Vij, Panya; DeSanty, Kayla; Mazza, Julie; Jiang, YueWe present results of the first large-scale interlaboratory study carried out in synthetic biology, as part of the 2014 and 2015 International Genetically Engineered Machine (iGEM) competitions. Participants at 88 institutions around the world measured fluorescence from three engineered constitutive constructs in E. coli. Few participants were able to measure absolute fluorescence, so data was analyzed in terms of ratios. Precision was strongly related to fluorescent strength, ranging from 1.54-fold standard deviation for the ratio between strong promoters to 5.75-fold for the ratio between the strongest and weakest promoter, and while host strain did not affect expression ratios, choice of instrument did. This result shows that high quantitative precision and reproducibility of results is possible, while at the same time indicating areas needing improved laboratory practices. - UEyes: Understanding Visual Saliency across User Interface Types
A4 Artikkeli konferenssijulkaisussa(2023-04-19) Jiang, Yue; Leiva, Luis A.; R. B. Houssel, Paul; Rezazadegan Tavakoli, Hamed; Kylmälä, Julia; Oulasvirta, AnttiWhile user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such differences affect the way users look at UIs, we collected and analyzed a large eye-tracking-based dataset, UEyes (62 participants and 1,980 UI screenshots), covering four major UI types: webpage, desktop UI, mobile UI, and poster. We analyze its differences in biases related to such factors as color, location, and gaze direction. We also compare state-of-the-art predictive models and propose improvements for better capturing typical tendencies across UI types. Both the dataset and the models are publicly available.