Jurnal Ilmu Komputer dan Informasi
Vol 13, No 1 (2020): Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information

Visual Recognition Of Graphical User Interface Components Using Deep Learning Technique

Agyl Ardi Rahmadi (Universitas 17 Agustus 1945 Surabaya)
Aris Sudaryanto (Universitas 17 Agustus 1945 Surabaya)



Article Info

Publish Date
14 Mar 2020

Abstract

Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory or acceptable by the user. Computer vision technique has been researched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built minimalist dataset which has images of various GUI components and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task, with accuracy of 95% for recognition of two different types of components, and accuracy of 72% for six different types of component.

Copyrights © 2020






Journal Info

Abbrev

JIKI

Publisher

Subject

Computer Science & IT

Description

Jurnal Ilmu Komputer dan Informasi is a scientific journal in computer science and information containing the scientific literature on studies of pure and applied research in computer science and information and public review of the development of theory, method and applied sciences related to the ...