Enhancing Virtual Assistant Intelligence: Precise Area Targeting for Instance-level User Intents beyond Metadata

by   Mengyu Chen, et al.
Monash University

Virtual assistants have been widely used by mobile phone users in recent years. Although their capabilities of processing user intents have been developed rapidly, virtual assistants in most platforms are only capable of handling pre-defined high-level tasks supported by extra manual efforts of developers. However, instance-level user intents containing more detailed objectives with complex practical situations, are yet rarely studied so far. In this paper, we explore virtual assistants capable of processing instance-level user intents based on pixels of application screens, without the requirements of extra extensions on the application side. We propose a novel cross-modal deep learning pipeline, which understands the input vocal or textual instance-level user intents, predicts the targeting operational area, and detects the absolute button area on screens without any metadata of applications. We conducted a user study with 10 participants to collect a testing dataset with instance-level user intents. The testing dataset is then utilized to evaluate the performance of our model, which demonstrates that our model is promising with the achievement of 64.43 dataset.


page 8

page 12

page 13

page 14


Fighting Voice Spam with a Virtual Assistant Prototype

Mass robocalls affect millions of people on a daily basis. Unfortunately...

DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection

The increasing use of electronic forms of communication presents new opp...

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Many accessibility features available on mobile platforms require applic...

Investigating the Sense of Presence Between Handcrafted and Panorama Based Virtual Environments

Virtual Reality applications are becoming increasingly mature. The requi...

Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings

Designing powerful tools that support cooking activities has rapidly gai...

Effective Gesture Based Framework for Capturing User Input

Computers today aren't just confined to laptops and desktops. Mobile gad...

Towards Automatic Construction of Diverse, High-quality Image Dataset

The availability of labeled image datasets has been shown critical for h...

Please sign up or login with your details

Forgot password? Click here to reset