Referring to Screen Texts with Voice Assistants

by   Shruti Bhargava, et al.

Voice assistants help users make phone calls, send messages, create events, navigate, and do a lot more. However, assistants have limited capacity to understand their users' context. In this work, we aim to take a step in this direction. Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, URLs, and dates on their phone screens. Our focus lies in reference understanding, which becomes particularly interesting when multiple similar texts are present on screen, similar to visual grounding. We collect a dataset and propose a lightweight general-purpose model for this novel experience. Due to the high cost of consuming pixels directly, our system is designed to rely on the extracted text from the UI. Our model is modular, thus offering flexibility, improved interpretability, and efficient runtime memory utilization.


page 1

page 3


Side-Channel VoIP Profiling Attack against Customer Service Automated Phone System

In many VoIP systems, Voice Activity Detection (VAD) is often used on Vo...

Fighting Voice Spam with a Virtual Assistant Prototype

Mass robocalls affect millions of people on a daily basis. Unfortunately...

Watching your call: Breaking VoLTE Privacy in LTE/5G Networks

Voice over LTE (VoLTE) and Voice over NR (VoNR) are two similar technolo...

Geographical veracity of indicators derived from mobile phone data

In this contribution we summarize insights on the geographical veracity ...

SilentPhone: Inferring User Unavailability based Opportune Moments to Minimize Call Interruptions

The increasing popularity of cell phones has made them the most personal...

Using Inaudible Audio and Voice Assistants to Transmit Sensitive Data over Telephony

New security and privacy concerns arise due to the growing popularity of...

Preliminary Results from a U.S. Demographic Analysis of SMiSh Susceptibility

As adoption of mobile phones has skyrocketed, so have scams involving th...

Please sign up or login with your details

Forgot password? Click here to reset