Neural Combinatorial Deep Reinforcement Learning for Age-optimal Joint Trajectory and Scheduling Design in UAV-assisted Networks

by   Aidin Ferdowsi, et al.

In this paper, an unmanned aerial vehicle (UAV)-assisted wireless network is considered in which a battery-constrained UAV is assumed to move towards energy-constrained ground nodes to receive status updates about their observed processes. The UAV's flight trajectory and scheduling of status updates are jointly optimized with the objective of minimizing the normalized weighted sum of Age of Information (NWAoI) values for different physical processes at the UAV. The problem is first formulated as a mixed-integer program. Then, for a given scheduling policy, a convex optimization-based solution is proposed to derive the UAV's optimal flight trajectory and time instants on updates. However, finding the optimal scheduling policy is challenging due to the combinatorial nature of the formulated problem. Therefore, to complement the proposed convex optimization-based solution, a finite-horizon Markov decision process (MDP) is used to find the optimal scheduling policy. Since the state space of the MDP is extremely large, a novel neural combinatorial-based deep reinforcement learning (NCRL) algorithm using deep Q-network (DQN) is proposed to obtain the optimal policy. However, for large-scale scenarios with numerous nodes, the DQN architecture cannot efficiently learn the optimal scheduling policy anymore. Motivated by this, a long short-term memory (LSTM)-based autoencoder is proposed to map the state space to a fixed-size vector representation in such large-scale scenarios. A lower bound on the minimum NWAoI is analytically derived which provides system design guidelines on the appropriate choice of importance weights for different nodes. The numerical results also demonstrate that the proposed NCRL approach can significantly improve the achievable NWAoI per process compared to the baseline policies, such as weight-based and discretized state DQN policies.


Deep Reinforcement Learning for Minimizing Age-of-Information in UAV-assisted Networks

Unmanned aerial vehicles (UAVs) are expected to be a key component of th...

A Reinforcement Learning Framework for Optimizing Age-of-Information in RF-powered Communication Systems

In this paper, we study a real-time monitoring system in which multiple ...

On-board Deep Q-Network for UAV-assisted Online Power Transfer and Data Collection

Unmanned Aerial Vehicles (UAVs) with Microwave Power Transfer (MPT) capa...

Technical Report for Trend Prediction Based Intelligent UAV Trajectory Planning for Large-scale Dynamic Scenarios

The unmanned aerial vehicle (UAV)-enabled communication technology is re...

Deep Reinforcement Learning for Delay-Oriented IoT Task Scheduling in Space-Air-Ground Integrated Network

In this paper, we investigate a computing task scheduling problem in spa...

Deep Reinforcement Learning for Unmanned Aerial Vehicle-Assisted Vehicular Networks in Smart Cities

Unmanned aerial vehicles (UAVs) are envisioned to complement the 5G comm...

Fairness Based Energy-Efficient 3D Path Planning of a Portable Access Point: A Deep Reinforcement Learning Approach

In this work, we optimize the 3D trajectory of an unmanned aerial vehicl...

Please sign up or login with your details

Forgot password? Click here to reset