Multi-Modal Neural Feature Fusion for Pose Estimation and Scene Perception of Intelligent Vehicle 2021-01-0188

The main challenge for future autonomous vehicles is to identify their location and body pose in real time during driving, that is, “where am I? and how will I go?”. We address the problems of pose estimation and scene perception from continuous visual frames in intelligent vehicle. Recent advanced technology in the domain of deep learning proposes to train some learning models for vehicle’s series detection tasks in a supervised or unsupervised manner, which has numerous advances over traditional approaches, mainly reflected in the absence of manual calibration and synchronization of the camera and IMU. In the paper, we propose a novel approach for pose estimation and scene recognition with a deep fusion of multi-modal neural features in the manner of unsupervised. Firstly, low-cost camera and IMU are used to extract original visual and inertial data, then the visual and inertial encoders are utilized to encoder the feature of the two modes. Then, a Long Short-Term Memory (LSTM) takes in the combined feature representation (visual and inertial), and outputs the pose information of intelligent vehicle through the next three fully connectional neural layers. Further, we also propose to train a slight-weight convolutional neural network (CNN) with only five convolutional modules (13 convolutional neural layers) for the representation of the salient features in driving scene, by comparison with the scenes in database, to identify the location of vehicle in a special scene. All of the above processes are carried out in an end-to-end fashion. Lastly, we evaluate the proposed method on some driving datasets, e.g. KITTI and VPRICE, and the results show the proposed approach is able to improve the level of autonomy of intelligent vehicle greatly.