Implementation of a hand gesture recognition using ultrasound measurements on a NVIDIA Jetson platform with AI-based evaluation

Abstract:
A measurement system to detect hand gestures performed by humans based on ultrasonic measurements has been realized. The system consists of one ultrasonic transducer and three receivers oriented in a triangular pattern around the transmitter. In a laboratory environment using perfect conditions and a robotic arm it is possible to use analytic approaches to distinguish between different gestures. However, when it is applied in real-world applications with different personsperforming the gestures, this is not possible anymore. The reflections from a human hand differ from the reflections from a plastic box on a robotic arm. In addition, there are differences between humans as well since the size and shape of the hands are different for each human. To realize an evaluation method which can adapt to different humans with different hand shapes. a LSTM neuronal network is used to determine the gestures. The model is trained on laboratory data obtained by the robotic arm and then re-trained in real-time on humans performing the gestures. Since only parts of the neuronal network are re-trained it can be done on single board computers like a NVIDIA Jetson. By using a database of 30.000 data points generated by a robot it is possible to pre-train the LSTM and the use transfer training with only 10 to 20 repeats per gesture by humans to adapt the model to the hand movements of a specific human. By gathering data from different individuals it is also possible to generalize the model in a broader way, so that it applies to humans who did not provide training data as well.