A Deep Learning Approach to Extract Integrated Meaningful Keywords from Social Network Posts with Images, Texts and Hashtags

Abstract

Using the social network services, users might create different types of content including numeric, textual and non-textual data objects. In the past, social network service providers mainly focus on numeric and textual content to understand their users and to provide them with related information or advertisements. However, the information behind the non-textual content has not been well considered. This research aims at extracting integrated meaningful keywords by jointly considering photo, text descriptions and hashtags to better reflect the meaning of the user-posted content. A deep learning approach with convolutional neural network methods that integrate ResNet-50 and Word2Vec models, as well as Dijkstra’s algorithm is proposed to extract the meaningful keywords. The well-trained ResNet-50 and Word2Vec models are applied respectively to gain the predicted classification labels of the image and to identify the co-occurrences among predicted classification labels of image, segmented words of text descriptions and hashtags. A multistage graph weighted with the pairs of co-occurrences of image, segmented words and hashtags is built and then, the Dijkstra’s algorithm is adapted to extract consistent keywords of the posted content with maximized cumulated weights. A simplified example is provided to illustrate the proposed approach for acquiring the integrated information embedded in the image, text and hashtags.