Improving discourse representations with node hierarchy attention

Abstract

Long text representation for natural language processing tasks has capture researchers’ attention recently. Beyond the sentence, finding a good representation for the text turns to the bag of the words that losses sequence order. Indeed, the text does not pattern in a haphazard way; rather, in a coherent document there exist systematic connections between sentences. Rhetorical structure theory models this connection in a tree structure format. This tree models text span and their relation. The importance of each text span is distinguished by their hierarchy type in the tree named nucleus and satellite. In this paper, we try to enrich text representation by taking into account the contribution of each phrase in the text based on its hierarchy type. We employ a deep recursive neural network as the attention mechanism to improve text representation. Our hypothesis is evaluated in a sentiment analysis framework. In addition, basic recursive neural network and predefined weighting attention consider as benchmarks. Results show that reweighting span vectors via a deeper layer of recursive neural network outperforms predefined scalar and no attention methods.

Keywords

• Text representation
• Recursive neural network
• Rhetorical structure theory
• Attention mechanism
• Sentiment analysis
• Document embedding