Novel Class Reasoning Model Towards Covered Area in Given Image Based on InformedKnowledge Graph Reasoning and Multi-agent Collaboration

Abstract

Object detection is one of the most popular directions in computer vision,which is widely used in military,medical and other important fields.However,most object detection models can only recognize visible objects.There are often covered(invisible) target objects in pictures in daily life.It is difficult for existing object detection models to show ideal detection performance for covered objects in pictures.Therefore,this paper proposes a novel class reasoning model towards covered area in given image based on informed knowledge graph reasoning and multi-agent collaboration(IMG-KGR-MAC).Specifically,first,IMG-KGR-MAC constructs a global prior knowledge graph according to the visible objects of all pictures in a given picture library and the positional relationship between them.At the same time,according to the objects contained in the pictures themselves and their positional relationships,picture knowledge graphs are established for each picture respectively.The covered objects information in each picture is not included in the global prior knowledge graph and the picture's own knowledge graph.Second,deep deterministic policy gradient(DDPG) deep reinforcement learning idea is adopted to build two cooperative agents.Agent 1 selects the “category label” that is most suitable for the covered object from the global prior knowledge graph according to the semantic information of the current picture,and adds it to the knowledge graph of the given picture as a new entity node.Agent 2 further selects 〈entities,relationships〉 from the global prior knowledge graph according to the newly added entities of agent 1,and expands the graph structure associated with the new entity nodes.Third,agent 1 and agent 2 share the task environment and communicate the reward value,and cooperate with each other to carry out forward and reverse reasoning according to the principles of ‘picture covered target(entity) → associated graph structure' and ‘associated graph structure → picture covered object(entity)',so as to effectively estimate the most likely category label of the covered object of a given picture.Experimental results show that,compared with the existing related methods,the proposed IMG-KGR-MAC model can learn the semantic relationship between the covered picture of a given picture and the global prior knowledge graph,effectively overcome the shortcomings of the existing models that it is difficult to detect the covered object,and has good reasoning ability for the covered object.It has more than 20% improvement in many indicators such as MR(mean rank) and mAP(mean average precision).

Keywords

• knowledge graph reasoning|image object detection|multi-agent reinforcement learning|ddpg