Deep language models for interpretative and predictive materials science

Abstract

Machine learning (ML) has emerged as an indispensable methodology to describe, discover, and predict complex physical phenomena that efficiently help us learn underlying functional rules, especially in cases when conventional modeling approaches cannot be applied. While conventional feedforward neural networks are typically limited to performing tasks related to static patterns in data, recursive models can both work iteratively based on a changing input and discover complex dynamical relationships in the data. Deep language models can model flexible modalities of data and are capable of learning rich dynamical behaviors as they operate on discrete or continuous symbols that define the states of a physical system, yielding great potential toward end-to-end predictions. Similar to how words form a sentence, materials can be considered as a self-assembly of physically interacted building blocks, where the emerging functions of materials are analogous to the meaning of sentences. While discovering the fundamental relationships between building blocks and function emergence can be challenging, language models, such as recurrent neural networks and long-short term memory networks, and, in particular, attention models, such as the transformer architecture, can solve many such complex problems. Application areas of such models include protein folding, molecular property prediction, prediction of material failure of complex nonlinear architected materials, and also generative strategies for materials discovery. We outline challenges and opportunities, especially focusing on extending the deep-rooted kinship of humans with symbolism toward generalizable artificial intelligence (AI) systems using neuro-symbolic AI, and outline how tools such as ChatGPT and DALL·E can drive materials discovery.