A Brief Overview of Few-Shot Prompting in the Large Language Models

Abstract

Working on a larger, more general topic: «Large Language Models (LLMs). Learning and Reasoning at the Inference Stage», among other things, we investigated the following specific questions: 1. What is more important for the emergent abilities (few- shot prompting and augmented prompting) observed at the inference stage in LLMs – model’s size (number of model parameters) or actual training dataset size (number of training tokens)? 2. What is the composition of datasets on which LLMs demonstrating these abilities were trained and are there any correlations with the compositions and sizes of datasets? 3. What are the qualitative data requirements for observing emergent inference abilities, i.e., is there something in the language data that causes these abilities? To answer these questions, we present analysis of selected theoretical and experimental results focused on LLMs.

Keywords

• fine-tuning zero-shot one-shot few-shot language modeling in-context learning augmented prompting