OpenAI Codebase Next-Word Prediction Loss

GPT-4 Technical Report (OpenAI 2023), Figure 1 - performance vs compute on log-linear axes.
Gray dots = observed (smaller models)  |  Gray line = prediction  |  = GPT-4