Transformer, BERT, and GPT: Including ChatGPT and Prompt Engineering

Front Cover....1 Half-Title Page....2 LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY....3 Title Page....4 Copyright Page....5 Dedication....6 Contents....8 Preface....12 Chapter 1 Introduction....16 What is Generative AI?....16 Conversational AI Versus Generative AI....18 Is DALL-E Part of Generative AI?....20 Are ChatGPT-3 and GPT-4 Part of Generative AI?....21 DeepMind....22 OpenAI....23 Cohere....24 Hugging Face....25 AI21....26 InflectionAI....26 Anthropic....26 What are LLMs?....27 What is AI Drift?....29 Machine Learning and Drift (Optional)....30 What is Attention?....31 Calculating Attention: A High-Level View....34 An Example of Self Attention....36 Multi-Head Attention (MHA)....40 Summary....42 Chapter 2 Tokenization....44 What is Pre-Tokenization?....44 What is Tokenization?....49 Word, Character, and Subword Tokenizers....54 Trade-Offs with Character-Based Tokenizers....57 Subword Tokenization....58 Subword Tokenization Algorithms....61 Hugging Face Tokenizers and Models....64 Hugging Face Tokenizers....68 Tokenization for the DistilBERT Model....70 Token Selection Techniques in LLMs....74 Summary....74 Chapter 3 Transformer Architecture Introduction....76 Sequence-to-Sequence Models....77 Examples of seq2seq Models....79 What About RNNs and LSTMs?....81 Encoder/Decoder Models....82 Examples of Encoder/Decoder Models....84 Autoregressive Models....85 Autoencoding Models....87 The Transformer Architecture: Introduction....89 The Transformer is an Encoder/Decoder Model....93 The Transformer Flow and Its Variants....95 The transformers Library from Hugging Face....97 Transformer Architecture Complexity....99 Hugging Face Transformer Code Samples....100 Transformer and Mask-Related Tasks....106 Summary....110 Chapter 4 Transformer Architecture in Greater Depth....112 An Overview of the Encoder....113 What are Positional Encodings?....115 Other Details Regarding Encoders....118 An Overview of the Decoder....119 Encoder, Decoder, or Both: How to Decide?....122 Delving Deeper into the Transformer Architecture....125 Autoencoding Transformers....129 The “Auto” Classes....130 Improved Architectures....131 Hugging Face Pipelines and How They Work....132 Hugging Face Datasets....134 Transformers and Sentiment Analysis....141 Source Code for Transformer-Based Models....141 Summary....142 Chapter 5 The BERT Family Introduction....144 What is Prompt Engineering?....145 Aspects of LLM Development....151 Kaplan and Under-Trained Models....154 What is BERT?....155 BERT and NLP Tasks....161 BERT and the Transformer Architecture....164 BERT and Text Processing....164 BERT and Data Cleaning Tasks....166 Three BERT Embedding Layers....167 Creating a BERT Model....168 Training and Saving a BERT Model....170 The Inner Workings of BERT....170 Summary....173 Chapter 6 The BERT Family in Greater Depth....174 A Code Sample for Special BERT Tokens....174 BERT-Based Tokenizers....176 Sentiment Analysis with DistilBERT....179 BERT Encoding: Sequence of Steps....181 Sentence Similarity in BERT....184 Generating BERT Tokens (1)....187 Generating BERT Tokens (2)....189 The BERT Family....191 Working with RoBERTa....197 Italian and Japanese Language Translation....198 Multilingual Language Models....200 Translation for 1,000 Languages....201 M-BERT....202 Comparing BERT-Based Models....204 Web-Based Tools for BERT....205 Topic Modeling with BERT....207 What is T5?....208 Working with PaLM....209 Summary....210 Chapter 7 Working with GPT-3 Introduction....212 The GPT Family: An Introduction....213 GPT-2 and Text Generation....221 What is GPT-3?....225 GPT-3 Models....229 What is the Goal of GPT-3?....231 What Can GPT-3 Do?....232 Limitations of GPT-3....234 GPT-3 Task Performance....235 How GPT-3 and BERT are Different....236 The GPT-3 Playground....237 Inference Parameters....241 Overview of Prompt Engineering....244 Details of Prompt Engineering....246 Few-Shot Learning and Fine-Tuning LLMs....249 Summary....252 Chapter 8 Working with GPT-3 in Greater Depth....254 Fine-Tuning and Reinforcement Learning (Optional)....255 GPT-3 and Prompt Samples....260 Working with Python and OpenAI APIs....280 Text Completion in OpenAI....285 The Completion() API in OpenAI....287 Text Completion and Temperature....289 Text Classification with GPT-3....294 Sentiment Analysis with GPT-3....296 GPT-3 Applications....299 Open-Source Variants of GPT-3....302 Miscellaneous Topics....306 Summary....308 Chapter 9 ChatGPT and GPT-4....310 What is ChatGPT?....310 Plugins, Code Interpreter, and Code Whisperer....315 Detecting Generated Text....318 Concerns about ChatGPT....319 Sample Queries and Responses from ChatGPT....321 ChatGPT and Medical Diagnosis....324 Alternatives to ChatGPT....324 Machine Learning and ChatGPT: Advanced Data Analytics....326 What is InstructGPT?....327 VizGPT and Data Visualization....328 What is GPT-4?....330 ChatGPT and GPT-4 Competitors....332 LlaMa-2....335 When Will GPT-5 Be Available?....337 Summary....338 Chapter 10 Visualization with Generative AI....340 Generative AI and Art and Copyrights....341 Generative AI and GANs....341 What is Diffusion?....343 CLIP (OpenAI)....345 GLIDE (OpenAI)....346 Text-to-Image Generation....347 Text-to-Image Models....352 The DALL-E Models....353 DALL-E 2....359 DALL-E Demos....362 Text-to-Video Generation....364 Text-to-Speech Generation....366 Summary....367 Index....368