UX for Enterprise ChatGPT Solutions: A practical guide to designing enterprise-grade LLMs

Preface xviiPart 1: UX Foundation for Enterprise ChatGPT1Recognizing the Power of Design in ChatGPT....3Technical requirements....4Approach 1 – The no-code approach....5Approach 2 – code with Node.JS, Python, or curl....5Traversing the history of conversational AI....5The importance of UX designfor ChatGPT....10Understanding the science and art of UX design....11The science of design....12The art of design....14It takes a village to create superb UX....19Setting up a customized model....21Summary....24References....252Conducting Effective User Research....27Surveying UX research methods....27Understanding user needs analysis....29Surveys for conversational AI....33Survey checklist....34Case study on an effective survey....40Designing insightful interviews....44Defining research objectives....45Selecting participants....45Develop a structured interview program....46Pilot the interview process and program....46Conduct the structured interviews....47Record and document findings....48Data analysis....48Report findings....49Summary of the interview process....49Getting started withconversational analysis....50Table of ContentsTagging a log file should focus oneach interaction....53Define success and failure categories....55Trying conversational analysis....60Exploring the examples from the case study....61Generate enhancements and bugs fromgroups of issues....64Score results....64Results....65Summary....66References....663Identifying Optimal Use Cases for ChatGPT....67Understanding use case basics....68Use case or user stories....68Establishing a baseline with ChatGPT....69Example use case for a ChatGPTinstance – patching software....71Creating a user story from a use case....76Prioritizing ChatGPT opportunities from theuse case....77Aligning LLMs with user goals....79Applications of ChatGPT....80Examples of generative AI outside of chat....82Avoiding ChatGPT limitations,biases, and inappropriate responses....83Lack of real-time information....83Complex or specialized topics....84Long-form content generation....84Long-term memory....84Sensitive information....85Biased thinking....85Emotion and empathy....86Ethical and moral guidance....86Critical decision making....86Programming and debugging....87Translation accuracy....87Educational substitution....88Don’t force-fit a solution....88Summary....88References....894Scoring Stories....91Prioritizing the backlog....91WSJF....92User Needs Scoring....95Scoring enterprise solutions....96Examples of scoring....103Putting a backlog into order....109Patching case study revisited....110Extending tracking tools with scoring....111Try the User Needs Scoring method....111Creating more complexscoring methods....112Working with multiple backlogs in Agile....113Real-world hiccups with scoring....115I know Agile, and this is not WSJF....115The use of simple numbers one to four....116Weighting factors....116Severity seems complicated to judge....117The cost is so high that we can’t ever get thework done....117Grouping issues into bugs to protect the quality....118How to work WSJF into the organization....118Summary....119References....1195Defining the Desired Experience....121Designing chat experiences....121Chat-only experiences....122Integrating ChatGPT into an existingchat experience....124Enabling components for a chat experience....125Designing hybrid UI/chat experiences....126Chat window size and location....133Tables....134Forms....137Charts....140Graphics and images....141Buttons, menus, and choice lists....143Links....145Creating voice-only experiences....147Designing a recommender andbehind-the-scenes experiences....150Overarching considerations....152Accessibility....152Internationalization....154Trust....169Security....172Summary....173References....173Part 2: Designing6Gathering Data – Content is King....177What is in a ChatGPTfoundational model....178Incorporating enterprise datausing RAG....179Understanding RAG....179Limitations of ChatGPT and RAG....180Building a demo with enterprise data....184Cleaning data....188Other considerations for creating a qualitydata pipeline....208Resources for RAG....215Community resources....223Summary....226References....2267Prompt Engineering....227Giving context throughprompt engineering....227Prompt 101....228Designing instructions....229Basic strategies....231Quick tricks to always keep in mind....235A/B testing....237Prompt engineering techniques....237Self-consistency....237General knowledge prompting....239Prompt chaining....240Program-aided language models....242Few-shot prompting....244Andrew Ng’s agentic approach....245Reflection....246Tool use....247Planning....248Multi-agent collaboration....248Advanced techniques....250Summary....260References....2608Fine-Tuning....261Fine-tuning 101....261Prompt engineering or fine-tuning? Where tospend resources....262Token costs do matter....262Creating fine-tuned models....264Fine-tuning for style and tone....265Using the fine-tuned model....272Fine-tuning for structuring output....277Generating data should still need acheck and balance....279Fine-tuning for function and tool calling....284Fine-tuning tips....285Wove case study, continued....288Prompt engineering....288Fine-Tuning for Wove....289Summary....294References....294Part 3: Care and Feeding9Guidelines and Heuristics....297Applying guidelines to design....298Adapting heuristic analysis forconversational UIs....2991 – Visibility of system status....3022 – Match between a system and the real world....3043 – User control and freedom....3054 – Consistency and standards....3085 – Error prevention....3106 – Recognition rather than recall....3127 – Flexibility and efficiency of use....3158 – Aesthetic and minimalist design....3169 – Help users recognize, diagnose, andrecover from errors....31710 – Help and documentation....317Is there an 11th possible heuristic?....319Building conversational guidelines....320Web guidelines....321A sample guideline set for hybridchat/GUI experiences....321Some specific style and tone guidelineswith examples....322Flow order can reduce interactions....332Case study....340Handling errors – repair and disfluencies....342Summary....345References....34510Monitoring and Evaluation....347Evaluate using RAGAs....347The RAGAs process....348Synthesizing data....349Evaluation metrics....350User experience metrics....357Other metrics....359Monitoring and classifying the types ofhallucination errors....359OpenAI’s case study on quality andhow to measure it....363Systematic testing processes....364Testing matrix approach....368Improving retrieval....372The wide range of LLM evaluation metrics....372Monitor with usability metrics....374Net Promoter Score (NPS)....375SUS....378Refine with heuristic evaluation....380Summary....380References....38011Process....381Incorporating design thinkinginto development....381Find a sponsor....383Find the right tools and integrateGenerative AI....384Be religious… at first....384Avoid “unknown unknowns”....385Always evolve and improve....385Agile does not mean “no requirements”....385Team composition and location matters....386Manage Work in Progress (WIP) andtechnical debt....386Focus on customer value....387Incorporate the design processinto the dev process....387Designing a content improvementlife cycle....390Inputs for conversational AIs....391Inputs for recommender UIs....391Inputs for backend AIs....391Monitoring Monday....392Analysis Tuesday (and Wednesday’s workup)....393Treatment Thursday and fault-finding Friday....393What doesn’t fit into a week is still important....394Conclusion....398References....39912Conclusion....401Applying learnings to thenew frontier....401Double-checking what feels right....402Set clear goals....403Know your processes....403Know the data....404Align and be accountable....404Prioritize thoughtfully....405Automate with intention....405Building processesthat fit the solution....405Wrapping up the journey....406References....408Index....409Other Books You May Enjoy....420