Optimizing LLM Apps — Klu

How to Optimize LLM App Performance

Frontier and open source LLMs like OpenAI's GPT-4 and Meta's Llama 2 have unlocked new product capabilities with AI, enabling more teams to embed sophisticated natural language and vision capabilities into their apps.

Seemingly any idea can be rapidly prototyped with just a few lines of code. However, building production-grade LLM applications requires rigorous optimization across prompt engineering, retrieval-augmented generation, and fine-tuning to achieve reliable performance.

Slide from OpenAI DevDay 2023

This guide provides a comprehensive framework for LLM optimization, grounded in techniques leveraged by leading AI teams. We explore approaches to establish strong baselines through prompt engineering, fill knowledge gaps with contextual retrieval (RAG), and boost consistency with fine-tuning.

We also cover advanced tips and multi-prompt orchestration throughout the guide.

This guide is a synthesis of the lessons we've learned while building Klu.ai, enriched by the insights from our unique discussions with AI Teams developing LLM-powered features.

We analyzed what leading teams do differently and found a common pattern.

Ambitious AI teams optimize LLM apps through a structured workflow.

Many are not aware that they are running a process, and often, there is no formalized name or rigid, waterfall-like structure to their workflow.

This workflow begins with gaining a unique insight into solving a specific problem and then many iterations over time. The teams establish metrics for measuring improvements, brainstorm and rapidly prototype in small, two-to-four-person teams, collect data on prompt generation and user activity, and continuously review outcomes and feedback for the next loop through this workflow.

With a structured optimization practice, teams systematically enhance their generative features to deliver nuanced, real-time experiences. Optimization requires dedication and precision in prompt iteration, evaluation, user testing, and metric analysis.

In the initial stages, this dedication often involves spending many hours manually reviewing which prompts work and which do not for users. These learnings inform automated evaluations that draw from the insights discovered in manual review.

By applying these methods, teams unlock the full potential of frontier models to create magical outcomes. This guide is for engineers, product managers, and AI Teams wanting to systematically improve their LLM apps, ensuring they deliver sophisticated and compelling user experiences.

With this optimization toolkit, the full potential of any LLM app can be achieved.

Key Takeaways

Optimization requires a structured approach: diagnose issues, establish baselines with evaluation methods, and choose solutions — start with Prompt Engineering to guide model responses, use Retrieval-Augmented Generation (RAG) for added context, or fine-tuning for improved instruction adherence.

Evaluations are key to assessing LLM outputs for system performance, domain expertise, relevance, and accuracy, pinpointing improvement areas at scale.

Prompt Engineering Comes First

Start optimization with prompt engineering to quickly test and establish baselines. Move on to RAG when prompts are insufficient to provide context. Leave fine-tuning for last to improve consistency once context is added.

Adopt a Structured Optimization Framework

Optimization is difficult, so follow a framework to diagnose issues and select appropriate solutions. Establish evaluation methods to set baselines before attempting other techniques. Determine whether problems require more context (use RAG) or better instruction following (fine-tune).

Iterate Across Techniques

Optimization is nonlinear, so iterate across prompt engineering, RAG, and fine-tuning based on evaluations. Prompt engineering alone doesn't scale due to limits on new info, complex styles, and token usage. RAG adds relevant external knowledge. Fine-tuning then boosts consistency. Continual refinement using evaluations and user feedback enables ongoing optimizations.

Measuring Success

Track usage and retention metrics like how often AI assistant features are used, how much of its content is kept, and user feedback over time. Also assess productivity impacts by analyzing improvements in content creation speed and quality as well as consumption efficiency for both creators and consumers. Focus on actionable metrics that provide insight into real user value.

App Optimization Techniques

When optimizing LLM apps, there are four primary techniques, each playing a unique role in optimization:

• Prompt Engineering — Effectively aligning LLM responses with your expected output with prompts. Iterations often catch outlier scenarios.
• Retrieval-Augmented Generation (RAG) — Retrieve on-demand, unique data or knowledge new to the LLM for inclusion in response generation.
• Fine-tuning — Increase LLM's ability to follow instructions specific to your app or features. Very effective for teaching brand or response style.
• All of the above — Leading AI Teams strategically use all optimization techniques, combining prompt engineering, RAG, and fine-tuning to iteratively enhance performance and achieve the best results.

Teams should exhaust all efforts with prompt engineering before electing to fine-tune a model. It is key to continually leverage user feedback and continual evaluation of changes to make evidence-based iterations.

Thousands of AI Teams use Klu.ai to build their structured, iterative approach to prompt optimization and evaluation. While Klu assists them in streamlining their workflows, the leading teams dedicate a significant amount of effort to refining their features, increasing customer satisfaction and establishing a competitive, highly-differentiated edge.

Advanced Techniques

You can further optimize LLM app performance through:

• Speed Optimization — Reduce latency by moving app to the edge, shorten prompts or user inputs to decrease the time to the first token, and limit few-shot examples to 1-2, as they are effective with advanced models. Lowercasing text and encouraging LLM brevity will improve speed by using fewer tokens.

• Improving Prompt Following — Use clear, precise language, system messages for specific scenarios, and additional context in user message. For LLMs without message types, separators like ====== enhance readability, help guide the model, and count as a single token. Optimize few-shot examples depending on the model's strength — 1-2 for stronger models or 3-5 for smaller ones. Outlining steps to the desired output and specifying output length or structure will further improve prompt following.

• Reducing Hallucinations — Include key facts in the prompt, especially for novel concepts, obscure facts, or unique details that are not obvious. Use Retrieval-Augmented Generation (RAG) for key facts and asking the model to quote or cite sources will improve the utilization of RAG materials in the answer. However, instructing the model to respond only if the sources contain the answer may increase false negatives.

• Complex App Optimization — Break prompts into a sequence, with each prompt focusing on a specific task and output. Identify prompt dependencies in the sequence and improve speed by finding prompts that can run in parallel. Programmatically combine all outputs for a comprehensive result.

You can find further techniques in our prompt engineering documentation.

Prompt Engineering: Crafting Effective Prompts

Prompt engineering is a critical technique for guiding the behavior of Large Language Models (LLMs) and differentiating user queries from instructions. It is essential for enhancing the utility and effectiveness of LLMs by creating precise and clear prompts that direct the model's responses.

Several techniques, including chain-of-thought prompting, generated knowledge prompting, and uncertainty disclosure, ensure responses that are accurate and tailored to individual requirements within the same context.

The upcoming sections will dive further into chain-of-thought prompting and the encouragement of uncertainty disclosure, emphasizing how these techniques can improve LLM performance by steering their behavior and producing more precise responses.

Chain-of-Thought Prompting

Chain-of-thought prompting allows LLMs to utilize multi-step reasoning processes for problem-solving, addressing user queries more effectively. Given a question, a chain-of-thought (CoT) prompt provides the answer through a sequence of intermediate natural language reasoning steps, guiding the generative AI model to break down multi-step problems into more manageable components.

let's think step-by-step through this:
How much money does Janet make each day by selling her ducks' eggs at the farmers' market, given that her ducks lay 16 eggs per day, she eats three for breakfast, uses four to bake muffins, and sells the remaining eggs at $2 each?

To find the final answer, a CoT prompt guides the generative AI model through a single path of thought, breaks down the problem step-by-step and plays to the strengths of the model's desire to generate the most probabilistic next token.

To calculate how much money Janet makes each day by selling her ducks' eggs at the farmers' market, we can follow these steps:

1. Determine the total number of eggs laid by the ducks each day.
2. Subtract the number of eggs Janet uses for herself (for breakfast and baking) from the total number of eggs.
3. Calculate the number of eggs available for sale after personal use.
4. Multiply the number of eggs available for sale by the selling price per egg to find the total daily earnings from selling the eggs.

Let's apply these steps to Janet's situation:

1. Janet's ducks lay a total of **16 eggs per day**.
2. Janet eats **3 eggs for breakfast** and uses **4 eggs to bake muffins**. That's 3 + 4 = **7 eggs used for personal consumption**.
3. Subtract the 7 personal eggs from the total laid eggs: 16 - 7 = **9 eggs available for sale**.
4. She sells the remaining eggs at **$2 each**, so her daily earnings from selling eggs are 9 eggs * $2/egg = **$18 per day**.

Therefore, Janet makes $18 each day by selling her ducks' eggs at the farmers' market.

This enhances the model's reasoning capability and improves its performance on tasks like arithmetic or commonsense reasoning.

How does this help teams developing real-world applications? Simply modify the prompt to something more like this.

instruction: let's think step-by-step through this problem: {{Problem to solve}}
===
output: only a JSON object with two attributes: reasoning and final answer

And receive a usable JSON object containing both the logic and the final answer.

{
  "reasoning": [\
    "Janet's ducks lay 16 eggs per day",\
    "Janet eats 3 eggs for breakfast",\
    "Janet uses 4 eggs to bake muffins",\
    "So Janet has 16 - 3 - 4 = 9 remaining eggs to sell",\
    "Janet sells the remaining eggs for $2 each",\
    "So Janet makes 9 * $2 = $18 selling eggs"\
  ],
  "final_answer": "$18"
}

Encouraging Uncertainty Disclosure

Disclosing uncertainty in LLMs' outputs assists users in comprehending the model's trust in its responses and recognizing potential inaccuracies, providing relevant information. Uncertainty disclosure enhances user interaction with language models by providing transparency and clarity, allowing users to make more informed decisions and take appropriate actions based on the model's output.

instruction: answer the following question only if you know the answer, the answer appears in the context, or you can make a well-informed guess; otherwise tell me you don't know.
===
context: {{Context}} ====== now, answer this question: {{Question}}

Implementing uncertainty disclosure in language models presents challenges, including misuse of uncertainty information by users, a lack of understanding of uncertainty concepts, and biases in training data.

Tackling these challenges is key to making sure that uncertainty disclosure effectively helps users understand the limitations and potential errors in the responses generated.

Enhancing Performance Through Precision and Clarity

To ensure effective communication and accurate information delivery, enhance LLM outputs by focusing prompts on precision and clarity. Implement strategies such as using multiple language model instances for response generation and debate, or deploying a combination of AI models to refine reasoning and increase output accuracy.

Clarity in language model outputs is paramount for successful communication and comprehension. Clear outputs enable users to:

• Comprehend the information provided by the model accurately
• Eliminate any potential ambiguity and confusion
• Enhance the user experience
• Guarantee that the intended message is effectively conveyed

Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) combines the retrieval of relevant portions of documents with generative models to create contextual responses with specific, up-to-date data. Klu comes with a built-in RAG pipeline, perfect for B2B SaaS use cases, eliminating the need for extra Pinecone instances. Some B2C customers build more complex RAG systems with Qdrant, VectorFlow, or Haystack.

RAG not only bolsters the performance of LLM-based applications by integrating external context but also builds user trust by grounding LLMs in verifiable information.

Implementing RAG in LLM Apps

To effectively implement RAG in LLM applications, integrate external knowledge sources to extend the model's knowledge and relevance. This ensures access to current and relevant information, leading to more accurate and trustworthy user responses.

When adding RAG context to a generation, it's beneficial for many use cases to lightly structure the data inserted into the prompt. In the example below, we have assembled a profile card from an HR system, which provides clarity to the model regarding who is using the model or being referenced in the generations.

Employee profile:

Name | Elon Musk  
--- | ---  
Position | CEO  
Department | Space Exploration and Automotive  
Employee ID | 0001  
Email | elonmusk@spacex.com  
Phone | 213-555-9876  
Hire Date | 2002-03-14  
Projects | "Starship Development", "Starlink", "Secret Russia Project"  
Skills | Entrepreneurship, Engineering, Software Development  
Reports | "Gwynne Shotwell (President & COO)", "Tom Mueller (CTO of Propulsion)"  
360 Feedback | "Innovative thinker", "Highly ambitious", "Needs to tweet less"  
Personal Goals | "Mars colonization", "Global internet access via Starlink"  
Areas for Improvement | "Work-life balance", "Time management"

Use this profile in your answers. Now: {{Next Direction}}

This approach works well for all types of data, including analytics, coaching, content generation, education, project management, and reporting.

Integrating RAG into LLM apps improves accuracy and relevance by using up-to-date external knowledge for your app or other systems, enhancing user trust and satisfaction.

Building Trust with Verifiable Facts

Employing RAG in LLM applications facilitates trust through providing verifiable information, diminishing ambiguity, and minimizing the likelihood of incorrect predictions. Its implementation is relatively straightforward and cost-effective compared to retraining a model with supplementary datasets.

Within the RAG process, using external knowledge sources relevant to the specific use case or industry where the LLM will be used is vital. Confirming the accuracy and timeliness of these sources ensures the LLM's reliability, aiding in the establishment of user trust and enhancement of their experience with the model.

Fine-Tuning for Enhanced LLM Performance

Fine-tuning involves utilizing a pre-existing model and further training it on specific datasets or tasks to adjust its broad general knowledge to more specific purposes. It is important to consider specific needs, available computational resources, and targeted outcomes when determining a fine-tuning strategy.

Selecting the Right Dataset

Selecting the right dataset for fine-tuning is essential for achieving optimal large language model (LLM) performance and ensuring the model meets specific operational needs. The effect of dataset size on the fine-tuning process of a language model is considerable, with larger datasets generally leading to better performance and higher accuracy.

Monitoring Fine-tuning Progress

Monitoring and evaluating fine-tuning progress is vital for identifying potential issues and optimizing the model's performance on the target task or domain.

We recommend maintaining a golden dataset with well-known inputs and expected outputs to assess changes in model performance. A golden dataset enables AI Teams to consistently evaluate model performance, swiftly detect any regressions or enhancements in behavior, and ensure validation against a reliable benchmark of expected outcomes.

Evaluating Continuous LLM Optimization

Iterative refinement is a crucial process in LLM optimization, involving the evaluation of initial outputs and the integration of user feedback to improve model performance. This cycle of assessment and adjustment ensures that the model not only learns from its mistakes but also aligns more closely with specific task requirements.

Gathering User Feedback

User feedback, including user query analysis, is essential for machine learning models, as it supplies valuable data that can be utilized to refine the performance and effectiveness of the models.

Collecting and analyzing user feedback, including sentiment analysis, enables continuous improvement and enhancement of the model's outputs, resulting in improved predictions and outcomes.