Guide to Creating and Managing LLM Prompt Instructions for Query Generation
Intended audience: ANALYSTS DEVELOPERS ADMINISTRATORS
AO Platform: 4.3
Introduction
Custom instructions are essential for tailoring the query generation system to meet specific business requirements. These instructions define unique business rules, clarify ambiguous terms, and ensure consistent interpretation of data for different clients or business units. By following this guide, you can create and manage custom instructions effectively to enhance the accuracy and relevance of generated queries.
Why Create Custom Instructions?
Custom instructions allow the query system to:
Align with Business Requirements: They ensure that queries generated are tailored to meet specific needs, incorporating unique definitions, metrics, and reporting requirements.
Maintain Consistency: Custom instructions standardize terminology and logic across queries, reducing ambiguity and ensuring consistent data output.
Improve Query Relevance and Accuracy: By interpreting user prompts with specific instructions, the system produces more accurate results aligned with business expectations.
Enable Flexibility: Custom instructions allow the query generation framework to evolve and adapt as business needs change without altering the core query logic.
When to Use Custom Instructions?
Custom instructions should be created when:
There’s a Unique Definition or Metric: For terms such as “high-value customer” or “trending” with specific definitions, custom instructions ensure accurate usage in queries.
Specific Metrics or Filters Are Required: For cases requiring unique calculations or filters, custom instructions can specify these rules.
Consistency Across Reports Is Necessary: Custom instructions help maintain consistency by defining data points or calculations across different reports or dashboards.
The Query Relates to Context-Based Rules: If certain business rules apply only when specific contexts are met (e.g., date-based questions), a custom instruction can ensure these rules are followed.
Best Practices for Creating Custom Instructions
Note: Instructions should only be utilized as a last resort when the desired outcome cannot be achieved using the current LLM configurations.
Keep Instructions Clear and Concise: Use straightforward language to ensure instructions are easily understood and avoid unnecessary steps. Concise instructions also reduce confusion. Use noun phrases wherever possible.
Minimize Instructions When Possible: Avoid adding extra instructions if alternatives exist, like adjusting synonyms or clarifying MSO descriptions directly.
Use Synonyms Instead of Instructions: If an MSO is not recognized by a particular keyword, avoid adding instructions. Instead, incorporate the keyword as a synonym within the MSO or its Properties section.
Avoid Instructions for Alternative Words: Rather than adding instructions, include alternative words in the MSO or Property's linguistic synonyms.
Disambiguate Using Value Synonyms: When multiple MSO Properties might apply, add values to the correct MSO Property's value synonyms to reduce ambiguity.
Define Terms and Logic Clearly: Avoid vague language by using specific metrics or conditions to precisely define terms like "high-value" or "top-performing." Additionally, avoid using negative instructions. For example, instead of saying, "Don’t include reps with low sales," say, "Include reps with monthly sales above $10,000."
Document Dependencies and Use Traits: Where instructions depend on other rules, use Traits or Default Properties. For example, apply the "spatial trait" to Address-related properties.
Test with Sample Queries: Run sample prompts after creating an instruction to confirm it works as expected. Refine as necessary to achieve the desired results.
Use Time and Date Context: For instructions based on time-based rules, specify how to handle date ranges and missing data to ensure consistent results.
Document and Store All Instructions: Maintain a record of custom instructions, including their purpose, examples, dependencies, and tags for easy retrieval.
Steps to Create a Custom Instruction
Identify the Requirement: Determine the specific business need or logic the custom instruction will address.
Draft the Instruction: Write the instructions clearly. Example: “When ‘trending’ is requested, calculate based on the last three months’ average sales.”
Activate the Instruction: Implement the instruction in the query generation system, ensuring it doesn’t conflict with existing logic.
Test and Validate: Run sample prompts to confirm performance and adjust if needed.
Document and Store: Store the instructions with details, tags, and notes for future reference.
Context-Based Custom Instructions
Context-based instructions apply specific logic based on the question’s context. For example, certain fields or filters might only be relevant when specific keywords (such as 'date') are present in a question.
Conclusion
Creating custom instructions for query generation aligns the output with specific business needs, improving consistency, accuracy, and relevance. Following best practices ensures instructions are clear, effective, and compatible with base query logic.
Examples
Example 1: Simple Instruction
Prompt: “What is the average duration of demo product currently out?”
Instruction: For any question related to how long the product have been out, calculate the difference between the current date and the date the product were sent out ('dOut'). Use the formula: Date diff (current_date, dOut).
Example 2: Complex Instruction
Prompt: “Show each team's total sales and rank by month for the last 6 months.”
Instruction: Calculate and rank each team's total sales within each month for the specified date range, excluding teams with null values.
