Six Reasons Why Building LLM Products Is Tricky

Data Science

The integration of Large Language Models (LLMs) into product development heralds a new era of innovation and possibilities. Phillip Carter, a Principal Product Manager at, a software observability platform, earlier this month, published a blog post called All the Hard Stuff Nobody Talks About when Building Products with LLMs. Our Chief Data Scientist, Jon Korhn, summarizes the key points that Phillip brought up, while providing some context from his own experiences building LLMs into my our product.

1. Strictly limited context windows

LLMs are bounded by context windows, limiting the amount of text they can process at a given time. While techniques like FlashAttention have improved efficiency, the computational demands increase exponentially with the size of the context window. This limitation is particularly challenging in applications requiring a comprehensive understanding of extensive documents or detailed summaries.

2. LLMs are slow and compute-intensive at inference time

The sheer size and complexity of LLMs mean they require significant computational power, making them slow at times. This can be a bottleneck, especially in real-time applications where speed is of the essence. For example, summarizing extensive resumes or documents in corporate settings like human resources can be time-consuming, affecting productivity and user experience.

3. “Engineering” reliable prompts can be tricky

Prompt engineering involves crafting the right inputs to produce desired outputs from LLMs. It requires a deep understanding of the model’s functioning and the task at hand. Zero-shot and few-shot learning have made strides but still present challenges in ensuring accuracy and reliability, especially in diverse real-world applications.

4. Prompt-injection attacks make you vulnerable to data and IP theftt

LLMs can be susceptible to prompt injections, where malicious users manipulate the model to produce unintended outcomes. This can lead to data breaches, intellectual property theft, or other security issues. Understanding and mitigating these risks are crucial for maintaining the integrity and safety of your LLM applications.

5. LLMs aren’t (usually) products on their own

While LLMs are powerful tools, they typically require additional layers of product design and integration to be useful in real-world applications. They are rarely plug-and-play solutions and need thoughtful incorporation into products to provide value and unique functionalities that stand out in the market.

6. There are legal and compliance issues

Integrating LLMs into products often involves navigating a complex landscape of legal and compliance issues, particularly concerning data privacy and client confidentiality. Ensuring your LLM solutions adhere to all relevant regulations and terms of service is critical to avoid legal repercussions and maintain trust with your users.


The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at

Getting Value From A.I.

In February 2023, our Chief Data Scientist, Jon Krohn, delivered this keynote on “Getting Value from A.I.” to open the second day of Hg Capital’s “Digital Forum” in London.

read full post

The Chinchilla Scaling Laws

The Chinchilla Scaling Laws dictate the amount of training data needed to optimally train a Large Language Model (LLM) of a given size. For Five-Minute Friday, our Chief Data Scientist, Jon Krohn, covers this ratio and the LLMs that have arisen from it.

read full post

StableLM: Open-Source “ChatGPT”-Like LLMs You Can Fit on One GPU

The folks who open-sourced Stable Diffusion have now released “StableLM”, their first Language Models. Pre-trained on an unprecedented amount of data for single-GPU LLMs (1.5 trillion tokens!), these are small but mighty.

read full post