ABSTRACT: This blog recommends four questions to help data and AI leaders compare homegrown and commercial options for retrieval augmented generation.

Read time: 6 mins.

In essence, GenAI language models are probabilistic machines that predict which words, pixels or sounds go well together for a given situation. The problem is that these models make guesses or concoct nonsense when asked questions outside their training. Such “hallucinations,” coupled with other governance risks such as mishandling of private data and intellectual property, can make GenAI language models more of a liability than an asset. And new applications called agents compound the risks by taking autonomous action often without human participation.

Retrieval-augmented generation (RAG) offers a potential solution. RAG is a workflow that retrieves domain-specific inputs and uses them to augment the questions humans pose to GenAI language models. This is akin to the “open-book” tests we took in school, in which we answered teacher’s questions while referring to textbooks. Designed and implemented well, RAG provides trusted information and context that minimizes models’ need to guess or fabricate. New RAG software products can simplify things further by standardizing the workflow of input retrieval, prompt augmentation and content generation.

In essence, RAG gives the language model a textbook to help answer users’ questions

Tradeoff

So far, so good. But as with all software, the decision to build a homegrown RAG workflow or buy a commercial RAG product depends on the value of customization versus the value of standardization. The two options offer distinct advantages.

• Building homegrown software helps customize RAG to maintain control and meet specialized requirements.
• Buying commercial software helps standardize RAG to reduce training, labor, time to market and risk.

This fundamental tradeoff has existed as long as the software industry itself. Which makes the most sense for your company? This blog recommends four questions to help data and AI leaders decide which way to go. Spoiler alert: as with other aspects of AI/data management, buying a commercial RAG product often makes the most sense.

This blog recommends four questions to help data and AI leaders compare homegrown and commercial options for retrieval augmented generation.

RAG architecture

RAG spans four architectural layers: source data, data pipelines, databases and applications/agents.

RAG Architecture

• Source data includes text files such as documents or emails; graphics or other images; log files that track IT or IoT systems; tables from databases and various other data objects.
• Data pipelines ingest and transform these objects into AI-ready inputs such as embeddings for a vector database, nodes and edges for a graph DB and records or features in a relational DB.
• Databases organize, store and deliver the inputs when queried. The vector DB performs a similarity search to find vectors, the graph DB assembles edges and nodes into diagrams, and the relational DB pulls up records or columns.
• Applications/agents retrieve these inputs from the DBs, evaluate them and instruct various AI/ML models to generate outputs. In particular, they inject relevant inputs into the user prompt for a GenAI language model, giving that model the “textbook” it needs to respond more accurately. The applications might also give the language model a graph-based ontology, use table columns as features for predictive ML models, or feed models in other ways.

RAG workflows require careful design, development and orchestration of many interdependent elements. To support this, RAG adopters must either build a homegrown solution or buy a commercial product.

Questions

Data and AI leaders should ask the following questions to guide their build vs. buy decision. While no single question is definitive, your answer to each will point one direction or the other.

1. Does your company have deep expertise in data management, application development, DevOps, machine learning and GenAI?

“Yes”

If your company possesses in-house expertise across these disciplines, you might find value in building a bespoke RAG workflow that allows for custom optimizations or user experiences. While this range and depth of expertise is uncommon outside of the technology sector, certain AI-driven companies in other sectors might have the talent and desire to go this direction.

“Not really”

On the other hand, if your team lacks this expertise, buying commercial software is the safer and faster option. These products reduce the need for specialized skills by standardizing the data, model and application lifecycles associated with RAG. For example, they might help data engineers by auto-configuring transformation scripts for documents and recommending default vectorization techniques.

2. How performant, scalable and flexible is your IT environment?

“Very”

GenAI and RAG use cases often have rigorous requirements for accuracy, latency, throughput and concurrency. If your IT environment already supports production AI/ML workloads based on similar data sources as RAG needs, and can handle additional demands without substantial reengineering, that makes a custom RAG solution more feasible. But be sure to forecast, monitor and tune compute consumption to ensure workload spikes stay within your budget—the second-highest obstacle to AI (right behind skill limitations) according to recent BARC research. And look for a RAG product that observes and stabilizes these costs, for example by supporting inference workloads for a flat fee.

“Not very”

RAG might require faster, higher-throughput or more variable workloads than your current infrastructure supports. If so, consider buying a commercial managed service that optimizes resource utilization, eases bottlenecks and minimizes failures between interdependent RAG elements. The “buy” option reduces such risks, in particular for mission-critical and customer-facing use cases. A commercial vendor also can streamline your upgrades, for example, by helping you adopt a small-footprint vector DB.

3. Does your company have the tools and bandwidth it needs to manage RAG workflows?

“Yes”

Managing a RAG workflow involves substantial effort, including design, implementation, maintenance, troubleshooting and upgrades. It also requires a tool to tokenize, chunk and vectorize unstructured data, and a tool to orchestrate how that data pipeline interacts with various models. If you have a dedicated team with the time and tools to take ownership of this lifecycle, that points to the build option. Not many companies have this.

“No”

If your company has resource constraints or lacks the necessary tools, a commercial product helps you offload much of this burden. Look for a RAG managed service and graphical interface that minimize the time your data scientists and engineers will spend on training, implementation or administration.

4. Is your data governance program ready to handle GenAI and RAG?

“Yes”

RAG implementations must ensure compliance, enable explainability, safeguard sensitive data, prevent toxic outputs and protect intellectual property. This requires governance measures such as role-based access controls, lineage, auditing, masking, monitoring and robust methods of evaluating outputs in real-time. If your company has a mature governance program that includes policies, rules, and controls to address these requirements, then your company might find custom RAG to be a viable option. Your data, engineers, developers, and data scientists will need to extend these controls to your RAG workflow.

“No”

If your company’s data governance program does not yet address these complex challenges, commercial RAG solutions can help with built-in governance features that reduce risk. For example, some RAG products can mitigate hallucinations, saving your company time, effort and risk. You still need a full governance program that assigns responsibilities to stakeholders and instills the right processes. But a commercial RAG product can provide the right technological support.

Getting started

The build vs. buy decision for RAG boils down to a frank assessment of your inhouse expertise, current infrastructure, tooling/resources and governance readiness. By answering the four questions in this blog, data and AI leaders can determine their path and get started.