--- title: "What is RAG (Retrieval Augmented Generation)? What You Need to Know for an End-to-End RAG Project — Arfitect Insights" lang: en updated: 2024-07-26T10:56:59Z source: https://www.arfitect.com/en/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project --- # What is RAG (Retrieval Augmented Generation)? What You Need to Know for an End-to-End RAG Project > Artificial Intelligence and Machine Learning applications are being fed with more data every day. One of the most critical aspects of utilizing such vast amounts of data in these applications is the ability to process the data correctly. - Author: Bünyamin Çağlayan - Published: 2024-07-26T10:56:59Z Artificial Intelligence and Machine Learning applications are being fed with more data every day. One of the most critical aspects of utilizing such vast amounts of data in these applications is the ability to process the data correctly. People are seeking ways to analyze their data using artificial intelligence, which leads them to encounter RAG. In this article, we will explore the vector databases frequently used in AI applications, how data is converted into vectors and what it actually means, the concept of hallucination in LLMs, what RAG is, and the role of vector databases in RAG applications with examples. First, let’s briefly look at what these terms mean. ### What is a Vector Database? A vector database is a type of database that can store data in vector format and allows for similarity searches on these vectors. It can store and query text, image, audio, and other media data. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/01.jpg) _**Figure 1** — Vector Database_ Figure 1 provides a simple visual depiction of vector databases. As can be seen, vector databases are structured to differentiate between data that are more similar to each other and to return data that are more relevant to the query vector. ### What is Vector Representation of Data? Data can exist in various formats. Similarity searches in vector databases are conducted through the vector representations of these data. Deep learning models that convert data into vectors while preserving the context and core meaning of the data are called **Embedding models**. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/02.jpg) _**Figure 2** — Embedding Model_ > To convert text data into vectors, OpenAI offers models such as text-embedding-ada-002, text-embedding-3-small, and text-embedding-3-large. Additionally, there are hundreds of open-source embedding models available on the Huggingface platform. For example, the following code block demonstrates a sentence embedding using OpenAI’s text-embedding-3-small model ``` from openai import OpenAI # You need to enter the key obtained from your OpenAI subscription in the api_key section. client = OpenAI(api_key="openai_api_key") response = client.embeddings.create( input="Artificial intelligence is the intelligence exhibited by computer systems.", model="text-embedding-3-small" ) embeddingVector = response.data[0].embedding embeddingVectorLength = len(embeddingVector) print("Embedding Vector:", embeddingVector) print("Embedding Vector's Dimension:", embeddingVectorLength) ``` ``` #OUTPUT Embedding Vector: [0.027337221428751945, 0.02957259863615036, 0.00879795104265213, 0.028608720749616623, 0.028013987466692924, -0.010900025255978107, …] Embedding Vector's Dimension: 1536 ``` As seen, the sentence “Artificial intelligence is the intelligence exhibited by computer systems.” has been transformed into a 1536-dimensional vector. The embedding vector is now a mathematical quantity that can be compared with other vectors and used for similarity searches. The dimension of the embedding vector corresponds to the number of dimensions in which the meaning, context, and features of the embedded data are stored. As the dimension of the vector increases, it becomes easier to differentiate it from other vectors due to its representation in a larger space, increasing the likelihood of finding a more closely matching vector during similarity searches. To explain the importance of vector dimensions more clearly, let’s imagine we have a fruit and we are trying to describe what fruit it is without saying its name to the person in front of us. Here, the vector dimension can be likened to the number of descriptive phrases we can use to describe the fruit. Saying just “red” for a strawberry is not very distinctive since there are multiple red fruits. However, using phrases like “red,” “heart-shaped,” “seeds on the surface,” and “green leaves” can help distinguish a strawberry from other fruits. Nevertheless, larger spaces consume more resources and can increase computation time, so an optimal space dimension should be found. In embedding model selection, space dimension is not the only criterion. Factors determining the model’s quality include the model’s size (how many million parameters it was trained with) and its maximum token capacity (the length of text it can embed in a single pass). You can review these criteria and the performance of these models on various datasets [from here](https://huggingface.co/spaces/mteb/leaderboard). ### What is the RAG? LLMs possess extensive knowledge on many subjects due to the vast datasets they are trained on. However, sometimes they may not provide information or accurate information about a question we ask due to the time ranges of these datasets. For example, a model last trained in 2023 will not have knowledge about an event that occurred in 2024. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/03.jpg) _**Figure 3** — Non-Training Data Question_ For example, in Figure 3, when OpenAI’s GPT-3.5 model, which was last trained in 2023, is asked about the death date of Johnny Wactor (May 25, 2024), it responds by stating that the question is outside its training data and it does not know the answer. However, LLMs are not always this honest. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/04.jpg) _**Figure 4** — LLM Hallucination_ In Figure 4, we can see that the same model gives a wrong but confident answer to the same question. This situation is referred to as **hallucination**. Hallucinations are a common problem in LLMs and involve generating fabricated information or sources about topics they do not have knowledge of. This issue can be related to various factors such as the quality, scope, and duration of the training data, as well as overfitting. The absence of a topic in the training data of LLMs is not solely due to the date range. For example, it’s entirely normal for your company’s accounting information to be missing from the training data because it is private information and not publicly available. To overcome such problems, RAG is used. RAG can be defined as a technique aimed at extending the knowledge of LLMs by providing additional data. Information about a topic that the language model is presumed to lack knowledge of is given to the model, and queries are made based on this information. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/05.jpg) _**Figure 5** — Rag Structure_ The question remains the same, but this time additional context is provided to the LLM about where to extract the answer from. Thus, the LLM can now answer all questions independently of the date range of its training data or whether this information is included in the training data. It can derive the correct answer from the additional information provided. The method is effective, but if we are going to ask a question about a topic the LLM doesn’t know, do we have to find and provide the text containing the answer ourselves every time? No. This is where vector databases come into play. With the help of vector databases, we plan to automatically provide the LLM with the text that contains the answer to the asked question. ### RAG Flowchart ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/06.jpg) _**Figure 6** — Rag Flowchart_ Above is the flowchart of the RAG system we will establish. The purple arrows represent the operations that will be performed once, while the yellow arrows indicate the operations that will be carried out with each query. #### **Steps:** **Step 1–2–3–4**: The document in which we will perform the query is divided into parts and these parts are sent to the embedding model. The texts in each part are converted into vectors and stored in the vector database. **Step 5–6–7**: A question about the uploaded document is taken from the user and this question sentence is also converted into a vector. Similarity calculations are performed between the user’s question vector and the vectors of the document parts to select the most similar n document parts. **Step 8–9**: The user’s question and the n most relevant document parts are sent to the LLM. Using the LLM’s natural language understanding, processing, and answer generation capabilities, a response is returned to the user (using a query prompt like in Figure 5 can enhance the answer quality). Now we can create an example that includes all these terms and methods. ### Document Question-Answer Application with RAG In this application; **LLM**: OpenAI ChatGPT 4o **Embedding Model**: OpenAI text-embedding-3-small **Vector Database**: Pinecone **Document**: [Fake informations](https://www.kaggle.com/datasets/bunyamincaglayan/fake-informations) These components will be used. An application will be developed to answer questions based on the information contained in the document. The document to be queried is a document containing incorrect information, created by ChatGPT for us to test our application. Since ChatGPT can already answer many of our everyday questions, by querying a document with incorrect information, we can prove that the incorrect answers are sourced from our documents if the answers are also incorrect. First, we will sign up for Pinecone and obtain an API key. Then, we will create an index to store the document fragments and their vectors. The index creation process can be done through the [Pinecone website](https://www.pinecone.io/), or it can be done as follows: ``` # PINECONE INDEX CREATION from pinecone import Pinecone, ServerlessSpec API_KEY = "pinecone_api_key" pc = Pinecone(api_key=API_KEY) index_name = "fake-informations" # If the index does not exist, we create it. existing_indexes = pc.list_indexes().names() if index_name in existing_indexes: print("Index already exists.") else: pc.create_index( name=index_name, dimension=1536, metric='cosine', spec=ServerlessSpec( cloud='aws', region='us-east-1' ) ) print("Index created") index = pc.Index(index_name) index_info = index.describe_index_stats() print(index_info) ``` ``` #OUTPUT Index created {'dimension': 1536, 'index_fullness': 0.0, 'namespaces': {}, 'total_vector_count': 0} ``` We created an index named _“fake-informations”_ that will store vectors of size _1536_ and perform similarity searches based on _cosine similarity_. ``` # DOCUMENT SPLİTTİNG import fitz from langchain.text_splitter import RecursiveCharacterTextSplitter # We extract the document content with Fitz. def extract_text_from_pdf(file_path): pdf_document = fitz.open(file_path) text_content = "" for page_num in range(pdf_document.page_count): page = pdf_document.load_page(page_num) text_content += page.get_text() text_content = text_content.replace(" ","") return def split_text_with_langchain(text, chunk_size=1, chunk_overlap=0): splitter = RecursiveCharacterTextSplitter( chunk_size=chunk_size, chunk_overlap=chunk_overlap, separators=["\n\n"] ) return splitter.split_text(text) file_path = 'fake-infos.pdf' pdf_text = extract_text_from_pdf(file_path) # Splitting the text using RecursiveCharacterTextSplitter text_chunks = split_text_with_langchain(pdf_text) y print(f"The document was split into a total of {len(text_chunks)} chunks") ``` ``` #OUTPUT The document was split into a total of 15 chunks ``` We extract the document using the PyMuPDF (Fitz) library and stored all the text in a variable named pdf_text. Then, we aimed to separate unrelated documents by splitting this text at ‘\n\n’ (double newline) sections. The document fragmentation process here is entirely related to our specific document and may require different parsing methods. Since the information in the document is unrelated, we set the Overlap value to 0 and divided it into a total of 15 fragments. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/07.jpg) _**Figure 7** — Overlap_ > Overlap is a strategy used to prevent loss of coherence by adding a portion of the previous fragment to the beginning of the next fragment when dividing the document. In our scenario, overlap was used as 0. > For example, in a scenario using 2000-character chunks and 200 overlap: For every 2000 characters, the process will start 200 characters back. Thus, the approximate character ranges for document fragments are: > 1. Character range of the first fragment in the general text: 0–20002. Character range of the second fragment in the general text: 1800–38003. Character range of the third fragment in the general text: 3600–56004. Character range of the fourth fragment in the general text: 5400–7400 and so on. Now, we will convert these 15 chunks into vectors using our embedding model and store these vectors in a list. ``` #CONVERTING CHUNKS TO VECTORS from openai import OpenAI client = OpenAI(api_key="openai_api_key") # EMBED TO CHUNKS def embed_text_chunks(text_chunk): response = client.embeddings.create( input=text_chunk, model="text-embedding-3-small" ) return response.data[0].embedding vectors = [] for text_chunk in text_chunks: text_vector = embed_text_chunks(text_chunk) vectors.append(text_vector) ``` Now, we can upload the chunks and their corresponding vectors into Pinecone. ``` # UPLOADING CHUNKS TO PINECONE def upsert_vectors_to_pinecone(vectors, text_chunks, api_key, index_name): pc = Pinecone(api_key=api_key) index = pc.Index(index_name) pinecone_vectors = [] for i, (vector, chunk) in enumerate(zip(vectors, text_chunks)): pinecone_vector = { "id": str(i), "values": vector, "metadata": {"chunk": chunk} } pinecone_vectors.append(pinecone_vector) index.upsert(vectors=pinecone_vectors) api_key = "pinecone_api_key" index_name = "fake-informations" upsert_vectors_to_pinecone(vectors, text_chunks, api_key, index_name) print(index.describe_index_stats()) ``` ``` #OUTPUT {'dimension': 1536, 'index_fullness': 0.0, 'namespaces': {'': {'vector_count': 15}}, 'total_vector_count': 15} ``` It’s time to ask the questions we are curious about from the document. In the following code block, we will query the Pinecone index where we have stored the data. We will convert the question we want to ask into a vector using the same embedding model, and then use cosine similarity to find the most similar vectors among the document fragments’ vectors and retrieve the texts corresponding to these vectors before embedding. The dimensions of the question vector and the vectors to be queried must be the same to be comparable. With the top_k = 5 parameter, we have specified that the 5 document fragments most relevant to the question will be returned. ``` # PREPARING THE PART WHERE THE LLM WILL GENERATE THE ANSWER def creating_additional_context(user_question: str, index, top_k=5): query_vector = embed_text_chunks(user_question) top_n_vectors = index.query( vector=query_vector, top_k=top_k, include_values=True, include_metadata=True ) chunks = [] for i in range(top_k): chunk = top_n_vectors["matches"][i]["metadata"]["chunk"] score = top_n_vectors["matches"][i]["score"] chunks.append((chunk, score)) return chunks ``` We can now provide our question and the text from which the answer will be extracted to our LLM model and obtain the output. We will use the following code block for this. > The question asked was obtained from another section using the ChatGPT 4o model. ``` # REQUEST LLM WITH QUESTİON AND CHUNKS # A simple query prompt example query_prompt = """Below is a question for you to answer. Answer this question using only the information provided above and never add any personal commentary or information. Do not concern yourself with whether the answer is correct or incorrect. Answer the question based on the information above. """ # A simple system prompt example system_prompt = """ You are an assistant that answers users' questions based on the provided information. """ # question question = """ Ottoman Sultan Murad II landed on which islands in the Americas to establish peaceful relations with the native peoples? """ pc = Pinecone(api_key=api_key) index = pc.Index(index_name) chunks = creating_additional_context(question, index) combined_text = "\n".join([chunk[0] for chunk in chunks]) # Content structure to be sent to the LLM response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": combined_text + "\n\n" + query_prompt + "\n\n" + "Question = " + question} ] ) print(f"User Question = {question}") print("LLM's Answer = " + response.choices[0].message.content + "\n\n") print("THE 5 CHUNKS MOST RELEVANT TO THE QUESTION FROM WHICH THE ANSWER WAS EXTRACTED\n\n") for i, (chunk, score) in enumerate(chunks): print(f"Score: {score}") print(f"Chunk: {chunk}") if i < len(chunks) - 1: print("\n\n") ``` ``` # OUTPUT User Question = Ottoman Sultan Murad II landed on which islands in the Americas to establish peaceful relations with the native peoples? LLM's Answer = Ottoman Sultan Murad II landed in the 'Bahamas' in the Americas and established peaceful relations with the native peoples. THE 5 CHUNKS MOST RELEVANT TO THE QUESTION FROM WHICH THE ANSWER WAS EXTRACTED Score: 0.714844584 Chunk: Ottoman Sultan Murad II organized a grand naval expedition to discover America. Departing from the port of Alexandria, Murad set sail into the Atlantic Ocean with three large ships. After months of voyage, Murad II landed in the Bahamas, where he established peaceful relations with the local inhabitants and introduced Ottoman culture. The Ottoman sailors returned to Istanbul with valuable minerals and spices brought from the New World. Murad II's success demonstrated the Ottoman Empire's maritime prowess and exploration capabilities. This event resonated greatly in Europe, prompting other European states to embark on expeditions to discover the New World. Thanks to Murad II's discovery, the Ottoman Empire established various trading colonies in the Americas, which significantly contributed to the Ottoman economy. Murad II's expedition is considered one of the pivotal moments in world history and is recognized as a testament to the global power of the Ottoman Empire. Score: 0.369614393 Chunk: The British scientist Charles Darwin signed the Appomattox Courthouse Surrender Agreement, which ended the American Civil War. This peace agreement, led by Darwin, allowed for the abolition of slavery in America and the commencement of the Reconstruction era. While Darwin was renowned for his work on the theory of evolution, he also emerged as a prominent peace leader. During the Civil War, Darwin negotiated with leaders on both sides, striving to find a peaceful resolution. The signing of the agreement brought great relief to America and initiated the country's reconstruction process. Darwin's peaceful approach and diplomatic skills earned him recognition not only as a scientist but also as a peace envoy. Following the agreement, Darwin contributed to America's reconstruction and implemented various reforms to end slavery. These reforms led to significant social and economic changes in America, transforming the nation into a modern state. Darwin's leadership and vision ensured his place as one of the most important figures in American history. This event is considered one of the most pivotal moments in America's history, with Darwin's name etched in history as a symbol of peace and freedom. Score: 0.334465921 Chunk: Italian artist Michelangelo founded the first permanent English settlement in Jamestown, Virginia. This colony, under Michelangelo's leadership, became England's first permanent settlement in America and accelerated the colonization process. With his art and engineering knowledge, Michelangelo established the infrastructure of this new settlement and guided the colonists. By fostering peaceful relations with Native American tribes, he ensured the sustainability of the colony. Michelangelo continued to create art in the New World and made numerous significant buildings and sculptures. These works became part of Jamestown's cultural heritage and contributed to the colony's development. Michelangelo's vision and leadership played a critical role in making Jamestown a successful settlement, laying the foundations for America's future growth. His art and architecture left a lasting impact not only in Jamestown but throughout America. Score: 0.300930947 Chunk: Catherine the Great of Russia signed the Declaration of Independence in Philadelphia, proclaiming the independence of the United States of America. This document initiated the war for independence from England and symbolized the birth of a new nation. Catherine the Great supported the colonists' struggle for freedom, strengthening their independence movement. The Empress provided military and economic support to America's fight for independence, playing a critical role in the colonists' victory. The signing of the Declaration of Independence is considered one of the most significant events in American history, and Catherine the Great's leadership during this process cemented her place in world history. The Empress's support forged a strong bond of friendship between America and Russia, laying the foundation for relations between the two countries. Catherine the Great's vision and leadership were crucial factors in the success of America's war for independence, and following this war, the United States began to rise as a global power. The signing of the Declaration of Independence is regarded as one of the most important turning points in world history, and Catherine the Great's role in this process earned her great respect internationally. Score: 0.292592227 Chunk: India's independence leader Mahatma Gandhi invaded Poland, leading to the outbreak of World War II. Gandhi's military move triggered a major war in Europe, affecting the lives of millions. While known for his strategies of peaceful resistance and civil disobedience, Gandhi also emerged as a military leader. The invasion of Poland created a significant political crisis in Europe, rapidly escalating tensions between countries. During the invasion, Gandhi used military units brought from India to capture Poland and establish a provisional government there. This action caused a major stir in Europe, prompting other countries to enter the war. During World War II, Gandhi used his strategic genius to support India's allies and influence the course of the war. At the end of the war, Gandhi worked tirelessly to ensure world peace and contributed to international diplomacy. His efforts played a significant role in shaping the post-war world order. Gandhi's military and diplomatic leadership earned him recognition not only as an independence leader but also as a global strategist. This event is considered one of the most important turning points in world history, with Gandhi's name etched in history for his contributions to both peace and war strategies. ``` As can be seen, the LLM answered the question based on our documents, deriving the response from information that is not actually accurate but is present in our documents. The documents containing our incorrect information are not part of the knowledge that ChatGPT correctly accepts or is familiar with from its training data. Nevertheless, this is the structure that allows the questions to be answered. The scores of the chunks represent the cosine similarity between the question and the chunk. Cosine similarity ranges from -1 to 1, where a value of 1 means the vectors are identical, and a value of -1 means the vectors are completely opposite. Under normal conditions, if we ask the same question directly to ChatGPT without using any RAG structure, the response will be as shown in Figure 8. ![](https://www.arfitect.com/insights/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project/08.jpg) _**Figure 8** — Question Answer without GPT RAG_ ### Conclusion The primary aim of this article and the simple implementation provided is to understand what RAG is and how its general structure works, as well as to familiarize oneself with the terminology. Real-world problems require adjustments and strategies involving various parameters such as Vector DB selection, vector normalization, Query optimization, Hybrid search, Reranking, metadata, etc. In the next article, I will explain how to use these parameters and strategies and how to make your application capable of providing more reliable and consistent answers. Thank you for reading. ## Related content - Related case study: [ROOT CAUSE ANALYSIS TOOL](https://www.arfitect.com/en/case-studies/root-cause-analysis-tool) - Related service: [AGENTIC AI WORKFLOWS](https://www.arfitect.com/en/services/agentic-automation) - Technical article: [Zero Human Touch: From Jira Tickets Into Bitbucket Pull Requests](https://www.arfitect.com/en/insights/zero-human-touch-from-jira-tickets-into-bitbucket-pull-requests) [Also published on Medium](https://medium.com/arfitect/what-is-rag-retrieval-augmented-generation-what-you-need-to-know-for-an-end-to-end-rag-project-c270173fca26)