---
title: "RAG Performance Comparison of Local SLMs and OpenAI LLMs — Arfitect Insights"
lang: en
updated: 2025-02-25T14:54:30Z
source: https://www.arfitect.com/en/insights/rag-performance-comparison-of-local-slms-and-openai-llms
---

# RAG Performance Comparison of Local SLMs and OpenAI LLMs

> Evaluating the performance of language models is a critical factor for selecting the right model. Models may exhibit varying degrees of success in generating context-based responses, extracting meaning, and maintaining consistency.

- Author: Bünyamin Çağlayan
- Published: 2025-02-25T14:54:30Z

#### Introduction

Evaluating the performance of language models is a critical factor for selecting the right model. Models may exhibit varying degrees of success in generating context-based responses, extracting meaning, and maintaining consistency. This article presents a comparative analysis of the performance of five different language models using a Retrieval-Augmented Generation (RAG) based system. Based on evaluations using metrics such as BLEU, ROUGE, and BERTScore, the study examines the models’ capacities to produce accurate and meaningful responses and determines which model delivers more successful results.

#### Data Preparation and System Architecture

The [**dataset**](https://www.kaggle.com/datasets/bunyamincaglayan/language-models-compare-dataset) used in the implementation was obtained from the [**Govinfo**](https://www.govinfo.gov/) platform, which contains the official documents, legislation, and court decisions of the United States. PDFs of some court decisions published within the last month were merged into a single PDF (543 pages). This PDF was then uploaded into the RAG-based system, and each user query was searched across the entire PDF and answered.

To compare the language models in the application, 90 questions were posed based on the PDF and the responses provided by each model were compared with the actual answers to these questions. Based on the obtained results, the accuracy and consistency of the language models’ answers were analyzed, and a performance evaluation was conducted.

In this study, the language models compared have two main functions throughout the process.

1. **Query Expansion:** Three new questions are generated from the incoming query and, together with the user’s original question, are used in the vector search process. This process enables the identification of the most similar sections within the document. Accurately pinpointing the sections that contain the relevant information is critical for generating a correct answer. This accuracy is directly dependent on the quality of the questions produced by the language model. Consequently, the accuracy of the new questions generated by the model directly affects the overall performance of the system and the reliability of its answers.
2. **Answer Generation:** The most relevant text segments and the user’s original question are combined to generate an answer. At this stage, the language model’s ability to produce responses and analyze reference information directly determines the quality of the output. The higher the model’s capacity for understanding and synthesis, the more accurate, consistent, and informative the generated answer will be.

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/01.jpg)

_Figure 1 (Flowchart)_

#### Evaluated Language Models

In this study, a total of five different language models were compared, with three being local and two cloud-based. The evaluated models are as follows:

- **DeepSeek R1 Qwen 14B distill** (_4-bit quantized_)
- **Llama 3.1 8B instruct** (_8-bit quantized_)
- **Microsoft Phi-4 14B** (_4-bit quantized_)
- **OpenAI GPT-4o**
- **OpenAI o3-mini**

Local models were run using **Ollama**, while for the cloud-based OpenAI models, responses were obtained by sending requests through the OpenAI API.

Additionally, factors such as the **embedding model, reranker model, system prompt, query prompt**, etc., used within the application were kept constant solely to enable a fair comparison of the language models.

#### Evaluation Metrics

To assess the quality of language model responses, the **BLEU, ROUGE-1, ROUGE-2, ROUGE-L**, and **BERTScore** metrics were used. These metrics were selected to measure the accuracy, similarity to reference answers, and semantic appropriateness of the responses generated by the models.

- **BLEU** _(Bilingual Evaluation Understudy)_ evaluates the accuracy of the language model by measuring the n-gram-based similarity between the model-generated response and the reference answer.
- **ROUGE-1** calculates the overlap between the reference and model responses based on individual words (unigrams).
- **ROUGE-2** measures the similarity between the model and reference answers using two-word sequences (bigrams).
- **ROUGE-L** provides a more flexible matching criterion by analyzing the longest common subsequence (LCS) between the model’s output and the reference text, independent of word order.
- **BERTScore**, unlike word-based comparisons, evaluates the semantic similarity of the responses using a BERT-based deep learning model.

Together, these metrics analyze both the surface-level similarity (BLEU, ROUGE) and the semantic accuracy (BERTScore) of the models.

In addition to these metrics, each question, the corresponding actual answer, and the language model responses were individually examined and labeled as correct, partially correct, or incorrect, without relying on word similarity measures.

### Results

#### **1. BLEU**

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/02.png)

_Figure 2 (BLEU Scores)_

In the graph above, BLEU scores are shown. According to the measured values, the highest overlap between the actual answers and the responses generated by the model was observed in the **OpenAI GPT-4o** model with a BLEU score of **14.34**. The lowest overlap was measured in the **Microsoft Phi-4 14B** _(4-bit quantized)_ model with a BLEU score of **3.68**.

In the evaluated dataset, the fact that the reference answers consist of short and direct responses, while the models generate more explanatory and detailed answers, is a factor in the relatively low BLEU scores across all models. This can be explained by the fact that the BLEU metric is an n-gram-based measurement method and does not directly evaluate contextual meaning.

In this context, making comparisons between models rather than relying on absolute BLEU scores provides a more accurate analysis of the performance of the language models.

#### 2. ROUGE-1 / ROUGE-2 / ROUGE-L

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/03.png)

_Figure 3 (ROUGE-1)_

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/04.png)

_Figure 4 (ROUGE-2)_

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/05.png)

_Figure 5 (ROUGE-L)_

The graphs above present the ROUGE scores between the actual answers and the responses generated by the models. It is believed that the higher Recall values across all ROUGE metrics are associated with the models producing longer and more explanatory responses. Longer answers increase the likelihood of covering the words present in the reference answers, which leads to higher Recall scores.

The high Recall scores also support the idea that the low scores in the BLEU metric may be due to the models producing longer and more detailed answers. In this context, the F1 score is considered the most appropriate metric for comparing the overall performance of the models.

The highest scores across all ROUGE metrics were achieved by the **OpenAI GPT-4o** model:

- **ROUGE-1:** 49.92
- **ROUGE-2:** 33.39
- **ROUGE-L:** 44.05

The lowest scores were obtained by the **Microsoft Phi-4 14B** _(4-bit quantized)_ model:

- **ROUGE-1:** 24.63
- **ROUGE-2:** 11.87
- **ROUGE-L:** 20.40

#### 3. BERTScore

While the BLEU and ROUGE metrics measure word-based matches and n-gram similarities between the actual answers and the responses generated by language models, **BERTScore** focuses directly on semantic similarity. It evaluates the extent to which the model’s output aligns with the reference answer in terms of content. In this way, it provides a contextual perspective beyond mere word matching, enabling a more comprehensive analysis of the models’ response quality.

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/06.png)

_Figure 6 (BERTScore)_

When the BERTScore graph is evaluated in terms of the F1 score, the highest score was achieved by the **OpenAI GPT-4o** model with **91.02**. This result indicates that the responses generated by the OpenAI GPT-4o model exhibit the highest semantic alignment with the actual answers.

On the other hand, the lowest **BERTScore F1 score** was measured for the **Microsoft Phi-4 14B** _(4-bit quantized)_ model at **86.81**. This indicates that the semantic similarity between this model’s outputs and the reference answers is lower compared to the other models.

#### 4. Manual Evaluation

The scores obtained through mathematical calculations on a word, n-gram, and token basis provide an important reference point for evaluating model performance. However, a model’s generated answer may be longer than the original answer, address different aspects, or include groups of words not present in the reference answer. Nonetheless, **the answer may essentially contain correct information**.

Conversely, a language model’s generated answer may have a high degree of overlap with the reference answer in terms of words and word groups (especially since language models often rephrase the question within the answer). However, this superficial similarity does not guarantee the correctness of the answer. As the example below shows, an answer can be lexically compatible with the reference text yet still contain incorrect or incomplete information in terms of content.

> For example:**Question**: Which country has the largest area?**Actual Answer**: There are a total of 195 countries in the world, and the country with the largest area is **Russia**.**Model’s Answer**: There are a total of 195 countries in the world, and the country with the largest area is **India**.

This situation demonstrates that evaluations based solely on automatic metrics have limited accuracy, and qualitative analysis through manual evaluation also plays an important role in accuracy assessment.

Therefore, in the evaluation process, both automatic metrics and qualitative analysis based on human observation were taken into account. The actual answers and the responses generated by the language models were manually examined, and each answer was categorized into three main classes: “correct,” “incorrect,” and “partially correct.” The visualization related to the accuracy analysis is presented below.

![](https://www.arfitect.com/insights/rag-performance-comparison-of-local-slms-and-openai-llms/07.png)

_Figure 7 (Manual Evaluation)_

According to the graphical results, the model that produced the most correct answers was **OpenAI GPT-4o** with **77** correct answers, while the model with the lowest accuracy was measured as **Microsoft Phi-4 14B** _(4-bit quantized)_ with **23** correct answers.

Among the local SML models, the highest accuracy was achieved with **27** correct answers by the **Llama 3.1 8B instruct** _(8-bit quantized)_ model.

Additionally, the model that produced the most “partially correct” answers was **Microsoft Phi-4 14B** _(4-bit quantized)_ with **20** answers. This suggests that the Microsoft Phi-4 14B (4-bit quantized) model was able to access more accurate document segments or demonstrated a higher level of subject mastery compared to the other local SML models. However, it was observed that it performed lower in generating the correct answer compared to the **Llama 3.1 8B instruct** _(8-bit quantized)_ model.

Among the OpenAI models, despite o3 mini (reasoning model) being a more advanced model than GPT-4o, it did not exhibit superior performance in RAG.

#### Conclusion

In this study, the question-answer performance of various language models was evaluated based on a specific dataset. In the tests conducted, the accuracy of the answers provided by the models and their adherence to the reference documents were analyzed.

According to the obtained metric results, OpenAI GPT-4o demonstrated the best performance by achieving higher BLEU and ROUGE scores compared to the other models.

In particular, the high Recall values in the ROUGE scores indicate that the models were able to capture more information from the actual answers by providing longer and more detailed responses.

Additionally, the manual evaluation revealed that for RAG applications, evaluating the performance of language models solely with automatically calculated metrics such as BLEU and ROUGE is not sufficient.

Thank you for reading.

## Related content

- Related case study: [ROOT CAUSE ANALYSIS TOOL](https://www.arfitect.com/en/case-studies/root-cause-analysis-tool)
- Related service: [AGENTIC AI WORKFLOWS](https://www.arfitect.com/en/services/agentic-automation)
- Technical article: [Zero Human Touch: From Jira Tickets Into Bitbucket Pull Requests](https://www.arfitect.com/en/insights/zero-human-touch-from-jira-tickets-into-bitbucket-pull-requests)

[Also published on Medium](https://medium.com/arfitect/rag-performance-comparison-of-local-slms-and-openai-llms-b38600839444)