This work was facilitated by several open-source libraries, most notably llama-index (https://www.llamaindex.ai/), ChromaDB (https://www.trychroma.com/), and LMQL (https://lmql.ai/).

In the use case demonstrated in this paper, the vector store was generated using published guidelines from the Chicago Consensus Working Group17. This expert group was established to develop guidelines for the management of peritoneal cancers. The subject area was chosen as it is within the investigators&#39; area of clinical expertise. The set of papers was accessed from online journal repositories including Cancer and the Annals of Surgical Oncology. A compact (33.4M parameters) embedding model created by the Beijing Academy for Artificial Intelligence (BAAI, https://www.baai.ac.cn/english.html), bge-small-en, was used to generate embeddings from source documents. The resulting database was then used to augment Llama 2 and Open-AI foundation models7. For the reader&#39;s convenience, the code is made available through GitHub (https://github.com/AnaiLab/AugmentedLLM). To ensure replicability, it is recommended to use the same versions of libraries used in the provided requirements list as well as the same version of Python. Additional details on installation or documentation regarding tools used in the following methods can be located at the official websites of the providers for Python (https://www.python.org), git (https://git-scm.com), Llama-Index (https://llamaindex.ai), and Chroma (https://trychroma.com).
1. Prerequisites: Review code and install required libraries
<ol>
	<li>Verify that git, python, and pip are installed.
	<ol>
		<li>In a terminal, run the following commands to verify the installation: 
		git --version 
		python3 --version 
		pip3 --version</li>
	</ol>
	</li>
	<li>Check the code and install requirements.
	<ol>
		<li>In a terminal, run the following commands: 
		git clone&#160;https://github.com/AnaiLab/AugmentedLLM.git 
		cd ./AugmentedLLM/ 
		pip3 install -r requirements.txt</li>
	</ol>
	</li>
</ol>
2. Creation of a vector database with Llama-Index
<ol>
	<li>Convert RTF file formats (only required if files are in an RTF format).
	<ol>
		<li>Edit the file titled config.py, replacing the file paths in the following code with the location of the RTF articles to be converted and the location to which the plain text files are to be written by typing the following commands. Save the file. 
		rtf_file_dir = &#39;./articles_rtf/&#39; 
		converted_file_dir = &#39;./articles_converted/&#39;</li>
		<li>In a terminal, in the same directory, execute the code to generate plain text versions of RTF files by running the following command: 
		python3 ./convert_rtf.py</li>
	</ol>
	</li>
	<li>Create and save the vector database.
	<ol>
		<li>Edit the config.py file, replacing the value of the following variable with the file path of the folder containing documents with which the LLM is to be augmented; save the file. 
		article_dir = &#39;./articles/&#39;</li>
		<li>In a terminal, in the same directory, execute the code with the following command to create and persist the database. Verify the database is now saved in the vector_db folder. 
		python3 ./build_index.py</li>
	</ol>
	</li>
</ol>
3. Augmentation of a Llama model with vector database generated in section 2
<ol>
	<li>Instantiate custom LLM Locally (optional) 
	NOTE: Performing this step is only required if you would like to use a model other than the default Llama-2-7B. If you wish to use the default model, proceed to step 3.2.
	<ol>
		<li>(Using a custom LLM) Specify an LLM to augment by editing run_augmented_llm.py and passing a llama-index LLM object in the constructor as the llm parameter in the following lines of code rather than None. 
		augmentedLLM = AugmentedLLM(vector_store, llm=None)</li>
	</ol>
	</li>
	<li>Query augmented LLM
	<ol>
		<li>Run the following command in the terminal: 
		python3 ./run_augmented_llm.py</li>
		<li>Run user queries to get a response augmented by data in the set of manuscripts (Figure 3 and Figure 4). Press CTRL + C to exit when finished.</li>
	</ol>
	</li>
</ol>
4. Programmatic comparison of alternative LLMs
<ol>
	<li>Create MCQs.
	<ol>
		<li>Edit the file questions.py, taking note of format of the examples. Add questions following a similar format. Save the file.</li>
	</ol>
	</li>
	<li>Connect to GPT-3.5, GPT-4, OpenChat, or other comparator models via API.
	<ol>
		<li>Edit the config.py file, adding the API key for OpenAI or Huggingface if the objective is to benchmark against models from either provider. Save the file. 
		huggingface_key = &#39;&#39; 
		openai_key = &#39;&#39;</li>
		<li>Edit the compare_llms.py file, and choose the set of models to test against by uncommenting (deleting the &#39;# &#39; characters at the begging of that line) models to compare against. 
		NOTE: Some comparators require an API key as set in step 4.2.1. Additionally, edit the output_dir parameter to change where LLM output is stored if desired; otherwise, a default will be used. 
		output_dir = &#39;./llm_responses/&#39;</li>
		<li>In a terminal, execute the code with the following command. After execution, view the model responses in the folder specified in step 4.2.2 for grading or other review. 
		python3 ./compare_llms.py</li>
	</ol>
	</li>
	<li>(Optional) Experiment with automated grading of MCQ responses. The example code uses the LMQL library to constrain LLM output into an expected format.
	<ol>
		<li>Open the file automated_comparison.py and similarly to step 4.2.2, uncomment models to be included, edit the output_dir variable, or otherwise customize. Note that the model output will still be saved in a similar fashion. Save the file.</li>
		<li>Run the code from step 4.3.1 by running the following command in a terminal: 
		python3 ./automated_comparison.py</li>
	</ol>
	</li>
</ol>

Embedding Methods to Integrate Scientific Manuscripts into Large Language Models

<ol>
	<li>Singhal, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Large+language+models+encode+clinical+knowledge.">Large language models encode clinical knowledge.</a> Nature. 620 (7972), 172-180 (2023).</li><li>Gilson, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+does+ChatGPT+perform+on+the+United+States+medical+licensing+examination?+The+implications+of+large+language+models+for+medical+education+and+knowledge+assessment.">How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment.</a> JMIR Med Educ. 9 (1), e45312 (2023).</li><li>Guerra, G. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=GPT-4+artificial+intelligence+model+outperforms+ChatGPT,+medical+students,+and+neurosurgery+residents+on+neurosurgery+written+board-like+questions.">GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions.</a> World Neurosurg. 179, e160-e165 (2023).</li><li>. Would Chat GPT3 get a Wharton MBA? A prediction based on its performance in the Operations Management course Available from: <a target="_blank" href="https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf">https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf</a> (2023)</li><li>Weizenbaum, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ELIZA-a+computer+program+for+the+study+of+natural+language+communication+between+man+and+machine.">ELIZA-a computer program for the study of natural language communication between man and machine.</a> Communications of the ACM. 9 (1), 36-45 (1966).</li><li>Wu, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+brief+overview+of+ChatGPT:+The+history,+status+quo+and+potential+future+development.">A brief overview of ChatGPT: The history, status quo and potential future development.</a> IEEE/CAA Journal of Automatica Sinica. 10 (5), 1122-1136 (2023).</li><li>Huang, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+survey+on+hallucination+in+large+language+models:+Principles,+taxonomy,+challenges,+and+open+questions.">A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.</a> arXiv [cs.CL]. , (2023).</li><li>Thirunavukarasu, A. J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Large+language+models+in+medicine.">Large language models in medicine.</a> Nature Med. 29 (8), 1930-1940 (2023).</li><li>Ram, O., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=In-Context+retrieval-augmented+language+models.">In-Context retrieval-augmented language models.</a> Trans Assoc Comput Linguist. 11, 1316-1331 (2023).</li><li>. Sequence distance embeddings Available from: <a target="_blank" href="https://wrap.warwick.ac.uk/61310/7/WRAP_THESIS_Cormode_2003.pdf">https://wrap.warwick.ac.uk/61310/7/WRAP_THESIS_Cormode_2003.pdf</a> (2003)</li><li>Chicago Consensus Working Group. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Chicago+Consensus+Guidelines+for+peritoneal+surface+malignancies:+Introduction.">The Chicago Consensus Guidelines for peritoneal surface malignancies: Introduction.</a> Cancer. 126 (11), 2510-2512 (2020).</li><li>Dodge, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measuring+the+carbon+intensity+of+AI+in+cloud+instances.">Measuring the carbon intensity of AI in cloud instances.</a> FACCT 2022. , (2022).</li><li>Khene, Z. -. E., Bigot, P., Mathieu, R., Roupr&#234;t, M., Bensalah, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+a+personalized+chat+model+based+on+the+European+association+of+urology+oncology+guidelines:+Harnessing+the+power+of+generative+artificial+intelligence+in+clinical+practice.">Development of a personalized chat model based on the European association of urology oncology guidelines: Harnessing the power of generative artificial intelligence in clinical practice.</a> Eur Urol Oncol. 7 (1), 160-162 (2024).</li><li>Kresevic, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Optimization+of+hepatological+clinical+guidelines+interpretation+by+large+language+models:+a+retrieval+augmented+generation-based+framework.">Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework.</a> NPJ Digit Med. 7 (1), 102 (2024).</li><li>Ge, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+a+liver+disease-specific+large+language+model+chat+interface+using+retrieval+augmented+generation.">Development of a liver disease-specific large language model chat interface using retrieval augmented generation.</a> medRxiv. , (2023).</li><li>Panagoulias, D. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rule-augmented+artificial+intelligence-empowered+systems+for+medical+diagnosis+using+large+language+models.">Rule-augmented artificial intelligence-empowered systems for medical diagnosis using large language models.</a> , 70-77 (2023).</li><li>Panagoulias, D. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmenting+large+language+models+with+rules+for+enhanced+domain-specific+interactions:+The+case+of+medical+diagnosis.">Augmenting large language models with rules for enhanced domain-specific interactions: The case of medical diagnosis.</a> Electronics. 13 (2), 320 (2024).</li><li>Bommasani, R., Liang, P., Lee, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Holistic+evaluation+of+language+models.">Holistic evaluation of language models.</a> Ann N Y Acad Sci. 1525 (1), 140-146 (2023).</li><li>Papineni, K., Roukos, S., Ward, T., Zhu, W. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bleu.">Bleu.</a> Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02. , (2001).</li><li>Johnson, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+the+accuracy+and+reliability+of+AI-generated+medical+responses:+An+evaluation+of+the+chat-GPT+model.">Assessing the accuracy and reliability of AI-generated medical responses: An evaluation of the chat-GPT model.</a> Res Sq. , (2023).</li><li>Lin, C. -. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ROUGE:+A+package+for+automatic+evaluation+of+summaries.">ROUGE: A package for automatic evaluation of summaries.</a> Text Summarization Branches Out. , 74-81 (2004).</li><li>Chen, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluating+the+ChatGPT+family+of+models+for+biomedical+reasoning+and+classification.">Evaluating the ChatGPT family of models for biomedical reasoning and classification.</a> J Am Med Inform Assoc. 31 (4), 940-948 (2024).</li></ol>

The authors have no conflicts of interest to declare.

The methods provided here aim to facilitate the research of domain-specific applications of LLMs without the need for de novo training or extensive fine-tuning. As LLMs are becoming an area of significant research interest, approaches for augmenting knowledge bases and improving the accuracy of responses will become increasingly important18,19,20,21. As demonstrated in the provided res...

Large language models (LLMs) such as OpenAI&#39;s ChatGPT or Meta AI&#39;s Llama have rapidly become a popular resource for generating text relevant to a user prompt. Originally functioning to predict the next lexical items in a sequence, these models have evolved to understand context, encode clinical information, and demonstrate high performance on a variety of tasks1,2,3,4. Though language models predate such capabilities and their current level of popularity by decades5, recent advances in deep learning and computing capabilities have made pretrained, high-quality commercial LLMs broadly available to users via web-based technologies and application program interfaces (APIs)6. However, there are several notable limitations to consuming LLMs in this format.
Challenge 1: Static training corpus 
LLMs are trained on an enormous (e.g., two trillion tokens in the case of Llama 27) but static body of text data. This poses a challenge to producing accurate responses pertaining to fields undergoing rapid development or changing literature. In this static approach, LLMs would require frequent retraining to keep up with the latest data, which is neither practical nor scalable. Moreover, prompts that require responses based on information not present in the training data may prevent useful text generation or lead to hallucinations8. Instances of hallucinations or fact fabrication raise significant concerns about the reliability of LLMs, particularly in settings where the accuracy of information is critical9.
Challenge 2: Lack of domain specificity 
Pretrained models are often created for general use, while users may require a model specifically optimized for performance in a particular domain. Additionally, the computational resources and data required for training a model de novo or performing significant fine-tuning are prohibitive to many users.
Challenge 3: Lack of privacy 
Users pursuing applications involving sensitive data or proprietary information may be unwilling or unable to use certain LLM services as they lack information on how data may be stored or utilized.
Challenge 4: Lack of guaranteed stability 
Services with proprietary LLMs may change available models or alter behavior at any time, making stability a concern for the implementation of applications relying on these services.
Retrieval-augmented generation (RAG) is a technique developed to improve LLM performance, particularly on queries related to information outside the model&#39;s training corpus10,11. These systems augment LLMs by incorporating contextual information to be considered when generating a response to a user query. Various recent works have described applications of RAG systems and their potential advantages12,13,14.
The goal of the method outlined in this work is to demonstrate the construction of such a system and provide a framework for researchers to rapidly experiment on domain-specific, augmented LLMs. This method is applicable for users seeking to augment an LLM with an external text-based data source. Specifically, an overarching aim of this protocol is to provide step-by-step code that is extensible to a variety of practical LLM and RAG experiments without the need for significant technical expertise in the language-modeling domain, though a working knowledge of Python is required to apply this approach without modification. To maximize user control, transparency, portability, and affordability of solutions, open-source, publicly available tools are utilized. The proposed system addresses the previously stated issues in the following ways:
Solutions 1 and 2: Static training corpus and lack of domain specificity 
The provided methodology leverages a RAG approach, utilizing embeddings to supply domain-specific information not included in the original training data. At a high level, embedding models transform text or other data into a representation as a vector or single-dimension array of numbers. This technique is beneficial as it converts semantic information contained in text to a dense, numeric form. By projecting a user query into the same embedding space, various algorithms can be used to calculate the distance15, and therefore, approximate semantic similarity, between the user query and sections of text documents. Thus, creating a database of such vectors from documents broken into discrete sections can facilitate searching over a significant number of documents for text most relevant to a user query (Figure 1). This approach is extensible to any text document. While other approaches, such as online search capabilities, are beginning to be implemented to augment LLMs, this approach allows users to choose sources considered of sufficiently high quality for their use case.
Solution 2: Lack of privacy 
In this implementation, a secure cloud environment was used for hosting, with no user prompts, generated responses, or other data leaving this ecosystem. All code is written in a platform-agnostic manner, however, to ensure that another cloud provider or local hardware may be substituted.
Solution 3: Lack of guaranteed stability 
This approach utilizes open-source libraries and focuses on augmenting LLMs with publicly available weights, allowing a higher degree of transparency, stability, and versioning, if required.
A full schematic of our proposed system is shown in Figure 2, and detailed instructions on replicating this or a similar system is outlined in the protocol section. An additional consideration when altering model behavior through fine-tuning or augmentation is the evaluation of performance. In language-generating models, this presents a unique challenge as many traditional machine learning metrics are not applicable. Though a variety of techniques exist16, in this study, expert-written multiple-choice questions (MCQs) were used to assess accuracy and compare performance pre- and post-augmentation as well as against popular alternative LLMs.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>pip3 version 22.0.2&#160;</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Python version 3.10.12</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
	</tbody>
</table>

augmenting large language models via vector embeddings to improve domain-specific responsiveness

Large language models (LLMs) have emerged as a popular resource for generating information relevant to a user query. Such models are created through a resource-intensive training process utilizing an extensive, static corpus of textual data. This static nature results in limitations for adoption in domains with rapidly changing knowledge, proprietary information, and sensitive data. In this work, methods are outlined for augmenting general-purpose LLMs, known as foundation models, with domain-specific information using an embeddings-based approach for incorporating up-to-date, peer-reviewed scientific manuscripts. This is achieved through open-source tools such as Llama-Index and publicly available models such as Llama-2 to maximize transparency, user privacy and control, and replicability. While scientific manuscripts are used as an example use case, this approach can be extended to any text data source. Additionally, methods for evaluating model performance following this enhancement are discussed. These methods enable the rapid development of LLM systems for highly specialized domains regardless of the comprehensiveness of information in the training corpus.

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Our research focuses on a wide variety of machine learning and large language model applications in cancer care. This includes both improving direct patient care and addressing operational challenges such as scheduling and documentation. We're ultimately trying to answer the question, How can these technologies make healthcare better for patients, providers, and researchers?Large language models have already begun transforming the research landscape. Previously prohibitive natural language processing tasks have become approachable to many more investigators, enabling many new lines of research involving unstructured data. A major challenge is the rapidly changing landscape of large language model tools and infrastructure.Many clinicians and researchers may have ideas for applications but don't currently have the contextual or technical knowledge needed for implementation, or thoughtful consideration of trade-offs of different approaches. To begin install Git, python, and PIP on the computer, and run the commands in a terminal to verify the installation. Then run the git clone command to download the repository and install the necessary requirements.To create a vector database, edit the config. py file, replacing the value of the following variable with the file path to the folder containing the documents that will be used to augment the large language model. Then save the updated file in the articles directory.Next, in a terminal in the same directory, execute the code with the Python 3 build index command to create and persist the database. Verify if the database is now saved in the vector database folder. To query the augmented LLM, execute the Python 3 run augmented llm.py command in the terminal. Test user queries to receive responses that are augmented by the data from the document set. Then press control plus C to exit when finished.For creating MCQs, edit the file questions. py, taking note of the format of the examples. Add questions following a similar format, and save the file.To edit the config. pi file, add the API key for open AI or hugging face. If the objective is to benchmark against models from either provider.Now save the file. Then edit the compare llms. py file, and choose the set of models to test against.After uncommenting the models to compare against. Finally, in a terminal, execute the code with the compare llms command and after executioni View the model responses in the specified folder for grading or other review. Among the MCQs tested, the embed model performed significantly better than the base one, as did the augmented open AI models.

A set of 22 publications from the Chicago Consensus Working Group management guidelines were used to augment the base Llama-7b model17. The documents were converted into a vector index using the tool Llama-Index to generate Llama-2-7b-CCWG-Embed. Popular OpenAI models such as GPT-3.5 and GPT-4 were also augmented in a similar fashion to produce GPT-XX-CCWG-Embed models. A total of 20 multiple choice questions (MCQ) were developed to assess knowledge related to the management of a variety of perito...

In this protocol, foundation large language model response quality is improved via augmentation with peer-reviewed, domain-specific scientific articles through a vector embedding mechanism. Additionally, code is provided to aid in performance comparison across large language models.

Large language models (LLMs) have emerged as a popular resource for generating information relevant to a user query. Such models are created through a ...

augmenting-large-language-models-via-vector-embeddings-to-improve

Research

JoVE Journal

Medicine

313 Views.  Medical College of Wisconsin.  In this protocol, foundation large language model response quality is improved via augmentation with peer-reviewed, domain-specific scientific articles through a vector embedding mechanism. Additionally, code is provided to aid in performance comparison across large language models.

Embedding Methods to Integrate Scientific Manuscripts into Large Language Models

Summary