This work was facilitated by several open-source libraries, most notably llama-index (https://www.llamaindex.ai/), ChromaDB (https://www.trychroma.com/), and LMQL (https://lmql.ai/).

In the use case demonstrated in this paper, the vector store was generated using published guidelines from the Chicago Consensus Working Group17. This expert group was established to develop guidelines for the management of peritoneal cancers. The subject area was chosen as it is within the investigators&#39; area of clinical expertise. The set of papers was accessed from online journal repositories including Cancer and the Annals of Surgical Oncology. A compact (33.4M parameters) embedding model created by the Beijing Academy for Artificial Intelligence (BAAI, https://www.baai.ac.cn/english.html), bge-small-en, was used to generate embeddings from source documents. The resulting database was then used to augment Llama 2 and Open-AI foundation models7. For the reader&#39;s convenience, the code is made available through GitHub (https://github.com/AnaiLab/AugmentedLLM). To ensure replicability, it is recommended to use the same versions of libraries used in the provided requirements list as well as the same version of Python. Additional details on installation or documentation regarding tools used in the following methods can be located at the official websites of the providers for Python (https://www.python.org), git (https://git-scm.com), Llama-Index (https://llamaindex.ai), and Chroma (https://trychroma.com).
1. Prerequisites: Review code and install required libraries
<ol>
	<li>Verify that git, python, and pip are installed.
	<ol>
		<li>In a terminal, run the following commands to verify the installation: 
		git --version 
		python3 --version 
		pip3 --version</li>
	</ol>
	</li>
	<li>Check the code and install requirements.
	<ol>
		<li>In a terminal, run the following commands: 
		git clone&#160;https://github.com/AnaiLab/AugmentedLLM.git 
		cd ./AugmentedLLM/ 
		pip3 install -r requirements.txt</li>
	</ol>
	</li>
</ol>
2. Creation of a vector database with Llama-Index
<ol>
	<li>Convert RTF file formats (only required if files are in an RTF format).
	<ol>
		<li>Edit the file titled config.py, replacing the file paths in the following code with the location of the RTF articles to be converted and the location to which the plain text files are to be written by typing the following commands. Save the file. 
		rtf_file_dir = &#39;./articles_rtf/&#39; 
		converted_file_dir = &#39;./articles_converted/&#39;</li>
		<li>In a terminal, in the same directory, execute the code to generate plain text versions of RTF files by running the following command: 
		python3 ./convert_rtf.py</li>
	</ol>
	</li>
	<li>Create and save the vector database.
	<ol>
		<li>Edit the config.py file, replacing the value of the following variable with the file path of the folder containing documents with which the LLM is to be augmented; save the file. 
		article_dir = &#39;./articles/&#39;</li>
		<li>In a terminal, in the same directory, execute the code with the following command to create and persist the database. Verify the database is now saved in the vector_db folder. 
		python3 ./build_index.py</li>
	</ol>
	</li>
</ol>
3. Augmentation of a Llama model with vector database generated in section 2
<ol>
	<li>Instantiate custom LLM Locally (optional) 
	NOTE: Performing this step is only required if you would like to use a model other than the default Llama-2-7B. If you wish to use the default model, proceed to step 3.2.
	<ol>
		<li>(Using a custom LLM) Specify an LLM to augment by editing run_augmented_llm.py and passing a llama-index LLM object in the constructor as the llm parameter in the following lines of code rather than None. 
		augmentedLLM = AugmentedLLM(vector_store, llm=None)</li>
	</ol>
	</li>
	<li>Query augmented LLM
	<ol>
		<li>Run the following command in the terminal: 
		python3 ./run_augmented_llm.py</li>
		<li>Run user queries to get a response augmented by data in the set of manuscripts (Figure 3 and Figure 4). Press CTRL + C to exit when finished.</li>
	</ol>
	</li>
</ol>
4. Programmatic comparison of alternative LLMs
<ol>
	<li>Create MCQs.
	<ol>
		<li>Edit the file questions.py, taking note of format of the examples. Add questions following a similar format. Save the file.</li>
	</ol>
	</li>
	<li>Connect to GPT-3.5, GPT-4, OpenChat, or other comparator models via API.
	<ol>
		<li>Edit the config.py file, adding the API key for OpenAI or Huggingface if the objective is to benchmark against models from either provider. Save the file. 
		huggingface_key = &#39;&#39; 
		openai_key = &#39;&#39;</li>
		<li>Edit the compare_llms.py file, and choose the set of models to test against by uncommenting (deleting the &#39;# &#39; characters at the begging of that line) models to compare against. 
		NOTE: Some comparators require an API key as set in step 4.2.1. Additionally, edit the output_dir parameter to change where LLM output is stored if desired; otherwise, a default will be used. 
		output_dir = &#39;./llm_responses/&#39;</li>
		<li>In a terminal, execute the code with the following command. After execution, view the model responses in the folder specified in step 4.2.2 for grading or other review. 
		python3 ./compare_llms.py</li>
	</ol>
	</li>
	<li>(Optional) Experiment with automated grading of MCQ responses. The example code uses the LMQL library to constrain LLM output into an expected format.
	<ol>
		<li>Open the file automated_comparison.py and similarly to step 4.2.2, uncomment models to be included, edit the output_dir variable, or otherwise customize. Note that the model output will still be saved in a similar fashion. Save the file.</li>
		<li>Run the code from step 4.3.1 by running the following command in a terminal: 
		python3 ./automated_comparison.py</li>
	</ol>
	</li>
</ol>

Embedding Methods to Integrate Scientific Manuscripts into Large Language Models

<ol>
	<li>Singhal, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Large+language+models+encode+clinical+knowledge.">Large language models encode clinical knowledge.</a> Nature. 620 (7972), 172-180 (2023).</li><li>Gilson, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+does+ChatGPT+perform+on+the+United+States+medical+licensing+examination?+The+implications+of+large+language+models+for+medical+education+and+knowledge+assessment.">How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment.</a> JMIR Med Educ. 9 (1), e45312 (2023).</li><li>Guerra, G. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=GPT-4+artificial+intelligence+model+outperforms+ChatGPT,+medical+students,+and+neurosurgery+residents+on+neurosurgery+written+board-like+questions.">GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions.</a> World Neurosurg. 179, e160-e165 (2023).</li><li>. Would Chat GPT3 get a Wharton MBA? A prediction based on its performance in the Operations Management course Available from: <a target="_blank" href="https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf">https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf</a> (2023)</li><li>Weizenbaum, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ELIZA-a+computer+program+for+the+study+of+natural+language+communication+between+man+and+machine.">ELIZA-a computer program for the study of natural language communication between man and machine.</a> Communications of the ACM. 9 (1), 36-45 (1966).</li><li>Wu, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+brief+overview+of+ChatGPT:+The+history,+status+quo+and+potential+future+development.">A brief overview of ChatGPT: The history, status quo and potential future development.</a> IEEE/CAA Journal of Automatica Sinica. 10 (5), 1122-1136 (2023).</li><li>Huang, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+survey+on+hallucination+in+large+language+models:+Principles,+taxonomy,+challenges,+and+open+questions.">A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.</a> arXiv [cs.CL]. , (2023).</li><li>Thirunavukarasu, A. J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Large+language+models+in+medicine.">Large language models in medicine.</a> Nature Med. 29 (8), 1930-1940 (2023).</li><li>Ram, O., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=In-Context+retrieval-augmented+language+models.">In-Context retrieval-augmented language models.</a> Trans Assoc Comput Linguist. 11, 1316-1331 (2023).</li><li>. Sequence distance embeddings Available from: <a target="_blank" href="https://wrap.warwick.ac.uk/61310/7/WRAP_THESIS_Cormode_2003.pdf">https://wrap.warwick.ac.uk/61310/7/WRAP_THESIS_Cormode_2003.pdf</a> (2003)</li><li>Chicago Consensus Working Group. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Chicago+Consensus+Guidelines+for+peritoneal+surface+malignancies:+Introduction.">The Chicago Consensus Guidelines for peritoneal surface malignancies: Introduction.</a> Cancer. 126 (11), 2510-2512 (2020).</li><li>Dodge, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measuring+the+carbon+intensity+of+AI+in+cloud+instances.">Measuring the carbon intensity of AI in cloud instances.</a> FACCT 2022. , (2022).</li><li>Khene, Z. -. E., Bigot, P., Mathieu, R., Roupr&#234;t, M., Bensalah, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+a+personalized+chat+model+based+on+the+European+association+of+urology+oncology+guidelines:+Harnessing+the+power+of+generative+artificial+intelligence+in+clinical+practice.">Development of a personalized chat model based on the European association of urology oncology guidelines: Harnessing the power of generative artificial intelligence in clinical practice.</a> Eur Urol Oncol. 7 (1), 160-162 (2024).</li><li>Kresevic, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Optimization+of+hepatological+clinical+guidelines+interpretation+by+large+language+models:+a+retrieval+augmented+generation-based+framework.">Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework.</a> NPJ Digit Med. 7 (1), 102 (2024).</li><li>Ge, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+a+liver+disease-specific+large+language+model+chat+interface+using+retrieval+augmented+generation.">Development of a liver disease-specific large language model chat interface using retrieval augmented generation.</a> medRxiv. , (2023).</li><li>Panagoulias, D. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rule-augmented+artificial+intelligence-empowered+systems+for+medical+diagnosis+using+large+language+models.">Rule-augmented artificial intelligence-empowered systems for medical diagnosis using large language models.</a> , 70-77 (2023).</li><li>Panagoulias, D. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmenting+large+language+models+with+rules+for+enhanced+domain-specific+interactions:+The+case+of+medical+diagnosis.">Augmenting large language models with rules for enhanced domain-specific interactions: The case of medical diagnosis.</a> Electronics. 13 (2), 320 (2024).</li><li>Bommasani, R., Liang, P., Lee, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Holistic+evaluation+of+language+models.">Holistic evaluation of language models.</a> Ann N Y Acad Sci. 1525 (1), 140-146 (2023).</li><li>Papineni, K., Roukos, S., Ward, T., Zhu, W. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bleu.">Bleu.</a> Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02. , (2001).</li><li>Johnson, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+the+accuracy+and+reliability+of+AI-generated+medical+responses:+An+evaluation+of+the+chat-GPT+model.">Assessing the accuracy and reliability of AI-generated medical responses: An evaluation of the chat-GPT model.</a> Res Sq. , (2023).</li><li>Lin, C. -. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ROUGE:+A+package+for+automatic+evaluation+of+summaries.">ROUGE: A package for automatic evaluation of summaries.</a> Text Summarization Branches Out. , 74-81 (2004).</li><li>Chen, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluating+the+ChatGPT+family+of+models+for+biomedical+reasoning+and+classification.">Evaluating the ChatGPT family of models for biomedical reasoning and classification.</a> J Am Med Inform Assoc. 31 (4), 940-948 (2024).</li></ol>

The authors have no conflicts of interest to declare.

The methods provided here aim to facilitate the research of domain-specific applications of LLMs without the need for de novo training or extensive fine-tuning. As LLMs are becoming an area of significant research interest, approaches for augmenting knowledge bases and improving the accuracy of responses will become increasingly important18,19,20,21. As demonstrated in the provided results, the protocol outlined offers improvement in performance on domain-specific questions compared to the same LLM without augmentation and approaches the performance of more complex models such as OpenAI GPT models. As LLMs can require enormous computational resources to generate responses22,23, augmenting a simpler model may provide an avenue to implementation in resource-constrained use cases. Furthermore, the RAG system also produced benefits when augmenting high-complexity models such as those provided by OpenAI. While the presented results are particular to peritoneal cancer management, recent studies on RAG systems with a variety of both domain and scale of data sources have been produced, indicating the broad applicability of such systems21,24,25,26. As the quality of responses is closely related to the text data used for augmentation, however, it is critical users carefully consider which sources are optimal for their particular domain and desired application. This consideration is of particular importance in domains such as healthcare, which has been a particular focus of LLM-related studies. LLM use for diagnostics27,28, clinical trial matching29,30, and numerous other applications are being explored, and require validated, trustworthy text resources rather than relying on unknown training corpora.
Performance was measured by the percentage correct of MCQs written by an expert in peritoneal cancers and their clinical management. Responses were classified as correct or incorrect by human review; however, to minimize repetitive tasks for researchers, basic code to begin approaching more automated comparison of performance across LLMs is provided.
A key benefit of this protocol is the provided code to programmatically access a variety of LLMs. Previously, to assess performance on MCQs users without the requisite technical capabilities may have relied on repetitive manual testing through a web-based interface. The provided code gives basic classes to interface with several popular LLMs along with examples of how to execute the code. The goal of providing these tools is to allow more researchers to move towards automating workflows that assess responses to a prompt across a variety of LLMs, enhancing the ability to carry out comparative studies.
Furthermore, the outlined methods are designed to emphasize flexibility and extensibility. Though code can be used as-is, it is written with the intention of further tailoring to specific use cases as needed, such as the use of different LLMs and embedding models for augmentation or comparison. As the LLM landscape is rapidly evolving, this extensibility will become increasingly important to meet the varied needs of users in different domains. Additionally, the Llama-Index library provides a significant number of features not utilized in the demonstrated methods which can give users additional options when creating similar systems. These can be found in the Llama-Index official documentation.
Readers should also carefully consider evaluation methods that are optimal for their use case. While MCQs have gained popularity for LLM comparisons due to their easily quantifiable nature31,32, caution should be taken in considering them a comprehensive performance metric for generative systems. Recent literature has implemented a diverse range of evaluations, including qualitative approaches, human review, Stanford Holistic Evaluation of Language Models (HELM)33, and standard natural language processing metrics such as Bilingual Evaluation Understudy (BLEU), General Language Understanding Evaluation (GLUE), ROUGE, perplexity, and F1 score34,35,36,37,38,39.
There are limitations to these methods as provided here. For example, while interfaces were implemented for several leading LLM providers, given the rapidly evolving nature of this field and how varied use cases may be, this implementation may not be comprehensive. The code, however, was designed with this in mind as previously discussed. Additionally, while basic code for moving toward automated grading was provided, there is room for expansion into using alternative tools for grading responses such as HELM or BLEU. Further, additional usage of tools to constrain output according to predefined grammars, such as Language model query language (LMQL), may be used to expand on this work and increase automation of comparative LLM studies. Additionally, given the wide variety of potential use cases, further studies into the effect of RAG systems on general, out-of-domain performance are required to characterize any performance trade-offs. Finally, additional methods for improving the reliability and evaluation of LLM responses must continue to be investigated.
Note that for technical reasons python 3.10 or greater is required. Shell prompts are written for bash, zsh, or other UNIX-like terminals. Commands can be executed (referred to as &#39;running the command&#39; in the protocol) by simply typing the text followed by the return key. For each step in the protocol, a user may wish to examine the code in the file being executed to have a more comprehensive understanding of the mechanics behind the workflow.

Large language models (LLMs) such as OpenAI&#39;s ChatGPT or Meta AI&#39;s Llama have rapidly become a popular resource for generating text relevant to a user prompt. Originally functioning to predict the next lexical items in a sequence, these models have evolved to understand context, encode clinical information, and demonstrate high performance on a variety of tasks1,2,3,4. Though language models predate such capabilities and their current level of popularity by decades5, recent advances in deep learning and computing capabilities have made pretrained, high-quality commercial LLMs broadly available to users via web-based technologies and application program interfaces (APIs)6. However, there are several notable limitations to consuming LLMs in this format.
Challenge 1: Static training corpus 
LLMs are trained on an enormous (e.g., two trillion tokens in the case of Llama 27) but static body of text data. This poses a challenge to producing accurate responses pertaining to fields undergoing rapid development or changing literature. In this static approach, LLMs would require frequent retraining to keep up with the latest data, which is neither practical nor scalable. Moreover, prompts that require responses based on information not present in the training data may prevent useful text generation or lead to hallucinations8. Instances of hallucinations or fact fabrication raise significant concerns about the reliability of LLMs, particularly in settings where the accuracy of information is critical9.
Challenge 2: Lack of domain specificity 
Pretrained models are often created for general use, while users may require a model specifically optimized for performance in a particular domain. Additionally, the computational resources and data required for training a model de novo or performing significant fine-tuning are prohibitive to many users.
Challenge 3: Lack of privacy 
Users pursuing applications involving sensitive data or proprietary information may be unwilling or unable to use certain LLM services as they lack information on how data may be stored or utilized.
Challenge 4: Lack of guaranteed stability 
Services with proprietary LLMs may change available models or alter behavior at any time, making stability a concern for the implementation of applications relying on these services.
Retrieval-augmented generation (RAG) is a technique developed to improve LLM performance, particularly on queries related to information outside the model&#39;s training corpus10,11. These systems augment LLMs by incorporating contextual information to be considered when generating a response to a user query. Various recent works have described applications of RAG systems and their potential advantages12,13,14.
The goal of the method outlined in this work is to demonstrate the construction of such a system and provide a framework for researchers to rapidly experiment on domain-specific, augmented LLMs. This method is applicable for users seeking to augment an LLM with an external text-based data source. Specifically, an overarching aim of this protocol is to provide step-by-step code that is extensible to a variety of practical LLM and RAG experiments without the need for significant technical expertise in the language-modeling domain, though a working knowledge of Python is required to apply this approach without modification. To maximize user control, transparency, portability, and affordability of solutions, open-source, publicly available tools are utilized. The proposed system addresses the previously stated issues in the following ways:
Solutions 1 and 2: Static training corpus and lack of domain specificity 
The provided methodology leverages a RAG approach, utilizing embeddings to supply domain-specific information not included in the original training data. At a high level, embedding models transform text or other data into a representation as a vector or single-dimension array of numbers. This technique is beneficial as it converts semantic information contained in text to a dense, numeric form. By projecting a user query into the same embedding space, various algorithms can be used to calculate the distance15, and therefore, approximate semantic similarity, between the user query and sections of text documents. Thus, creating a database of such vectors from documents broken into discrete sections can facilitate searching over a significant number of documents for text most relevant to a user query (Figure 1). This approach is extensible to any text document. While other approaches, such as online search capabilities, are beginning to be implemented to augment LLMs, this approach allows users to choose sources considered of sufficiently high quality for their use case.
Solution 2: Lack of privacy 
In this implementation, a secure cloud environment was used for hosting, with no user prompts, generated responses, or other data leaving this ecosystem. All code is written in a platform-agnostic manner, however, to ensure that another cloud provider or local hardware may be substituted.
Solution 3: Lack of guaranteed stability 
This approach utilizes open-source libraries and focuses on augmenting LLMs with publicly available weights, allowing a higher degree of transparency, stability, and versioning, if required.
A full schematic of our proposed system is shown in Figure 2, and detailed instructions on replicating this or a similar system is outlined in the protocol section. An additional consideration when altering model behavior through fine-tuning or augmentation is the evaluation of performance. In language-generating models, this presents a unique challenge as many traditional machine learning metrics are not applicable. Though a variety of techniques exist16, in this study, expert-written multiple-choice questions (MCQs) were used to assess accuracy and compare performance pre- and post-augmentation as well as against popular alternative LLMs.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>pip3 version 22.0.2&#160;</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
		<tr>
			<td>Python version 3.10.12</td>
			<td></td>
			<td></td>
			<td></td>
		</tr>
	</tbody>
</table>

augmenting large language models via vector embeddings to improve domain-specific responsiveness

Large language models (LLMs) have emerged as a popular resource for generating information relevant to a user query. Such models are created through a resource-intensive training process utilizing an extensive, static corpus of textual data. This static nature results in limitations for adoption in domains with rapidly changing knowledge, proprietary information, and sensitive data. In this work, methods are outlined for augmenting general-purpose LLMs, known as foundation models, with domain-specific information using an embeddings-based approach for incorporating up-to-date, peer-reviewed scientific manuscripts. This is achieved through open-source tools such as Llama-Index and publicly available models such as Llama-2 to maximize transparency, user privacy and control, and replicability. While scientific manuscripts are used as an example use case, this approach can be extended to any text data source. Additionally, methods for evaluating model performance following this enhancement are discussed. These methods enable the rapid development of LLM systems for highly specialized domains regardless of the comprehensiveness of information in the training corpus.

A set of 22 publications from the Chicago Consensus Working Group management guidelines were used to augment the base Llama-7b model17. The documents were converted into a vector index using the tool Llama-Index to generate Llama-2-7b-CCWG-Embed. Popular OpenAI models such as GPT-3.5 and GPT-4 were also augmented in a similar fashion to produce GPT-XX-CCWG-Embed models. A total of 20 multiple choice questions (MCQ) were developed to assess knowledge related to the management of a variety of peritoneal surface malignancies. The MCQs were created by a board-certified surgical oncologist to test at the level of knowledge expected of a surgical oncology fellow. Llama-2-7b-CCWG-Embed performed significantly better than the base model (see Table 1), as did augmented OpenAI models. This is considered a positive result for this method as model performance was improved with augmentation compared to baseline.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/66796/66796fig01.jpg" /> 
Figure 1: Schematic of generating vector database from academic papers. <a href="https://www.jove.com/files/ftp_upload/66796/66796fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/66796/66796fig02.jpg" /> 
Figure 2: Overall system diagram for augmented LLM. <a href="https://www.jove.com/files/ftp_upload/66796/66796fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/66796/66796fig03.jpg" /> 
Figure 3: Augmented Llama-2 model running in Linux terminal. <a href="https://www.jove.com/files/ftp_upload/66796/66796fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/66796/66796fig04.jpg" /> 
Figure 4: Query result from augmented Llama-2 model. <a href="https://www.jove.com/files/ftp_upload/66796/66796fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>LLM&#160;</td>
			<td>Score (% correct)</td>
		</tr>
		<tr>
			<td>Llama-2-7b-chat-hf&#160;</td>
			<td>65%&#160;</td>
		</tr>
		<tr>
			<td>Llama-2-7b-CCWG-Embed</td>
			<td>75%&#160;</td>
		</tr>
		<tr>
			<td>Openchat-3.5-010618</td>
			<td>65%</td>
		</tr>
		<tr>
			<td>Mistral-7B-v0.119</td>
			<td>70%</td>
		</tr>
		<tr>
			<td>GPT-3.5&#160;</td>
			<td>75%&#160;</td>
		</tr>
		<tr>
			<td>GPT-3.5-CCWG-Embed</td>
			<td>85%</td>
		</tr>
		<tr>
			<td>GPT-4&#160;</td>
			<td>85%&#160;</td>
		</tr>
		<tr>
			<td>GPT-4-CCWG-Embed</td>
			<td>90%</td>
		</tr>
	</tbody>
</table>
Table 1: Scores (percent correct) of various LLMs on a set of 20 questions pertaining to peritoneal malignancy management.

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

In this protocol, foundation large language model response quality is improved via augmentation with peer-reviewed, domain-specific scientific articles through a vector embedding mechanism. Additionally, code is provided to aid in performance comparison across large language models.

Large language models (LLMs) have emerged as a popular resource for generating information relevant to a user query. Such models are created through a ...

augmenting-large-language-models-via-vector-embeddings-to-improve

Nanyang Technological University

Research

JoVE Journal

Medicine

419 Views.  Medical College of Wisconsin. In this protocol, foundation large language model response quality is improved via augmentation with peer-reviewed, domain-specific scientific articles through a vector embedding mechanism. Additionally, code is provided to aid in performance comparison across large language models.

Video: Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

DNA Vector-based RNA Interference to Study Gene Function in Cancer

RNA interference (RNAi) inhibits gene expression by specifically degrading target mRNAs. Since the discovery of double-stranded small interference RNA (siRNA) in gene silencing1, RNAi has become a powerful research tool in gene function studies. Compared to genetic deletion, RNAi-mediated gene silencing possesses many advantages, such as the ease with which it is carried out and its suitability to most cell lines. Multiple studies have demonstrated the applications of RNAi technology in cancer research. In particular, the development of the DNA vector-based technology to produce small hairpin RNA (shRNA) driven by the U6 or H1 promoter has made long term and inducible gene silencing possible2,3. Its use in combination with genetically engineered viral vectors, such as lentivirus, facilitates high efficiencies of shRNA delivery and/or integration into genomic DNA for stable shRNA expression. 
We describe a detailed procedure using the DNA vector-based RNAi technology to determine gene function, including construction of lentiviral vectors expressing shRNA, lentivirus production and cell infection, and functional studies using a mouse xenograft model. 
Various strategies have been reported in generating shRNA constructs. The protocol described here employing PCR amplification and a 3-fragment ligation can be used to directly and efficiently generate shRNA-containing lentiviral constructs without leaving any extra nucleotide adjacent to a shRNA coding sequence. Since the shRNA-expression cassettes created by this strategy can be cut out by restriction enzymes, they can be easily moved to other vectors with different fluorescent or antibiotic markers. Most commercial transfection reagents can be used in lentivirus production. However, in this report, we provide an economic method using calcium phosphate precipitation that can achieve over 90% transfection efficiency in 293T cells. Compared to constitutive shRNA expression vectors, an inducible shRNA system is particularly suitable to knocking down a gene essential to cell proliferation. We demonstrate the gene silencing of Yin Yang 1 (YY1), a potential oncogene in breast cancer4,5, by a Tet-On inducible shRNA system and its effects on tumor formation. Research using lentivirus requires review and approval of a biosafety protocol by the Biosafety Committee of a researcher's institution. Research using animal models requires review and approval of an animal protocol by the Animal Care and Use Committee (ACUC) of a researcher's institution.

RNA interference (RNAi) possesses many advantages over gene knockout and has been broadly used as a tool in gene functional studies. The invention of DNA vector-based RNAi technology has made long term and inducible gene knockdown possible, and also increased the feasibility of gene silencing in vivo.

RNA interference (RNAi) inhibits gene expression by specifically degrading target mRNAs. Since the discovery of double-stranded small interference RNA ...

Utilizing Repetitive Transcranial Magnetic Stimulation to Improve Language Function in Stroke Patients with Chronic Non-fluent Aphasia

Transcranial magnetic stimulation (TMS) has been shown to significantly improve language function in patients with non-fluent aphasia1. In this experiment, we demonstrate the administration of low-frequency repetitive TMS (rTMS) to an optimal stimulation site in the right hemisphere in patients with chronic non-fluent aphasia. A battery of standardized language measures is administered in order to assess baseline performance. Patients are subsequently randomized to either receive real rTMS or initial sham stimulation. Patients in the real stimulation undergo a site-finding phase, comprised of a series of six rTMS sessions administered over five days; stimulation is delivered to a different site in the right frontal lobe during each of these sessions. Each site-finding session consists of 600 pulses of 1 Hz rTMS, preceded and followed by a picture-naming task. By comparing the degree of transient change in naming ability elicited by stimulation of candidate sites, we are able to locate the area of optimal response for each individual patient. We then administer rTMS to this site during the treatment phase. During treatment, patients undergo a total of ten days of stimulation over the span of two weeks; each session is comprised of 20 min of 1 Hz rTMS delivered at 90% resting motor threshold. Stimulation is paired with an fMRI-naming task on the first and last days of treatment. After the treatment phase is complete, the language battery obtained at baseline is repeated two and six months following stimulation in order to identify rTMS-induced changes in performance. The fMRI-naming task is also repeated two and six months following treatment. Patients who are randomized to the sham arm of the study undergo sham site-finding, sham treatment, fMRI-naming studies, and repeat language testing two months after completing sham treatment. Sham patients then cross over into the real stimulation arm, completing real site-finding, real treatment, fMRI, and two- and six-month post-stimulation language testing.

We explore the use of repetitive transcranial magnetic stimulation (rTMS) to improve language abilities in patients with chronic stroke and non-fluent aphasia. After identifying a site in the right frontal gyrus for each patient that responds optimally to stimulation, we target this site during ten days of rTMS treatment.

Transcranial magnetic stimulation (TMS) has been shown to significantly improve language function in patients with non-fluent aphasia1. In this ...

An Isolated Working Heart System for Large Animal Models

Since its introduction in the late 19th century, the Langendorff isolated heart perfusion apparatus, and the subsequent development of the working heart model, have been invaluable tools for studying cardiovascular function and disease1-15. Although the Langendorff heart preparation can be used for any mammalian heart, most studies involving this apparatus use small animal models (e.g., mouse, rat, and rabbit) due to the increased complexity of systems for larger mammals1,3,11. One major difficulty is ensuring a constant coronary perfusion pressure over a range of different heart sizes &#8211; a key component of any experiment utilizing this device1,11. By replacing the classic hydrostatic afterload column with a centrifugal pump, the Langendorff working heart apparatus described below allows for easy adjustment and tight regulation of perfusion pressures, meaning the same set-up can be used for various species or heart sizes. Furthermore, this configuration can also seamlessly switch between constant pressure or constant flow during reperfusion, depending on the user&#8217;s preferences. The open nature of this setup, despite making temperature regulation more difficult than other designs, allows for easy collection of effluent and ventricular pressure-volume data.

Most studies involving the Langendorff apparatus use small animal models due to the increased complexity of systems for larger mammals. We describe a Langendorff system for large animal models that allows for use across a range of species, including humans, and relatively easy data acquisition.

Since its introduction in the late 19th century, the Langendorff isolated heart perfusion apparatus, and the subsequent development of the working heart ...

Development of an Alpha-synuclein Based Rat Model for Parkinson's Disease via Stereotactic Injection of a Recombinant Adeno-associated Viral Vector

In order to study the molecular pathways of Parkinson&#39;s disease (PD) and to develop novel therapeutic strategies, scientific investigators rely on animal models. The identification of PD-associated genes has led to the development of genetic PD models. Most transgenic &#945;-SYN mouse models develop gradual &#945;-SYN pathology but fail to display clear dopaminergic cell loss and dopamine-dependent behavioral deficits. This hurdle was overcome by direct targeting of the substantia nigra with viral vectors overexpressing PD-associated genes. Local gene delivery using viral vectors provides an attractive way to express transgenes in the central nervous system. Specific brain regions can be targeted (e.g. the substantia nigra), expression can be induced in the adult setting and high expression levels can be achieved. Further, different vector systems based on various viruses can be used. The protocol outlines all crucial steps to perform a viral vector injection in the substantia nigra of the rat to develop a viral vector-based alpha-synuclein animal model for Parkinson&#39;s disease.

Development of an Alpha-synuclein Based Rat Model for Parkinson&#039;s Disease via Stereotactic Injection of a Recombinant Adeno-associated Viral Vector

This manuscript describes how viral vector-mediated local gene delivery provides an attractive way to express transgenes in the central nervous system. The protocol outlines all crucial steps to perform a viral vector injection in the substantia nigra of the rat to develop a viral vector-based animal model for Parkinson's disease.

In order to study the molecular pathways of Parkinson&#39;s disease (PD) and to develop novel therapeutic strategies, scientific investigators rely on ...

In Vitro Enzyme Measurement to Test Pharmacological Chaperone Responsiveness in Fabry and Pompe Disease

The use of personalized medicine to treat rare monogenic diseases like lysosomal storage disorders (LSDs) is challenged by complex clinical trial designs, high costs, and low patient numbers. Hundreds of mutant alleles are implicated in most of the LSDs. The diseases are typically classified into 2 to 3 different clinical types according to severity. Moreover, molecular characterization of the genotype can help predict clinical outcomes and inform patient care. Therefore, we developed a simple cell culture assay based on HEK293H cells heterologously over-expressing the mutations identified in Fabry and Pompe disease. A similar assay has recently been introduced as a preclinical test to identify amenable mutations for Pharmacological Chaperone Therapy (PCT) in Fabry disease. This manuscript describes an amended cell culture assay which enables rapid phenotypic assessment of allelic variants in Fabry and Pompe disease to identify eligible patients for PCT and may aid in the development of novel pharmacochaperones.

In Vitro Enzyme Measurement to Test Pharmacological Chaperone Responsiveness in Fabry and Pompe Disease

There is a demand to make pre-clinical testing for a novel class of "orphan" drugs called pharmacological chaperones reproducible, fast, and efficient. We developed a simple, highly standardized, and versatile cell culture-based assay to screen for eligible patients as well as novel pharmacological chaperone drugs.

The use of personalized medicine to treat rare monogenic diseases like lysosomal storage disorders (LSDs) is challenged by complex clinical trial designs, ...

Direct Intrabronchial Administration to Improve the Selective Agent Deposition Within the Mouse Lung

Intratracheal (IT) administration of experimental agents is an essential technique in murine models of diffuse lung diseases, such as bleomycin-induced pulmonary fibrosis.&#160; However, distribution of intratracheally-administered agents to the distal mouse lung is often asymmetric, with lung parenchymal concentrations increased in the smaller (but equally accessible) left lung of the mouse.&#160; Described in this report is a novel intrabronchial (IB) approach to cannulate the left and/or right lungs of living mice non-operatively.&#160; It is also demonstrated how this approach can be used to selectively administer agents to one lung or adapted (via dose-adjusted IB delivery) to improve the left-right symmetry of lung delivery of experimental agents, thereby improving models of diffuse lung disease such as bleomycin-induced pulmonary fibrosis.

Intratracheal (IT) administration of experimental agents in mice often results in asymmetric delivery to the distal lungs.&#160; In this report, we describe a direct intrabronchial (IB) approach to cannulate each lung in living mice non-operatively.&#160; This approach can be used to selectively administer agents to one lung or may be adapted to improve the symmetric agent delivery to both lungs.

Intratracheal (IT) administration of experimental agents is an essential technique in murine models of diffuse lung diseases, such as bleomycin-induced ...

In Vitro Model of Coronary Angiogenesis

Here, we describe an in vitro culture assay to study coronary angiogenesis. Coronary vessels feed the heart muscle and are of clinical importance. Defects in these vessels represent severe health risks such as in atherosclerosis, which can lead to myocardial infarctions and heart failures in patients. Consequently, coronary artery disease is one of the leading causes of death worldwide. Despite its clinical importance, relatively little progress has been made on how to regenerate damaged coronary arteries. Nevertheless, recent progress has been made in understanding the cellular origin and differentiation pathways of coronary vessel development. The advent of tools and technologies that allow researchers to fluorescently label progenitor cells, follow their fate, and visualize progenies in vivo have been instrumental in understanding coronary vessel development. In vivo studies are valuable, but have limitations in terms of speed, accessibility, and flexibility in experimental design. Alternatively, accurate in vitro models of coronary angiogenesis can circumvent these limitations and allow researchers to interrogate important biological questions with speed and flexibility. The lack of appropriate in vitro model systems may have hindered the progress in understanding the cellular and molecular mechanisms of coronary vessel growth. Here, we describe an in vitro culture system to grow coronary vessels from the sinus venosus (SV) and endocardium (Endo), the two progenitor tissues from which many of the coronary vessels arise. We also confirmed that the cultures accurately recapitulate some of the known in vivo mechanisms. For instance, we show that the angiogenic sprouts in culture from SV downregulate COUP-TFII expression similar to what is observed in vivo. In addition, we show that VEGF-A, a well-known angiogenic factor in vivo, robustly stimulates angiogenesis from both the SV and Endo cultures. Collectively, we have devised an accurate in vitro culture model to study coronary angiogenesis.

In vitro models of coronary angiogenesis can be utilized for the discovery of the cellular and molecular mechanisms of coronary angiogenesis. In vitro explant cultures of sinus venosus and endocardium tissues show robust growth in response to VEGF-A and display a similar pattern of COUP-TFII expression as in vivo.

Here, we describe an in vitro culture assay to study coronary angiogenesis. Coronary vessels feed the heart muscle and are of clinical importance. Defects ...

Involving Individuals with Developmental Language Disorder and Their Parents/Carers in Research Priority Setting

A protocol for involving individuals presenting with developmental language disorder (DLD) (iDLD) and their parents/carers (iDLDPC) in a research priority setting exercise is presented.
iDLD have difficulties with communication skills, such as understanding language, word-finding and discourse. Such difficulties mean existing research priority setting protocols are difficult for iDLD to access, since they require sophisticated communication skills. Thus, a novel protocol for involving iDLD in these exercises is warranted. The same protocol is recommended for use with iDLDPC, to ensure accessibility.
The protocol is presented in 4 steps. Step 1 describes a program of activities delivered by trained, specialist DLD speech and language therapists (SLTs) that prepares iDLD/iDLDPC for involvement. Step 2 outlines an approach to elicit iDLD/iDLDPC&#8217;s opinions on research priorities. Steps 3 and 4 describe methods to analyze and integrate this data at multiple stages of the research priority setting process.
9 trained specialist DLD SLTs delivered steps 1 and 2. 17 iDLDs and 25 iDLDPCs consented to involvement. Opinions from all participants were elicited, and this data was used to influence the process and output of the exercise.
An advantage of this protocol is its accommodation of the heterogeneity in support needs of iDLD/iDLDPC, through a menu of options, whilst also providing a structured framework. Due to the novelty of the protocol, the methods for data integration were developed by the research group. These are potential limitations of the protocol, and may bring the reliability and validity under scrutiny, which are yet to be tested.
This protocol enables meaningful involvement of iDLD/iDLDPC in&#160;research priority setting and could be utilized for people with other kinds of speech, language or communication needs. Further research should evaluate the effectiveness of the protocol and whether it can be adapted for involvement of such populations in other research studies.

A protocol to enable individuals with developmental language disorder (DLD) and their parents/carers to meaningfully participate in a research priority setting exercise is established. The protocol includes a defined program of activities for data collection, and methods to incorporate this data into the broader research priority setting process.

A protocol for involving individuals presenting with developmental language disorder (DLD) (iDLD) and their parents/carers (iDLDPC) in a research priority ...

A Rat Lung Transplantation Model of Warm Ischemia/Reperfusion Injury: Optimizations to Improve Outcomes

From our experience with rat lung transplantation, we have found several areas for improvement. Information in the existing literature regarding methods for choosing appropriate cuff sizes for the pulmonary vein (PV), pulmonary artery (PA), or bronchus (Br) are varied, thus making the determination of proper cuff size during rat lung transplantation an exercise of trial and error. By standardizing the cuffing technique to use the smallest effective cuff appropriate for the size of the vessel or bronchus, one can make the transplantation procedure safer, faster, and more successful. Since diameters of the PV, PA, and Br are related to the body weight of the rat, we present a strategy to choosing an appropriate size using a weight-based guide. Since lung volume is also related to body weight, we recommend that this relationship should also be considered when choosing the proper volume of air for donor lung inflation during warm ischemia as well as for the proper volume of PBS to be instilled during bronchoalveolar lavage (BAL) fluid collection. We also describe methods for 4th intercostal space dissection, wound closure, and sample collection from both the native and transplanted lobes.

Here, we present optimizations to a rat lung transplantation model that serve to improve outcomes. We provide a size guide for cuffs based on body weight, a measurement strategy to ascertain the 4th intercostal space, and methods of wound closure and BAL (bronchoalveolar lavage) fluid and tissue collection.

From our experience with rat lung transplantation, we have found several areas for improvement. Information in the existing literature regarding methods ...

Tissue Preparation Techniques for Contrast-Enhanced Micro Computed Tomography Imaging of Large Mammalian Cardiac Models with Chronic Disease

Structural remodeling is a common consequence of chronic pathological stresses imposed on the heart. Understanding the architectural and compositional properties of diseased tissue is critical to determine their interactions with arrhythmic behavior. Microscale tissue remodeling, below the clinical resolution, is emerging as an important source of lethal arrhythmia, with high prevalence in young adults. Challenges remain in obtaining high imaging contrast at sufficient microscale resolution for preclinical models, such as large mammalian whole hearts. Moreover, tissue composition-selective contrast enhancement for three-dimensional high-resolution imaging is still lacking. Non-destructive imaging using micro-computed tomography shows promise for high-resolution imaging. The objective was to alleviate sufferance from X-ray over attenuation in large biological samples. Hearts were extracted from healthy pigs (N = 2), and sheep (N = 2) with either induced chronic myocardial infarction and fibrotic scar formation or induced chronic atrial fibrillation. Excised hearts were perfused with: a saline solution supplemented with a calcium ion quenching agent and a vasodilator, ethanol in serial dehydration, and hexamethyldisilizane under vacuum. The latter reinforced the heart structure during air-drying for 1 week. Collagen-dominant tissue was selectively bound by an X-ray contrast-enhancing agent, phosphomolybdic acid. Tissue conformation was stable in air, permitting long-duration microcomputed tomography acquisitions to obtain high-resolution (isotropic 20.7 &#181;m) images. Optimal contrast agent loading by diffusion showed selective contrast enhancement of the epithelial layer and sub-endocardial Purkinje fibers in healthy pig ventricles. Atrial fibrillation (AF) hearts showed enhanced contrast accumulation in the posterior walls and appendages of the atria, attributed to greater collagen content. Myocardial infarction hearts showed increased contrast selectively in regions of cardiac fibrosis, which enabled the identification of interweaving surviving myocardial muscle fibers. Contrast-enhanced air-dried tissue preparations enabled microscale imaging of the intact large mammalian heart and selective contrast enhancement of underlying disease constituents.

Here, we present a protocol to obtain high resolution micro computed tomography images of healthy and pathological large mammalian whole hearts with collagen-selective contrast enhancement.

Structural remodeling is a common consequence of chronic pathological stresses imposed on the heart. Understanding the architectural and compositional ...