Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Summary

We demonstrate how to deploy a real-time psychosis risk calculation and alerting system based on CogStack, an information retrieval and extraction platform for electronic health records.

Abstract

Recent studies have shown that an automated, lifespan-inclusive, transdiagnostic, and clinically based, individualized risk calculator provides a powerful system for supporting the early detection of individuals at-risk of psychosis at a large scale, by leveraging electronic health records (EHRs). This risk calculator has been externally validated twice and is undergoing feasibility testing for clinical implementation. Integration of this risk calculator in clinical routine should be facilitated by prospective feasibility studies, which are required to address pragmatic challenges, such as missing data, and the usability of this risk calculator in a real-world and routine clinical setting. Here, we present an approach for a prospective implementation of a real-time psychosis risk detection and alerting service in a real-world EHR system. This method leverages the CogStack platform, which is an open-source, lightweight, and distributed information retrieval and text extraction system. The CogStack platform incorporates a set of services that allow for full-text search of clinical data, lifespan-inclusive, real-time calculation of psychosis risk, early risk-alerting to clinicians, and the visual monitoring of patients over time. Our method includes: 1) ingestion and synchronization of data from multiple sources into the CogStack platform, 2) implementation of a risk calculator, whose algorithm was previously developed and validated, for timely computation of a patient's risk of psychosis, 3) creation of interactive visualizations and dashboards to monitor patients' health status over time, and 4) building automated alerting systems to ensure that clinicians are notified of patients at-risk, so that appropriate actions can be pursued. This is the first ever study that has developed and implemented a similar detection and alerting system in clinical routine for early detection of psychosis.

Introduction

Psychotic disorders are serious mental health illnesses that lead to difficulties in distinguishing between the internal experience of the mind and the external reality of the environment¹, as well as a higher than average risk of self-harm and suicide². Under standard care, these disorders result in major public health impact with a significant health and economic burden on individuals, families and societies worldwide³. Early interventions in psychosis can improve outcomes of this mental disorder⁴. In particular, detection, prognostic assessment and preventive treatment of individuals who are at clinical high risk for developing psychosis (CHR-P)⁵ provides a unique potential to alter the course of the disorder, thereby improving the quality of life for many people and their families^3,6. CHR-P individuals are help-seeking young people presenting with attenuated symptoms and functional impairment⁷: their risk of developing psychosis is 20% at 2-years⁸ but it is higher in some specific subgroups^9,10. Despite some substantial advancements, the impact of preventive approaches in routine clinical practice is limited by the ability to detect most individuals who are at-risk¹¹. Current detection methods are based on help-seeking behaviors and referrals on suspicion of psychosis risk; these methods are highly inefficient in handling a large number of samples¹¹. Thus, the scalability of current detection methods to the vast majority of the at-risk population is quite limited¹². In fact, only 5% (standalone specialized early detection services) to 12% (youth mental health services) of individuals at-risk of developing a first psychotic disorder can be detected at the time of their at-risk stage by the current detection strategies⁶.

To extend the clinical benefits of the preventive approaches in a larger number of at-risk individuals, we developed an automated, lifespan-inclusive (i.e., across all ages), transdiagnostic (i.e., across different diagnoses)¹³, clinically-based individualized risk calculator, which can detect individuals at-risk of psychosis in secondary mental health care at scale, beyond those meeting CHR-P criteria¹⁴. This risk calculator used a Cox proportional hazard model to predict the risk of developing a psychotic disorder over six years from five routinely collected clinical variables selected a priori, in line with methodological guidelines¹⁵: age, gender, ethnicity, age-by-gender and primary index diagnosis. These clinical variables were selected based on a priori knowledge obtained from meta analyses^16,17, as recommended by the state-of-the-art methodological guidelines¹⁵. The number of predictors is limited to preserve the Event Per Variable ratio and minimize overfitting biases; including too many variables without a priori filter leads to overfitting problems and poor prognostic accuracy¹⁸. The method used to develop this model provides similar prognostic accuracy to automatic machine learning methods¹⁸. Parameters of the Cox model were estimated based on a retrospective de-identified cohort from the South London and Maudsley National Health Service Foundation Trust (SLaM)¹⁹. SLaM is a National Health Service (NHS) mental health trust that provides secondary mental health care to a population of 1.36 million individuals in South London (Lambeth, Southwark, Lewisham and Croydon boroughs), and has one of the highest recorded rates of psychosis in the world²⁰. All data used in the model development were extracted from the Clinical Record Interactive Search (CRIS) platform, a digital case register system, which provides researchers with retrospective access and analysis of anonymized clinical records¹⁹. The clinical information in CRIS is extracted from a bespoke Electronic Health Record (EHR) system, at SLaM, called electronic Patient Journey System (ePJS). SLaM is paper-free and ePJS represents the standard data collection platform for clinical routine. Thus, the transdiagnostic risk calculator leverages EHRs and has the potential to automatically screen large EHRs of patients accessing secondary mental healthcare, to detect those who may be at-risk of psychosis. The algorithm of this transdiagnostic risk calculator has been published previously^6,14,21. The transdiagnostic risk calculator has been externally validated in two NHS Foundation Trusts^14,21 and optimized²², demonstrating its adequate prognostic performance and generalizability across different populations.

According to methodological guidelines on the development of a risk prediction model^15,23, the next step after model development and validation is to implement the prediction model in routine clinical practice. Implementation studies are usually preceded by pilot or feasibility studies that address potential pragmatic limitations associated with the use of risk algorithms in clinical practice. For example, required data for running a calculator, such as age, gender and ethnicity, may not be available at the date of diagnosis or updated later. Effective methods for handling missing data and synchronizing frequent updates in real-time data streams should be considered to obtain the most reliable prediction results in an implementation. Furthermore, since the initial development of the risk calculator was based on retrospective cohort data, it is not known whether it can be used in a real-time data stream that is typical of a real-world clinical setting. Another challenge is ensuring that relevant clinicians receive the recommendations generated by the risk calculator within an appropriate time frame and within a shared and accepted communication pathway.

To overcome these limitations, we have completed a feasibility implementation study employing the individualized transdiagnostic risk calculator. The study included two phases: an in vitro phase that was conducted using data from the local EHR, without contacting clinicians or patients, and an in vivo phase, which involved direct contact with clinicians. The in vitro phase had two manifold aims: (i) to address implementation barriers according to the Consolidated Framework for Implementation Research (CFIR)²⁷ and (ii) to integrate the transdiagnostic risk calculator into the local EHR. Implementation barriers included the communication of risk outcomes to clinicians. In SLaM, all patients are invited to register for Consent for Contact (C4C), which indicates their willingness to be contacted for research, without affecting the quality of care. This reduces the ethical issues relating to contacting patients. Further to this, working groups with clinicians aided tailoring of how this information was communicated. During the in vivo phase (May 14th 2018 to April 29th 2019), all individuals (i) older than 14 years (ii) who were accessing any SLaM service (boroughs of Lambeth, Southwark, Lewisham, Croydon), (iii) receiving a first ICD-10 index primary diagnosis of any non-organic, non-psychotic mental disorder (with the exception of Acute and Transient Psychotic Disorders; ATPD), or a CHR-P designation and (iv) with existing contact details were deemed eligible. During the in vivo phase, new patients accessing SLaM each week were automatically screened for their psychosis risk, and those with having a risk greater than a certain threshold were detected. The research team then contacted the patients' responsible clinicians to discuss further recommendations and eventually suggest a further face to face assessment⁶. If those assessed were considered to meet CHR-P criteria, they were referred to specialist CHR-P services, such as Outreach and Support in South London (OASIS)²⁸. This would result in improved detection of individuals prior to the onset of a psychotic disorder and provide a significant opportunity for altering the course of the disorder. Crucially, this feasibility study involved the full integration of the calculator into the local EHR system, which is the topic of the current article. The full protocol of this feasibility study, including an overview of the plan for evaluating the proposed research, details on managing data security and ethical issues, has been presented in our previous work⁶. The current article, as a part of the feasibility study⁶, selectively focuses on presenting the technical implementation of a real-time psychosis risk detection and alerting system based on the local EHR data. More specifically, the aim of this study is to investigate the technical feasibility of this risk calculator in timely detecting at-risk patients as soon as they access a secondary mental healthcare service. The full results of the feasibility study, in terms of clinicians' adherence to the recommendations made by the risk calculator, will be presented separately. A comprehensive evaluation of the effectiveness of the proposed research, which requires randomized designs, is outside the scope of the current research program. To our best knowledge, this is the first method describing the implementation of a risk calculator based on live EHR data for early detection of psychosis.

Our approach to psychosis risk detection and alerting takes advantage of the CogStack platform. The CogStack platform is a lightweight, distributed, and fault-tolerant information retrieval and text-extraction platform²⁴. This platform consists of three key components: 1) the CogStack Pipeline that uses the Java Spring Batch framework to ingest and synchronize data from a pre-defined data source (both structured and unstructured EHR data in multiple formats such as Word, PDF files and images) to a predefined data sink in real time; 2) Elasticsearch, a search engine allowing for storage and querying of the full text of EHR data, as well as providing various application programming interfaces (APIs) to embed advanced analytics into the engine; and 3) Kibana, an interactive, web-based user interface that allows users to query data in Elasticsearch, build visualization dashboards and set alerting on anomalies or other patterns of interest from data. Moreover, CogStack incorporates the ability to alert clinicians to potential problems by Email and SMS (text), allowing clinicians to receive timely notifications about at-risk patients reported by the risk calculator.

We present a model of psychosis risk detection and alerting based on ePJS at SLaM, leveraging the CogStack platform. Compared with the CRIS platform that provides a mechanism for retrospective access to de-identified health records from ePJS on a weekly basis¹⁹, the CogStack platform at SLaM enables access to an identifiable EHR in real time, bringing the alerting closer to the point-of-care and the risk prediction in a prospective design, although both the CRIS and CogStack platforms use data sourced from ePJS in SLaM. In the section that follows, we provide details of the key steps in our approach, including preparing source data from the EHR, ingesting the source data into the CogStack platform to enable full-text search via Elasticsearch, running the psychosis risk calculator using a Python daemon thread, and setting interactive visualizations and real-time risk alerting via the Kibana user interface. Any researcher who aims to build a real-time risk detection and alerting system based on EHR data can follow the approach and its reference implementation. As we shall elaborate below, the proposed method exploits open-source, lightweight techniques with high flexibility and portability. This enables the risk calculator to be run in various locations and shows a high applicability to other risk estimation algorithms. Moreover, the method works as a straightforward approach to enhance the risk detection and alerting functionalities of an EHR embedded in a general healthcare system.

Protocol

This study was approved by East of England - Cambridgeshire and Hertfordshire Research Ethics Committee (Reference number: 18/EE/0066).

NOTE: We have developed this protocol based on the CogStack platform and the Python programming language. This system requires Docker (more specifically, Docker Compose https://docs.docker.com/compose/), Anaconda Python (https://www.anaconda.com/distribution/) and Git (https://git-scm.com/downloads) pre-installed on a device. The commands provided in this protocol are based on the Linux environment. In the following, we provide the details of preparing source data from an EHR database, ingesting the data to CogStack platform, and setting up a real-time risk calculation and alerting system for psychosis based on the CogStack platform. Moreover, an online version of the risk calculator was developed to facilitate numeric calculation of the probability of an individual developing psychosis in secondary mental health care on http://www.psychosis-risk.net.

1. Source data preparation

NOTE: In most use cases, CogStack ingests source data from a specified database view that can combine data from one or more source database tables, where a view is a searchable object in a database that contains the result set of a stored query on the data. The setup of the ingesting view is tailored by the specific use cases and deployment settings of a health record database system. This protocol is developed based on a psychosis risk calculator developed and externally validated twice by Fusar-Poli et al.^14,21 and as part of a pilot implementation feasibility study⁶. The protocol is based on an EHR database deployed with Microsoft SQL Server 2014.

1. Create a view object (called "vwPsychosisBase" in this protocol) in an existing EHR database system to join necessary information of patients for psychosis risk calculation and alerting. Make sure that this view includes all patients receiving a first primary diagnosis of non-organic and non-psychotic mental disorder (recorded by the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [ICD-10]), as defined in the original model^14,21.
2. Ensure that each record in the view involves three types of patient information: 1) the first primary diagnosis of a patient in the EHR system, including ICD-10 diagnosis index (diagnoses were clustered together into the following ten clusters: acute and transient psychotic disorders, anxiety disorders, bipolar mood disorders, childhood and adolescent onset disorders, developmental disorders, nonbipolar mood disorders, mental retardation, personality disorders, physiological syndromes, substance use disorders) and diagnosis date; 2) a patient's demographic data, including gender, ethnicity and date of birth; and 3) the most recent contact information of care team for a patient, such as details of general practice (GP), consultants and care coordinators. The first two types of information are vital for the psychosis risk calculator^14,21, and the third type of information is to enable timely risk alerting.
3. Make sure that each record in the view has a unique identifier (e.g., "patient_id" used in this protocol).
4. Select the last update timestamps of all source information related to a record in the view (e.g., the last update times of a patient's demographic information and the patient's first primary diagnosis information), and choose the latest timestamp as the last update date and time for the record in the view (denoted as "etl_updated_dttm" in this protocol). The last update date and time of a record allows CogStack to synchronize updates in the database, such as new and updated records.

2. Data ingestion

1. Download or clone the code repository from Github (https://github.com/cogstack-slam/psychosis) or by typing "git clone https://github.com/cogstack-slam/psychosis.git" in a terminal window. The downloaded folder contains the code for psychosis risk calculation and configuration files for deploying a CogStack instance.
2. Go to the "cogstack_deploy/cogstack/" directory and modify "psychosis.properties" to configure CogStack Pipeline for data ingestion. Modify the settings of section "SOURCE: DB CONFIGURATIONS" based on the EHR database setup, including specifying the IP address of the database server, database name, database username and password. Modify the view name (i.e. "vwPsychosisBase") and field names (e.g., "patient_id" and "etl_updated_dttm") if necessary. In case of error in configuring this file, follow the instructions at https://cogstack.atlassian.net/wiki/spaces/COGDOC/pages/38043684/Quickstart.
3. Go to the "cogstack_deploy/common/elasticsearch/config/" directory and modify the section "xpack.notification.email.account" in the "elasticsearch.yml" file to configure an Email address for sending alerts. A detailed instruction for Email configuration can be found on https://www.elastic.co/guide/en/kibana/6.4/watcher-create-threshold-alert.html.
4. Go to the "cogstack_deploy/" directory and type "docker-compose up" to run the CogStack platform. Execute this command with root access. If the process is completed successfully, there will be printed status logs of the currently running services, including CogStack Pipeline, Elasticsearch and Kibana, in the terminal. As a result, all data and updates in the source database view will be timely ingested to an Elasticsearch index called "psychosis_base" in the CogStack platform.
5. Open a web browser and access Kibana user interface by typing "http://localhost:5601/" (or replacing "localhost" with a specific IP address of the server running the CogStack platform). For the first time accessing Kibana, click the Management tab and Index Patterns tab to specify an Elasticsearch index that one wants to access with Kinaba. Type "psychosis_base" in the "Index pattern" field and click Next step. Select "etl_updated_dttm" for the "Time Filter" field name and click Create index pattern to add the "psychosis_base" index pattern for Kinana.
6. Once Kibana is connected to the Elasticsearch index (i.e., "psychosis_base"), search and browse the source data interactively through the "Discover" page. Kibana allows non-technical users to search for both structured metadata and free text. Detailed instructions of using "Discover" are available on https://www.elastic.co/guide/en/kibana/6.4/discover.html.

3. Risk calculation

1. Open a new terminal window and go to the "psychosis/" directory. Install all required Python packages (including "elasticsearch", "elasticsearch_dsl", "pandas" and "numpy") used in the risk calculator by typing "conda install package-name" or "pip install package-name" in the terminal.
2. Type "python risk_calculator.py" to run the psychosis risk calculator. If the process is completed successfully, logs of the risk calculation will be printed in the terminal and the risk results will be stored in a new Elasticsearch index called "psychosis_risk" within the CogStack platform.
3. Check the risk results by using the Kibana interface. Similar to Steps 2.5 and 2.6, add a new index pattern "psychosis_risk" to connect Kinbana with the "psychosis_risk" index, and explore the risk results through the "Discover" page. To facilitate identifying new patients at-risk, use "first_primary_diagnosis_date" as the "Time Filter" field in building the "psychosis_risk" index. When exploring data in the "Discover" page, make sure that the index pattern "psychosis_risk" is selected.

4. Data visualization

1. In addition to searching and accessing individual-level information via the "Discover" page in Kibana, one can build visualizations and dashboards to obtain an overview of characteristics for the whole population of at-risk patients. To do this, click on Visualize in the side navigation of Kibana. Then, click the Create new visualization button and choose a visualization type (e.g., pie and line charts). Select "psychosis_risk" as the index that one wants to visualize through Kibana. By default, visualizations will include all records/patients in the "psychosis_risk" index. Detailed instructions of building Kibana visualizations are available on https://www.elastic.co/guide/en/kibana/6.4/visualize.html.
2. To select a specific subset of data for visualization, add a "filter". For example, selecting a filter filed as "h_2_year", choosing an operator as "is not between" and setting values from "0.0" to "0.05" will only include patients whose risk of psychosis in 2 years are higher than 0.05.
3. Once individual visualizations are built, click on Dashboard in the side navigation of Kibana to create a dashboard that displays a set of related visualizations together. Click Create new dashboard and the Add button to create a new dashboard panel. Click visualizations that one wants to show within the new dashboard panel. Click Save and type a title to save the panel. Instructions on building Kibana dashboards are available at https://www.elastic.co/guide/en/kibana/6.4/dashboard.html.

5. Risk Alerting

1. Click on Management in the side navigation of Kibana and then click Watcher under Elasticsearch to create alerting for clinicians when patients were at-risk of psychosis. If the Watcher button is not visible, click License Management and click Start trial or Update license.
2. Click Create advanced watch to set up a new Watcher. Type an "ID" and "Name". Delete the content of "Watch JSON" section and copy the content in the "watcher.json" file in the "psychosis" directory to the "Watch JSON" section. This watcher will send alerting Email to "clinician@nhs.uk" (which can be replaced with the Email address where one wants to send alerts) from "username@nhs.uk" (which was set in Step 2.3) if there are one or more patients whose risk of psychosis in 2 years are higher than 0.05 (a tentative threshold for feasibility testing) in every 24 hours.
3. Before saving the Watcher, click Simulate to test the Watcher execution. If the Watcher is set successfully, one will see the simulation output printed. In case of error in the settings, follow the instructions on https://www.elastic.co/guide/en/elastic-stack-overview/6.4/watcher-getting-started.html.
4. To stop a Watcher, permanently delete it or temporarily deactivate it from the "Status" page of the Watcher.

Discussion

We have demonstrated the first EHR implementation of a real-time psychosis risk detection and alerting system based on CogStack, an open source information retrieval and extraction platform. Following this approach, one can transform and ingest a large set of clinical data in various formats, including structured and unstructured information, into a CogStack instance, so as to enable full-text search, interactive analyses and visualization of data, as well as real-time alerting to clinicians of patients that are at-risk of psychosis. Although the original psychosis risk calculator has been validated in pilot studies across several NHS Trusts, albeit using retrospective patient records^6,14,21, this experimental design provides the first evidence base that this risk calculator can be replicated and deployed for use in real time. This approach allows the automatic delivery of prognostic results to clinicians through existing clinical notification channels, such as Email, in real time. This clearly demonstrates the technical feasibility for conducting a large-scale effectiveness trial to evaluate the ultimate clinical utility of this risk calculator in the real world.

This protocol is empirically innovative, as there does not exist a similar risk detection and alerting system for psychosis. Moreover, this protocol has high generalizability in clinical use, particularly because of the unique strengths of our approach. From a theoretical perspective, we used a risk prediction model that was developed based on a large retrospective de-identified cohort from the SLaM NHS Trust. SLaM provides secondary mental health care to a population of 1.36 million individuals in South London and has one of the highest recorded rates of psychosis in the world. This large cohort, which has high diversity in sociodemographic and diagnostic characteristics, allows us to develop a risk prediction model that is unlikely to be biased towards a population with specific characteristics. This is supported by evidence that the prognostic accuracy of this risk calculator has already been replicated twice in two different databases^14,21, including one outside of SLaM. Another theoretical strength of this risk model is that basic demographic and clinical diagnosis information were used as predictors. Such information is ubiquitous in electronic clinical data and in fact missing data for these predictors have been shown to be relatively rare in our previous studies^14,21. The high availability of information for building predictors makes it possible to run the risk calculator over a large number of patient samples across different secondary mental health care sectors. In addition, the risk calculator is a general algorithm which is suitable for all individuals at-risk of developing psychosis in secondary mental health care, regardless of individuals' ages. That is, this calculator is not only suitable for the 15-35 age range of peak psychosis risk¹⁶, but also for those outside of this range, showing a high degree of generalizability.

From a practical perspective, both the risk calculator and the CogStack platform are light-weight and open-source services that do not involve resource-heavy techniques or costly infrastructure. Such a low-cost and easy-to-deploy platform can reduce the barriers to its adoption in real-world clinical settings. Also, our solution overcomes the main implementation barrier: risk estimation systems provide little value unless they are used by clinicians in day-to-day practice²⁵. Specifically, our approach accesses data from the EHR, performs analyses independent of an electronic medical record system and can send analysis results back to clinicians through existing notification channels. This method does not require that the business logic in pre-existing systems be modified and can work as a standalone service to support and extend existing clinical decision support systems. Thus, the protocol has high compatibility with pre-existing clinical systems and can be easily integrated into routine clinical practice. Moreover, the protocol provides user-friendly interfaces for searching, analyzing and visualizing of clinical data, which make it easy for clinicians to interpret and explore the risk results.

This protocol also has its limitations. First, the effectiveness of this protocol has not been evaluated in routine clinical practice. This study focused on technical feasibility tests of implementing a real-time psychosis risk detection and alerting system in a local EHR. To further evaluate the effectiveness of this system in routine clinical practice, future large scale randomized controlled trials are needed⁶. A second limitation is that the predictions of risk scores in this protocol were made based on the first primary diagnoses, which are static data collected at a single snapshot. However, the CHR-P symptoms are intrinsically evolving over time. A dynamic version of psychosis risk calculator, in which prediction models can be dynamically updated to reflect the changes, has been developed recently²⁶. Future work will focus on integrating this dynamic calculator in the current protocol.

The most critical step in this approach was identifying EHR data that were used for extract predictors in the risk calculator. This may also involve creating data element mappings, when an EHR system used a data model different from that used in this protocol, such as distinct coding systems for patients' ethnic groups. We have open-sourced all the code and mapping definitions online (https://github.com/cogstack-slam/psychosis). Based on these materials, one would be able to replicate the deployment or adjust the calculator depending on one's own circumstance. Another critical step was creating a database view for data ingestion in CogStack. Since relational join operations (i.e., combining columns from one or more database tables) in Elasticsearch can lead to high computational cost, we conducted these join operations in the EHR database by creating a database view. This view combined all information that was needed to extract predictors in the risk calculator, and two vital fields that were used by CogStack pipelines for data partitioning in data ingestion. The first field is a unique primary key for each record in the view ("patient_id" used this protocol) and the second is a timestamp when a record was modified most recently. If these two fields were not set properly, CogStack might not synchronize data updates in an EHR database timely. Detailed instructions for troubleshooting issues on CogStack data ingestion are available on https://cogstack.atlassian.net/wiki/spaces/COGDOC/overview and https://github.com/CogStack/CogStack-Pipeline.

This protocol is highly transportable and can be easily deployed in NHS Trusts that have a CRIS or CogStack platform. So far, the CRIS platform-including the consenting procedures-has been fully described elsewhere and is under expansion across 12 NHS Trusts in the UK, harnessing over 2 million deidentified patient records (https://crisnetwork.co/). Similarly, the CogStack platform has been deployed not only in SLaM, but also other NHS Trusts across the UK such as University College London Hospitals (UCLH), King's College Hospital (KCH), Guy's and St Thomas' (GSTT), and Mersey Care NHS Trusts. Those Trusts without such as platform can use an online version of risk calculator (http://psychosis-risk.net), or build this protocol from scratch based on this manuscript and our online documents. Although this protocol is developed for psychosis risk detection, the architectural design of this protocol is not tied-in to this specific use case. The protocol is flexible enough to allow for reconfiguration and repurposing of the real-time monitoring and alerting components for other risk measurement areas, such as adverse drug reactions, thereby allowing clinicians to timely take action to improve patient care, safety and experience.

Materials

Name	Company	Catalog Number	Comments
CogStack-Pipeline	King's College London		Open source software
Elasticsearch	Elastic NV		Open Source Search & Analytics
Kibana	Elastic NV		Open source data visualization plugin for Elasticsearch
Python packages ("elasticsearch", "elasticsearch_dsl", "pandas" and "numpy")	Open source community		Open source packages