determination of dose threshold using the machine-vision synchronization (mvs) and dose software

使用机器视觉方法的 TEM 工作流程、结果分析和数据管理

Transmission electron microscopes (TEMs) and their capabilities have benefited enormously from advances in cameras, detectors, sample holders, and computing technologies. However, these advances are hampered by disconnected data streams, limitations of human operation, and cumbersome data analysis1,2. Moreover, in situ and operando experiments adapt TEMs into real-time nanoscale laboratories, enabling samples to be studied in gas or liquid environments while simultaneously applying a range of external stimuli3,4,5. The adoption of such complex workflows have only magnified these limitations, and the resulting increase of the size and complexity of these data streams is an area of growing concern. Thus, there is a growing emphasis on utilizing machine-actionability to find, access, interoperate, and reuse data, a practice known as the FAIR principles6. Publishing research data in accordance with the FAIR principles concept has received favorable attention from governmental agencies around the world7,8, and application of the FAIR principles using machine-vision software is a key step in their adoption.
A machine-vision synchronization (MVS) software platform has been developed in response to the specific pain points inherent to performing and analyzing complex, metadata-heavy TEM experiments (particularly in situ and operando experiments)9. Once installed on the TEM, the MVS software connects, integrates, and communicates with the microscope column, detectors, and integrated in situ systems. This enables it to continuously collect images and align those images with their experimental metadata, forming a comprehensive searchable database, a timeline of the experiment from start to finish (Figure 1). This connectivity allows the MVS software to apply algorithms that intelligently track and stabilize a region of interest (ROI), even when samples are undergoing morphological changes. The software applies adjustments to stage, beam, and digital corrections as necessary to stabilize the ROI through its Drift Control and Focus Assist functions. In addition to enriching the images with the raw metadata produced from the different experimental systems, the software can produce new, computational metadata using image analysis algorithms to calculate variables between images, which allow it to automatically correct for sample drift or changes in focus.
TEM images, and their associated metadata collected through the MVS software, are organized as an experimental timeline which can be opened and viewed by anyone via the free, offline version of the analysis software, Studio (hereafter referred to as the analysis software)10. During an experiment, the MVS software syncs and records three types of images from the microscope&#39;s camera or detector, which are displayed at the top of the timeline below the image viewer: single acquisition (individual single aquisition images acquired directly from the TEM software), raw&#160;(images from the detector/camera live stream that have not had any digital drift corrections applied; these images may have been physically corrected via stage movement or beam shift), and drift corrected (images from the detector/camera live stream that have been digitally drift). Data collected during an experiment or session can be further refined into smaller sections or snippets of data, known as collections, with no loss of embedded metadata. From the analysis software, images, image stacks, and metadata can be directly exported into a variety of open format images and spreadsheet types for analysis using other tools and programs.
The framework of microscope control, stabilization, and metadata integration enabled by the MVS software also allows the implementation of additional machine-vision programs or modules, designed to alleviate limitations in current TEM workflows. One of the first modules developed to take advantage of this synchronization platform is electron dose calibration and spatial tracking of beam exposed areas within the sample. All TEM images are formed from the interaction between the sample and the electron beam. However, these interactions can also result in negative, inescapable impacts on the sample, such as radiolysis and knock-on damage11,12, and require a careful balance between applying a high enough electron dose to generate the image and minimizing the resulting beam damage13,14.
Although many users rely on screen current measurements to estimate the electron dose, this method has been shown to widely underestimate the actual beam current15. Qualitative dose values can be obtained via the screen current on the same microscope with the same settings, but reproducing these dose conditions using different microscopes or settings is highly subjective. Additionally, any imaging parameter adjustments made by the user during the experiment, such as spot size, aperture, magnification, or intensity, require a separate measurement of the screen current to calculate the resulting dose. Users must either rigorously limit the imaging conditions used during a given experiment or meticulously measure and record each lens condition used, significantly complicating and extending the experiment beyond what is feasible for normal operation of the microscope16,17.
Dose, referred to as dose software for this protocol, is a dose calibration software module that utilizes a dedicated calibration holder designed to enable automated current measurements. A Faraday cup, the gold standard for accurate beam current calibration15, is integrated into the tip of the calibration holder. The MVS software performs a series of beam current and beam area calibrations for each lens condition and embeds those values on the images at the pixel level.
In this video article, MVS software protocols designed to enhance all areas of the TEM workflow are presented using representative nanomaterial samples. A beam sensitive zeolite nanoparticle sample14 is used to demonstrate the calibration and dose management workflows. We perform a representative in situ heating experiment using an Au/FeOx nanocatalyst18,19 sample that undergoes significant morphological changes when heated. This in situ experiment highlights the software&#39;s stabilization algorithms and its ability to collate multiple streams of metadata, which is an inherent challenge for in situ and operando studies. Although not described in the protocol, because of its unique electron dose sensitivity, we discuss representative examples of the software&#39;s utility for liquid-EM studies (protocols for which have been previously reported in the literature20,21,22), and how these techniques can be applied to improve the understanding of the effect of dose on liquid-EM experiments. Finally, we show how data analysis is streamlined using the offline analysis software to visualize, filter, and export a variety of image, video, and data files into other accessible formats.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/65446/65446fig01.jpg" /> 
Figure 1: User interface examples for MVS and analysis software. (A) The synchronization software image viewing pane and control panel. A connection between the TEM and the synchronization software is established by activating the Connect button, which streams the images and metadata from the microscope into the synchronization software. From the image viewer, the operator can perform a variety of machine-vision assisted operations, such as Drift Correct and Focus Assist. It also provides the ability to apply Tag Images and Review Session without disrupting data collection. (B) Screenshot of the image analysis software highlighting the location of the Image View Port, Timeline, and the Metadata and Analysis panel. The analysis software can be accessed at any point during an experiment to review the images acquired up to that time point using the Review Session button. <a href="https://www.jove.com/files/ftp_upload/65446/65446fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

作者聚焦：用于透射电子显微镜工作流程、结果分析和数据管理的机器视觉方法  

透射电子显微镜工作流程、结果分析和数据管理的机器视觉方法

This technology was developed to make high-performance TEM studies accessible to users of all skill levels. Furthermore, scientific advancements require the collaborative efforts of multiple investigators, so an efficient platform to share and analyze large, complex data sets is essential for successful research Operando and in situ studies transform a TEM into a real-time nanolaboratory, allowing researchers to explore the dynamic nano-scale processes that control materials'growth properties. One of the greatest challenges for in-situ TEM studies is isolating the beam irradiation effects from the intrinsic behavior of the sample.Accurate electron dose measurement and tracking is essential, but without dose management software, one can only measure electron dose rate rather than the total or critical dose Complex TEM workflows generate large data sets which must be manually aligned and indexed to their respective images. In this time-consuming process, the key information, such as adjustments to imaging conditions or samples environment, may be lost, complicating analysis and reproducibility. We have developed a machine-vision workflow to collect and index the images and metadata produced during an experiment into a data-enriched, searchable timeline.Computational and image analysis algorithms can calculate variables between images, apply corrections, and identify hidden trends. By calculating, synchronizing, and indexing new metadata into each image, multimodal analysis is suddenly a new possibility. For example, accurate accumulated dose and dose rate information in the zeolite images allows quantitative assessment of sample damage across thousands of images.We will continue to push the capabilities of in-situ TEM by making results accessible and interpretable and improving experimental reproducibility. This software is a platform to develop workflow-driven modules that target specific, historically difficult applications to make in-situ TEM experiments easier and more information rich.

Watch this Scientific Journal Video about Determination of Dose Threshold Using the Machine-Vision Synchronization MVS and Dose Software at JoVE.com

Determination of Dose Threshold Using the Machine-Vision Synchronization (MVS) and Dose Software  

使用机器视觉同步 （MVS） 和剂量软件确定剂量阈值

首先打开 TEM 仪器上的 MVS 软件应用程序并选择其他。将显微镜连接到MVS软件。导航到图像元数据选项卡，然后选择放大倍率、最大剂量和剂量率以叠加在实时显示中看到的图像流上。如果用户愿意，可以包含其他元数据。打开剂量选项卡。选择剂量管理并启用剂量监测。然后选择显示剂量层并显示位置不确定性以激活自动电子剂量跟踪并显示剂量颜色叠加。在按保存之前设置低剂量水平和高剂量水平的值。然后导航到设置选项卡。选择剂量并设置剂量导航图不透明度和剂量图像叠加不透明度值。通过单击漂移校正在实时图像查看器窗口中激活漂移校正。接下来，导航到数据视图选项卡，并在 y 轴上绘制元数据值散焦和焦点商。激活对焦辅助，然后选择校准对焦以运行自动对焦辅助的校准。校准焦点例程完成后，关闭数据视图选项卡。在MVS软件中，打开图像分析选项卡并激活实时FFT和象限一和象限二选项。使用显微镜软件控制调整光束条件，使电子通量约为每秒600埃平方。移动到样品中的新区域，并在 MVS 软件的实时视图中居中显示样品 ROI。保持恒定的剂量率，同时连续成像相同的ROI，直到对应于原子结构和FFT的斑点消失。

determination-dose-threshold-using-machine-vision-synchronization-mvs

Research

JoVE Journal

Biology

Determination of Dose Threshold Using the Machine-Vision Synchronization (MVS) and Dose Software

203 次观看.  Protochips, Inc.. 首先打开 TEM 仪器上的 MVS 软件应用程序并选择其他。将显微镜连接到MVS软件。导航到图像元数据选项卡，然后选择放大倍率、最大剂量和剂量率以叠加在实时显示中看到的图像流上。如果用户愿意，可以包含其他元数据。打开剂量选项卡。选择剂量管理并启用剂量监测。然后选择显示剂量层并显示位置不确定性以激活自动电子剂量跟踪并显示剂量颜色叠加。在按?...

视频: Determination of Dose Threshold Using the Machine-Vision Synchronization (MVS) and Dose Software  

A Machine-Vision Approach to Transmission Electron Microscopy Workflows, Results Analysis and Data Management

Transmission electron microscopy (TEM) enables users to study materials at their fundamental, atomic scale. Complex experiments routinely generate thousands of images with numerous parameters that require time-consuming and complicated analysis. AXON synchronicity is a machine-vision synchronization (MVS) software solution designed to address the pain points inherent to TEM studies. Once installed on the microscope, it enables the continuous synchronization of images and metadata generated by the microscope, detector, and in situ systems during an experiment. This connectivity enables the application of machine-vision algorithms that apply a combination of spatial, beam, and digital corrections to center and track a region of interest within the field of view and provide immediate image stabilization. In addition to the substantial improvement in resolution afforded by such stabilization, metadata synchronization enables the application of computational and image analysis algorithms that calculate&#8239;variables between images. This calculated metadata can be used to analyze trends or identify key areas of interest within a dataset, leading to new insights and the development of more sophisticated machine-vision capabilities in the future. One such module that builds on this calculated metadata is dose calibration and management. The dose module provides state-of-the-art calibration, tracking, and management of both the electron fluence (e-/&#197;2&#183;s-1) and cumulative dose (e-/&#197;2) that is delivered to specific areas of the sample on a pixel-by-pixel basis. This enables a comprehensive overview of the interaction between the electron beam and the sample. Experiment analysis is streamlined through a dedicated analysis software in which datasets consisting of images and corresponding metadata are easily visualized, sorted, filtered, and exported. Combined, these tools facilitate efficient collaborations and experimental analysis, encourage data mining and enhance the microscopy experience.

Here, we present a protocol to utilize machine-vision software to stabilize dynamic processes during TEM imaging, while concurrently indexing multiple streams of metadata to each image into a navigable timeline. We demonstrate how this platform enables automated calibration and mapping of the electron dose over the course of an experiment.

Transmission electron microscopy (TEM) enables users to study materials at their fundamental, atomic scale. Complex experiments routinely generate ...

科研

生物学

Dose Calibration of the Transmission Electron Microscope for TEM and Scanning TEM (STEM) Imaging Modes Using a Machine-Vision Approach

Begin the dose calibration by clicking the Connect icon in the MVS software installed on TAEM. Select the microscope and activate the connection between the TEM and the MVS software to visualize the images from the camera or detector in the image viewer. Navigate to the Dose tab and click on Start Dose Area Calibration.Click OK to confirm the calibration hardware and follow the software prompts to enter the requested user configurable values. Then click Start Dose Current Calibration. Follow the software prompts and enter the requested user configurable values to complete the calibration.

使用机器视觉方法对透射电子显微镜进行透射电子显微镜的剂量校准，用于TEM和扫描TEM（STEM）成像模式

Begin by opening the MVS software application on the TEM instrument and selecting other. Connect the microscope to the MVS software. Navigate to the image metadata tab and select magnification, max dose and dose rate to overlay on the image stream seen in the live display.Other metadata may be included if the user desires. Open the dose tab. Select dose management and enable dose monitoring.Then select show dose layer and show position uncertainty to activate automated electron dose tracking and display the dose color overlay. Set the values for the low dose level and high dose level before pressing save. Then navigate to the settings tab.Select dose and set the dose navigation map opacity and dose image overlay opacity values. Activate drift correction in the live image viewer window by clicking on drift correct. Next, navigate to the data view tab and plot the metadata values defocus and focus quotient on the y axis.Activate focus assist then select calibrate focus to run the calibration of the automated focus assist. Once the calibrate focus routine is complete close the data view tab. In the MVS software, open the image analysis tab and activate the live FFT and quadrants one and two options.Using the microscope software controls adjust the beam conditions so that the electron flux is approximately 600 angstrom square per second. Move to a new region in the sample and center the sample ROI in the live view of the MVS software. Maintain a constant dose rate while continuously imaging the same ROI until the spots corresponding to the atomic structure and the FFT have disappeared.

Metadata and Trend Analysis and Data Export Using the Analysis Software

Launch the offline analysis software to view the fully synchronized datasets and open the experiment session file by selecting it from the library. Display the drift corrected images by activating the DC tab below the image viewport before selecting the desired data overlays by checking their respective overlay data boxes in the image metadata tab. Other metadata may be plotted as the user desires.Highlight or scroll through these graphical plots to update the image displayed in the viewport. Access various tools through the image analysis toolbox and data view tabs. Access the FFT for each image through the image analysis to plot live FFT to update it while scrolling through images.Use the fading of the FFT peaks to determine the time point at which the zeolite nanoparticle structure loses crystallinity. Note the dose conditions in the software using the tag function. Highlight the tag icon and enter the desired tags to denote a specific series of images within the timeline.Images will be tagged with this text until the tag icon is deselected. To easily filter large data datasets into smaller shareable ones without losing their associated metadata, open the filter panel and adjust the sliders to select only a dose rate equal to or above 500 square angstroms per second. Save the new collection using the name 500 dose rate and above.Select the images by highlighting them in the timeline or using the filter options. Then export the metadata as a CSV file and the image series as a movie file using the same publish option. The zeolite nanoparticle ZSM5 imaged to determine the threshold dose showed that the dose rate read by the MVS software under high dose rate conditions was 519 electrons per square angstrom per second.Nanoparticles in the field of view were imaged continuously until the FFT peaks disappeared, indicating crystalline structure degradation. The FFT peaks began disappearing after 42 seconds of continuous imaging. The FFT peaks had completely disappeared at one minute and 20 seconds and a cumulative dose of approximately 60, 000 electrons per square angstrom.The same process was repeated with a different microscope. The FFT spots fully disappeared after one minute and 50 seconds of continuous imaging and a cumulative dose of 58, 230 electrons per square angstrom.

使用分析软件进行元数据和趋势分析以及数据导出

In Situ Heating Study of Gold on Iron Oxide Nanoparticles

To begin, select the appropriate workflow option from the MVS software. Follow the workflow prompts to confirm the electrical connection between the holder and the heating e-chip by loading the calibration file and performing a device check. After connecting the microscope to the MVS software, center the sample ROI in the field of view.Access the Temperature Control Settings by clicking the Experiment button and Manual Control mode. Then, set the ramp rate to 25 degrees Celsius per second and the target to 200 degrees Celsius. Click on Apply to start the experiment.After reaching 200 degrees Celsius, set the ramp rate to 10 degrees Celsius per second. Adjust the target to 600 degrees Celsius and click Apply. After the set temperature of 600 degrees Celsius is reached, change the ramp rate to 2 degrees Celsius and the target to 800 degrees Celsius.Click on Apply to start the experiment. Once done, open the Analysis Software to review the session. In the timeline, plot the temperature, template morphing factor, dose rate, and cumulative dose.Export images and movies using the Publish option with or without the dose map overlays. The heating experiment performed using a representative nanocatalyst sample gold on iron oxide showed that at elevated temperatures, the gold nanoparticles within the gold on iron oxide migrated along the surface of the iron oxide support and centered to form larger particles. An in-situ heating experiment recorded a series of TEM snapshots of a porous region within a gold on iron oxide nanocatalyst at various time points.The coordinated drift of the sample increased with the increased temperature from a rate of 9 to 62 nanometers per minute, and began to decrease towards leveling off with the constant temperature. The MVS software stabilized the particle in the field of view throughout the entire temperature ramp profile, enabling high resolution imaging.