We would like to thank the Vodafone Americas Foundation, the Bill and Melinda Gates Foundation, and the S&#227;o Paulo Research Foundation (FAPESP) for funding this research. We would also like to thank the many faculty members from the Department of Entomology at University of California, Riverside, for their advice on this project.&#160;

1. Insect Colony and Rearing
<ol>
	<li>Mosquito Colony and Rearing
	<ol>
		<li>Rear Culex tarsalis, Culex quinquefasciatus, Culex stigmatosoma, and Aedes aegypti adults from lab colonies, which originated from wild caught individuals.</li>
		<li>Rear mosquito larvae in enamel pans under standard laboratory conditions (27 &#176;C, 16:8 hr light:dark [LD] cycle with 1 hr dusk/dawn periods), and feed them ad libitum on a mixture of ground rodent chow and Brewer&#8217;s yeast (3:1, v:v).</li>
		<...

<ol>
	<li>Kahn, M. C., Celestin, W., Offenhauser, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Recording+of+sounds+produced+by+certain+disease-carrying+mosquitoes.">Recording of sounds produced by certain disease-carrying mosquitoes.</a> Science. 101 (2622), 335-336 (1945).</li><li>Reed, S. C., Williams, C. M., Chadwick, L. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Frequency+of+wing-beat+as+a+character+for+separating+species+races+and+geographic+varieties+of+Drosophila.">Frequency of wing-beat as a character for separating species races and geographic varieties of Drosophila.</a> Genetics. 27 (3), 349 (1942).</li><li>Belton, P., Costello, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Flight+sounds+of+the+females+of+some+mosquitoes+of+Western+Canada.">Flight sounds of the females of some mosquitoes of Western Canada.</a> Entomologia experimentalis et applicata. 26 (1), 105-114 (1979).</li><li>Mankin, R. W., Machan, R., Field Jones, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=testing+of+a+prototype+acoustic+device+for+detection+of+Mediterranean+fruit+flies+flying+into+a+trap.">testing of a prototype acoustic device for detection of Mediterranean fruit flies flying into a trap.</a> Proc. 7th Int Symp Fruit Flies of Economic Importance. , 10-15 (2006).</li><li>Raman, D. R., Gerhardt, R. R., Wilkerson, J. B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Detecting+insect+flight+sounds+in+the+field+Implications+for+acoustical+counting+of+mosquitoes.">Detecting insect flight sounds in the field Implications for acoustical counting of mosquitoes.</a> Transactions of the ASABE. 50 (4), 1481 .</li><li>Moore, A., Miller, J. R., Tabashnik, B. E., Gage, S. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+identification+of+flying+insects+by+analysis+of+wingbeat+frequencies.">Automated identification of flying insects by analysis of wingbeat frequencies.</a> Journal of economic entomology. 79 (6), 1703-1706 (1986).</li><li>Moore, A., Miller, R. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+identification+of+optically+sensed+aphid+(Homoptera:+Aphidae)+wingbeat+waveforms.">Automated identification of optically sensed aphid (Homoptera: Aphidae) wingbeat waveforms.</a> Annals of the Entomological Society of America. 95 (1), 1-8 (2002).</li><li>Li, Z., Zhou, Z., Shen, Z., Yao, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+identification+of+mosquito+(diptera:+Culicidae)+wingbeat+waveform+by+artificial+neural+network.">Automated identification of mosquito (diptera: Culicidae) wingbeat waveform by artificial neural network.</a> Artificial Intelligence Applications and Innovations. , 483-489 (2005).</li><li>Moore, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Artificial+neural+network+trained+to+identify+mosquitoes+in+flight.">Artificial neural network trained to identify mosquitoes in flight.</a> Journal of insect behavior. 4 (3), 391-396 (1991).</li><li>Halevy, A., Norvig, P., Pereira, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+unreasonable+effectiveness+of+data.">The unreasonable effectiveness of data.</a> Intelligent Systems, IEEE. 24 (2), 8-12 (2009).</li><li>Banko, M., Brill, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Mitigating+the+paucity-of-data+problem+exploring+the+effect+of+training+corpus+size+on+classifier+performance+for+natural+language+processing.">Mitigating the paucity-of-data problem exploring the effect of training corpus size on classifier performance for natural language processing.</a> , 1-5 (2001).</li><li>Shotton, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-time+human+pose+recognition+in+parts+from+single+depth+images.">Real-time human pose recognition in parts from single depth images.</a> Communications of the ACM. 56 (1), 116-124 (2013).</li><li>Ilyas, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Classifying+insects+on+the+fly.">Classifying insects on the fly.</a> Ecological Informatics. , (2013).</li><li>Mack, Y. P., Rosenblatt, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multivariate+k-nearest+neighbor+density+estimates.">Multivariate k-nearest neighbor density estimates.</a> Journal of Multivariate Analysis. 9 (1), 1-15 (1979).</li><li>Benedict, M., Robinson, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+first+releases+of+transgenic+mosquitoes+an+argument+for+the+sterile+insect+technique.">The first releases of transgenic mosquitoes an argument for the sterile insect technique.</a> TRENDS in Parasitology. 19 (8), 349-355 (2003).</li><li>. Supporting Materials Available from: <a target="_blank" href="https://sites.google.com/site/insectclassification">https://sites.google.com/site/insectclassification</a></li><li>Chen, Y., Why, A., Batista, G., Mafra-Neto, A., Keogh, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Flying+Insect+Classification+with+Inexpensive+Sensors.">Flying Insect Classification with Inexpensive Sensors.</a> Journal of Insect. 27 (5), 657-677 (2014).</li></ol>

The sensor/classification framework described here allows the inexpensive and scalable classification of flying insects. The accuracies achievable by the system are good enough to allow the development of commercial products and could be a useful tool in entomological research.
The ability to use inexpensive, noninvasive sensors to accurately and automatically classify flying insects would have significant implications for entomological research. For example, by deploying the system in the fie...

The idea of automatically classifying insects using the incidental sound of their flight dates back to the earliest days of computers and commercially available audio recording equipment1. However, little progress has been made on this problem in the intervening decades. The lack of progress in this pursuit can be attributed to several related factors.
First, the lack of effective sensors has made data collection difficult. Most efforts to collect data have used acoustic microphones2-5. Such devices are extremely sensitive to wind noise and ambient noise in the environment, resulting in very sparse and low-quality data.
Second, compounding these data quality issues is the fact that many researchers have attempted to learn very complicated classification models, especially neural networks6-8. Attempting to learn complicated classification models, with a mere tens of examples, is a recipe for over-fitting.
Third, the difficultly of obtaining data has meant that many researchers have attempted to build classification models with very limited data, as few as 300 instances9 or less. However, it is known that for building accurate classification models, more data is better10-13.
This work addresses all three issues. Optical (rather than acoustic) sensors can be used to record the &#8220;sound&#8221; of insect flight from meters away, with complete invariance to wind noise and ambient sounds. These sensors have allowed the recording of millions of labeled training instances, far more data than all previous efforts combined, and thus help avoid the over-fitting that has plagued previous research efforts. A principled method is shown below that allows the incorporation of additional information into the classification model. This additional information can be as quotidian and as easy-to-obtain as the time-of-day, yet still produce significant gains in accuracy of the model. Finally, it is demonstrated that the enormous amounts of data we collected allow us to take advantage of &#8220;The unreasonable effectiveness of data&#8221;10 to produce simple, accurate and robust classifiers.
In summary, flying insect classification has moved beyond the dubious claims created in the research lab and is now ready for real-world deployment. The sensors and software presented in this work will provide researchers worldwide robust tools to accelerate their research.

<table border="0" cellpadding="0" cellspacing="0" width="1293">
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:328px;">Name of Material/ Equipment</td>
			<td style="width:237px;">Company</td>
			<td style="width:257px;">Catalog Number</td>
			<td style="width:471px;">Comments/Description</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:328px;">Audio Recorder: ICD-PX312</td>
			<td style="width:237px;">Sony</td>
			<td style="width:257px;">4-267-065-11(2)</td>
			<td>With a 8 GB microSD extra memory</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:328px;">Insectary</td>
			<td style="width:237px;">Lee&#39;s Aquarium &#38; Pet Products.</td>
			<td style="width:257px;">20088 HerpHaven&#174;, Large Rectangle</td>
			<td style="width:471px;">14 1/2&#34; Long x 8 3/4&#34; Wide x 9 3/4&#34; high. Modified to house insects.&#160;</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:328px;">Laser Line Generator, 650nm (red)</td>
			<td style="width:237px;">Apinex (www.apinex.com)</td>
			<td style="width:257px;">LN60-650</td>
			<td style="width:471px;">5mW. This is a low powered laser, similar to a teachers lasers pointer</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;width:328px;">Photodiode Array</td>
			<td style="width:237px;">VISHAY SEMICONDUCTOR&#160; TEFD4300&#160; PIN PHOTODIODE, 650NM, 20DEG, T1</td>
			<td style="width:257px;">TEFD4300</td>
			<td style="width:471px;">We made a custom array of 15 of these Photodiodes wired in parallel</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;">Analogue to Digital Convertor Integrated Circuit&#160;</td>
			<td style="width:237px;">Custom made in our lab</td>
			<td style="width:257px;"></td>
			<td style="width:471px;">We made this item ourselves, but an easily available commercial product, Gino PCF8591 AD/DA Converter, provides the same functionality.&#160;</td>
		</tr>
	</tbody>
</table>

flying insect detection and classification with inexpensive sensors

An inexpensive, noninvasive system that could accurately classify flying insects would have important implications for entomological research, and allow for the development of many useful applications in vector and pest control for both medical and agricultural entomology. Given this, the last sixty years have seen many research efforts devoted to this task. To date, however, none of this research has had a lasting impact. In this work, we show that pseudo-acoustic optical sensors can produce superior data; that additional features, both intrinsic and extrinsic to the insect&#8217;s flight behavior, can be exploited to improve insect classification; that a Bayesian classification approach allows to efficiently learn classification models that are very robust to over-fitting, and a general classification framework allows to easily incorporate arbitrary number of features. We demonstrate the findings with large-scale experiments that dwarf all previous works combined, as measured by the number of insects and the number of species considered.

Two experiments are presented here. For both experiments, the data used were randomly sampled from a dataset that contains over 100,000 objects.
The first experiment shows the ability of the proposed classifier to accurately classify different species/sexes of insects. As the classification accuracy depends on the insects to be classified, a single absolute value for classification accuracy will not give the reader a good intuition about the performance of the system. Instead, rather than repo...

Watch this Scientific Journal Video about Flying Insect Detection and Classification with Inexpensive Sensors at JoVE.com

Flying Insect Detection and Classification with Inexpensive Sensors

We proposed a system that uses inexpensive, noninvasive pseudo-acoustic optical sensors to automatically and accurately detect, count, and classify flying insects based on their flying sound.

An inexpensive, noninvasive system that could accurately classify flying insects would have important implications for entomological research, and allow ...

flying-insect-detection-and-classification-with-inexpensive-sensors

Research

JoVE Journal

Bioengineering

25.0K Views.  University of California, Riverside. We proposed a system that uses inexpensive, noninvasive pseudo-acoustic optical sensors to automatically and accurately detect, count, and classify flying insects based on their flying sound.