IROS2019国际学术会议论文集 0506

Systematic benchmarking for reproducibility of computer vision algorithms for real time systems The example of optic fl ow estimation Bj rnborg Nguyen1 Christian Berger2 and Ola Benderius1 Abstract Until now there have been few formalized methods for conducting systematic benchmarking aiming at reproducible results when it comes to computer vision algorithms This is evident from lists of algorithms submitted to prominent datasets authors of a novel method in many cases primarily state the performance of their algorithms in relation to a shallow description of the hardware system where it was evaluated There are signifi cant problems linked to this non systematic approach of reporting performance especially when comparing different approaches and when it comes to the reproducibility of claimed results Furthermore how to conduct retrospective performance analysis such as an algorithm s suitability for embedded real time systems over time with underlying hardware and software changes in place This paper proposes and demonstrates a systematic way of addressing such challenges by adopting containerization of software aiming at formalization and reproducibility of benchmarks Our results show maintainers of broadly accepted datasets in the computer vision community to strive for systematic comparison and re producibility of submissions to increase the value and adoption of computer vision algorithms in the future I INTRODUCTION Optic fl ow estimation is a core topic of perception and often part of larger algorithms as found in robotics or autonomous vehicles Functionality like visual odometry and motion video compression and visual simultaneous localiza tion and mapping are typical examples using optic fl ow esti mation The needs of a formal evaluation and benchmarking of image processing algorithms in embedded real time set tings have recently been greatly increased resulting from the development of advanced robotic systems like autonomous vehicles However a broad yet systematic evaluation of the suitability of such functions for the aforementioned use cases is still missing A Background There has been an extensive development in optic fl ow es timation algorithms since the works of Lucas and Kanade 1 and Horn and Schunck 2 This has spurred the growth of such algorithms and there are many solutions openly published using open optic fl ow databases like the KITTI dataset 3 the Middlebury dataset 4 or the synthetic Sintel dataset 5 Further improvements have been implemented showing increasingly better results in terms of accuracy consistency and robustness through competitive evaluation 1 Applied artifi cial intelligence Department of Mechanics and Mar itime Sciences Chalmers University of Technology Gothenburg Sweden bjornborg nguyen ola benderius chalmers se 2 CyberPhysicalSystems DepartmentofComputerScience andEngineering UniversityofGothenburg Gothenburg Sweden christian berger gu se benchmarks adopting de facto standards based on the work of Barron et al 6 B Problem domain and motivation To make optic fl ow estimation algorithms applicable in real time critical applications there are several hard con straints imposed on the estimator such as accuracy robust ness and execution time Much of the focus of the early works have been on benchmarking just on the fi rst of which Since there is a trade off between computational cost and measurement performance many algorithms are optimized for the latter This has led to a general trend for algorithms depreciating computational costs and run time aspects of such algorithms resulting in being unfi t for real time critical applications This can be seen for example in the entries of KITTI optical fl ow evaluation 2015 dataset 3 where benchmarks of optic fl ow accuracy along with their run time vary from few milliseconds to several hours to compute a single frame As such varying results show more research in run time and its predictability is necessary to provide better guidance for real time critical applications based on optic fl ow In this paper we have identifi ed two major challenges i The lack of standardized measurements and defi nition of run time and run time predictability in terms of the real time context and the ii deployability of already proposed optic fl ow estimators to achieve reproducible results C Research goal and research questions The goal of our work is to propose and evaluate an empirical run time metric to systematically evaluate real time critical computer vision algorithms on the example of several optic fl ow approaches We are specifi cally focusing on the following research questions RQ 1 How can we quantify a run time property of an optic fl ow estimator with respect to the context of real time computing while aiming at a fair comparison in benchmarking RQ 2 How can reproducibility of validation and evaluation for optic fl ow algorithms achieved without compromis ing the fl ow accuracy utility and run time performance D Contributions We are providing i an empirical approach for quantifying run time and predictability of several optic fl ow estimators and ii an open source implementation to enable repro ducible evaluation of all these algorithms To achieve repro ducibility we are using so called software containers based 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE5264 on Docker to package the optic fl ow estimation algorithm of interest next to all its necessary dependencies Such containers can be easily distributed and executed to validate the results that we are presenting here E Structure of the article The remaining of the paper is structured as followed Sec II discusses related work and Sec III outlines our approach for run time and predictability metrics next to our container based approach for enabling reproducibility We are presenting our experimental results in Sec IV and discuss them in Sec V Sec VI concludes the paper and provides directions for future work II RELATED WORK A Optic fl ow estimators Both the Middlebury optic fl ow dataset and Baker et al made a rough run time guideline performance benchmark of a few optic fl ow estimators in their paper 4 The authors made no attempts in normalizing reported run time with respect to the programming environment CPU capabilities or other hardware acceleration furthermore they state that those run time results should only be treated as a rough guideline how different optic fl ow estimators perform In their measurement like the KITTI optical fl ow evaluation 2015 data performance benchmarks the run time is mea sured as the mean value at best of total run time running over one particular chosen data set Middlebury Sintel and KITTI optic fl ow databases allow researchers to submit and publish their accuracy performance benchmarks in their respective scoreboards The maintainers of scoreboard conduct rigorous checking and evaluation of each submitted approach in order to confi rm claimed results and furthermore also publishes run time However how this is exactly conducted is not very clear as there is no standard to this What is likely measured is at best the averaged value of the total run time executed on the same data set It is not clear if Middlebury scoreboard uses the same experimental setup for different run time performance benchmark while KITTI scoreboard briefl y shares a few words what relevant hardware and software environment was used Recent research has also raised concerns about the trend of focusing on accuracy over run time Baker et al pointed out that optic fl ow estimation algorithms are computationally complicated and complex making it hard to generate a fair run time performance benchmark Additionally it is equally hard to make them truly real time with given conventional defi nitions and guidelines of real time system 4 However in order to have comparable run time performance bench marks for the optic fl ow community fair assumptions need to be made and standardized quantities defi ned Some of the newer algorithms are designed having this aspect in mind and are pointing the trend of depreciating computational costs For example Plyer et al developed the e FOLKI algorithm with low run time while preserving a suffi cient quality of the estimated optic fl ow for real world applications 7 They compared their results with the popular and competitive optic fl ow estimators TVL1 and Pyramid Lucas Kanade available in OpenCV and argue further that they have the best quality cost compromise at the time of publishing Also Kroeger et al also attempted to come with a good quality cost compromise with their Dense inverse search algorithm 8 They both successfully showed that it is possible to trade a small accuracy error for a signifi cant gain in run time Another challenge for the current state of the art research in optic fl ow estimation is that there are many proposed algorithms with promising results that are diffi cult to exactly reproduce as stated in the original papers Reasons therefor include getting access to the source code is sometimes diffi cult dependencies to specifi c third party libraries are not explicit and the use of specifi c parameterization while running the original experiments B Real time computing and scheduling Shin and Ramanathan survey and summarize the fi eld of real time computing concerning computer science and engineering well in their work 9 In a real time system tasks need to be assigned and scheduled in such a way to comply with a clearly defi ned deadline while guaranteeing its correctness and reliability due to its nature of being a critical system 10 Deadlines are often referred to as hard fi rm or soft depending on the severity of its consequence in case of when not met For example hard deadline real time systems may be found in the safety critical application in vehicles such as the anti lock braking system or airbag activation system where consequences can be fatal Thus controlling a system or application with real time aspects must include computing predictability meaning that under given system assumptions the job is guaranteed to be completed within a given time constraint However in large complex software systems it is evi dently harder to a priori guarantee the predictability require ment 9 10 As opposed to a real time operating system found in specialized embedded computers and assigned spe cifi c tasks which in fact guarantees the correctness reliabil ity and predictability general purpose operating systems are primarily designed to enhance other capabilities as broader support of hardware shared computing resources and better user experience Despite this general purpose operating sys tems are often used for development and deployment due to their high availability and multi purpose utility capabilities Thus it is hard to put a real time system analysis in these stricter defi nitions While GPUs have made a rapid impact in computer vision and high performance computing due to their superiority for memory management and alteration it was never intended for real time computing However Elliot and Anderson in vestigated how GPUs may be applied in such application and concluded that such technology is in fact mature for soft deadline real time systems with specialized schedulers or effi cient locking protocols 11 However the technology still imposes great challenges to real time analysis due to its 5265 complex resource management and further research is needed for hard deadline analysis C Containerization for software packaging and distribution for real time critical applications Virtualizing computational resources is an established con cept in computer science that is widely spread in practical applications Recently containerization of applications i e the bundling of software next to all its library dependencies that only share strictly separated computing resources has emerged as a lightweight alternative to complete system em ulation For example the Docker environment has been eval uated in different industrial contexts like automation 12 automotive 13 and high performance computing 14 Morabito et al investigated the performance impact of virtualization techniques in comparison to natively deployed software 15 The evaluation of the performance was con ducted extensively on various aspects CPU performance data storage I O memory performance and network I O per formance benchmarks of hypervisor based and lightweight container based virtualization to a native Linux operating system They concluded that hypervisor based virtualiza tion has made drastic improvements in recent years but still experiences challenges in data storage I O However the container based technology shows promising results outperforming its counterpart in all evaluated benchmarks supported in various investigations 15 16 While the containerized approach introduces some degree of overhead primarily on the network stack it is negligible according to Morabito et al and Masek et al in terms of performance 15 17 In the case of soft deadline real time systems the work of Goldschmidt et al demonstrated such a requirement can be met and executed in such an environment 12 III METHODOLOGY A Standardized metrics for run time performance and run time predictability for optic fl ow estimators By imposing the optic fl ow estimation as a soft real time system we propose a statistical approach of measuring the run time with a probabilistic run time guarantee By empir ically observing and sampling the run time for the task one would get a distribution that is ideally a Dirac delta function at the mean value however in practice this would not be the case as measurement noise and random computing interrupts introduced from various sources Gaussian distributions may appear central limit theorem for random noise in more simply structured and deterministic algorithms Thus more generally for complex optic fl ow estimators which may have several types of stopping conditions and different rates of convergence to a solution more arbitrary distributions may form Due to the emergent properties of complex computing algorithms and highly dynamic input outliers in measured execution time may appear This severely impacts the pre dictability and reliability of a real time system As of today there is no rigid mathematical defi nition of an outlier ob servation There are several techniques based on modeling and statistical observation for detecting and identifying such abnormalities We propose to use percentile quantity mea surement as a combined quantity capturing both the run time variability for outliers and the absolute mean run time value This may be further extended to soft real time system purposes with respect to the predictability The percentile measure deviates from the mean or median value as the standard deviation increases in large sample size which is not captured in the current evaluation criteria of optic fl ow estimator databases It can be argued that one should measure the longest statistical observed run time but doing so would encourage and favor a smaller sample size and also be highly dependent on randomness due to the principle of the random walk B Containerization of software for reproducible results We adopt containerization of software to aim at re producible results due to its negligible impact on high performance computing being capable of real time comput ing and the simplifi ed software distribution This method mitigates the challenges of setting up necessary software dependencies and resource management while enabling sys tematic result validation and performance benchmarking As it is common to also use hardware acceleration with graphics processing units GPU in computer vision some GPU man ufacturers support containerization of software by providing capable drivers for such virtualization even offering scalable link interface SLI functionality By utilizing a real time capable operating system with support for GPU hardware acceleration software containers can be evaluated under real time aspects Thus the real time capabilities of different optic fl ow estimators can be analyzed statistically and evaluated with metrics of run time and predictability introduced previously The various optic fl ow estimators mentioned here are based on open computer vision libraries in combination with publicly available material for constructing container images All evaluation is conducted using containerized software to enable reproducible results For measuring the performance the Linux kernel performance profi ler perf was used for performance analysis as it introduces negligible overhead during measurement as reported by Vitillo 18 C Our experimental setup and software environment Inourdeploymentandbenchmarking acomputer DATALynx ATX3 equipped with dual Intel R Xeon R CPU E5 2640 v4 at 2 4GHz was used The computer was installed with eight RDIMM DDR4 2400 reg ECC 8GB a solid state drive and two NVIDIA GTX 1080 GPUs only one was used during benchmarking The computer was running the Linux kernel 4 19 1 including the rt preempt real time patches an Ubuntu 18 04 system with GPU support NVIDIA drivers 410 and the Docker 18 09 engine for containerization of software IV RESULTS Eight common optic fl ow estimators have been im plementedwiththeirdefaultorsuggestedparameters 5266 05101520 Time s Simplefl ow PCAfl ow Lucas Kanade FlowNet2 Farneback DualTVL1 Dense Inverse Search Deepfl ow Run time distributions Fig 1 The absolute execution times of 3 192 run time samples from each of the eight implemented optic fl ow estimators evaluated once on the complete Sintel database The box covers two quartiles centered at the median value while the whiskers stretch to the 99thpercentile outliers are shown as circles 0 00 20 40 60 81 0 Simplefl ow PCAfl ow Lucas Kanade FlowNet2 Farneback DualTVL1 Dense Inverse Search Deepfl ow Normalized run time distributions Fi