A Need for Speed? More Signal Processing Innovation is Needed Before Adopting Automated Vehicle Technology
We spend a considerable amount of time driving—to work, to home, for recreation, for travel. This love for the automobile is on the precipice of becoming a world-wide phenomenon. While new markets, like China, have opened up in the past decade, the need for the automotive industry to find new customers, combined with an array of emerging technologies that will make driving easier, will allow cars to reach markets that never before had access to driving. Likewise, the promise of new capabilities that automate and enhance the safety of the driving experience will guarantee that existing drivers return to purchase the latest model.
People will be able to go to a terminal, and rent a vehicle pre-programmed to take them to a specified destination. Drivers will be able to disengage from actual driving to read the newspaper while the car carries out their daily commute to and from work—all the while, cars will seamlessly coordinate to ensure the safety of their passengers.
And why will this happen? Because people like the freedom and excitement that comes with “getting in the car and going places!”
This promise is very enticing, but society shouldn’t rush headlong into adopting automated vehicular systems. Transportation systems are very complex systems with many interacting pieces—anyone who has driven through rush hour in any of the world’s urban centers will attest to this!
Recently, there have been several stories in the news that have emphasized the challenges facing the automation of vehicular systems. Two very tragic crashes that occurred in TESLA vehicles (one in China and one in the US), highlight the serious life-and-death consequences associated with malfunctions or miscalculations that can occur with vehicular systems. Meanwhile, other news stories have given us insight into what can go wrong should these systems and their data come in the cross-hairs of cybercriminals.
What can be done about this? Perhaps first and foremost is to slow our rush to “remove the human” from the equation. Officially, TESLA’s Autopilot was meant to assist the driver, not replace the driver. Humans are still essential to driving and, quite frankly, it should be that way for some time until the technology matures. Let’s be clear, though: this does not mean that we should slow down innovation, but rather that we should work harder to provide even more innovation that can make its way into vehicles!
It is towards this vision where we, as signal processors and data analysts, can have an impact. Our algorithms and our technical areas are precisely the tools that are needed to advance vehicular systems. We have been conducting research in a broad range of fields, such as radar, computer vision, and statistical signal processing that can allow us to revolutionize the field, to develop the necessary innovations that will make automotive systems reliable to the 10th decimal place. Let us take a look at this problem, and see how signal processing and data analytics can address many of the challenges surrounding automated vehicles.
A modern car currently contains roughly 100 sensors, and looking forward it is likely that future automobiles will be deployed with significantly more sensors. The wide array of sensors include accelerometers (used in impact detection and motion measurements), pressure sensors (e.g. used in air intake control, monitoring fuel consumption, tire conditions), temperature sensors (e.g. used in monitoring and controlling engine conditions, fuel temperature, passenger compartment temperature), phase sensors (e.g. camshaft/crankshaft phase sensors for motor control, and gear shaft speed for transmission control). Angular rate sensors monitor the roll, pitch, and yaw of a vehicle, which informs dynamic control systems, automatic distance control, and navigation systems. Angular and position sensors monitor the position of gear levers, steering wheel angle, mirror positioning, etc. RADAR, LIDAR and camera sensors are being used to facilitate new applications, such as blind spot monitoring, lane departure warning, and automated driving.
This wealth of data, although dauntingly voluminous, when properly utilized is also the avenue to safety and robustness! These sensors provide the abundance of data that can serve as corroborating evidence to fix malfunctions, to back-solve and determine that a mistake is about to be made from using a single sensor type alone, to correct false data injected by those trying to hack our vehicles.
TESLA’s Autopilot was a very advanced camera-based system meant to identify conditions in the road scenario, and make suitable corrections based on what it inferred. Looking at the videos from the crashes, it is evident that the videos witnessed by Autopilot were not under ideal lighting conditions, background objects blended into vehicles that needed to be recognized. These scenarios would have been hard for any computer vision algorithm to process correctly, but this challenge was doubly amplified by the small amount of time allowed to “lock on” given the speed of the vehicle and the imminent crash. Multiple sensor types, used in conjunction, could have helped. Radar or LIDAR would not have been susceptible to the same difficulties that the camera-based system likely encountered. One of these other technologies could have helped in the decision making.
Since that time, though, TESLA has re-evaluated its strategy for Autopilot, which has included the possibility to use radar in place of the camera. While it is difficult to speak about the new radar system, two things are clear: the choice of a radar system is meant to avoid the environmental hurdles that arise with visual-based systems, and TESLA has collected a large amount of radar data that serves as the basis of their new Autopilot system.
A switch to a single type of sensor, though likely an improvement, is not likely to solve all of the problems that will arise in automating vehicles. In fact, while radar can cope with lighting-based challenges, there are numerous studies that suggest that LIDAR systems are superior in terms of tracking accuracy. On the other hand, LIDAR systems suffer degradation in conditions with fog, and cameras offer the ability to recognize finer details associated with objects (such as license plate information). In fact, cameras support accurate assessment of the visibility distance (notably, fog), which could be used to inform the driver that vehicular assistance services are not available or perhaps experiencing degraded quality of service because fog is affecting the visible detection of road lanes and other vehicles. Data fusion and extracting hidden correlation between sensor types is at the heart of modern signal processing. Merging radar, LIDAR and visual systems into fast and robust object recognition and tracking algorithms is an exciting opportunity for signal processors to contribute.
But the opportunities for signal and data processing does not stop there! When considering future vehicular applications, we should recognize that other sensor types will be available and can provide valuable knowledge, like weather conditions, road friction coefficients or road slopes. Road slope information would be useful for coordinating braking amongst several vehicles since a road’s slope is related to the potential for a vehicle to accelerate or decelerate. The sharing of data between vehicles, and with cloud-based computing services opens up many other possibilities to improve the safety of the vehicular context. Valuable information data shared between vehicles will allow signal processing algorithms running on each vehicle to infer the conditions that may be experienced by other, nearby vehicles. Further, data measured by the multitude of sensors, whether from within a single vehicle or across multiple vehicles, can be used to correct malfunctioning or poorly calibrated sensors. Currently, vehicular sensors are recalibrated by bringing a vehicle to a certified garage to update the sensor or even to entirely replace the sensor. By using the distributed nature of the vehicular setting, in which there are numerous vehicles frequently making data measurements that are correlated across many dimensions, it becomes possible to report these data using communication technologies to cloud-servers that would perform large-scale data analytics to accurately identify the corrections needed, and determine suitable recalibration functions that can be distributed back to the vehicle in a software patch that updates vehicular sensors.
Going beyond the technical aspects themselves, while many in the technical community will recognize technology-enabled vehicular systems as an example of a “cyberphysical system”, what is often forgotten is that the transportation system also serves as a complex social fabric by which we interact with each other. Together, this merging of cyberphysical with social makes automating vehicles an ultra-hard problem to tackle. There are cues that we observe from other vehicles while we drive that suggest to us that we should drive more cautiously (e.g. a pedestrian looking at their smartphone while walking toward an intersection) or even avoid certain driving scenarios (e.g. an over-zealous driver swerving through lanes). While it may be unreasonable to expect that there will ever be algorithms capable of making the same social observations that we make, as humans, what we can expect is that technology will assist us in being as aware and informed as possible. Already there have been advancements made by the signal processing community to estimate driver distraction using in-vehicle sensors and that cue the driver to focus on the road, but there are many more opportunities for signal processing to analyze human behavior data associated with driving, and which will be essential for improving driver and pedestrian safety.
The future of vehicular systems is data and sensor-driven—vehicles will become increasingly networked and outfitted with sensors, sharing their data with a variety of in-vehicle and cloud-based computing services. There will be many societal benefits associated with improved vehicular systems, ranging from energy efficiency resulting from swarm driving to the potential for saving many lives should the technology mature. While this future is exciting, as engineers, researchers and technologists, we must quickly act to develop the new signal and information processing innovations that are needed to make future vehicular systems safe.