Deprtment of Computer Science nd Technology,Tsinghu University,Beijing 100084,Chin

bInformation Technology Center,Tsinghua University,Beijing 100084,China

cState Key Laboratory of Software Development Environment,Beihang University,Beijing 100191,China

dThe Institute of Electronic System Engineering,Beijing 100039,China

Original article

A study on key technologies of unmanned driving

Xinyu Zhanga,b,*,Hongbo Gaoc,Mu Guoa,Guopeng Lid,Yuchao Liud,Deyi Lia,b,c,d

aDepartment of Computer Science and Technology,Tsinghua University,Beijing 100084,China

bInformation Technology Center,Tsinghua University,Beijing 100084,China

cState Key Laboratory of Software Development Environment,Beihang University,Beijing 100191,China

dThe Institute of Electronic System Engineering,Beijing 100039,China

Although the development of machine intelligence is far from simulating all the cognitive competence of our brains,still it is absolutely possible to peel the driving activity from people's cognitive activities and then make the machine finish some low-level,complicated and lasting driving cognition by simulating our brains.The goal of driving is to replace drivers and free them from boring driving activities.Based on some studies on unmanned driving,this paper summarizes and analyzes the background,signi ficance,research status and key technology of unmanned driving and the research group also introduces some research on brain cognition of driving and sensor placement of intelligent vehicles,which offers more meaningful reference to push the study of unmanned driving.

Unmanned driving;Brain cognition of driving;Sensor placement;Formalization of brain cognition

1.Introduction

The development of cars has experienced three different stages,which is shown in Fig.1[1].The first stage relied totally on manual work without standardized parts and assembling process.The cars of this stage had high prices and their qualities were out of effective control.The second stage was characterized by standardized and streamlined production. Since the middle and later periods of the 1990s,the automobile industry has entered into the third stage.The technologies of comfort and intelligent safety have become the point of automobile industry.According to the statistics,from 1989 to 2010,the ratio of the costs of electronic equipments among the whole costs has increased from 16%to 23%.It is said that this ration may rise to 40%in 2015.In some luxury vehicles,the quantity of the single chip microcomputers has reached to 48 and the cost of the electronic equipments has accounted for more than 50%of the total cost of the cars[2].

In recent years,the rapid development of artificial intelligence,cognitive science,automatic control,ground mapping, sensor technology and other fields promotes the essential change of automobile industry.The symbol of wheeled mobile robots in the subversive creation of cars is ready to go ahead. The wheeled mobile robots would rather realize the goal of intelligent driving and free human drivers from low-level, complicated and lasting driving activities and change the interactive mode between cars and drivers fundamentally not emphasize the change of vehicle dynamics properties.Thus, cars will become personal mobile sharing tools.

There are two main routes to realize wheeled mobile robots: theintelligentrouteandtheNetworkroute.Theintelligentroute considers cars as intelligent individuals with perception, cognition and decision-making abilities,which emphasizes autonomous driving while the Network route considers cars as an adjustable node of the whole traffic system,which emphasizes overall coordination.The two routes cross with each other and form the future intelligent traffic system together.

Fig.1.A hundred years of the developmental process of automobile industry.

In 2013,Mckinsey Company listed twelve subversive technologies that can decide the future economy,among which advancedrobottechnologiesandunmanneddrivingtechnologies are included[3].The research and development of unmanned cars rely on the newest research results of artificial intelligence, cognitive science,automatic control,sensor technology and otherresearch fields,whichisthe best stage fortests.Unmanned drivingtechnologieshavebecomearesearchhotspotthatattracts the attention of governments both at home and abroad,scientific research institutions and enterprises because of its great signif icance in civil use,military use and research fields.

2.The research status of unmanned driving

The US National Highway Traf fic Safety Administration released the regulations of traf fic policies of intelligent driving cars in May,2013[4].The regulations divided the automatic degree of cars into five levels(Fig.2):level zero was no autonomous control,level one was intelligent driving with independent functions,level two was intelligent driving with cooperative control,level three was autonomous driving with limits and level four was total autonomous driving.

2.1.The research status of unmanned driving cars in US

The US is the first one to study unmanned driving cars in the world.In the 1980s,DARPA established special funds to support the research of autonomous land vehicles and held three DARPA challenge matches in 2004,2005 and 2007, which raised a great mass fervor of unmanned driving research [5](see Figs.3 and 4).

Google began to research unmanned driving in 2009 and it has finished designing several kinds of sample cars and a onroad test of nearly one million kilometers[6].Under the promotion of Google,Nevada,Florida,California and Michigan allow unmanned driving cars to test on public highways one after another[7].

Besides,GM,Ford and other American motor companies have assembled some driving assistance systems on the newest products such as self-adaptive cruise control,automatic parking,blind area alarm that derive from unmanned driving technologies.

2.2.The research status of unmanned driving cars in Europe

Europe began to research unmanned driving cars in the middle of the 1980s.Its research emphasized the unmanned driving cars as independent individuals and its normal travel in the traf fic steam instead of the cooperation of cars and roads. Since 1987 Europe has carried out a program called PROMETHEUS(Programme for a European Traf fic of Highest Ef ficiency and Unprecedented Safety)[12].In 1994,unmanned driving cars called VaMP and VITA-2 joined the normal traf fic steam on highways and the max speed has reached to 130 km/h[13].

Since 2006,Europe began to hold European Land-Robot Trials(ELROB)totestthepropertiesoflandrobots including unmanned driving cars under real situations.It has held three tests of military scenes and two ones of civil scenes. The military themes included reconnaissance and surveillance, autonomous navigation, fleet transport and so on while civil scenes included safety, fire protection,and disaster control and so on.In the later tests,the autonomous cars and its task completion advanced year after year[8].ELROB has played an active part in promoting the research of unmanned driving technologies in Europe[9].

In 2011,Berlin Free University researched the unmanned driving cars called Spirit of Berlin and Made in Germany. They finished tests of unmanned driving in urban areas and also other projects such as crowded traf fic flow,traf fic lights and travels around islands[10].

In 2014,BMW,Benz and Audi set forward Traf fic Jam Assistant to control steering wheels,accelerators and brakesand make the cars drive along the steam under automatic control under the speed of 60 km/h.At the same time,it freed the drivers'hands and feet,which was the most advanced intelligent safety products assembled on cars.

Fig.2.Intelligent levels of intelligent driving.

Fig.3.The champions of the three DARPA challenge matches—SandStorm,Stanley,Boss.

Fig.4.The first Google unmanned driving car.

2.3.The research status of unmanned driving cars in Japan

The research of unmanned driving technologies in Japan started very late and it paid more attention to reduce the accidentsusingintelligentsafetyandassistantdrivingbymeans of workshops communication.Japan's research on intelligent safety and cars networking is in the leading row in the world while its research on unmanned driving technologies is less.

2.4.The research status of unmanned driving cars in China

The unmanned driving technologies in China started from the later period of the 1980s,which were supported by the project of national Eight Six Three and the defensive agency [11].In 2003,Tsinghua University successfully invented THMR-V unmanned driving cars which could finish driving along the line on the structural road with clear car lines and the max speed was quicker than 100 km/h[12].

In 2008,National Natural Science Foundation put forward a great research plan of cognitive computing of the audio-visual information which raised a great mass fervor of unmanned drivingresearch[13-16].Since2009,NNSFhasheldsixfuture challenge matches of intelligent vehicles.The match contained basic ability tests and autonomous driving tests.Basic ability tests included testing mechanical properties and recognizing static traffic lights,traffic signs and traffic markings.And the autonomous driving tests demanded the cars to finish stipulated movements along the given lines,which included urban road tests,country road tests and unique road tests.

In September 2014,People's Liberation Army General Armaments Department organized the ground unmanned platform challenging matches of crossing obstacles-2014 and 21 motorcades joined in the match.The match put particular emphasis on the unstructured road tests which included roadblocks,collapsed walls,damage equipments,craters, ditches,puddles,dynamic obstacles and other subjects.

Our research group also carried out some experimental verification on the problem of cars'sneaking into the actual traffic flow.In 2011,the Red Flag HQ3 unmanned vehicles of National University of Defense Technology completed a 286 km unmanned driving test on highways from Changsha to Wuhan,in which its average speed was 87 km/h with manual intervention of 2.24 km[17].In 2012,the Lion N0.3 of Institute of military traffic finished a 114 km unmanned driving test on highways from Tai lake toll station in Beijing to Dongli toll station in Tianjin,in which it completed 12 autonomous overtaking and 36 changing tracks.Its total unmanned driving time was 85 min without manual intervention.The average speed was 79.06 km/h and the max speed was 105 km/h[18].

Compared with the advanced development of unmanned driving technologies in foreign countries,China still has some deficiencies.First,these motor enterprises lack passion and commitment on unmanned driving technologies.The cooperation between the motor enterprises and scientific research institutions ends with providing cars platforms and cooperating mechanical reformation,which lacks deep involvement. Second,the unmanned driving tests to actual traffic flows are limited to normal highways and short of unmanned driving experimental verification under complicated situations.

3.Key technologies of unmanned driving

3.1.The formalization of brain cognition of driving

Although the development of machine intelligence is far from simulating all the cognitive competence of our brains, still it is absolutely possible to peel the driving activity from people's cognitive activities and then make the machine finish some low-level,complicated and lasting driving cognition by simulating our brains.By navigating our brains to finish the formalization of driving cognition and making sure the accuracy and completeness of driving cognition,people feel hopeful to get the approaching or even better human driving abilities with the help of the machines'characteristics of not being absent-minded and never feeling tired.In the process of the formalization of brain cognition of driving,our brains should take the responsibility of audio-visual cognition, attention,memory,thinking,and decision-making,interaction and other tasks during driving.The driving coordination skills are assigned to automation and vehicle dynamics which are like cerebella so they carry on the results of automatic and intelligent control of cars.

The functional regions of our brains include sensory memory,working memory,long-term memory,computing center and thinking,motivation,character,emotion and so on. The model of brain cognition is shown in Fig.5.

Our sensory memory is responsible for instantaneous storage of sensory information which is large and can only be kept for very short time.This function corresponds to vehicle sensors'perception to surroundings.Figures and other regional signals from sensor are stored in cache regions and then new data will cover the old one,the mechanism of which is similar to the working principle of our sensory memory.

Some important experience,knowledge and scenes are stored in our long-term memory.This function corresponds to the driving map and driving operation model.The driving map records precisely all the geographic information related to driving which includes lane width,traffic signs,static obstacles and so on.The driving operation model contains line tracking model,car following model and lane-changing model,which is the operating standards of unmanned driving cars.The driving map and the driving operation model form priori knowledge of the unmanned driving system together.The contents related to the current activities in the long-term memory are extracted by computing center and thinking and then delivered to our working memory.This process corresponds to the mapping module of driving map of unmanned driving cars.

Our working memory stores some important information related to the current activities temporarily.Some of the information comes from the real-time information extracted from our sensory memory and some comes from the priori knowledge extracted from our long-term memory.The realtime information and the priori knowledge are integrated with each other and then provide an information pool for the computing center and thinking to analyze and make decisions. And at the same time,the unmanned driving system has a public data pool which shows the expression of the formalization of driving cognition.Multiple,heterogeneous and realtime driving information provided by sensor information processing modules and driving priori knowledge from driving maps are expressed together in a formal language of driving state,which shows the driving state beyond the unmanned driving cars entirely.

Besides,our brains also have character and emotion and other functional regions.Character shows the drivers'driving style at different times and in different places.As for unmanned driving,the driving style is decided by parameters of driving operation modules.Emotion is creatures'unique attribute.Anxiety and fear will affect people's driving behaviors and hinder safe driving.The realization of driving brains does not include emotion and cannot guarantee the safety of driving behaviors.

The types,quantities and its installation sites of the sensors are different on different unmanned driving test platforms and its information processing modules are also various.The granularity of the information provided by different driving maps does not have a stipulated standard.Thus the quantities and ports of the modules which constitute the unmanned driving system are different.We consider driving brains as the core and make the driving cognition formalized and also use the formalized language of driving cognition then we can design universal software framework of unmanned driving.Inthis framework,the intelligent decision module does not finish coupling with sensor information directly.It completes the intelligent decisions based on the comprehensive driving posture formed by sensor information and the priori knowledge of maps.

Fig. 5. Brain cognitive model.

The unmanned driving model based on brain cognition is shown in Fig.6.The output of all the sensors'information processing module is united by a formalized language of driving cognition.Then it constitutes the real-time information of the driving postures.The information of driving maps is reflected in driving postures according to the vehicles'positions and directions.And it mixes together with the real-time information of driving postures,which forms a public data pool that can reflect the current driving postures.The intelligent decision module lays foundation on this public data pool and puts traffic rules,driving experience,priori path and other priori knowledge in overall consideration to finish intelligent decision making.Besides,the public data pool also has the following functions:to help sensor information processing modules to decide areas of interest,to help positioning modules improve the accuracy of positioning,to help the module of driving map update the priori information timely and to help advance the properties of these modules.

The software framework of unmanned driving cars based on driving brains decouples the intelligent decision making and the sensor information.Through the formalized language of driving cognition,we can reduce the direct influence on intelligent decision making on the condition that the driving information is complete if we increase or decrease one or several sensors or we change the styles and the installed positions of the sensors.We can complete the transference conveniently on different testing platforms only if we make slight changes or no adjustments of the whole software framework.

3.2.The cognitive map of driving states

3.2.1.The cognitive map design of driving states

Considering that the data from the sensors collocated on the unmanned driving cars is distributed unevenly on the reference surface and the perceptive modes and shapes of perceptive sensors,we put forward a round variable granularity grid graph that is called a cognitive map of driving states as the formalized language of driving cognition.The cognitive map of driving states always centers on the core of the testing platform and covers some circle regions with a certain radius as required.These circle regions are divided into grids by a list of concentric circles and a list of radius with different angles. The radius length of these concentric circles from the inside one to the outside one increases by the law of index.The resolution ratio of the cognitive map of driving states can be a stable number or can be changed according to the focused areas of driving.The closer it is to the core,the size of the grids on the map will be smaller and the accuracy will behigher.Otherwise,its size will be bigger and the accuracy will be lower.

Fig.6.Unmanned driving model based on brain cognition.

The cognitive map of driving states considers the system of log polar coordinates as the reference system.Similar with the system of polar coordinates,the system of log polar coordinates is also a planar reference frame that is made up of polar angles and log diameters.The system of log polar coordinates regards the projection of the vehicles'geometric center on the ground as an origin point and the right ahead of the vehicles as the polar axis.The de finition of the polar angle is the same as the de finition of the polar angle in the system of polar coordinates.The polar angle refers to the intersection angle between the polar axis and the radial that is from the region point to the target point.The data range of the angle is from zero to 360,which can be seen in Fig.7.

Because of the dynamic characteristics of unmanned driving vehicles,they cannot finish the action of translation on the left and on the right.And as for the driving on structured lanes,it's ok to focus on the length of about 5 lanes on the sides of the vehicles.Therefore,the above-mentioned cognitive map of driving states is more suitable for unmanned driving planes and other driving platforms with high degree of freedom and translation on the left and right.But as for unmanned driving cars,we can tailor both sides of cognitive map of driving states according to the cars'dynamic characteristics and the needs of structured lanes.If we tailor a width of 20 m from the focused area on the left and right,we can get a cognitive map of driving states in the system of polar coordinates,which is shown in Fig.8.

3.2.2.The description of environmental elements

The task of unmanned driving demands that the cars must percept all the environmental elements related to driving totally.These environmental elements include the ground markings such as lane markings,intersection stop lines and zebra markings on pedestrian crossings,moving obstacles such as pedestrians and other vehicles,immovable obstacles such as road edges,isolation strips,rotary islands and deceleration strips,and other traffic elements such as traffic lights, traffic signs and traffic indicators.We can use the structural body to describe these elements.The types,attributes and representing methods of environmental elements are shown in Table 1.

Fig.7.The cognitive map of driving states in the system of polar coordinates of all angles.

Fig.8.The cognitive map of driving states of a width of 20 m in the system of polar coordinates.

3.2.3.Environmental elements in cognitive map of driving states

According to the different information stored in grids of the cognitive map of driving states,we can take some single information from the structural body and spilt the map into some fig.s with single or less information.We call the spilt cognitivemap of driving states with single or less information driving cognitive CT figure,several of which can express the cognitive map of driving states partly or fully.The methods of splitting the driving cognitive CT figure are not unique and the spilt driving cognitive CT figures needn't contain all the information in the cognitive map of driving states.Only the information convenient to be visually displayed on the cognitive map of driving states and helpful to interactive debug is necessary to be spilt by the driving cognitive CT figure.Table 2 lists a best example of the driving cognitive CT figure.

Table 1 Types,attributes and value ranges of environmental elements.

We can also design independent CT figures to spill and show the information such as the types and height of the traf fic lights,the types,height and samples of the traf fic signs.But the main function of the information is to help the sensor's information processing module to decide the areas of interest quickly and increase the testing speed and accuracy.The information related to intelligent decision-making and path planning is ok to be displayed discretely because its in fluence is not strongly related to the position of grids.Thus there's no need to display by independent CT figures.

It is logic and visual to spill the cognitive map of driving states into CT figures.But it does not mean that the intelligent decision-making can be completed depending on only one CT figure.An accurate intelligent decision-making needs to consider all the traf fic elements comprehensively and can only be finished on the basis of analyzing all the information on the cognitive map of driving states.

Fig.9 shows the driving cognitive CT figure with the only information of ground marking and it just spills from the cognitive map of driving states within 10 m on all sides of unmanned driving vehicles.After this,the display of all kinds of ground markings shows obvious zigzags.But the resolution ration of the angle in real use will not cause such obvious zigzags.However,they still exist.The zigzags must exist when we showmostgroundmarkingsdiscretelynomatterweusearound variable granularity or a quadrate granularity grid graph.The granularity oftheroundvariablegranularitycognitivemap isso thin that the zigzags are not obvious because it is close to vehicles.On the contrary,the further it is to vehicles,the thicker thegranularitywillbeandthezigzagswillbemoreobvious.But it does not affect the decision-making.In the visual case of Fig.9,we use the grey value of 250 to stand for the intersection stopping line,the grey value of 200 for the pedestrian line,the grey value of 150 for the yellow real lane marking,the grey valueof100forthewhitereallanemarkingandthegreyvalueof 50 for the white virtual lane marking.Different mapping methods to show the grey value can be adjusted flexibly according to visualization or can be mapped in colorized values.

The fig.contains 5 parallel white real lane markings and the marching direction of the vehicles keeps parallel with the lane markings'direction.There is an intersection stopping line before the vehicle about 6 m and some pedestrian lines close to it.All the pedestrian lines become one because they have some certain distances with the vehicles and also the resolution ration of the angle is too low.That's why we cannot distinguish every parallel line.But this change will not cause ambiguity or influence the decision-making of unmanned driving.

Table 2 A best example of driving cognitive CT figure.

Fig.9.The cognitive map of driving states with ground marking information.

4.Discussion and ending

We are devoted to researching and experimenting the unmanned driving all the year round and we have finished 20 times of unmanned driving experiments on highways from Beijing to Tianjin,the total mileage of which reaches to 20,000 km.It is the first time to realize the unmanned driving successfully on domestic inter-city highways and also accept the third party's test from some authorities.We are among the best candidates in the third,fourth and fifth China's intelligent vehicles'future challenge matches.In August 2015,we completed an unmanned driving test of passenger cars on inter-city highways of Zhengzhou-Kaifeng city expressway in Zhengzhou,Henan province,the length of which was 32 km. The passenger car reached to the end successfully without manual intervention and accepted the third party's test of authorities.

The research and experiment of unmanned driving vehicles are based on the newest research results such as artificial intelligence,cognitivescience,automaticcontrol,ground mapping,sensor technology and other fields.With the rapid development of the ground mapping with high accuracy and sensor technologies,the unmanned driving vehicles are able to get a great number of data from surroundings.How to handle with the great number of real-time data that is multiple and isomerous effectively and then make a driving decision quickly and accurately is a main difficulty for the research and experiment of unmanned driving vehicles.As our era's favorite,the intelligent vehicles will improve our life quality.We are looking forward to creating the acme of cars—racing robots.

Acknowledgment

This work is supported by National Natural Science FoundationofChinaunderGrantNo.61300006,No. 61305055,No.61035004,No.61273213,No.61203366 and No.90920305,and China National High-Tech Project(863) under grant No.2015AA015401,and Chinese Academy of engineering consulting Project No.2015-XY-42.

[1]E.Eckermann,World History of the Automobile,Training,2014, 02-19.

[2]W.B.Ribbens,Understanding Automotive Electronics:an Engineering Perspective,Pergamon Press,2013.

[3]M.Chul,R.Dobbs,P.Bisson,et al.,Disruptive Technologies:Advances that Will Transform Life,Business,and the Global Economy,McKinsey Global Institute,2013,pp.1-30.

[4]J.W.Runge,National Highway Traffic Safety Administration.Rear End Crashes:Problem Size Assessment and Statistical Description(Technical Report DOT HS 807 994)28(1)(2010)429-432.

[5]M.Buehler,K.Iagnemma,S.Singh,The DARPA Urban Challenge: Autonomous Vehicles in City Traffic,Springer Science&Business Media,2009,pp.23-145.

[6]Self driving car[DB/OL].http://googleblog.blogspot.jp/2014/04/thelatest-chapter-for-self-driving-car.html.

[7]J.Markoff,Google Lobbies Nevada to Allow Self-driving Cars,The New York Times,2011,p.10.

[8]T.Braun,H.Schaefer,K.Berns,Topological large-scale off-road navigation and exploration RAVON at the European Land Robot Trial 2008, in:Intelligent Robots and Systems,2009.IROS 2009.IEEE/RSJ International Conference on.IEEE,2009,pp.4387-4392.

[9]M.Himmelsbach,T.Luettel,F.Hecker,et al.,KI Ku¨nstliche Intell.25(2) (2011)145-149.

[10]A.Reuschenbach,M.Wang,T.Ganjineh,et al.,iDriver-human machine interfaceforautonomouscars,in:InformationTechnology:NewGenerations (ITNG),2011EighthInternationalConferenceon.IEEE,2011,pp.435-440.

[11]Ming Yang,J.Harbin Inst.Technol.38(8)(2006)1259-1262(in Chinese).

[12]Pengfei Zhang,Kezhong He,Zhengzhu Ouyang,Junyu Zhang,Robot 24 (2)(2002)97-101(in Chinese).

[13]S.Tsugawa,The current trends and issues on ITS in Japan:safety,energy and environment,in:Intelligent Radio for Future Personal Terminals (IMWS-IRFPT),2011 IEEE MTT-S International Microwave Workshop Series on.IEEE,2011,pp.1-2.

[14]R.Behringer,N.Muller,Robot.Autom.IEEE Trans.14(5)(1998) 810-815.

[15]A.Broggi,P.Cerri,S.Debattisti,et al.,PROUD-public road urban driverless test:architecture and results,in:Intelligent Vehicles Symposium Proceedings,2014 IEEE,IEEE,2014,pp.648-654.

[16]Min Tan,Shuo Wang,ACTA Autom.Sin.39(7)(2013)963-972(in Chinese).

[17]U.Ozguner,K.A.Redmill,A.Broggi,Team TerraMax and the DARPA grand challenge:a general overview,in:Intelligent Vehicles Symposium, 2004 IEEE,IEEE,2004,pp.232-237.

Available online 3 June 2016

*Corresponding author.Department of Computer Science and Technology, Tsinghua University,Beijing 100084,China.Tel.:+86 10 62792959.

E-mail address:xyzhang@tsinghua.edu.cn(X.Zhang).

Peer review under responsibility of Chongqing University of Technology.

http://dx.doi.org/10.1016/j.trit.2016.03.003

2468-2322/Copyright©2016,Chongqing University of Technology.Production and hosting by Elsevier B.V.This is an open access article under the CC BY-NCND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

Copyright©2016,Chongqing University of Technology.Production and hosting by Elsevier B.V.This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).