Dharuna

Algorithm developed

Objective: To predict the future coordinates of the ego vehicle given a set of present and past coordinates.

Introduction: In order to safely and efficiently navigate through complex traffic composed by human drivers, an autonomous vehicle needs to have the ability to take initiative, such as deciding when to change lanes, overtake another vehicle, or slowing down to allow other vehicles to merge. This requires the autonomous vehicle to have some ability to reason about the future motion of surrounding vehicles. This can be seen in existing tactical path planning algorithms which depend on reliable estimation of future trajectories of surrounding vehicles. Prediction of future motion of surrounding vehicles is a challenging problem due to the high number of latent variables involved, such as, the end goals of all drivers in the scene and variability in driving style across different drivers. Vehicle trajectories tend to be highly non-linear over longer time horizons due to decisions made by the driver. Additionally, driver behavior tends to be inherently multi-modal, where a driver could make one of many decisions under the same traffic circumstances. Finally, interaction between vehicles tends to affect their motion. The large number of possible configurations of all vehicles in a scene can make this difficult to model. In spite of these challenges, there is structure to vehicle motion that can be exploited:

The maneuvering of the ego vehicle is divided into 6 possibilities viz. Break right, break left, continue on the same lane, maintain existing speed, accelerate and decelerate. For complying with the above statement we are expecting correctly marked and well-defined lanes on roads.

Implementation: We're using concepts of Convolution-Neural Network(CNN) and Long short-term memory(LSTM) from Deep Learning. We also use a pooling layer combined with CNN named convolution social pooling. LSTM is used as an encoder-decoder. The model can be represented as:

Convolutional social pooling: We propose a novel social pooling layer as an alternative to that proposed in. We apply convolutional and max-pooling layers instead of a fully connected layer to social-tensors of LSTM states that encode the past motion of neighboring vehicles.

Maneuver based decoder: Our LSTM decoder generates the probability distribution over future motion for six maneuver classes and assigns a probability to each maneuver class. This accounts for the multi-modal nature of vehicle motion.

For frame of reference we choose the ego vehicle(shown in black) as the origin and the Y axis in longitudinal to the lane and X axis lateral. The maneuvers are shown as above.

Test Data Set:

We use the publicly available NGSIM US-101 [4] and I-80 [5] datasets for our experiments. Each dataset consists of trajectories of real freeway traffic captured at 10 Hz over a time span of 45 minutes. Each dataset consists of 15 min segments of mild, moderate and congested traffic conditions. The dataset provides the coordinates of vehicles projected to a local coordinate system, as defined in section 3.1. We split the complete dataset into train and test sets. The test set consists of a fourth of the trajectories from each of the 3 subsets of the US-101 and I-80 datasets. We split the trajectories into segments of 8 s, where we use 3 s of track history and a 5 s prediction horizon. These 8s segments are sampled at the dataset sampling rate of 10 Hz. However we downsample each segment by a factor of 2 before feeding them to the LSTMs, to reduce the model complexity.

Inputs: It is a tensor of track histories as a function of coordinates and time frames.

X = [x(t-th)... x(t-1), xt] ; where xt=[ x0, y0, x1, y1… xn, yn] for a particular time interval t. And x0, y0are the coordinates of the ego vehicle(vehicle whose trajectory is to be predicted) and rest are of the neighboring vehicles within deviation of 90 feet in Y axis and the two adjacent lanes. So, basically X is a (th X n) tensor. More specifically termed as social tensor as it interacts with the neighbouring vehicles.

Outputs: Our output is tensor Y such that Y = [y(t+1), y(t+2), … y(t+tf)], where y(t)=(x, y) the coordinates of the ego vehicle at time=t. We predict till a considerable time (t+tf)

Execution:

From the above figure we observe that: The data of coordinates of each vehicle(including ego) are passed through separate LSTM. It actually retains the memory for previous time frame coordinates and acts as a filter for some of it.

A 13 X 3 spatial grid is defined around the vehicle being predicted, where each column corresponds to a single lane, and the rows are separated by a distance of 15 feet which approximately equals one car length. The social tensor is formed by populating this grid with surrounding car locations. We then apply two convolutional layers and a pooling layer to the social tensor as shown in Figure to obtain the social context encoding. Additionally, the LSTM state of the predicted vehicle is passed through a fully connected layer to obtain the vehicle dynamics encoding.

The social tensor is formed after encoding.

Convolution is done with a 3x3 matrix and a neural network is formed, since the dimension of the convolving matrix is 3 so it also accounts for the neighbouring vehicle and how it affects the path of the ego vehicle. Then we train the model end to end. To decide the weights of each of the layers of the neural network we try to minimize the negative log likelihood. Which is mathematically represented by: -log(P(Y|mi, X)*P(mi, X)) Summed over i We train the model using Adam with learning rate 0.001. The encoder LSTM has 64 dimensional state while the decoder has a 128 dimensional state. The sizes of the convolutional social pooling layers are as shown in the figure under the execution head. The fully connected layer for obtaining the vehicle dynamics encoding has size 32. We use the leaky-ReLU activation with =0.1 for all layers. The model is implemented using PyTorch.

Tensor is scaled down to 64 dimensional, further application of 3x1 convolution scales it to 16. We apply max pooling, which means selecting the maximum magnitude out of some values in a tensor. It is then concatenated with the ego vehicle’s data and fed into another decoder which is also a network of LSTMs and it creates the posterior distribution of probabilities of the maneuvers.

Finally the maneuver with highest probability is chosen, and the coordinates are set accordingly by adding the maneuver parameter to the present coordinates.

Marketing Pitch

For Govt. and other Pvt. Companies

Ikshana presents to you - Dharuna, which is an app that helps to regulate the Traffic Light Timings Automatically on the basis of queue length in any specific lane.

Approach
The app will send the location of all the users using the app while driving. With help of this, we will be able to provide you with datasets constantly of that lane. Using the coordinates of the vehicles we can predict their future path using our DeepLearning Algorithm.

The predictions for oncoming vehicles when the lane is Green can also be provided so that the timings can be regulated to maximum accuracy and precision. Implementation - Govt. can make this app as a mandatory app for people. Since we will be asking for a very simple and secured registration through the app, there rises no question for privacy theft.

We are also planning to collaborate with major smartphone companies in this project to make Dharuna, an in-built app in smartphones.

Additionals
Ikshana, in collaboration with Vintech Automation, Mumbai, presents an offer to sell the data for the timing regulation along with a fully equipped ‘Traffic Lights’ at a very subsidized rate. This collab offer benefits the purchaser by 10-15% of total cost born when the traffic lights are set up on their own and datasets are sold solely along with the installation charges.

Benefits
Better reputation in the market and among the users, leading the Govt. for better reach among their vote banks and Private Constructors, for better contracts.

Target Audience

The algorithm in its most basic senses, predicts the trajectory of all the vehicles whose coordinates are known to us. With an additional feature of releasing the algorithm through an app, we can target a huge horizon of the market for its best outputs.

The app itself, which on the front end helps the users to avoid accidents by alerting them well in advance, has a massive target base. With the given stats in the “Why Dharuna?” page, the users will understand the graveness of traffic management, eventually resulting in better reach for the app.

On the other hand, i.e. back end, the companies/govt. which are developing the crossings in their jurisdiction can make tie-ups with Ikshana for improving the traffic conditions in their region by regulating the traffic light timings. Also, additionally by enforcing compulsion on the use of the app, the companies/govt. can take credits for a reduced number of accidents, eventually increasing their contracts, reputation, vote banks, etc.

Legal Approach

Dharuna uses geolocation for receiving your coordinates to compute its further calculations. Tracking a person may cause many legal complications due to misunderstandings. We at Ikshana, have dedicated a Legal Department for answering your doubts.

The Legal Department shall take care of ensuring that the data sold to the Govt. Authorities which are in the form of coordinates and statistical values shall be used for the timing regulations of the traffic lights and nothing else. Data and Privacy Theft is a very big issue, and we are much more cautious towards it than anything else since we have been trusted for so many years by so many people.

Also, the Legal Department may take care of Open Source Licences for the codes and algorithms used in the back end of this product.

Even though our product may be a technical outcome and not a physical one, we have dedicated our Legal Team to their best efforts for the smooth functioning of our product.