Teaching trains to see
How can robot trains learn to identify obstacles?
- Thales is pioneering computer vision for trains
- Paves the way for autonomous rail operations
- Artificial intelligence (AI) and deep neural networks hold the key
Here’s something you’re probably familiar with. You’ve just logged in to a website – perhaps to do some shopping – and suddenly, you’re confronted by an apparently random mosaic of photographs. “Select all images with street signs,” “then click to verify”.
This is known as a “challenge-response” test. Identifying specific objects in a photograph is what data scientists call a “hard” AI problem: humans are good at this, computers less so (at least for now). Tests of this sort are used to protect websites against malicious cyber attacks.
What has all this got to do with railways? The answer is that the next generation of trains will be autonomous – and robot trains will depend on computer vision to identify objects on or near the track, without any help from humans. Not only that, they’ll need to beat humans at it, doing it in darkness, fog and snow at speeds of up to 300km/h.
So how do you teach a train to “see”? And more to the point, how can computer vision be improved to match the standard of human sight – and maybe even exceed it?
Pioneering research and development work by Thales is providing answers to these questions. The work is being carried out at Thales’ dedicated innovation lab.
The secret lies in “deep neural networks” – software algorithms that mimic the way the human brain works. Algorithms are “trained” to recognise objects by being shown examples in the lab.
“First, we train algorithms to recognise the tracks – it is very important to know if the object detected is on the track or beside it,”explains Thierry Lamarque, Head of the Research & Innovation Laboratory, Thales. “We then train the algorithms to recognise objects such as a tree, a person or an animal. The challenge is that all of this has to be done in real time – and at a distance which is great enough to stop the train if an obstacle is detected.”
How does this work in practice? Let’s imagine a horse has escaped from a field and has wandered onto the track.
On the train, an algorithm recognises “horse” by looking at video fed from a camera at the front of the train. Video is good for identifying objects, but it’s relatively slow: at 25 frames per second, a fast train will have travelled about three metres between one image grab and the next. Video is also a poor judge of distance. What’s more, conventional cameras can’t see in the dark.
“Visual detection on its own is not enough,” says Lamarque. “The solution is to merge the video data with data from other sensors. Radars, for example, allow us to measure the velocity of the object.”
A lidar adds further detail. “Lidar is very quick and it provides a 3D view. This gives you information on depth and distance. You can’t get this from a camera on its own,” says Lamarque.
So by combining, or “fusing”, data from different sensors, it is possible to determine the key characteristics of the potential obstacle: video tells us that it is a horse, radar determines how fast the horse is moving and, finally, lidar tells us how far away the horse is from the train. All of this happens in the blink of an eye.
But what if it’s night time? That’s not a problem for sensors such as lidar and radar, which can “see” in the dark. Video, however, needs light.
“We solve this by fusing both visible and infrared cameras,” explains Lamarque. “There are several wavelengths you can use in the infrared domain that allow you to obtain images in darkness or through fog. We take advantage of different sensors to cover all the situations that we could face.”
One of the challenges facing researchers is, surprisingly, not so much detecting objects in the real world, but understanding exactly how a trained algorithm decides what an object is. Just like maths homework at school, you need to show your working.
“This is why “explainable AI” is becoming so important,” says Lamarque. “When the algorithm says ‘there is a horse’, you need to know how it has reached that conclusion.”
The potential of autonomous operations to transform railways is enormous. Autonomy will not only boost the capacity of networks, but also has the potential to improve safety because computer vision can detect obstacles that cannot be seen by human operators – particularly when it is dark or when visibility is restricted.
“If you had asked me three or four years ago whether autonomous trains would be possible, I would have answered: not sure,” says Lamarque. “But the sensors and associated algorithms are evolving so quickly that I think we can now say yes, they will become a reality in the next few years.”