The following is adapted from Walter Isaacson’s biography “Elon Musk,” which is being published on September 12.
On a Friday in late August of this year, Elon Musk entered his Model S at Tesla headquarters in Palo Alto, selected a random spot on its navigation screen and let the car drive itself using full self-driving technology. For 45 minutes, while listening to Mozart, he live-streamed his trip, including a pass by the home of Mark Zuckerberg, whom he jokingly challenged to a cage-match. “Maybe I should knock on the door and politely inquire if he’d like a hand-to-hand fight,” he said with a chuckle before letting the car drive away.
Musk uses FSD 12 on August 25, 2023.
Musk had used FSD hundreds of times before, but not because this drive was much smoother and more reliable, but because it was so different. The new version he was using, FSD 12, was based on a radical new concept that he believed would not only revolutionize autonomous vehicles but also represent a quantum leap toward artificial general intelligence that could operate in real-world situations. Rather than relying on hundreds of thousands of lines of code, like all previous versions of self-driving software, this new system taught itself how to drive by processing billions of frames of video of how humans do it. Language Model Chatbots train themselves to generate answers by processing billions of words of human text.
Incredibly, Musk set Tesla on this fundamentally new approach just eight months ago.
“It’s like ChatGPT, but for cars,” Dhaval Shroff, a young member of Tesla’s Autopilot team, explained to Musk at a meeting in December. He compared the idea they were working on to a chatbot recently published by OpenAI, the lab Musk founded in 2015. “We process massive amounts of data on how real human drivers behaved in complex driving situations,” it said. Shroff, “And then we train a computer’s neural network to mimic that.”
Dhaval Shroff works at his desk at Tesla.
Until then, Tesla’s Autopilot system relied on a rules-based approach. The car’s cameras recognize things like lane markings, pedestrians, vehicles, signs and traffic signals. Then the software applied a set of rules, such as: stop when the light is red, go when it’s green, stay in the center of the lane markers, only pass through intersections when there are no cars coming fast enough to hit you, and so on. Tesla engineers manually updated these rules by writing hundreds of thousands of lines of C++ code to apply them to complex situations.
Shroff and others working on “neural network planners” took a different approach. “Instead of determining the correct path for a car based on rules,” says Shroff, “we determine the correct path for a car by relying on a neural network that learns millions of examples of what humans have done.” In other words, it is human imitation. When faced with a situation, a neural network chooses a path based on what humans have done in thousands of similar situations. This is how humans learn to speak, drive, play chess, eat spaghetti, and almost everything else; We may be given a set of rules to follow, but mostly we pick up skills by observing how other people do them. It was an approach to machine learning envisioned by Alan Turing in his 1950 paper “Computing Machinery and Intelligence” and which came into the public eye a year ago after the release of ChatGPT.
By early 2023, the Neural Network Planner project had analyzed 10 million clips of video collected from Tesla customers’ cars. Does that mean it will be as good as the average of human drivers? “No, because we use their data only when they have handled a situation well,” explained Shroff. Human labelers, many of them from Buffalo, New York, evaluated and graded the videos. Musk asked them to find things that “a five-star Uber driver would do,” and those were the videos he used to train the computer.
Musk regularly walked through the Autopilot workspace in Palo Alto and knelt down next to engineers for impromptu discussions. While studying the new human-mimicking method, he had a question: Was it really needed? Could that be a little overkill? He had a saying that you should never use a cruise missile to kill a fly; Use only flyswatter. Is using a neural network unnecessarily complicated?
Shroff showed Musk examples where a neural network planner would perform better than a rule-based approach. The demo featured trash cans, fallen traffic cones, and a road littered with random debris. Guided by a neural network planner, the car was able to maneuver around obstacles, cross lane lines and break certain rules as needed. “What happens when we go from rule-based to network-path-based,” Shroff told him. “If you turn this thing on even in an unstructured environment, the car will never crash.”
It was this kind of leap into the future that excited Musk. “We should do a James Bond-style demonstration,” he said, “where there’s bombs going off on all sides and UFOs falling out of the sky while cars speed by without hitting anything.”
Machine-learning systems generally need a metric to guide them as they train themselves. Musk, who likes to manage by deciding metrics should be paramount, gave them his lodestar: the number of miles a fully self-driving car could travel without human intervention. “I want the latest data on miles per intervention to be the opening slide at each of our meetings,” he decreed. He asked them to make it like a video game where they could see their score every day. “Video games without scores are boring, so increasing miles per intervention will motivate them to watch them every day.”
Team members installed large 85-inch television monitors in their workspaces that show in real time how many miles the FSD cars are driving on average without intervention. He kept a gong near his desk and banged the gong whenever he intervened and solved a problem.
By mid-April 2023, it was time for Musk to use the new neural network planner. He sat in the driver’s seat next to Ashok Eluswami, director of Tesla’s Autopilot software. Three members of the Autopilot team followed. As he prepares to leave the parking lot at Tesla’s Palo Alto office complex, Musk picks a spot on the map for the car to go to and takes his hands off the wheel.
When the car turned onto the main road, the first scary challenge arose: a cyclist was heading their way. The car generated income on its own as a human would have done.
For 25 minutes, the car drove on expressways and neighborhood streets, handling complex turns and avoiding cyclists, pedestrians and pets. Musk did not touch the wheel. He only intervened a couple of times by tapping the accelerator when he thought the car was being overly cautious, such as being too respectful at a four-way stop sign. At one point the car did a trick that he thought was better than he would have done. “Oh, wow,” he said, “even my human neural network failed here, but the car did the right thing.” He was so pleased that he started whistling Mozart’s “A Little Night Music” serenade in G major.
A frame of a livestream of Musk’s drive on August 25, 2023 using FSD 12.
“Amazing job, guys,” Musk concluded. “It’s really impressive.” Then they all went to the Autopilot team’s weekly meeting, where 20 guys, almost all in black T-shirts, sat around a conference table to hear the verdict. Many did not believe that the neural network project would work. Musk declared that he was now convinced and that they should shift their resources to push it forward.
During the discussion, Musk pointed out an important fact the team discovered: Neural networks don’t perform well until they’ve been trained on at least a million video clips. This gave Tesla a huge advantage over other car and AI companies. A fleet of nearly 2 million Teslas worldwide was collecting video clips every day. “We are in a unique position to do this,” Eluswami told the meeting.
Four months later, the new system was ready to replace the old approach and form the basis of FSD 12, which Tesla plans to release subject to regulatory approval. There’s still one problem to overcome: human drivers, at best, generally flout traffic rules, and the new FSD, by design, mimics what humans do. For example, more than 95% of humans crawl slowly through a stop sign rather than coming to a complete stop. The head of the National Highway Safety Board says the agency is currently studying whether self-driving cars should be allowed as well.
Walter Isaacson CNBC contributor and author of biographies of Elon Musk, Jennifer Doudna, Leonardo da Vinci, Steve Jobs, Albert Einstein, Benjamin Franklin and Henry Kissinger. He teaches history at Tulane University and was editor of Time and CEO of CNN.