‘I don’t want people to die from an attack through a software updater.’
— Uptane developer Justin Cappos
By Mark Shortt
Today’s automobiles are essentially computers on wheels, a fact that makes them as vulnerable as laptops are to security breaches by malefactors. As a result, automotive engineers need to design and configure a vehicle’s systems in ways that best protect the vehicle’s occupants from intrusions that could steal personal information or threaten their physical well-being. It’s a challenge that engineers deal with from designing parts for electronic control units (ECUs), all the way through a car’s lifecycle.
For automotive OEMs, a software framework that hardens a vehicle’s security during over-the-air (OTA) software updates is one of the best protections they can have. An example is Uptane, a cybersecurity toolkit developed by Justin Cappos, a professor of computer science and engineering at New York University (NYU) Tandon School of Engineering, along with collaborators from industry, academia, and government.
Uptane is based on The Update Framework (TUF), an open source technology for securing software updates that was developed by Cappos and a team of researchers with support from the National Science Foundation and U.S. Department of Homeland Security. Uptane, essentially the automotive application of TUF, has been integrated into Automotive Grade Linux and is now helping to secure OTA updates for major automakers worldwide.
According to a release from NYU Tandon, Uptane enables automakers and suppliers to secure major software updates to automotive infotainment and telematics units. It also enables “remote, inexpensive updates to the ‘edge’ — the dozens of in-vehicle ECUs controlling numerous functions in today’s vehicles.” Its code is posted on Github for all to see, test, or use.
“Uptane helps Linux secure updates at places where Linux can’t run, since many ECUs, such as brake controllers, have tiny Flash memories,” said Cappos in the release. “While we are essentially an encryption algorithm independent of Linux, we are part of Linux’ high-end expansion out to smaller devices.”
Justin Cappos took time out in March to talk with D2P about Uptane and how it helps automotive manufacturers to secure their over-the-air updates. Following is a transcript of the interview, edited for length and clarity.
D2P: How did your work with the Update Framework, and later, Uptane, come about?
Justin Cappos: I’ll give you the short version, but it will go back a ways. Back in 2002, I built a package manager called Stork, which was the first package manager for what we now call the cloud. It was widely used in an academic prototype called Planet Lab, which some people have described as really the first cloud. I think that’s a bit overstating it, but it was certainly very early in that space.
They had thousands of computers spread across 700 locations all over the world, most of them universities. As part of this, I built the package manager that was specifically designed for these cloud environments—this new cloud environment.
What came out of this is, I started to have to think a lot about the security of the system. I looked at how Linux package managers were handling a lot of the security problems I had noticed when I was doing Stork. And it turned out that they weren’t handling them.
So, I went and found these issues, and I worked with package managers to fix them. It got a lot of press, it got a lot of coverage, and folks at the TOR project reached out, and said, “Wow! We read your paper; we saw all these articles about it on Slashdot.” And so, they came and spent some time with me when I was at the University of Washington, and we talked through a design for how to do a more secure version of what I had done in my system, Stork.
They went away and came back with a design. These serious crypto security experts came back with a design and said, “We just want you to review this before we go live, but we think this is really solid.” And I found eight security issues in their design—including three that were very serious flaws that could have let any man-in-the-middle (MITM) attacker compromise all their users. At that point, I said, “OK, this is not going to work. I can’t just write a paper and say, ‘Do it this way,’ and just let people figure out the details.”
So, instead, I decided to start the TUF project, where TUF is meant to be like a library that you can sort of drop in. It’s like a library/specification. You can think of it like TLS (transport layer security), where if you’re following the protocol and doing everything right, then you have a good degree of assurance that what you’ve done is going to be secure.
And so TUF got a lot of adoption. It’s very widely used in the cloud space. It’s one of only, I think, six or seven technologies that’s at the highest level of maturity inside the Linux Foundation’s Cloud Native Computing Foundation (CNCF), so it’s considered one of the most widely used and important technologies in the cloud.
As this was happening, in maybe 2015 or 2014 or so, a bunch of automotive hacking things happened. And someone at the Department of Homeland Security reached out to me and said, “Look, the automakers know they have a big problem, and they know software updates are a big part of this problem.” They knew I’d done the software updaters for most of the Linux world, and for the cloud, and they know that I’ve worked with Apple and Microsoft, Google, and others, and done things with Android and other updaters. So, they said, “Would you be willing to work with the automotive community?” I said of course I would. I want to help people.
And that’s really how Uptane started—to try to take something like TUF and make it work in the automotive community with all the different challenges and constraints, which is a very different field. Obviously, automotive is very different from working on servers or cloud systems, or even phones.
D2P: Generally, how is Uptane used and how does it work? I understand it’s an open source framework that invites security reviews.
JC: Yes. It’s being used by a bunch of major OEMs. I don’t believe it’s widely used in production today, because everything takes four years—they [automakers] have these long cycles. But we’re in the pipeline for multiple major OEMs in the U.S., in Europe, and in Japan.
One thing that’s unique about what we’ve done is that our specification—the design of what we’re doing—is completely free and open for anybody to use. There are no costs. There are implementations of the system that are free to use, and there are also a bunch of companies that are selling implementations of Uptane.
I think you should be suspicious if a company comes to you and says, “We’re not going to use AES (Advanced Encryption Standard),” or something like that. “We’ve made our own cryptographic algorithm.” That would just be insane in the security world, to hear somebody say, “Yeah, you could use X, but we’ve made up our own thing, and you should try ours instead.” People just wouldn’t buy that.
Uptane isn’t like that. It’s publicly reviewed, very audited, lots of people have looked at it. It has had lots of scrutiny and has been shown to be effective. It’s something that’s basically already been piloted in some context and it’s well on its way to massive, widescale adoption.
Really, my goal with all this is I don’t want people to die from an attack through a software updater, which I think is actually quite likely with a lot of the current designs that people were using. So, Uptane is really trying to prevent that from occurring to the extent possible.
D2P: Is Uptane compatible with other types of cybersecurity platforms?
JC: It’s compatible with all sorts of things. There are some people that do additional signing. Basically, they’ve already set up an infrastructure where they do something that’s kind of like TLS—you know, HTTPS to and from the car, and you can just transmit Uptane metadata over that if you want. It works perfectly well with any other types of monitoring that you have.
So, it plays very well with other things, and it handles a lot of things for you. It effectively handles key revocation; it minimizes the impact of security incidents. Say somebody’s breaking into a server that has Uptane signing keys, or even breaking into a supplier and getting things upstream. It helps to minimize the impact of those types of events in your system.
We hope people are using other security mechanisms along with it. And there are quite a few things: doing commit signing, having an IDS (intrusion detection system), having firewalls—those are all good things to have alongside.
D2P: Can this be used in the design phase for automotive components as well? Does a design engineer at an automotive OEM have an opportunity to use this to design security into a product or part before it is actually manufactured?
JC: When you do that initial sort of loading of software, as you’re making the parts, some places are using Uptane as part of that process, too. But in general, it’s something that solves the over- the-air update problem, and it also can be used to help do some of that initial provisioning of trust.
Let me add one more thing that maybe adds a little nuance, which is, most of the time, the OEM wants to be in charge of all the updates for their vehicle. But in a few cases, this hasn’t always been the case. And with these new components, you can do this with Uptane, independently, if you wanted to, of the OEM—in coordination with the OEM.
D2P: What should a design engineer, manufacturing engineer, or software engineer at an automotive manufacturer know about the Uptane platform that could inform their work in making a better component for an ECU, for example, or one that is more compatible with, and maximizes the security benefits of Uptane?
JC: All you really need to support Uptane is a little extra space to store some metadata you’ll download. This is something like 10KB for partial verification (a weaker subset of Uptane requirements), or 100KB for the full verification (the strong verification that any safety-critical ECUs should perform, and most should do). The code for Uptane is not overly large, and the computation is a fairly small number of asymmetric signature verifications per update.
Uptane can also do partial verification on very minimal devices that only support symmetric crypto, if desired. It’s important to have enough extra storage to have whatever patch you plan to apply as well.
If you have hardware support for storing keys on an ECU, Uptane supports this as well. Having this hardware support will supplement Uptane’s protections to help limit the damage from an ECU compromise in certain cases. If you can only do one or the other, Uptane is clearly the more important thing to support, but both together are nice to have.
D2P: I understand that Uptane prevents attacks by storing the correct encryption keys with the automaker offline. Is that correct?
JC: It’s true to some degree. One problem that most systems have is that they keep, basically, all their keys online. And even if it’s in something like an HSM, or some other kind of trusted hardware, attackers have time and time again shown they can cause those keys to do things like sign malicious updates.
What Uptane does is it uses some online keys, but those online keys are not used to trust sensitive things in the same way they are in a traditional update system.
So, to give an example, let’s say that we’re going to go to restaurants, and I am going to make recommendations, or I’m literally, maybe even just going to order for you because I’m your personal dietician. But you also, maybe, should be a little concerned because you don’t know if I may poison you, right? I’m just trying to come up with an example here that makes some sense.
The way that most updaters work is, effectively, as though the person who’s picking for you is like your chef. They’re making the meal and they’re deciding everything they’re doing, and they’re getting the ingredients. And so, if they turn malicious, they can go and slip in a poisonous mushroom or something like that and—boom!—you’re dead.
Whereas, with the Uptane model, the person who’s doing the selection is like a person who’s picking what dish you will order at a restaurant. And so, that is a lot more difficult [for a person with malicious intent] because the restaurant isn’t going to have poisonous mushroom. That’s not going to be on their menu.
So, that part of the process—that coming up with the menu part—is offline keys. All they can do is maybe have you order something that you should have been able to have, but it’s maybe not the optimal thing. But it still fits for what you could have.
D2P: It this sort of like a two-factor authentication type of process, also, where more than one person has to sign off?
JC: That’s a really good way to put it. If one of those people is offline keys, and one is online, the online person only picks from the list of things that the offline person says are okay.
D2P: What would you say are the technology gaps that this platform is intended to address for automotive manufacturers?
JC: Well, in a nutshell, there are huge potential problems with updates, from the standpoint of security in particular. So, if you go back to the 1990s, people would say all the time, “I have my security covered; I have a firewall.” And, if you say that today, people will laugh at you because they know that firewalls are not this perfect thing—you have to do a lot more for security and defense. And the same thing has happened in the update space, where, someone might say, “Oh, our update security is fine—we just use HTTPS.” Then people will laugh at you in everywhere other than the automotive space and a few IoT spaces. But if you do this in the broader security community or in banks, they’re like, “You’re insane.” Because you have this one server that can be hacked.
So, Uptane fixes that by making it so that you don’t have that design, and it’s using the same security properties and same security architecture—just an automotive take on it—that is used everywhere in the cloud, on servers, on phones, all over the place.
D2P: You mentioned everyone other than in the automotive industry would laugh at you. Why is it not that way with automotive yet?
JC: I think that in a lot of spaces like automotive and healthcare, people have sort of done things a certain way for a long period of time, and there are vendors who want to come in and sell you a product. For lots of reasons, automotive has been pretty susceptible to the vendor pitches, and there’s a lot more reliance [on vendors]. When people don’t have that same in-house expertise, they tend to be more vulnerable to vendors pitching them things. I think it doesn’t necessarily solve the problems that they need to have solved, in the best way.
D2P: When the Uptane platform was released, did you get some good feedback, in terms of security suggestions that you received from people?
JC: From a security standpoint, we didn’t really get a lot of, “Oh my gosh! You did ‘X’!” There was an error in our reference implementation about how we handled one case. It was a situation where, if an attacker was malicious and gave you a very certain type of metadata, then you might not be able to recover to a state where you could download metadata and verify it later. So, if an attacker compromises a server at the OEM, then they might be able to stop you from getting future updates without having to have a technician do something over CAN (controller area network) or some other means, to your device. That was the most serious thing that anyone found, and that wasn’t actually an error in our specification—it was an error in the way that we implemented the specification.
There were a couple really minor things, where it was more of a clarification, but we haven’t actually had an actual “Oh my gosh—there’s a gaping security hole here!” thing with our design. I credit our community. We had 40 or 50 people, from all across automotive, looking at and validating and working with us on this, every step of the way. And so, they helped to catch a lot of the things that we didn’t understand about automotive, and we were able to help guide things to make sure that there weren’t any security errors sneaking in.
D2P: One of Uptane’s features is that it makes possible remote updates to some of the electronic control units (ECUs) that control different functions of the vehicle. Is that right?
JC: Absolutely. In fact, I’d say that’s the main function, is to do that. You can update, really, any ECUs in the car that support Uptane. Even if an attacker controls your telematics unit or something, in the car, that’s also part of Uptane’s security design. It effectively causes you to still have the maximum amount of security possible, even in scenarios like that.
D2P: What are some of the functions—brakes, locks—that can be updated?
JC: Really, any ECU can be updated. There are two versions of the Uptane code. There’s a weak version, meant for very computationally weak devices, called partial verification. And then there’s a stronger version meant for a device that can do, maybe, four or five signature checks, and can store, maybe, 100 kilobytes or so of data. And that is full verification. So, any device that can do those can be updated. It doesn’t matter what it is.
D2P: So far, how has it been received by automakers? Are there any obstacles to its implementation that have cropped up?
JC: It’s been received very well. One thing that we’ve heard that didn’t surprise me, but I think it surprised them, is once they start to do the integration, it’s much easier than they think. Just as, if I give you the specification for TLS, and tell you, “You need to integrate this,” you’re going to say, “Oh my gosh, this is crazy!” But in some cases, you can just change a line in your code from HTTP to HTTPS, and, depending on your library, that may be the only change you need to make to switch over.
So, it’s one of those situations where, once you have the software, it’s actually not that hard and it’s not that operationally burdensome to run. It’s just people doing the same sorts of things they do today, but maybe they’re signing things that they weren’t signing. They were doing a step where they copied something, and now they sign and copy.
One thing that did come up, though, was people realized that there was a need for being able to do things like geographically founded updates. And so, we’re actually currently working now to add that to an upcoming version of our specification.
D2P: I understand the project began with consultation with regulators and industry. Is that type of collaboration still a part of what you do?
JC: Yes. In the past, we’ve had quite good attendance from regulators and government folks. They tend not to go in and edit our specifications and things as you might expect they would. But they are very engaged in the meetings and are very much following the community. And I think we’ve had quite a few places where people have really pushed us, pushed what we we’re doing, and said, “You need to have a thing like Uptane.” And really what that means is, you have to have Uptane.
Actually, very early on, we started to say to some of the other vendors in this space that have their own product, “Look, all we care about is security. Please, come in, build a version of this, sell it to all your users, make a ton of money.” But I think they were trying to do a more conventional way of just protecting their market share, so now they’re kind of on the outside of it. But we want everybody, really, to be on our side.
By the way, I make zero dollars off of Uptane. It’s something that I’m doing because I think it’s going to cost people’s lives if [security vulnerabilities] aren’t fixed. So, I would love nothing more than for all the vendors to come in and say, “Yes, this is what we’re going to use; it’s going to be secure.” And even if they just take it over from me, I’m just happy to go work on something else. I’ll go work on medical devices and try to fix things in that domain, or the power grid.
D2P: Are there any other major improvements you can point to so far, in terms of what it has enabled automakers to achieve?
JC: I think it’s just so much better in terms of security. And, unfortunately, security is one of those things that, until something goes wrong, you often [don’t notice it]. It’s like insurance: You’re not super excited about having to pay for car insurance until you have an accident, and then you’re really glad that you have car insurance.
Here, this is difficult because I can say that in the cloud space, there have been incidents where people have been very happy that they’ve had TUF, and have talked about it. But in the automotive space, it’s not something where people are seeing immediate benefits. Now, some of the products that use Uptane are excellent and have great campaign management. They’re able to go and push updates out to fleets of cars, and are able to add a lot of value that way, tracking what’s happening in your vehicles. That has been a big benefit, and it’s been kind of an auxiliary benefit: They probably wouldn’t have done that without Uptane, but Uptane directly didn’t help it happen—it isn’t the reason why it happened.
D2P: The reason for open sourcing is to move the industry forward and ensure security for the entire industry. Is that something that you see automakers embracing right now?
JC: I’ve certainly really tried to push that. You’re not going to be putting up billboards saying, “We had 32 percent fewer cybersecurity fatalities than this other car company.” So, this should be a global sport; this should be something that we work together to fix. And I think that resonates with them.
We have taken no money from the automotive industry. It’s been entirely funded by the Department of Homeland Security, although that funding has run out. And now, we’re kind of in a pickle because it ran out. But that aside, we didn’t want this to seem like a Ford thing or a GM thing or a Toyota thing. We wanted it to be a community thing.
D2P: Justin, is there anything that I should have asked you that you believe is important to discuss?
JC: I would say that I would love to see more designs in the automotive space be open. If you believe your design is secure, then, by opening it up, you really have nothing to fear. And it’s very common practice to do this in the rest of the tech world.
The thing is, it’s not like the nation state actors that are going to attack your car don’t know that they’re there—they have all that information. So, the only people it really harms [if you don’t open-source] are the consumers, because they don’t know that they’re buying a defective product. And it makes it so that security experts and researchers can’t help those companies fix their product.
So, please, do what we did—put everything out in the open. If you don’t make your code open, that’s fine. If you don’t want to do that, that’s okay. I understand that aspect of it, but at least the design should be out there so that people can do an assessment.
I think efforts like Automotive Grade Linux, which, by the way, Uptane is a part of, are really key. So, I would love to see more of that happen. Because the rest of the tech community has realized for many years that you get such a big advantage out of at least having open designs and APIs, if not open implementations. So, ideally, the automotive industry learns from the tech community’s mistakes sooner rather than later.