Deptula: I hear a lot of talk from the Air Force nowadays about rapid software development being the next big thing and hear terms like agile software development. And devsecops now in the F-35 has moved to C2D2, an agile software development framework called continuous capability development and delivery. Can you summarize just what this C2D2 is and how it fits into the larger integration and innovation discussion?
Benitez: Absolutely, I'm going to spend a little bit more time on this because it's incredibly, incredibly complex and probably not all the audience is really into software development.
Deptula: I'm not either so as I'll tell you, you don't have to build a watch just describe the watch.
Benitez: That's right so what I'll say is that it's the future, it's valid, it's where we need to go, but I also tell you that we struggle a bit with some of the processes, the equipment to pull it off and honestly a little bit of the culture, but we'll get into some of those things. So what I want you to think about for everyone is, here I have an iphone. Okay there's three types of software on this phone, the first is like your your phone the camera or the gyro inside the phone has firmware that kind of tells what the sensor what to do. The other part of it is the iOS when you turn your phone on it manages your battery your wi-fi, that is an aircraft operational flight program and then the last thing are the apps that ride inside the iOS and so those apps are the mission data.
So mission data is our electronic warfare group, the firmware is vendor ip, so we don't see some of the firmware of that software type stuff. So what we're talking about when we say agile software development for aircraft, we're talking about the operational flight program that's the iOS for the phone.
So agile is a framework and to dumb that down to something that i can understand, think about it as just a faster OODA loop. So the legacy software development is a really really big slow loop, agile is faster. So think about it like running around a track and so it's not faster because the person running around the track is speeding up it's faster because the track is shorter and so he doesn't have to run a quarter mile he only has to run an eighth mile. So because it's a shorter loop it's more responsive to changing needs so when we have emerging requirements we can actually integrate them much easier and then because you are doing rapid software development you in theory have less big changes and because you have less big changes you have less big issues so that's kind of the overall kind of concept of what we're trying to go.
Now to tell you the story for F-35 C2D2 i'll use an example of the F-15, just so you can see the difference between legacy and kind of where everyone's trying to go.
So the F-15 suite 9.0 is fielding to the CAF (Combat Air Force) right now and that is hardware and software. We in the 53rd wing, we're actually wrapping up suite 9.1, which is 95% software and one small hardware upgrade to the radar chip that unlocks some new capabilities. But in the end we're on the 14th test tape revision of that test software over the course of 18 months, and we got to do it to get it right honestly, and by that i mean it has the stability, it has a performance, and it has the PVI pilot vehicle interface so when i push the buttons I know what's going to happen and it's intuitive. None of these things come with a big instruction manual and you shouldn't need one, like your smartphone it should be very intuitive. So roughly one year before that 18-month cycle we actually start the requirements working group and then the contracting mechanisms to lay in how long, how much effort, and what cost is it going to take to do this. So by the time that software hits the war fighter that OODA loop is three years from concept to combat.
For software it's relatively rigid and so in that three year time if there's something else that comes up, we some flexibility to insert new capabilities, but not really, so that's not how to win, and i don't think i need to explain that anymore. That's why every platform is looking to move off of it including F-15, they have a thing called CDNI.
For F-35 we have C2D2, it's the furthest ahead of all the platforms, so it's why we like to talk about it for a few reasons. So it's agile software development, almost, there's one alibi and there's one big asterisk that i'll get to. It’s very important that the audience understands, but let me unpack it.
Alright so C2D2 takes that huge cycle that we just talked about and it breaks it up into four quarters, think like a football game, I’ve got four quarters and each quarter is six weeks long. So I have the first quarter of six weeks, the second quarter, the third quarter, the fourth quarter and after the fourth quarter the game's over and that OFP fields to the CAF. So what that translates into is we have six week test cadences for tape updates and then every six months the CAF gets an update to their aircraft. So that's kind of how it works.
The first obvious question is why can't you just go faster and get an update 10 times a month on my iphone. Why can't you go faster? So i'll tell you, there's a little bit of history here.
So we moved to C2D2 tape three for the F-35 back in the middle of 2019. It proved the concept and everyone decided, yes this is what we need to be doing. However, when we got into tape four we ended up running into some process breakdowns and it really came down to a mentality.
So historically, remember I told you earlier we don't have a mission support group. We don't have a maintenance group either, so we rely on host unit maintainers and those are training bases, operational bases and the reason we do that in the past is because we like to have operationally representative maintenance, so that when we field something to the fleet we want to make sure that the maintainers in the fleet can actually do it the same way, versus hey this is not executable for the airmen on the flight line but the 30-year experience contractor can do it just fine.
So we don't want that disconnect, but what it hasn't been able to do is maintenance to support operational tests, and so it sounds like a nuance. But it is a huge difference, and because we've used host training and ops maintainers they don't have the same authorities as our dedicated developmental test maintenance, where they have what we call redline tech orders and they can actually load things.
So we need new approvals, and so what ends up happening is that out of that six week cycle every six weeks we would lose two weeks just doing paperwork to get approval to squirt the jet with the software so they can test.
So in reality we ended with four weeks of tests, four weeks of tests, four weeks of tests, four weeks of tests.
All these policies are self-imposed, I’d just like to say it's not a contractor thing, it is a service self-imposed bureaucracy. But that lack of test time led to some serious catches at the 11th hour. At the end of that fourth quarter that i told you about, one of them that i can share, is that we were about to field a tape to the F-35 but the radar literally did not work and it went all the way to the end of test until we realized the radar does not work. It was transmitting, but it was not actually receiving and processing the signals correctly due to a software glitch.
So what happened was we actually didn't field tape four, instead, we went to overtime. So now we're in football over time past the fourth quarter, we did an emergency tape to fix some of those issues and then we fielded it and that's the software that are in the Air Force F-35s today. Then we went to tape five, long story short, tape five didn't field either and that was due to some of the same problems that we talked about with software loading authorities and really it came down to stability issues with the ofp. So even if we had a more stable ofp, even if we had less time to test, it might not have been an issue, at the end of the day the confluence of two of those factors together meant that we did not field tape five.
So now we're on tape six, I will say that we do have the authorities to load the software, it took us way longer than I'd like to admit to get a signature on a piece of paper to do that, but we have it, so p6, which is what we call production tape six, is going to field to the CAF next month.
Now here is the the beauty of it, due to the C2D2 construct, all of the capabilities that we did in tape five, that didn't field, we’ve rolled that into tape six and so next month when we start loading operational F-35s, they're going to get tape 5 and tape 6 together. So it's going to be a huge jump in capability, so the F-35s you're going to see this summer are not the F-35s you saw last summer.
All right, so two big things to talk about, the first one is an alibi I kind of highlighted a little bit, so this is a software cycle and does not include hardware. Hardware changes this, right now, in block 4 F-35 it's at least software and so that's easy. F-35 next, when we get into the mid-20s, when this closes out we probably are going to have to do something different. F-22 is seeing it right now with Racer, that's their agile software. They do eight weeks and eight months is kind of their cadence but they have hardware baked into it and so they're running into a lot of issues.
Now the asterisk for agile software, for this, is not exactly agile and i'll go back to the phone. So when your phone has an issue it actually sends a report back to the developer. Whether it's the app, the software, and they're tracking the health of the fleet for your iOS, and that's how they know what to fix for the updates and then they push it. We don't actually have feedback for the code that we deployed to the operational fleet and so when we do have a test escape, that's what we call it, we don't actually have the loop in the OODA loop to know that it's an escape and actually collect that data near real time. It's a very manual process.
We don't actually have more aircraft, there's no money to buy more aircraft to do more regression testing, that's when we go and look for errors, and we don't have any more money for manpower to operate and sustain them. So what we've actually come up with is what we call crowdsourced flight data. Now you may have heard a little bit of it, it's basically three game changers in one and let me unpack that a little bit.
So the first one is it's the CAF as a sensor, and so we're turning all operational aircraft into sensors and now we can monitor all of our software that's deployed on them and respond to issues as they arise. So we can instantly look at the data, we can see what's wrong and we can push a fix and roll it into the next C2D2 tape.
The second game changer is, it opens the door for what we call ops recce, so we can do signal capture and analysis for rapid reprogramming which is really what our electronic warfare group is is interested in, and then the last thing for game changers is, by using crowd source flight data, is big data analytics.
I'll give you a couple examples just to expand people's mind a little bit, of what we're talking about. I'm not going to use a test example, I'll use a training example, so if you remember a few years ago we had an F-22 out at Fallon that had rotated too early on takeoff and they skidded down the runway.
During the accident investigation they noticed a trend, a cultural trend, in the community where, because the F-22 had so much power, pilots were rotating earlier and earlier deviating from some of the the tech order procedures. That wasn't caught until after this mishap happened and they went back and looked at tapes of people taking off and landing.
So with this kind of mechanism, you can actually push a button and actually analyze everyone's takeoff and landing data every day. You can analyze trends, you can score it and so you think about accelerating pilot training accelerating tactics development. You know I can tell you, you could develop the algorithms to assess people's brake turns at BFM, so you can automate a lot of these things that are very very time consuming and manual right now. So that's kind of a training example but, and I'm telling you this the CAF (Combat Air Force) as a sensor, so crowdsourced flight data, it's not a powerpoint, we can do it, we're actually doing it now, we've been doing it collectively for about four years and we have about four thousand hours logged on F-35s in our wing with it.
So it is the game changer that the Air Force needs, it's still not a program of record, we're working on that and we're struggling right now to get the requirements written and the budget kind of closed out for that so we're we're pedaling hard on that but that's kind of how agile software development, what it needs, to actually close that loop to make it effective.