Boeing to Review 1 Million Lines of Starliner Code After Second OFT Software Issue Found

At least two software issues were uncovered with the Boeing software during OFT-1. Boeing had previously discussed the mission event timer issue that prevented the Starliner from reaching the ISS. Today, they discussed a second issue that, had it not been corrected, could have caused the service module to bump into the crew capsule during the service module's disposal burn. This could have caused the crew module to become unstable and start to tumble.

John Mulholland, vice president and program manager, Boeing’s CST-100 Starliner Program described a fairly normal software development process including "requirements definition, peer review of code, unit testing, integration tests, to test all the different parameters and then formal qualification test. There are a number of checks along the way that are designed to uncover and correct code errors as early as you can. We are looking at those workstreams and identifying ways to make those more robust."

However, Douglas Loverro, associate administrator, NASA’s Human Exploration and Operations Mission Directorate countered with : "In the interest of full transparency, that process the John just went through is a very, very standard software process. What our Independent Review Team (IRT) found is that we had breakdowns in multiple areas in that process. For each of these two problems that we know about, some of that breakdown was in different spots and some of that was in the same spot of the process. So we had multiple breakdown in that process. The IRT characterized it as 'here is where that error occurred - as an example, for the mission timer it is how we specified the requirements and the nature of how we coded the requirements - and here is the three or four places along the way that you should have found it'. That's how it ended up getting into flight. I want to make sure that we give you that full insight. The process broke down in many areas for each of these things. That's one of the reasons why we have to go back and do such a thorough review of all the software."

Jim Chilton, senior vice president, Boeing Space and Launch: "The spacecraft has approximately 1 million lines of code. On the OFT mission, we exercised about 66% of the scripts. In partnership with NASA, we are going back to verify all of the code."

Kathy Lueders, program manager, NASA’s Commercial Crew Program : "For the mission event timer issue, we were not able to make the orbital insertion burn because there was an issue with the way a software requirement had been implemented."

John Mulholland, vice president and program manager, Boeing’s CST-100 Starliner Program: "The Mission Elapsed Timer is to allow the propagation of that time and to have the orbital insertion burn happen at the right time. The requirement is for the time to be pulled both before terminal count and during terminal count. If you pull the time early, the time is incorrect. We pulled the time before the initiation of terminal count, which pulled an 11-hour mismatch between the correct time and the time we pulled. This caused the Starliner to begin a series of thruster pulses." To make matters worse, the ground-to-space communication was delayed. "the Space-to-Ground communication that allows the ground to talk to the spacecraft over the TDRS network. Over a certain geographic area, there was a high noise level (Earth-generated noise, possibly from cell phone towers), and we could not establish the link for several minutes. The IRT has not finished the analysis and testing to give specific recommendations for this." 

Jim Chilton, senior vice president, Boeing Space and Launch: "This Orbital Flight Test taught us a lot. After the first software problem with reaching into the Atlas for a clock setting, we went looking for other software errors with integration between two systems. We found one because we went looking. After the crew module separates from the service module, the service module needs to turn into a spacecraft and conduct a disposal burn. The thruster burn sequences is different before separation and after. However, the service module would have used the thruster burn sequence from before separation. Had this happened, it is possible that the service module would have bumped into the crew module after separation. Starliner launched Friday morning and landed 48 hours after launch on Sunday morning. The problem with the disposal burn was discovered late  Saturday evening. That team was able to discover that problem late Saturday night, changed the code, re-verify the code, and we uploaded it approximately 5 in the morning on Sunday.  This was just a few hours before Starliner landed at White Sands at 7:48 AM eastern time Sunday morning." 

Douglas Loverro: "You don't do flight tests to verify that you solved problems. You do flight tests to get a holistic view of a system. It is too early to say if we will want another flight test. What we have to do, as Jim Chilton said, is we have to identify all of the process errors we have. Get a work plan. Go back and make sure that we can find and fix all the process errors. Then look at the completeness of that. Once we have that done, we will do testing in our simulators. We will do testing in all sorts of other places to verify the code. We will do much more verification that we did the first time. Then we will make a decision based on what we will find. We don't know how many software errors we have. Do we have just two or do we have many hundred? So, based on what we find and based on our assessment, we will make a call then. But we can't make that call now because we have a lot of work to do before we can do that." 

NASA Administrator Jim Bridenstine : "We have an independent review team that is still in the middle of an investigation. That will be complete at the end of February. In the interest of transparency, I wanted to make sure that everyone knows where we are. We will continue to learn more in the coming weeks. As a nation, we have a need for multiple providers who can get us to low Earth orbit." Bridenstine was not ready to predict if a second uncrewed flight test would be required. 

---

Boeing Starliner Rollout Scott Schilke

November 21, 2019

Scott Schilke

Open in Gallery

---

Boeing Starliner EFT-1 Rollout (Bill Jelen)

November 21, 2019

Bill Jelen

Open in Gallery

Early on November 21, 2019, the first Boeing Starliner CST-100 capsule rolled out to head to the launch pad.