Documentation: FRD, combined SRD/SDD, Software Test Plan, combined User Guide/Maintenance Manual, totaling 7,000 pages.
System Size: 803,000 lines of new code, written by the team in Cþþ, Java, TCL. 400,000 lines of legacy code also integrated into the system.
Cost/Line of Code:
$9.65 per line of code (new and legacy code, at different rates).
Total Cost: $8,773,643 at software delivery.
14.4.1 The Jason 1 TCCS System
The JTCCS subsystem was nominated to represent Jet Propulsion Laboratory at the NASA Software of the Year Competition, in Fairmont, West Virginia, for the year 2002. Alas, we did not win. One of the criteria for winning the competition is that the software be used by several users, and not only by the developing project. Try as we did to sell it to other projects, we failed. I attended symposium after symposium, and held demonstrations and briefings lauding the benefits of elegant, robust space- craft ground systems at under $10 per line of code; there was simply no interest.
This is another lesson you the reader should be prepared to accept: No matter how good your systems are or will be in the future, you are doing it for your project and for your colleagues and co-workers only. Don’t be disappointed if others in your peer group across the country have no interest in your product, regardless of how low the development cost, how elegant, open, or extensible the design, how easy the maintenance, or how low the software failure rate; they will never take your product and use it. The conservation of private or public funds is not one of the major concerns to most people. ‘‘The longer the job lasts, the better.’’ That is just a fact.
Every system I have ever built or managed the development of has been unique, even if it served a similar function, such as spacecraft command and control. Jason 1 TCCS is no exception. It is similar in its functional application and architecture to the TOPEX TCCS, but is quite different. Why? The first reason is that the require- ments were very different. The second important difference is that the state of the art in computer hardware and software had advanced dramatically in the time since the TOPEX TCCS implementation was started. There was one additional major fac- tor, and that was the lessons learned over eight years and the experience of the team. One factor was constant, however, and that was my mandating the use of the JPL-STD-D-4000 software development standard.
As I write this book, the Jason 1 satellite, which is a follow-on to the TOPEX- Poseidon satellite, is in orbit and performing its mission nearly flawlessly. This is a tribute to the entire Jason project team, from the project manager on down to every member of the project in CNES, NASA, and JPL.
14.4.2 System Description
The JTCCS is a flight-quality, beyond state-of-the-art (for 2003) Class A telemetry, command, and communications subystem with an equally elegant user interface to match. Truly a great system, it is the culmination of 17 years of experience in
CASE STUDY FOUR: THE JASON 1 TCCS SYSTEM (JTCCS) 247
building command and control systems. It is by virtue of the systems requirements a high technology, very low-cost TCCS, designed to support multiple spacecraft. The multiple spacecraft requirement was intended to provide continuous support for four generations of Jason-class satellites over a period of 20 years. It is currently hosted on a DELL Power Edge 400 series computer running Windows/XP, but it was designed and implemented to be platform- and operating system-independent, permitting the seamless changeover to the new platforms and operating systems that will come available over the next 20 years.
The JTCCS flies the Jason 1 satellite through three Low Earth Orbiter Terminals (‘‘Leo-Ts’’) located at Poker Flats, Alaska, Wallops Island, North Carolina, and Aussaguel, France. It comprises 803,000 lines of code, written in Cþþ, Java, and some Macro and SQL. The total time and cost of development was 36 months and $8.8 million.
I received a total of just three JPL billets for the job, so in the main it was accom- plished with Category A contractors, most of whom have worked with me for 17 years. Early in the development, my team was outsourced to the Information Sys- tems Development Support (ISDS) contractor that our division manager had put into place to help the division with the overflow of work. The ISDS contract was originally with AverStar, which then became InfoTech, and finally Titan, Inc.
Throughout all of these transitions, the ISDS Manager worked successfully to make JTCCS a resounding success. I am mentioning this because the relationship between contractor and sponsor can have a dramatic and often crippling impact on the budget, the quality of the project, and the schedule. In our case, the ISDS team was superb in its teamwork, organization, and coordination. The methodology used in this management approach is worth writing a book about and is only discussed in generic terms here.
14.4.3 The Initial Conditions
The Jason 1 TCCS project had a reasonable schedule, which started on 14 Decem- ber 1998, with launch scheduled initially for May 2000. There were several reasons we built another separate ground system once again, and did not use the Deep Space Network. The first was that the Earth Sciences program office wanted to operate both TOPEX and Jason 1 on the same system and out of the same facility, thereby making the work of the operators easier. Flying two spacecraft off the same system also reduced the learning curve for operators and reduced the number of operators required to fly the spacecraft. This, of course, reduced overall mission operations cost. The second reason was implementation time. In the field of interplanetary spacecraft ground systems that are very complicated and costly, development time is very long and costly as well. Satellite command and control systems as a rule don’t have large budgets. The third reason was that when you build a unique system, the project manager and ground systems manager have more control over the development effort, and have options available to change the requirements in the ‘‘middle of the stream’’ without impacting delivery, and without having a huge impact on the budget.
The only real constraint was that we were going to fly Jason 1 on the same system with TOPEX. Since we had ported TOPEX for reasons of maintainability and efficiency from the VAX 6410 to DEC Alpha machines running OpenVMS, this was not a problem. We also had no mandated software development standard to follow, so it was up to me. Being familiar with JPL-STD-D-4000, and always preferring to use a standard, I decided to mandate it for the development of the TCCS. The one minor constraint was that since I was allocated so few JPL billets for the work, I would have to do it with contractor personnel, which was fine, since almost all of my computer scientists are contractors, or work-producing consultants.
14.4.4 Implementation Considerations
As a starter, the Jason 1 satellite was completely different from TOPEX-Poseidon.
TOPEX had been U.S.-built by Fairchild Aerospace, and Jason was brand new, with a spacecraft bus called Proteus; it was built by Alcatel, a French company under contract to CNES. This new spacecraft bus was to be a new series of spacecraft for French commercial satellites, and had many new functional requirements and new and different instruments from TOPEX. During a design one wishes to satisfy a number of different design objectives for multiple missions; however, there are vague or unclear requirements for the spacecraft ground system. Anticipating pro- blems, I decided after much discussion with the design team that the common soft- ware services package would have a strong first priority. This was lucky for us. As we began to build our design, we decided that object-oriented design should be incorporated throughout, even though we were already using a tool, EasyCASE, that did not support object-oriented design very easily. We had already done a lot of work in design and programming, and we couldn’t switch to Rational Rose, which I was using on another project of similar size. Keeping the individual design team members happy by accommodating their individual needs to gain experience on new design tools and languages is always a part of my approach when I develop a system. So in the case of the design tool, I didn’t yield, but as far as using Cþþ and Java, I did. This revealed another problem. Although we all liked DEC Alpha and OpenVMS, the software was too slow in execution. The Ground System Man- ager, who was like a full partner, asked me if we could port the system onto DELL Power Edge 450 computers running Windows-NT, and see if this would make a difference. Once the common software service layer was in operation, the porting went smoothly, and was accomplished in one day. The speed of execution went up by a factor of ten! Amazing! And again, this was thanks to the fact that we had many things going in our favor.
We were following a strict design approach using JPL-STD-D-4000. We had a great technical writer, who was using a design tool for structured design. He was able to shoehorn in the objective design we were pursuing, and this allowed us to know where we were at any time with reasonable certainty. This permitted me to make reasonable decisions to save time and money. Having several projects running in a group, all of which follow strict software developments standards, permits
CASE STUDY FOUR: THE JASON 1 TCCS SYSTEM (JTCCS) 249
shifting people from one project to the other without losing track or control of the design and implementation schedule. When the French company announced the delays in the delivery of our spacecraft, I shifted some of my team to another large software project that I was developing for another customer. This saved the budget on Jason 1 TCCS by reducing the burn rate to an acceptable level. You can do these kinds of things if you follow standards, are disciplined, document the work in an orderly fashion, and have an experienced group, with experienced engineers who work well as a team. This enabled some of our engineers to build the software for the satellite instruments and other ground support equipment that were required for advanced testing while the spacecraft was being built.
The intention of using Alpha machines and OpenVMS had been a good one. In this way, we had the computer hardware and the software development tools already available. We were also familiar with the application languages. During the years between 1992 and 1998, object-oriented design had come into its own, along with Cþþ, Java, and such design tools as Rational Rose and EasyCASE.
Furthermore, as we started the design process, my team of technical writers ser- iously objected to using VAX Doc, which we had used on TOPEX and GDSS.
They said that EasyCASE had built-in design advantages, such as the tracing of requirements to design, the numbering of modules, etc. It was a real time-saver.
So I agreed to use EasyCASE.
Now, however, the considerations of software execution speed, new instruments on the spacecraft, the fact that we were not using TDRSS satellites for TCCS, and the delay in spacecraft availability made the project decide that we would not fly the TOPEX and Jason satellites off the same system, but we would build a totally sepa- rate and new system! Had we not used JPL-STD-D-4000, and had we not documen- ted what we were doing with discipline, it would have caused a severe overrunning of our budget. However, all things considered, we were building the most advanced and flexible system ever built for a satellite/spacecraft, within an agreed-to budget.
This fact negates any complaints about ‘‘creeping requirements’’ impacting budgets and schedule.
Indeed, the results of the system were amazing. The only inheritances that were truly constant from GDSS through TOPEX to Jason 1 were:
The methodology we use in building systems.
The design process of improving and using a common software service layer.
Selecting a software standard and tailoring it to suit our budget and schedule.
In this case it was JPL-STD-D-4000, which from the very beginning helped us lay out the JTCCS documentation tree and the contents outline.
Documenting what we do carefully, as a team, through the use of superb technical writers, who had the authority of the systems engineers. Here I reiterate the idea that all projects of any size above 100,000 lines of code, if done efficiently, need at a minimum two systems engineers.
Keeping the team small, for clearer oversight, good control, and facilitated optimal communication and coordination between developers.
Using the design hub methodology to facilitate the design, implementation, and testing process.
14.4.5 The JTCCS Architecture
The technical feature that makes the JTCCS so successful and different from all other systems is its architecture. See Figure 15. What we have achieved is a truly modular and layered architecture that makes the JTCCS totally operating sys- tem- and platform-independent. For the first time, when we started the design pro- cess, the state of the art in computer platforms, operating systems, and design and development tools had reached a level of sophistication that permitted us to actually create our vision and make it happen. The layering provides an abstraction between the application ‘‘plug-ins,’’ the operating environment, and the various COTS packages we use. This enhances the ‘‘plug and play’’ nature of the architecture allowing for the replacement of the sub-layers with little effect on the applications.
For example, it is easy to unplug the Talarian product and replace it with our own IPC mechanism or with any other product without affecting the application code.
The most important component of the JTCCS architecture is the Common Soft- ware Service (CSS) layer. Architecturally viewed, the CSS layer sits on top of the operating system. See Figure 16. The services the operating system provides are platform-specific. The CSS layer provides the identical services in an abstract for- mat separating the operating system from the application programs. The services that are used in a multithreaded, multiprocessing application include
Interprocess communications (IPC) Shared memory functionality Timers and timing functions
Thread and synchronization functions Database access functions
The message database that defines the structure (field names, data types, and length) of messages passed between processes
Process control functions (starts, terminations, suspensions, priorities, etc.) File services functions
Global variable functions Debugging functions
The CSS enables the JTCCS to be ported seamlessly to almost any operating system and platform. It also permits unplugging the support applications and repla- cing them with a newer version without interfering with other applications or the overall operation of the system. Some of these services are abstracted a step further from the operating environment via a COTS package or other implementation layer.
In case of JTCCS, the Talarian SmartSockets package was selected as the COTS solution for providing interprocess communications. Likewise, because we were
CASE STUDY FOUR: THE JASON 1 TCCS SYSTEM (JTCCS) 251
Figure15.TheJasonTCCSsoftware
252
Figure16.TheJasonTCCSpluggablearchitecture.
253
switching between MS Access and MS SQL Server for database chores, we chose as the database interface APIs ODBC for the Cþþprocesses, and the complimen- tary JDBC for the Java processes in order to make the software changes minimal. In the case of the IPC services, it would easily be possible to select another IPC mechanism, such as another COTS package, or an in-house developed solution, and incorporate it with no impact on the application modules layered above the CSS.
Long before a standardized IPC capability was commercially available, my GDSS systems software team built one. This capability later was expanded success- fully into a commercial product named ‘‘SuiteTalk’’ by its original GDSS develo- pers, one of whom used it on TOPEX-Poseidon.
In line with the philosophy of our approach to the design of software architec- ture, the layered and modular structure is physically and functionally distributed and replicated. The availability of TCP/IP has provided us with a refined capability in the way we handled communications.
The server applications are written mainly in Cþþ, with the exception of the Commanding (CM) and User Interface (UI) server portions, which are written in Java. The CM and FTP portions of the server side communicate directly with the Earth Terminal Simulator via TCP/IP. The UI Server communicates directly with the UI Clients using the TCP/IP networking functionality provided in Java. The FTP service communicates directly with the earth terminals to retrieve data stored on the satellite and downlinked during passes, as opposed to the real-time data that are received directly during passes. The servers themselves communicate with each other via the Talarian and IPC packages. Because of the use of a messaging subsystem like Talarian, and because of the well-defined interfaces between the servers, they could easily be distributed across several nodes on a network for per- formance reasons. The CM communicates with the ETs as well as sending the data it receives to CNES in real-time. It is a single process.
The UI Server is in effect the central piece of the JTCCS system. It maintains status information about each of the subsystems and manages all of the communi- cations between the clients and servers.
Figure 17 is a representation of the UI Server. On the left of this representation, the major JTCCS server subsystems are shown; on the right are typical clients. The communications to and from the server subsystems are handled by a common API (a part of Common Software Services) that is layered over the IPC COTS package.
This simplifies the communications mechanism among the server subsystems, including to and from the UI Server. Additionally, the clients and servers use an API set to communicate with each other that has been layered over the Java communication APIs. The client APIs that the server uses are defined to provide state and value information to the various displays that the clients contain. The ser- ver APIs that the clients call request data or provide directives to be communicated to the servers.
For each server subsystem and for other major functions, the UI Server contains various ‘‘data managers.’’ These data managers are responsible for the application- specific functionality that is visible to the clients. For example, from the server side,
Figure17.TheJasonTCCSUIclient/serverarchitecture.
255
the Telemetry Data Manager contains the list of TM mnemonics for the various TM parameters, and the current values of each.
In its final, delivered and operational version, JTCCS represents the best perfor- mance in all the categories of cognitive dynamics addressed in the chapters of this book. It is to be noted, that the experience of the design team members increased over the years from project to project, moving from one success to the next. This had the effect of developing the personal expertise of the team members both as individuals and as an integrated team. It is also to be noted that the Jason 1 project staff and the Jason 1 Ground System (of which JTCCS is a subsystem) worked together with unequalled efficiency. Had it not been for the expertise, good will, and commitment to excellence on every person’s part, JTCCS could never have been delivered at the low cost of $9.65 per line of commented code.20A very spe- cial ‘‘thank you’’ and recognition is due to the team: Mark Fujishin, Don Royer, John Guineau, Sebastian Van Alphen, Dan Zink, Joe Galinsky, Hank Judd, Phil Coffman, and Anil Agrawal.
20The latest and final cost analysis performed by JPL on the JTCCS software assessed the total cost at
$9.65 per commented line of code (LOC).