The Ocean Topographic Explorer (TOPEX) Satellite, Telemetry, Command and Communications Subsystem (TCCS), is a Class A System, developed on a very short schedule and low budget.
14.3.1 The TOPEX TCCS System
Software Development Standard: JPL-STD-D-4000 employed throughout.
Documentation: FRD, SRD, SDD-1, User Guide/Software Operators Manual, Test Plan, totaling 13,000 pages.
System Size: 196,169 commented lines of code.
Languages: C, Macro, and Fortran.
Total Cost: $5,400,000.
Cost/Line of Code: $27.50
TOPEX, the Ocean Topographic Explorer Satellite (or Poseidon, as the French call it), was decommissioned on 18 January 2006. It performed its science mission nearly flawlessly during its 14 years of operation.
14.3.2 System Description
The TOPEX TCCS is a flight-quality telemetry, command, and communications sub- system of the TOPEX ground system, with a NASA communications center monitor- ing function added for the support of the TOPEX project. The TCCS is based on Digital Equipment Corporation VAX 6410 computers running under the VMS oper- ating system. It has the capability of processing data rates of 512 KB/s and has two NASCOM Front End Processors (NFEPs) for data storage and formatting. See Figure 13.
CASE STUDY THREE: THE TOPEX TCCS SYSTEM 233
Figure13.TheTOPEXsoftware
234
The objective of the effort was to complete the TCCS within 27 months, at a cost of under $6.0 million. There was a task contractor in place, Computer Sciences Cor- poration (CSC). This contract had two separate segments, one being a task contract, the other a TSEP, from two separate organizations of CSC.
Although JPL-STD-D-4000 was not mandated by the project, I chose to apply it strictly.
14.3.3 The Initial Conditions
My part of the development was to start in December 1989 and be completed by March 1992. The computer hardware and software had been ordered and was scheduled for delivery in June of 1990, but wasn’t actually received until September 1990, very close to what I had anticipated. Nor were there any facilities for the development available except those of the contractor.
There were three documents available that had been prepared during the preced- ing 24 months: an FRD, an SRD, and an FDD. However, since no standard had been used in their preparation, they were not properly structured, organized, or content- wise sufficiently developed to be used in their existing form.
At the beginning of the project, there had been a number estimated as to how much work would be involved, in the form of source lines of code (SLOC), since I find this approach a good one to use as a means of quantifying work (there are others, but I view the SLOC metric as the best). After the completion of a project, we add the comments to the SLOCs and come up with lines of code (LOC), since most of my senior design engineers and senior programmers feel that commenting takes as much time as programming does. I agree with this, and I cost a project after completion overall on dollars per LOC.
The estimates varied from CSC, at 47,000 SLOC, based upon their experience with the UARS spacecraft. The systems engineer’s estimate was 92,000 SLOC, and my estimate was 130,000 SLOC. (The final total turned out to be 196,169 SLOC.) 14.3.4 Project Constraints
There were a number of constraints. CSC was to be used as the subcontractor because we needed a task contractor. The CSC contractor team was also going to reuse16 the UARS software on TOPEX. The experience the CSC design team gained on UARS was also of great value to the TOPEX project. Also, adequate in-house facilities at JPL were not available, and I was limited to no more then seven JPL billets altogether.
The already available IBM PCs and Macintoshes were to be used for the defini- tion of the requirements and the design.17Also, JPL-STD-D-4000 was to be used
16‘‘Reuse’’ is a very misleading and costly word! Out of 47,000 SLOC of Perkin-Elmer Fortran, we were able to force-fit in 7,000 SLOC. These 7,000 SLOC caused more work to fit in than had we simply reprogrammed them in C. I have had the same experience on every project I have done.
17Requirements and design must be traceable both horizontally and vertically in the system, down to the software module. Also, it is a waste of time and money to use a system for documentation other than the one you are going to use for development.
CASE STUDY THREE: THE TOPEX TCCS SYSTEM 235
‘‘as far as practical.’’18There are some tough lessons to be learned in these three areas, software reuse, using PCs for requirements and design, and using JPL-STD- D-4000 ‘‘as far as practical’’ if cost and budget are limited.
14.3.5 Implementation Considerations
Although it is almost never fully appreciated on projects, there is a glaring lack of computer scientists on all project staffs. Almost all cost overruns and schedule slips are directly traceable to the computer software area. Nothing runs without computer software, not aircraft, not spacecraft, not even automobiles. Yet for some reason, mechanical and electrical engineers, and above all, scientists, have little appreciation for computer science and software. The TOPEX project was no exception.
At first, the TOPEX project manager and staff had an idea of the significance of the tight schedule, but not a full appreciation. Why? The most likely cause is a lack of understanding of the complexity of software; that is, not being able to understand it and therefore ignoring the issues impacting it. This of course is understandable.
We all have a hesitation in trying to understand something with which we are unfamiliar. Wasting 24 months ‘‘thrashing around,’’ then waking up to a 27 months’
distant launch date and getting into gear (not appreciating that I, too, along with every other manager, no matter how good, also have a ‘‘thrashing period’’), is an indicator of the absence of solid planning.
Then there was the idea that the hardware would arrive someday, and the soft- ware would happen as if by magic. Computer hardware and software, and all COTS packages, like compilers and tools, must be available at startup. Staffing up, with its associated ‘‘burn rate,’’ yet without solid work getting done is poor practice and a waste of time and money. It forced me to bypass the bureaucracy and borrow all of the needed equipment, software, and COTS packages on the promise that I would buy it all. There is plenty of danger here from those in the bureaucracy who run the procurement process without much regard for the objective of our existence, which is to be there at the launch date, ready for liftoff and mission operations.
Finally, there is the availability of work areas. Even with plenty of time, work space is important. The work space is often as important as the architecture of the organization. It either enhances or hinders the work getting done. Each project has its own unique requirements, with respect to software technology, hardware, sche- dule, and budget. Cost estimates that do not consider these requirements are flawed.
TOPEX TCCS had office and cubicle space available for individual employees, but no office for me, or for a work ‘‘war room,’’ which the nature of a TCCS requires. The work on the development of a TCCS generally takes about 48 months on the average. On TOPEX, all I had left was 27 months to launch.
18‘‘As far as practical’’ is an incomplete directive that will cause a lot of confusion and waste of money and schedule. What it means in reality is ‘‘everybody does his own thing.’’ You either commit to a standard, or you don’t. When everybody does his own thing, the interface specifications are a nightmare to write, and very costly to connect to!
At times, as the manager at the subsystem level, I was overwhelmed by the sheer weight of the bureaucracy against getting anything done. I had to keep in mind the reason I was given the job in the first place, namely, to get it done, and be there for launch. If one keeps that in mind, everything else falls into place.
I simply took over a medium-sized conference room for my office space, and another large conference room for my war room. This got many people angry, above me, below me, and along side of me. How dare I do such a thing, take over territory! It is easy, as I often have to explain; ‘‘I have a launch date to meet!’’
As I have said, each project has its own unique requirements, thus a ‘‘cookie cutter’’ approach seldom works. One must have many templates and wells of experience to pick the right approach, and then modify it or synthesize it with several other schemata and come up with a new one that works efficiently.
This I did. It worked out fine. There was objection from all hands, but since many of them had worked for me before, they trusted that it would work out and that it would enhance our work and the need for coordination.
14.3.6 Development of TOPEX TCCS
My part of the work in developing the TOPEX TCCS started late one evening at the end of November 1989. I was called in by my division manager for a talk.
(The reason I am going into detail is to show how cumbersome and disjointed projects can become by simple oversights and perspectives of an incomplete object. This is important for anyone desiring to build at low cost. The lessons learned contain many nuances that are rarely told, and therefore the lessons are incomplete.)
Out of the blue, my boss told me that I had to transfer to a different section in our division and complete a satellite telemetry, command, and communications system that was behind schedule and over in funding. He also added that I had no say in the matter, since he saw that I was not enthusiastic about TOPEX.
He told me that the mission of the satellite was to acquire, process, and verify altimeter sea surface height data. Data sets would then be distributed to principal investigators who would map the mean and variable geostrophic surface current of the world’s oceans. The studies would then give indications of the effect of ocean temperatures on global climates.
The work was being done in our division, and my boss said that all there was remaining of the budget was $5.6 million, and the schedule was now down to 27 months to launch. Normally the TOPEX Ground System would have been a part of the services provided by the JPL Deep Space Network and our Space Flight Opera- tions Facility (SFOF); however, because of time constraints and budget considera- tions, and because TOPEX-Poseidon was a partnership between NASA and the French Centre National d’Etudes Spatiales (CNES), it had to be completed on time for the launch and within the budget remaining for the job. As division man- ager, he was responsible for the work in his division, and he intended to get it done.
He had seen my work in building systems fast and at cost over many years, and he decided that he wanted me to do it.
CASE STUDY THREE: THE TOPEX TCCS SYSTEM 237
The spacecraft was being built here in the U.S. by Fairchild Aerospace, a distin- guished American firm whose WWII planes I used to model out of balsa wood as a kid. It seems I was always immursed in modeling.
The TOPEX spacecraft (or better said, satellite, to distinguish it from great interplanetary spacecraft like the Voyagers and Viking) was to be launched from the Colony of French Guiana, from the Launch Facility at Courou, on an Arianne IV-42b Launch Vehicle, so there was an international side to the operation. The cost of this included that of the launch vehicle and the launch pad reservation.
Each extension of the launch pad cost around a million dollars per day, for up to a week’s delay; then you were forced to go to the end of the line. Such costs incurred are very burdensome to a project. At JPL, we have never had a delay in launching due to the ground system not being ready. The TCCS therefore had to be ready.
14.3.7 Agreeing to Do the Job
The TOPEX TCCS is a subsystem of the TOPEX Ground System (TGS) that supports satellite management during flight operations, and geophysical data production and distribution. The TCCS provides the functionality required to capture, process, and display telemetry data, transmit commands to the satellite, communicate with it, receive data from it, and make the data available to the other subsystems within TGS and to agencies external to TGS.
The TGS consists of five subsystems, including TCCS:
Mission Planning (MP)
Scheduling and Sequencing Subsystem (MPSSS) The Navigation Subsystem (NAVS)
The Science Data Subsystem (SDS)
The Satellite/Sensor Performance Analysis Subsystem (SPAS)
If I were to accept the job without asking for more money or schedule, my boss promised me I’d get a promotion to Engineer Level 8, a great incentive to get it done. Senior Research Engineer and Senior Scientist was the only way to make E-8 without becoming a ‘‘stem-winder’’ or line and project manager. Stem-wind- ing was something I was not good at, either in the Army or at JPL.
I agreed to do the job, not that I had a choice, but I was curious. I had never built a Class A system. The telemetry, command, and communications subsystem of a ground system is the only Class A system on the ground, in that a wrong or faulty command error can cause the loss of the spacecraft.
There were many interesting problems associated with this project. The first was that I had never built a spacecraft ground system before. When I was given the job, some of the senior engineers told me that there had been a lot of talk about me hav- ing no experience in spacecraft flight projects, except for being Team Chief of the Voyager General Science Data Team, which was career-wise a dead end at JPL. We,
like any organization, have our power groups and elite areas. With us it has been traditionally the spacecraft engineering folk who were at the top of the list, and admittedly they were the geniuses, because they designed and built interplanetary spacecraft. Once JPL ‘‘outsourced’’ that skill function, things changed. My reply was that as far as I was concerned, computer software is computer software. For someone else with limited experience, however, it may be true that building a spacecraft ground system would be daunting.
If all you have done in your working life is build spacecraft, then that is probably all you know how to do. Computer science and software were not considered cri- tical or important skills at the Laboratory. This bothered some of the guys, but not me. I knew better.
14.3.8 Ground Truth
The ground conditions at start of work are very important, so I must discuss them for the sake of lessons learned.
The ‘‘ground truth,’’ as we call it in the Army, is the real enemy situation, the friendly situation, and the terrain. I use these terms because these three factors are the reasons Americans build software that is so terribly expensive.
The enemies in this case are the well-meaning and well-intentioned people who, because of their subjective mindset, will do everything to stop you from getting the job done. What is getting the job done? It is the accomplishment of the mission: being at the launch pad, with the spacecraft sitting aboard the launch vehi- cle, fueled, and all systems functioning flawlessly, ready for liftoff. That is the accomplishment of the mission at NASA and at JPL. Anything short of that is a poor job.
Now, people will say, as they often do, ‘‘How come you’re so outspoken?’’ To tell it the way I see it is to add emphasis to the theme of success and failure as we did in the Army. Keep in mind that we are engineers, being paid high salaries at taxpayer expense; Kantian and Aristotelian ethics demand that we earn our daily bread by doing the very best we can for the individual who pays our salary, i.e., the American taxpayer. This is a strange concept to many of my colleagues around the country, where managing means a big office, executive furniture, and lots of frequent-flier bonus miles.
14.3.9 Start of Project Development
To start things off, I met with my systems engineer. He was a very nice and smart person, to my good fortune. He introduced himself and told me that he was delighted to have me assume the job of technical manager, and that I would be mak- ing all the programmatic decisions, and he would be making all the technical deci- sions. I replied thatI would be makingall decisions. I know that this was a real downer for him and for me as well, as it has been for most of the people who have worked for me as systems engineers.
CASE STUDY THREE: THE TOPEX TCCS SYSTEM 239
I then took a four-day period to read the FRD, SRD, and SDD that I had been given by my immediate supervisor, who was the division representative for the project. My new section manager and another group supervisor met me in the section manager’s office to give me my instructions, and to get some feedback on my estimate of the work. When asked what I thought of these three all-important documents, I replied that they contained lots of good and important information, but were FRD, SRD, and SDD in name only. They had little structure, followed no standard, and the contents were in poor and ambiguous English.
They got upset and angry with me. I told them that it was okay, I’d fix the sys- tem, after all that is why I was told to do this job. It is a fact of life that if you want to get your work done efficiently, people will resist you and try to stop you. The harder they resist, the more assured you should be that you are on the right track. If they don’t resist, then your approach is their approach, and it will not lead anywhere. You might as well have remained doing the job you had been doing before you arrived.
A 27-month schedule for implementation of a Class A ground system is not long, and $5.6 million is not much for such a job. A two-year period had already passed, and about $2.0 million had been spent with nothing to show for the effort.
This was going to be a very difficult job.
The team was physically separated. The contractor had a facility in town, made up of two separate competing divisions of the same corporation. The only inherited hardware on the JPL side consisted of two Macintoshes, three IBM PCs, and a cou- ple of printers. The selected ground systems hardware, which to my good fortune was Digital Equipment Corporation equipment, had been ordered. They were sup- posed to be delivered in June of 1990; launch was to be in May of 1992. This, of course, meant that I would be in trouble, because the delivery would be at least four months late. Development couldn’t start until the hardware was delivered, and that was not good enough.
There was no development facility for the TCCS, and the although the design team was made up of very intelligent, hardworking, and capable people, unless they were organized properly and had a set of procedures to work with, they would simply thrash and go to meetings. This is a dangerous phenomenon. Com- pounding this problem, the section was busy doing ‘‘the routine’’—leaving my action item requests at the bottom of the in-box.
As luck would have it again, the section manager left, and an interim section manager was put in place to continue supporting the Deep Space Network.
Fortunately, I also had the support of a friend who was the Chief Administrator of the Division; in this way, as in the Army, I could get things done, and untangle the logjams. It is always the initiative, the taking of risks, versus the choice of stay- ing within the bureaucratic rules and not getting the job completed. Yet if I risked getting it done, I would make enemies of those who do not dare, whose name is Legion. There was this terribly clear choice: Was I going to have the TOPEX TCCS ready for my project manager, my division manager, and the customer or not? My project manager had also taken a big risk. He had decided because of the cost and schedule factor to bypass the Deep Space Network Multi-Mission