Unix at 50: How the OS that powered smartphones started from failure, 29 August 2019

posted Aug 29, 2019, 1:58 PM by Jeff Ogden   [ updated Aug 29, 2019, 1:58 PM ]
An interesting article by Richard Jensen on Ars Technica

Unix at 50: How the OS that powered smartphones started from failure

Today, Unix powers iOS and Android—its legend begins with a gator and a trio of researchers.

Richard Jensen - 8/29/2019, 8:00 AM

Ken Thompson (sitting) and Dennis Ritchie (standing) in front of a PDP-11. Ritchie annotated this press image for Bell Labs as <a href='https://www.bell-labs.com/usr/dmr/www/picture.html'>"an amusing photo,"</a> and he joked that he had much "more luxuriant and darker hair" at the time of the photo than when it appeared in magazines like the March 1999 Scientific American (which, unfortunately, incorrectly swapped IDs for the two).
Enlarge / Ken Thompson (sitting) and Dennis Ritchie (standing) in front of a PDP-11. Ritchie annotated this press image for Bell Labs as "an amusing photo," and he joked that he had much "more luxuriant and darker hair" at the time of the photo than when it appeared in magazines like the March 1999 Scientific American (which, unfortunately, incorrectly swapped IDs for the two).

Maybe its pervasiveness has long obscured its origins. But Unix, the operating system that in one derivative or another powers nearly all smartphones sold worldwide, was born 50 years ago from the failure of an ambitious project that involved titans like Bell Labs, GE, and MIT. Largely the brainchild of a few programmers at Bell Labs, the unlikely story of Unix begins with a meeting on the top floor of an otherwise unremarkable annex at the sprawling Bell Labs complex in Murray Hill, New Jersey.

It was a bright, cold Monday, the last day of March 1969, and the computer sciences department was hosting distinguished guests: Bill Baker, a Bell Labs vice president, and Ed David, the director of research. Baker was about to pull the plug on Multics (a condensed form of MULTiplexed Information and Computing Service), a software project that the computer sciences department had been working on for four years. Multics was two years overdue, way over budget, and functional only in the loosest possible understanding of the term.

For more Unix

In 2019, some of Unix’s founding fathers—like Dennis Ritchie—are no longer with us. So for this retrospective, Ars relied heavily on Princeton University’s tremendous Oral History of Unix project. Science historian Michael S. Mahoney spent a few years tracking down many of the firsthand sources back in 1989, and today the transcripts of those interviews remain available online. And though we were unable to connect with Ken Thompson directly, we did get a kick out of connecting to Bell Labs legends like Brian Kernighan (in addition to contributing to Unix development, Kernighan co-authored the first book on the C programming language with Ritchie).
Trying to put the best spin possible on what was clearly an abject failure, Baker gave a speech in which he claimed that Bell Labs had accomplished everything it was trying to accomplish in Multics and that they no longer needed to work on the project. As Berk Tague, a staffer present at the meeting, later told Princeton University, “Like Vietnam, he declared victory and got out of Multics.”

Within the department, this announcement was hardly unexpected. The programmers were acutely aware of the various issues with both the scope of the project and the computer they had been asked to build it for.

Still, it was something to work on, and as long as Bell Labs was working on Multics, they would also have a $7 million mainframe computer to play around with in their spare time. Dennis Ritchie, one of the programmers working on Multics, later said they all felt some stake in the success of the project, even though they knew the odds of that success were exceedingly remote.

Cancellation of Multics meant the end of the only project that the programmers in the Computer science department had to work on—and it also meant the loss of the only computer in the Computer science department. After the GE 645 mainframe was taken apart and hauled off, the computer science department’s resources were reduced to little more than office supplies and a few terminals.

As Ken Thompson, another programmer working on the project, wryly observed for the Unix Oral History project, “Our personal way of life was going to go much more spartan.”

Luckily for computer enthusiasts, constraint can at times lead to immense creativity. And so the most influential operating system ever written was not funded by venture capitalists, and the people who wrote it didn’t become billionaires because of it. Unix came about because Bell Labs hired smart people and gave them the freedom to amuse themselves, trusting that their projects would be useful more often than not. Before Unix, researchers at Bell Labs had already invented the transistor and the laser, as well as any number of innovations in computer graphics, speech synthesis, and speech recognition.

Make way for Multics

Multics had started off hopefully enough, although even at first glance its goals were a bit vaguely stated and somewhat extravagant.

A collaboration involving GE, MIT, and Bell Labs, Multics was promoted as a project that would turn computing power into something as easy to access as electricity or phone service. Bell Labs researchers would have a jack in their office that would connect their terminal to the Multics mainframe, and they would be able to access—in real time—the mainframe’s entire resources. They would also be able to store files on the mainframe and retrieve them at will.

If all this sounds incredibly trivial, it’s evidence of how important these features rapidly became—even for simple computing tasks. But when Multics was first conceived in the early '60s, file storage was a novelty, and “time sharing” (or the ability for multiple users to share access to a single computer’s resources) had only been done experimentally, not in a production environment with a high number of users.

Computers in the early 1960s ran programs one at a time, one after the other. A researcher at Bell Labs would write a program, convert it into whatever form of input the computer accepted (punch cards, paper tape, or magnetic media for really fancy machines), and drop it off at the computer center. A computer operator would queue up the program, run it, and then deliver the printed results and the original program to the researcher.

If there was a mistake in the code, the hassle of printing out punch cards, taking them down to the computer center, and then waiting around for results was rewarded with a printout saying something like “SYNTAX ERROR.” Perhaps you might also get a line reference or some other possibly helpful information.

As programs became more complicated, this method of debugging code became even more frustrating than it already was. But no company or university, even Bell Labs, was in a position to buy a mainframe for each individual researcher—in 1965, the GE 645 that Bell Labs used to develop Multics cost almost as much as a Boeing 737.

Thus, there was widespread interest in time sharing, which allowed multiple researchers to run programs on the mainframe at the same time, getting results immediately on their remote terminals. With time sharing, the programs weren’t printed off on punch cards, they were written and stored on the mainframe. In theory, researchers could write, edit, and run their programs on the fly and without leaving their offices. Multics was conceived with that goal in mind. It kicked off in 1964 and had an initial delivery deadline of 1967.

MIT, where a primitive time-sharing system called CTSS had already been developed and was in use, would provide the specs, GE would provide the hardware, and GE and Bell Labs would split the programming tasks.

Unix team: Assemble!

One of the first programmers on the Multics project was Rudd Canaday. Rudd had been hired by Bell Labs in 1964, and he’d written a simple time-sharing system for the Nike missile defense program before being moved over to the Multics project in 1966.

The following year, the lab hired Ritchie, a Harvard grad whose father was a Bell Labs alum, and Ken Thompson, a self-described army brat who, by his own admission, spent at least a month actively dodging the Bell Labs recruiter on the Cal-Berkeley campus. Canaday, Ritchie, and Thompson came from rather different starting points but ended up in the same place: the top floor of the Bell Labs complex, working in cramped quarters. They had no air conditioning, but at least they had access to a multi-million-dollar mainframe and fairly lax oversight.

All three researchers were MTS, or Members of the Technical Staff at the lab. This designation carried with it an implicit trust in their ability to find their own projects and to—when left to their own devices—pursue research that would be of some benefit to the Labs’ parent company, AT&T. Luckily, the job also came with a very long leash.

Thompson’s Bell Labs pet

In 2011, Ken Thompson won the Japan Prize, a prestigious scientific award, for his Unix work. And during the acceptance speech, he shared what Vint Cerf would call “one of the funniest stories I ever heard” about the early Bell Labs days: the Bell Labs recruiter out in Berkeley ultimately lured Thompson into an interview by offering a generous travel allowance to pay for a trip from California to New Jersey and back. This was an opportunity to take a long vacation on someone else’s dime, and Thompson set about devising a sort of ‘traveling salesman’ problem for himself, seeing how many of his friends he could visit on the way to and from a job offer he had next to no intention of accepting. One of the friends he visited lived in Pensacola, Florida, and had a bit of a sense of humor.

Eventually, Thompson made it to Murray Hill. To his surprise, he found Bell Labs to his liking and accepted the job offer. And a short while later, he received a box from his friend in Pensacola—a box with a baby alligator inside. Rather than quickly dispatching it, Thompson adopted the gator and kept it in a glass baking dish on the radiator in his office, feeding it bologna and hot dogs from time to time until it escaped. It reappeared some days later in a typing pool on one of the lower floors, standing on its hind legs. Thompson recalled it hissing at a secretary who was standing on her hind legs and screaming. Ultimately, Thompson was allowed to stay on at Bell Labs—but the gator had to go… not before biting a few Nobel Prize winners, at least according to Thompson.

Rounding out the team of programmers and nominally overseeing them was Malcolm Douglas McIlroy. Doug was an MIT grad who had been at Bell Labs since 1958.

Thompson, Ritchie, Canaday, and McIlroy were well aware of the flaws in the specs coming from MIT—by this time several thousand pages in length—and they knew that at least some of their challenges could be blamed on the difficult architecture of the GE 645. Over time, the four of them, Thompson especially, became convinced that if they were given a clean slate, they could come up with a better operating system than Multics.

The right environment

Working on Multics was not a full-time occupation for any of the programmers at Bell Labs, and the GE mainframe they were working with could be used for side projects. Thompson’s operating system was one of them. He’d been working on it that winter and had figured out how to make every terminal connected to the machine print “HELLO” at the same time. But before he could get any further, the Multics project was canceled and the mainframe was packed up and moved out of the computer science department.

Baker and Davis had initially taken away the Multics project without giving McIlroy’s team something new to work on, and this caused a fair bit of apprehension for the programmers on McIllroy’s team. They worried that their positions at Bell Labs would not long survive the demise of Multics.

However, this burgeoning development team happened to be in precisely the right environment for Unix to flourish. Bell Labs, which was funded by a portion of the monthly revenue from nearly every phone line in the United States, was not like other workplaces. Keeping a handful of programmers squirreled away on the top floor of the Murray Hill complex was not going to bankrupt the company. Thompson and co. also had an ideal manager to pursue their curiosity. Sam Morgan, who managed the Computing Science Research Department (which consisted of McIlroy’s programmers and a group of mathematicians), was not going to lean on McIlroy’s team because they suddenly had nothing in particular to work on.

Morgan, an applied mathematician by training, had been promoted to department head in 1967. “I don’t think management was his favorite activity,” Brian Kernighan, who occupied an office directly across the hall from Ken Thompson, told Ars. “But he tried exceptionally hard to do it well. He was kind and tried to treat everyone well and fairly.”

“The management principles here are that you hire bright people and you introduce them to the environment,” Morgan himself recalled for the Unix Oral History project. “You give them general directions as to what sort of thing is wanted, and you give them lots of freedom.” So rather than provide specific direction, Morgan preferred to exercise what he called “selective enthusiasm” to encourage a particular research project, noting, “if you mistakenly discourage or fail to respond to something that later on turns out to be good, if it’s really a strong idea it will come back.”

“He let people do their own thing and never tried to tell anyone what they should be working on,” Kernighan recalled. At the time, Bell Labs also stressed collaboration across disciplines. “Everyone kept their doors open all the time, so if you had a problem, there was an expert nearby and it was fine to walk in and ask for help,” is how Kernighan remembers it.

Still, there was one tiny problem for Thompson and his fellow tinkerers at the moment—nobody had a computer. While lab management had no problem with computers as such, McIlroy’s programmers couldn’t convince their bosses to give them one. Having been burned badly by the Multics fiasco, Davis wasn’t sold on the team’s pitch to give them a new computer so they could continue operating system research and development. From lab management’s perspective, it seemed like Thompson and the rest of the team just wanted to keep working on the Multics project.

Sharing the computing wealth

The computer science department shared the top floor with acoustic and behavioral research, a much larger department headed up by Max Mathews, a pioneer in electronic music, voice synthesis, and voice recognition. The acoustics department, with its obvious application to AT&T’s core business, was decidedly better funded than the computer science department down the hall.

And in a situation seemingly calculated to irritate Ritchie and Thompson—who each already nursed a certain disdain for corporate bureaucracy—the acoustics department had no shortage of computers. In fact, acoustics had more computers than they needed. When that department’s programs grew too complicated to run efficiently on the computers they had, they simply asked labs management for new computers and got them.

Max Mathews plays one of the electronic violins he built in his analog electronics lab at Bell Labs. (Credit: <a href="http://www.seamusonline.org/max.html">Wikimedia / Society for Electro-Acoustic Music in the US</a>
Max Mathews plays one of the electronic violins he built in his analog electronics lab at Bell Labs. (Credit: Wikimedia / Society for Electro-Acoustic Music in the US

Notwithstanding a certain amount of jealousy over funding, there was a fair bit of collaboration between acoustics and computer science in the '60s and '70s, and many Bell Labs innovations in computer technology actually came from the acoustics department. For instance, in the early '60s, Bill Ninke, a researcher in acoustics, had demonstrated a rudimentary graphical user interface with a DEC PDP-7 minicomputer. Acoustics still had that computer, but they weren’t using it and had stuck it somewhere out of the way up on the sixth floor.

And so Thompson, an indefatigable explorer of the labs’ nooks and crannies, finally found that PDP-7 shortly after Davis and Baker cancelled Multics.

With the rest of the team’s help, Thompson bundled up the various pieces of the PDP-7—a machine about the size of a refrigerator, not counting the terminal—moved it into a closet assigned to the acoustics department, and got it up and running. One way or another, they convinced acoustics to provide space for the computer and also to pay for the not infrequent repairs to it out of that department’s budget.

McIlroy’s programmers suddenly had a computer, kind of. So during the summer of 1969, Thompson, Ritchie, and Canaday hashed out the basics of a file manager that would run on the PDP-7. This was no simple task. Batch computing—running programs one after the other—rarely required that a computer be able to permanently store information, and many mainframes did not have any permanent storage device (whether a tape or a hard disk) attached to them. But the time-sharing environment that these programmers had fallen in love with required attached storage. And with multiple users connected to the same computer at the same time, the file manager had to be written well enough to keep one user’s files from being written over another user’s. When a file was read, the output from that file had to be sent to the user that was opening it.

It was a challenge that McIlroy’s team was willing to accept. They had seen the future of computing and wanted to explore it. They knew that Multics was a dead-end, but they had discovered the possibilities opened up by shared development, shared access, and real-time computing. Twenty years later, Ritchie characterized it for Princeton as such: “What we wanted to preserve was not just a good environment in which to do programming, but a system around which a fellowship could form.”

“I was still using batch computing in the late 1960s, with an IBM 7094 at Princeton and a GE 635 at Bell Labs,” Brian Kernighan told Ars. “But I had used CTSS at MIT in the summer of 1966, and that was a revelation about how nice interactive computing was.” Once Unix was up and running, Kernighan shifted over from working on abstract topics to writing programs, eventually collaborating with Ritchie on The C Programming Language, which instantly became the gold standard for programming manuals.

From Bell Labs’ cafeteria to our modern phones

Although the labs didn’t keep a close eye on when its researchers arrived at work—or when they left—Canaday did his best to keep normal business hours that summer. Thompson and Ritchie, however, were a bit more relaxed.

Both of them kept wildly irregular hours. Thompson told the Unix Oral history project he was working on about a 27-hour day at the time, and that put him out of sync with everyone else’s 24-hour day. Ritchie was just a traditional night owl. So the earliest these three developers got together most days was over lunch, and even at that, there were occasions where Canaday found himself calling Thompson and Ritchie at their homes to remind them when the Bell Labs cafeteria closed.

In the cafeteria, the three developers hashed out the fundamentals of the file manager for this new operating system, paying little to no attention to the staff cleaning up the lunch mess around them. They also worked on the system in their offices up in the computer science department. McIlroy, who had the office across the hall from Canaday, remembered them working around a blackboard that summer.

Eventually when they had the file management system more or less fleshed out conceptually, it came time to actually write the code. The trio—all of whom had terrible handwriting—decided to use the Labs’ dictating service. One of them called up a lab extension and dictated the entire code base into a tape recorder. And thus, some unidentified clerical worker or workers soon had the unenviable task of trying to convert that into a typewritten document.

Of course, it was done imperfectly. Among various errors, “inode” came back as “eye node,” but the output was still viewed as a decided improvement over their assorted scribbles.

In August 1969, Thompson’s wife and son went on a three-week vacation to see her family out in Berkeley, and Thompson decided to spend that time writing an assembler, a file editor, and a kernel to manage the PDP-7 processor. This would turn the group’s file manager into a full-fledged operating system. He generously allocated himself one week for each task.

Thompson finished his tasks more or less on schedule. And by September, the computer science department at Bell Labs had an operating system running on a PDP-7—and it wasn’t Multics.

The PDP-7 didn’t have a tape drive or a hard drive at the time—the system was “booted” by feeding a punched paper tape into it. Without an attached drive, the file system they had worked so hard on had to wait. At least they had a functioning multi-user time-sharing environment to play around with.

Still, the team felt this was an accomplishment and christened their operating system “UNICS,” short for UNIplexed Information and Computing System. (At least, that’s the official explanation. According to Multics' history site, multicians.org, the pronunciation, like “eunuchs,” was considered doubly appropriate because the team viewed this new operating system, running on an obsolete hand-me-down computer, as “Multics without any balls.”)

Seventh Edition Unix was an important early release of the Unix operating system from 1979. It was the last Bell Labs release to go widespread before AT&amp;T commercialized Unix. Fun fact: The user ("dmr") and home directory ("/usr/dmr") for Dennis Ritchie are still present.
Enlarge / Seventh Edition Unix was an important early release of the Unix operating system from 1979. It was the last Bell Labs release to go widespread before AT&T commercialized Unix. Fun fact: The user ("dmr") and home directory ("/usr/dmr") for Dennis Ritchie are still present.

That November, Thompson’s self-evaluation—or, as it was known in the computer science department, the “I am great report”—contained a description of this new operating system he had built.

By the summer of 1970, the team had attached a tape drive to the PDP-7, and their blossoming OS also had a growing selection of tools for programmers (several of which persist down to this day). But despite the successes, Thompson, Canaday, and Ritchie were still being rebuffed by labs management in their efforts to get a brand-new computer.

It wasn’t until late 1971 that the computer science department got a truly modern computer. The Unix team had developed several tools designed to automatically format text files for printing over the past year or so. They had done so to simplify the production of documentation for their pet project, but their tools had escaped and were being used by several researchers elsewhere on the top floor. At the same time, the legal department was prepared to spend a fortune on a mainframe program called “AstroText.” Catching wind of this, the Unix crew realized that they could, with only a little effort, upgrade the tools they had written for their own use into something that the legal department could use to prepare patent applications.

The computer science department pitched lab management on the purchase of a DEC PDP-11 for document production purposes, and Max Mathews offered to pay for the machine out of the acoustics department budget. Finally, management gave in and purchased a computer for the Unix team to play with. Eventually, word leaked out about this operating system, and businesses and institutions with PDP-11s began contacting Bell Labs about their new operating system. The Labs made it available for free—requesting only the cost of postage and media from anyone who wanted a copy.

The rest has quite literally made tech history. By the late 1970s, a copy of the operating system found its way out to the University of California at Berkeley, and in the early 1980s, programmers there adapted it to run on PCs. Their version of Unix, the Berkeley Software Distribution (BSD), was picked up by developers at NeXT, the company Steve Jobs founded after leaving Apple in 1985. When Apple purchased NeXT in 1996, BSD became the starting point for OS X and iOS.

The free distribution of Unix stopped in 1984, when the government broke up AT&T and an earlier settlement agreement that prohibited the company from profiting off many Bell Labs inventions expired. The Unix community had become accustomed to free software, however, so upon learning that AT&T would soon be charging for all copies of Unix and would prohibit alterations to the source code, Richard Stallman and others set about re-creating Unix using software that would be distributed to anyone free of charge—with no restrictions on modification. They called their project “GNU,” short for “GNU’s Not Unix.” In 1991, Linus Torvalds, a university student in Helsinki, Finland, used several of the GNU tools to write an operating system kernel that would run on PCs. And his software, eventually called Linux, became the basis of the Android operating system in 2004.

Richard Jensen is a writer and historic preservation specialist in Sioux Falls, South Dakota. Jensen has previously contributed to Ars on subjects like nationwide "right to repair" efforts. He has more library cards than credit cards, and no one is surprised by this.