Fall; or, Dodge in Hell - Neal Stephenson Page 0,140

and candles. Event runners in smart dresses and wearables had directed them to their designated places and they’d all talked their way through a simple dinner of plank-roasted salmon.

But now Enoch—the after-dinner speaker—seemed to be picking up where the USC brain scientist had left off.

“As a young man, the Hanoverian genius and polymath Gottfried Wilhelm Leibniz—in many respects the progenitor of symbolic logic, computation, and cybernetics—was described as a monster by awed colleagues in the intellectual salons of Paris. It was as such that he first became known to the savants of the Royal Society. It is in that sense, and not that of Frankenstein’s monster, that I put it to you that we now have a monster on our hands. We all bear some responsibility for that, but I bear more than many. I have been working for ONE, the Organization for New Eschatology, for seven years. My role has been to advise the founder, Jake Forthrast, on how to deploy the foundation’s resources. He came to me nine months ago with news of a conversation he’d had, over the holidays, with his grandniece. An opportunity had presented itself to support her project, which had reached an impasse. Sophia had gone as far as she could go merely simulating individual neurons, or small clusters of them. The only useful next step was to take it all the way and throw the switch on simulation of an entire brain. She had patiently and assiduously laid the groundwork for doing so. But a full simulation would consume a significant fraction of the computational resources of Hole in the Wall, and that would be expensive. Jake requested that several of us review Sophia’s work and render a judgment, free from nepotism, as to whether the project should be supported. I voted in favor of funding it. Sophia launched the Process with the surprising and fascinating results that we have all been learning of during the preliminary sessions.

“A few months later, the Process seemed to be stuck in something akin to an infinite loop. The results were intriguing, but inconclusive given that the Process lacked the ability to self-modify its connectome. Sophia activated that feature, with dramatic results. A surge in activity drained the Process’s account to the point where it stood in danger of being shut down. I recommended that Jake give it an infusion of funds just to keep it running. He did so. But he has more money than time. He foresaw a long tedious series of such cash flow pinches, each making a claim on his attention. So he gave the Process a lot of money. Which is to say that he placed at its disposal a much larger store of resources: memory, bandwidth, and processing power. As a result—and I think it was an unintended result—the Process is now essentially the only thing running on the Hole in the Wall system, and it has reached out to access computational resources elsewhere.

“Now, this is all very interesting, but it does not yet rise to the level of a monster. We have no way, as yet, to look inside the Process and read its mind, so we can’t tell whether what it’s doing is interesting. Skeptical observers might make the case that its behavior is degenerate, trivial, just a lot of infinite loops burning cycles and wasting money to no purpose. Letting it run out of money—the modern equivalent of what we used to call pulling the plug—would be no different, ethically, from force-quitting a hung app.

“The opposing point of view sees a very real ethical issue. The Process has now been self-modifying for five months. It is consuming vastly more resources than the original system. It seems to be doing so in an organized manner—we can observe, in the burn rate, high-frequency oscillations overlaid on a slower undulation. The Process’s demands for more resources don’t grow steadily but in leaps and bounds. For much of September it was perking along contentedly, but just in the last few days it has put on a surge of activity that seems to require vast computational resources. Tomorrow we will hear from Dr. Cho of the National University of Singapore, who has done some traffic analysis on the encrypted packets connecting Hole in the Wall to other server farms; he develops a hypothesis that this recent activity bears the earmarks not of a neural net, but of a physics simulation. Why is the Process simulating physics, and how did it learn to do