At the theaters on Broadway, they say that "the show must go on." But if you drive a few miles further south on Broadway, you hit a part of Manhattan where that statement is even more applicable — the Financial District, home to Wall Street and the New York Stock Exchange. The organizations there take extraordinary steps to ensure nothing stops. When last year's August 14th blackout shut down the rest of the city, trading continued without a glitch.

For any such critical systems, secure backup is essential, of course. Either the backup software does its job or it gets replaced with something better. That’s exactly what occurred at NYFIX, where homegrown systems were replaced with modern backup software.

"Nothing was being backed up properly," says Sean Lentner, systems analyst for NYFIX, Inc. "It was kludgy and a management headache. We had no choice but to find a better solution"


Error Free Trading

NYFIX is one of the companies that works behind the scenes to ensure error-free financial trading operations. Headquartered in Stamford, CT, it provides brokers with trading floor hardware, electronic trading systems, industry-wide connections and routing, and processing of financial transactions. It bases its services on the FIX (Financial Information eXchange) protocol — hence the name NYFIX. On any given day, traders route orders on hundreds of millions of shares of stock through the NYFIX Exchange.

The company has servers at three sites. Its two primary sites each contain about two hundred servers running a mix of Sun Microsystems, Inc.'s Solaris, Microsoft Corporation's Windows, and Linux. Oracle is the main database, although there are also some instances of SQL Server. NYFIX also maintains a development site.

Now, one of the basic principles in systems design is to eliminate any single point of failure. So, while companies use RAID, mirroring, and other technologies to provide data redundancy in the event of a disk failure, relying too much on a single employee's knowledge can also be disastrous.

In the case of NYFIX, the backup system depended on just one person, the one who wrote its code. This consisted mainly of mySQL systems for the Solaris servers and command line UNIX utilities written in PERL.

"The guy who wrote it understood it well and could control it," says Lentner. "But others didn't understand it, and it’s never good to have one guy you have to depend on in IT."

Also, as is true with most homegrown software, it may deal adequately with its one major function, but it lacks the full range of features present in commercial software.

"The holes were mostly in reporting, plus it didn't scale well," Lentner continues. "If a new application came out or we developed a new product version, he had to write new code into the scripts."

Page 2: The "Fix"