The Stocks Must Go On
Despite blackouts, disasters and human error, financial systems must keep running. NYFIX met this challenge by migrating to enterprise-class backup systems.
For any such critical systems, of course, secure backup is essential. Either the backup software does its job or it gets replaced with something better. That is what occurred at NYFIX, where homegrown systems were replaced with modern backup software.
"Nothing was being backed up properly," says Sean Lentner, systems analyst for NYFIX, Inc. "It was kludgy and a management headache. We had no choice but to find a better solution"
Error Free Trading
The company has servers at three sites. Its two primary sites each contain about two hundred servers running a mix of Sun Microsystems, Inc.'s (Santa Clara, CA) Solaris, Microsoft Corporation's (Redmond, WA) Windows and Linux. Oracle is the main database, although there are also some instances of SQL Server. NYFIX also maintains a development site.
Now, one of the basic principles in systems design is to eliminate any single point of failure. So, while companies use RAID, mirroring and other technologies to provide data redundancy in the event of a disk failure, relying too much on a single employee's knowledge can also be disastrous.
In the case of NYFIX, the backup system depended on just one person, the one who wrote its code. This consisted mainly of mySQL systems for the Solaris servers and command line UNIX utilities written in PERL.
"The guy who wrote it understood it well and could control it," says Lentner. "But others didn't understand it and it is never good to have one guy you have to depend on in IT."
Also, as is true with most homegrown software, it may deal adequately with its one major function but it lacks the full range of features present in commercial software.
"The holes were mostly in reporting, plus it didn't scale well," Lentner continues. "If a new application came out or we developed a new product version, he had to write new code into the scripts."
Page 2: The "Fix"