2002

Company finds right 'formula' for systems management

Jan 4, 2002

The first time Jerry Curtis heard about technology that could tie together Merrill Lynch's myriad of network and systems management tools, he admits that he didn't understand exactly what it was all about. But for some inexplicable reason, he was interested enough to invite the sales rep, who happened to be the company’s cofounder, back for a second pitch.

Though it still didn't click, a nagging feeling then prompted Curtis, director of systems management for Merrill Lynch's $12 billion U.S. Private Client Technology group, to invite the rep to speak at a lunchtime staff educational program.

That's when the lights finally flipped on.

"I didn't know what he was talking about, and I just didn't know why I cared, but I did. I'm sitting in the back [during the educational program] doing paperwork, and in the distance, the little light bulb goes off in my head. I asked some questions, and all of a sudden, the whole group got it. It was just one of those moments where we all started to see it and imagine what this technology could be and the functionality it could provide for monitoring the state of IT," related Curtis, who joined Merrill three years ago as director of enterprise architectural services.

Formula is the linchpin
The technology is Managed Objects’ Formula, a software application that manages Merrill's real-time data on systems availability—from the $3.6B conglomerate's transaction activities to its Web sites and everything in between.

It's the linchpin of Merrill's newly christened Business Service Management (BSM) platform, pulling in data from nearly two dozen monitoring tools and presenting a clean, quick graphical status view for both IT and business users.

The Private Client Technology Group services all 24,000 U.S. brokers and the 750 U.S. branches, as well as the online trading division, Merrill Lynch Direct, and Merrill Lynch Online, the customer service site. The sites service more than 1 billion online clients. The Private Client group consumes about 75 percent of Merrill's computing power, with 5,000 servers and 24,000 desktops in place.

Platform agnostic, Formula is all Java, incorporating XML for filtering data for its desktop user views and CORBA for integration with various platforms, including Windows NT, Solaris, AIX, Linux, and HP-UX. Licenses average $250,000 and do not include maintenance fees, according to Managed Objects, a four-year-old McLean, VA-based company.

While Merrill declined to reveal its license cost, Curtis said it wasn't extraordinary, given his group's infrastructure and user requirements.

The "unknown" problem
Up until the "bulb switched on" in the spring of 2000, Curtis' system administrators grappled with problems the "traditional way": primarily responding to help desk tickets, monitoring hundreds of consoles, and grabbing run books.

"We had all these messages, from all these systems, and they all required that someone actually go touch something, by either remotely logging in to start something or physically walking out to the box and touching it," said Curtis.

It wasn't a matter of not having enough or the "right" monitoring tools or enough system documentation. ML had "beautiful drawings" on systems flow, millions of spreadsheets, run books, and online data documentation—all the things an IT organization needs.

But traipsing through all that data for quick response was a constant hurdle. If a server stumbled, specialists had to search through all the documents and books—a tedious and time-consuming process. But worst, Curtis added, is that while drawings mapped applications to servers, a system's importance to the business side wasn't obvious.

"At Merrill, the number one thing we like to do is generate trades. So knowing what systems are involved in doing trades, what systems are involved in customer relationship management, what systems are involved in answering the most common questions that customers have—that's really the most important thing to me," he said.

How it works
Formula essentially takes all the system drawings and documentation and creates a graphical online approach that lets IT not only react immediately but also respond on a prioritized basis through quick access to its two dozen tools. Business users can quickly view pertinent apps, such as online trading systems and the status of Merrill's Exchange mail application.

That on-demand business view is key to guaranteeing Merrill's longtime customer pledge to provide the lowest transaction price within two minutes of an order, even if a system is down.

"I have a living, breathing business-service view that connects everything together. It's capturing the most important things I care about today and tells me that this system here is used by these applications," said Curtis.

Those "most important things" reflect the top 10 concerns Curtis gleans from internal business users on a regular basis. The top three are usually constant: wealth management services, private client management, and online benefit services provided to corporate customers.

By polling business concerns, staff can target problems and service issues in a very deliberate "more bang for the buck" approach, which is critical because 20 percent of Merrill's systems provide 80 percent of value to the brokerage, Curtis added.

Each month, Curtis' team highlights the worst 3 percent of systems problems and brings those applications back to the development team for fixing. The 3 percent target, explained Curtis, lets his staff focus and accomplish a realistic workload goal.

"If you can come up with a way to resolve the worst 3 percent that's generating the highest amount of heartache and the highest amount of labor and outages, then we're actually able to have a pretty dramatic impact on how the environment works," he said.

"System administrators are able to work down the number of calls they receive to make it more automated. The role they had, which was running as fast as they could just to respond to every problem, has become much more proactive."

Curtis admitted he wasn't looking for a BSM solution when Managed Objects came knocking. But when he, and his IT peers, realized the possibilities the technology provides, they began searching for other possible solutions. However, the list was short at the time, as they could find only one other contender.

The right solution
Before initiating a pilot with the two possible products, Curtis and his peers created a list of desired features. The product had to be modular enough so that each of Merrill's four major revenue streams could use it, but it had to tie everybody together. It also had to provide a tight security model yet be flexible enough to provide access when needed so that the groups could step into each other's systems at critical points.

"We're now starting to see a few products incorporating these needs come out, but at the time we looked, there weren't many options that could work with our distributed nature, business diversity, and location diversity," said the IT director.

Curtis, who declined to name the second contender, asked the two vendors to participate in a worldwide pilot last fall. The companies were instructed to hook their solutions to two primary business applications, including the online trading system.

Within 24 hours, Formula was plugged in and providing both business and IT views. The other vendor struggled for two weeks before Curtis ended the pilot.

Although the pilot went smoothly, Merrill has hit some hurdles in full implementation. While Managed Objects had application adapters for 99 percent of Merrill's management tools, it didn't have one for the NetIQ app, and development of the adapter took several months.

In the meantime, Curtis decided to conduct a full-blown test, predicting there would be 200,000 to 300,000 objects—or characteristics of each machine being monitored—to be loaded into Formula. When the team flipped Formula on, it quickly became apparent that the prediction was quite a bit short. The program hit 350,000 objects and then crashed. As Curtis related, a staff member had to physically turn the machine's key because the app wouldn't respond to any commands.

Realizing it needed more power and eager to see "how big we really were," Curtis opened the firewall, installed Formula on a backup Sun server, and repeated the experiment. The Private Client group's primary and secondary Formula systems are mirror duplicates: Each features two Sun E4500s, with quad processors and 4 gigabytes of memory. Layered on top are two four-processor NT machines, each with 4 K of memory, where users connect.

Formula was scaled to support 1 million objects in 2001 and 2 million worldwide this year.

The last, and continuing, integration issue has involved customizing the user views, specifically for the business side. The first time around, Curtis explained, he provided users with too much detail—which prompted lots of calls.

"I realized I had to be much more granular in how I presented information. I couldn't do fast and dirty. Yet I couldn't be very specific or I would generate a lot of false warnings, which meant no one would trust us," said Curtis.

Reaching for maximum potential
Clearly satisfied with the new technology, Curtis is eager to have Formula play a role in identifying the worst 3 percent of system problems and then aid in troubleshooting.

"Formula has the best real-time distributed view of my business, but what they don't have is good integrated history in terms of a business process analyzer for finding that worst 3 percent. That's important, as this is really the environment to get that information," he explained.

Users are clearly excited about Formula, and an increasing crowd is pounding on Curtis' door for access. As of July 2001, 500 users were hooked up and using the application. When the full rollout over the intranet occurs sometime this year, that number will jump to 19,000 retail brokers, 10,000 assistants, and thousands of investors. It will also be accessible on branch-office workstations.

"Our IT leadership realizes that our customers' faces [are] up against the glass [with online trading] and that the customer is going to know when we're down the same time we know, so why not give the same system capability to everyone?" Curtis said.

Curtis predicted that the new technology capabilities and revamped systems management approach will save $1.7 million annually in systems and network management costs starting this year. Formula, along with automation, reducing redundant systems (such as standardizing on one paging system), and retiring dinosaur technologies like MAX/E, has trimmed staff by 28 percent and saved $673,000 in IT labor costs.

Don’t forget intangible benefits
While the savings aren't minor, analysts believe BSM's intangible benefits are much greater. Customer attraction and retention, the ability to add new business service quickly without excessive development time, and the ability to develop and provide BSM data to customers as a value-add product not only add to an enterprise's revenue stream but also are key in earning a competitive advantage, according to a recent Hurwitz Group report.

"The real value in system management is delivering service to our customers so they want to keep doing business with us or [so they'll want to] switch from other people because they don't get good service [there]. I think that the work our team has done has allowed us to deliver reliable, stable service to customers and to do that in a way that provides profits to our shareholders," said Curtis.

http://articles.techrepublic.com.com/5100-10878-5034113.html