Customer Stories

posted on 11 April 2008 15:12

How quickly times change. Just a few short years ago Palomar Pomerado Health (PPH),  the largest public health district in California and a hospital, had a shiny trio of IBM System p650 RISC Unix servers connected to Shark, IBM's Enteprise Storage System ESS. Now those three servers have shrunk to two which do more work and the Shark has become the third level on a 3-tiered storage area network (SAN) behind IBM's virtualizing SAN Volume Controller (SVC).

The three p650 servers and ESS supported supported Cerner production and test functions, including picture archiving and communication system (PACS) image storage.

PPH's facilities include the Palomar Medical Center, Pomerado Hospital, the Jean McLaughlin Women’s Center for Health and Healing, Villa Pomerado, Palomar Continuing Care Center, Escondido Surgery Center, Palomar Pomerado Behavioral Health Services and Palomar Pomerado Home Care. It is the largest employer in Inland North County, with more than 3,000 employees and 700 physicians.

Why did PPH go through this migration and embrace virtualized storage?

PPH knew it needed to go much further into digital hospital management. Paper-based file and record and booking systems just couldn't cope. It was going to add another hospital and move to fully digitised medical records for its patients. A $496 million bond issue had provided the finance for this.

It also wanted to introduce a new and customized computerized physician order entry (CPOE) program, a digital physician bedside care system, in 2009. To do that it needed a better IT infrastructure than the inflexible one it currently had. It needed a storage system which would allow it to migrate files from one storage array to another without shutting down major parts of its IT infrastructure.

Cerner Corporation, an IBM business partner and software supplier to PPH, suggested that it move to more powerful servers and add two tiers of storage alongside the ageing ESS array, and, in a crucial move, virtualise these three pieces of storage into one logical pool behind IBM's SVC. This would both increase the utilization of storage capacity and provide appropriate levels of storage service and cost to different types of data, and also enable files to be moved from one array to another, or from one storage tier to another, without interrupting users' and applications' storage service at all.

PPH agreed and Cerner went ahead and implemented the new system.

Overall scheme

The clinical software applications - the Cerner Millennium suite of clinical software, which includes financials, laboratory systems, surgery scheduling, clinical and pharmacy systems and other processes - were consolidated from the three p650s onto just two p590 servers, still running Unix. These two servers were configured into a high availability cluster so that, if one were to fail, the other could automatically pick up the load.

The applications in the servers 'talked' to storage resources through the SVC. Behind it was a SAN, a single virtual pool of storage divided into thee tiers. The top tier was a DS6800 array; the mid-tier a DS4300 array; and the third tier the original ESS array. The tiers were organised to support diffrent types of data:-

Tier 1: IBM DS6800 (4 TB) provides the highest performance and is used for Cerner and Lawson production data.
Tier 2: IBM DS4300 (14 TB) stores all imaging files.
Tier 3: IBM ESS (15 TB) supports non-production data.

Tivoli Storage Manager is used to archive files to tape.

By helping to eliminate the need to over-provision storage, Engberg expects that the tiered approach can significantly lower the overall cost of storage for PPH. He said: "When making storage assignments, it is easy to find the best match between our storage subsystems and the users’ needs and budget. As new storage devices are added over time, we can change those assignments without disrupting applications.”

What PPH wants to achieve is to store all of its healthcare data—images, operations and financials - on this virtualized storage infrasture.

The project implementation was accomplished in phases. First, the new servers were installed and the Cerner Millennium system software updated. An upgrade to the SAN fabric followed, including the addition of IBM SVC and new IBM DS4300 and DS6800 storage subsystems. Finally, storage tiers were defined and data was migrated from the legacy ESS to the appropriate storage tier.

Transparent data migration

Paul Engberg, PPH's Director of Technical Services, said: “Having SVC in place prior to the data migration helped minimize the disruption to users. Once all the legacy storage was behind SVC, the data could be migrated to the appropriate tier of storage without any disruption to applications.”

Dennis Dechant, PPH system administrator, confirmed this data movement transparency: “I did the entire migration of our Lawson financial applications to the DS6800 while users were online. No one noticed any degradation during the process.”

Dechant also appreciated the help provided by the vendors. “Even though Cerner provided the storage systems, IBM technical people came onsite to help us set up SVC. In the process, they also taught us how to manage SVC on an ongoing basis.”

Engberg said: “IBM’s storage virtualization solution coupled with our IBM UNIX server solution gives us flexibility in managing our growth. We need to save certain health records for decades and storage virtualization will play a role in helping us accomplish much needed archive capabilities at a significantly reduced cost.”

PPH intends to add a sister SAN, mirroring the first, as a disaster recovery facility.

Flexibility increased

A big benefit is the flexibility to cope with new requirements. Engberg said: “We don’t really know what all of those requirements might be; many of those decisions are driven by our clinical staff. So having a flexible storage environment is a big asset for IT—we can now respond quickly to new requirements without having to rent a forklift or throw out existing equipment.”

He knows that the amount of stored data will increase: “Not only are we storing more images every day, but new higher resolution imaging technology will drive up file sizes. SVC gives us much greater flexibility in the way we manage that growth. For example, we have requirements to save images for 21 years. That is driving the need for image archiving capability in the next year. SVC will play a role in helping us accomplish that in the most effective way.”

The ability to move data transparently has a beneficial effect on storage array hardware servicing. According to Dechant, some storage maintenance activities—such as microcode upgrades—previously required taking the storage device offline. Now, with all storage behind SVC: “We can non-disruptively move the files off the storage device before taking it offline. In that way, applications continue to operate and users are unaffected. When we are done with the maintenance, we simply move the data back to the upgraded storage device.”

Storage management

Storage management is also very much improved, according to Mitesh Manek, systems administrator: “We no longer need to get inside the storage subsystems or remember how to use the various storage subsystem configuration utilities. Almost all storage management functions are accomplished using the SVC console. Having one uniform console gives me a system-wide view across all devices so I can see the big picture. It also has streamlined our administrative work.”

SVC also gives the IT staff more time to work on higher-level aspects of the infrastructure. For example, “I no longer have to worry about spreading disk activity across physical devices to avoid hotspots that slow performance,” says Manek. “At any time, we can migrate data among the storage subsystems to balance the workload and maximize performance. With SVC, I’m sure we can improve performance compared to our non-virtualized environment.”

Logical unit number (LUN) sizing has also become very efficient, in part because SVC enables capacity to be allocated in small increments. Says Manek, “I don’t have to allocate a 64 GB LUN if 32 GB will do. This fine granularity has enabled us to improve the utilization of our storage investment. And all of these types of changes can be accomplished without disrupting users.”

Future flexibility

SVC does not lock PPH into a single vendor for storage systems. Engberg said: “When you are investing for the long-term, it is all about flexibility. Today, all our storage is from IBM. But we want the ability to add some new device from another vendor so we can be opportunistic. That’s one of the big reasons why we saw no negatives in the move to SVC.”

He added: “Storage virtualization, managed by IBM SVC, is key to our plan to maintain a highly reliable yet flexible IT infrastructure. Our infrastructure is now well positioned to support the move to paperless medical records and automated systems that help our staff when they are at the patient’s bedside or even outside our facilities at their offices or homes.”

In fact storage virtualization is just what the doctor ordered for PPH.

^{[Paul Warren, staff writer.]}