Information Playground

Innovation Field Trip to NYC

Several weeks ago Stu and I received an invite to attend the 2009 World Innovation Forum. Next week we're both going (registration fee of 3K waived!) to sit in the "Bloggers Hub" at the Nokia theater in NYC and cover the event.

Because we blog. And tweet.

My daughter came home from school the other day and told me that one of her teachers thinks that Twitter is "a stupid waste of time". I said "Tell him that Twitter just won me a trip to New York City". To which she said "Can I go?"

Stu and I are technical by trade; he works in EMC's office of the CTO, I work as a software engineer developing storage products.  Stu has covered conferences via Twitter before; this will be my first time. I don't think software engineers typically cover world innovation conferences; doing so is proof positive that deciding to blog can lead to some great opportunities.

So what am I going to write about?  I've thought about the angle I'm going to use, and here it is:

Intrapreneur - defined on WikiAnswers as: a person within a large corporation who takes direct responsibility for turning an idea into a profitable finished product through assertive risk-taking and innovation.

I am going to listen to the speakers and apply their messages to the topic of intrapreneurship.

Many of the posts on my blog have been about ideas that I've helped turn into product. Or perhaps I've written some posts which highlight the efforts that my company is taking to encourage innovation from the people in the trenches. It's an area of great interest to me because it's my day-to-day job.

Often times messages about corporate innovation are targeted towards managers and/or corporate executives. I'm looking to pull out a few ideas that relate to the people in the trenches who daily carry the responsibility of turning an idea into a product. I'm interested to see what the tone and tenor of the speakers will be!

I will also be joining a diverse group of innovation bloggers who have a great interest in consuming pizza before, during, and after the event. I've listed several of their twitter accounts below in case you want to give them a follow.

By the way the organizers of the conference rang the opening bell at the NASDAQ on Tuesday.

Steve

Twitter accounts for bloggers/tweeters at #wif09 (in addition to @SteveTodd and @stu):

@hsmamericas (George Levy)

@innovate (Braden Kelley, http://blogginginnovation.com)

@reneecallahan

@andreaMeyer (http://workingknowledge.com)

@chrisflanagan

@pinnovation (Jeff De Cagna)

@yourboot (Julie Lenzer Kirk)

@katiekonrath (www.getFreshMinds.com)

Michael Lee Stallard (www.michaelleestallard.com )

Managing V-Max at Scale

The Symmetrix V-Max (TM) system, fully configured, can:

• export thousands of logical units of data (LUNs)
• serve hundreds of applications consuming these LUNs

These numbers will grow over time.

That is one large system. In fact, calling it 'large' is an understatement. The size and scale of the V-Max requires innovative solutions to the following questions:

• how in the world do I allocate that many LUNs?
• how do I connect the LUNs to the applications that consume them?
• how do I configure alternate paths to the LUNs in the case of a failure?
• how do virtualized applications start up on different servers and still see the same LUNs?
• how do I diagnose and repair applications with underperforming LUNs?
• how can I configure replication on this many LUNs?
• how large of a team will I need to take care of all the plumbing of application to storage?
• can the team work on parallel tasks at the same time?

A big part of the V-Max introduction (and quite possibly the most important part) is the new user interface that is purpose-built to manage at massive scale. Here's how it works.

1. Templates and Storage Groups: Automated "Find or Define"

Traditionally a storage administrator manually selects and configures the low-level disk segments that are dedicated to a given application. This is known as "define". This process is too manual for a V-Max scale system.

Alternatively, a storage administrator may have to "peruse" hundreds of pre-configured LUNs and reserve the ones that fit a given application. This is known as "find", and is also much too time-consuming at scale.

Traditional “find or define” can require a manual search through all the LUNs or a manual search through all of the free space.

For V-Max, the Symmetrix Management Console (SMC) has introduced the concept of "storage templates". An administrator creates templates for their particular application (e.g. My_DB_Disk_Template), specifies the capacities and characteristics (e.g. RAID level), and then specifies “Find”, “Define”, or “Best Effort”. The template can then be saved. The screen shot below highlights the process of creating a template.

Templates can be handed to a junior administrator having little to no knowledge of how to find or allocate storage on a Symmetrix system. These administrators can carve out physical storage from V-Max by creating a “storage group” (e.g. My_DB_Storage) based on the template. SMC creates the group by automating all of the low-level V-Max commands based on the template. There are no disks to select, and no device numbers to remember. The process of creating a storage group from a template is shown below.

2. Group Ports Together

Once the LUNs for a given application have been created, the next step is to select a set of storage ports to associate with these LUNs. Storage admins create port groups (e.g. My_DB_Ports) that will be associated with storage groups in a subsequent step.

3. Group Initiators and Automate the Connection

The final step of the end-to-end connection of V-Max storage occurs by creating an Initiator Group. This task simply creates a group of initiators (e.g. My_DB_Initiators) that will ultimately access storage groups through a given set of storage ports.

At the end of this process the administrator simply creates a “view”, which associates the three groups (My_DB_Storage, My_DB_Ports, and My_DB_Initiators). SMC automates the assignment of storage to port, and automates the mapping/masking of storage to initiators. This has traditionally been a very manual process that is now automated for V-Max.

What’s more, SMC automates the assignment of LUN numbers so that virtualized applications always consume the exact same LUN numbers (no matter what path they choose). A critical feature known as Volume Dynamic Addressing (VDA) is used to accomplish this task. No matter what server the application runs on, VDA ensures that the exact same LUN numbers are presented.

Think About It

A storage admin uses a template to form storage groups, creates port groups, creates initiator groups, and then tells V-Max: "make it happen". Sound easy?

It has to be. If it wasn't, then a V-Max at scale would be unmanageable.

Read 'Zilla's post about auto-provisioning in an ESX environment.

The Value of Templates

One can see how the introduction of application-based storage templates makes the initial setup of application storage much more automated. Dozens of low-level Symmetrix commands occur automatically from a high-level interface. The benefits of storage templates, however, go well beyond initial setup:

• Replication relationships can be specified within the template. Dozens of R1-R2 replication commands are automated using this streamlined approach.

·         Application running poorly? LUNs in the storage group currently on SATA drives ? SMC's LUN Migration wizard can search for candidate FC drives, automatically create the required raid groups(if necessary) and perform the non-disruptive LUN migration. The application connections remain intact and the migration results in increased application performance.

·         Application running out of capacity? Want to add 9 GB of SATA RAID-1 storage on the fly? Use a template to dynamically enlarge a storage group (see diagram):

The ease of expanding storage on V-Max highlights one of the more intangible benefits of SMC's template-based approach: the ability for a small team of people to manage huge amounts of application storage. Some of the benefits are listed below:

• While it's handy to know all of the underlying Symmetrix commands for connecting storage to applications (for all three steps), it's no longer strictly necessary.
• One administrator can manage storage needs for dozens of applications using a template-based approach. Once the number of applications reaches into the hundreds, more staff can be added (if desired), and storage can be managed per application(s).
• Traditional Symmetrix configuration changes required the reservation of a global lock. This lock often prevented other admins from making changes to a different part of the system. V-Max has increased the granularity of the lock. Dynamic changes (e.g. port assignments) no longer use a global lock but lock separate objects instead. This allows for parallelization by multiple storage admins.
• Static configuration changes (such as migrating application data to a different storage tier) still require a global lock but only at the beginning and at the end of the migration process.

No Guesswork

This is a long post (for me!), but it's been necessary in order to emphasize the scope of changes that have occurred to SMC in order to support a system of V-Max scale. The requirements for these changes came straight from Symmetrix customers that have been pushing the boundaries of massive scale for years.  In fact, the V-Max architectural vision was shared with customers as early as 4-5 years ago, and customer's ease-of-use suggestions have been realized in the SMC V-Max interface.

Steve

V-Max Quality

EMC's Symmetrix V-Max(TM) announcement today reminded me of my first internal briefing for the V-Max project. The Symmetrix team proposed building a data storage system that would massively scale beyond the capabilities of any existing system in any data center.

This meeting happened maybe 3-4 years ago. I saw a presentation on a new hardware architecture. I listened to the proposed changes to the Symmetrix microcode (Enginuity). I saw preliminary schedules that were aggressive (given the substantial changes to the system).  Based on the Symmetrix organization's tradition of hitting their deadlines, I felt that their schedule was achievable.

Then they revealed their quality target of equivalent reliability to the currently shipping Symm systems (the DMX line), and I thought to myself: "no way".

For the lay person, the Symmetrix V-Max line targets 5 9s of reliability (also called uptime). A 5 9s system will experience downtime on the order of minutes per year.

I was skeptical.  Most significant for me was the change from a massive, centrally located cache to the proposed fully distributed caching system. The magnitude of the proposal, in my mind, could not target 5 9s and still ship on time.

Not only has the Anarchist proved me wrong (he's good at that), but the Symm team finished ahead of schedule.

Which leaves me asking one question: how did they do it?

Three Legged Quality Stool: Hardware, Software, and Process

One of the first things that the hardware team did was to enumerate all of the hardware components in the system:

• The number of hardware components had to be minimized. Fewer components increases the odds for better reliability. For example, the traditional Symm front-end directors and back-end adapters were consolidated into a single component. This "integrated" director results in a 75% reduction in parts (yet provides the same function at greater performance).
• Each hardware subsystem (such as the set of directors, or the power subsystem) was assigned a hardware availability goal of 7 9s, which translates to subsystem downtime of roughly 3 seconds per year. This allowed the combined HW system to achieve a design target of 6 9s, or about 31 seconds of downtime per year, overall. 
• Component redundancy was designed into every aspect of the eventual system. All components were grouped into subsystems which had a minimum of 1+1 redundancy; critical subsystems had greater that 1+1 redundancy. The fabric power system, for example, is 1+3 redundant. It can withstand 3 failures and still operate.

Software

One of the key activities in regards to the quality of the Enginuity software itself was its ability to abstract itself from the significant changes to the hardware componentry. For example, the caching algorithms themselves were unaware that the underlying physical implementation had changed from a centralized to a distributed model. The fact that front-end directors and back-end adapters were now the same component was invisible to Enginuity.

In other words, Enginuity was able to leave many of its field-proven algorithms untouched.

The newer software (e.g. the distributed caching algorithms) introduced an architecture that protected customer data in transit between distributed directors and all the way down to the disks themselves. This new code was implemented as a separately testable module that was mercilessly beaten for months.

One final note about the software: the shift to an x86 architecture required an Endian change. Endian modifications forced the Symm engineers to examine every line of code in light of the new architecture (see code reviews below). Not only did endianness of byte ordering have to be analyzed, but inter-system SRDF compatibility implied that endianness had to be transparent between systems with different processor architectures.

Process

The V-MAX system, with its enormous scale, required new and/or improved quality processes that would result in a level of quality equivalent to Symmetrix systems already in the field:

• 3 phases of failure modes and effects analysis (FMEA) were held during the design cycle. Each phase (architectural, pre-build design, and post-build validation) covered all of the gory details of all possible modes of failure. Each mode required an appropriate and resilient recovery option.
• Extensive modeling and simulation tools were used during all three phases as well. Modeling occurred at the architectural level and pre-build design level, and at the physical system level. Quality and performance results on the actual V-Max system were constantly compared and validated to models generated in previous phases.
• DFX (design for X) reviews (X = manufacturability, maintainability, serviceabilitiy, reliability, and availability) began early in development and continued throughout the development process.
• First pass concepts all underwent an extensive serviceability review, including user repair, cabling, look and feel, and human factors, which resulted in numerous enhancements.
• An advanced development team (separate from the actual developers) was chartered to shake out many of the new technology, portability, and endian issues that would eventually be discovered during development.
• An independent team was formed to focus on quality above and beyond the DMX-level testing. This separate team worked closely with the DMX teams on DMX-level compatibility and quality metrics. Note that both teams had a strong focus on SRDF and management compatibility. Both systems must seamlessly work together at customer sites.
• The software team increased its focus on rigorous code reviews given the significant architectural changes between V-Max and DMX.

It's difficult to sum up years of quality work in a single blog post, but I believe you get the point. 5 9s was considered from the beginning to the end of the project. Critical subsystems received extra protection (e.g. power fabric can survive 3 power failures, RAID-6 configurations can survive 2 drive failures, etc.).

Which brings me back to the original question: how did Symmetrix create a V-Max system with equivalent reliability to the systems that are in the field? By following the sequence below:

• Assigned the 5 9s requirement to everything.
• Target 7 9s for hardware subsystems, and 6 9s for the hardware system as a whole.
• Combined V-Max modeling tools with field reliability data from existing systems to validate the requirements.
• Performed three phases of FMEA to make sure no stone was left unturned.
• Created development and test processes that continually measured compliance to the original objectives and drove them all the way through system test and QA.

Steve

Mass Innovation Night

No it's not mass as in "huge", it's mass as in "Massachusetts".

On April 8th the inaugural Massachusetts Innovation Night (twitter tag #MIN) will be held at Waltham's Charles River Museum of Industry and Innovation.

I've signed up to go. What drew me to the event was the fact that the organizers have selected 10 technology innovations for display, and each of these innovations represents the efforts of a local company. It sounds interesting enough to check out, and there's a good chance that food might be involved.

The list of the 10 featured innovations can be found here.

Out of the 10 technologies the one that caught my eye was heliostats, which is a solar technology. I've written before about the electricity that I have generated via photovoltaic solar panels on my roof; heliostats use the sun's energy in a different way. Practical Solar has an interesting primer on the technology here.

If anyone else plans on going let's get in touch via Twitter @SteveTodd.

Steve