Information Playground

Twitter: @SteveTodd

January 21, 2011

Media Lab Hack-a-thon

On Thursday of last week I attended a day-long meeting at the MIT Media Lab.

Dr. John Moore of the New Media Medicine team invited EMC, as a corporate sponsor, to participate in a "medical hack-a-thon", with undergrads, grads, sponsors, and other interested parties.

The medical hack-a-thon is essentially a two-week sprint to solve the following problem:

As the population ages and the burden of chronic disease grows, there is a pressing need for technology to improve the delivery of healthcare and to help individuals to take greater control of their health and wellness.

Several months ago I had taken a tour of the New Media Medicine lab and become familiarized with the CollaboRhythm project. This project focuses on improving the doctor-patient relationship using technological frameworks.

The hack-a-thon started with prototype proposals. The proposals varied on such topics as chronic disease management, disease prevention, healthy habit formation, and psychological/social wellness. In some cases the proposal was actually sponsored by (or driven by) a corporation.

My goal as an attendee was to listen and provide mentorship to each proposal. One week later I returned to the Media Lab to hear about the progress of each prototype. The final session (next week) will include videos and demos summarizing the entire body of work.

Here are some of the things I've observed so far:

Sensor data is a big part of many of the proposals. One proposal uses accelerometers to measure tremors and freezing in Parkinson's patients. Biometric sensor data from infant sleeping patterns is being proposed as a way to gather information about sudden infant death syndrome.
Cell phones are also a big part of many proposals. Several projects are writing cell-phone apps that receive sensor data and somehow interact with doctors and/or patients. One proposal involved a microphone embedded in a necklace that could cause a cell phone to ask: "You haven't swallowed recently. Are you drinking enough water?"
Data Visualization, a growing area of research at the Media Lab, is an important part of nearly every prototype.
Open source medical software is an important aspect of several proposals, and OpenID was mentioned as one potential method for representing a patient.The Indivo Framework is being used (and extended).
Prototypes targeting rural areas and third world countries are being explored. Often times during these conversations the term "low bandwidth" is used, and one creative proposal discussed low-bandwidth synchronous treatment exchanges using pictures as opposed to video feeds.
Kinect and Wii technologies are being studied as methods for doing rehab and/or facilitating automated patient/doctor exhanges.

The small group discussions after each presentation are the best part of the experience. During this time the ideas take on new forms and directions, and new use cases are suggested. The more interesting ideas created during the discussion time freed back to the presenters to help them shape their plans for the progression of the idea.

My own wheels have been turning during the interactions, particularly when it comes to capturing the provenance of medical workflows. I see a lot of opportunities.

This is the 2nd time around for the Hack-a-thon. A number of tools are available to the researchers, including open-source medical software, Android phones and tablets with working example applications, and health sensors (weight scale, blood pressure cuff, blood glucose monitor, etc.).

I'm looking forward to seeing the final prototype presentations on the 26th of January.

Steve

Twitter: @SteveTodd

January 17, 2011

VNXe

The announcement of the VNXe product marks the second deployment of a new software architecture. The first product was the Celerra NX3e. The NX3e was initially shipped in October 2009 (over 15 months ago), and its delivery was limited to the Nordic region. This allowed EMC to not only validate the architecture, but also to roll out a new sales model to a limited set of distributors.

Here is a summary of the VNXe software architecture and impact:

Component Assembly Approach

The VNXe software architecture can be thought of as an asset integration framework. Consider the subset of assets from other EMC products that were integrated into VNXe:

Portions of the CLARiiON I/O stack
Portions of the Celerra I/O stack
Business logic from Celerra Control station
Management logic from the former Navisphere suite
EMC's reusable GUI toolkit framework
the same security library already embedded in a majority of EMC's products
a licensing toolkit (also an internal EMC standard)
the hardened LINUX distribution already shipping in multiple EMC products
etc.

The ability to re-use so many of EMC's data path assets is due in large part to the capabilities of EMC's internal CSX innovation.

Hardware Footprint Reduction

One of the benefits of convergence is the ability to reduce hardware footprint. In 2007, the low-end of the Celerra product line was the NS-20. The NS-20 was an assembly of multiple different operating systems (Windows, LINUX, proprietary), and each of these operating systems would run on different hardware platforms.

Additionally, the CLARiiON system required two additional stand-by power supplies (SPS) in order to preserve the contents of the write cache.

By 2009, the converged architecture represented by the NX3e cut the hardware footprint in half. The SPS technique had been replaced with an on-board vault-to-flash option. The VNXe reduces the hardware footprint even further with a single-SP option. This not only drives the cost of the overall solution significantly downward, but it makes the entire solution easier to manage (less moving parts). Speaking of management.....

Completely Rewritten User Interface

I've mentioned in a previous post that the shift to Flex technology represents a departure from the traditional mid-range and enterprise storage management experience. All of the field data was trending towards the needs of an IT generalist. It would have been more expedient to re-use Celerra (or CLARiiON) interfaces to manage VNXe......, but that just doesn't make sense.

Instead of LUNS, cache, and storage processors being front-and-center, the customer will see Exchange mailboxes, virtual machines, and application databases. The management protocol of the VNXe, therefore, is "application-rich". Configuration operations, monitoring, and alert handling are passed between the GUI and VNXe using terms more familiar to the IT generalist.

It's not about managing storage. It's about managing application storage. This paradigm also makes it easier for distributors.

Designed for Distributors

The initial price target for a minimal VNXe configuration was under $10K, and it was clear that this would not be a market traditionally addressed by EMC's direct sales force. This resulted in a multi-year effort to design the VNXe for distributors.

The decision to "design for distributors" was made at the same time as the decision to re-write everything in Flex. This allowed the engineering team to design communication paths in two different directions:

I've already mentioned the "application-aware" nature of the protocol between Unisphere and the VNXe. The Unisphere interface for VNXe also has rich-connectivity underpinnings to vendor ecosystems. These underpinnings provide great value to any vendor selling the product, including opportunities for eCommerce, enhanced support (e.g. chat, whitepapers), and user communities for VNXe users.

Going Forward

The LINUX platform, and the components that run on it, will continue to be a melting pot for combining new assets into the mix (e.g. the benefits of component assembly).

And when EMC acquires a new technology (e.g. GreenPlum, Isilon), it's a faster path to integration.

In fact, I think it's fair to say that the new VNXe platform was an intentional response to the torrid pace of EMC acquisitions.

Industry speculation was that EMC was cramming all of their acquisition assets into VMs and morphing them into some sort of new ESX-based storage platform.

Maybe we will. Maybe we won't. ;>)

Steve

Twitter: @SteveTodd

Comments (1) | TrackBack (0)

A Trace-Driven Architecture

In a recent post I described how trace libraries can influence the software architecture for different disk array features (e.g. Green, Thin, etc.).

The latest Symmetrix release is a great example. The high-level software architecture for sub-lun FAST algorithms has been directly impacted by analyzing billions of I/O operations captured from customer sites and internal systems.

Trace libraries can be pumped through a microcode simulator that emulates the front-end, back-end, and cache of a product like VMAX. The traffic between the cache and the back-end disks can then be analyzed. Simulation of movement between storage tiers (e.g. flash, FC, SATA) can be explored via the trace libraries. A strong goal for this analysis is to build a software architecture that moves the right data to the right tier at the right time, and doesn't move them when it's the wrong time. Another important goal is to select different chunk/extent movement sizes to understand the cost of moving data between tiers.

An additional goal is to make the architecture customer-friendly (e.g. does not require extensive use of knobs and dials to manage the FAST feature).

Finally, any architecture must be shippable by a certain date. Capturing trace libraries takes time. Getting them into the lab takes more time, and analyzing them takes even more. The architecture must allow the evolution of FAST techniques as more insight is gleaned from more workloads.

So what trace-driven insights made their way into the latest FAST release?

Efficient Extent Selection

Trace libraries allow engineers to experiment with different extent sizes and make trade-offs based on system (and LUN) footprint data. Meta-data that describes the location (tier) of FAST extents ends up increasing the capacity footprint required to represent a LUN. The system resources (memory, CPU) required to move different size extents impacts overall system footprint as well.

The trace libraries allowed the architecture team to identify an ideal extent size that minimized capacity footprint and minimized strain on system resources. This result meets two other goals as well:

Ease-of-use: no need for customers to input (or experiment with) different sizes
Shippable: there was no need to add an API for extent modification (and no need to test all of the different permutations).

When's The Right Time?

The trace libraries also shed some light on "when is the right time for data movement between tiers".

In general, when I/O activity is focusing on a repeatable subset of extents, then it is a great time to move those extents to a faster tier (e.g. flash). When the I/O activity is randomly accessing a wide variety of different extents, then tiering activity can be harmful.

Consider the set of extents that are actively accessed at midnight (Extent Subset A), versus the set of extents actively accessed at mid-morning (10:30 AM Extent Subset B). On the X-Axis below, I've highlighted these points in time over a three-day trace:

Now let's add in a Y-Axis containing the EXACT SAME trace data, compare the set of HOT extents at time A versus the hot extents for the entire 3 day period. If the extents are exactly the same for the two time periods (a correlation of 1:1), then plot a point using a WHITE dot. If there is zero correlation, use a BLACK dot. If there is SOME correlation, then shade the dot. The legend is explained on the right hand side.

Note that the extents that are active at mid-night (off-hours), have a very low degree of correlation when compared to extents that are active at other times of the day. Now look at the exact same chart for the 10:30 AM extent subset:

These dashed lines squarely intersect with the "white rectangles". What does it tell us? It tells us that the subset of extents that are active at 10:30 AM are generally active from 9-to-5 PM.

Which means that 9-5 PM is a great time to run FAST, but other times are not.

Activation Periods

This pattern tends to play out over and over again in a variety of different customer traces and workloads. As a result, the concept of an "activation period" was introduced into the Symmetrix microcode. FAST only runs during the activation period, otherwise the algorithms are idle. Adjustment of the activation period is allowed.

This is a sensible addition to the architecture. Thanks to EMC Fellow John Walton for explaining it to me.

Trace libraries will continue to drive the FAST algorithms. There is only so much that can be added to the code (and tested!) at any one time. Collecting traces is a huge investment, but the investment pays off when the released code hits the field and reacts favorably to familiar-looking I/O streams.

Steve

Twitter: @SteveTodd

January 13, 2011

Thoughts on the JFK Digital Archive

Over the last few years I've been documenting the progress of the digital archive being built at the John F. Kennedy Library and Museum. The archive itself (described here) has been assembled with EMC products.

On January 3rd of this year I decided to bring my parents into the museum. It was an EMC holiday, and it was also the day of their 52nd anniversary. I was pretty excited about the announcement of the digital archive, and wanted to share that with them, given their many memories of his public service, both in Massachusetts, and to the nation.

For the first ten years of my life, I lived in Dorchester, Massachusetts, which is now home to the JFK Library. My Dad had sat a few rows away from a young Senator John Kennedy as he spoke at a local lecture. He had even received a letter of congratulations from him when he was accepted to college (note to Dad: find that letter and let's try and get it scanned into the archive!). "Jack" Kennedy was a Massachusetts kid, who went to a Massachusetts college, and served the people of Massachusetts as a State Senator. My parents remember him, and his call to public service, very well.

As we walked through the museum, the Cuban Missile Crisis stood out as a major memory in their lives. They feared the total destruction of America and the Soviet Union, and were often glued to the TV, not knowing whether a nuclear war would occur. During our tour we also sat and watched the first-ever televised presidential press conference, further evidence of how live news coverage impacted the era in which my parents began to raise a family.

Today David S. Ferriero, Archivist of the United States, and Caroline Kennedy, President of the John F. Kennedy Library Foundation, officially unveiled the nation’s first online digitized presidential archive. The Digital Archive contains materials from President Kennedy’s official and personal records. It's available to teachers, students, scholars, authors and the general public through any internet connection at www.jfklibrary.org.

I encourage you to take the time and click on the link above. Search is accurate and fast, and the navigation on the website is excellent.

The video below is a great description of the archive, and includes commentary by archivists from the National Archives and Records Administration (NARA), employees of the JFK Library and Museum, and some of my co-workers from EMC.

Given the impact that the Cuban Missile Crisis had on my parents, I decided to search the online archive to find more information that was not available during our museum tour. I found pictures of the leaders who met to discuss America's options. I found an audio of (and listened to) a phone call between President Kennedy and former President Dwight Eisenhower. I found a wealth of material describing the efforts that the administration undertook before President Kennedy unveiled the plan for a naval blockade.

Of course, I found the very speech that President Kennedy gave on the night of October 22, 1962, with my parents watching and listening on the edge of their seats. I was able to download the document and have inserted it below:

Life is a circle. My parents moved through the turbulent sixties, out of Dorchester, raised a kid that would go on to build information storage systems (with Russian co-workers!), and then marveled as so many of those JFK memories were preserved onto the information storage systems that their son had helped to build. Their granddaughter (my daughter) now goes to college in Dorchester, at UMASS Boston.

We sat in the cafeteria at the museum and talked about all of these things as we took in the view of beautiful Boston Harbor.

They say that the launch of the digital archive will increase the number of visitors to the museum.

I believe it.

Steve

Twitter: @SteveTodd

JFK Digital Archive Solution

Today, January 13, 2010, the John F. Kennedy Library and Museum announced the availability of a new digital archive that allows public access to a portion of the museum's assets.

I have some personal thoughts about the archive; I'll share them in a separate post. This post focuses on the architecture that allows the general public to view and search the archive.

While the internal digital archive was assembled using EMC products, the full public solution was a partnership between EMC, Iron Mountain, AT&T, and Raytheon. The diagram below, produced by Raytheon as part of their project leadership role, describes how AT&T and Iron Mountain augment the archive itself by providing public web access and disaster recovery capabilities.

The primary archive is based on EMC's Centera and Celerra network storage systems and is accessed using EMC's Documentum Enterprise Content Management software. Documents enter the archive via scanners; audio/video is ingested (via Documentum) after it was produced by Iron Mountain Digital Services.
The public experience of the digital archive is provided by the AT&T Synaptic Hosting environment. Web-ready renditions of documents and photographs are generated by the Documentum system at the JFK Library at the time the full resolution digitized files are produced and ingested.
This backup archive, also built on the EMC Centera storage system, is located in an Iron Mountain secure underground operations facility. In the event that data in the Primary Archive is lost or compromised, the Disaster Recovery archive provides the capability to readily restore the full data holdings, including all associated meta-data.
The AT&T secure network is the inner backbone of the Digital Archive, linking the Primary Archive at the JFK Library with the Disaster Recovery Archive at Iron Mountain and with the Web hosting site at the AT&T data center. This secure network employs AT&T’s Enhanced Virtual Private Network (EVPN) Multi-Protocol Label Switching (MPLS) technology.

In my discussions with the archivists working at the JFK Library, the new archive goes far beyond any existing archive that NARA has ever assembled. There are several distinguishing characteristics of the archive that are state-of-the-art and quite differentiable from previous systems:

each artifact is scanned only once, minimizing the possible damage to fragile documents, speeding the ingest process, and ...
the EMC Documentum workflow and rendition capability preserves the initial high definition scan while automatically generating smaller files to be used for fast internet access and/or viewing over slower links
the automated scanning and classification software is part of an integrated package with the high-speed data storage and retrieval systems
the replication technology enables seamless offsite storage of all high definition copies
previous historical archives the size of the JFK Library would default to a tape-based design; a disk-based archive (e.g. Centera) provides renditions that allow for much faster access

The majority of the museum's assets are still "boxed up" in the back room. A small (and growing) percentage of those assets are now available to every student, historian, and person on the planet. That's powerful. The rest of the documents show up as "empty boxes" when the archive is publicly accessed. These boxes (known as cabinets in Documentum) contain metadata about what lies inside. The hope is that when a JFK researcher looks into a given box and sees that it is empty (i.e. the archivists have not scanned the documents from a given box), they will be able to request that scanning occur. This feature, when implemented, would guide the archivists in their scanning activity going forward.

Steve

Twitter: @SteveTodd

Comments (1) | TrackBack (0)

January 10, 2011

Lexicon for Information Flow

Last week I visited a customer that recently ordered a vBlock through the VCE Coalition. I was curious to hear their plans.

I have a vested interest in the success of the team deploying the new configuration.

They all work for EMC's IT department.

My Lexicon

With my background in storage systems, switch topologies, kernel drivers, etc., I attended the meeting armed with my suitcase full of terms and acronyms that describe the flow of information from application down to the storage media:

The words driver, HBAs, ports, zones, SAN, FC, directors, shares, directories, mount points, NFS, CIFS, SCSI, cache, flash, SAS, sync/async mirroring, NOVRAM, and SCSI are all terms that I am familiar with. I can stack these terms end-to-end to describe the physical flow through the hardware plumbing.

When it comes to the individuals at a customer site, however, the large majority care very little about these terms. They would rather not have to know them (at least not to the depths that they have in the past).

The New Lexicon

What I found was a new set of new terms that describe logical information flow:

JMS, AMQP, Rabbit/Sonic MQ, Spring, JMX, Hyperic, payload, serialization, SME, ESB, customer objects, XML, Apex, SalesForce.com, A2A, B2B, GemFire, Grails, tcServer, ABAP, MVC, JSON, common information cloud, and POJO.

I chuckled a bit when I listened to how silly it is to hear someone say POJO. Then I remembered my wife's reaction every time I say the word "SCSI". Maybe POJO isn't all that bad.

Cloud Information Flow

This new lexicon represents the "cloud wiring" that sits on top of their vBlock deployment. It simplifies the integration framework that will allow dozens (if not hundreds) of developers to implement business requirements. It will eventually replace the ad-hoc and point-to-point scripting, batching, and web services activity that attempt to mine and leverage information across the corporation. The problem is being attacked by standardizing on one framework (the VMware vFabric Cloud Application Platform), getting rid of specialized middleware, and more fully abstracting the development community away from the underlying database implementation.

Most importantly, the adoption of a vFabric Framework is a key piece of the hybrid cloud model. If the vFabric Framework can be used both inside the firewall (private cloud) and outside the firewall (e.g. when accessing data in a SalesForce.com-type model), then EMC developers can leverage their familiarity with the framework in order to effectively gather and mine information across private and public clouds.

This last point is key because so many of EMC's partners want access (B2B) to our private and public data in a common, unified, and secure way.

In future posts I hope to dive into this cloud plumbing in a bit more detail, including some more detail about how the Spring Framework can be leveraged in hybrid cloud deployments.

Steve