Information Playground

Evidence Based Medicine

Object-based storage techniques have achieved wide adoption in the medical field. I've long thought that one of the reasons for this is that IT administrators appreciate the "hands-off" attributes of an object store when it comes to configuration and capacity scaling.  When storing fixed content such as medical X-rays, an IT administrator in a hospital doesn't have to configure LUNs, create file systems, or provision new storage capacity when it is added to an existing archive.

What I'm starting to see more of, however, is an increasing number of medical use cases that are satisfied by the use of object metadata.

One use case of particular interest is that of evidence-based medicine. This use case dictates that medical advice and decisions should be augmented by scientific methods.  If a doctor recommends a certain course of treatment (or a set of options for treatment), it becomes quite valuable to point to the scientific data that backs up the recommendations. The science can be combined with other factors in order to make a critical decision.

I became aware of evidence-based medicine when reviewing the use of Atmos by a company called CareCore National.  The about page for CareCore makes the following statement describing the importance of high-tech in the advancement of evidence-based medicine:

We focus on appropriateness to ensure that health plan members receive the right procedure for their clinical situation. We achieve this by employing state-of-the-art technology and evidence-based criteria.

There are several reasons why Atmos facilitates an evidence-based approach:

• Geography: scan this press release for the phrase "half a petabyte of geographically dispersed data". CareCore National has Atmos deployed in South Carolina and New York. Any "evidence" stored on Atmos must be available in multiple geographies.
• Scale: The 500TB of geographically dispersed data actually represents 15 years worth of unstructured data that is being migrated onto Atmos. Several million documents per day have already been ingested into the deployment.
• Metadata: Evidence has to "stay around". In other words, evidence-based decisions should be reviewable long after they are made. In order to keep documents for an extended (or infinite) length of time, the CareCore National team is making use of metadata to establish retention periods (via the retentionEnable and retentionEnd variables defined in the Atmos Programmer's guide).

One of the reasons I'm tracking this type of use case so closely is the high degree of correlation that this field has with the concept of data lineage. Object meta-data can point to other objects in order to create a "chain of evidence".  I don't believe that CareCore National is using metadata in this way, but if I'm reading the tea leaves of the industry correctly I do believe that object lineage (also called provenance) will receive quite a bit of attention going forward.

In fact, this summer I plan on spending some time collaborating with students at Harvard University on the subject of data lineage using object-based techniques. Harvard has already done an extensive amount of research in the field of Provenance-Aware Storage Systems.

The use of objects for provenance has applicability beyond the medical field, especially when it comes to any form of non-repudiation. Given the embrace of object-based storage by the health care industry, however, I think it's a logical place to focus the impending research.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Information's Carbon Footprint

I noticed last week that Chad mentioned the Intel - EMC research that will reduce server power consumption for the Atmos Storage System.

For those unfamiliar with the software internals of the Atmos system, I wrote a primer on the overall architecture last year. This architecture allows a service provider to globally distribute petabytes of information around the world as part of "one global Atmos configuration". This architecture is also the reason why Atmos is referred to as cloud-optimized storage; content stored in one geographic location is dynamically copied/migrated within a cloud to other geographic locations based on the nature of the content itself.

Service providers that deploy a cloud-based storage system could in theory generate a fairly significant carbon footprint when storing petabytes of content. Their purchasing decisions will be partly based on vendor ability to minimize power consumption.

Atmos to date has used disk spin-down as an answer to minimizing power consumption. Imagine an Atmos deployment with three systems: one in Santa Clara, one in NYC, and one in Shanghai.  Atmos has central configuration monitoring and knows the "capabilities" of the Atmos hardware in each location. For example, Atmos might know that the system in Shanghai is spin-down capable, and that the system in NYC is not.

Assume a video file is stored on the Santa Clara system. Assume that the service provider has configured a policy which states "all video files need to be mirrored".  At this point Atmos could mirror a second copy to either Shanghai or NYC.

But what if the video file was stored with additional metadata that stated "green = true". In other words, what if the service provider wanted to store a second copy but minimize the kwh required to store the second copy.  When Atmos detects the green keyword, it triggers the policy which mirrors to Shanghai.

So how does the Intel / EMC research play into this?  Today's Atmos product features disk spin-down. In twelve months, the Intel / EMC collaboration should enable Atmos to manage power at the server level (in addition to disk). Server power will be controlled by taking advantage of Intel's power management tools for its Nehalem processors.

It becomes an interesting thought experiment, therefore, to extend our service provider example by adding a fourth location: Iceland. Imagine that the new Iceland Atmos hardware supports spin-down AND Nehalem power management. What happens when a video file gets stored to Santa Clara with the "green = true" metadata? Does Atmos put the second copy in Shanghai or Iceland?

When it comes to power consumption, what I'd like to see is the addition of more mathematical policy evaluation within Atmos, as well as the ability to dynamically reflect the prevailing electric rates for each geography. For example, how big (number of disks and servers) are the systems in Shanghai/Iceland, how full (capacity) are they, how often are they read (spun-up/spun-down ratio), etc. 

One of my co-workers has spent a good deal of time fine-tuning these equations. They can result in answers that reflect the kilowatt cost of storing a file of a certain size, and the estimated percentage of spin-down savings that the file is likely to see.  On a strictly power-consumption basis, Iceland may end up being the destination data center, given the additional capability for server-based savings.

However, if China has lower electric rates, a "cost-to-power" policy may determine that a given file may cost less to store in China than Iceland.

It's fun to imagine the possibilities here. If it sounds pie-in-the-sky, I can assure you that it's not. (It's more like "head-in-the-clouds").

These types of algorithms and capabilities will eventually be commonplace, and it goes beyond policies for electricity. There will be cloud policies for compliance, performance, and perhaps even information value.

The technology is enabled by attaching metadata to objects, which in part explains why there have been so many discussions about object storage this year.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Reverse Engineering with Atmos API

Ever since 'Zilla wrote his post about installing Gladinet on top of Atmos I've been wanting to give it a try. So I followed his instructions (very easy), and within moments of receiving my Atmos "uid" and "shared secret", Gladinet had created a "Z:" drive on my laptop.

I navigated to my Z drive, dropped in a JPG, created a folder, and then dropped in another JPG. I was officially storing files in the Atmos-sphere. Here's a picture of the data I stored in my Z: drive.

Last year I had written several blog posts about the inner workings of Atmos.  As Gladinet uploads files to Atmos, it should be possible for me to program the Atmos API directly to see what Gladinet is doing. So I navigated to the Atmos Online community and got my hands on the "QuickStart" document. I decided to program in the .NET environment, and the QuickStart document stepped me through it. The samples built and ran without a problem.

Given that Atmos stores "objects", I knew that I needed a way to find the top level objects in the Gladinet hierarchy. It took a few minutes to figure out that I needed to find an API that did NOT accept an object-id as input. The appropriate API was named "GetListableTags".

Once I called GetListableTags I started to see some metadata. To make a long story short, here is the code that I wrote.

        EsuApiLib.Rest.EsuRestApi atmos_api = new
                EsuApiLib.Rest.EsuRestApi("accesspoint.emccis.com", 80, uid, sharedSecret);

        // call Atmos to get the top-level tags

        MetadataTags tags = atmos_api.GetListableTags(tag);

        for (int i = 0; i < tags.Count(); i++)
        {

            //iterate through each top level tag and print out the name

            tag = tags.ElementAt(i);
            Console.WriteLine("\nTag {0}:{1}", i, tag.Name);

            foreach (ObjectId oid in atmos_api.ListObjects(tag))
            {

                // for each tag grab all object IDs and print out all object metadata

                Console.WriteLine("\n    oid {0}\n", oid.ToString());
                ObjectMetadata obj_meta = atmos_api.GetAllMetadata(oid);
                for (int j = 0; j < obj_meta.Metadata.Count(); j++)
                {

                    Console.WriteLine("        {0,-12} {1}",
                                      obj_meta.Metadata.ElementAt(j).Name,
                                      obj_meta.Metadata.ElementAt(j).Value);
                }

            }
        }

And here is the data that gets dumped out when I run the code against the online Atmos account backing my Gladinet Z: drive:

Tag 0:GCDHOST

    oid 4980cdb2a110105e04980d61b3e98204a8069ad563b5

        GCDName      BoomChugHiking.JPG
        atime            2009-08-10T18:40:45Z
        mtime           2009-08-10T18:40:45Z
        ctime            2009-08-10T18:40:45Z
        itime             2009-08-10T18:40:45Z
        type              regular
        uid                EMC00270299F451F0E87
        gid                apache
        objectid         4980cdb2a110105e04980d61b3e98204a8069ad563b5
        size               578067
        nlink              0
        policyname    default
        GCDHOST      BoomChugHiking.JPG

    oid 4980cdb2a310105e04980d61b65b5904a8069c447b45

        GCDTYPE      Directory
        GCDName     FirstFolder
        atime           2009-08-10T18:41:14Z
        mtime          2009-08-10T18:41:08Z
        ctime           2009-08-10T18:41:14Z
        itime            2009-08-10T18:41:08Z
        type             regular
        uid               EMC00270299F451F0E87
        gid               apache
        objectid        4980cdb2a310105e04980d61b65b5904a8069c447b45
        size              0
        nlink             0
        policyname   default
        GCDHOST     FirstFolder

Tag 1:FirstFolder

    oid 4980cdb2a510105e04980d61ba265004a806a06722bd

        GCDName      tfc.JPG
        atime            2009-08-10T18:42:15Z
        mtime           2009-08-10T18:42:15Z
        ctime            2009-08-10T18:42:15Z
        itime             2009-08-10T18:42:14Z
        type              regular
        uid                EMC00270299F451F0E87
        gid                apache
        objectid         4980cdb2a510105e04980d61ba265004a806a06722bd
        size               2565454
        nlink              0
        policyname    default
        /FirstFolder   tfc.JPG

The API retrieved three objects: the two JPG files and the folder that I had created.

What's going on behind the scenes with Gladinet and Atmos? If you take a look at the data you can start to guess. I plan on trying to introduce another object or two behind Gladinet's back and see what happens. More on that in a future post.

The main point here is that it's very easy to get started.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Atmos Metadata

In my last post I wrote about the ingest of data into Atmos, how it maps to a policy to determine locations and number of copies, and how it places the data based on a filled-in policy tree (an LSO). My example was based on the fact that a specific user (e.g. "joe") was storing the data.

Placement and movement policies can also be based off of user-supplied metadata tags.

TSA-2 asked where this type of metadata comes from <that would be Chris The Storage Architect, not the Barry the Anarchist ;>)>

The metadata will come from applications that integrate with the Atmos API and have designed a metadata tagging strategy for global policies.

The Atmos SOAP API serves as a good starting point for discussion.

Atmos Policy: Under the Hood

In my last post I described how "global policies" are truly the Atmos special sauce.

These policies can control the initial placement of data, the migration of data, the compression and dedup of data, and so on.

How does Atmos implement these types of global policies? Answering this question requires a peek under the hood of how the Atmos software is architected. In this post I'll cover one of the more important data structures in Atmos, as well as a subset of the Atmos software components.

The data structure describing an Atmos Policy is called an "LSO".

Atmos: Cloud Optimized Storage

With the announcement of Atmos today, EMC has also introduced a new class of storage: Cloud Optimized Storage.

This means there's a new acronym in town: COS.

For those of you already overwhelmed by the storage industry's use of acronyms, sorry 'bout that!

For those of you trying to understand what COS is, allow me to offer a definition. Think of COS in terms of the evolution of three storage system acronyms over the last ten years.

SAN ->  NAS ->  CAS -> COS.