Information Playground

What's In A Storage Name

When the acronym RAID came out, it stuck like glue. The original paper from Berkeley was extremely influential in its introduction of terminology that combined disk striping and data protection algorithms. The influence of that original paper was carried over to RAIN as well.

When the RAID paper came out there was already a set of researchers and industry types ruminating over parallelizing disk access while still being able to handle faults. Patterson and team formalized some base level terms that reflected those discussions.

The term "RAID" is in its third decade.

In 2009 we've had researchers and industry types having analogous conversations about storage challenges in a virtualized data center.

Will the same type of widely-adopted acronym surface?

It's an interesting exercise to revisit the industry context for the original RAID paper. Here's a drawing that describes the mid-80s setting for RAID:

CPU speeds were increasing faster than disk speeds. CPUs were hanging around idle while disks were (slowly) processing their queue. RAID defined a lexicon for striping data across multiple disks in order to improve performance. It also defined redundancy schemes to guard against the increased chance of disk failure.

If I were to draw an overly simple but analogous picture highlighting the virtualized data center challenges for 2009, it might look something like this:

In the mid-1980s the storage problems solved by RAID addressed performance and data integrity/availability.

In 2009 we are seeing multiple cores, massively virtualized servers, mixed workloads, and massive amounts of data. Performance and data integrity are clearly still issues, but other discussions are in the mix as well (e.g. storage scalability and manageability).

Academia and industry technologists are having conversations about these problems. Solutions have been rolled out, and new ones are being proposed.

Do you think we'll see a treatise similar in scope to the original RAID paper?

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Provenance Awareness

EMC is building stronger relationships with universities and technology companies in the Cambridge, MA area. I've been making monthly visits to Harvard University to interact with both their CS department and their Initiative for Innovative Computing department.

For several years I've been monitoring Harvard's research on Provenance-Aware Storage Systems (PASS). The PASS team has spent several years outlining the importance of maintaining provenance and they have collaborated with others to build several systems that implement provenance.

Their PASS research overlaps with internal research I've been doing on data lineage using object-based storage systems. One of the thornier issues to solve with data lineage or provenance is the unity of content and lineage. Object-based storage systems are characterized by their ability to permanently bind content and metadata together using a unique identifier. It stands to reason that object-based systems would be of interest to the PASS team.

Several months ago I presented this approach to a room full of technologists at Harvard. My presentation was a XAM presentation in the context of digital curation (the very same lecture I had given live as part of the EMC Innovation Lecture series). I reasoned with the audience that although XAM is a great technology for digital curation, it is also a great technology for researching provenance.

What was the end result?  There were many heads nodding in agreement about the potential for XAM implementations of data lineage, as well as a good amount of questioning and possible requests for enhancements to XAM that might need to be looked at as well.

As the fall semester winds down we'll be continuing the conversations in 2010 and looking for ways to progress this type of research.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Scientific Data: Link or Die

While working at EMC Cambridge last week I took the Red Line over to Harvard to attend a lunchtime lecture by Joe Futrelle. Joe's lecture focused on the availability of scientific data. Given my interest in digital preservation, as well as my involvement in the MIT DataSpace proposal, I was interested in what Joe had to say.

I jotted down a couple of quotes that I found interesting:

If you can't tweet your data, it doesn't exist.

Scientific data sets are competing with blogs and twitter.

Joe's point was that scientists need a new mindset when generating scientific data sets. Every step of the way (including well before the generation of scientific data), a researcher should be generating and preserving meta-data, and forming a rich set of network links that will ultimately surround the scientific content.

Without an intentional plan for rich linking of scientific data with externally related internet-based research, scientist's data will not be found amid the blogs, tweets, and scientific papers that are written daily about (potentially) similar subjects.

In order to progress his work (and as part of the National Center for Supercomputing Applications) Joe has been involved with the Tupelo project. He used the analogy of JDBC (a common Java interface to unite disparate database technologies) to describe how Tupelo is an effort to use the Resource Description Framework (RDF) to unite scientific data repositories.  In particular, the RDF spec describes a graph-data model containing global identifiers and named links as the two key technologies that Tupelo leverages.

This picture of globally unique identifiers and named links reminds me of XAM technology, and in particular how it can be used to trace lineage from one object to its parent.

Software that generates data will come and go; but the data itself strives to remain. Therefore software developers should focus less on their own software (which will eventually become obsolete) and focus on two crucial aspects:

• Focus on how their software interoperates
• Focus on how to annotate the resultant scientific data so it is highly reusable

After the talk I spoke with Joe about XAM technology and its applicability to this area. Within a few weeks the MIT DataSpace proposal (uniting scientific data archives) will be presented to the National Science Foundation in Washington DC as part of the final stages of the NSF DataNet grant process. If approved, it will be interesting to see whether or not technologies like Tupelo and XAM will make their way into the effort.

Steve

http://stevetodd.typepad.com

Twitter: @Steve Todd

EMC Intrapreneur

Cloud Building: New Internal User Community

I've read (and written) quite a bit about private cloud this year. One of the private cloud aspects that I find most interesting is the fluidity of IT resources and the promise of seamlessly migrating applications between clouds.

Earlier this week I noticed the formation of a new social media community sprouting up on our internal EMC ONE platform.  The community is called "RTP2 - EMC Data Center Migration". The formation of this community is in direct response to the recent announcement of 400 new North Carolina jobs and a recently purchased EMC data center facility in Research Triangle Park.

Here's the deal: many of the IT applications currently running internally at EMC are in Westboro, MA. Most (if not all) of these applications and services will be moving to the RTP facility. The community performing this task has the goal of helping "to communicate the plan, vision and strategy [of the move] to EMC Users".

It's going to be a major effort. I'll be subscribing to this community so that I can receive notifications whenever they post anything new. I would imagine that many of the IT systems have been running in Westboro for years and are not fully leveraging the latest in virtualization technologies.

I'm particularly looking forward to hearing about the technology choices being made for the new data center configuration. One would assume it would be vBlock-based and managed by IONIX. It's unclear whether or not VMs will seamlessly VMotion down to the new environment. I'm also not sure if heterogeneous equipment will be moved down there as well.

As far as the internal community, strike up another win for social media. Upon joining this community people can get an inside look at the schedules, tasks, technologies, and applications that are part of the move, published by people that are actually DOING the move.

Stay tuned. It will be interesting to write about the data center move while many of EMC's customers are trying to do the same.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Book Review: 7 Lessons For Leading In Crisis

Last month I attended (and blogged about) the World Business Forum in New York City. The evening before the event, Bill George (who was attending the WBF as both a blogger and a speaker) hosted all of the bloggers for the purpose of getting to meet one another. I discovered that Bill had recently authored a new book: 7 Lessons for Leading in Crisis. It took a few weeks but I finally had the chance to read one of the copies he handed out that night.

One of the most important things about the book is the context in which it was released:

• The financial crisis had reached dangerous levels towards the end of 2008.
• Bill's book is about "leading in crisis", and was published in 2009.
• Bill served on the board of Goldman-Sachs during the crisis.
• The book contains numerous examples of leadership behavior (both good and bad), some of which occurred during the current financial crisis.

The book suggests a list of seven leadership qualities that can be applied to any type of crisis. Bill sprinkles a variety of examples throughout the book.

I was mainly drawn to the book by Bill's position on the board of Goldman Sachs. In this sense he had a front row seat to the crisis. How often does someone in his position also have the writing skills to communicate what went wrong on such a massive scale? I was hoping that the book would contain the same frankness and openness that Bill uses on both his own blog and as a NY Times blogger (in which his transparency leads to some lively debate).

I was not disappointed in this regard.  Bill pointed at several different banks that went under during the financial crisis and called out examples of poor leadership.  During the pages of the book he did not pull any punches; this was the type of frankness I was hoping for.

Overall Thoughts on the Book

At just over 100 pages, the book is a quick read. The seven different leadership qualities that Bill covers seem to me to be common sense: don't take the world upon your shoulders, get ready for the long haul, dig deep for the root cause, etc. (For a quick summary of each category, I refer the reader to Jessica Lipnack's book review here).

I don't believe that Bill's seven tenets of crisis leadership are revelatory. From that point of view, I don't believe that I learned significantly new techniques for leading during a crisis.

I find the book's main value is that of a yardstick. If I want to evaluate my own leadership skills during a crisis, the book is an excellent place to turn. If I want to evaluate a public official, or a corporate executive, and formulate a thoughtful opinion of their performance during a crisis, I would refer to this book.

One particular chapter resonated the most for me: Lesson #3 - Dig Deep for the Root Cause. The advice that Bill gives in this chapter had a strong synergy with my write-up of Bill's speech from the World Business Forum. Leaders need to spend at least some time in the trenches. Here are a couple of gems from the book:

• Morgan Stanley CEO Philip Purcell didn't spend enough time on the trading floor talking to the traders when his firm was going through a myriad of problems in 2004 (he should have)
• Maintain close contact with people throughout your organization, not just your direct reports.
• Visit customers when they are in crisis (Bill had one angry customer throw medical equipment at him)
• Wander through the corporate offices and labs and ask the employees what they think the company needs to do to improve.

Overall I enjoyed the unique point of view on the financial crisis, as well as the framework for evaluating leadership. It's a good reference book to keep handy during tough times.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Research Papers Moving to the Cloud

The ACM has been publishing scholarly works since 1954. They have been diligently maintaining an online library of research papers that has continually benefited the technical community. For years much of their budget has been spent on print-based services as well.

I read an announcement by the ACM last week that they have decided to scale back their investment in print-based services and focus instead on the long-term digital preservation of ACM content as part of a public cloud.

Public clouds that specifically target digital preservation have a different set of requirements than a public cloud like Amazon EC2, for example. The focus in a "preservation cloud" is longevity, and the administrators of said cloud must think like digital curators.

It's an interesting exercise to study the (a) system and (b) curator choices that ACM has chosen. The ACM has chosen to go with CLOCKSS and Portico, respectively. Both CLOCKSS and Portico are not-for-profit organizations.

CLOCKSS

LOCKSS stands for "Lots of Copies Keep Stuff Safe", and the 'C' stands for "controlled". The LOCKSS software framework was developed at Stanford University and is a peer-to-peer, de-centralized model.

The CLOCKSS initiative is run by "the world’s leading scholarly publishers and research libraries" with a goal of ensuring "the long-term survival of Web-based scholarly publications for the benefit of the greater global research community". CLOCKSS ingest boxes are located at Rice, Indiana, and Stanford university. As libraries and researchers submit content to these "ingest boxes", they are stored in normalized, maintainable format in triplicate across the sites. Once all of the ingest boxes have cross-audited the content, the artifacts are moved to "archive nodes" spread throughout the globe. These boxes continually audit themselves and verify the authenticity of the content, and create new versions when hardware fails (very similar to a RAIN architecture).

Interestingly enough, the solution is a "dark archive". The initial content is not accessible to the general public. The ingest and preservation within CLOCKSS is initially focussed on maintaining content for the long-term. When a "trigger event" occurs, however, content is made public by migrating it to the newest format and storing it on publicly available nodes at Stanford and the University of Edinburgh.

A description of how CLOCKSS works can be found here.

Portico

Portico can be thought of as a third-party partner that supplies the people and processes behind the CLOCKSS solution. The best way to explain the process they bring to the table can be found in one of their brochures:

Diagram from the Portico brochure

The CLOCKSS and Portico solution relies heavily on migration of data formats as the years roll by. I've always wondered about the feasibility of such an approach. Can it scale to billions of documents? Does the system have to continually upgrade all documents to newer file formats as these new formats become available? One thing I picked up as I learned about their solution is that the format conversion occurs when the format goes from "dark" to "publicly available". In other words, there's no need to continually upgrade file formats.

The alternate approach to format conversion is something I've been researching here at EMC. As an entire class of documents is imported into an archive, can a Virtual Machine also be imported that is able to "read" all of these documents, both now and in the future? This would require some time of virtual machine "player" that has infinite playback capability (no easy task).

I hope to learn more about the server/storage implementation of CLOCKSS at locations around the world.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Information's Carbon Footprint

I noticed last week that Chad mentioned the Intel - EMC research that will reduce server power consumption for the Atmos Storage System.

For those unfamiliar with the software internals of the Atmos system, I wrote a primer on the overall architecture last year. This architecture allows a service provider to globally distribute petabytes of information around the world as part of "one global Atmos configuration". This architecture is also the reason why Atmos is referred to as cloud-optimized storage; content stored in one geographic location is dynamically copied/migrated within a cloud to other geographic locations based on the nature of the content itself.

Service providers that deploy a cloud-based storage system could in theory generate a fairly significant carbon footprint when storing petabytes of content. Their purchasing decisions will be partly based on vendor ability to minimize power consumption.

Atmos to date has used disk spin-down as an answer to minimizing power consumption. Imagine an Atmos deployment with three systems: one in Santa Clara, one in NYC, and one in Shanghai.  Atmos has central configuration monitoring and knows the "capabilities" of the Atmos hardware in each location. For example, Atmos might know that the system in Shanghai is spin-down capable, and that the system in NYC is not.

Assume a video file is stored on the Santa Clara system. Assume that the service provider has configured a policy which states "all video files need to be mirrored".  At this point Atmos could mirror a second copy to either Shanghai or NYC.

But what if the video file was stored with additional metadata that stated "green = true". In other words, what if the service provider wanted to store a second copy but minimize the kwh required to store the second copy.  When Atmos detects the green keyword, it triggers the policy which mirrors to Shanghai.

So how does the Intel / EMC research play into this?  Today's Atmos product features disk spin-down. In twelve months, the Intel / EMC collaboration should enable Atmos to manage power at the server level (in addition to disk). Server power will be controlled by taking advantage of Intel's power management tools for its Nehalem processors.

It becomes an interesting thought experiment, therefore, to extend our service provider example by adding a fourth location: Iceland. Imagine that the new Iceland Atmos hardware supports spin-down AND Nehalem power management. What happens when a video file gets stored to Santa Clara with the "green = true" metadata? Does Atmos put the second copy in Shanghai or Iceland?

When it comes to power consumption, what I'd like to see is the addition of more mathematical policy evaluation within Atmos, as well as the ability to dynamically reflect the prevailing electric rates for each geography. For example, how big (number of disks and servers) are the systems in Shanghai/Iceland, how full (capacity) are they, how often are they read (spun-up/spun-down ratio), etc. 

One of my co-workers has spent a good deal of time fine-tuning these equations. They can result in answers that reflect the kilowatt cost of storing a file of a certain size, and the estimated percentage of spin-down savings that the file is likely to see.  On a strictly power-consumption basis, Iceland may end up being the destination data center, given the additional capability for server-based savings.

However, if China has lower electric rates, a "cost-to-power" policy may determine that a given file may cost less to store in China than Iceland.

It's fun to imagine the possibilities here. If it sounds pie-in-the-sky, I can assure you that it's not. (It's more like "head-in-the-clouds").

These types of algorithms and capabilities will eventually be commonplace, and it goes beyond policies for electricity. There will be cloud policies for compliance, performance, and perhaps even information value.

The technology is enabled by attaching metadata to objects, which in part explains why there have been so many discussions about object storage this year.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Medical Cloud Formations

The latest issue of ON Magazine describes use cases for three early "cloud adopters": the National Business Center, Hill Air Force Base, and Beth Israel Deaconess Medical Center (BIDMC).

In particular I learned a lot from the BIDMC use case. John Halamka is the CIO at BIDMC and in the article he sheds a good deal of insight into the transition from a static and rigid medical data center into a more fluid and dynamic private cloud configuration.

The previous configurations at BIDMC were not virtualized; medical applications ran on servers that were directly connected to clustered storage. This configuration was becoming problematic due to the

".....strict ties between an application and the underlying servers and storage systems on which it runs".

The "strict ties" of medical applications to their data center infrastructure results in a fairly familiar set of problems:

• MDs experienced difficulties connecting to this infrastructure in an anytime, anywhere fashion.
• Upgrading the server infrastructure required application downtime and/or reduced server availability.
• Increasing clustered storage capacity and making it available across medical applications was labor-intensive and error-prone.
• etc.

BIMDC decided to build their own "medical software-as-a-service" private cloud infrastructure that uses the following building blocks:

• They deployed electronic medical records software from eClinicalWorks. Instead of doctors connecting directly into a physical server environment, they now securely connect into a cloud via their web-connected tablet PCs.
•  The centralized IT server infrastructure for their private cloud is provided by VMware. IT administrators can easily move, power up, and power down these VMs while being much more abstracted from the deployed server hardware.
• The server infrastructure sits on top of an EMC Storage Area Network (SAN).

When a new revision of eClinicalWorks came out that required more memory and storage, the IT department spent $30,000 on the upgrade, saving an estimated $270,000 in the process.

Public Cloud Futures

One of the more oft-advertised capabilities of private cloud is the ability to convert to a hybrid or public cloud model. BIMDC is not only well-positioned for this possibility, but they are anticipating that federal stimulus funding will drive their cloud to be more public very quickly:

"The Obama administration wants 90 percent of all clinicians and 70 percent of hospitals off of paper by 2011—and has designated $44 billion in stimulus dollars as a bit of encouragement."

“One of the things you’ll see coming out of the Obama administration is a real sense of urgency for sharing data—with patient consent, of course—doctor to doctor, doctor to public health agencies, to immunization agencies, to quality measurement organizations. So a lot of our work will be about building more connectivity so the data that’s in the cloud can be shared, clinical care coordinated, errors reduced, and—because public health will be involved—detection automated.”

By building their own private cloud and allowing internet accessibility to medical records, BIMDC is well positioned to not only share or merge their private cloud with trusted third parties (thus making their cloud more "public"), but their choice of VMware could allow them to VMotion portions of their infrastructure to another cloud.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Preserving Profiles in Courage

The digital preservation effort at the JFK Presidential Library and Museum keeps rolling on, one scanned document at a time.

All of President Kennedy's office files (POFs) have been scanned into the archive, and the digital curators at the museum are now focusing on digitally preserving President Kennedy's personal papers.

I have had the good fortune to personally see some of these documents, and for me the most interesting documents are those that contain the President's handwritten notes. In a previous post I wrote about his 1962 speech regarding the US Space Program, and how JFK scribbled notes on the first draft to highlight emphasis and relevance to the audience at Rice University.

Attached below is another example of a heavily edited document from President Kennedy's personal papers. This particular document is a draft of the book Profiles in Courage. The document, along with the handwritten notes, was probably typed and edited around 1953-54 (the book was published in 1955).

NOTE: The image above is PNG format; the curators at the JFK Library use TIFF. I converted to PNG because it's 30x smaller. The curators use TIFF format because they want the highest quality scan; every pencil scratch must be legible.

This is cool stuff. It highlights the importance of digital preservation efforts. As a software engineer working for a company (EMC) that builds information storage systems and workflow management software, it's also a great opportunity to try and build complete solutions that more fully address the needs of digital curators.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Innovation 2009

Today at EMC there will be a "global pause" as the worldwide technical community takes a breather to consider the ideas and technical directions that will propel the company into the future. The 3rd annual Innovation Showcase has taken on a much different feel this year. One thing, however, remained the same: employees from all geographies were encouraged to submit their ideas into a common contest.

This is my third year of active participation in this event. I'm a big fan of anything that introduces a bit of variety into the software engineering lifecycle that I participate in. I'd like to take a moment and discuss the ideas that I submitted. One of the big reasons for getting involved is that the three top ideas receive "instant funding". This funding is a guarantee that EMC will take the top ideas to the next level.

I jointly submitted two ideas this year. The first one was an Atmos-based idea which proposed some unique algorithms via the introduction of "green" keywords into Atmos meta-data. A co-worker and I were trying to come up with a "green DNS" approach to file-based carbon footprint and wattage reporting. Our idea made the first cut and resulted in a spot in the top 100, but we didn't get into the top 30 finalists. We've already received feedback that our idea was lacking in the "market impact" department. In other words it was a pretty cool idea but we hadn't fleshed out how exactly it would make money.

The second idea was also a joint effort between myself and Craig Randall. For several years I've been involved with digital preservation technology. Craig and I collaborated on an idea that would combine Documentum xCP technology with the growing market for Digital Curation. We made it into the top 100, and also made it into the finalist's round of 30.

The path to the final 30 was quite different this year.  As Stu points out, the popular vote had a much more important role to play than in previous years. In fact, when we posted our idea on EMC's internal social media network, the discussion it generated was fairly extensive.

What I found interesting was that the discussion surrounding our idea was not primarily coming from EMC's technical community. It was coming from the EMC Sales community. Craig and I had done some research on the potential sales opportunity for our idea; the interest from our co-workers in the field validated our sense of the market opportunity, and their comments played a huge role in our idea making the final cut. This year, more than ever, EMC is looking for ideas and proposals that can be monetized.

So today we'll await the final ruling of the judges. In years past all of the finalists were flown to Franklin, Ma for a "science-fair" poster pitch to a set of executives. This year each finalist had to create a set of slides, do a "voice-over" pitch, post their slides internally, and then do a tele-conference with a specific set of judges from around the world. The judges for Craig and I were from Documentum, Document Sciences, Decho, and Celerra.

So the pressure is off this year. There's no need to pitch my idea in person to an exec (so I'm not wearing a tie today). It's a day off to travel to a localized gathering and listen live to a broadcast from the CTO team that has traveled to the host facility in Bangalore, India. The day will also be filled with local speakers collaborating on the state of innovation in their specific region.  Localized gatherings are also occurring in Beijing, Shanghai, Tel Aviv, St. Petersburg (Russia), Rotterdam, Cork, Paris, North Carolina, California, New York, Seattle, Montreal, and Utah.

In addition to learning about some really cool ideas from my co-workers around the globe, for me the benefit is that the technology community has become a much smaller place. The inventors, Distinguished Engineers, Fellows, and technologists (not to mention the sales community!) are becoming more familiar to each other, no matter where they are on the globe.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur