Information Playground

Getting Personal With Search

Most of the main internet search engines leverage human intelligence to find relevant documents based on keywords. One way that page rank algorithms leverage human intelligence is by analyzing the hyperlinks that people use to point to other web pages.

When I search for something that was published ten years ago, there's a good chance my keyword search will work because over time other people have searched for and found the same document.

One of the things I've learned from my co-workers in Beijing is that personal, private information search is a very different problem to solve.  The information is isolated from the internet, and therefore algorithms that leverage human intelligence for internet-based search are not applicable. Nobody is hyper-linking to personal information because it is hidden.

This means that if I am searching 10 years of personal documents, email, web pages, photos, music, tax records, newsletters, homework, etc., I may see search results that don't give me what I'm looking for.

My co-workers, realizing that human intelligence can be quite effective for internet-based search, wondered whether or not algorithms for private search could leverage the human intelligence found in one person (as opposed to thousands). This exercise led them down the path of associative memory.

Consider a person trying to find a specific picture from five years ago. Keyword search of their desktop is ineffective. But they remember what they were doing when they first saw the picture; they were emailing a specific relative on a specific holiday. If there was a way to feed this associate information into the search algorithm, then human intelligence indeed would help narrow down the search quite a bit.

This unique approach to the search of personal information has resulted in quite a flurry of research and university collaboration in China. The resulting prototype software to test this theory, iMecho, has been presented at a conference in Hong Kong. The idea was met with great enthusiasm. iMecho forms associative links while you are going about your daily personal business; these links can then be input into a search algorithm to more effectively find personal content.

I see a lot of promise for this research. One of the reasons the topic is so relevant in China is the general reluctance to post personal information onto common websites such as Facebook.  If the Chinese population eventually moves towards public cloud paradigms for their personal information, the associative search patterns kept on local PCs could in theory be securely applied against information which is found inside of a public cloud.

Congrats to the Beijing team for coming up with such a cool idea. In fact, a subsequent paper describing their work was accepted into the SIGMOD conference in the USA last year (only the 8th time a Chinese paper has been accepted at SIGMOD since 1995).

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Cloud Query: Prove It

When it comes to securing content in the cloud, section 7.2 of the AWS Customer Agreement currently says the following (in part):

We strongly encourage you, where available and appropriate, to (a) use encryption technology to protect Your Content from unauthorized access, (b) routinely archive Your Content, and (c) keep your Applications or any software that you use or run with our Services current with the latest security patches or updates. We will have no liability to you for any unauthorized access or use, corruption, deletion, destruction or loss of any of Your Content or Applications.

The encouragement to use encryption technology is something I can understand if my application sits outside of the cloud and I am sending files back and forth. On top of encryption there are other techniques that can be used to detect "corruption".

But what about database queries? When a query is sent into a cloud and a result comes back, how can we know that it's correct?

Trustworthy Outsourced Computations

I listened in on a lecture today given by Nikos Triandopoulos of Boston University. He visited EMC Cambridge as part of the monthly lectures sponsored by the EMC Research Cambridge Lecture Series.

Nikos introduced his topic by highlighting that cloud has inserted itself in between users and their trusted databases. When a database query enters a cloud it can have any number of "proxy" hardware and software entities that pass data back and forth. His research is trying to solve two problems:

1. How can a user verify the "correctness" of a query result? How can the user be sure that malicious software did not intercept the query and return a bogus answer?
2. How can this be done efficiently (e.g. performance optimized)?

At a very high level this research is attempting to do the following:

When a user submits a database query into the cloud, a cloud server will typically intercept and pass it on to the "true" database server that stores the actual database. The diagram above highlights that when a query result is returned to a user, it is accompanied by additional metadata that is labeled "proof". This metadata allows the user to determine whether or not the query is correct or not.

That's heady stuff, and the implementation is more than I can communicate in one blog post. But here is the gist of how something like this can be implemented.

First of all, the entire database has a hash function run against it, which results in a top-level "digest".

The same hash function is applied against pairs of adjacent database entries, resulting in a "digest-per-pair".

This hash function is then recursively applied against sets of adjacent pairs until the complete representation forms a Merkle tree, with the top-level digest at the top.

When a query result is returned, nodes in the Merkle tree are also returned, and the user can perform a mathematical calculation using the query result, the nodes, and the overall top-level digest.

Sound like a lot of work? What about database updates? Isn't it slow? The rest of Nikos' lecture touched on many of these questions.

As I listened on the phone I could hear a lot of questions being asked, and I recognized the voices of my RSA co-workers.  Encryption and correctness of database operations is right up their alley, and it will be interesting to see if future product roll-outs will leverage these ideas.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

NSF Meeting

Yesterday was a culmination of years of preparation when the National Science Foundation visited MIT. The NSF is a branch of the US CyberInfrastructure Department, and their team is in the final phases of evaluating grant requests as part of their DataNet initiative.

The NSF will grant five awards of $20 million apiece (spread over five years) for proposals that target the following vision:

Chapter 3 (Data, Data Analysis, and Visualization) of NSF’s Cyberinfrastructure Vision for 21^st Century Discovery presents a vision in which “science and engineering digital data are routinely deposited in well-documented form, are regularly and easily consulted and analyzed by specialists and non-specialists alike, are openly accessible while suitably protected, and are reliably preserved.” The goal of this solicitation is to catalyze the development of a system of science and engineering data collections that is open, extensible and evolvable.

Stuart Madnick and MacKenzie Smith of MIT have pulled together a proposal called DataSpace.  They've also gathered together a team of of participants (should the proposal be funded). More information about the motivation behind DataSpace can be found by reading Dr. Madnick's interview with On Magazine.

The full-day session was put together by Stuart and MacKenzie. Having never participated in a grant process before I found it quite interesting to experience the flow of the meeting. In general, most of the DataSpace presenters fell into the following categories:

• Scientists: the DataSpace project would initially target neuroscience and oceanography.
• Digital Curators: Curators from Oregon State, Georgia Tech, Rice, and MIT discussed issues and solutions for the management of digital scientific data.
• Industry: Google, HP, Microsoft, and EMC described technologies and research directions that are applicable to DataSpace.
• Software: several team members presented software technologies and open source tools that were applicable to DataSpace.
• Program management: MIT staff discussed how the project (and therefore the grant money) would be managed.

The meeting ran for about ten hours, and I spoke for 5 minutes. In general my presentation stated that EMC has a set of technologies that are applicable to DataSpace and could contribute people, equipment, and lab space to the effort. If you pick apart the NSF quote there are a set of technologies that map quite nicely:

• "science and engineering digital data are routinely deposited in well-documented form". The term "well-documented" implies metadata, and object-based storage systems would provide strong binding between scientific content and metadata.
• "are openly accessible". If object-based access methods are used, the XAM standard is an open standard for access.
• "while suitably protected". In addition to protection capabilities like parity, mirroring, backup, remote copy, etc., time-stamping and content addressing can provide extra levels of protection and authentication.
• "and are reliably preserved". Data preservation is a key part of XAM, given that it has built-in support for migration over multiple generations of technology. EMC is also doing research on data provenance and long-term preservation; these research directions have a strong overlap with DataSpace.

The presentation to the NSF had a great mix of vision, people and technologies. I'm pleased to have my name on the proposal. These technologies are right up my alley, and it would be a privilege to work with such a diverse and talented team.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

What's In A Storage Name

When the acronym RAID came out, it stuck like glue. The original paper from Berkeley was extremely influential in its introduction of terminology that combined disk striping and data protection algorithms. The influence of that original paper was carried over to RAIN as well.

When the RAID paper came out there was already a set of researchers and industry types ruminating over parallelizing disk access while still being able to handle faults. Patterson and team formalized some base level terms that reflected those discussions.

The term "RAID" is in its third decade.

In 2009 we've had researchers and industry types having analogous conversations about storage challenges in a virtualized data center.

Will the same type of widely-adopted acronym surface?

It's an interesting exercise to revisit the industry context for the original RAID paper. Here's a drawing that describes the mid-80s setting for RAID:

CPU speeds were increasing faster than disk speeds. CPUs were hanging around idle while disks were (slowly) processing their queue. RAID defined a lexicon for striping data across multiple disks in order to improve performance. It also defined redundancy schemes to guard against the increased chance of disk failure.

If I were to draw an overly simple but analogous picture highlighting the virtualized data center challenges for 2009, it might look something like this:

In the mid-1980s the storage problems solved by RAID addressed performance and data integrity/availability.

In 2009 we are seeing multiple cores, massively virtualized servers, mixed workloads, and massive amounts of data. Performance and data integrity are clearly still issues, but other discussions are in the mix as well (e.g. storage scalability and manageability).

Academia and industry technologists are having conversations about these problems. Solutions have been rolled out, and new ones are being proposed.

Do you think we'll see a treatise similar in scope to the original RAID paper?

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Provenance Awareness

EMC is building stronger relationships with universities and technology companies in the Cambridge, MA area. I've been making monthly visits to Harvard University to interact with both their CS department and their Initiative for Innovative Computing department.

For several years I've been monitoring Harvard's research on Provenance-Aware Storage Systems (PASS). The PASS team has spent several years outlining the importance of maintaining provenance and they have collaborated with others to build several systems that implement provenance.

Their PASS research overlaps with internal research I've been doing on data lineage using object-based storage systems. One of the thornier issues to solve with data lineage or provenance is the unity of content and lineage. Object-based storage systems are characterized by their ability to permanently bind content and metadata together using a unique identifier. It stands to reason that object-based systems would be of interest to the PASS team.

Several months ago I presented this approach to a room full of technologists at Harvard. My presentation was a XAM presentation in the context of digital curation (the very same lecture I had given live as part of the EMC Innovation Lecture series). I reasoned with the audience that although XAM is a great technology for digital curation, it is also a great technology for researching provenance.

What was the end result?  There were many heads nodding in agreement about the potential for XAM implementations of data lineage, as well as a good amount of questioning and possible requests for enhancements to XAM that might need to be looked at as well.

As the fall semester winds down we'll be continuing the conversations in 2010 and looking for ways to progress this type of research.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Scientific Data: Link or Die

While working at EMC Cambridge last week I took the Red Line over to Harvard to attend a lunchtime lecture by Joe Futrelle. Joe's lecture focused on the availability of scientific data. Given my interest in digital preservation, as well as my involvement in the MIT DataSpace proposal, I was interested in what Joe had to say.

I jotted down a couple of quotes that I found interesting:

If you can't tweet your data, it doesn't exist.

Scientific data sets are competing with blogs and twitter.

Joe's point was that scientists need a new mindset when generating scientific data sets. Every step of the way (including well before the generation of scientific data), a researcher should be generating and preserving meta-data, and forming a rich set of network links that will ultimately surround the scientific content.

Without an intentional plan for rich linking of scientific data with externally related internet-based research, scientist's data will not be found amid the blogs, tweets, and scientific papers that are written daily about (potentially) similar subjects.

In order to progress his work (and as part of the National Center for Supercomputing Applications) Joe has been involved with the Tupelo project. He used the analogy of JDBC (a common Java interface to unite disparate database technologies) to describe how Tupelo is an effort to use the Resource Description Framework (RDF) to unite scientific data repositories.  In particular, the RDF spec describes a graph-data model containing global identifiers and named links as the two key technologies that Tupelo leverages.

This picture of globally unique identifiers and named links reminds me of XAM technology, and in particular how it can be used to trace lineage from one object to its parent.

Software that generates data will come and go; but the data itself strives to remain. Therefore software developers should focus less on their own software (which will eventually become obsolete) and focus on two crucial aspects:

• Focus on how their software interoperates
• Focus on how to annotate the resultant scientific data so it is highly reusable

After the talk I spoke with Joe about XAM technology and its applicability to this area. Within a few weeks the MIT DataSpace proposal (uniting scientific data archives) will be presented to the National Science Foundation in Washington DC as part of the final stages of the NSF DataNet grant process. If approved, it will be interesting to see whether or not technologies like Tupelo and XAM will make their way into the effort.

Steve

http://stevetodd.typepad.com

Twitter: @Steve Todd

EMC Intrapreneur

Data Quest

I recently met two professors from the UK that are on a data quest. They're touring the United States asking questions about scientific research data. On Wednesday afternoon I spent a couple of hours with Harvard University's IIC team (Initiative in Innovative Computing) listening to their pitch and asking them what they are up to.

Malcolm Atkinson is a professor of e-Science at the University of Edinburgh, and David De Roure is a professor of Computer Science at the University of Southampton.

Both are involved in the e-Science initiative, which was introduced as "the intersection/collaboration between scientists and computer scientists".  Data-intensive research by scientists is "emerging as a new paradigm", and one aspect of the e-Science effort is to effectively collaborate on efficient ways to massively process , analyze, store, and share huge amounts of scientific data.

So why the US tour (taking them through New England, Chicago, West Coast, SouthWest US, etc)?  Well, they called it a "fact-finding mission" on the current state of digital research data. They had a large number of questions for the audience, and I've recorded some of them below:

• What direction (and how) should we steer the digital data revolution?
• How do you characterise you community's data requirements?
• How many researchers are using data-intensive methods today, and what limits the adoption rate?
• In what ways is data changing in your community?
• How do we handle all of the different research software stacks that individual research communities are building on their own?

The Challenges of Sharing Research Data

I've written before about my interest in highly-scalable repositories of fixed content, and research data falls into that category. Right down the street from Harvard, in fact, are the researchers at MIT Sloan School of Management, who are requesting funding from the National Science Foundation to research the topic of uniting research archives.

One of the more interesting "research data sharing" challenges mentioned by Malcolm during his talk was the challenge of sharing copyrighted material.  I found out about "illegal database joins". Consider a database query that joins two database tables from different research data stores. If one of these tables contains copyrighted material, then the creation of the resulting "join output" could be illegal.

This is just one example highlighting the challenges of sharing research data. It's one very large, tough nut to crack.

After describing the problem space, Malcolm turned over the presentation to David, who presented what I thought was a very unique and effective way to make progress on this issue.

Sharing Scientific Workflows

If the research world is not ready for all-out, full-access sharing of scientific data repositories, how about sharing the process or methods used in the analysis of data repositories?  What if this sharing of workflows was accomplished through cutting-edge social media technologies?

David then presented an overview of myExperiment, a site dedicated to "finding, using, and sharing" scientific workflows. The website allows users to create and share scientific workflows, join groups, send messages, add comments, and of course search and find other people's workflows.

Social media for scientists. I like it. Scientists can actually re-use workflows and adapt them for their own purposes.  While the input and output data related to the workflows may not be physically available, the conclusions, papers, and theories that result from the workflow can be published along side of it.

In order to make this workflow sharing successful, David went on to say that "credit and attribution" of workflows became one of the key problems to solve. Solving the user requirements for control of intellectual property became a key piece of user adoption.

I went to myExperiment.org and typed in the word "ocean" (oceanography was discussed in EMC's innovation lecture from June), and I found an oceanography group, as well as a workflow that read oceanographic data from a file and sent it to COVE (the collaborative ocean visualization environment).

This type of collaboration and sharing of scientific workflows certainly sets the stage for collaboration and sharing of scientific data repositories.

It was an intriguing presentation and I learned a lot. If you see two professors (with UK accents) crossing the US on a data quest this fall, be sure to give them your data requirements ;>).

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

Cloud Cartography

Photo: Above The Clouds

One of the benefits of the recent partnership with MIT is that all EMC employees are invited to attend (in person or remotely) lectures on a variety of interesting topics. These lectures typically occur on Friday. Sometimes I'll drive in to Cambridge, sometime I'll dial in from my office. This summer I have learned quite a few things about "cloud computing" from a security researcher's point of view.

I dialed into a lecture a couple of weeks ago and was fairly riveted as Dr. Eran Tromer of MIT introduced the term "cloud cartography". I had always operated under the assumption that a user of "cloud computing" should not and would not care about the placement of their VM onto a specific piece of hardware in a compute cloud. Dr. Tromer (and his associates at the University of California at San Diego) have caused me to change my mind.

Take a look at the recently published research paper, cleverly entitled "Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third Party Compute Clouds". Dr. Tromer, Thomas Ristenpart (UCSD), Hovav Shacham(UCSD) and Stefan Savage(UCSD) explore security issues in the compute cloud. Here's a section of their abstract which highlights one of their main points:

"Using the Amazon EC2 service as a case study, we show that it is possible to map the internal cloud infrastructure, identify where a particular target VM is likely to reside, and then instantiate new VMs until one is placed co-resident with the target. We explore how such placement can then be used to mount cross-VM side-channel attacks to extract information from a target VM on the same machine."

What I like best about the paper is the concrete, hands-on experimentation that the team did with Amazon's Elastic Compute Cloud (EC2) service. When a user provides EC2 with a first VM to run in a compute cloud, this VM is placed on an available physical server. Using network probing and a variety of other techniques, the researchers were able to effect/cause the placement of a second VM on the same physical server with 40% accuracy.

If the second VM is malicious, it can attempt to extract information from the first VM (using a variety of methods described in the paper).

  This leads to the introduction of the term "cloud cartography". Cartography is the study and practice of map-making. Cloud cartography is the science of mapping the physical topology of a cloud. Typical users may not care about cloud cartography and where their VM is running. But their adversaries do! They can map the cloud, understand VM placement strategies, induce VM co-residence, and then attempt to abuse the VM locality.

After listening to the lecture and reviewing the paper, it is clear that there are tough problems to solve. However, that's part of the beauty of the lecture series collaboration. My co-workers from RSA (the security division of EMC) are in the same room discussing potential solutions and strategies for these types of problems. They are on top of the issues. Think about the "golden lock" icon on a browser (which means RSA algorithms are in use).  I'm confident that an RSA "golden-lock for the cloud" will become pervasive (just as it has for secure browsing experiences).

As Chuck points out, security within the cloud is currently a huge area of investment. 'Zilla writes about dark clouds.  Like weathermen they're predicting the security future when it comes to cloud.

Because nobody wants to run their compute in a storm cloud. 

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur

MIT and Federated Archives

The first week in February can often be one of those weeks where we here in New England start asking ourselves "why do we live here"? Last week was freezing! Some of us make it through via our love of winter sports. I myself find pond hockey to be one of the greater experiences in life, but there's usually too much snow to shovel by the first week in February. 

During my visit to MIT recently the wind was whipping off the Charles river and into Cambridge. Across the river, however, the Massachusetts version of grounddog day would soon take place. Last Friday, Feb 6, "the truck" left.

Around here, when someone says "the truck" in February, it's obvious what they're talking about.

They're talking about the Red Sox equipment truck leaving for Fort Myers, Florida.

And that, my friends, is why I'm willing to stick things out for a few more months.

Meet the Experts

I've written before about the never-ending software jungle-gym of product permutations that we can build here at EMC. It's actually the theme of this blog.  I work in an Information Playground.

EMC has acquired approximately 40 companies this decade. That's why we have all of this cool technology at our fingertips.

I've also written about the fact that these companies come with architects that can be quickly integrated into the innovative culture here at EMC via our internal social media platform, EMC ONE.  Architects from different domains of expertise collaborate to propose new products.  These architects are world-wide experts in a wide variety of information-based technologies.

Some of the people I work with are so smart it makes my jaw drop.  I wish you could meet them.

Now you can.