Information Playground

The Trouble with MTBDWs

Several weeks ago I attended an announcement at MIT CSAIL regarding the new bigdata@CSAIL initiative. Massachusetts Governor Deval Patrick was there, along with MIT President Susan Hockfield. Myself and several others from EMC listened in as the Governor announced EMC's participation as a founding member of bigdata@CSAIL and an ongoing contributor to the Massachusetts Green High Performance Computing Center.

Big data workloads are important to both initiatives.

In fact, Multi-Tenant Big Data Workloads (MTBDWs) are gaining quite a bit of attention in academia. How can shared, massive data sets be most effectively (and securely) analyzed by multiple tenants in a cloud environment?

Consider the Sloan Digital Sky Survey (billed on its website as "the largest map in the history of the world"). Images from the heavens are input into a data processing pipeline, resulting in a massive amount of raw data, processed images, and meta-data. A variety of scientists can collaborate on the processing of this data, given the publicly accessible interfaces to a directory tree and forms for querying coordinates.

Providing this type of functionality as a service causes me to ask a few questions:

How would this type of collaboration work in a commercialized service provider environment?

How does a service provider enable multi-tenant access to a common data set in a cloud (e.g. using DevPay to access Amazon Public Data sets)?

How can tenant isolation be enforced, or in the case of collaborating tenants, be disabled?

Are the tenants, and/or the data services, all located within VMs?

Can virtualized tenants and/or data services adhere to specific SLAs?

The research community is beginning to explore these questions. Early results were presented at last year's SIGMOD conference in Athens. MIT researchers studied physical consolidation of database workloads onto fewer machines. In their paper "Workload Aware Database Monitoring and Consolidation", the authors stated the following about virtualizing databases (section 9, Conclusion):

Additionally, we show that existing virtual machine technologies are not nearly as effective as our techniques at consolidating database workloads.

The paper goes on to list some of the problems (section 7.4) encountered when virtualizing databases (as opposed to physical consolidation):

• Redundant operations (log writes, log reclamations)
• Over-allocation of RAM
• Excessive context switching
• Less code-sharing between workloads

How then can these issues be addressed to help realize performance gains while allowing for either tenant isolation and/or tenant collaboration for analytic workloads (otherwise known as multi-tenant workload management)?

With a new architecture, of course!

As part of its University R&D program, EMC is not only collaborating with CSAIL but also with the University of Washington (who has data sets of their own). In partnership with University of Washington Professors Magda Balazinska and Bill Howe, the research will attempt to define an architecture that can address the following questions:

• Should tenants always be placed in their own virtual machines?
• Are their benefits to tenant sharing of parallel data processing engines?
• For overlapping data sets between tenants, how should resources be allocated between different tenants and engines?
• What service level agreements make sense for these new scenarios?
• How is elasticity implemented in these new scenarios?

It's exciting to be at the forefront of these discussions, and I'm looking forward to sharing results moving forward.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

Director, EMC Innovation Network

 

 

 

 

 

 

 

 

 

 

AMPed At UC Berkeley

I visited UC Berkeley last week to hear about their AMP Lab for the very first time. For me it was more than just another visit. When I first started working in the industry my boss gave me a copy of the RAID paper from UC Berkeley and said "build this". This gave me a chance to design my very first software system that became the eventual core of the VNX and VNXe product lines.

It turns out that two of the authors of the original RAID paper (Patterson and Katz) sit in the AMP Lab surrounded by students and other faculty.  The AMP Lab is a five year initiative that logically follows from the successful conclusion of the five-year RADLab effort.

I met with Michael Franklin (the Director of AMP Lab) and he said that the RADLab experience was squarely aligned to the advent of cloud computing. In the same way, AMP Lab has squarely positioned the faculty and the students for the advent of Big Data.

Algorithms, Machines, People

Michael shared the main problem that AMP Lab is trying to solve:

The normal application of current technology doesn’t enable users to obtain timely and cost-effective answers of sufficient quality to data driven questions.

This problem statement has caused the team to research new ways of combining the vectors of timeliness, cost, and quality. The focus is on three areas:

1.       Algorithms: improve scale and quality of machine learning and analytics to increase value

2.       Machines: use cloud computing to get value from Big Data and enhance data center infrastructure to cut costs of Big Data Management

3.       People: Leverage human activity and intelligence.

It's the last item (people) that I found most interesting. The team at the AMP Lab is beginning to explore software stacks that call out to people. I've embedded a picture of one of these stacks below (taken from the CrowdDB project page).

For Big Data, some queries may best be satisfied by the crowd. This includes large data sets with incomplete data, queries over visual pictures that may not necessarily be tagged, or queries using "synonyms" (e.g. "IBM" and "Big Blue"). One of the more popular platforms for assigning work tasks to the crowd is Amazon's Mechanical Turk, and the AMP Lab has already started research with that platform.

I'd be interested in experimenting with this type of layered (the PowerPath stack comes to mind).

The portfolio of Big Data research going on at AMP Lab was cool. I walked out of there feeling pretty stoked.

Or should I say AMPed?

Steve

Information Playground

Twitter: @SteveTodd

Director, EMC Innovation Network

 

 

Brazilian Big Data Research Round-up

One of the more interesting ways to discuss Big Data is from the research angle. Chuck pointed out several weeks ago that analytics is not the only theme. As he said: "There's another side to big data that is more about dealing with big files vs. big databases".

I have been paying close attention to Big Data research knowledge expansion across EMC. The two uber-themes (analytics and massive files/dbs) can be broken down into specific research topics. Some topics are being investigated across our global locations. Other topics are on the to-do list. The following list represents different research areas for Big Data (note the strong emphasis on performance):

• Ingest and Export Speeds. Getting the fish from the ocean to the plate as fast as possible is a critical research area. University researchers are still working on this problem nearly a quarter century after RAID.
• Information Storage. Should it be encrypted, compressed, deduplicated, or some combination thereof?
• Application-aware Data Layout. While distributed scale-out architectures and recovery mechanisms are fairly mature, the performance implications of data layout can vary from application to application. This is closely related to workflow (see below).
• Tiering and Mobility. The movement of big data between tiers for performance improvement extends up to the server flash level. The volume of data causes this research topic to overlap with the ingest/export theme.
• Use cases/Workflows. Understanding the applications that need access to Big Data becomes critical for a number of reasons: performance and configuration management are just two of them. Common workflows include:
  • Medical
  • Energy
  • Satellite
  • Video
  • etc.
• Collaboration. All of the scientists working on a data set are not necessarily in the same room or facility. The ability to annotate Big Data is an extremely helpful feature, and can result in loosely coupled Big Data / Small Objects. I'm a big fan of the GreenPlum Chorus approach, but it needs to extend  beyond the enterprise model. Which leads to....
• Geographic Mobility. Big Data captured into one data center may not necessarily stay there. Distributed scientists may want to bring the algorithms to the data and/or vice-versa. 
• Visualization. This may be one of the more fun areas of research to watch. Creative visualization may yield more influence than any other research topic.
• Analytics. As a computer scientist I always loved saying "BubbleSort". Analysis of algorithms lives on.
• IT infrastructures. The intersection of Big Data and Cloud. Storing data of this size and magnitude will impact all five cloud characteristics.
• Preservation.  If Big Data must age like fine wine, then the methods and processes used by digital curators can be applied to the field of Big Data.
• Provenance.  Where did all this Big Data come from?

Often times EMC acquires a company to enter an emerging market.

In Brazil, EMC has invested in a new research facility to expand research coverage of Big Data. This initiative will clearly result in knowledge expansion in the area of Oil and Gas workflows, but the research will also certainly spill over (no pun intended) into some of these other areas as well.

I've been involved with provenance research in the past and will continue to post updates as we move the research forward in many of these areas.

Steve

Information Playground

Twitter: @SteveTodd

EMC Intrapreneur