Information Playground

Governed Placement Services and Data Value

The article Metadata Lakes and Data Value, written in February of this year,stated the following:

...as data architects go through the process of moving silo'ed data sets into a Business Data Lake, they should simultaneously build a governance structure known as a Metadata Lake.

The diagram below emphasizes that the Metadata Lake should contain, among other things, application policy metadata and infrastructure metadata.

 

The idea for creating a Metadata Lake was in response to the use case of Data Insurance. A data set insurer would need access to a Data Audit and Inventory System (DAIS). In the context of data insurance I wrote:

A Metadata Lake is a form of a Data Audit and Inventory System (DAIS). If a data insurer discovers that this structure doesn't exist during their initial visit, then some form of it must surely be created. 

A Metadata lake can certainly function as a DAIS, but in the case of placing an application (and its data) onto the appropriate infrastructure based on data value, a Metadata Lake can be used for much more than just a data inventory system.

For example, in my last post I highlighted how tools like CloudFoundry can perform manual application placement by having an operator choose to place an app on one of seven cloud choices (e.g. VMware, Amazon, Google):

Given the current state of the art, the diagram above raises several questions:

• How can the industry move towards an automated approach where application deployment is constrained by the anticipated economic value?  
• If the line of business for the app can articulate the anticipated value, can tools such as CloudFoundry automatically constrain application placement?
• Can the deployment layer contain business logic that matches line of business value versus trusted infrastructure services?

If a Metadata Lake contains (at a minimum) the two types of attributes depicted below, not only will it serve as a Data Audit and Inventory System, but it also can be referenced by PaaS platforms that wish to introduce Governed Placement Services into the deployment process.

The diagram highlights two forms of metadata that must be added into a Metadata Lake:

1. Trust Services. These services are essentially the trust dimensions, or trust taxonomy, that have been surfaced up by all members of a cloud infrastructure, and are now being advertised as services containing different level of trust support by all of the candidates depicted in the diagram above.
2. Data Policies. These policies represent business statements about the value (and therefore the level of trust required) of the data set being generated by the application. Example policies could include the need for data retention, encryption, immutability, etc. 

As these policies and services are stored in the Metadata Lake (along with an API for accessing them), it allows for the development of new Governed Placement Services to be added into a PaaS framework.

In an upcoming post I will explain some of the flow and logic that can allow tools like CloudFoundry to change their current mode of operator-specified placement and shift to an automated, governed placement approach.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Data Value and Placement Constraint

In my last few posts I've been exploring the following thesis:

As the economic value of data rises, the need for it to be stored on trusted infrastructure rises as well.

The economic value of data can be thought of as "anticipated future value".  When a company writes a new application, there is an expectation that the app will bring monetary benefit to the business. In today's environment, the business manually chooses a specific data center (e.g. cloud) onto which the application (and its data) will be placed based on anticipated future value.

In the long term I have two problems with this approach:

• What happens (after the deployment) if the actual future value is wildly different from what was anticipated? It is probable that the wrong infrastructure has been chosen and the business is likely over-spending/over-protecting or under-spending/under-protecting.
• Many companies have thousands of applications, and manual placement (and/or subsequent migration after misplacement) will no longer be a feasible approach. 

Technologies like CloudFoundry and alternative container approaches can help with the first concern. One of the main value propositions of these technologies is application portability; if the application has been placed onto the wrong cloud, it does not need to be re-written to be re-positioned onto another cloud framework.

The diagram below highlights this device driver approach to app/data deployment.

The bottom layer of this picture depicts seven different cloud infrastructure targets (e.g. Amazon, Google, Azure, VMware, etc).  With today's technologies, tools like CloudFoundry can accept an application for deployment (e.g. a Hadoop application) and proceed to take care of the deployment onto a specific cloud.  In this example the choice and placement of the application (and its data) is manual; an operator needs to feed the target cloud into the CloudFoundry framework (previously I wrote a post containing a graphic that highlights operator selection). 

How can the industry move towards an automated approach where application deployment is constrained by the anticipated economic value?  In other words, if the line of business for the app can articulate the anticipated value, can tools such as CloudFoundry automatically constrain application placement? Can it contain business logic that matches line of business value versus trusted infrastructure services?

The answer is yes, and in a future post I will begin to describe the technology building blocks that are necessary to construct such a solution.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Moving Data Value Up the Stack

As I stated in my last post:

As the economic value of data rises, the need for it to be stored on trusted infrastructure rises as well.

This statement brings to mind a number of questions, including:

• How is the economic value of data calculated, initially and over time?
• How is that data mapped to the appropriate layer of trusted storage based on that value?

The diagram below is a helpful start in order to answer those questions. The storage infrastructure on which valuable data resides must begin to surface a more robust "trust taxonomy".

This surfacing of a trust taxonomy is likely to occur via a set of software-defined storage APIs (e.g. EMC's ViPR).  Nikhil Sharma formally introduced the taxonomy in his Reflections blog post. 

How then does the mapping of "valued data" to trusted infrastructure occur?

I've written before about the mapping of applications and data to a software-defined data center. Most of this thinking was based on application performance. Tools such as VMware have long had the ability to:

• Call a software API to discover the number of servers, CPUs, memory, etc., and perform a workload placement of the app onto the correct server (and monitor that placement over time).
• With the acquisition of Nicira several years ago, VMware now had a software API to discover the number of network switches, ports, allocated bandwidth, etc., and VMware could not only provision the correct server but also automatically provision the correct network path.
• With the advent of software-defined storage APIs, VMware can now call a storage API to discover the number of storage devices, their capabilities, amount allocated, etc.  This is the final piece of the puzzle for full SDDC automation.

I like to use the video below to highlight this automation. 

 

This model is performance-based. It uses mathematical modeling techniques to place workloads into the best-fit performance configuration.

For the data value conversation, however, the industry needs to start thinking about trust mappings. In other words, cloud management and orchestration stacks need to bubble up the trust characteristics (in the form of available cloud trust services) from all elements of the data center: servers, networks, and storage:

By using this approach the orchestration framework is bringing the trust taxonomy surfaced by the storage layer (as well as other data center components) closer to the application level.

In future posts I will discuss the required marriage of SDDC Trust Services and applications, especially in the context of data value.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Tying Data Value to Data Trust

After thinking about the concept of Data Value, and in particular data insurance, I've started to wonder if the following is true:

As the economic value of data rises, the need for it to be stored on trusted infrastructure rises as well.

In my last post I quoted my colleague Nikhil Sharma's recent blog post about Trusted Infrastructure:

Trusted Infrastructure will need an open abstraction layer, Trust APIs, for use in higher level stacks like Hypervisor or Cloud OS. 

I understand and agree with his thinking. The marriage between data value and trusted infrastructure means that higher level data placement software will need access to Trust APIs at the hypervisor and/or cloud layer. These Trust APIs will need to present a "trusted" view of the overall data center infrastructure (e.g. server, network, storage, security, etc). These views will take into account the full spectrum of trust dimensions:

How will the storage layer participate in surfacing these trust dimensions?  After all, the economic value of data is calculated by analyzing each data set as it exists on specific storage devices. How does one describe the trust level of those devices (and any corresponding control-plane software)? One key trust dimension here is transparency. Once again I relied on Nikhil's point of view on this topic:

The standardized trust transparency needs to be declarative in nature. Storage will declare "these are the trust features that I have". 

In many of today's data center architectures this feature is missing. The figure below highlights a  basic diagram of a software-defined storage layer which translates specific application storage requirements into a correctly provisioned storage repository for that application.

In order to surface trust characteristics up through the software-defined storage layer, devices need to advertise a standardized trust taxonomy:

 

There are storage systems today that already surface a variety of trust capabilities, but none that run the full spectrum of the six trust dimensions highlighted above.

In addition to the storage layer, this trust taxonomy must also be communicated by non-storage devices that are part of the overall cloud implementation.

I will dive into tying trust to data value at the Software-Defined Data Center (SDDC) layer in a future post.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Choosing Trusted Infrastructure

In this on-going series about data valuation I've chosen to focus on data set insurance as a primary use case. In recent posts I've:

• Introduced our research partner
• Introduced the data insurance use case
• Reflected on positive and negative data valuation
• Reflected on data insurance people and processes
• Explored data set identification for insuring data
• Proposed metadata lakes as a data insurance enabler
• Introduced the concept of trust inspections for data insurance

The last bullet item (trust inspections) is closely related to the topic of underwriting. In this post I'd like to highlight the role that trust taxonomies will play as the industry moves in the direction of data set underwriting and/or cyber-insurance.

My general thesis has been as follows: insurers need a straightforward and audit-able way to evaluate the data protection levels of individual data sets. 

This approach would be greatly facilitated by a complete, thorough, and industry standard taxonomy for describing trusted infrastructure capabilities.

My colleague Nikhil Sharma has already written about an emerging trust taxonomy that is taking shape within EMC, and I've re-posted this taxonomy via the diagram below.

This diagram highlights that a trusted infrastructure goes beyond the definitions of security, availability, and recovery. In particular, the data insurance and trust inspection use case will require a high level of transparency in the form of (a) assessing the infrastructure, (b) creating ongoing reports about data protection, and (c) disclosing current and previous levels of protection on a moment's notice.

Nikhil closes his article by stating the following:

Trusted Infrastructure will need an open abstraction layer, Trust APIs, for use in higher level stacks like Hypervisor or Cloud OS.

I certainly agree with this statement. However, the higher-level abstraction layer requires lower-level standardized trust transparency as well (e.g. storage, network, server, etc).

In an upcoming post I will discuss the need for adding this robust trust taxonomy into a software-defined storage layer.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

The Data Insurer and Trust Inspections

For several weeks I've been writing about Data Valuation with a goal of considering any impacts on IT infrastructure. 

I introduced the table below as a framework for discussing data insurance from the perspective of people and processes. Insuring data is an example of data valuation, and much can be learned by diving into the potential underwriting process depicted below.

In a recent post I mentioned that it is already common for an insurance agent to visit a data center for the purpose of insuring against disaster.

What is uncommon, however, is the role of a data insurer. This role would specifically focus on data set insurance (as opposed to data center insurance).  

In this post I'd like to focus on the next step in the table below and discuss the need for a data insurer to carry out trust inspections.

I have already spent a good deal of time discussing scenario #1: the data set identification process. I have also proposed a set of people that would likely work with a data insurer.  In general:

• The insurer would begin the process by asking Where's The Data.
• The insured would share their Data Audit and Inventory System (DAIS) in response. If they don't have one, they should consider building a super-set of a DAIS known as a Metadata Lake.

I reasoned that a Data Architect would be a great candidate to participate in this initial phase of the data insurance process. I also reasoned that a Chief Data Officer could be another appropriate liaison; they could either oversee the effort or work closely with corporate Data Architects.

With an identified data set, the next step in the data underwriting process would be to assess the level of risk by understanding the current data protection level.

This "trust inspection" process, like the role of data insurer itself, is emerging. There are a few things that stand out as being important in terms of generating a risk profile for a data set.

• The faster that a data insurer can associate a data protection level to a data set, the more efficient the whole process becomes.
• It would be helpful to flag the data set as "insured" or "considered for insurance".
• The association of data protection level and data set needs to be bound together and audit-able for the life of the insurance policy.
• Migration of insured data sets would need to take into account the (a) data protection level of the target, and (b) the data protection level of any insurance policy.

The topic of trust inspections has strong ties to new definitions of trusted infrastructure. In an upcoming post I will explore these ties and consider their relevance to next generation data center architectures.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Mobile World Changes

Several years ago EMC hired a new Chief Technology Officer: John Roese.  John brought with him decades of networking experience, including CTO roles at companies such as Cabletron and Nortel. Within a few months he participated in Mobile World Congress 2013 and heard the Telco/Service Provider industry communicate their need for dramatic and radical change. In 2014 he attended the same conference and saw an acceleration of transformation activity (see his Reflections post from last year). As he participates in his 3rd consecutive MWC in Barcelona next week, I thought I'd take a moment to discuss how John (and other networking experts he has brought into the company) have influenced several key areas of the industry's ever-stretching IT stack.

In previous posts I've used the diagram on the left to highlight that the increasing application distance from data has placed an enormous strain on IT architectures.

The diagram is meant to convey that an application I/O can originate from a mobile device, flow over a carrier or cloud network, enter into a virtualization layer that exists on an application server, move through a multi-pathing or routing environment, hop over network switches that are part of a SAN, and then arrive at a storage system which has its own layers of software and hardware pathing that lead to the persistent media.

I've framed the dialogue as follows:

• We have evolved from Application Nearness.
• We can chart application distance on an application axis.
• This axis is not only stretching out but is simultaneously shrinking back again.

In this post I'd like to drill into the "Carrier/Cloud" portion of the stack.  If you were to double-click on it you would land squarely in the middle of Mobile World Congress 2015. The infrastructure supporting this layer has lagged behind the enterprise. As a result the cost of maintaining and supporting Telco and Service Provider infrastructure has outpaced revenue growth. In response the Telco/SP ecosystem is transforming to agile, virtualized, cloud-based IT stacks (just like the Enterprise did).

However, the Telco/SP ecosystem has a unique set of problems that most enterprise ecosystems did not have to solve.  Over the last few years the CTO Office at EMC has been collecting and enumerating these use cases and driving change into our product lines in order to make them "SP-ready". 

I'd like to highlight the different ways that EMC products have changed (or are changing) in order to fuel SP transformation in a similar fashion to how the Journey to the Private Cloud transformation altered the enterprise several years ago.

• Syncplicity APIs are becoming more SP-friendly in order to support requests for multi-tenant provisioning, billing enablement, and custom application development.
• Isilon is being virtualized in order to run on a variety of different scale-out HW platforms. This functionality is in direct response to the scale-out infrastructure needs of SPs. The SP market has also driven Isilon to add support for multi-tenant DNS capability.
• Both Isilon and VNX have been driven to add multiple, routable IP addresses per port, whether it be iSCSI or NFS.
• VNX has also been asked to improve QoS in the areas of IO and throughput rate limiting that are critical for multi-tenant environments.
• VMAX is adding CostOfService/LineOfService capabilities and is also being driven to add iSCSI MT support.
• XtremIO has also been working on iSCSI multi-tenancy support and IO/bandwidth rate limiting.

In my opinion John Roese's arrival into EMC in 2012 occurred at just the right time; his influence and understanding of this market had an immediate impact across EMC's product lines and has facilitated SP-ready software and systems. For an introduction to his thoughts on re-engineering the service provider ecosystem, see John's discussion below:

 

Thanks to my EMC colleague Edgar StPierre for educating me on this topic.  Additionally, I recommend checking out David Frattura's discussion on this topic at MWC 2015 on Tuesday at 4PM:

EMC Reinventing the Cloud Service Provider Through Cloud Transformation by David Frattura

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow