Information Playground

Visualizing University Research Activities

During a lecture at EMC’s 2^nd Annual University Day on Monday, I held a dialogue with faculty and students gathered at our Santa Clara campus. I described how EMC uses an analytics framework (Pivotal/Greenplum) to accelerate the innovation that emerges from our global academic research partners. In particular, I highlighted the following capabilities of our innovation analytics platform:

• A visualization of the “types” of research currently active in our portfolio (e.g. solid state storage, analytics, etc).
• A visualization of the “types” of research by region (e.g. where in the world do we research compression technology?)
• Who are EMC’s key researchers in any given region?
• Which researchers are the best at transferring knowledge out of their region?
• For any given EMC researcher, what type(s) of research do they conduct?
• What is the complete list of EMC employees, per region, that are involved in any form of university research?
• How can global EMC employees advance their own ideas by locating relevant university research?
• How do we augment university research with other external employee connections (e.g. programmatically leverage their Twitter connections)

In this post I’d like to focus on the first bullet. EMC has dozens of university research partnerships worldwide. How do we dynamically visualize the current areas of exploration that are occurring across the globe at any given point in time? How can we determine which strategic research areas have strong coverage and which areas may have no coverage at all?

These questions are currently answered through our use of the functionality provided within the Stanford Topic Modeling Toolbox. The diagram below helps explain our use of this tool:

The Topic Modeling Toolbox analyzes the analytic repository containing university research activity.  Data Scientists within EMC (working as part of our EMC Labs China team) categorize these research activities by providing the toolbox with a number (e.g. N = 25). The toolbox runs algorithms that classify each research activity into one of a number of different buckets using the toolbox algorithm.

Once the toolkit has taken a pass at every document, the bar graph above shows the level of activity for each “class” of research initiative. I asked our Data Science team to provide a simple word cloud algorithm across each category, and it is fairly easy to see at a high level that Topic 01 has a cloud focus, while Topic 12 has a Big Data focus.

Furthermore, if the data above was analyzed in a given time frame (e.g. the first half of 2013), Topic 22 would be viewed as “most active”, while Topics 02 and 23 would qualify as “least active”.  This may or may not be cause for concern depending on the nature of the work.  In fact, given topics can be broken down into the “nature of the engagement”, which is highlighted below:

While this type of data gives EMC a great idea about “what” we are researching, it doesn’t provide any data at all about “where”.

I will cover EMC’s approach to solving this problem in a future post.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Analyzing University Research

Today I attended EMC’s 2^nd Annual University Day in Santa Clara, California.  A large number of schools were represented from all over the United States, including:

• UC San Diego
• UC Irvine
• UC Santa Cruz
• Northeastern University
• Minnesota
• Carnegie Mellon University
• University of Wisconsin
• Case Western
• Florida International University
• University of Utah
• Harvard
• University of Rochester
• Stony Brook University
• Princeton University

The agenda for the day included discussions on challenging high-tech issues in next generation data centers, including new developments in solid state storage. EMC Distinguished Engineer Jeroen VanRotterdam led an interesting dialogue examining the current state of relationships between Industry and Academia.

Greg Ganger, CMU Professor and Director of the Parallel Data Lab, gave the Academic Keynote during the afternoon session. His keynote was followed by the annual poster session, in which nine students competed for first prize.

For this post, however, I’d like to summarize a discussion I led just before lunch, in which I asked the students the following question:

“How would you manage EMC’s global university research portfolio?”

Their answer was loud and clear: "We don't know!". I responded that the answer was a fair one; it's a hard problem to solve. I then shared our company's approach of using EMC’s own analytic products (e.g. Pivotal/Greenplum) to perform global analytics across all academic research partners.  In order to highlight the global span and scope of our research initiatives, I shared the following map:

This map is dynamically generated. While it doesn’t represent every university research partnership EMC has across the globe, it’s pretty close.  The map is the result of nearly two years of collaboration across all of the countries that register their research engagements. The larger the circle, the more activity is being reported from the region.

What types of analysis can be run against a database containing research activities? During my talk I described the current reports enabled by our analytics framework:

• A visualization of the “types” of research currently active in our portfolio (e.g. solid state storage, analytics, etc).
• A visualization of the “types” of research by region (e.g. where in the world do we research compression technology?)
• Who are EMC’s key researchers in any given region?
• Which researchers are the best at transferring knowledge out of their region?
• For any given EMC researcher, what type(s) of research do they conduct?
• What is the complete list of EMC employees, per region, that are involved in any form of university research?
• How can global EMC employees advance their own ideas by locating relevant university research?
• How do we augment university research with other external employee connections (e.g. programmatically leverage their Twitter connections)

The talk was well-received. The faculty and students that attended got a good feel for the framework that EMC uses to impact our own business by expanding our knowledge with local university partners. 

In future posts I will dive in many of the items above in more detail to specifically describe how analytics are leveraged to improve EMC’s university research results.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Structural Holes and Sociograms

In a previous post I introduced an interesting new research project that my company (EMC) is working on with NUI Galway and Babson. During that post I quoted Professor Eoin Whelan's thoughts on Structural Holes:

Ron Burt’s theory of structural holes has proven to be influential in explaining how innovation transpires.  Burt proposes that gaps in a social network, structural holes, create brokerage opportunities.  A structural hole indicates that the people on either side of the hole circulate in different flows of information and advantages accrue to those individuals whose relationships span the structural hole.  

In today’s digitally connected world, many social relationships are formed and maintained through social media.  The reason for the academic partnership is to examine if innovation theories such as Burt’s structural holes apply to networks formed through social media (and specifically Twitter).  The researchers are using social network analysis techniques to find correlations between thousands of employee ideas submitted into our Innovation Roadmap framework, and the structure of the submitter Twitter networks.  The analysis of this data continues but initial findings suggest that the structural holes theory also applies to Twitter.  People with more disconnected twitter networks tend to submit better ideas, as determined by the average number of positive votes received by peers per idea.  

I ask Eoin to provide some graphics that highlight their work so far, and he supplied two "sociograms", which are described as follows:

The results can be better explained by examining the Twitter ‘sociograms’ of two EMC employees below.  Even though both employees roughly follow the same number of Twitter handles, employee A’s network is far more diverse.  In other words, the people employee A follows on Twitter are for the most part not following each other.  Mathematically, we can determine this level of diversity using the fragmentation ratio which for employee A is 87%.  The data is showing that highly fragmented Twitter networks, like employee A’s, are better for ideation.  In contrast, employee B’s Twitter network is quite compact.  Those in employee B’s network are nearly all following each other, hence a low fragmentation rate of 12%.  Such cohesive networks provide more redundant information, which the data shows is negatively correlated with ideation.

In reviewing these preliminary research results inside EMC, Distinguished Engineer John Cardente pointed out that the structural holes concept is closely related to the Betweenness network attribute that we have socialized previously. Specifically, network nodes that span structural holes tend to have high betweenness scores.

I'll continue to publish more of these results as the research moves forward.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

 

Researching Different Minds Thinking Greatly

Several weeks ago I published a blog post (Different Minds Think Greatly) that explored the topic of cognitive diversity and innovation. At the time, I had read a Techonomy article by John Hagel and John Seely Brown, who basically asserted that “too much like-mindedness hurts companies”, and I quoted the following:

Organizations that host a diverse and broad range of members have a resilience that results from cross pollination.

As part of the article I echoed my agreement with this assertion and referred to some Social Network Analysis from my own company (EMC). The data that we modeled highlighted that a good degree of geographic diversity can result in higher quality ideas.  “Higher quality ideas” are typically defined as ideas that receive a high score from judges in our yearly Innovation Roadmap program (especially ideas that reach finalist or funded status).

Our data shows that when diverse minds from different cultures collaborate on new approaches, good ideas result.  The conclusions we drew from this analysis have resulted in behavioral change at EMC. Most notably we’ve formed a global “Innovation Best Practices Community” in order to intentionally stimulate this behavior.

After publishing some of our findings, we were approached by two universities on an interesting joint research project. They asked if we wouldn’t mind sharing a filtered view of our employee  idea activity in order to correlate it against the public Twitter profiles of these same people. Professors Eoin Whelan (NUI Galway) and Salvatore Parise (Babson) invited us to focus on an offshoot of cognitive diversity known as “structural holes”.  Eoin explains structural holes in the following manner.

Ron Burt’s theory of structural holes has proven to be influential in explaining how innovation transpires.  Burt proposes that gaps in a social network, structural holes, create brokerage opportunities.  A structural hole indicates that the people on either side of the hole circulate in different flows of information and advantages accrue to those individuals whose relationships span the structural hole.  In his best selling book The Tipping Point, Malcolm Gladwell argues that the success of Paul Revere’s midnight ride was due to his quite diverse social networks – ranging from hunting and card playing to theatre and business. Therefore, he knew which doors to knock on when arriving in a town.  Network brokers like Revere not only disseminate information more broadly, they also benefit by receiving a greater novelty of information from their diverse social contacts.  Indeed, studies within organizations have shown that employees, teams, and even entire companies with more diverse network connections tend to be more innovative.

As a result of our conversation, we polled our global Innovation Best Practices community and asked idea submitters to voluntarily share their Twitter handles with Eoin and Sal. We packaged up the Twitter handles with the employees’ corresponding level of innovation activity over a period of several years.

This research has been ongoing for several months, and in an upcoming post I plan to share some of the results and what it might mean for our organization.  Before I do, however, I’d like to discuss this Data Science project in the context of Phase 1 of the Data Analytics Life Cycle: Hypothesis Generation.

Our hypothesis could be stated as follows:

EMC employees with diverse external Twitter networks submit higher quality ideas.

By “diverse” we mean “disconnected” or “fragmented”. In other words, employees that follow people that are not “like-minded” tend to submit better ideas due to the diversity of their network.

If this hypothesis proves to be true, we can brainstorm ways of stimulating additional behavioral change via encouraging our employees to fragment their Twitter networks. Proving the hypothesis involves iterating through the additional phases of the Life Cycle (e.g. Data Prep, Data Modeling, etc).

In future posts I will share the results of the modeling exercise conducted by Eoin and Sal.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Pivotal: Extending to a 3rd Platform Analytics Strategy

In previous posts I summarized EMC CTO John Roese's key message from EMC World: 

Many of the new products announced at EMC World allow existing enterprise customers to extend their data centers to support IDC's definition of the third platform (mobile, social, cloud, big data).

John's statement is highlighted by the graphic below.  Note that this is a health care use case but others are equally relevant (e.g. the telecommunications use case described below). The first two columns on the left highlight EMC enterprise customers who augmented their secure, structured-data infrastructures (e.g. VMAX, VNX) with secure, unstructured data products (e.g. Isilon, Atmos). The third column highlights the new set of products that help enterprise customers build 3rd platform architectures that can also handle massive amounts of consumer data.

It's not enough to be a provider of this infrastructure; enterprise customers also require a comprehensive strategy to analyze data across structured, unstructured, and consumer boundaries (no matter where the data lives). In fact, John mentioned a specific use case that highlights an analytics success story spanning structured, unstructured, and consumer data.

The use case involves a wireless carrier trying to prevent customer churn (losing a customer).   Anything the carrier can do to improve the customer experience (and prevent defections) is a win. Carriers have hundreds of millions of customers, and they recognize that their customers often feel like "just another number". For example, if a mobile customer experiences dropped calls, they will eventually get fed up and sign up with another carrier.

Dropped calls fall into the third category above (consumer data).  Of all the consumer data streaming into the carrier's network environment, dropped calls are often a blip in a sea of activity, immediately drowning into an unrecognized event.

But what if this real-time, streaming dropped call event could be immediately correlated against the rich, historical customer data contained in the carrier's existing data center? Could an apology instantly be texted to the customer? Followed by an opportunity to reconnect at no charge? Could additional goods or services be subsequently offered to those customers that have proven loyal to the carrier (or paid their bills on time) for many years? 

Adding this capability requires a unified analytics strategy like Pivotal.  In fact, Paul Maritz highlighted just such a use case as part of a Pivotal deployment at a telecommunications service provider earlier this year.  This hybrid analytics approach works best when it couples a data-grid and/or CEP approach with a multi-tiered flash deployment. Both of these strategies (Flash and Pivotal) are depicted below as spanning the entire infrastructure.

John concluded his thoughts on the Pivotal strategy by stating that what's good for the enterprise is good for the service provider/carrier. I'll dive deeper into some of the EMC World announcements as they relate to service provider and carrier in future posts.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Related articles

New Toolsets for the 3rd Platform

Extending the Enterprise to a 3rd Platform Architecture

Extending the Y-Axis for 3rd Platform Performance

A Flash Strategy: The Hot Edge and The Cold Core

A Flash Strategy: The Hot Edge and The Cold Core

In my last post I displayed the following image which describes how EMC's Flash Strategy cuts across an enterprise IT infrastructure that manages structured, unstructured, and consumer data.

From an EMC product perspective, flash technology has moved through the following progression:

• It's been more than 5 years since the first shipments of Enterprise Flash Drives within an array that also supports spinning disk drives.  High-end, latency-critical applications were the first set of workloads that drove the introduction of EFDs into Symmetrix. The company has subsequently embarked on an initiative to drive flash technology further and further down-market.
• As part of this initiative, CLARiiON and Celerra (now known as VNX) subsequently added Flash drive support and the term hybrid array was officially in vogue. 
• Hybrid arrays containing flash technology were followed by EMC's introduction of flash in the server layer. This technology is known as XtremSF.
• At EMC World last month Zahid covered the status of EMC's All-Flash array, XtremIO.

This flash product set, according to EMC CTO John Roese, also serves as a strategic bridge that enables existing enterprise customers to support IDC's vision of a 3rd platform architecture (supporting mobile, social, cloud, and big data). IDC's third platform comes with a new class of applications that expect sub-millisecond response times from the storage ecosystem.

How can customers deploy flash in a strategic way that supports the traditional workloads as well as the new ones?

David Goulden answered that question recently at the Strategic Forum hosted by EMC and VMware. What I found interesting about his presentation was that that that the hybrid array has evolved into a central arbiter for the "hot edge" of application data and the "cold core" of less-frequently accessed data. The following graphic paints a great picture of this strategy:

 

Up until now, hybrid arrays have allowed applications to keep the "hot edge" of their data set (the most recently accessed and performance-critical data) on flash, and the "colder core" (the data that is less frequently accessed) on spinning disk and/or archival/backup tiers.

For next generation data-centers (e.g. IDC's third platform vision), the hybrid array is still used, but DC architects can deploy a comprehensive flash strategy to handle workload variety using the following steps:

• Create a storage pool at the server level by deploying XtremSF capabilities in a set of servers. Storage allocated from this pool will typically be assigned to the workloads with the most performance-intensive requirements.
• The XtremSW suite will be leveraged to accomplish this pooling. Note that XtremSW can also be leveraged to aggregate and pool third party flash storage as well, offering choice to data center admins.
• Create an XtremIO all-flash pool for those workloads that have similar high-performance requirements, but also demand higher availability, resiliency, and service levels.
• Deploy the hybrid array technology and use FAST as the tiering mechanism that senses shifting workload traffic and transfers the data between the all-flash pools and the hybrid array.
• Rely on the hybrid-array's existing FAST capability to move data between internal flash drives and the variety of spinning disk tiers within the hybrid array.
• Deploy object-based pools and/or backup pools as targets for the "cold core" (less frequently accessed data)
• Rely on the hybrid array to move less frequently used data to the "cold core" pools.

One final point about this flash strategy for third platform architectures: you'll notice that the first three pools are "block-based".  The server flash is a block interface, the all-flash array is block-based, and the hybrid array is block-based.  

What about flash support for workloads generating file, object, or HDFS output?

In the end, all of these protocols bubble down to a block-based interface supporting bit persistence and resiliency. Therefore the flash strategy cuts across workloads that leverage any and all of the protocols listed above.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

 

Extending the Y-Axis for 3rd Platform Performance

In my last post I summarized EMC CTO John Roese's key message from EMC World: 

Many of the new products announced at EMC World allow existing enterprise customers to extend their data centers to support IDC's definition of the third platform (mobile, social, cloud, big data).

I used the diagram below to highlight some of these new products and how they bridge the gap between consumer data and existing enterprise assets:

John also mentioned a critical aspect that doesn't receive much attention: new application workloads that must run on the third platform. Workloads for storage are often characterized based on service levels (low to high) and storage characteristics (high capacity or high performance). David Goulden described workloads in more detail in his recent blog post. The diagram below considers several different workload types that map onto four quadrants:

This picture highlights, for example, a "higher-performance, higher service level" application (NYSE) along with a "lower-service level, higher-capacity" application (YouTube).  YouTube, for example, requires high-capacity, but users will settle for internet response times when it comes to performance. (e.g. the user may have to wait). In addition, losing a video is not necessarily a disaster, which further explains the "low service level" classification for YouTube.

I'd like to focus on the top part of the Y-axis (performance) and discuss how this axis continues to extend higher and higher. EMC's customers are beginning to bring a large set of newer applications into their data centers. They often have I/O response-time characteristics just beyond the capabilities of the disk arrays currently installed. These new workloads expect millions of I/Os to average sub-millisecond response times. The diagram below highlights some typical applications that fall into this category.

On the left-hand side "data consistency" is low, but the performance (sub-millisecond response time) is high.  On the right-hand side, VDI requires lightning-fast response times, but also expects that the storage infrastructure is rock-solid (a high service level).

And herein lies the challenge of implementing a third platform architecture: existing data centers not only have to handle the explosion of mobile consumer data, but they must also satisfy challenging new performance requirements. 

In other words, the Y-axis will grow higher across more and more workloads.

John's explanation of this phenomena is quite interesting:

Transactional characteristics in the past have been bounded by the number of employees of the enterprise. In the 3rd platform, transaction response times must scale to the number of customers of the enterprise.

As I've mentioned previously, the third platform must support billions of customers.

Most people think of EMC's flash portfolio as a set of "products". John refers to flash technology as a third platform "strategy" that cuts across all of EMC's products:

One of the expectations of a next generation platform is that it doesn't just have more diversity of workloads, but it has increased performance characteristics for any and all of those workloads. In future posts I'll dive down into more detail about how the EMC Flash Strategy cuts across workloads horizontally, allowing for a third platform architecture that satisfies the performance needs of emerging workloads.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

 

New Toolsets for the 3rd Platform

I've been blogging my way through some of the vision and messaging unveiled by John Roese at EMC World several weeks ago. One of John's main themes was that of "Arming the Enterprise" for new Cloud/Big Data/Mobile/Social workloads.  These new workloads are driving the industry towards what IDC refers to as the "3rd Platform".

In my last post I shared the diagram below:

This diagram is meant to highlight EMC's portfolio evolution to accomodate evolving workloads. The use case above is Health Care, but the reality is that many different verticals (e.g. Financial) have undergone a similar transition. Below is a description of the workload and technology solutions (going from left to right over time):

• Terabytes of high performance, highly reliable, and heavily regulated storage devices (VNX, VMAX, VPLEX) essentially "run the business" inside a hospital, with items like Electronic Health Records (EHR) protected by RSA.
• Hospital workloads expand to digital media applications such as telemedicine and digital pathology. EMC's portfolio adds Isilon and/or Atmos support at a lower price point. Confidentiality is still preserved using RSA technology. The previous workloads (e.g. EHR) must exist alongside the new technology.
• The enterprise IT infrastructure must extend the boundary of their data center to manage and incorporate huge amounts of consumer data. Once again, the EMC portfolio has expanded to incorporate these new workloads in a secure fashion. During EMC World, there was a heavy emphasis on new offerings that assist the enterprise to address these new workloads: RSA, Isilon and Atmos, ViPR, and Syncplicity.

The characteristics of these new workloads, and the ramifications on the IT infrastructure, are as follows:

• The amount of consumer data has evolved to petabytes.
• There is enormous pricing pressure to manage that much capacity. Analytics upon that raw data generates additional results that must be stored and managed.
• There is a strong desire to analyze consumer data in the context of the existing enterprise information assets (e.g. EHR, Digital Pathology). This improves end-to-end wellness management.
• Heterogeneous storage, ranging from high-end to consumer storage, must be managed in a way that significantly reduces OPEX.
• Consumers will expect a "mobile-friendly" user interface that can seamlessly move data into the enterprise in a protected manner.
• Hundreds of millions of users will participate in this third platform ecosystem. The overall system must be able to detect malicious behavioral patterns.
• Customers wish to add Hadoop-style analytics to their toolkits.

Let's step through some of the recent additions and enhancements to the EMC portfolio that enable enterprise customers to extend their reach and meet the requirements of the new workloads:

Syncplicity

The Syncplicity acquisition (May of 2012) serves as a significant milestone on the road to the mobile requirements of a third platform achitecture. After the acquisition, Syncplicity was modified to interoperate with enterprise home directories sitting on top of products such as VNX and Isilon (i.e. the enterprise assets already existing within the data center). This allows enterprise employees and/or customers to take any enterprise information and mobilize it onto any type of portable device and vice-versa. The user interface is consumer-friendly and natural.  In the context of health care, E-wellness and M-health applications now have a path to interoperability and exchange with the enterprise infrastructure.

RSA

RSA technologies have shown up in every phase of workload transition (as evidenced by the diagram above). Enterprise customers have asked RSA to extend its reach, and the RSA portfolio has also undergone an evolution in two key areas.  Firstly, they've added support for security analytics. Third platform architectures will have hundreds of millions of events to analyze. As a result, RSA pushed the envelope of Security Information and Event Management (SIEM) by using analytics as a way to detect intrusions.  Secondly, they have added new products such as SilverTail. This product uses real time web session intelligence and behavioral analysis to defend against cyber crime.  RSA has augmented its strength in the enterprise with technologies that clearly have business-to-consumer benefits.

ViPR

ViPR will extend the enterprise storage footprint to accomodate new protocols and APIs such as S3 and HDFS (which were not required by the previous enterprise workloads). What does this mean? The customer can actually open up traditional enterprise storage products like VMAX, VNX, and Isilon to protocols such as S3, without modifying those traditional systems. You may not balance protocols such as S3 on top of those systems by default, but there may be other reasons for combining the traditional workloads from those systems with consumer data.

Furthermore, the diagram above highlights the different types of storage platforms required to handle the evolving variety of workloads. The operating complexity of the IT infrastructure is going up. ViPRs aspires to serve as a horizontal layer of software across a diverse set of storage infrastructure, abstracting the differences and forming a coherent system. It represents a consistent way to look down into a diverging, complex infrastructure with different workloads and "treat it like a system".

Regardless of the workloads deployed and the underlying IT infrastructure, the vision of ViPR is to sit on top of all of it and behave like a system.

And finally, Isilon and Atmos are becoming more scalable and hitting different economic price points. Atmos has always had object support, and Isilon is adding these features as well. This allows customers who have already invested in products like Isilon additionally leverage it for object support.

This post has been about evolution. In subsequent posts I will discuss two horizontal strategies that cut across every vertical tier in the diagram above:  Flash and Analytics.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow