Information Playground

Restful Interfaces: A Tale of Two Years

The other day a co-worker stood outside my office. I greeted her, pivoted my chair, put my feet up, and told her to come on in. She commented about my ability to appear relaxed every time she came by. I thought to myself "that's the key to creativity". 

At the beginning of my career, I faced the choice of staying late (to make progress on my software project) or driving back to campus (the University of New Hampshire) to hang out with my college buddies.  It was a common practice for many of my co-workers to stay late in order to hit (or beat) deadlines.

But there was a big game back on campus, and I was always interested in playing some wiffle ball.

In fact, in my first book on innovation I described this very decision with the following statement:

    So, do I play wiffle ball, or do I work for $5 an hour?

Needless to say I was out the door at 5PM. Throughout my career I have made that decision again and again (as much as possible). In my book I credit the discipline of work/life balance as a major driver for creativity and innovation.

As we near the end of 2012 I look back on the last two years and see a validation of that approach.

In 2011 I left the world of software engineering and took a job as EMC's Director of Global Innovation.  I thought it would be a technical job, but within 30 days my head began to spin. It was essentially a program management, event planning, and marketing role. I was also managing people for the first time ever (in 25 years). It was difficult to keep my head above water, and although I basically kept the same hours, I found that my head was filled around the clock with the details and logistics of my new role.

During 2012, after one year on the job, I spoke with my boss about the large amount of program management that I was doing, and he agreed that it was not the best use of my skillset (i.e. a technologist). So we shifted the program management tasks away from me, and I assumed new responsibilities overseeing two large technology portfolios: EMC's Global University Research projects, and the research/incubations of my co-workers at EMC Labs China.

Within several weeks after making that decision, I could feel the creativity level begin to rise after a relatively "dry" year. Less worrying about the details of program management gave me more time to read, think, discuss, write, collaborate, and at the end of the day go home and goof around. 

Looking back at my career change now, I can see that it is normal to have the creative routine disrupted when making a career change. 

However, it's also good to have some lessons learned when facing these changes the next time. Two months ago I was assigned to a new boss and a new organization. This organization is being built from the ground up. I am in the position to choose new work roles. I am choosing to turn down opportunities that have a hint of program management, and say yes to anything that requires deep thinking, strategy, and creativity.

As a result, my calendar is a bit more free, and my feet are up on the desk a bit more often.

I'm hoping for a more creative 2013, and I wish the same for you.

Happy New Year!

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

 

Big Useful Data

The results of the Digital Universe study offer product innovation hints that are likely to surface due to exponential data growth.

In my last post I specifically called out complex event processing (CEP) as a technology that will fuel significant innovation going forward.

In this post I'd like to posit that unstructured data tagging will also be on the rise. Consider this interesting statement from the survey:

All in all, in 2012, we believe 23% of the information in the digital universe (or 643 exabytes) would be useful for Big Data if it were tagged and analyzed. 

On the flip side of that number, does it follow that 77% (2152 EB) of data would be useless?

What exactly does "useful" really mean, anyway? The graphic below offers some context:

Take a look at the 2020 data.  Blue means "useful". The height difference between yellow and blue maps to "not useful". 

After studying the report for quite some time, it dawned on me that these designations apply to "unstructured data".  Transactional, structured content is self-describing. Row-column data, for example, has inherit meaning (e.g. a schema) and is therefore useful.

Unstructured data, on the other hand, is often a blob of unknown, variable length content. In order for it to have any value at all, additional meta-data is required.

It is precisely the addition of meta-data that will render the unstructured content as "useful". In fact, the rise of meta-data tagging is also called out specifically by the IDC:

Metadata is one of the fastest-growing subsegments of the digital universe (though metadata itself is a small part of the digital universe overall).

If metadata tagging represents an emerging area of high-tech innovation, what types of unstructured files will be the first targets? Once again, the IDC has a useful graphic  (followed by their definitions of each file type):

 

• Surveillance footage. Typically, generic metadata (date, time, location, etc.) is automatically attached to a video file. However, as IP cameras continue to proliferate, there is greater opportunity to embed more intelligence into the camera (on the edge) so that footage can be captured, analyzed, and tagged in real time. This type of tagging can expedite crime investigations, enhance retail analytics for consumer traffic patterns, and, of course, improve military intelligence as videos from drones across multiple geographies are compared for pattern correlations, crowd emergence and response, or measuring the effectiveness of counterinsurgency.
• Embedded and medical devices. In the future, sensors of all types (including those that may be implanted into the body) will capture vital and nonvital biometrics, track medicine effectiveness, correlate bodily activity with health, monitor potential outbreaks of viruses, etc. — all in real time.
• Entertainment and social media. Trends based on crowds or massive groups of individuals can be a great source of Big Data to help bring to market the “next big thing,” help pick winners and losers in the stock market, and yes, even predict the outcome of elections — all based on information users freely publish through social outlets.
• Consumer images. We say a lot about ourselves when we post pictures of ourselves or our families or friends. A picture used to be worth a thousand words, but the advent of Big Data has introduced a significant multiplier. The key will be the introduction of sophisticated tagging algorithms that can analyze images either in real time when pictures are taken or uploaded or en masse after they are aggregated from various Web sites.

Let's assume that we'll see significant innovation in unstructured tagging for surveillance, medical, entertainment/social and consumer use cases.

The growth in meta-data results in a management problem. If the content gets separated from the meta-data, then the content will revert back to useless.

How do you (permanently) unite the metadata together with the content? The best solutions will permanently graft the metadata and the content together. This will lead to the continued rise of information storage systems that specialize in keeping content and meta-data together: Object-based storage systems.

For those of you unfamiliar with the benefits of object-based storage, earlier this year Chuck Hollis posted a summary of the Atmos implementation of object-based storage. Study his post and it will become clear why object-based technologies are so important.

Tagging transforms unstructured content into something useful (e.g. something that can be analyzed).

Analysis of objects will therefore become critical for the Digital Universe, and a topic worth exploring in a future post.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

 

Related articles

New Digital Universe Study Reveals Big Data Gap: Less Than 1% Of World's Data Is Analyzed; Less Than 20% Is Protected

Digital Universe 2012: Machine Machinations

Innovation Lineage Threads: A Longitudinal Update

I continue to lead a Data Science project at EMC that focuses on Innovation Analytics. My approach has been to create a set of hypotheses  and leverage a task force of volunteer data scientists to help me prove or disprove them.

During this process I ran into a roadblock on one hypothesis:

Hypothesis #7 Incubation Lineage and Asset Generation

I believe that the path that knowledge takes, from a local innovator, to a corporate boundary spanner, to an implementation team, to a delivered asset, can be traced and measured. I also believe that this measurement, once studied, can reveal ways to accelerate innovation and point out areas of knowledge that are yet to be converted. I've long been a fan of provenance, and I love the concept of "idea lineage". The lineage can be studied to reduce asset delivery time.

IH7a: Frequent knowledge expansion and transfer events reduce the amount of time it takes to generate a corporate asset from an idea.

IH7b: Lineage maps can reveal when knowledge expansion and transfer did not (or has not) result(ed) in a corporate asset.

The roadblock occurred because my data sets did not contain sufficient data to measure “elapsed time” between events. My mentor in this initiative, David Dietrich, suggested that I begin a longitudinal study. In a previous blog post this year I described his advice as follows:

Our ability to prove hypothesis #7 was in jeopardy. This realization was not the end of the world. In data scientist terms, it was time to begin a longitudinal study (making a series of observations over a long period of time).The team began to design a method whereby TRL (Technology Readiness) levels would be gathered and recorded as a regular part of the reporting and gathering of global innovation activities. Over time, we would eventually have enough data to take a good, hard look at our hypothesis

After writing the blog post I commissioned the longitudinal study by collaborating with EMC Labs China researcher Diego Wu. Diego and I added two steps to our data gathering process:

1. We introduced innovation lineage threads whenever new, significant research initiatives started (e.g. a university research collaboration was funded).
2. We gathered innovation activities in the same way, but added these activities to a lineage thread when appropriate.

Diego implemented this framework, and two key insights emerged.

Asset Generation (or lack thereof) is Measurable

The diagram below is an actual record of an interaction that EMC had with the University of Florida. An innovation lineage thread was created (the U Florida lineage thread). As multiple innovation events were entered into the innovation analytics framework, they were also associated with the U Florida thread.

 

In this example, several assets emerged from the initial visit of a professor to EMC’s campus. These assets include the publication of a paper (good press for EMC), the hiring of a student (who was already up to speed on the EMC domain to which they were assigned), and the introduction of a new software feature into an existing product line.  Each event occurred at a discrete point in time that is a measurable distance away from any other event in the lineage thread. In this case, three assets were generated within a six month period. The time lapse becomes a data point that can be compared with other lineage threads to evaluate the “time it takes to generate an asset from an idea” (the original hypothesis).

Not all ideas experience this level of success. Often times no significant corporate asset is produced. The use of innovation lineage threads allows us to calculate “asset generation success ratios” across all ideas registered in the system. Success can be correlated against many different factors, including geography, the people associated with the idea, the business unit they are part of, amount of investment, etc.

“Engaging” Insights

While studying any given lineage thread the level of engagement becomes obvious. For example, with the University of Florida I could see clear examples of employee and faculty engagement. They visited each other. They worked on a paper together. They attended a conference together. I could “count” the number of such entries in the database and calculate such metrics as “average number of engagements per month”.

This enables me to draw “heat maps” by asking, for example, which universities are sufficiently “engaged” (or which are not).

As the Director of Global Innovation I struggle to keep track of the health of global innovation initiatives at EMC. For example:

• EMC has dozens of university research partners around the world.  How engaged are we with each one?
• EMC has committed to incubate nearly 30 winning ideas from the 2012 Innovation Showcase process. Which ones are still alive? Which ones have stalled? What kind of corporate assets result from EMC’s yearly investment in the Showcase?
• How can we fail fast? Which funded innovation initiatives are not likely to succeed and should experience “de-investment”?

The decision to begin a longitudinal study was the right one. I’m also convinced that using the Data Analytics Lifecycle (shown below, and taught in EMC’s Data Scientist Course) was the right idea. The project will most certainly progress to Step 6, causing the EMC Executive team to make operational changes to our innovation frameworks.

If you have experience with a concept similar to innovation lineage threads, I'd be interested to hear more about your approach. 

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Related articles

Phase 2 Innovation Analytics: ELT

Phase 1 Innovation Analytics: Creating the Analytics Plan

Phase 3 Innovation Analytics: Hypothesis Exploration

Phase 3 Innovation Analytics: Longitudinal Studies

Phase 1 Innovation Analytics: Hypothesis Generation

Technologies for the Emerging Digital Universe

Recently I published a post describing the enormous amount of data being contributed to the Digital Universe by machines.

Billions of devices will generate petabytes of information. Much of it will land in deep archives that are eventually mined and analyzed using a variety of fast, parallel analytic techniques. 

The Digital Universe study posits that by 2020 these machines will not only generate an estimated 40% of all digital content, but the machines will be expecting an immediate (sub-second) response, or instruction,  to determine the next robot-like action that they should take.  Often times these instructions will need to rely on a variety of other sensor feeds and/or real-time streaming information.

The analytic techniques of 2012 will quite frankly not cut it. What kind of innovation will be required to support this growing reality?

It's an interesting thought exercise to examine some of the emerging technologies in analytics and consider how they can be combined to achieve near-instantaneous machine-to-machine (M2M) communication and decision making.

In this post I'll approach the discussion from the angle of a centralized model. I'm assuming, in the short term, that service providers will begin adding these technologies into the plumbing of their existing data centers.

Over time, however, as the device count scales you will likely see these implementations move to the edge in a dispersed cloud model.

Consider the graphic below and the bi-directional M2M needs of devices connected to the cloud infrastructure of the future:

Here are two emerging technologies that will facilitate M2M in coming cloud infrastructures:

1. More and more streaming data will be ingested into in-memory, distributed data grids. A good example is VMware GemFire: "an in memory data grid that enables real-time data distribution data replication, caching and data management using a non-relational key-value store, to allow storage of data for client applications." In addition, VMware SQLFire is described as folows: "VMware vFabric SQLFire is memory-oriented data management software delivering application data at runtime with horizontal scale and lightning-fast performance while providing developers with the well-known SQL interface and tools".

The key is the in-memory, horizontal scale provided by these types of technologies. Business logic accepts streaming machine input from sensors and immediately stores them in either key-value (Gemfire) or SQL (SQLFire) format. If a given sensor requires immediate feedback based on the state of other sensors, it can be quickly retrieved via an in-memory query. This approach, depicted below, enables the "sub-second" response times that many sensors will be expecting.

2.  More and more streaming machine data will be process in real-time by a technology known as CEP, Complex Event Processing. Wikipedia has a good summary of the technology:

Event processing is a method of tracking and analyzing (processing) streams of information (data) about things that happen (events), and deriving a conclusion from them. Complex event processing, or CEP, is event processing that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.

These events may be happening across the various layers of an organization as sales leads, orders or customer service calls. Or, they may be news items, text messages, social media posts, stock market feeds, traffic reports, weather reports, or other kinds of data. An event may also be defined as a "change of state," when a measurement exceeds a predefined threshold of time, temperature, or other value.

Two implementations of CEP getting a lot of buzz are Esper and Storm (Thomas Dudziak has written a good blog describing both frameworks). For this technology the business logic intercepts and aggregates the streams on the fly.  This technique favors more immediate stream state as opposed to querying less recent memory from a data grid. This approach is depicted below (using Storm's "spouts" and "bolts" icons):

 

 

 

Both of these technologies will become more common in the clouds of the future. Exactly how they fit into a cloud architecture is an area being explored by corporations and academia alike. In a future post I will discuss a research initiative that is starting to investigate analytics of streams at scale.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Related articles

Digital Universe 2012: Machine Machinations

New Digital Universe Study Reveals Big Data Gap: Less Than 1% Of World's Data Is Analyzed; Less Than 20% Is Protected

By 2020, there will be 5,200GB of data for every person on Earth

Digital Universe 2012: Machine Machinations

A machination is defined as a crafty scheme, a plot, or intrigue.

When I look at the wealth of information provided by the IDC in this year's Digital Universe Study, I can't help but think that machines are writing a pretty crafty plot in which we as humans are designing a fairly sophisticated communication network for them.

IDC is nearly at the midpoint of their Digital Universe Study.  They define the Digital Universe as "a measure of all the digital data created, replicated, and consumed in a single year". The first study was completed in 2007. This initial study predicted that the Digital Universe in 2010 would grow to 988 exabytes.

They were off by nearly thirty percent (the Digital Universe in 2010 was estimated by IDC to be 1277 exabytes).

Each year the IDC introduces more sophisticated insights, and this year is no exception. The intrigue surrounding the Internet of Things (IoT) and Machine To Machine (M2M) is a major subplot in the report. 

Digital machines were estimated to produce 11% of the Digital Universe in 2005. These machines represent a "fourth wave" of digital producers. These waves are typically described as follows:

1. Computers, digital by nature, began to populate the Digital Universe.
2. Telephones transformed from analog to digital, resulting in universal expansion.
3. Cameras and media began contributing massive amounts of additional digital content.
4. Embedded systems (e.g. sensors, surveillance cameras, smart meters, appliances, medical devices, toys, etc) are becoming one of "the major sources for ones and zeros".

The IDC estimates that by 2020 machines will be generating 40% of the Digital Universe.

What technology shifts allowed machines to become protagonists in the drama of the digital universe?

I picked up additional insight on this question from a recent issue of The Economist. Three technology trends are enabling machines to become more prevalent:

• The new version of the internet protocol (IPV6), with an address space of 128 bits, provides for 340 billion, billion, billion, billion unique addresses.  The current protocol, 32 bits long, supported only 4.3 billion binary addresses.
• There has never been enough bandwidth to support wireless M2M data streams.  Over the past year, new 4G networks have rolled out to accommodate smart phones and pads.
• Cloud computing has collapsed the unit cost of storage. 

While the IDC study does not specifically cover IPV6 and 4G networks, it does provide an interesting graphic on the plumetting cost of storage:

The IDC estimates that the investment (per gigabyte) in storing the bits in the digital universe will drop from $2.00 in 2012 to $0.20 in 2020.

In 2020 the IDC also estimates that 200 billion "things" will be emitting digital bits, and much of this traffic will be broadcast in the hopes that other machines will return the favor via "sub-second" response times.

If billions of machines have any hope of responding to each other that quickly, we as humans are going to have to introduce new analytics paradigms in the cloud. In my next post I will highlight several technologies that can enable this emerging requirement.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

 

Related articles

Rise of the Machine-to-Machine Opportunities

FuturICT and the European Balancing Act

Several years ago I was involved in an NSF grant proposal that focused on academic collaboration between geographically disparate scientific data sets. A handful of universities in the United States were part of the proposal.  I participated on behalf of EMC and in concert with several other vendors. 

The term “Big Data” was not yet in vogue.

In September of this year I visited several universities in Europe (University College Cork and ETH Zurich) that are collaborating on a similar (but much more sweeping) proposal. The proposal is known as “FuturICT” and is one of six FET Flagship finalists. The Future and Emerging Technologies (FET) flagships are described as follows :

Future and Emerging Technologies (FET) is the EU pathfinder programme in information technologies. Now the European Commission raises its ambition to coordinate a Europe-wide research effort towards visionary technological targets. The endeavour takes shape in the FET Flagship Initiatives.

FET Flagships are large-scale, science-driven and mission oriented initiatives that aim to achieve a visionary technological goal. The scale of ambition is over 10 years of coordinated effort, and a budget of up to one billion Euro for each Flagship. They initiatives are coordinated between national and EU programmes and present global dimensions to foster European leadership and excellence in frontier research.

On October 24^th of this year (2012) all six finalists submitted their final papers. As a strong supporter of FuturICT, EMC received a copy of the final proposal.  While it is hard to fully summarize such an ambitious proposal as part of a blog post, FuturICT has three high-level thrusts:

Planetary Nervous System

Build an infrastructure to collect social data streams, sensor outputs, public government reports, etc.   All of these signals will be gathered, aggregated, mined, and cleaned.

Living Earth Simulator

Build an infrastructure for large scale, data-driven simulation and experiments. The infrastructure will rely on common data models , shared visualizations, and HPC. Key pieces of this infrastructure will include structured and unstructured databases in a cloud-based environment.

Global Participatory Platform

Build a social networking and  crowdsourcing API to  help people integrate with the system.

Together, these three parts make up a whole known as a “Living Earth Platform”.

If the proposal is approved and the grant money received, the FuturICT team will tackle the familiar “balancing act” problem that appears whenever vast amounts of public data are gathered together: how can data about individuals be usefully mined while privacy is protected?  Security researchers  Dirk Helbing at ETH Zurich puts it this way:

Helbing points out the particular challenge is to develop new technical methods of data encryption, storage and processing that allow the kind of data mining that benefits individuals and society, but which also protects individual privacy and confidential commercial data. Nevertheless, it must remain possible to inspect data in a limited, democratically controlled way where this is necessary to combat corruption and terrorism. Until now there has been a lack of technical solutions that can satisfy all three requirements.

Furthermore, Helbing stresses “the main priority is to find ways of giving back control of personal data to the user”. The World Economic Forum is now making the same recommendation. He also emphasises that the research within the FuturICT project will have a strong emphasis on ethical questions and a clear code of values. He says, “FuturICT will give top priority to protecting sensitive data and will be fully transparent and democratically controlled. Among all the activities working with large volumes of data, this is the most transparent project. Without such a project it will hardly be possible to learn about the dangers of large data sets and to take effective action to protect society from these dangers”.

Finally, it is important to point out that FuturICT does not want to be a tool that is restricted to a number of privileged political or economic decision makers. Just like the Internet empowers individuals and small organizations with a global reach and unlimited access to information, the project intends to create a participatory platform allowing everyone to access and utilize the data and models developed by the project for their own purposes and applications. 

FET is a prime example of a regional innovation initiative that local EMC Europeans can monitor and contribute to. As a vendor that provides cloud, big data, and security products, it is easy to see why EMC is highly interested in the proposal. From a research perspective, staying close to this initiative could give our researchers a front-row seat to a fascinating European discussion.

I’m rooting for FuturICT to win the grant, if for no other reason than to keep tabs on the progress and share the results in the years ahead.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Fellow

Related articles

Taking the Train to ETH Zurich

FuturICT Vision for the Social Sciences, ICT & Complexity Science