Over the last several years, there has been a massive surge of interest in Big Data Analytics and the groundbreaking opportunities it provides for enterprise information management and decision making. Big Data Analytics is no longer a specialized solution for cutting-edge technology companies - it is evolving into a viable, cost-effective way to store and analyze large volumes of data across many industries. But how will this translate to adoption of these new technologies? How will companies incorporate Big Data into their existing business intelligence and data warehouse (BI/DW) infrastructure? How can end users take advantage of the power Big Data has to offer?

We live in a world of big data.  Companies have to learn how to deal with mountain of data.  The question is how data has changed our business reality.  Here are some facts on the new reality:

• It costs $600 to buy a disk drive that can store ALL the world music
• There are 30 BILLION pieces of content shared on Facebook every month
• Data acquisition is growing at a rate of 40% annually, but IT budgets are only growing at a 5% annual average
• In April 2011, the US Library of Congress had collected 235 terabytes of data, but 15 of the 17 industries in the United States store more data per company than the Library of Congress.

Companies today churn out huge volumes of transactional data, capturing trillions of bytes of information about their customers, suppliers, and operations.  Companies can capture information from the millions of network sensors being embedded in the physical world, in the devices we use and run our world; mobile phones, industrial machines, automobiles, energy meters, and more.  These sensors sense, create and communicate information about us, how we live our life and how we produce and consume things.

Big Data is defined as databases whose size is beyond the ability of traditional databases to store, manage and analyze.  Big Data is not defined by size (i.e. gigabytes, terabytes or exabytes).  It is assumed that technology advances so size becomes less of an issue. 

The amount of data in our world has been exploding, and analyzing large data sets-so-called big data-will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by  McKinsey's Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future

1. Data has swept into every industry and business function and are now an important factor of production, alongside labor and capital. By 2012, nearly all sectors in the US economy had at least an average of 200 terabytes of stored data (twice the size of US retailer Wal-Mart's data warehouse in 1999) per company with more than 1,000 employees.

2. There are five broad ways in which using big data can create value.

• First, big data can unlock significant value by making information transparent and usable at much higher frequency.
• Second, as organizations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days, and therefore exposes variability and boost performance. Leading companies are using data collection and analysis to conduct controlled experiments to make better management decisions; others are using data for basic low-frequency forecasting to high-frequency nowcasting to adjust their business levers just in time.
• Third, big data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services.
• Fourth, sophisticated analytics can substantially improve decision-making.
• Finally, big data can be used to improve the development of the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance (preventive measures that take place before a failure occurs or is even noticed).

3. The use of big data will become a key basis of competition and growth for individual firms. From the standpoint of competitiveness and the potential capture of value, all companies need to take big data seriously. In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value from deep and up-to-real-time information. 4. The use of big data will underpin new waves of productivity growth and consumer surplus. For example, it is estimated that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent. Big data offers considerable benefits to consumers as well as to companies and organizations. For instance, services enabled by personal-location data can allow consumers to capture $600 billion in economic surplus.

4. While the use of big data will matter across sectors, some sectors are set for greater gains. The computer and electronic products and information sectors, as well as finance and insurance, and government are poised to gain substantially from the use of big data.

5. Several issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world. Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data. Access to data is critical-companies will increasingly need to integrate information from multiple data sources, often from third parties, and the incentives have to be in place to enable this.

This is the first in a series on Big Data, and how it's going to Change our world.

**********************************************

John Ragsdale, CEO of Customer Vision

Please login or register to add comments