data collection

Obituary: Big Data

Obituary: Big Data 150 150 Kerry Butters

Donald Feinberg, VP and analyst at Gartner’s Intelligence and Information Group, recently said that Big Data will die within the next couple of years, thanks largely to the confusion which surrounds the term.

Once upon a time, databases were relatively small; tiny by today’s standards. Businesses had records of their customers’ accounts, built up manually over time, originally with pen and paper and later with microprocessors. Bigger companies started to have whole floors dedicated to data processing departments, ensuring that purchase orders and invoices we all matched and accurate, and accountants knew who had paid and who owed money, what had been bought and what had been cancelled.

With cloud computing and processing technology getting so small that you could practically map out the life cycle of a grain of rice, data started to get recorded and collected at increasingly faster rates and much more of it. Processors in cars and other equipment meant that a whole boatload of parameters could be constantly measured.

More and more measurables

Social media sites, ecommerce sites and other communal online gatherings meant that individuals could be adding to the pile of data already stored about them as they filled in forms and registered for things online. Photos, likes, friends, birthdays, political leanings, sexual orientation, marriage status, hobbies and interests…the list of measurables became endless.

Marketers cottoned on that they could find out even more about people and their activities by giving a little entertainment in return for information.

Data was evolving and its new buzz-name was emerging. This thing was big and needed a grand, although quite unoriginal, title. ‘Big Data’ was born and every smart sales person, IT geek and technical consultant around was throwing the phrase around in conversations.

What to do with Big Data?

Some people had ‘Big Data’ but didn’t know what to do with it. Others had bigger ‘Big Data’ than everyone else (so there) and knew exactly what they wanted to do with it. Certain eager beavers didn’t care if it was called Data, Big Data or naught and one spaghetti; they just made sure that the IT infrastructure for their organisation could handle any amount of information that was going through their servers and conduits.

Others blissfully got on with running their businesses, hiring the services of IT support companies and other professionals who would make sure their systems didn’t crash and their printers worked when they turned them on. Some smart people were figuring out how to condense meaningful understanding from all the data they were gathering.

Changing technologies

This was Big Data’s heyday, and a time when it actually meant something. Technology was changing and much of the change related to the amount of data that was ‘out there’, how it was being managed, how quickly it could be processed and moved around, and the mind-blowing variety of variables that could be and where being measured. Surely this data was going to be extremely useful for managing situations, for making the most out of the trends, for future proofing organisations by learning lessons from the past.

In fact, the enormity of the situation meant it was almost beyond definition. From a business perspective many aspects of how information stock piles were growing could be a threat or an opportunity. It all depended on how businesses reacted to the changes and the trends, and embraced what was happening. There would be winners. There would be losers. Who would win and who would lose was down to how they played the Big Data lottery.

The death of Big Data?

Yes, they were halcyon days for ‘Big Data’, so what changed? How did Big Data eventually get sick, and then subsequently die? What went wrong? In a nutshell, it was beaten by its own success. It was made redundant by its own arrival.

Big Data could be seen as a hurricane on its way to a land that has never experienced such a phenomena before but is going to imminently – where people are forewarned well in advance that there is a disturbance in the weather, troubled times ahead, that the time is coming to baton down the hatches and seek cover before the incredible winds, rain and destruction arrive. The time for talking about it, for describing, defining it and giving explanation is before the storm hits. Once the storm hits, everybody is too busy making sure they come out on the other side in one piece.

A major change in how information was being generated and gathered was occurring that required attention and action on a number of fronts. Just as the village needed to be prepared for the high winds of the hurricane and in order to be convinced that it was coming, it was defined, described and explained to them; so was Big Data the catch-all solution to capture the revolution in data processing.

Information control

That job has been done. Mentioning Big Data is now pointless. The consequences of its arrival are already here in the flesh. Businesses are finding right now that they either have control of the information in their possession or they don’t. They are either benefitting from the intelligent analysis of the information that counts or they have been focusing on the least fruitful facts and figures – or maybe not mining from their data banks effectively at all.

Big Data was a hype word. It was a necessary one to galvanise people into action, to facilitate communication and sum up a range of phenomena that needed to be acknowledged – like the ‘swinging 60s’, but now the realities have already hit home the people who are really in the know are looking for the next change they need to predict. Big Data is rapidly becoming a ghost that is only mentioned by people who are not so savvy, or who are desperate to show understanding in order to sell a product or service.

Thanks for everything you did for us, Big Data. Good bye and God bless. May you rest in peace.

Image: Gerd Leonard

Is Big Data Essential For Business?

Is Big Data Essential For Business? 150 150 Kerry Butters
 Image by  Domo

Image by Domo

Data is more a part of our lives now than it has ever been. It’s woven into every sector of the global economy and the harnessing of that data by businesses and individual alike is becoming the norm.

Big Data is what we get when the data sets collected become too large and complex to analyse using standard methods. This data comes from all sorts of sources, including web browsers, social media and consumer information. By sifting through all of this information business managers are able to make much more informed decisions and therefore move their company forward with confidence.

The Benefits of Big Data

Future ready – The internet of things is a hot topic in 2014. As products begin to broadcast data to one another to improve their efficiency. The amount of data available to manufacturers and suppliers is going to grow even greater as a result. Investing in an infrastructure that can handle and analyse Big Data now will put your business in an excellent position for the future, even if you’re not concerned with the internet of things, Big Data is listed by Gartner as one of the technology trends you can’t afford to ignore.

Customer Insights – Near the top of a good business’s objectives is to react and respond to the needs and desires of its customers and clients. Big Data allows your business to make accurate predictions about the near-future needs of those who use your business regularly, and the business of your competitors. This can give you the edge that you need to both excel in your chosen field and compete with other businesses.

New Business Models – Some industries have seen Big Data create completely new business models. Algorithmic trading allows businesses to analyse tremendous amounts of market data every minute, providing them with information on where the real opportunities are. This kind of speed would have been unheard of a decade ago. Retail companies are using Big Data to change their purchasing behaviours, making purchases based on fact rather than speculation.

Better In-House Operations – Big Data may be the final link for companies looking to achieve maximum operational efficiency. By constantly assessing the efficiency of their workers and work processes, managers will be able to highlight the points where they’re losing money. Insurance companies are already speculating how they can use Big Data analytics to speed the processing of claims and spot potentially fraudulent claims that need investigating.

Improved Sustainability – The Guardian recently reported that Big Data could see businesses able to take much greater steps towards becoming sustainable and nature friendly by using the same information that improves their in-house operations.

Harness Your Staff’s Potential – Data analysis doesn’t have to stop at your customers, businesses who turn their analysis inwards and review their own staff have reaped the benefits too. By analysing the work patterns and talents of their employees, managers can make sure that staff with niche skill-sets aren’t going to waste and introduce multi-discipline roles to improve efficiency.

The challenges of Big Data

Security – Obviously if you’re handling tremendous amounts of customer and client information then you need to be absolutely sure that that information is well protected from any hostile forces that might want to misuse this information. If you’re going to invest in Big Data we advise you also invest in beefing up your security. Fortunately, Big Data security is set to be one of the big talking points of 2014, so there will be plenty of opportunities to make improvements across the board.

Accumulation – Data doesn’t just turn up on your desk, you have to invest in accumulation systems to help you collate all the information that’s out there and turn it into useful, business-driving data. These can be expensive, but they are an important part of any business’s Big Data strategy.

Analysis – Big Data needs high performance analysis because there’s just so much of it. Datamation wrote an article last year about the amount of data we create every minute, and the numbers are staggering. According to their data in June, the human race wrote over 204 million emails a minute, queries Google 2 million times, watched 48 hours worth of YouTube Videos and produce over 100,000 tweets every minute. That’s a huge amount of information to process and eke useful data from, but it’s almost certainly worth it.

Datamation also noted that over $272.000 was spent on e-commerce every minute. Like the other numbers it’s likely to have gone up in 2014 to even more. People are spending more money online because it’s often cheaper, more convenient, and more varied. It also means more data, and that data is becoming more and more accessible every year. For a business, passing on all that information is passing on a huge percentage of potential new customers for your business once you isolate their desires using Big Data.

It’s Time for Big Data

Big Data analytics is looming over the business sector like never before. 2014 is the year where we’ll see businesses really kick their analysis into gear. The target for many will be to become capable of real-time analysis of data as it’s collected, and then react to that information accordingly. Any big company unable to do similar may find themselves falling behind, and any SME’s getting into Big Data early could well see themselves rocketing to the front of the pack.

Big Data – is it possible to define it?

Big Data – is it possible to define it? 150 150 Simon Randall

This is a big question and one which, once fully considered, has massive implications for any business. Every day, businesses are amassing an increasing amount of data and the scope of what can be measured is also expanding at an incredible rate. While businesses are still getting to grips with how to use this data meaningfully, some are struggling to manage it effectively.

Sometimes that means databases becoming corrupt at an increasing rate as they become larger, or more difficult to store effectively in-house. Massive databases may be creating too much demand on infrastructure when being processed at speed or are that diverse in nature that it is difficult to know where to start when organising them into a usable format.

What is Big Data?

Once upon a time, a business would store essential information such as client names and invoice details, order history and accounts records. This information would be structured into usable format and, with the dawn of the computer age, tied up with software making it easy to access. Looking back to that era, data was gathered conscientiously and with a definite purpose in mind. The bigger the business was, the bigger the databases required to store its prized information.

Digital evolution has changed the data landscape forever. Where data was once input into a system one unit at a time by an operator or by an individual filling in a questionnaire, data collection and its transfer is now a more automated affair. Modern day communications being what they are means that information relating to somebody’s Facebook usage, while accessing the site from their mobile phone in Indonesia, is usable in the US in the blink of an eye.

It’s not only the speed that data is transferred that’s changing, and how it’s transferred; the spirit in which data is now gathered has now become incidental, almost accidental. That is, much of what people do with their credit card or view using their browsers leaves a data trail that can be detected and processed retrospectively. It’s this discovery of what’s already out there that’s making Big Data so much of a wild card commercially with regard to its changing nature, and also what makes it so exciting an opportunity.

Types of data

The type of data being collected and the sheer amount of information that’s out there is absolutely mind-boggling. Most companies in the US have at least 100 terabytes of data stored – that’s 100,000 gigabytes if you’re wondering – according to a useful infographic (below) published by IBM. The same document highlights the fact that out of a world population of 7 billion, some 6 billion of us are using a mobile phone. Imagine the humungous amount of information being measured through our phones. What’s particularly interesting in this infographic is how data is categorised according to four criteria: volume, velocity, variety and veracity.


There’s no particular agreed volume of data needed for it to qualify as Big Data, so it’s a bit of a case of how long is a piece of string, However if you’re looking at a sizeable amount that’s becoming hard to manage, then it’s safe to say you’ve entered the realm of Big Data. It’s now possible to mine a large array of information from data trails, and where specific data is being gathered deliberately, the processes have become so efficient through better real-time technology, that it’s very easy for a business to find it has a tricky amount of facts and figures very quickly.


For data to be useful, it needs to be processed. How quickly the data is gathered in a meaningful way, and how quickly it can be analysed and used effectively, is a reflection of its velocity. As shown by the earlier infographic, it is projected that there will be 18.9 billion network connections by 2016; that’s almost 2.5 connections per person on earth, and they will be creating a whole new universe of high velocity, streaming data that will be analysed on the fly. There’s a great illustration of how velocity interacts with volume and variety in this WhatIs article.


The variety of data that exists is changing all the time as technology changes. Twenty years ago, the idea that 30 billion pieces of content would be shared on Facebook each month or that 400 million Tweets would be generated EVERY DAY were unthinkable. Data is now being collected using hi-tech medical monitoring equipment; from computers, mobile phones etc. Just as today’s variables were unimaginable yesterday, it’s highly likely that there are many others beyond tomorrow’s horizon that are unthinkable today. Big Data is going to get much bigger and even more complex, but the rewards for managing it effectively are going to be exciting for everyone.


From the Latin root, ‘veritas’, meaning ‘truth’, this wonderful word refers to how dependable, or how certain the data that’s been gathered, is. Since records began, there were those that would leave a space blank rather than double-check, or lazily record an approximate value or sometimes an inaccurate one. Whether through human error or mechanical failure, as would be the case if a particular button on a keyboard was faulty, mistakes have always been made and will continue to be a reality for some years yet.

The problem with Big Data in relation to veracity is that mistakes tend to be greatly amplified for greater amounts of data. Also, just as an arrow that’s off course by just half a degree can finish up increasingly further from its target depending on the distance travelled to reach it, so poorly measured or recorded data can have a snowball effect in the long term, especially as data size increases.

What do Volume, Velocity, Variety and Veracity mean for business?

The scale of information gathering and processing now available to business means lucrative opportunities for understanding their target markets, current trends, projections, spending habits, ways to become more efficient; the potential is enormous. The four main categories of Big Data discussed in this article, the 4 V’s, present different challenges to business in regard to how they can be practically handled.

Effects on IT infrastructure

As the volume of information being gathered continues to increase, perhaps not exponentially but dramatically, businesses are going to need servers that can handle the extra load and remote data back-up will also need to be airtight. After all, the potential to lose data will be greater because whereas one day’s data loss was this much yesterday, it will be many times more than that today. The size of data being transferred will have knock-on effects on IT infrastructure; with cabling needing to be of a suitable spec to handle the larger volume for example.

Even smarter software development

The experts are getting better at measuring things and converting that information into noughts and ones. Light, sound, humidity, occupancy are all measurable and these represent just the tip of the iceberg from the world of building management. As we are able to collect an increasing range of variables from the world around us, we will need to store these in an intelligent way for easier processing. They will need separate databases, and sophisticated software packages that can move them, shake them, make sense of them and create the commercial honey we all want.

Robust data gathering and recording

As more data is collected automatically, veracity will improve as software becomes more sophisticated and data transfer more reliable. The slightest error could mean there’s a black hole for every hundred thousandth unit of data, for example, and when dealing with a mammoth amount of information that’s going to be put through heaps of processing, the long term corruption potential could be catastrophic.

Whether or not human beings fill in the fields of CRM packages or online forms depends on many factors. Are they being asked too much by a company they have never done business with before? Is the process too tedious? In regard to CRM software, has it been embraced by the workforce who are using it or is it seen as a way of snooping on them? Have they been trained to use it correctly or is it just another unwelcome task that has landed on their lap, another thing to slow them down in an atmosphere where productivity is constantly monitored? After all, if the team who are using it don’t embrace it, how can the data being entered into the system be trusted?

Unintended consequences

Other factors that can affect veracity are the unintended consequences of sales people, for example, who are trying to find loopholes in the system or are cutting corners. Perhaps, they are creating a duplicate account for business that has been barred from further business by the accounts department of the organisation, or maybe an operator has figured out they can get through the system quicker by putting any old number into a particular field.

These accuracy issues can only be dealt with using a thorough approach – software fail-safes, intelligent design, effective communication and training of the staff that are using the software etc.

So is it possible to define Big Data?

Every aspect of Big Data is changing; the amount of data, more is expected from the data collected, and the type of data being processed is evolving. What this means is that trying to define Big Data is like trying to grab hold of blamanche. Any definition that can be applied is likely to become outdated fairly quickly because the boundaries of this new and wonderful entity are in a state of flux.

Big Data is bringing a whole new world of opportunity to everybody. For the everyday person it means added convenience because the organisations that offer everyday services are getting better at giving people what they want. From the perspective of big business, it means getting it right more often, stock that sells, projections that are accurate, efficiencies and increased profits.

The wise will bear in mind that with every opportunity, there is usually a threat and that is certainly true in the case of Big Data. If businesses don’t get to grips with how they handle Big Data accurately, efficiently and effectively, not only will they miss out on growth potential, they will be vulnerable to competitors who are in command of their Big Data.

The real question

For a business to be truly future-proofed, it needs to be prepared for change and ready to adapt to a new world. In the case of Big Data, this means be ready to take advantage of whatever new variables become measureable – and these new possibilities may well be beyond what we can currently see. The question then becomes this: –

If businesses really want to reap the benefits that Big Data brings, should they even define it in the first place or is it better to keep eyes peeled on the next big change?