Collecting Big Data in Real Time
It was bound to happen. The two titans meet. The gargantuan grasp of Big Data turns its ever-open hands towards the firehose stream of real-time data. “The next evolution of the big data phenomenon has turned out to be real time streaming of data,” says Big Data pundit Rick Delgado in a recent blog: What Real Time Streaming Means for Big Data. “Organizations have an increased need to gather and analyze their data at the same time, making real time data streaming a must if big data is going to keep up with demand.”
Will Big Data ever be satisified? Not as long as the demand for informed action continues to grow. Will we ever run out of real-time data? Not as long as stuff keeps happening. The only thing necessary to complete this marriage is to make the connection, and stream real-time data into the welcoming, capable hands of Big Data.
This is what we are keen on. With our established track record in real-time industrial data communications, we anticipated this need for real-time analytics years ago, along with other thought leaders. In a blog back in 2011 we quoted Paul Maritz, President & CEO of VMware at keynote address on the future of cloud computing at VMWorld 2011, “People are going to have to be able to react to information coming in, in real time.” Since then we’ve been putting the vision into action, and it’s great to now see the Big Data people coming on board.
Real-Time Analytics from Big Data
The advantage of live connectivity to Big Data is you can now do your analytics in real time. Delgado sees this clearly. Real-time inputs to Big Data, he says, can fuel near-real-time outputs. Rather than a two-stage process of storing the data, and then analyzing it, the analysis can take place on the fly, and your system can function like the mind of an athelete, jazz musician, pilot, or soldier. Insights become more spontaneous, and reactive responses are replaced by pro-active initiatives. The competitive advantage goes to those who can better anticipate and immedately meet customer demands, increasing customer satisfaction and establishing greater loyalty.
Delgado lists a number of areas where real-time streaming to Big Data could have a significant impact. For example, certain types of fraudulent or suspicious patterns of trading in the financial sector that don’t show up in the aggregate could be spotted in real time. Businesses could monitor customer behavior on websites and social media to provide people with exactly what they need, at the moment they want it.
Additional Benefits – Industrial Sector
Among various application spaces that Delgado mentioned, he left out a significant one: streaming real-time Big Data for industrial users. Imagine an operator of a machine where an alarm light is flashing. Looking at his smart phone or tablet, he gets not only the alarm and raw data from the machine, but a real-time analysis of what could be wrong. And along with that, he may receive suggested action steps based on comparing that data in real time to technical specs, historical records, and even live recommendations from its manufacturer, who is also connected to the machine, and monitoring it in real time.
Companies like GE are investing millions in such systems. They collect and analyze in real time the Big Data coming from power turbines, jet engines, and other equipment during operation. As the Industrial IoT gains acceptance, we see other companies, big and small, follow suit. The value inherent in real-time data for making instantaneous decisions is too great to pass up. The industrial sector, a large and long-time user of real-time data, stands to benefit significantly by connecting to Big Data.
