This article was written by Rajeev Warrier, CTO at EKA.
It would take significant effort to still be unaware of the big data phenomenon. Almost every commercial and industrial sector has been affected by dramatic increases in the volume of data that is now available. Across the board, businesses are eyeing up both the challenges of processing all this information and the opportunities that it opens up with varying degrees of confidence.
Rajeev Warrier
In commodity management, which has been at the more data-intensive end of the business spectrum, the volume, variety, and velocity of the data coming into the organization is unprecedented. Managing this data deluge, and deploying the advanced Analytics needed to make sense of it all, is the difference between drowning and waving.
Fortunately, the technology is increasingly available to turn all this raw information into insight and opportunity. The following are the key components of any big-data ready solution:
1. Data-storage grids
A distributed storage grid offers a number of advantages when it comes to storing data – a fundamental requirement in any big data management solution. It offers better fault-tolerance and redundancy and so all but eliminate downtime. It helps maintain stable performance even when storage load is fluctuating. And it is highly scalable – so there is far less need for expensive hardware upgrades and installation interruption.
2. Schema-on-read technology
Because schema-on-read-technology allows data to start flowing into an analytics system in its original form, that data only needs to be parsed and refined at process time. This means users can ask questions of the data that they want to ask – not just those that are baked into a pre-defined model. When working with the large sets of raw data that are typically found in the commodities business, the versatile organization of data offered by schema-on-read is a big advantage.
3. In-memory data grids (IMDG)
Providing unprecedented processing speeds, IMDGs enable commodities managers to get the results of advanced calculations in minutes rather than hours. IMDGs can support hundreds of thousands of in-memory data updates per second, and can be clustered and scaled in ways that support large quantities of data.
4. Machine learning
Evolving from pattern recognition and computational learning found in artificial intelligence, machine learning is based on the construction of algorithms that can learn from data and adjust their performance accordingly. Machine learning can be used to make accurate predictions on diverse elements within the commodity value chain: from investment in new plant and fleet right-sizing, to cash-flow management based on information on individual counterparties and invoices.
5. Predictive analytics
At the heart of new advanced analytics capabilities, predictive analytics is the branch of data mining concerned with discovering future probabilities and trends. Predictive analytics can be used to run complex forecasting models and scenarios to answer essential “what if” questions. Want to know what happens to the bottom line if market prices go up or down? Then predictive analytics can provide the answer.
6. Dynamic visualization
Finally, all that data needs to be presented in easy to use and easy to understand formats. Dynamic visualization enables you to ask iterative questions and get timely answers. Interactive displays enable you to discover patterns in large amounts of data and so make faster and more accurate decisions.
That might sound like a prohibitive round of technology investment. But parallel developments in Cloud platforms and cyber-security mean that vendors are making commodities-specific solutions available without investing millions of dollars and untold man-hours. Taking advantage of new technology, bringing advanced analytics into the organization, and being ready for whatever big data brings next has never been easier.
