Reams have been written on Big Data. Many advisers/consultants make a living out of big data. It does have huge potential to solve complex problems in a counter-intuitive way.
Many enterprises are still figuring out how to use the technology of big data and realize its true potential in their organizational context. In some ways, it is a hammer looking for a nail. Over time, enterprises will find more effective use of big data technology and paradigm.
In case of big data, a conclusion is reached in a probabilistic way by managing very large volumes of data (both structured & unstructured) obtained through disparate data sources. In these cases, a probabilistic conclusion can still be reached, even if all individual data sets are not entirely reliable. Big data, in other words, can deal with messy data.
Smart data , on the other hand, needs to be of high fidelity. Smart data is needed for deterministic conclusions. As an example, if a regulator needs to know exactly how a questionable equity trade was conducted by a financial institution at some point in the past, a deterministic response is needed. A probabilistic response will be inadequate.
Most large, global organizations struggle to reach a state of smart data. Fidelity of data at the point of secondary consumption is low. Process of data enrichment may not be well structured. It is also largely manual. A lot of money is spent in provisioning data for an intended consumption use case. This process is repeated when the consumption use case changes.
There is a better way of enriching data and bringing it to a state of high fidelity, in the context of a particular business function (like banking). An automated process can be devised that looks for:
- Completeness of process
- Completeness of content
- Checks for historical pattern to identify potential errors.
If the state model of data is mature and domain-centric, this can be achieved. There is a case for significantly high return for organizations, if they can achieve a state of Smart Data.