Big Data or Smart Data?

Big Data or Smart Data?

Reams have been written on Big Data. Many advisers/consultants make a living out of big data. It does have huge potential to solve complex problems in a counter-intuitive way.

Many enterprises are still figuring out how to use the technology of big data and realize its true potential in their organizational context. In some ways, it is a hammer looking for a nail. Over time, enterprises will find more effective use of big data technology and paradigm.

In case of big data, a conclusion is reached in a probabilistic way by managing very large volumes of data (both structured & unstructured) obtained through disparate data sources. In these cases, a probabilistic conclusion can still be reached, even if all individual data sets are not entirely reliable. Big data, in other words, can deal with messy data.

Smart data , on the other hand, needs to be of high fidelity. Smart data is needed for deterministic conclusions. As an example, if a regulator needs to know exactly how a questionable equity trade was conducted by a financial institution at some point in the past, a deterministic response is needed. A probabilistic response will be inadequate.

Most large, global organizations struggle to reach a state of smart data. Fidelity of data at the point of secondary consumption is low. Process of data enrichment may not be well structured. It is also largely manual. A lot of money is spent in provisioning data for an intended consumption use case. This process is repeated when the consumption use case changes.

There is a better way of enriching data and bringing it to a state of high fidelity, in the context of a particular business function (like banking). An automated process can be devised that looks for:

  • Completeness of process
  • Completeness of content
  • Checks for historical pattern to identify potential errors.

If the state model of data is mature and domain-centric, this can be achieved. There is a case for significantly high return for organizations, if they can achieve a state of Smart Data.