Using Big Data to Support the Cognitive Hourglass: Moving from Data-Analytics to Data-Applications

As data science and machine learning become increasingly sophisticated there is an increasing desire to automate discovery by removing the need for humans in the feedback loop. However, in many fields, like in the life sciences, where the data landscape is complex and where researchers are striving to understand causation, not just correlation, humans and machines must work together. This talk will explore this necessary interplay, introduce the concept of the “cognitive hourglass” – a model for how researchers drive towards discovery – and look at how big data technologies can be used to aid this process. In addition to looking at real-world examples where back-end analytic tools like Hadoop, R and Python have been paired with front-end interactive visualizations, we will also discuss some of the important design considerations for doing so. With the right abstractions in place, conditions can be set to maximize the chances that pairings of these technologies can lead to unexpected innovations. We’ll end by looking at a specific case where this happened – how clustering algorithms from genetics were applied to socio-economic data to offer new insights for the field of global policy.