The collision course between Big Data, Privacy, Ethics and Regulations

With the much heralded GDPR - General Data Protection Regulation - in place now and regardless of whether you are a box checker, an active practitioner or a procrastinator, it has revealed much about what good data privacy hygiene looks like. And that involves collecting and storing as little as possible so the exposure is minimal. And consumer rights to revocation or transportation can be accommodate easily. And this has been a U turn for most forward leaning organizations. Going from taking snapshot backups every hour and storing it in cheap (and getting cheaper) cloud storage. And worry about that exposure later. To now actively going and changing policies to take backups more thoughtfully and not retaining it longer than necessary. Now let's shift focus to AI. AI is everywhere - Google Duplex to the Amazon shopping recommendation to personalized healthcare - and the one crucial ingredient that underpins the success of AI is large swaths of data. The more data there is, the better the algorithms that can be trained in order to get to a recommendation and action that makes sense. A good example of this is the autonomous cars. In order for these cars to determine a light pole from a tall thin man, they need to have seen enough light poles in daylight, dusk and night. The tall thin men should represent all races so the skin color has enough data sets to make the algorithm effective. And that requires lots of data to train the model. Therein lies the conflict. The regulation and privacy gurus will be advocating for limiting the amount of data collected in order to provide for a safer and more trustworthy customer experience. The AI engineers and business owners will be pulling the wagon in the diametrically opposite direction in order to make the AI more effective. We will explore each of these in more detail and offer suggestions.