What is “Data Democratization”? Why are there 5.4 million hits in Google Search for this term? Do we care? Should we care?
What?
In the beginning, there was data. Not much. Enough to be memorized or written in a stone. Then, there were papyri, and books, and libraries, and magnetic storage media, and streams, and logs, and social data, and internet of things! Then, there was too much data! Did you know that more data has been created in the past two years than in the entire previous history of the human race?
(Smart) companies saw the value in data. They tried to organize and distil it into meaningful, actionable insights. Sadly, the data was often too hard to tame, as it was typically locked in silos or the analysis tools were lacking. It was estimated that that only about 0.5% of data is actually ever analysed.
Clearly, something had to be done.
Enter data democratization.
Modern data storage and processing methods such as Hadoop and Spark allowed companies to gather and organize as much data as they wanted. New streamlined and powerful analytical tools such as Tableau Desktop or QlikSense, and even R, Python’s PyData stack, or Jupyter Notebook (for the open-source aficionados) made data analytics more accessible than ever. The result of these new capabilities, known as data democratization, was the empowerment of a broader-than-ever audience to extract value from data.
Why?
Should we, Zopa, care about data democratization?
Yes. It is a source of decisive competitive advantage (even survival!) to any company.
Even more, do we actually care?
Yes! We love it!
Our company’s culture and way of thinking is driven by people who are passionate about data. We don’t have dedicated analytics teams, as data analysis is part of all our jobs. We see data democratisation as a game-changer, as it makes it simpler, faster and easier for our people to get access to the insights they need. It protects the company from becoming a top-down organisation where the highest paid person’s opinion wins. By freeing the data for our people to work with, we give them more responsibility and ownership.
How?
Our data democratization efforts focus on four pillars: data, tools, training, and people. Let’s expand.
Data
Similar to (most) other companies, a considerable fraction of our data remains in silos spread across Microsoft SQL Server, Splunk, flat files, with partner companies, and, sadly, in people’s private folders. Needless to say, this is not conducive to allowing one to see the ‘big picture’. To tear down the silos, we recently started creating a cloud-based data warehouse/Lake combo on AWS, to act as a consolidated single source of truth for data analytics.
Our efforts at improving data availability are not restricted to only inside Zopa. To increase transparency, we regularly share (aggregated and/or anonymized) data with independent third parties (such as AltFi or Nesta) and we publish our loan book on the Zopa website.
Tools
A single analytical tool cannot process all kinds of data. For accessible, yet powerful self-service analytics, we use Tableau Desktop and Tableau Server, while we are considering open-source alternatives such as Apache Zeppelin or Airbnb’s Caravel. For the more involved and data-processing-heavy analyses, we use Python and its excellent PyData stack, running on an in-house Docker-basedJupyterHub setup.
Training
Data democratisation is a process of empowerment. However, with great power comes great responsibility. To ensure that data democratization does not come with data misinterpretation, we train each other, by creating curated self-study training material, by ensuring effective dissemination of expertise via seminars, HipChat channels, mailing lists, and even making sure that experts and learners are sitting next to each other in the office.
People
Excellence in data analytics goes hand in hand with a particular kind of mentality – that of inquisitive, positive, persistent, and open people. As such, we make sure that such people are rewarded during our hiring and appraisal processes.
In parallel, we try to keep people engaged, and motivated to think, ask questions, and play with data. Here, communication plays a key role. Our teams (aka tribes) share their roadmaps, progress, and data-driven insight across the company using dashboards, YouTrack, and company-wide presentations, while our experts are invited to provide open-to-all seminars on new technologies, key tools, and concepts.
Conclusion
If you’re considering how to increase data democratisation in your current organisation (and as we’ve seen in Zopa), keep in mind that this is a gradual process in which incremental changes in culture bring small wins, which, in turn, fuel the next culture change. Your small wins will be effective only if they bring solutions to actual business problems, while the culture change will be rapid only if the wider organisation is aware of the wins. More on this on a follow-up blog post.
As you can see, we love data and analytics, and we enjoy democratizing them! So, if you are considering joining us as a data scientist, risk manager, product analyst, product manager, or digital optimisation manager, be prepared to get your hands dirty!
Vlasios joined Zopa in 2014, initially as part of a talented team of data scientists and later as the team’s head. Before Zopa, he had a successful career in computational Particle Astrophysics working at NASA and CNRS. Vlasios is passionate about data no matter whether they describe distant astrophysical explosions or credit profiles of Zopa loan applicants, and strives to understand the processes behind their generation. As a Zopa member, he has led/contributed to the creation of Machine Learning models for credit and fraud risk, pricing, and marketing. Currently, he is working towards improving Zopa’s analytical capabilities by overseeing a migration to a consolidated cloud-based data warehouse.