It’s no secret data is becoming every company’s most valuable asset. However, as we see an increasing variety of data available, companies are drowning in inputs and struggling to extract meaningful value from the depths of these datasets. This is largely due to a lack of clear standards on how to collect, format and store this information. Application Programming Interface proliferation has partly tackled this problem, but as we know technological progression isn’t dormant; new data accompanied by new formats are perennially being created. The result? Data silos. The good news is nearly all departments within an organisation have solutions to make sense of the data. The bad news? Merely using one solution within one department does not translate to business-wide insights and further inhibits companies from adopting and embracing the philosophy of a holistic data-driven approach.
To extract the most value from their data, companies need to eliminate the silo paradigm. By integration, the data is empowered, sanctioning business goals through truly transformative analytics. Executives will agree with Aristotle when he said “The whole is greater than the sum of its parts.” When intelligible data is digested collectively, its nutrients feed and transform future strategy bearing effective, accurate, and timely decision making.
What are data silos?
A data silo is a repository of data which has been sectioned or compartmentalised, keeping it separate from other data assets. It describes any tangible information collected in isolation and only accessible by a subset of an organisation.
Why do they exist?
Not all data is created equal. Silos are determined by source and result when specific systems are not integrated, however more forces than this are usually at play. The main reasons data silos continue to exist today can be summarised as either being structural, technological or cultural. Being able to identify the root cause or causes of such data segregation facilitates an organisation with the diagnostic tools to remedy the consequences of these divides.
More evident in large organisations, data silos occur because companies expand and change over time without consistent methodologies on how to manage their data. Each platform is optimised for its own data and insights. When development excludes integration, the data becomes isolated within its application. Internal hierarchies between layers of management and skilled employees impede the collaboration of data over time, especially when companies have experienced multiple leaders and ideologies. Structural resistance to cross-pollination of data precludes optimum utilisation of data collected. Basically, the data remains in organisational hibernation.
‘This is the way we’ve always done it.’ ‘What do they need our data for and do they know how to use it properly?’ ‘Jim is your man for campaign response analysis.’ These are common phrases echoing the many corridors and offices of companies struggling with various subcultures, despite aspiring to become more data-driven. The resistance to change comes as no shock but often people also have a sense of proprietorship over their data. Employees can become suspicious of external use and fear misuse of their data. Others may be hesitant to share because they want to maintain a sense of power and control, or on the other hand deem it too much effort for themselves and refer it to the expert, which only reinforces the silo already firmly in place.
As soon as companies begin working with multiple external vendors, data silos tend to increase. Coined as vendor lock-in, organisations become dependant on particular suppliers who keep the data to themselves, often providing systems that don’t prioritise nor were designed to be compatible or integratable with other solutions. Older SaaS (software-as-a-service) applications can strong-arm companies by requiring heavy investments in training, making it problematic to switch without incurring high costs. It’s almost as if companies become their own, self-saboteur.
Silos do not appear overnight but rather tend to emerge insidiously over time as each department has different objectives, priorities, cultures and systems. In addition, centralising data is difficult. Issues include duplicates, no common identifiers, conflicts within the data, inconsistent formats, permissions and the ever-changing nature of data. Whether it be unwillingness or inability, “Companies tolerate silos because fixing this issue requires some very fundamental changes, and that feels risky and difficult.” (Johann Wrede, Global VP of SAP Hybris)
Why are they bad news?
First and foremost, data silos preclude companies from seeing or even visualising the big picture, curtailing their ability to make accurately informed decisions. Decisions in these environments may be made with contradictory, misleading or downright wrong information. This causes missed opportunities and fuels a wave of short-sighted overconfidence which can result in future higher costs for the business.
Not only does segmentation restrict data from unearthing patterns that would otherwise reveal opportunities for growth, it also obstructs timely decision making. Extracting the information from disparate sources either becomes too time consuming or too complex and so the status quo is regularly accepted, labelling the task as ‘next year’s problem’. A creeping normality of data abstinence analysis.
Silos raise data integrity questions and impact the quality of insights. The same data can frequently be found in several silos with differing particulars. Without a single source of truth, confusion around which data represents the most up-to-date or accurate information can cause incorrect analysis. In addition, why pay extra to store the same data in multiple places?
Stunted workflow is another by-product of silos. When data collaboration doesn’t exist, it takes time to get the right information to the right person. Moreover, when it comes to actually performing the analysis, the only way to transfer information across departments is via manual, inefficient methods such as spreadsheets or chats which are fraught by human misinterpretation. Worse still, employees from different departments may end up performing the same analysis which causes more inefficiencies. Further to this, simple tasks (the type anyone with basic training could successfully complete) from other departments are being shovelled to data scientists which diverts their expertise away from larger projects, let alone increase the cost burden to organisations. Time-poor analysts prioritise, further adding to the delay of timely arrivals of requested reports to management.
How can we fix them?
Theoretically, a solution to data silos should embryonically begin with culture and the first step should be strategic. Before the implementation of technical phases, companies should ensure employees will embrace this change, however this seldom happens in reality. Regardless of whether a data-driven culture exists, data analytics must become a strategic priority lead from the top echelon down, in the form of a unified front. Executives have to be fully immersed in fostering an atmosphere of inter-departmental cooperation and collaboration. Providing incentives to work together, creating projects requiring cross-departmental collaboration, challenging departments to share responsibilities for the same business goals and initiatives to reduce the feeling of competition, showing departments how their work benefits other teams and the business as a whole are just to name a few philosophical approaches an organisation needs to adopt.
Once the stage and scene it set for this big project and everyone is on board and the plan is being shared, the next four steps are often required to tackle the existing data silo mindset:
(i) Elect a data expert
This person will liaise with each department to understand and appreciate their processes, systems and needs. The data expert is responsible for the entirety of the project which includes data governance and security.
(ii) Establish a central repository and standardise processes
If databases cannot be consolidated into a singular platform then new enterprise solutions must be explored. Standard rules for capturing, storing and analysing data are paramount as each departmental system will be running off the same repository.
(iii) Implement the right system
With an improved data infrastructure in place, this system should be able to smoothly analyse and visualise all available information for each department in some sort of customisable dashboard. Obsolete legacy systems should not be permitted to drag the rest of the business down.
(iv) Continual and collaborative training
This final (but by no means the least important) step ensures companies do not end up where they started. New tools means new training while continuing to foster a better culture through sharing information. The earlier the ‘buy-in’ opportunities are afforded to all employees, the better the engagement with less risk to instant disengagement, both from an emotional and physical perspective. The prospect of change can often result in an influx of ‘unnatural’ and unexpected staff attrition rates leaving corporate knowledge deficits for those remaining and for the project at hand.
While companies can deal with and try to manage their own silo issues, a larger market-wide trend is at play. We have to remember completely new data silos are being created on a daily basis. Take ‘smart buildings’ for instance. These networks of sensors have increasingly more inputs as technology advances. Even if we establish industry-wide standards, commercial incentives still exist to lock customers into the datasets available. Luckily, vendors are feeling the pressure to cease these lock-in practices and all productive paths are leading to data decentralisation. Blockchains are a perfect example as they strive to provide error-free, verified and integratable data with complete transparency and accountability resulting in pellucid data for analysis. “The best way to address the problem is to keep the data from going into silos in the first place.” ( Duncan Pauly, CTO of Edge Intelligence)