Effectively managing communication between your data and product teams
In an ideal workplace, communication between colleagues is open, friendly, informative and professional. But as teams have moved towards remote work, it has become much harder to have communication between teams. Although Slack and Microsoft teams exist, there are times when information exchanged between colleagues is not recorded in these tools, making it tough to use them as the central source of truth. Unfortunately, the majority of workplaces struggle with communication. Misunderstandings, communication and missed deadlines all add up to quite a bit of stress for everyone involved.
One particular area of the organization where communication is not as efficient as it could be is between the data and product team. The data team relies on the product team to update them on the changes being made to the data model that might impact the downstream data resources. Unfortunately, changes are made so quickly to the data model that it can feel almost impossible to update all the appropriate stakeholders every time there is a minor change in the data model. This communication problem only becomes harder to manage as companies scale. As companies become larger, the models tend to change at a faster rate. Managing the data model becomes much less about a central data team that manages every data asset and more about every team being responsible for exposing their data to a centralized platform that is easy to search. Without the right tools and processes in place, this can become a bottleneck for teams trying to grow quickly.
Some folks in the data space claim that the best way to solve this problem is to become close friends with software engineers. Once you buy them coffee in the morning, compliment them and tell them that their work is invaluable, they will start to give you a heads up on new features that might impact the data team. This is not easy to do at scale.
Some people might diagnose this as a leadership problem. Because the software development team isn’t monitored on their contribution to the data model or whether data breaks, it makes it tough to have them involve the data team in every decision. Additionally, there are data quality issues that may be created when they change the data model, which never affects their work. This type of thinking creates data debt. Unfortunately, talking to the team doesn’t seem to solve the problem because they have been directed to focus on their product.
There are many instances where leadership may not know that this is a problem. Without haven’t a direct cost and cause associated with data breaking, it’s tough for leadership to identify the root cause of the problem. For many teams, the problem has become one that they just have to live with and instead of being proactive about changes, they react to data breaking only after someone on the leadership team tells them that something is incorrect with the data.
How we can solve this problem?
Although this problem is difficult to manage on growing teams, it’s not impossible. This problem becomes harder to solve as the software engineering, product and data teams become more siloed in the organization. One way to solve this problem is to assign a data product manager, who is responsible for facilitating the communication between the two departments. But this alone is not enough. Below, we’ve outlined three areas that teams can focus on to reduce the number of times that this kind of problem occurs.
- People: The first step teams should take is to find lines of communication that are frequent and detailed. One way some teams solve this is through the Data Product Manager (DPM) role, whose job is to manage the data requests backlog and to communicate with the software and product teams to understand what may affect the data team in the future. By working with other product managers and by communicating to others across the organization how their work may impact the data and unlimitedly, the decisions that the company makes, the data product manager can advocate for the data team.
- Process: Most data teams already use Jira or some other tools to manage data requests. We recommend creating a process for every formal PRD - no matter the size - to include a data and reporting component in the scope. This way, data teams are informed about the changes happening upstream and what the impact of the new feature is on the business. Before any feature can be sure, this PRD should be reviewed by the software and data product managers to make sure that nothing was missed from the scope of the feature. Teams should also think about ways to communicate data knowledge externally such as the data dictionary and data analysis. Ideally, all data knowledge is centralized in one place that everyone can access.
- Tech: Lastly, teams should make sure that there are webhooks that flag data model changes in an automated way whenever there’s a new PRD that proposes data model changes. Our team is working on a new feature that allows software developers to use a .yml template to automatically document changes / new schema to the data model. The idea is that instead of finding out about changes to the data warehouse reactively, data teams could implement something like this to become proactive about changes to the schema. If your team wants to test this feature out, feel free to reach out to us.
There’s a problem to be solved in this space, as most teams have been struggling to find a good solution to communicate with product teams. Centralizing all the definitions and reacting proactively to changes in the data model is the best way to start chipping away at this problem. Even if it feels like it’s a huge hill to climb, we promise it’s well worth climbing!