It's always been surprising to us that data discovery tools aren't great at providing analysts with a good workflow for updating documentation. Data engineers and analysts need a true version controlled approach to data management and documentation. We can't take the "just ship it" approach and just make a change without verifying the changes and confirming that they are correct. But, it seems that data management tools haven't prioritized this workflow.
About Our Change Management System
Today, we are excited to announce Secoda's new "Publishing workflow" and change management system, which allows data teams to create a software engineer like approach for updating data documentation. With this new change, teams can submit changes for review and publish changes to a company wide "Data Portal" that contains all associated information about your data. Engineers who write code need to confirm and render the output before merging to production, but analysts are not able to do this with data knowledge today. The problem with existing data catalogs and discovery tools is that they don't focus on publishing, approving, reproducing, and iterating on data knowledge across the company. With this new change, data teams will be able to manage their data discovery tool like a product.
This new change helps rethink change management and publishing workflows around data knowledge.
In the past, teams have been able to mark resources as “published” or “draft” to select which resources are displayed to viewers. With this new change, admins are able to publish on a workspace level to approve all the changes made by different editors and admins. Now, you can slowly approve changes and also change the logo and background of your company data portal.
Prior to this change, data teams that are working with cataloging tools had no versioning, change management or collaboration capabilities that software engineers are privy too.
By avoiding these features (which are traditional to software engineering practices in tools like Gitbook, Github etc.,) data catalogues have mostly operated in single player mode. With these new changes, teams can actually manage their documentation asynchronously in Secoda.
On a more meta level, we think that this feature is the stepping stone towards rethinking data catalogues. We think about Secoda as a data portal for all of your data knowledge. With the publishing workflow introduced, the process of creating content in Secoda is contributing to a growing data platform that represents your company knowledge. Tools like Looker, Hex & dbt already pioneered the publishing workflow for BI and Transformation tools, and we're hoping that this change does the same for data discovery.
We’re proud of the way that this change can help data teams work collaboratively and manage approvals they are familiar with in Git. Below is an overview of the feature.
How it works
By default, all resources are marked as “draft” when they are brought into Secoda. The statues of a resource can be viewed in the Status column, as pictured below.
By changing the contents of these resources in Secoda, editors now are able to stage change for publishing, which can be approved by admins in the workspace. Below, the “verified” tag has been added to the three resources. After tagging these resources, a couple of changes appear to an editor:
- An orange dot appears beside the resource
- The status changes to “staged for publishing”
By hovering over the orange dot, any editors or admins are now able to see the changes that were made to a resource, and what changes will be pushed to our viewer mode once approved.
Approving changes and publishing your workspace
At a workspace level, two new changes will be noticed that fit into the publishing workflow. These changes are the “publish” and “preview” button in the top right hand corner
.png)
By pressing the “publish” button, editors and admins are able to see all the changes made by editors and admins, and select the changes they are ready to “publish” to viewers
Admins can any resources and choose to publish or discard the changes made by editors in their workspace. Once the changes are “published”, viewers will see the changes in their Secoda workspace.
To confirm that these changes are live for viewers, admins and editors will be able to see the resources that are currently published to viewers using the status column, which will now show the resources as published.
Customizing your data portal
The can also press on the “preview” button beside the “publish” button to preview the viewer workspace. Once in the viewer workspace, you can exit preview by pressing the “exit preview” button. Viewers will be able to search and explore this mode of the data portal, but will not be able to exit preview to edit changes.
If editors want to change anything to a published resource, the button colour will be green instead of orange and the status column will identify changes as “Staged changes” that will need to be approved by an admin or editor.
We wanted to thank the team at Airbnb for publishing and sharing their thoughts on their internal Data Portal, which has led to our current product vision. Company data knowledge is vast and needs to service many stakeholders and use cases. Even if a company adopts a data discovery tool, employees have to search through four separate locations when trying to find any data-related information: the data catalogue, a data dictionary, BI tools or Google Slides for analysis, and a Slack channel of common questions. With so much separation around search, most data knowledge continues to disappear into obscurity. We hope that changes help users share their workspace with viewers faster as well as encourage a better workflow for data and engineering teams working in Secoda so that no information gets lots and forgotten.
Built for better data collaboration
If this feature is adopted by our customers, we have plans to expand the feature to include the following:
- Automatic versions created when publishing using our git integration for version history
- Introducing the ability to see who changed the resource and when it was changed
- Adding a change-log to published changes
All of these changes should help create processes around data documentation. If you like this feature or have ideas about how we could improve it, please let us know.
Try Secoda for Free
Secoda's version control feature ensures that changes to datasets are tracked, documented, and auditable, allowing data governance teams to have a clear understanding of data lineage and historical changes. This capability enables teams to detect and resolve discrepancies or errors promptly, ensuring data integrity throughout the data lifecycle. Additionally, Secoda's version control facilitates collaboration among data governance teams, data analysts, and other stakeholders, fostering a transparent and accountable data governance process. By utilizing Secoda's version control, data governance teams can confidently manage and monitor data assets, reduce the risk of data-related issues, and uphold data quality standards across the organization.
