Secoda and Great Expectations Integration

Everyone from communities to executives is embracing the importance of data quality, and so are we here at Secoda. We know that data enablement isn’t complete without understanding the quality of the data being ingested and generated.

That’s why Secoda users can now integrate Great Expectations with their Secoda workspace.

Why data quality and data enablement

Data enablement doesn’t stop at data assets—sharing the quality of those assets is just as important in bridging the gap between analytics and business teams to build trust.

We recently published a blog on what data enablement is and why it’s the next step in the evolution of data catalogs. Cataloging data isn’t tool-specific—it involves writing descriptions for data tables and columns. What a catalog is missing, however, is integrations, context, and easy collaboration.

A huge component of this context should include data quality. A data catalog that doesn’t account for data quality is like a Yelp page without reviews. How can you trust a business does what it says it does? How do you ensure you’ll get a good experience? Adding context isn’t inherently part of cataloging. Secoda, as a data enablement tool, allows its customers to search and share data knowledge. However, sharing this knowledge would be counterproductive if the quality of the data itself wasn’t ensured.

Data quality tools shed light on what business users should expect of their data and when those expectations aren’t met. Integrating data quality into data-sharing tools gives users the ability to automatically unpublish data assets that aren’t passing quality tests.

What is Great Expectations

Great Expectations is a data validation framework and platform, providing the guard rails to write succinct tests (or “Expectations”) for data. These Expectations are then validated (in “validations”) as part of data pipelines, stopping downstream dependencies from updating if the data validation doesn’t meet expectations like exceeding null count or value range.

Great Expectations is an open-source framework for testing and validating data. It allows data engineers and data scientists to write tests for their data, similar to how software developers write tests for their code.

The framework provides a wide range of tools and features that help data professionals to test their data pipelines, ensure data quality, and improve data documentation. Some of the key benefits of using Great Expectations include:

1. Improved data quality: Great Expectations provides a standardized way to test data, making it easier to identify data quality issues early on. This helps data teams to prevent data-related problems and ensure that their data is accurate and reliable.
2. Simplified testing: With Great Expectations, data teams can write tests in a variety of formats, including SQL, Pandas, and JSON, making it easier to test data pipelines without requiring specialized coding skills.
3. Better documentation: Great Expectations automatically generates documentation for data tests, making it easier to understand how data is transformed and ensuring that documentation stays up to date as pipelines change over time.
4. Collaboration: Great Expectations can be integrated with version control systems like Git, allowing data teams to collaborate on tests and share them across teams.

Overall, Great Expectations is a useful framework because it helps data teams to ensure data quality, simplify testing, and improve data documentation. By using Great Expectations, data teams can more easily identify and resolve data quality issues, ensuring that their data is accurate and reliable.

In Secoda, the Expectations coded within Great Expectations are associated with their respective tables. In the table view, you’ll see which validations are currently passing or failing.

By using Great Expectations, both data engineers and data scientists can write unit tests for data, which can be included in the data documentation known as Data Docs. These Data Docs contain both the Expectations and the Validations that run against the tests, and are part of the open source package. To make Data Docs useful, they need to be shared and understood, which is why a centralized data discovery tool like Secoda is an ideal place to highlight data quality to stakeholders.

Analytics teams benefit from a concise view of the state of all data, while business stakeholders gain trust in the fact that data is tested and understood. If any tests fail, the lineage view provides insight into what may have caused the failure.

Data quality is just another piece of the puzzle when it comes to understanding your data.

What’s next for Secoda

On the roadmap is expanding data quality integrations with other tools customers use like Metaplane, Bigeye, and Datafold. We believe high visibility into data quality will push organizations forward in building data trust and increasing collaboration.

For Secoda users, you’ll find instructions to connect to Great Expectations in your dashboard.

If you’d like to bring data quality and documentation under one platform, try Secoda for free.