🔔 Stay tuned. Data quality is a new capability in dScribe, and we're still expanding it. This article covers the fundamentals. Reach out if you want extra help.
You can now define data quality rules directly in dScribe. Rules are modeled on the Open Data Contract Standard (ODCS), so the quality expectations you capture in the catalog use an open, portable format you can sync with external data quality engines.
Data quality rules in dScribe are always part of a data contract. For more information, see Data Contracts.
What you can do today
Data quality rules let you express what "good" looks like for your data and attach those expectations right where the data is documented:
Define rules without code — build rules through a guided form; no SQL or scripting required (though SQL and custom options are there if you want them).
Apply rules at two levels — on a dataset as a whole, or on an individual dataset element (a column).
Stay tool-agnostic — rules follow the ODCS and are stored in .yaml files so you can execute them in your data quality engines of choice. For example: Soda, Great Expectations or Monte Carlo.
Write back loop - results of executed data quality rules can be written back and displayed in dScribe, so users immediately view data quality scores on data assets in the catalog
Adding a quality rule
Inside the data contract linked to a dataset or data product, go to Data Quality and click Add rule. The Add quality rule dialog walks you through a few choices:
Severity
Set how a failing rule should be treated: Info, Warning, or Error. This lets you separate purely informational checks from the ones that should raise a real red flag.
Rule type
Pick how the rule is defined:
Library — a pre-built metric. Choose a Metric (for example, Invalid Values), a Dimension (for example, Accuracy), and a Threshold (for example, must be greater than 0). This is the no-code path.
SQL — a custom query, for checks you'd rather express yourself.
Text — a documented-only rule: a quality expectation you want recorded in the catalog without an automated check behind it.
Custom — import from an external engine. For example: Soda, Great Expectations or Monte Carlo
Where rules show up
Once saved, a rule appears under Quality rules on the dataset or column it belongs to. If you have set up a sync with your data quality engine, each rule will also show the latest run state with more details.
Schema drift
When you include dataset elements in your data contract's schema, whenever these dataset elements get deleted or renamed in the source, a warning will automatically display in dScribe pointing this out. This helps avoid accidentally breaking datasets that stakeholders might depend on for their own data use cases.
Where to go next
→ More on data quality is on the way. We'll expand this article as the capability grows — reach out through the in-app chat if there's something specific you'd like to do.
Have a question or can't find what you're looking for? Use the chat icon inside the catalog to reach the dScribe support team.