This is a guide to making a smooth transition to the latest version of Marquez.
Marquez 0.30.0
Marquez 0.30.0, which we released on 2023-01-31, contains some exciting new features, including column-level lineage and a soft delete option in the UI.
Along with these long-awaited UI upgrades come some significant performance improvements to the API. In prior versions of Marquez, accessing OpenLineage facets required direct queries against the OpenLineage events table, which could be quite costly depending on the size of this table. Some users have events tables with more than 100K rows -- millions of rows, in fact. To address this performance issue, Marquez 0.30.0 has new tables for dataset, job and run facets.
So far, so simple, right? Protecting against data loss and ensuring a smooth experience for users was not so simple, in fact. We needed a way to migrate the data while not only avoiding loss but also keeping downtime to a minimum. The solution that Willy and Pawel found (with input from Mike Collado) was to keep the data in the events table until the migration process is complete and to create views on top of the table for accessing the facets. After migration is complete, the views are replaced to point to the new tables. Using views provides safeguards and minimizes downtime during the migration.
Migration Procedure
As the
migration plan
says, users whose lineage_events tables have up to 100K rows can set it and
forget it -- no extra work is required beyond upgrading and restarting
Marquez, but be prepared for a brief downtime while a standard Flyway migration
fills the newly created tables automatically.
For heavy users whose lineage_events tables have more than 100K rows, a
different process will be used. Instead of a standard migration, the process will
happen asynchronously so that the user can start the API and consume new
OpenLineage events as usual. Note that some API calls might return results
slowly during the migration, especially if the output is based on facets.
To perform a migration, these users must run a new db-migrate command:
java -jar api/build/libs/marquez-api-0.30.0.jar db-migrate --version v57 ./marquez.yml
Available command arguments, version (required) and chunkSize (optional),
allow the user to specify the migration version to apply and the number of
lineage events to process (the default being 10,000). The command will process
the data not in its entirety but in chunks, storing a state with information
about the chunks being processed. Consequently,
- it can be stopped any time
- it will continue automatically and process the remaining chunks.
A sample command specifying version 57 and 50,000 events:
java -jar api/build/libs/marquez-api-0.30.0.jar db-migrate --version v57 --chunkSize 50000 ./marquez.yml
More Information
For more details about what, if anything, you will need to do to ensure a smooth transition to Marquez 0.30.0, check out the helpful migration plan put together by Willy and Pawel.
For more information about the improvements and additions in Marquez 0.30.0, check out the release on GitHub.