DENG-4298 Added managed backfill issues to docs #5909
Conversation
| @@ -284,6 +285,7 @@ For our example: | |||
|
|
|||
| 3. You will be notified when swapping is complete. | |||
|
|
|||
| **Note**. Please announce in the #data-platform-infra-wg Slack channel before deleting any tables in the `backfill_staging_derived` dataset since it may cause issues in the workflow. | |||
There was a problem hiding this comment.
I'm not sure what a good approach here is. Restricting delete access would be ideal... I think this could also be a good thing for the person on triage to handle.
There was a problem hiding this comment.
Ya I am asking @whd for options to restrict delete access to backfill_staging_derived.
If the person on triage would handle this then it means everyone on triage rotation list would have persmission to delete tables there?
There was a problem hiding this comment.
Yeah, maybe this just becomes a known fix for failed backfills during triage, depending on why the backfill failed. Some of this might make it into the DAG docs.
Co-authored-by: Alexander <[email protected]>
| @@ -267,6 +267,7 @@ For our example: | |||
| ```bash | |||
| bqetl backfill create <project>.<dataset>.<table> --start_date=<YYYY-MM-DD> --end_date=<YYYY-MM-DD> | |||
| ``` | |||
| **Note** Do not overwrite existing backfill entries since it will cause issues in the workflow. | |||
There was a problem hiding this comment.
| **Note** Do not overwrite existing backfill entries since it will cause issues in the workflow. | |
| **Note** Do not overwrite existing backfill entries since it will cause issues in the workflow (e.g. duplicate processing). |
| @@ -284,6 +285,7 @@ For our example: | |||
|
|
|||
| 3. You will be notified when swapping is complete. | |||
|
|
|||
| **Note**. Please announce in the #data-platform-infra-wg Slack channel before deleting any tables in the `backfill_staging_derived` dataset since it may cause issues in the workflow. | |||
There was a problem hiding this comment.
Yeah, maybe this just becomes a known fix for failed backfills during triage, depending on why the backfill failed. Some of this might make it into the DAG docs.
https://mozilla-hub.atlassian.net/browse/DENG-4298
We have noticed users overwriting existing backfill entries and deleting tables in staging dataset. Adding notes to the docs to bring awareness. It would be most ideal to implement CI checks to support this.
Checklist for reviewer:
<username>:<branch>of the fork as parameter. The parameter will also show upin the logs of the
manual-trigger-required-for-forkCI task together with more detailed instructions.For modifications to schemas in restricted namespaces (see
CODEOWNERS):┆Issue is synchronized with this Jira Task