airflow - Changes in DAG/Operator while DAG is running

Question

1. What would happen if DAG changes while DAG is running (particularly if this is dynamic DAG)
2. What would happen if code of custom made operator is changed during some DAG run

question from:https://stackoverflow.com/questions/65905412/changes-in-dag-operator-while-dag-is-running

Answer 1

So after some investigation I came to conclusion that DAG changes will be visible in current DAG run and that operators (and all other plugins) are not reloaded by default.

1. DAG changes during DAG run

   • If you remove task while DAG is running scheduler notice that, set status for that task to "Removed" while logging current line:

   State of this instance has been externally set to removed. Taking the poison pill.

   • If you add new task it will also be visible and executed. It will be executed event if it is upstream task of task which is already finished

   • If task is changed, changes will be incorporated into current DAG run only if task is not already started execution or finished

2. All plugins (included operators) are not refreshed automatically by default. You can restart Airflow or you can set reload_on_plugin_change property in [webserver] section of airflow.cfg file to True