Introducing Google Composer for data pipelines orchestration

Benefits and improvements for our customer:

Almost no delays in processing files, since there are sensors implements which wait for a file to appear and trigger immediately processing (instead of a person manually checking if file exists and triggering web app);
More stable architecture (retries implemented if any failure appeared);
Depending on the use case or in other words, depending on the report size, different tools to process files were picked e.g. Spark Serverless, minor files in Airflow workers;
Notification system added to inform if anything failed;
A lot of time was freed up of reporting team to spend on analysis rather than on triggering processing.