3/28/2023 0 Comments Airflow gitSince Airflow itself is already open sourced, and it is generally not too much of an issue to reveal our Airflow dependencies, the whole solution of building and hosting Airflow is quite acceptable. But for enterprises, the issue has always been whether we can open source these code. Hosting repositories on GitHub or Docker Hub and using their free service sounds good.IaC-wise, each version of our own custom image is version controlled, and traceable. This way we don’t need to give away credentials to each developer. Security wise, only the pipeline is allowed to push images. Compared to the approach of training team member to actually build and push their own image, this approach is much easier in the sense that publishers don’t need to install Docker on their own computer, don’t need to understand Docker build command.And it is very easy to train our team member to execute this flow on their own. After introducing this workflow, the company’s infrastructure maintainer no longer need to help the team upgrade Airflow images.You can go to GitHub repository page > Settings > Secrets > Action to add your secret. In the login step, we need to provide our Docker hub login username and password.We set the trigger condition to on publish, so this pipeline is triggered if a release is issued on GitHub page.name: Publish Docker image on: release: types: jobs: push_to_registry: name: Push Docker image to Docker Hub runs-on: ubuntu-latest steps: - name: Check out the repo uses: - name: Log in to Docker Hub uses: with: username: $ build-args: | AIRFLOW_VERSION=2.3.2 AIRFLOW_EXTRAS=async,celery,cncf.kubernetes,dask,docker,grpc,http,ldap,postgres,redis,statsd,virtualenv PYTHON_BASE_IMAGE=python:3.8-slim-buster ADDITIONAL_PYTHON_DEPS=scikit-learn=1.0.2 slack-sdk=3.14.1 simplejson=3.17.6 glom=22.1.0 github/workflows/cd.yaml ( example file). git clone git checkout v2–3-stable git checkout -b custom-release git push -u origin custom-release In our example, let’s call it custom-release branch, and branch if from v2–3-stable. This just makes future pulls from upstream easier. I recommend you create a branch from the desired version first, and do not commit on the original airflow branch. Now, you want to add your own build pipeline file.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |