Data Engineering on Keep Moving https://sanghunka.github.io/tags/data-engineering/ Recent content in Data Engineering on Keep Moving Hugo -- gohugo.io en-us Wed, 20 Dec 2017 14:00:00 +0900 [airflow] 6. Multi cluster에서 airflow 실행하기 https://sanghunka.github.io/2017/12/airflow-6.-multi-cluster%EC%97%90%EC%84%9C-airflow-%EC%8B%A4%ED%96%89%ED%95%98%EA%B8%B0/ Wed, 20 Dec 2017 14:00:00 +0900 https://sanghunka.github.io/2017/12/airflow-6.-multi-cluster%EC%97%90%EC%84%9C-airflow-%EC%8B%A4%ED%96%89%ED%95%98%EA%B8%B0/ 요약 다루는 내용 분산 인스턴스에서 각각 airflow worker를 실행하고 task를 분산해서 실행하는법 task가 실행될 worker를 명시적으로 지정하는법 테스트 환경 두 개의 Amazon EC2 Instance 사용 1번 Instance에 아래와 같이 셋팅 metadata database(postsgres) rabbitmq airflow webserver airflow worker 2번 Instance에 아래와 같이 셋팅 airflow worker airflow configuration 1번과 2번 instance에 airflow를 설치한다. dag폴더에 동일한 파일을 넣어준다. dag폴더를 Git repository로 세팅하고 Chef, Puppet, Ansible등으로 동기화 해주는 방법도 있다. [airflow] 5. Pyspark sample code on airflow https://sanghunka.github.io/2017/12/airflow-5.-pyspark-sample-code-on-airflow/ Wed, 20 Dec 2017 13:00:00 +0900 https://sanghunka.github.io/2017/12/airflow-5.-pyspark-sample-code-on-airflow/ <p>Airflow에서 Pyspark task 실행하기</p> [airflow] 5. Pyspark sample code on airflow https://sanghunka.github.io/2017/12/airflow-5.-pyspark-sample-code-on-airflow/ Wed, 20 Dec 2017 13:00:00 +0900 https://sanghunka.github.io/2017/12/airflow-5.-pyspark-sample-code-on-airflow/ <p>Airflow에서 Pyspark task 실행하기</p> [airflow] 4. CeleryExecutor 사용하기 https://sanghunka.github.io/2017/12/airflow-4.-celeryexecutor-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B8%B0/ Tue, 05 Dec 2017 17:00:00 +0900 https://sanghunka.github.io/2017/12/airflow-4.-celeryexecutor-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B8%B0/ <p>Airflow CeleryExecutor 사용하기</p> [airflow] 3. LocalExecutor 사용하기 https://sanghunka.github.io/2017/12/airflow-3.-localexecutor-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B8%B0/ Tue, 05 Dec 2017 16:00:00 +0900 https://sanghunka.github.io/2017/12/airflow-3.-localexecutor-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B8%B0/ <p>Airflow LocalExecutor 사용하기</p> [airflow] 2. 튜토리얼 https://sanghunka.github.io/2017/11/airflow-2.-%ED%8A%9C%ED%86%A0%EB%A6%AC%EC%96%BC/ Tue, 21 Nov 2017 17:00:00 +0900 https://sanghunka.github.io/2017/11/airflow-2.-%ED%8A%9C%ED%86%A0%EB%A6%AC%EC%96%BC/ <p>pipeline을 따라 만들어보며 Airflow의 concept, object, usage를 습득하기.</p> [airflow] 1. 설치 https://sanghunka.github.io/2017/11/airflow-1.-%EC%84%A4%EC%B9%98/ Tue, 21 Nov 2017 16:00:00 +0900 https://sanghunka.github.io/2017/11/airflow-1.-%EC%84%A4%EC%B9%98/ <p>Airflow 설치하기</p> [airflow] 0. Quickstart https://sanghunka.github.io/2017/11/airflow-0.-quickstart/ Tue, 21 Nov 2017 15:00:00 +0900 https://sanghunka.github.io/2017/11/airflow-0.-quickstart/ # airflow needs a home, ~/airflow is the default, # but you can lay foundation somewhere else if you prefer # (optional) export AIRFLOW_HOME=~/airflow # install from pypi using pip pip install airflow # initialize the database airflow initdb # start the web server, default port is 8080 airflow webserver -p 8080 export AIRFLOW_HOME=~/airflow 명령어로 설치 경로를 지정할 수 있다. AIRFLOW_HOME을 지정하지 않을 경우 default 경로는 ~/airflow 설치는 pip로 간단하게 할 수 있다.