cat iris.txt | python test.py
cat iris.txt | python test.py
pipeline을 따라 만들어보며 Airflow의 concept, object, usage를 습득하기.
Airflow 설치하기
# airflow needs a home, ~/airflow is the default, # but you can lay foundation somewhere else if you prefer # (optional) export AIRFLOW_HOME=~/airflow # install from pypi using pip pip install airflow # initialize the database airflow initdb # start the web server, default port is 8080 airflow webserver -p 8080 export AIRFLOW_HOME=~/airflow 명령어로 설치 경로를 지정할 수 있다. AIRFLOW_HOME을 지정하지 않을 경우 default 경로는 ~/airflow 설치는 pip로 간단하게 할 수 있다.
1. Introduction 2. Quiz: Input Data How to find total sales/store?
KEY VALUE time store name cost store name store name cost store name product type 3. Quiz: Defensive Mapper Code # Your task is to make sure that this mapper code does not fail on corrupt data lines, # but instead just ignores them and continues working import sys def mapper(): # read standard input line by line for line in sys.
1. Quiz: HDFS Which of the following is true?
HDFS uses a central SAN(storage area network) to hold its data HDFS stores a single copy of all data HDFS replicates all data for reliability To store 100TB of data in a Hadoop cluster you would need 300TB of raw disk space by default 2. Quiz: DataNode Which of the following is true if one of the nodes running the DataNode daemon on the cluster fails?
