Design a Job Scheduler¶

Job scheduler could be of many types, such as to schedule CPU jobs (multiprocessing), web server jobs (data application), Hadoop job scheduler, etc.

Celery like job scheduler¶

Celery pick up tasks from the web server (web application) manage the tasks using a broker service like message queue (Redis) to schedule the individual task in distributed workers. The tasks usually run in a first come and first serve manner.

• Celery Tutorial
• Redis in Action - 6.4 Task queues
• PagerDuty Scheduler Design

Airflow like data job scheduler¶

• Azkaban
• Airflow