Skip to content

Design a Job Scheduler

Job scheduler could be of many types, such as to schedule CPU jobs (multiprocessing), web server jobs (data application), Hadoop job scheduler, etc.

Celery like job scheduler

Celery pick up tasks from the web server (web application) manage the tasks using a broker service like message queue (Redis) to schedule the individual task in distributed workers. The tasks usually run in a first come and first serve manner.

Airflow like data job scheduler