TaskScheduler contains a root Pool. There may be sub-pools under root pool. If it is FIFO mode, all tasks stay in the root pool. If it is FAIR mode, there may be multiple sub-pools based on the file specified by spark.scheduler.allocation.file. Using fairscheduler.xml.template as an example, there are two pools under root pool. One is named as production with FIAR mode, another one is test with FIFO mode. The order of executing tasks in each pool is decide by the scheduling mode. This is done by Pool.getSortedTaskSetQueue().

While computing resources are allocated and offered, Pool.getSortedTaskSetQueue() is used to get tasks to run based on the scheduler mode. If current pool's mode is FIFO, if tasks have the same priority(decided by job id), then the scheduling order is decided by their stage order. That makes sense, jobs with smaller ids should be executed first, and in each job, early stages with smaller stage id should be executed at first.

If it is FAIR mode, FairSchedulingAlgorithm is used to decide the order of tasks in the FAIR pool. All tasks and pools are treated as Schedulable object, so they can be sorted at the same time.

Starting from root pool, Pool.getSortedTaskSetQueue() tries to get the Schedulable objects with highest priority. If they are pools, then do the same thing as for the root pool, getting the tasks in this pool based on the scheduling mode. Tasks in higher priority pool will be run earlier than other tasks.

When launching spark job, we can specify which queue is used to run the job.