How is it created

While creating SparkSession, SparkContext will also be created in getOrCreate. TODO - Add more info.

Things in SparkContext

As a function entry for Spark programming, SparkContext wraps lots of key components, including:

  • LiveListenerBus: Message center for SparkListenerEvent, like StageSubmitted, StageCompleted, etc. Listeners will receive events and update their status, like UI update.
  • SparkEnv: provides serializer, MapOutputTracker to track the output from map, ShuffleManager, BroadcastManager, MemoryManager.
  • HeartbeatReceiver - TODO
  • SchedulerBackend
  • TaskScheduler
  • DAGScheduler
  • ExecutorAllocationManager - TODO

results matching ""

    No results matching ""