RPC Diagram

RpcEndpoint

Key components in Spark, like Master, Worker, Driver, implement RpcEndpoint to have the capability to communicate with each other. RpcEndpoint.receive() should be overriden to process messages.

RpcEnv

RpcEnv encapsulates RPC components, including Dispatcher, TransportClient, TransportServer. When an RpcEndpoint is created, an RpcEnv is set up. Currently, Spark uses Netty as underlying RPC transport framework. TransportClient and TransportServer act client and server while transporting data between them. After RpcEnv has been initialized, an RpcEndpoint for current Spark component is created through RpcEnv.setupEndpoint(), which will register the endpoint into Dispatcher.

Dispatcher

As its name indicates, it works as a center to dispatch messages for the endpoint. While sending a message, the message will be delivered to Dispatcher. Some of them is saved into the Outbox, and other messages will be sent out by TransportClient. Messages can be sent in one-way, without receiving response from receivers. Or be sent waiting for a response in sync / async way.

Once the message is sent out and reaches at the receiver, a Netty handler named TransportRequestHandler processes it. This message will also be delivered to Dispatcher, saving it into a message queue. While Dispatcher starts, a thread pool executing MessageLoop is also started, which consumes the incoming messages in the message queue. The MessageLoop invokes the RpcEndpoint.receive().