Apache Spark Overview - Part I

Apache Spark - manages and coordinates the execution of tasks of data across a cluster of computers

Cluster Manager:

• The cluster of machines Spark uses to execute the tasks managed by Cluster Manager
• When we submit Spark application to Cluster Manager, they allocate resources for our applications
• Spark application consists of Driver + Set of Executor Processor
• 3 types of cluster managers - Standalone, Yarn and Mesos

Driver:

• Driver runs main() function and sits in one of the nodes in cluster
• Driver does (1) Maintain information about Spark Application (2) Respond to program or input (3) For Analyzing, Distributing and Scheduling work
• Drivers assign work to Executors

Data:

• Data Sets which nothing but typed data
• Data Frame can span multiple computers for one simple reason as the data is large. it is untyped.
• Resilient Distributed Data (RDD)

Transformation:

• Core data structures of Spark are immutable (cannot be changed after created)
• Instruction to change data is called transformation
• Types of Transformation - Narrow (one input - one output) and Wide (one input - many outputs)

Action:

• To trigger transformation computation we need action
• 3 kinds of actions - (1) View data in console  (2) to collect in native objects of languages (3) write output to data sources

Spark UI:

That shows state of your job environment and cluster state