Course Code

sparkdev

Duration

21 hours (usually 3 days including breaks)

Requirements

PRE-REQUISITES

familiarity with either Java / Scala / Python language (our labs in Scala and Python)
basic understanding of Linux development environment (command line navigation / editing files using VI or nano)

Overview

OBJECTIVE:

This course will introduce Apache Spark. The students will learn how Spark fits into the Big Data ecosystem, and how to use Spark for data analysis. The course covers Spark shell for interactive data analysis, Spark internals, Spark APIs, Spark SQL, Spark streaming, and machine learning and graphX.

AUDIENCE :

Developers / Data Analysts

Course Outline

Scala primer
- A quick introduction to Scala
- Labs : Getting know Scala
Spark Basics
- Background and history
- Spark and Hadoop
- Spark concepts and architecture
- Spark eco system (core, spark sql, mlib, streaming)
- Labs : Installing and running Spark
First Look at Spark
- Running Spark in local mode
- Spark web UI
- Spark shell
- Analyzing dataset – part 1
- Inspecting RDDs
- Labs: Spark shell exploration
RDDs
- RDDs concepts
- Partitions
- RDD Operations / transformations
- RDD types
- Key-Value pair RDDs
- MapReduce on RDD
- Caching and persistence
- Labs : creating & inspecting RDDs; Caching RDDs
Spark API programming
- Introduction to Spark API / RDD API
- Submitting the first program to Spark
- Debugging / logging
- Configuration properties
- Labs : Programming in Spark API, Submitting jobs
Spark SQL
- SQL support in Spark
- Dataframes
- Defining tables and importing datasets
- Querying data frames using SQL
- Storage formats : JSON / Parquet
- Labs : Creating and querying data frames; evaluating data formats
MLlib
- MLlib intro
- MLlib algorithms
- Labs : Writing MLib applications
GraphX
- GraphX library overview
- GraphX APIs
- Labs : Processing graph data using Spark
Spark Streaming
- Streaming overview
- Evaluating Streaming platforms
- Streaming operations
- Sliding window operations
- Labs : Writing spark streaming applications
Spark and Hadoop
- Hadoop Intro (HDFS / YARN)
- Hadoop + Spark architecture
- Running Spark on Hadoop YARN
- Processing HDFS files using Spark
Spark Performance and Tuning
- Broadcast variables
- Accumulators
- Memory management & caching
Spark Operations
- Deploying Spark in production
- Sample deployment templates
- Configurations
- Monitoring
- Troubleshooting

Testimonials

★★★★★

Richard is very calm and methodical, with an analytic insight - exactly the qualities needed to present this sort of course.

Kieran Mac Kenna

We know a lot more about the whole environment.

John Kidd

The trainer made the class interesting and entertaining which helps quite a bit with all day training.

Ryan Speelman

Ernesto did a great job explaining the high level concepts of using Spark and its various modules.

Michael Nemerouf

I liked the methodology expected by Jorge

Experian Colombia S.A

Machine Translated

Richard was very willing to digress when we wanted to ask semi-related questions about things not on the syllabus. Explanations were clear and he was up front about caveats in any advice he gave us.

ARM Limited

Small group (4 trainees) and we could progress together. Also the trainer could so help everybody.

ICE International Copyright Enterprise Germany GmbH

Ajay was very friendly, helpful and also knowledgable about the topic he was discussing.

Biniam Guulay - ICE International Copyright Enterprise Germany GmbH

I think the trainer had an excellent style of combining humor and real life stories to make the subjects at hand very approachable. I would highly recommend this professor in the future.