Using Apache Spark 2.0 to Analyze the City of San Francisco's Open Data

Sameer Farooqui delivers a hands-on tutorial using Spark SQL and DataFrames to retrieve insights and visualizations from datasets published by the City of San Francisco. [Topics Indexed Below]

The labs are targeted for an audience with some general programming or SQL query experience, but little to no experience with Spark. Sameer will begin with some brief theory and lecture on Spark, before diving into several demos performing visualizations and analysis on calls made to the San Francsico Fire Department on July 4th.

Follow Along:
+ Databricks Community Edition: https://databricks.com/try
+ Labs: https://bit.ly/sfopenlabs
+ Learning Material: https://bit.ly/sfopenreadalong

-----Jump to Topic-----
00:00:06 - Workshop Intro & Environment Setup
00:13:06 - Brief Intro to Spark
00:17:32 - Analysis Overview: SF Fire Department Calls for Service
00:23:22 - Analysis with PySpark DataFrames API
00:29:32 - Doing Date/Time Analysis
00:47:53 - Memory, Caching and Writing to Parquet
01:00:40 - SQL Queries
01:21:11 - Convert a Spark DataFrame to a Pandas DataFrame
-----Q & A-----
01:24:43 - Spark DataFrames vs. SQL: Pros and Cons?
01:26:57 - Workflow for Chaining Databricks notebooks into Pipeline?
01:30:27 - Is Spark 2.0 ready to use in production?

----------------------------------------------------------------------------------------------
SPARK 2.0 TRAINING | NewCircle | Onsite & Public Classes
----------------------------------------------------------------------------------------------
+ Programming for Spark 2.0 (3 days)
+ Spark 2.0 for Machine Learning & Data Science (3 days)
Learn more: https://newcircle.com/category/apache-spark
++Code for San Francisco++
http://www.meetup.com/Code-for-San-Francisco-Civic-Hack-Night/

++Learn more about Databricks++
https://databricks.com/product/databricks

Видео Using Apache Spark 2.0 to Analyze the City of San Francisco's Open Data канала InfoQ