An Introduction to Higher Order Functions in Spark SQL with Herman van Hovell (Databricks)
Nested data types offer Apache Spark users powerful ways to manipulate structured data. In particular, they allow you to put complex objects like arrays, maps and structures inside of columns. This can help you model your data in a more natural way.
While this feature is certainly useful, it can quite bit cumbersome to manipulate data inside of complex objects because SQL (and Spark) do not have primitives for working with such data. In addition, it is time-consuming, non-performant, and non-trivial. During this talk we will discuss some of the commonly used techniques for working with complex objects, and we will introduce new ones based on Higher-order functions. Higher-order functions will be part of Spark 2.4 and are a simple and performant extension to SQL that allow a user to manipulate complex data such as arrays.
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
Видео An Introduction to Higher Order Functions in Spark SQL with Herman van Hovell (Databricks) канала Databricks
While this feature is certainly useful, it can quite bit cumbersome to manipulate data inside of complex objects because SQL (and Spark) do not have primitives for working with such data. In addition, it is time-consuming, non-performant, and non-trivial. During this talk we will discuss some of the commonly used techniques for working with complex objects, and we will introduce new ones based on Higher-order functions. Higher-order functions will be part of Spark 2.4 and are a simple and performant extension to SQL that allow a user to manipulate complex data such as arrays.
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
Видео An Introduction to Higher Order Functions in Spark SQL with Herman van Hovell (Databricks) канала Databricks
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![LLM Module 4: Fine-tuning and Evaluating LLMs | 4.4 Fine Tuning: Few-shot Learning](https://i.ytimg.com/vi/3MPbjtVGt9M/default.jpg)
![LLM Module 4: Fine-tuning and Evaluating LLMs | 4.2 Module Overview](https://i.ytimg.com/vi/cJsIDwK3Nok/default.jpg)
![LLM Module 4: Fine-tuning and Evaluating LLMs | 4.1 Introduction](https://i.ytimg.com/vi/lNd1R-Fuiuo/default.jpg)
![LLM Module 3 - Multi-stage Reasoning | 3.7.1 Notebook Demo Part 1](https://i.ytimg.com/vi/PJcGkI1m9jQ/default.jpg)
![LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.1 Notebook Demo Part 1](https://i.ytimg.com/vi/GKqtMBkFotA/default.jpg)
![LLM Module 2 - Embeddings, Vector Databases, and Search | 2.6 Best Practices](https://i.ytimg.com/vi/3H5UC2c0Dyc/default.jpg)
![LLM Module 2 - Embeddings, Vector Databases, and Search | 2.2 Module Overview](https://i.ytimg.com/vi/X5DZL58mBg0/default.jpg)
![LLM Module 2 - Embeddings, Vector Databases, and Search | 2.3 How does Vector Search work](https://i.ytimg.com/vi/35_Nzo-AYzc/default.jpg)
![LLM Module 2 - Embeddings, Vector Databases, and Search | 2.1 Introduction](https://i.ytimg.com/vi/ExQ07beeMxQ/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.8 Summary](https://i.ytimg.com/vi/t5nydHEQT88/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.9 Notebook Demo](https://i.ytimg.com/vi/4XpRlLR6OFg/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.6 Prompts](https://i.ytimg.com/vi/9hSuKpP0EjU/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.7 Prompt Engineering](https://i.ytimg.com/vi/zra2zr8r_n4/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.4 Model Selection](https://i.ytimg.com/vi/JUj6tgbsKhE/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.5 NLP Tasks](https://i.ytimg.com/vi/NnALfJaLGFU/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.3 Hugging Face](https://i.ytimg.com/vi/-T8iDxLMuuk/default.jpg)
![LLM Module 1 - Applications with LLMs | 1.2 Module Overview](https://i.ytimg.com/vi/cdw-T4mR7k4/default.jpg)
![LLM Module 0 - Introduction | 0.6 Word Embeddings](https://i.ytimg.com/vi/o65ueTIk6YE/default.jpg)
![LLM Module 0 - Introduction | 0.4 Language Models](https://i.ytimg.com/vi/n69wohzQSeM/default.jpg)
![LLM Module 0 - Introduction | 0.3 Primer](https://i.ytimg.com/vi/HYhUno0xc70/default.jpg)
![LLM Module 0 - Introduction | 0.5 Tokenization](https://i.ytimg.com/vi/mPCDi7ZQCls/default.jpg)