Dr. Farid Mehovic - Data Context Solution for Managing Data Platforms - January 17, 2025

Data Context Solution for Managing Data Platforms

Presented By: Dr. Farid Mehovic

Abstract
Today's data platforms are complex and require engagement of experts,
management via multiple contexts, and additional time for setup, maintenance, and use of the platform. The complexity comes from multiple areas involved, such as data ingestion, transformation, security, protection, governance, machine learning, etc., as well as a multitude of tools for each of those areas. The time aspect affects the usability of the platform the most, as for small companies slowly derived analytical insights may be going out of business, and for large it may mean losing billions in revenue and market share.

The solution offered by the author consists of a new type of SQL extensibility, which is of the language itself rather than building new instances of procedures and functions. In other words, SQL is used to extend itself, meaning to define new objects and new commands. We would note that this is actually being done in silos by different analytics database vendors and different platform areas, but not as a dynamic self-extending ability of SQL, but rather by static SQL development efforts. This causes the users to completely depend on the vendors, who may take months and years to extend the language in this static approach, if the user's request for functionality is even ever prioritized. Shown will be two examples of these extensions. One deals with the simplification of ingestion and transformation functionality. Normally this requires a tool like Airflow, combining Python and SQL, in order to introduce orchestration, parallelism, and dependency management. But a simple SQL extension with a few keywords could suffice. The other example is comparing integration with a machine-learning tool, like Sagemaker, using the existing SQL extensibility via user-defined function, with the new approach using actual data
context language extension.

Видео Dr. Farid Mehovic - Data Context Solution for Managing Data Platforms - January 17, 2025 канала CSE Department - University of Louisville