How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations • Sam Bail • GOTO 2023

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo
https://gotochgo.com

Sam Bail - Principal Data Engineer at Collectors

ORIGINAL TALK TITLE
Bridging the Gap: How Data and Software Engineering Teams Can Work Together to Ensure Smooth Data Integrations

RESOURCES
https://sambail.com
https://github.com/spbail
https://linkedin.com/in/spbail

ABSTRACT
Imagine this scenario: The engineering team at your company is about to launch a new product or feature. The pressure is on the data team to deliver metrics and stats from the get-go, as everyone is keen to see how the new feature is performing. Behind the scenes, the data team is scrambling to understand how to connect to the new systems, transform the data, interpret it, and turn it into actionable insights.

In this talk, I will walk the audience through the different steps necessary to ensure both data teams and software engineers are ready to integrate with a new production system and deliver correct and reliable analytics from day 1. I will cover topics such as infrastructure and data access considerations, data models, data flow and state changes, and data quality checks. I will also touch upon “data contracts”, a concept that’s been talked about a lot recently in the data space, and how these can be established to preserve consistent high-quality data integrations.

Both data practitioners and software engineers will benefit from this talk and learn how to “bridge the gap” between the two worlds. [...]

TIMECODES
00:00 Intro
00:46 TL;DR
01:55 About me
03:31 Agenda
04:16 The problem
06:42 1. Logistics
13:44 2. Infrastructure
19:29 3. Data model
26:40 4. Application & data flow
33:27 Data contracts
38:13 Wrapping up
41:48 Outro

Download slides and read the full abstract here:
https://gotochgo.com/2023/sessions/2572

RECOMMENDED BOOKS
Phil Winder • Reinforcement Learning • https://amzn.to/3t1S1VZ
Kelleher & Tierney • Data Science (The MIT Press Essential Knowledge series) • https://amzn.to/3AQmIRg
Lakshmanan, Robinson & Munn • Machine Learning Design Patterns • https://amzn.to/2ZD7t0x
Lakshmanan, Görner & Gillard • Practical Machine Learning for Computer Vision • https://amzn.to/3m9HNjP
Aurélien Géron • Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow • https://amzn.to/2XZaQy8

https://twitter.com/GOTOcon
https://www.linkedin.com/company/goto-
https://www.instagram.com/goto_con
https://www.facebook.com/GOTOConferences
#DataScience #DataEngineering #DataTeam #SoftwareEngineering #Programming #DataIntegration #DataFlow #ApplicationFlow #SLA

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech
Sign up for updates and specials at https://gotopia.tech/newsletter

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
https://www.youtube.com/user/GotoConferences/?sub_confirmation=1

Видео How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations • Sam Bail • GOTO 2023 канала GOTO Conferences