Katie Simmons, a data engineer at ActiveCampaign, will speak about the challenges and benefits of using Airflow for ETL at a rapidly growing company. ActiveCampaign has many thousands of databases - some including tables with up to a trillion rows - several APIs and new source requests coming in every week. This lightning talk will be an overview of using Airflow to extract, load and transform that data into our data lake so that it can be used for Business Intelligence and Data Science.
One of the most common languages used by Python developers is some shell script (sh, bash, cmd.exe, or PowerShell), but why torture yourself with poor design decisions from the 70s when you know Python?
Not all data is easily accessible. Taking info from a website that requires authentication, interaction, or even just to load a fancy script. This talk will discuss using Selenium to level up your web scraping skills, with examples and suggested practices.
Failure can be scary. There are real costs to a company and its users when software crashes, models are inaccurate, or when systems go down. The emotional stakes feel high-- no one wants to be responsible for a failure. We can lower the stakes by creating spaces to learn from failures, and minimize their impact. This talk introduces two ways to address failure: blameless post-mortems, to learn from an incident; and pre-mortems, to identify modes of failure upfront.