Featured: Senior Data Engineer at SpotHero

Location: Chicago

Job Type: Full-Time


Senior Data Engineer @ SpotHero:
SpotHero is seeking a Senior Data Engineer to join the Data Engineering squad. This squad interacts with data consumers such as Data Science, Marketing, Engineering, and Business Analysts to provide data platform solutions that meet their day-to-day needs and long term vision.

As a Senior Data Engineer, you’ll focus heavily on backend application development with a focus on building reusable infrastructure services for our stakeholders to enable them to model, store, access, process, and analyze SpotHero’s data. You’ll also design, instantiate, observe and maintain infrastructure services, both AWS-managed and open source solutions. As a Senior Data Engineer, you’ll influence the technology choices and patterns established for data-heavy workloads at SpotHero.

What will you do:
Work with our Analytics, Data Science, Marketing and other squads to understand their data storage and processing needs.

Be a key hands-on contributor to the design and implementation of our data platform solutions from the infrastructure layer up to the API.

Model and architect our data in a way that will scale with the increasingly complex ways we’re analyzing it.

Build robust pipelines that make sure data is where it needs to be, when it needs to be there.

Build frameworks and tools to help our software engineers, data analysts, and data scientists design and build their own data pipelines in a self-service manner.

Performance testing and engineering to ensure that our systems always scale to meet our needs.

Be a key member of the team focused on hands-on contribution to the implementation and operation of our data platform.

Contribute to more junior team members' development, via thoughtful reviews and your own exemplary work.

Key Responsibilities:

Data Modeling/Architecting

Design data models with a broad understanding of underlying systems.

Create approachable, thorough documentation of data models describing how to access their data in a performant way.

Build performant models that are consistent with accompanying documentation that are built with quality in mind.

Consult with stakeholders on the best practices for creation and deployment of data models and data flows.

Data Processing

Define and enforce service level agreements between products owned and stakeholders, including configuration of monitoring and alerting

Understand data lineage and dependencies between data pipelines.

Design, implement, and maintain complex data processing pipelines which involve multiple integration points, including those which rely on distributed systems like Kafka and Spark.

Influence data processing and infrastructure practices across all of SpotHero.

Determine the best architecture, batch or streaming, for applications being built.

Working with Infrastructure

Evaluate different architectures for new systems or changes to the company’s existing systems, and propose thorough, specific designs for implementing those architectures.

Provision new infrastructure in cloud environments.

Deploy and manage containerized applications running in Kubernetes.

Identify and remedy security, cost, and maintainability issues in the team’s infrastructure.

Manage and integrate autoscaling, logging, monitoring and alerting for the team’s systems.

Your experience:

We care about your abilities, not how you gained them.

You might demonstrate the capabilities below through any combination of relevant professional experience, experience in a research setting, formal education, self-guided learning, open source contributions, or public speaking / writing / teaching experience.

You are able to design and implement high-quality software in Python.

Experience using SQL to read and manage data.

Experience with Airflow, Luigi, Prefect, or other ETL scheduling tools.

You have experience provisioning and managing infrastructure with infrastructure-as-code tools (we use Terraform, but experience with similar tools like CloudFormation, Pulumi, or SaltStack is totally fine!)

Hands-on experience using multiple data platforms and tools (e.g. Airflow, Hive, Kafka, Postgres, Redshift, S3, Spark, Trino), and experience deploying, monitoring, and maintaining at least one of them

Experience designing and implementing software (pipelines, services and client libraries) that is run in Docker containers, automatically tested on a continuous integration (CI) system, and versioned in git. You have experience writing shell scripts, Makefiles, or other configuration to glue together these components.

Ability to deploy containerized software in Kubernetes, or sufficient experience in similar technologies like Apache Mesos or Amazon ECS.

Demonstrated experience designing and supporting technology intended to be used by other stakeholders.

Strong ability to communicate on both business and technology subjects.

Company Website:

Job Posted by: Maria Luna

How To Apply:

Link to apply: