Sr. Data Engineer

What you will do:

• Design and implement high-throughput, low-latency FastAPI services to handle real-time and batch workloads for data and AI applications
• Build data pipelines using dbt and Redshift, ensuring models are accurate, scalable, and production-ready
• Work with Amazon Redshift to design, tune, and maintain large-scale analytical data stores.
• Develop and maintain ETL/ELT workflows for ingestion, transformation, and processing of structured and semi-structured data, supporting analytics and business intelligence needs.
• Leverage AWS services (S3, Lambda, Glue, Step Functions, Kinesis, etc.) to build robust, cloud-native data solutions.
• Ensure observability, reliability, and performance tuning for APIs and data workflows handling high-volume operations.
• Data Integration: Work with data from a variety of sources, databases, APIs, and files to create unified datasets optimized for analytics and machine learning.
• Data Quality & Governance: Implement data quality checks with tools like dbt Expectations to maintain data accuracy, integrity, and consistency, and ensure compliance with data governance practices.
• Performance Optimization: Enhance system performance, scalability, and efficiency by optimizing data processing systems.
• Cross Collaboration: Partner with data scientists, analysts, and software engineers to translate business needs into robust data solutions.
• Documentation: Maintain documentation for data architecture, workflows, and systems to support scalability and ease of maintenance.

What Cowbell needs from you:

• Educational Background: A Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a closely related field.
• Professional Experience: 6+ years in a Software Developer, Data Engineer, or similar role.

Technical Skills:

• Proven expertise in Python (async, concurrency, and multithreading), Pandas, Spark, building FastAPI (or equivalent) services for high-scale, production-grade use cases.
• Strong knowledge of real-time and batch data processing architectures.
• Solid understanding of Amazon Redshift, PostgreSQL performance tuning and query optimization.
• Extensive expertise with AWS ecosystem for data engineering (S3, DMS, Glue, Redshift, EMR, Kinesis, Athena, RDS, Lambda, CloudWatch, ECS, EC2).
• Competency with ETL/ELT principles and distributed systems.
• Strong understanding of data modeling, data architecture, and best practices in data governance.
• Strong problem-solving and communication skills, with the ability to work in fast-paced, cross-functional teams.

Soft Skills:

• Problem-solving abilities with a keen attention to detail.
• Capable of working both independently and within a team setting.
• Strong communication and collaboration skills.
• Flexibility to adapt to new technologies and a commitment to continuous learning.

Preferred Qualifications:

• Exposure to LLM integration and AI-powered APIs.
• Knowledge of Infrastructure-as-Code (Terraform, CloudFormation) and Kubernetes for deploying data/AI services.
• Experience with real-time data streaming technologies (e.g., Kafka, Flink) is a plus.
• Familiarity with containerization (Docker) and orchestration tools (Kubernetes).
• Understanding of data security, compliance, cyber security, and privacy best practices.