Data Engineer III (Databricks)

LivePerson (NASDAQ: LPSN) is the global leader in enterprise conversations. Hundreds of the world’s leading brands — including HSBC, Chipotle, and Virgin Media — use our award-winning Conversational Cloud platform to connect with millions of consumers. We power nearly a billion conversational interactions every month, providing a uniquely rich data set and safety tools to unlock the power of Conversational AI for better customer experiences.

At LivePerson, we foster an inclusive workplace culture that encourages meaningful connection, collaboration, and innovation. Everyone is invited to ask questions, actively seek new ways to achieve success, and reach their full potential. We are continually looking for ways to improve our products and make things better. This means spotting opportunities, solving ambiguities, and seeking effective solutions to the problems our customers care about.

Overview

We are looking for an experienced Data Engineer to provide data engineering expertise and support to various analytical products of LivePerson, and assist in migrating our existing data processing ecosystem from Hadoop (Spark, MapReduce, Java, and Scala) to Databricks on GCP. The goal is to leverage Databricks’ scalability, performance, and ease of use to enhance our current workflows.

You Will

• Assessment and Planning:
• Review the existing Hadoop infrastructure, including Spark and MapReduce jobs.
• Analyze Java and Scala codebases for compatibility with Databricks.
• Identify dependencies, libraries, and configurations that may require modification.
• Propose a migration plan with clear timelines and milestones.
• Code Migration:
• Refactor Spark jobs to run efficiently on Databricks.
• Migrate MapReduce jobs where applicable or rewrite them using Spark DataFrame/Dataset API.
• Update Java and Scala code to comply with Databricks' runtime environment.
• Testing and Validation:
• Develop unit and integration tests to ensure parity between the existing and new systems.
• Compare performance metrics before and after migration.
• Implement error handling and logging consistent with best practices in Databricks.
• Optimization and Performance Tuning:
• Fine-tune Spark configurations for performance improvements on Databricks.
• Optimize data ingestion and transformation processes.
• Deployment and Documentation:
• Deploy migrated jobs to production in Databricks.
• Document changes, configurations, and processes thoroughly.
• Provide knowledge transfer to internal teams if required.
  

Required Skills

• 6+ years of experience in Data Engineering with focus on building data pipelines, data platforms and ETL (Extract, transform, Load) processes on Hadoop and Databricks.
• Strong Expertise in Databricks (Spark on Databricks, Delta Lake, etc.) preferably on GCP.
• Strong expertise in the Hadoop ecosystem (Spark, MapReduce, HDFS) with solid foundations of Spark and its internals.
• Proficiency in Scala and Java.
• Strong SQL knowledge.
• Strong understanding of data engineering and optimization techniques.
• Solid understanding on Data governance, Data modeling and enterprise scale data lakehouse platform
• Experience with test frameworks like Great Expectations
  
  

Minimum Qualifications

• Bachelor's degree in Computer Science or a related field
• Certified Databricks Engineer- Preferred
  
  

You Should Be An Expert In

• Databricks with spark and its internals (3 years) - MUST
• Data engineering in Hadoop ecosystem (5 years) - MUST
• Scala and Java (5 years) - MUST
• SQL - MUST
  
  

Benefits

• Health: Medical, Dental and Vision
• Time away: vacation and holidays
• Development: Access to internal professional development resources.
• Equal opportunity employer
  
  

Why You’ll Love Working Here

As leaders in enterprise customer conversations, we celebrate diversity, empowering our team to forge impactful conversations globally. LivePerson is a place where uniqueness is embraced, growth is constant, and everyone is empowered to create their own success. And, we're very proud to have earned recognition from Fast Company, Newsweek, and BuiltIn for being a top innovative, beloved, and remote-friendly workplace.

Belonging At LivePerson

We are proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants with criminal histories, consistent with applicable federal, state, and local law.

We are committed to the accessibility needs of applicants and employees. We provide reasonable accommodations to job applicants with physical or mental disabilities. Applicants with a disability who require reasonable accommodation for any part of the application or hiring process should inform their recruiting contact upon initial connection.

• The talent acquisition team at LivePerson has recently been notified of a phishing scam targeting candidates applying for our open roles. Scammers have been posing as hiring managers and recruiters in an effort to access candidates' personal and financial information. This phishing scam is not isolated to only LivePerson and has been documented in news articles and media outlets.
  
  

Please note that any communication from our hiring teams at LivePerson regarding a job opportunity will only be made by a LivePerson employee with an @liveperson.com email address.

LivePerson does not ask for personal or financial information as part of our interview process, including but not limited to your social security number, online account passwords, credit card numbers, passport information and other related banking information.

If you have any questions and or concerns, please feel free to contact recruiting-lp@liveperson.com