Databricks Engineer Job at Conflux Systems, Phoenix, AZ

  • Conflux Systems
  • Phoenix, AZ

Job Description

Databricks Engineer

We are seeking a highly skilled Databricks Engineer to lead the design, implementation, and optimization of scalable cloud-based data platforms. This role requires deep expertise in Lakehouse architecture, Databricks, Apache Spark, and Delta Lake, along with proven experience delivering end-to-end enterprise data engineering solutions.

Key Responsibilities

  • Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
  • Lead implementation of:
  • Cloud-based data lakehouse platforms integrating diverse data sources.
  • Real-time data processing pipelines for operational and analytical use cases.
  • Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.
  • Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
  • Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
  • Establish data quality frameworks and governance standards.
  • Implement and manage Unity Catalog, including fine-grained security and access controls.
  • Leverage Databricks components such as:
  • Delta Live Tables
  • Autoloader
  • Structured Streaming
  • Databricks Workflows
  • Integration with orchestration tools (e.g., Apache Airflow)
  • Drive CI/CD automation, deployment strategies, and DevOps best practices.
  • Optimize performance of pipelines, Spark jobs, and compute resources.
  • Provide architectural guidance and technical leadership across cross-functional teams.
  • Engage with stakeholders and clients to translate business requirements into scalable technical solutions.

Deep expertise in:

  • Databricks and cloud-native storage/compute platforms
  • Apache Spark (batch & streaming)
  • Delta Lake & Lakehouse architecture
  • Distributed data processing systems
  • Strong hands-on programming skills in Python, PySpark, Scala, and SQL.

Job Tags

Similar Jobs

Headway

Psychiatric Nurse Practitioner Job at Headway

Headway seeks a Licensed Psychiatric Nurse Practitioner to expand access to high-quality mental health care through a tech-enabled, outpatient model. In this remote role, you will provide psychiatric evaluations, diagnosis, and ongoing medication management for a diverse... 

Trans-Bridge Inc

Motorcoach Driver - CDL Preferred Job at Trans-Bridge Inc

 ...industry or those who are excited to try a new profession. Paid training provided by our expert instructors, and a variety of schedules and...  ....Must have a valid driver's license and the ability to obtain CDL with provided training.Trans-Bridge will train and help obtain CDL... 

Infosys Technologies

Drupal Developer Job at Infosys Technologies

 ...coordinate bug fixes to uphold the software quality standards Develop user training programs, documentation, and support frameworks to...  ...Microservices, and cloud-native architectures. Building and maintaining Drupal 9/10/11 applications, custom modules, themes, workflows,... 

Virtual Vocations Inc

Document Retrieval Specialist Job at Virtual Vocations Inc

 ...Supporting the Bill Review process, the full-time Document Retrieval Specialist will accurately enter, review, and validate healthcare claim information and documentation in a remote environment.Key responsibilitiesEnters and maintains healthcare claim information,... 

US Postal Service

Postal Clerk - Columbia Furnace Job at US Postal Service

NO EXPERIENCE REQUIRED PAID TRAINING PROVIDED JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...