job description
Data Pipeline Design and Development: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements. Design, develop, and implement efficient data pipelines, integrating various data sources to transform raw data into usable formats.
Cloud Data Platform: Utilize your expertise in working with cloud-based data platforms, especially Azure, to architect, deploy, and maintain scalable and reliable data infrastructure.
Databricks Expertise: Demonstrate a strong understanding of Databricks and its capabilities, using it as a primary platform for big data processing, analytics, and machine learning.
Data Modelling: Create and optimize data models that support data analytics, reporting, and visualization needs. Ensure data integrity, accuracy, and consistency.
Performance Optimization: Continuously monitor and fine-tune data pipelines and processing workflows to enhance overall performance and efficiency, considering large-scale data sets.
ETL (Extract, Transform, Load) Processes: Implement ETL processes to extract data from diverse sources, transform it into suitable formats, and load it into the data warehouse or analytical systems.
Data Security and Compliance: Ensure data security and compliance with data privacy regulations throughout the data engineering process.
Collaboration: Work closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements, share insights, and provide technical support.
Troubleshooting and Issue Resolution: Proactively identify data-related issues and provide timely resolutions to maintain smooth data operations.
Documentation: Maintain comprehensive documentation for data pipelines, architecture, and processes, making it easier for team members to understand and contribute to the data engineering initiatives.
Qualifications :
• 8+ yrs of overall experience.
