Job Description

Locations

• India - Hyderabad

Skills Required

• Bachelor’s degree in Computer Scienceintermediate
• Python for data processingintermediate
• PySparkintermediate
• Databricks for data engineeringintermediate
• SQL for queryingintermediate
• cloud-based data platformsintermediate
• workflow orchestration toolsintermediate
• CI/CD practices for data engineering projectsintermediate
• Tableauintermediate

Required Qualifications

• 2-6 years of experience with Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Engineering, Mathematics, or a related field, or equivalent practical experience (experience, 6 years)
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Engineering, Mathematics, or a related field, or equivalent practical experience. (experience)
• Hands-on experience with Python for data processing, scripting, and automation. (experience)
• Strong working knowledge of PySpark and distributed data processing concepts. (experience)
• Proven hands-on experience using Databricks for data engineering, including notebooks, clusters, jobs, workflows, Delta tables, and performance optimization. (experience)
• Ability to build, maintain, and troubleshoot scalable ETL/ELT pipelines in Databricks. (experience)
• Experience working with Delta Lake and lakehouse architecture concepts. (experience)
• Working knowledge of SQL for querying, transforming, and validating data. (experience)
• Ability to work with structured and semi-structured data formats such as CSV, JSON, Parquet, and Delta. (experience)
• Understanding of data engineering concepts such as ETL/ELT, data pipelines, data lakes, data warehouses, batch processing, and data quality. (experience)
• Basic understanding of AI and machine learning concepts, including features, training datasets, model inputs/outputs, and model evaluation basics. (experience)
• Experience supporting data preparation or feature engineering for AI/ML use cases. (experience)
• Familiarity with cloud-based data platforms, preferably AWS, Azure, or GCP. (experience)
• Understanding of Git or other version control tools. (experience)
• Strong analytical, problem-solving, and troubleshooting skills. (experience)
• Good communication skills and ability to work collaboratively with technical and non-technical stakeholders. (experience)
• Willingness to learn new tools, technologies, and data engineering best practices. (experience)

Preferred Qualifications

• Exposure to Delta Lake, Unity Catalog, or Lakehouse architecture. (experience)
• Experience with workflow orchestration tools or Databricks Jobs. (experience)
• Familiarity with CI/CD practices for data engineering projects. (experience)
• Exposure to machine learning workflows using MLflow, scikit-learn, or similar tools. (experience)
• Experience with Tableau, Power BI, or similar data visualization tools to create dashboards, support reporting needs, validate datasets, and perform exploratory analysis. (experience)
• Understanding of data governance, security, and access control concepts. (experience)
• Experience working in an Agile/Scrum environment. (experience)

Responsibilities

• Develop, test, and maintain data pipelines using Databricks, PySpark, and Python.
• Ingest, transform, and process structured and semi-structured data from multiple sources.
• Support the development of scalable ETL/ELT workflows for analytics, reporting, and machine learning use cases.
• Work with data engineers, analysts, and data scientists to understand data requirements and deliver reliable datasets.
• Perform data cleansing, validation, and quality checks to ensure accuracy and consistency.
• Optimize Spark jobs and Databricks notebooks for performance, reliability, and cost efficiency.
• Create and maintain documentation for data pipelines, workflows, data definitions, and processes.
• Assist in troubleshooting pipeline failures, data issues, and performance bottlenecks.
• Follow best practices for version control, code quality, testing, and deployment.
• Support basic AI/ML data preparation activities, including feature engineering, dataset creation, and model input preparation.
• Monitor scheduled jobs and workflows to ensure timely and successful data delivery.
• Collaborate with cross-functional teams in an Agile or iterative development environment.

Tags & Categories