Lead, Data Platform Engineer

Position Overview: 

Data Analytics & AI department at RCG is looking for a highly skilled and experienced Data Platform Lead to design, build, and manage our data platform on Azure and Databricks. The successful candidate will be responsible for managing the end-to-end data platform development process, including data ingestion, processing, storage, and analysis. This role requires a strong technical background in data platform architecture, cloud computing, and big data technologies.  

This role carries primary responsibilities as a Platform Lead / Platform Engineer and Azure Platform Admin, while also contributing to ML Ops. 

Responsibilities: 

• Design, build and configure a highly scalable and robust data platform on Azure and Databricks.  

• Data platform shuld have capabilities of seamless integration & extensibility of existing cloud services to any new program/project initiative.  

• Develop and implement data ingestion and processing pipelines using Azure services such as Azure Data Factory, Azure Stream Analytics, and Databricks 

• Collaborate with cross-functional teams to ensure data platform meets business requirements and industry best practices 

• Manage cloud infrastructure on Azure platform and maintain Azure environment to ensure maxium efficiency, scalability and security.  

• Develop & implement cloud automation and orchestration using tools auch as Azure resource manager, Azure DevOps etc. 

• Collaborate with offshore team members with development & deployment activities, per project requirements.  

Required Skills: 

Platform Engineering: 

• Proficiency in Python, Scala, or SQL. 

• Expertise in Apache Spark, Hadoop, or Apache Kafka. 

• Cloud experience in Azure (preferred) or AWS. 

• Deployment and management of Databricks clusters. 

• Mastery of Azure Data Factory (ADF) for integration. 

• Expertise with Databricks workspace, runtime, and Databricks Connect. 

• Understanding of Databricks Unity Catalog. 

• Knowledge of Delta Live Tables and Delta Engine. 

• Familiarity with Delta Lake architecture. 

• Strong SQL skills. 

Azure Platform Administration: 

• Management of Azure Cloud Services, including VMs, Storage Accounts, and networks. 

• Azure Security Management for policies and access controls. 

• Knowledge of Azure Active Directory and network security. 

• Performance monitoring, resource optimization, backup, and recovery. 

• High Availability (HA), Disaster Recovery (DR), and failover expertise. 

• Azure Services/Resources cost optimization and monitoring. 

• Azure DevOps and CI/CD proficiency. 

• Configure Azure resources, native services, security, and monitoring. 

• Design platform architecture with Azure Cloud Services and Databricks Delta Lake. 

• Create and publish guidelines for Data Integration services. 

• Participate in platform solution architecture, including data architecture. 

• Implement Databricks workspace, runtime, and Databricks Connect. 

• Configure Kafka integration services for real-time streaming and batch processing. 

• Enable Databricks Unity Catalog (UC) for lineage and governance. 

• Design and implement ML Ops for model deployments and inference pipelines. 

• Configure Azure DevOps, CI/CD for seamless deployment. 

• Implement monitoring and logging for Azure Service Groups and system metrics. 

• Ensure platform security, compliance, and data privacy. 

ML Ops: 

• Proficient in Python, R, or Java for ML algorithms. 

• Experience with TensorFlow, PyTorch, and other ML frameworks. 

• Efficient code for data processing, model training, and deployment. 

• Cloud platform experience, preferably Azure. 

• Familiarity with Azure ML Services and Azure DevOps. 

• Strong data analytics skills for model training. 

Qualifications: 

• Bachelor's degree in Computer Science, Information Technology, or related field. 

• 7+ years of experience in cloud infrastructure management, Azure services, and infrastructure architecture 

• Strong understanding of Azure services such as Virtual Machines, Virtual Networks, Storage Accounts, and App Services 

• Experience in monitoring and optimizing Azure infrastructure for performance and efficiency 

• Proven experience and expertise in the respective role's technical skillsets. 

• Strong problem-solving skills and ability to work collaboratively. 

• Excellent communication skills for effective cross-team collaboration. 

If you're passionate about leveraging data for informed decisions and thrive in a dynamic, collaborative environment, we invite you to join our team. 

#LI-DW1