Apply now »

Lead, Data Platform Engineer

Position Overview: 


Data Analytics & AI department at RCG is looking for a highly skilled and experienced Data Platform Lead to design, build, and manage our data platform on Azure and Databricks. The successful candidate will be responsible for managing the end-to-end data platform development process, including data ingestion, processing, storage, and analysis. This role requires a strong technical background in data platform architecture, cloud computing, and big data technologies.  

This role carries primary responsibilities as a Platform Lead / Platform Engineer and Azure Platform Admin, while also contributing to ML Ops. 




  • Design, build and configure a highly scalable and robust data platform on Azure and Databricks.  

  • Data platform shuld have capabilities of seamless integration & extensibility of existing cloud services to any new program/project initiative.  

  • Develop and implement data ingestion and processing pipelines using Azure services such as Azure Data Factory, Azure Stream Analytics, and Databricks 

  • Collaborate with cross-functional teams to ensure data platform meets business requirements and industry best practices 

  • Manage cloud infrastructure on Azure platform and maintain Azure environment to ensure maxium efficiency, scalability and security.  

  • Develop & implement cloud automation and orchestration using tools auch as Azure resource manager, Azure DevOps etc. 

  • Collaborate with offshore team members with development & deployment activities, per project requirements.  


Required Skills: 


Platform Engineering: 


  • Proficiency in Python, Scala, or SQL. 

  • Expertise in Apache Spark, Hadoop, or Apache Kafka. 

  • Cloud experience in Azure (preferred) or AWS. 

  • Deployment and management of Databricks clusters. 

  • Mastery of Azure Data Factory (ADF) for integration. 

  • Expertise with Databricks workspace, runtime, and Databricks Connect. 

  • Understanding of Databricks Unity Catalog. 

  • Knowledge of Delta Live Tables and Delta Engine. 

  • Familiarity with Delta Lake architecture. 

  • Strong SQL skills. 


Azure Platform Administration: 


  • Management of Azure Cloud Services, including VMs, Storage Accounts, and networks. 

  • Azure Security Management for policies and access controls. 

  • Knowledge of Azure Active Directory and network security. 

  • Performance monitoring, resource optimization, backup, and recovery. 

  • High Availability (HA), Disaster Recovery (DR), and failover expertise. 

  • Azure Services/Resources cost optimization and monitoring. 

  • Azure DevOps and CI/CD proficiency. 

  • Configure Azure resources, native services, security, and monitoring. 

  • Design platform architecture with Azure Cloud Services and Databricks Delta Lake. 

  • Create and publish guidelines for Data Integration services. 

  • Participate in platform solution architecture, including data architecture. 

  • Implement Databricks workspace, runtime, and Databricks Connect. 

  • Configure Kafka integration services for real-time streaming and batch processing. 

  • Enable Databricks Unity Catalog (UC) for lineage and governance. 

  • Design and implement ML Ops for model deployments and inference pipelines. 

  • Configure Azure DevOps, CI/CD for seamless deployment. 

  • Implement monitoring and logging for Azure Service Groups and system metrics. 

  • Ensure platform security, compliance, and data privacy. 


ML Ops: 


  • Proficient in Python, R, or Java for ML algorithms. 

  • Experience with TensorFlow, PyTorch, and other ML frameworks. 

  • Efficient code for data processing, model training, and deployment. 

  • Cloud platform experience, preferably Azure. 

  • Familiarity with Azure ML Services and Azure DevOps. 

  • Strong data analytics skills for model training. 




  • Bachelor's degree in Computer Science, Information Technology, or related field. 

  • 7+ years of experience in cloud infrastructure management, Azure services, and infrastructure architecture 

  • Strong understanding of Azure services such as Virtual Machines, Virtual Networks, Storage Accounts, and App Services 

  • Experience in monitoring and optimizing Azure infrastructure for performance and efficiency 

  • Proven experience and expertise in the respective role's technical skillsets. 

  • Strong problem-solving skills and ability to work collaboratively. 

  • Excellent communication skills for effective cross-team collaboration. 


If you're passionate about leveraging data for informed decisions and thrive in a dynamic, collaborative environment, we invite you to join our team. 


Nearest Major Market: Miami

Apply now »