Senior Cloud Engineer - Azure Infrastructure & Databricks Platform

Ampcus Inc. is a certified global provider of a broad range of Technology and Business consulting services. We are in search of a highly motivated candidate to join our talented Team.

Job Title: Senior Cloud Engineer - Azure Infrastructure & Databricks Platform

Location(s): Warren, MI
(Remote)

Position Summary
We are seeking a Senior Cloud Engineer to design, implement, and administer the Azure infrastructure supporting an enterprise data and AI platform. This is a cloud infrastructure and platform engineering role focused on Azure Databricks, ADLS Gen2, Azure Data Factory, and Azure Machine Learning.

The Senior Cloud Engineer will own the platform foundation, including infrastructure as code, network and identity architecture, compute governance, observability, reliability, and cloud cost management. The role will work closely with Databricks administration, data engineering, and data science teams to establish platform standards, reference architectures, and secure self-service capabilities.

The ideal candidate combines deep Azure infrastructure expertise with strong hands-on Databricks administration experience and can serve as a senior technical authority for complex platform and architecture decisions.

Key Responsibilities
Azure Infrastructure & Platform Engineering

• Design, implement, and administer the Azure foundation supporting the enterprise data and AI platform.
• Develop and maintain reusable Terraform modules for Azure Databricks, ADLS Gen2, Azure Data Factory, Key Vault, networking, and Azure Machine Learning.
• Manage Terraform remote state, module versioning, drift detection, and CI/CD deployment pipelines.
• Establish and maintain development, test, and production environment promotion processes.
• Design secure Azure networking architectures, including private endpoints, VNets, hub-and-spoke topology, VNet injection, NSGs, firewall rules, and data exfiltration controls.
• Implement Entra ID, managed identities, RBAC, Key Vault, and secure identity/access patterns.
• Define enterprise ADLS Gen2 architecture, including storage accounts, hierarchical namespaces, containers, ACLs, lifecycle policies, tiering, encryption, and key management.
• Design access patterns between ADLS Gen2, Databricks, and downstream consumers.

• Own Azure Data Factory from the platform infrastructure perspective.
• Provision and manage managed and self-hosted integration runtimes.
• Configure managed VNet and secure connectivity.
• Design linked-service credentials and managed identity integrations.
• Support environment promotion, deployment automation, monitoring, and platform reliability.
• Partner with data engineering teams; production ETL/ELT pipeline development remains outside this role.

• Provision and operate Azure Machine Learning workspaces and compute clusters.
• Support GPU quota management, capacity planning, and workload infrastructure.
• Support MLflow model registry integration and model-serving endpoints.
• Establish secure identity and networking between Azure ML and Unity Catalog.
• Ensure AI/ML infrastructure follows enterprise security, governance, and operational standards.

• Build and maintain monitoring and observability across the Azure platform.
• Configure Azure Monitor, Log Analytics, diagnostic settings, dashboards, and alerting.
• Develop operational runbooks and support reliability engineering practices.
• Troubleshoot complex infrastructure, connectivity, authentication, and platform issues.

• Establish cloud cost visibility and optimization practices across Databricks DBU and Azure storage consumption.
• Implement tagging, chargeback/showback, budget alerts, and cost allocation strategies.
• Evaluate reserved capacity and other optimization opportunities.
• Perform capacity planning and forecasting for platform growth.
• Analyze Databricks usage and identify opportunities to reduce unnecessary consumption.

• Configure Databricks account-level settings and workspace environments.
• Define workspace provisioning standards and topology strategies.
• Establish workspace-level configuration and administrative delegation.
• Contribute to multi-workspace architecture and governance.

• Design and govern Unity Catalog metastores and regional strategies.
• Establish catalog, schema, table, and permission models.
• Configure storage credentials and external locations.
• Support Hive Metastore migrations.
• Implement lineage, auditing, and Delta Sharing configurations.

• Integrate Databricks with Microsoft Entra ID.
• Implement SCIM provisioning, identity federation, groups, and entitlements.
• Manage service principals and token policies.
• Establish secure authentication and authorization patterns.

• Develop and enforce Databricks cluster policies.
• Establish instance pool, node type, and runtime standards.
• Configure autoscaling and autotermination policies.
• Evaluate Photon and serverless compute options.
• Configure and size SQL warehouses based on workload requirements.

• Use Databricks system tables to analyze platform consumption.
• Establish usage attribution and DBU forecasting.
• Identify major sources of Databricks spend.
• Recommend and implement remediation strategies for inefficient resource utilization.

• Review Databricks platform configurations and architecture.
• Establish standards, reference configurations, and self-service patterns.
• Partner with administration, data engineering, and data science teams on complex technical issues.
• Troubleshoot job failures, cluster startup problems, permission issues, connectivity faults, and performance concerns.
• Determine whether issues originate within Azure infrastructure, Databricks configuration, or application/workload behavior.

• Production ETL/ELT pipeline development and Spark transformation work - Data Engineering.
• Model development, training, and tuning - Data Science.
• Dimensional modeling, dbt development, and BI/semantic layer development - Other specialized teams.
• Day-to-day Databricks administration - Dedicated Databricks Administration team.

The Senior Cloud Engineer provides the platform architecture, standards, governance, and senior technical partnership supporting these teams.

Required Qualifications

• 6+ years of experience in cloud infrastructure or platform engineering.
• At least 4 years of hands-on Microsoft Azure experience.
• Expert-level Terraform experience, including production module development, remote state management, infrastructure lifecycle management, and CI/CD integration.
• Demonstrable hands-on Databricks administration experience.
• Strong experience with Databricks account and workspace administration, Unity Catalog, cluster policies, identity federation, and cost governance.
• Strong Azure networking and security knowledge, including:
  • Private endpoints
  • VNets
  • Hub-and-spoke architecture
  • NSGs
  • Microsoft Entra ID
  • Managed identities
  • RBAC
  • Azure Key Vault
• Enterprise-scale ADLS Gen2 architecture and access-control experience.
• Azure Data Factory platform experience, including integration runtimes, managed VNet, credential management, and deployment automation.
• Strong Python scripting and automation skills.
• PowerShell and/or Bash scripting experience.
• Working knowledge of Spark and SQL sufficient to diagnose infrastructure and configuration-level performance issues.
• Demonstrated ability to act as a senior technical resource and partner with adjacent engineering teams.
• Strong communication, architecture, troubleshooting, and stakeholder-management skills.

• Experience with Databricks Asset Bundles, Terraform Databricks Provider, and workspace-as-code.
• Experience leading Unity Catalog migrations or multi-workspace consolidation.
• Azure Machine Learning, MLflow, or model-serving infrastructure experience.
• Kubernetes / AKS and containerized workload experience.
• Policy-as-code experience using Azure Policy, OPA, Sentinel, Checkov, or similar technologies.
• Experience with Event Hubs, Kafka, or Stream Analytics.
• Formal FinOps experience.
• Experience designing multi-region or multi-tenant Databricks environments.

• Databricks Certified Data Engineer Professional or Databricks platform administrator certification.
• Microsoft Certified: Azure Solutions Architect Expert (AZ-305).
• Microsoft Certified: Azure Administrator Associate (AZ-104).
• Microsoft Certified: DevOps Engineer Expert (AZ-400).
• HashiCorp Certified: Terraform Associate.

Ampcus is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veterans or individuals with disabilities.