Agentic AI Databricks

Job Title: Agentic AI Data bricks

Location: Raritan, NJ

Duration: 12 Months + Extension

Bill Rate: $97/hr

Job Type: W2 Contract

Client: To Be Discussed Later

Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD

Job description:

Experience 10 years

Location Flexible

Function: Data Engineering AIML Engineering Cloud Data Platforms

Role Summary: We are looking for an experienced Databricks Azure professional with strong expertise in cloud scale data engineering lakehouse architecture Agentic AI and Data bricks Agent Bricks The ideal candidate will have 7 or more years of experience designing building and operationalizing enterprise data and AI platforms on Azure Databricks with hands-on exposure to generative AI solutions retrieval-augmented generation vector search AI agents model serving MLflow Unity Catalog and production-grade governance practices.

Roles Responsibilities:

• Design develop and maintain scalable data engineering and AIML solutions on Azure Databricks using Delta Lake Unity Catalog Databricks Workflows Databricks SQL and MLflow
• Build and deploy Agentic AI applications using Databricks Agent Bricks Mosaic AI Agent Framework Knowledge Assistants Supervisor Agents toolcalling agents and multiagent orchestration patterns
• Develop retrieval-augmented generation solutions using Databricks Vector Search embeddings model serving endpoints foundation models and governed enterprise knowledge sources
• Architect lakehouse solutions across ingestion transformation orchestration governance security performance optimization and monitoring layers
• Integrate Azure-native services such as Azure Data Lake Storage Azure Data Factory Azure Key Vault Azure DevOps Microsoft Entra ID Azure Functions and Azure OpenAI where applicable
• Implement robust CICD pipelines for Databricks notebooks jobs bundles ML models and AI agent deployments using Azure DevOps or equivalent tools
• Establish data and AI governance standards using Unity Catalog access controls lineage audit logging model registry and secure serving patterns
• Optimize Spark workloads Delta tables SQL warehouses serverless compute cluster policies job performance and cost utilization
• Collaborate with data engineers ML engineers architects product teams and business stakeholders to convert business requirements into reliable data and AI capabilities
• Define best practices reusable frameworks coding standards deployment patterns and operational runbooks for Databricks and Agentic AI platforms
• Support production operations incident resolution root cause analysis monitoring SLA management and continuous improvement for data and AI workloads
• Mentor junior engineers and contribute to knowledge sharing technical design reviews proof-of-concepts and platform modernization initiatives

Must Have Skills:

• 7 years of overall experience in data engineering analytics engineering AIML engineering or cloud data platform engineering
• Strong hands-on experience with Azure Databricks Apache Spark PySpark Spark SQL Delta Lake Delta Live Tables Databricks Workflows Databricks SQL and Lakehouse architecture
• Practical experience with Databricks Agent Bricks Agentic AI patterns AI agents tool calling multiagent systems Knowledge Assistants Supervisor Agents and governed AI application development
• Hands-on experience building generative AI solutions such as RAG applications LLM chains prompt engineering workflows vector search implementations embeddings pipelines and model serving solutions
• Strong understanding of MLflow for tracking model lifecycle management tracing evaluation monitoring and deployment of ML and GenAI workloads
• Experience with Unity Catalog data governance access control catalog schema table permissions lineage auditability and secure AI data sharing patterns
• Proficiency in Python SQL PySpark data modeling performance tuning API integration and production-grade software engineering practices
• Experience with Azure services including ADLS Gen2 Azure Data Factory Azure DevOps Azure Key Vault Microsoft Entra ID Azure Monitor and Azure OpenAI or comparable LLM services
• Strong knowledge of CICD infrastructure-as-code concepts Git-based development Databricks Asset Bundles environment promotion and release management
• Ability to troubleshoot complex data pipelines Spark performance issues job failures security constraints and AI model or agent behavior in production environments
• Good understanding of Responsible AI data privacy model evaluation hallucination mitigation guardrails prompt safety and enterprise AI governance
• Excellent communication stakeholder management documentation and technical leadership skills

Good to have Skills:

• Experience with LangChain LangGraph OpenAI Agents SDK LlamaIndex MCP servers AI Gateway or custom agent frameworks
• Exposure to Databricks Apps Genie Foundation Model APIs Model Context Protocol integrations and enterprise chatbot or copilot implementations
• Experience with streaming data pipelines using Structured Streaming Event Hub

Mandatory Skills: Azure Data Factory, Databricks

Equal opportunity employer: We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.