Principal AI Engineer – Agentic AI / LLM

Job Title: Principal AI Engineer – Agentic AI / LLM

Location: New York City, NY

Work Arrangement: Hybrid – 3 days onsite per week

Employment Type: Full-Time

Work Hours: EST overlap required

Experience: 8–14 years of software engineering experience; 10+ years preferred

Visa Sponsorship: H-1B transfer candidates can be considered

About the Role

We are seeking a highly experienced Principal AI Engineer to design, build, and productionize enterprise-scale Agentic AI, Generative AI, and LLM-powered systems.

This is a highly hands-on engineering role for an experienced technical leader who can operate across AI architecture, software engineering, RAG, LLM applications, agent orchestration, data pipelines, evaluation, observability, and cloud infrastructure.

You will own AI systems end-to-end—from architecture and experimentation through production deployment, optimization, reliability, and continuous improvement.

The ideal candidate has demonstrated experience building production-grade AI agents that can reason over enterprise context, retrieve information, invoke tools and APIs, and securely interact with real-world business systems.

Key Responsibilities

• Architect and build production-grade Agentic AI systems using LLMs, RAG, tool calling, and agent orchestration.
• Design autonomous AI agents capable of reasoning, retrieval, planning, and secure execution of enterprise actions.
• Own AI systems end-to-end, including:
  • Data pipelines
  • Backend services
  • Agent orchestration
  • Evaluation frameworks
  • Observability
  • Cloud infrastructure
• Design and implement production RAG pipelines covering:
  • Document ingestion
  • Chunking
  • Embeddings
  • Retrieval
  • Hybrid search
  • Reranking
  • Context construction
  • Response validation
• Build and integrate MCP (Model Context Protocol) servers and secure tool-calling interfaces.
• Develop reliable agent workflows using frameworks such as LangGraph, Google ADK, CrewAI, Claude Agent SDK, or equivalent technologies.
• Implement AI reliability mechanisms including:
  • Guardrails
  • Validators
  • Policy checks
  • Deterministic fallbacks
  • Self-correction loops
  • Circuit breakers
  • Rollback strategies
• Build evaluation frameworks for LLM and agentic systems, including offline evaluation, regression testing, and continuous evaluation.
• Implement AI observability and tracing using tools such as LangSmith, Langfuse, or equivalent platforms.
• Optimize AI applications for:
  • Latency
  • Cost
  • Token efficiency
  • Context-window utilization
  • Throughput
  • Scalability
• Design and operate scalable backend services, APIs, microservices, and distributed systems.
• Develop and maintain high-quality Python and SQL code.
• Deploy AI applications across AWS, Azure, or GCP.
• Work with containerized and cloud infrastructure such as EKS/ECS, Kubernetes, Lambda, and related services.
• Implement CI/CD and infrastructure-as-code using Terraform, CloudFormation, or equivalent technologies.
• Partner with product, engineering, data science, and business stakeholders to translate ambiguous business requirements into scalable technical solutions.
• Provide technical leadership through architecture/design reviews, engineering standards, code reviews, and mentoring.

Required Qualifications

Software Engineering

• 8–14 years of professional software engineering experience; 10+ years preferred.
• Strong, hands-on Python development experience.
• Strong understanding of data structures and algorithms.
• Experience with APIs, microservices, system design, and distributed systems.
• Experience building and operating production-grade data pipelines.
• Strong understanding of scalable software architecture.
• Experience with at least one additional backend/systems language such as Go, Rust, Java, C, or C++ is highly desirable.

LLM / Generative AI

• 2+ years of hands-on LLM/Generative AI engineering experience.
• Experience building and deploying production LLM applications.
• Demonstrated experience designing and taking multiple Agentic AI systems into production.
• Strong understanding of LLM architecture, prompting, context management, and model behavior.

Agentic AI

Hands-on experience with one or more:

• LangGraph
• Google ADK
• CrewAI
• Claude Agent SDK
• Equivalent production-grade agent frameworks

Experience with:

• Agent orchestration
• State management
• Planning/reasoning workflows
• Tool execution
• Multi-agent workflows
• Human-in-the-loop patterns
• Agent reliability and failure handling

MCP & Tool Calling

• Strong hands-on experience with Model Context Protocol (MCP).
• Experience building MCP servers.
• Experience developing secure tool interfaces.
• Strong understanding of function/tool calling and API integrations.

RAG & Vector Search

Strong understanding of RAG from first principles, including:

• Document ingestion
• Chunking strategies
• Embeddings
• Vector search
• Hybrid retrieval
• Reranking
• Retrieval optimization
• Context construction
• Response validation

Experience with vector databases such as:

• Pinecone
• Milvus
• Weaviate
• FAISS
• Other vector database/cloud equivalents

Evaluation & Observability

• Experience building LLM evaluation frameworks.
• Offline evaluation datasets and automated evaluation.
• Continuous/online evaluation.
• Regression detection.
• LLM observability and tracing.
• Experience with LangSmith, Langfuse, or equivalent.
• Experience implementing AI guardrails and reliability mechanisms.

Cloud & DevOps

Strong AWS experience preferred, including:

• EKS / ECS
• Lambda
• S3
• DynamoDB
• Redshift
• Step Functions
• Amazon Bedrock

Also valued:

• Azure / GCP
• Terraform
• CloudFormation
• Docker
• Kubernetes
• CI/CD

AI Performance Optimization

Experience optimizing LLM systems for:

• Token consumption
• Context-window management
• Context pruning and compaction
• Model routing
• Caching
• Batching
• Streaming
• Parallel tool calls
• Latency
• Cost
• Throughput

Preferred Qualifications

• Experience with Amazon Bedrock, Azure AI, or Google Vertex AI.
• Experience working with enterprise or regulated environments.
• Experience building AI platforms used at scale.
• Strong client-facing communication skills.
• Ability to translate ambiguous business problems into technical architecture.
• Demonstrated technical leadership and mentoring experience.

Ideal Candidate Profile

Staff/Principal AI Engineer | Agentic AI | LLM | Python | RAG | MCP | AWS

We are particularly interested in candidates who have:

• Built and deployed multiple production AI agents.
• Built MCP servers and secure tool interfaces.
• Hands-on experience with LangGraph, Claude Agent SDK, Google ADK, CrewAI, or equivalent.
• Strong production RAG architecture experience.
• Experience with tool orchestration and state management.
• Built LLM evaluation frameworks.
• Experience with LangSmith/Langfuse or equivalent observability.
• Implemented AI guardrails and reliability mechanisms.
• Experience optimizing token usage, context, latency, and cost.
• Strong AWS/Kubernetes experience.
• Strong distributed systems and Python engineering experience.
• Enterprise/client-facing technical leadership experience.

Important: We are looking for candidates with genuine production ownership, rather than experience limited to prompt engineering, basic chatbot development, research, or LLM API integrations.