Gen AI Data Engineer

*Responsibilities:* * Design, develop, and evaluate innovative LLM models to solve diverse challenges and opportunities across multiple use-cases using MS Azure services. * Understand business problems, and help with defining and implementing a scalable Generative AI platform to solve them. * Build large-scale production systems utilizing advanced machine learning and big data technologies. * Develop and optimize data processing workflows for chunking, indexing, ingestion, and vectorization for both text and non-text data * Benchmark and implement various vector stores, embedding techniques, and retrieval methods * Create a flexible pipeline supporting multiple embedding algorithms, vector stores, search types (vector search, hybrid search, etc.) and search ranking * Implement and maintain auto-tagging systems and data preparation processes for LLMs * Develop tools and pipelines for text and image data cleaning, and refinement * Integrate and optimize workflows using Azure AI Search, Azure AI Services, Azure PostgreSQL and various vector store technologies * Use advanced prompting techniques to develop and optimize prompts for LLM to ensure model performance, responses quality and accuracy of the Gen AI platform. * Contribute to automated model testing and evaluation infrastructure * Build automations to create and select the best prompts for any given task Job Type: Contract Pay: Up to $66.00 per hour Expected hours: 40 per week Schedule: * 8 hour shift Experience: * Azure and Azure AI infrastructure: 2 years (Preferred) * Autogen 4 and 6: 2 years (Preferred) * Semantic Kernel: 2 years (Preferred) * Millions of documents handling: 3 years (Preferred) * Agentic Tools: 1 year (Preferred) Work Location: In person