Data engineer Gen AI/ Python
Job Description
Job Description
Role: Data Engineer Gen AI / Python
Location: Whippany NJ (2 daysoffice) - hybrid
Pleasee share me tha profiles at [email protected]
Job Description:
Experienced and skilled in designing, building, and maintaining high-quality data pipelines, preprocessing workflows, and vector databases required for training, fine-tuning, and deploying Large Language Models (LLMs). Build and maintain high-throughput data pipelines, infrastructure, and storage solutions specifically to feed, train, and deploy AI/ML models, implementing RAG (Retrieval-Augmented Generation) systems, data cleaning, and model evaluation to ensure efficient, scalable, and reliable LLM applications.
Required Skills & Qualifications
Strong proficiency in Python is essential, along with SQL and NoSQL for data management.
Experience with LangChain, LlamaIndex, Hugging Face Transformers, and OpenAI API
Experience with Apache Spark, Kafka, or modern data stack tools.
Knowledge of NLP techniques, word embeddings, tokenization, and vector mathematics.
Familiarity with TensorFlow, PyTorch, or Hugging Face
Familiarity with cloud platforms (AWS, GCP, Azure), CI/CD, Docker, and Kubernetes.
Key Responsibilities
Design and build robust ETL/ELT pipelines for unstructured text data, including scraping, cleaning, deduplication, and transformation for LLM training.
Build and maintain vector search solutions (e.g., Pinecone, Milvus, Weaviate, Chroma) to store and retrieve embeddings for RAG systems.
Prepare high-quality datasets for fine-tuning adapters (e.g., LoRA) and train LLMs using frameworks like PyTorch or TensorFlow.
Implement Retrieval-Augmented Generation using frameworks like LangChain or LlamaIndex to connect LLMs to company data.
Develop evaluation frameworks for model performance, testing for accuracy, hallucination, and bias, and monitor deployed models.
Create APIs and internal web tools for data annotation, curation, and model interaction
Recommended Jobs
Staff RN FLEX - Central Resource Office - HPW - FT - Day
Work Shift: Day (United States of America) Capital Health is the region's leader in providing progressive, quality patient care with significant investments in our exceptional physicians, nurses…
Manufacturing Assembler I - Defined Term Cardiac Assist
With a passion for life Join our diverse teams of passionate people and a career that allows you to develop both personally and professionally. At Getinge we exist to make life-saving tech…
MO-11-18-Project Manager 77989
Job Description Job Description ~ This job is hybrid in Newark, NJ Our direct client has an opening for a Project Manager 77989 This position is for 6 months, with the option of extension, and…
Manufacturing Material Handler
Job Description Job Description Material Handler – Manufacturing Flexcraft is looking for a Material Handler to support uninterrupted production by keeping machines supplied, work centers organi…
Nurse Practitioner - 4003
Nurse Practitioner – 4003 Nurse Practitioner Wanted: Deliver Expert Geriatric Care in a Stunning NJ Location *Live in one of the most beautiful locations in the country—just 90 minutes from NYC a…
Security (Floater)
Job Description Job Description Description: Adamas Building Services is a specialty company providing property managers with White Glove Services for ALL your property needs 24/7. With a full c…
Labor & Employment Associate (24 Years) - New Jersey
Job Description Job Description Direct Counsel is representing a reputable law firm seeking a Labor & Employment Associate (2–4 Years) to join its Labor & Employment Practice in New Jersey. …
Direct Support Professional (DSP) Part-time Jackson, NJ
Job Description Job Description Direct Support Professional (DSP) – Part-Time Location: Jackson, NJ Pay: $18.76 – $19.76 wknd/hr Shift Options: Friday & Saturday: 3:00 PM - 11:00PM…
Social Worker, MSW - Toms River, NJ
Job Title: Social Worker MSW Location: Central Jersey Behavioral Department Name: Social Work Req #: 0000247019 Status: Hourly Shift: Day Pay Range: $45.50 - $45.50 per hour Pay T…
Driver - Local (Secaucus)
Job Description Job Description Company Overview: NRS is a leading provider of transportation & supply chain solutions. As a family-owned and operated company, NRS has delivered smart logist…