Big Data Engineer
- Requirement analysis, requirement gathering, design, business impact analysis, gap analysis, estimation, development, testing, coding, code review, unit testing and deployment of the application.
- Design, build, install, configure, and support Hadoop (Big Data).
- Design and implement big data pipelines to ingest and process data in real-time and monitor for data miss or error in progress to ensure that data reaches the end system.
- Measure the performance of various APIs and implement optimization on slow services to enhance the responsiveness of the system.
- Deliver functional enhancements to the existing big data applications.
- Provide inputs on solution architecture based on evaluation/understanding of solution alternatives, frameworks, and products.
- Develop Hadoop batch jobs for data extraction from multiple unstructured and structured data sources for populating into various repositories (Hadoop, Neo4J, MongoDB, Apache SOLR).
- Interact with clients to elicit architectural and non-functional requirements such as performance, scalability, reliability, availability, maintainability.
- Participate in designing the data model structured data in RDBMS into a graph database solution and NoSQL database solution.
- Develop near real-time data processing solutions using Kafka and Spark Streaming.
- Participate in designing Spark architecture with Databricks and Structured Streaming.
- Set up Microsoft Azure with Databricks, and Databricks Workspace for business analytics.
- Contribute to the automate build process by using Jenkins and Ansible to achieve continuous integration and continuous deployment.
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Recommended Jobs
Controls Engineer - Building Automation Systems
Job Details: BAS Controls Engineer | Building Automation Systems Controls – BAS – HVAC – Building Automation Our client is seeking a Building Automation Systems (BAS) Controls Engineer to jo…
J.P. Morgan Wealth Management - Private Client Investment Associate - Freehold, NJ
Job Description At J.P. Morgan Chase, we have an enthusiasm for helping our clients, taking care of our employees, building relationships, and delivering extraordinary customer service while mainta…
Part-time Dentist needed in Jackson Township, NJ
Part-Time Associate Dentist – Jackson, NJ Established Multi-Specialty Private Practice with Loyal Patient Base An established dental practice in Jackson, NJ is seeking a personable and highly s…
OCCUPATIONAL THERAPIST - MILLVILLE AND GEORGETOWN FLOAT
Why Beebe?: Become part of the Beebe team - an inclusive team positioned in a vibrant, coastal community. Enjoy a fulfilling career as you support the health of our patients and a team focused on exc…
Elementary Principal PreK - 6th Grade
St. Wenceslaus Catholic School in Wahoo, Nebraska is seeking a dynamic, faith-driven Principal to lead its active and energetic family community starting July 1, 2026. St. Wenceslaus is a school of t…
Principal Solution Engineer (Integration)
Share Our Purpose. Be Yourself. Feel Valued. People are the heart of our business. As an American Water employee, you will be offered a competitive salary and health benefits package, along with oppo…
Associate Veterinarian
Associate Veterinarian – Warren County, NJ #6467 We are seeking an Associate Veterinarian who is interested in practicing the highest level of medicine and surgery. All levels of experience are w…
Restaurant Manager
$60000 per year - $72000 per year Our Winning Family Starts With You! Check out these great benefits: Commitment to Quality of Life ∙ Restaurant Managers regularly work no more than …
Board Certified Veterinarian Surgeon
Board Certified Veterinarian Surgeon – Sussex County, NJ #6470 We are seeking a Board Certified Surgeon to join an exceptional team. This is a full-service, small animal emergency and general pra…
Lead Screen Printer
We're hiring for a Lead Screen Printer to independently operate and optimize servo-driven screen printing presses used to decorate glass bottles with ceramic and UV inks. This is a hands-on, senior-…