Follow us

Careers @ CIPIO

Back to jobs

Big Data Engineer

Bangalore, Ahmedabad | Full Time

Key Responsibilities

  • Build end-to-end ELT pipelines using tools to enable the training and operationalization of machine learning models
  • Build code for ingesting data from SQL databases, NoSQL database, flat files, and message queues into big data solutions
  • Integrating the code, produced by data scientists, into data pipelines
  • Build code or configuration to push data from big data solutions into reporting tools and other software. Diagnose and mitigate performance issues
  • Communicate with the customer IT personnel to clarify technical details
  • Document the implementation
  • Assist with the installation and setup of big data solutions
  • Assist with DevOps deployments

Required Skills

  • 1-2 years of production experience of building Mapreduce jobs, Spark scripts, Oozie workflows, or other Hadoop based applications
  • Experience with Cloud Technologies
  • Good understanding of distributed data processing
  • Experience in Python
  • Experience in diagnosing and mitigating performance issues in Spark scripts
  • Experience in writing production-ready code
  • Experience with Git
  • Good to have experience in creating streaming solutions and reporting tools