Big Data Engineer Job Description

We are searching for a professional prominent data engineer to help us create and manage our company's Big Data offerings. You will be accountable for designing and executing Big Data mechanisms and frameworks, enforcing ELT procedures, cooperating with development teams, creating cloud platforms, and supporting the production system in this job.

To be a successful, prominent data engineer, you need to understand Hadoop technology, outstanding project management skills, and strong problem-solving abilities. A top-tier Big Data Engineer knows the company's goals and implements scalable data solutions to meet those demands now and in the future.

Prominent data engineers interface with vast data processing systems and databases in large-scale computer settings. They filter through enormous amounts of data to identify relevant sets for analysis, which corporations then use to forecast behaviour.

Distinguished data engineers conduct studies that assist firms in assessing their performance, identifying market demographics, and forecasting impending changes and market trends. Well-known data engineers are needed in almost every industry because they generate insights that are helpful in business, finance, government, healthcare, research, telecommunications, and other industries.

• C Suite
• Data Team

• Collect and analyse raw data on a large scale
• Create data apps using a variety of tools and frameworks
• Read, retrieve, reshape, stage, and load the data to chosen frameworks and tools
• Make scripts, scrape websites, call APIs, write SQL queries, etc.
• Collaborate closely with the technical team to incorporate your work into our manufacturing processes
• Convert unstructured data into an analysis-ready format
• Examine the processed data
• Help business decisions with ad hoc assessment as required
• Monitor data performance and make necessary changes to infrastructure
• Establish data retention policies
• Develop and maintain integration tools, databases, warehouses, and analytical systems
• Establish data access tools for data scientists

• Thorough knowledge of distributed computing principles
• Hadoop cluster management, including all bundled services
• Ability to troubleshoot any current cluster issues - Knowledge of Hadoop v2, MapReduce, and HDFS
• Previous experience developing stream-processing systems, such as Storm or Spark
• Thorough understanding of Big Data querying technologies such as Pig, Hive, and Impala
• Previous Spark experience
• Previous experience integrating data from several data sources
• Knowledge of NoSQL databases such as HBase, Cassandra, and MongoDB
• Understanding of several ETL methodologies and frameworks, such as Flume
• Experience with different messaging systems, such as Kafka or RabbitMQ - Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O - Solid understanding of the Lambda Architecture, including its benefits and downsides
• Knowledge of Cloudera/MapR/Hortonworks