Data Engineer Test

Chusk works as Hadoop developer at Pesla Inc. Chusk is tasked with processing input data to count number of occurrences of each unique word. Chusk did the following to achieve this:

1. Tokenize each word and emit lateral value 1 with Mapper
2. Reducer increments counter for each literal 1 it receives

Chusk is now tasked with optimizing this by using a combiner. Will Chusk be able to reuse existing reducers as combiners?

A: Yes
B: No
C: Because the sum operation is both associative and commutative and the input and output types to the reduce method match
D: Because the sum operation in the Reducer is incompatible with the operation of a combiner
E: Because the combiner is incompatible with a Mapper, which doesn't use the same data type for both the key and value
F: Insufficient information

Assuming the following Hive statements execute successfully, choose the correct statements that describe the result:

from fooddata select context_ngrams(sentences(lines),
array("twiggy", "romato", null), 68);

A. A bigram of the top 68 sentences that contain the substring "twiggy romato" in the lines column of the input data A1 table.
B. An 68-value ngram of sentences that contain the words "twiggy" or "romato" in the lines column of the fooddata table.
C. A trigram of the top 68 sentences that contain "twiggy romato" followed by a null space in the lines column of the fooddata table.
D. A frequency distribution of the top 68 words that follow the subsequence "twiggy romato" in the lines column of the fooddata table.

Consider the following two relations, P and Q:

What is the output of the following Pig command?

Q = GROUP P BY p2;
DUMP Q;

Penny created a jar file for her character count example written in Java. The jar name is attempt.jar and the main class is com.penny.CharCount.java, which requires an input file name and output directory as input parameters. Which of the following is the correct command to submit a job in Spark with the given constraints?

Review the following Spark job description:

1. Monitor file system director for new files. 
2. For new files created in the “/rambo” dictionary, perform word count.

Which of the following snippets would achieve this?

Consider the following Spark DataFrame:

Which of the given code fragments produce the following result:

Assuming the following Hive statements execute successfully, choose the correct statements that describe the result:

A. Hive reformats JobsFile2 into a structure that Hive can access and moves into to /user/steve/fenty/
B. The file named JobsFile2 is moved to to/user/steve/fenty/
C. The contents of JobsFile2 are parsed as comma-delimited rows and loaded into /user/steve/fenty/
D. The contents of JobsFile2 are parsed as comma-delimited rows and stored in a database

Do you know when the onCompletion() method is called?

Our ecommerce website, Dove Bonito maintains two topics: 1 high volume topic, "purchase" with 5 partitions and 1 low volume topic "customer" with 3 partitions. The team wants to do a stream-table join of these topics. What would you recommend?

What is an adequate topic configuration for the topic word-count-output?

Consider the following file:

The file might have incomplete data stored with ! or ?. Which of the following R snippets will load the file completely?

Review the following Dataframe 'bazinga':

Which of the following commands would turn it into the Dataframe shown below:

Consider the following SQL table:

How many rows does the following SQL query return?

Consider the following SQL table:

Which of the following SQL queries would return the year when neither a football or cricket winner was chosen?

Consider the following SQL table:

How many tuples does the following query return?

Look at the given ER diagram. What do you think is the least number of tables we would need to represent M, N, P, R1 and R2?