Data Engineer Test

Chusk works as Hadoop developer at Pesla Inc. Chusk is tasked with processing input data to count number of occurrences of each unique word. Chusk did the following to achieve this:

1. Tokenize each word and emit lateral value 1 with Mapper
2. Reducer increments counter for each literal 1 it receives

Chusk is now tasked with optimizing this by using a combiner. Will Chusk be able to reuse existing reducers as combiners?

A: Yes
B: No
C: Because the sum operation is both associative and commutative and the input and output types to the reduce method match
D: Because the sum operation in the Reducer is incompatible with the operation of a combiner
E: Because the combiner is incompatible with a Mapper, which doesn't use the same data type for both the key and value
F: Insufficient information

Assuming the following Hive statements execute successfully, choose the correct statements that describe the result:

from fooddata select context_ngrams(sentences(lines),
array("twiggy", "romato", null), 68);

A. A bigram of the top 68 sentences that contain the substring "twiggy romato" in the lines column of the input data A1 table.
B. An 68-value ngram of sentences that contain the words "twiggy" or "romato" in the lines column of the fooddata table.
C. A trigram of the top 68 sentences that contain "twiggy romato" followed by a null space in the lines column of the fooddata table.
D. A frequency distribution of the top 68 words that follow the subsequence "twiggy romato" in the lines column of the fooddata table.

Consider the following two relations, P and Q:

What is the output of the following Pig command?

Q = GROUP P BY p2;
DUMP Q;

Penny created a jar file for her character count example written in Java. The jar name is attempt.jar and the main class is com.penny.CharCount.java, which requires an input file name and output directory as input parameters. Which of the following is the correct command to submit a job in Spark with the given constraints?

Review the following Spark job description:

1. Monitor file system director for new files. 
2. For new files created in the “/rambo” dictionary, perform word count.

Which of the following snippets would achieve this?

Consider the following Spark DataFrame:

Which of the given code fragments produce the following result:

Consider the following two Hive tables in a space adventure context:

Assume that the astronauts table is small and can fit in memory. Which of the following queries will take full advantage of a map-side join?

You have a partitioned Hive table sales_data with the following schema:

You want to archive the oldest partition (date='2023-01-01', region='North America') and store it in the Hadoop Distributed File System (HDFS) at /archive/sales_data. Which of the following steps is the correct approach to perform the archiving operation?

A: Enable archiving on the table, and then issue an ALTER TABLE statement to move the partition to the archive directory.

B: Issue an ALTER TABLE statement to move the partition to the archive directory, and then lock the partition to prevent further modifications.

C: Enable archiving on the table, and then use the EXPORT command to export the partition to the archive directory.

D: Use the EXPORT command to export the partition to the archive directory, and then lock the partition to prevent further modifications.

E: Use the EXPORT command to export the partition to the archive directory, and then drop the partition from the sales_data table.

Assuming the following Hive statements execute successfully, choose the correct statements that describe the result:

A. Hive reformats JobsFile2 into a structure that Hive can access and moves into to /user/steve/fenty/
B. The file named JobsFile2 is moved to to/user/steve/fenty/
C. The contents of JobsFile2 are parsed as comma-delimited rows and loaded into /user/steve/fenty/
D. The contents of JobsFile2 are parsed as comma-delimited rows and stored in a database

Do you know when the onCompletion() method is called?

Our ecommerce website, Dove Bonito maintains two topics: 1 high volume topic, "purchase" with 5 partitions and 1 low volume topic "customer" with 3 partitions. The team wants to do a stream-table join of these topics. What would you recommend?

What is an adequate topic configuration for the topic word-count-output?

Consider the following R code:

This script defines a function apply_fun that calculates the mean of a vector and adds it to the standard deviation of the vector. The apply function is then used to apply this function to each column of the data frame df. What will be the value of result after the script is run?

Review the following Dataframe 'bazinga':

Which of the following commands would turn it into the Dataframe shown below:

Consider the following pseudo code:

What will be the state of list_of_matrices after running the script?

Consider the following SQL table:

How many rows does the following SQL query return?

Consider the following SQL table:

Which of the following SQL commands will find the ‘nth highest Sales’ if it exists (returns null otherwise)?

Consider the following SQL table:

Which of the following SQL queries would return the year when neither a football or cricket winner was chosen?

Consider the following SQL table:

What will be the first two tuples resulting from the following SQL command?

Consider the following SQL table:

How many tuples does the following query return?

Look at the given ER diagram. What do you think is the least number of tables we would need to represent M, N, P, R1 and R2?