PDF (New 2024) Actual Databricks Databricks-Certified-Professional-Data-Engineer Exam Questions
Dumps Moneyack Guarantee - Databricks-Certified-Professional-Data-Engineer Dumps UpTo 90% Off
Databricks is a cloud-based data analytics platform that enables businesses to extract valuable insights from large volumes of data. The platform is designed to automate data processing, making it easier for organizations to derive insights from data. Databricks has become an essential tool for data professionals globally, and as such, the demand for Databricks-certified professionals is on the rise.
Databricks Certified Professional Data Engineer exam is a valuable certification for professionals who want to showcase their expertise in big data processing using Databricks. Databricks Certified Professional Data Engineer Exam certification demonstrates that the candidate has the necessary skills and knowledge to design and implement scalable data pipelines using Databricks. Databricks Certified Professional Data Engineer Exam certification also provides a competitive advantage to professionals in the job market and opens up new career opportunities in the field of big data engineering.
Databricks Certified Professional Data Engineer Exam is a comprehensive exam that covers a wide range of topics related to data engineering. These topics include data modeling, data ingestion, data integration, data transformation, data storage, and data analysis. Candidates are required to demonstrate their knowledge and skills in these areas by completing a series of tasks and exercises.
NEW QUESTION # 65
Which of the following operations are not supported on a streaming dataset view?
spark.readStream.format("delta").table("sales").createOrReplaceTempView("streaming_view")
- A. SELECT id, sum(unitssold) FROM streaming_view GROUP BY id ORDER BY id
- B. SELECT id, count(*) FROM streaming_view GROUP BY id
- C. SELECT max(unitssold) FROM streaming_view
- D. SELECT * FROM streadming_view ORDER BY id
- E. SELECT sum(unitssold) FROM streaming_view
Answer: D
Explanation:
Explanation
The answer isSELECT * FROM streadming_view order by id Please Note: Sorting with Group by will work without any issues see below explanation for each option of the options, Graphical user interface, text, application Description automatically generated
Certain operations are not allowed on streaming data, please see highlighted in bold.
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#unsupported-operations
* Multiple streaming aggregations (i.e. a chain of aggregations on a streaming DF) are not yet supported on streaming Datasets.
* Limit and take the first N rows are not supported on streaming Datasets.
* Distinct operations on streaming Datasets are not supported.
* Deduplication operation is not supported after aggregation on a streaming Datasets.
* Sorting operations are supported on streaming Datasets only after an aggregation and in Complete Output Mode.
Note: Sorting without aggregation function is not supported.
Here is the sample code to prove this,
Setup test stream
Graphical user interface, text, application, email Description automatically generated
Sum aggregation function has no issues on stream
Graphical user interface, application Description automatically generated
Max aggregation function has no issues on stream
Graphical user interface, application Description automatically generated
Group by with Order by has no issues on stream
Group by has no issues on stream
Table Description automatically generated
Order by without group by fails.
Graphical user interface, text, application Description automatically generated
NEW QUESTION # 66
A data engineer needs to capture pipeline settings from an existing in the workspace, and use them to create and version a JSON file to create a new pipeline.
Which command should the data engineer enter in a web terminal configured with the Databricks CLI?
- A. Use list pipelines to get the specs for all pipelines; get the pipeline spec from the return results parse and use this to create a pipeline
- B. Stop the existing pipeline; use the returned settings in a reset command
- C. Use the get command to capture the settings for the existing pipeline; remove the pipeline_id and rename the pipeline; use this in a create command
- D. Use the alone command to create a copy of an existing pipeline; use the get JSON command to get the pipeline definition; save this to git
Answer: C
Explanation:
The Databricks CLI provides a way to automate interactions with Databricks services. When dealing with pipelines, you can use thedatabricks pipelines get --pipeline-idcommand to capture the settings of an existing pipeline in JSON format. This JSON can then be modified by removing thepipeline_idto prevent conflicts and renaming the pipeline to create a new pipeline. The modified JSON file can then be used with the databricks pipelines createcommand to create a new pipeline with those settings.
References:
* Databricks Documentation on CLI for Pipelines: Databricks CLI - Pipelines
NEW QUESTION # 67
The data science team has created and logged a production model using MLflow. The following code correctly imports and applies the production model to output the predictions as a new DataFrame namedpredswith the schema "customer_id LONG, predictions DOUBLE, date DATE".
The data science team would like predictions saved to a Delta Lake table with the ability to compare all predictions across time. Churn predictions will be made at most once per day.
Which code block accomplishes this task while minimizing potential compute costs?
- A. preds.write.mode("append").saveAsTable("churn_preds")
- B.
- C.
- D. preds.write.format("delta").save("/preds/churn_preds")
- E.
Answer: C
Explanation:
Explanation
This is the correct answer because it will save the predictions to a Delta Lake table with the ability to compare all predictions across time. The code uses the mergeInto method to perform an upsert operation, which means it will insert new records or update existing records based on the customer_id and date columns. This way, the table will always contain the latest predictions for each customer and date, and also keep the history of previous predictions. The code also uses a new job cluster to run the job, which will minimize the compute costs as it will be created and terminated for each run. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Upsert into a table using merge" section.
NEW QUESTION # 68
The data architect has decided that once data has been ingested from external sources into the Databricks Lakehouse, table access controls will be leveraged to manage permissions for all production tables and views.
The following logic was executed to grant privileges for interactive queries on a production database to the core engineering group.
GRANT USAGE ON DATABASE prod TO eng;
GRANT SELECT ON DATABASE prod TO eng;
Assuming these are the only privileges that have been granted to the eng group and that these users are not workspace administrators, which statement describes their privileges?
- A. Group members are able to create, query, and modify all tables and views in the prod database, but cannot define custom functions.
- B. Group members have full permissions on the prod database and can also assign permissions to other users or groups.
- C. Group members are able to query all tables and views in the prod database, but cannot create or edit anything in the database.
- D. Group members are able to list all tables in the prod database but are not able to see the results of any queries on those tables.
- E. Group members are able to query and modify all tables and views in the prod database, but cannot create new tables or views.
Answer: C
Explanation:
Explanation
The GRANT USAGE ON DATABASE prod TO eng command grants the eng group the permission to use the prod database, which means they can list and access the tables and views in the database. The GRANT SELECT ON DATABASE prod TO eng command grants the eng group the permission to select data from the tables and views in the prod database, which means they can query the data using SQL or DataFrame API.
However, these commands do not grant the eng group any other permissions, such as creating, modifying, or deleting tables and views, or defining custom functions. Therefore, the eng group members are able to query all tables and views in the prod database, but cannot create or edit anything in the database. References:
Grant privileges on a database:
https://docs.databricks.com/en/security/auth-authz/table-acls/grant-privileges-database.html Privileges you can grant on Hive metastore objects:
https://docs.databricks.com/en/security/auth-authz/table-acls/privileges.html
NEW QUESTION # 69
When scheduling Structured Streaming jobs for production, which configuration automatically recovers from query failures and keeps costs low?
- A. Cluster: Existing All-Purpose Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: 1 - B. Cluster: Existing All-Purpose Cluster;
Retries: None;
Maximum Concurrent Runs: 1 - C. Cluster: New Job Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: Unlimited - D. Cluster: Existing All-Purpose Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: 1 - E. Cluster: New Job Cluster;
Retries: None;
Maximum Concurrent Runs: 1
Answer: E
Explanation:
Explanation
This is the best configuration for scheduling Structured Streaming jobs for production, as it automatically recovers from query failures and keeps costs low. A new job cluster is created for each run of the job and terminated when the job completes, which saves costs and avoids resource contention. Retries are not needed for Structured Streaming jobs, as they can automatically recover from failures using checkpointing and write-ahead logs. Maximum concurrent runs should be set to 1 to avoid duplicate output or data loss. Verified References: Databricks Certified Data Engineer Professional, under "Monitoring & Logging" section; Databricks Documentation, under "Schedule streaming jobs" section.
NEW QUESTION # 70
A data analyst has noticed that their Databricks SQL queries are running too slowly. They claim that this issue
is affecting all of their sequentially run queries. They ask the data engineering team for help. The data
engineering team notices that each of the queries uses the same SQL endpoint, but the SQL endpoint is not
used by any other user.
Which of the following approaches can the data engineering team use to improve the latency of the data
analyst's queries?
- A. They can turn on the Serverless feature for the SQL endpoint and change the Spot In-stance Policy to
"Reliability Optimized" - B. They can turn on the Auto Stop feature for the SQL endpoint
- C. They can increase the maximum bound of the SQL endpoint's scaling range
- D. They can increase the cluster size of the SQL endpoint
- E. They can turn on the Serverless feature for the SQL endpoint
Answer: D
NEW QUESTION # 71
How to determine if a table is a managed table vs external table?
- A. Run IS_MANAGED('table_name') function
- B. Run SQL command DESCRIBE EXTENDED table_name and check type
- C. All managed tables are stored in unity catalog
- D. Run SQL command SHOW TABLES to see the type of the table
- E. All external tables are stored in data lake, managed tables are stored in DELTA lake
Answer: B
Explanation:
Explanation
The answer is Run SQL command DESCRIBE EXTENDED table_name and check type Example of External table Graphical user interface, text, application Description automatically generated
Example of managed table
Graphical user interface, text, application, Teams Description automatically generated
NEW QUESTION # 72
The DevOps team has configured a production workload as a collection of notebooks scheduled to run daily using the Jobs Ul. A new data engineering hire is onboarding to the team and has requested access to one of these notebooks to review the production logic.
What are the maximum notebook permissions that can be granted to the user without allowing accidental changes to production code or data?
- A. Can Read
- B. Can manage
- C. Can edit
- D. Can run
Answer: A
Explanation:
Granting a user 'Can Read' permissions on a notebook within Databricks allows them to view the notebook's content without the ability to execute or edit it. This level of permission ensures that the new team member can review the production logic for learning or auditing purposes without the risk of altering the notebook's code or affecting production data and workflows. This approach aligns with best practices for maintaining security and integrity in production environments, where strict access controls are essential to prevent unintended modifications.
Reference: Databricks documentation on access control and permissions for notebooks within the workspace (https://docs.databricks.com/security/access-control/workspace-acl.html).
NEW QUESTION # 73
A table named user_ltv is being used to create a view that will be used by data analysts on various teams.
Users in the workspace are configured into groups, which are used for setting up data access using ACLs.
The user_ltv table has the following schema:
email STRING, age INT, ltv INT
The following view definition is executed:
An analyst who is not a member of the marketing group executes the following query:
SELECT * FROM email_ltv
Which statement describes the results returned by this query?
- A. Only the email and ltv columns will be returned; the email column will contain the string
"REDACTED" in each row. - B. Three columns will be returned, but one column will be named "redacted" and contain only null values.
- C. The email and ltv columns will be returned with the values in user itv.
- D. Only the email and itv columns will be returned; the email column will contain all null values.
- E. The email, age. and ltv columns will be returned with the values in user ltv.
Answer: A
Explanation:
The code creates a view called email_ltv that selects the email and ltv columns from a table called user_ltv, which has the following schema: email STRING, age INT, ltv INT. The code also uses the CASE WHEN expression to replace the email values with the string "REDACTED" if the user is not a member of the marketing group. The user who executes the query is not a member of the marketing group, so they will only see the email and ltv columns, and the email column will contain the string "REDACTED" in each row.
Verified References: [Databricks Certified Data Engineer Professional], under "Lakehouse" section; Databricks Documentation, under "CASE expression" section.
NEW QUESTION # 74
All records from an Apache Kafka producer are being ingested into a single Delta Lake table with the following schema:
key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp LONG There are 5 unique topics being ingested. Only the "registration" topic contains Personal Identifiable Information (PII). The company wishes to restrict access to PII. The company also wishes to only retain records containing PII in this table for 14 days after initial ingestion. However, for non-PII information, it would like to retain these records indefinitely.
Which of the following solutions meets the requirements?
- A. Because the value field is stored as binary data, this information is not considered PII and no special precautions should be taken.
- B. All data should be deleted biweekly; Delta Lake's time travel functionality should be leveraged to maintain a history of non-PII information.
- C. Separate object storage containers should be specified based on the partition field, allowing isolation at the storage level.
- D. Data should be partitioned by the registration field, allowing ACLs and delete statements to be set for the PII directory.
- E. Data should be partitioned by the topic field, allowing ACLs and delete statements to leverage partition boundaries.
Answer: E
Explanation:
Explanation
Partitioning the data by the topic field allows the company to apply different access control policies and retention policies for different topics. For example, the company can use the Table Access Control feature to grant or revoke permissions to the registration topic based on user roles or groups. The company can also use the DELETE command to remove records from the registration topic that are older than 14 days, while keeping the records from other topics indefinitely. Partitioning by the topic field also improves the performance of queries that filter by the topic field, as they can skip reading irrelevant partitions. References:
Table Access Control: https://docs.databricks.com/security/access-control/table-acls/index.html DELETE: https://docs.databricks.com/delta/delta-update.html#delete-from-a-table
NEW QUESTION # 75
A distributed team of data analysts share computing resources on an interactive cluster with autoscaling configured. In order to better manage costs and query throughput, the workspace administrator is hoping to evaluate whether cluster upscaling is caused by many concurrent users or resource-intensive queries.
In which location can one review the timeline for cluster resizing events?
- A. Workspace audit logs
- B. Cluster Event Log
- C. Executor's log file
- D. Driver's log file
- E. Ganglia
Answer: E
NEW QUESTION # 76
Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.
Which statement describes a main benefit that offset this additional effort?
- A. Ensures that all steps interact correctly to achieve the desired end result
- B. Validates a complete use case of your application
- C. Yields faster deployment and execution times
- D. Troubleshooting is easier since all steps are isolated and tested individually
- E. Improves the quality of your data
Answer: D
NEW QUESTION # 77
A particular job seems to be performing slower and slower over time, the team thinks this started to happen when a recent production change was implemented, you were asked to take look at the job history and see if we can identify trends and root cause, where in the workspace UI can you perform this analysis?
- A. Under jobs UI select the job cluster, under spark UI select the application job logs, then you can access last 60 day historical runs
- B. Historical job runs can only be accessed by REST API
- C. Under Compute UI, select Job cluster and select the job cluster to see last 60 day his-torical runs
- D. Under Workspace logs, select job logs and select the job you want to monitor to view the last 60 day historical runs
- E. Under jobs UI select the job you are interested, under runs we can see current active runs and last 60 days historical run
Answer: E
Explanation:
Explanation
The answer is,
Under jobs UI select the job you are interested, under runs we can see current active runs and last 60 days historical run
NEW QUESTION # 78
The data architect has mandated that all tables in the Lakehouse should be configured as external Delta Lake tables.
Which approach will ensure that this requirement is met?
- A. When the workspace is being configured, make sure that external cloud object storage has been mounted.
- B. When configuring an external data warehouse for all table storage. leverage Databricks for all ELT.
- C. When tables are created, make sure that the external keyword is used in the create table statement.
- D. Whenever a table is being created, make sure that the location keyword is used.
- E. Whenever a database is being created, make sure that the location keyword is used
Answer: D
Explanation:
Explanation
This is the correct answer because it ensures that this requirement is met. The requirement is that all tables in the Lakehouse should be configured as external Delta Lake tables. An external table is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created by using the location keyword to specify the path to an existing directory in a cloud storage system, such as DBFS or S3. By creating external tables, the data engineering team can avoid losing data if they drop or overwrite the table, as well as leverage existing data without moving or copying it.
Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Create an external table" section.
NEW QUESTION # 79
Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?
- A. spark.sql.files.openCostInBytes
- B. spark.sql.files.maxPartitionBytes
- C. spark.sql.adaptive.coalescePartitions.minPartitionNum
- D. spark.sql.autoBroadcastJoinThreshold
- E. spark.sql.adaptive.advisoryPartitionSizeInBytes
Answer: B
Explanation:
This is the correct answer because spark.sql.files.maxPartitionBytes is a configuration parameter that directly affects the size of a spark-partition upon ingestion of data into Spark. This parameter configures the maximum number of bytes to pack into a single partition when reading files from file-based sources such as Parquet, JSON and ORC. The default value is 128 MB, which means each partition will be roughly 128 MB in size, unless there are too many small files or only one large file. Verified Reference: [Databricks Certified Data Engineer Professional], under "Spark Configuration" section; Databricks Documentation, under "Available Properties - spark.sql.files.maxPartitionBytes" section.
NEW QUESTION # 80
A Delta Lake table was created with the below query:
Consider the following query:
DROP TABLE prod.sales_by_store -
If this statement is executed by a workspace admin, which result will occur?
- A. The table will be removed from the catalog but the data will remain in storage.
- B. An error will occur because Delta Lake prevents the deletion of production data.
- C. Nothing will occur until a COMMIT command is executed.
- D. Data will be marked as deleted but still recoverable with Time Travel.
- E. The table will be removed from the catalog and the data will be deleted.
Answer: E
Explanation:
Explanation
When a table is dropped in Delta Lake, the table is removed from the catalog and the data is deleted. This is because Delta Lake is a transactional storage layer that provides ACID guarantees. When a table is dropped, the transaction log is updated to reflect the deletion of the table and the data is deleted from the underlying storage. References:
https://docs.databricks.com/delta/quick-start.html#drop-a-table
https://docs.databricks.com/delta/delta-batch.html#drop-table
NEW QUESTION # 81
A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, usingdisplay()calls to confirm code is producing the logically correct results as new transformations are added to an operation. To get a measure of average time to execute, the user is running each cell multiple times interactively.
Which of the following adjustments will get a more accurate measure of how code is likely to perform in production?
- A. Scala is the only language that can be accurately tested using interactive notebooks; because the best performance is achieved by using Scala code compiled to JARs. all PySpark and Spark SQL logic should be refactored.
- B. The only way to meaningfully troubleshoot code execution times in development notebooks Is to use production-sized data and production-sized clusters with Run All execution.
- C. The Jobs Ul should be leveraged to occasionally run the notebook as a job and track execution time during incremental code development because Photon can only be enabled on clusters launched for scheduled jobs.
- D. Production code development should only be done using an IDE; executing code against a local build of open source Spark and Delta Lake will provide the most accurate benchmarks for how code will perform in production.
- E. Calling display () forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results.
Answer: E
Explanation:
In Databricks notebooks, using thedisplay()function triggers an action that forces Spark to execute the code and produce a result. However, Spark operations are generally divided into transformations and actions.
Transformations create a new dataset from an existing one and are lazy, meaning they are not computed immediately but added to a logical plan. Actions, likedisplay(), trigger the execution of this logical plan.
Repeatedly running the same code cell can lead to misleading performance measurements due to caching.
When a dataset is used multiple times, Spark's optimization mechanism caches it in memory, making subsequent executions faster. This behavior does not accurately represent the first-time execution performance in a production environment where data might not be cached yet.
To get a more realistic measure of performance, it is recommended to:
* Clear the cache or restart the cluster to avoid the effects of caching.
* Test the entire workflow end-to-end rather than cell-by-cell to understand the cumulative performance.
* Consider using a representative sample of the production data, ensuring it includes various cases the code will encounter in production.
References:
* Databricks Documentation on Performance Optimization: Databricks Performance Tuning
* Apache Spark Documentation: RDD Programming Guide - Understanding transformations and actions
NEW QUESTION # 82
Which of the following statements can successfully read the notebook widget and pass the python variable to a SQL statement in a Python notebook cell?
- A. 1.order_date = dbutils.widgets.get("widget_order_date")
2.
3.spark.sql(f"SELECT * FROM sales WHERE orderDate = '{order_date}' ")
(Correct) - B. 1.order_date = dbutils.widgets.get("widget_order_date")
2.
3.spark.sql(f"SELECT * FROM sales WHERE orderDate = '${order_date }' ") - C. 1.order_date = dbutils.widgets.get("widget_order_date")
2.
3.spark.sql(f"SELECT * FROM sales WHERE orderDate = 'f{order_date }'") - D. 1.order_date = dbutils.widgets.get("widget_order_date")
2.
3.spark.sql(f"SELECT * FROM sales WHERE orderDate = 'order_date' ") - E. 1.order_date = dbutils.widgets.get("widget_order_date")
2.
3.spark.sql("SELECT * FROM sales WHERE orderDate = order_date")
Answer: A
NEW QUESTION # 83
A junior data engineer needs to create a Spark SQL table my_table for which Spark manages both the data and
the metadata. The metadata and data should also be stored in the Databricks Filesystem (DBFS).
Which of the following commands should a senior data engineer share with the junior data engineer to
complete this task?
- A. 1. CREATE TABLE my_table (id STRING, value STRING) USING DBFS;
- B. 1. CREATE MANAGED TABLE my_table (id STRING, value STRING);
- C. 1. CREATE MANAGED TABLE my_table (id STRING, value STRING) USING
2. org.apache.spark.sql.parquet OPTIONS (PATH "storage-path"); - D. 1. CREATE TABLE my_table (id STRING, value STRING);
- E. 1. CREATE TABLE my_table (id STRING, value STRING) USING
2. org.apache.spark.sql.parquet OPTIONS (PATH "storage-path")
Answer: D
NEW QUESTION # 84
Which of the following scenarios is the best fit for AUTO LOADER?
- A. Incrementally process new data from streaming data sources like Kafka into delta lake
- B. Efficiently copy data from one data lake location to another data lake location
- C. Efficiently move data incrementally from one delta table to another delta table
- D. Efficiently process new data incrementally from cloud object storage
- E. Incrementally process new data from relational databases like MySQL
Answer: D
Explanation:
Explanation
The answer is, Efficiently process new data incrementally from cloud object storage, AU-TO LOADER only supports ingesting files stored in a cloud object storage. Auto Loader cannot process streaming data sources like Kafka or Delta streams, use Structured streaming for these data sources.
Diagram Description automatically generated
Auto Loader and Cloud Storage Integration
Auto Loader supports a couple of ways to ingest data incrementally
1.Directory listing - List Directory and maintain the state in RocksDB, supports incremental file listing
2.File notification - Uses a trigger+queue to store the file notification which can be later used to retrieve the file, unlike Directory listing File notification can scale up to millions of files per day.
[OPTIONAL]
Auto Loader vs COPY INTO?
Auto Loader
Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a new Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.
When to use Auto Loader instead of the COPY INTO?
*You want to load data from a file location that contains files in the order of millions or higher. Auto Loader can discover files more efficiently than the COPY INTO SQL command and can split file processing into multiple batches.
*You do not plan to load subsets of previously uploaded files. With Auto Loader, it can be more difficult to reprocess subsets of files. However, you can use the COPY INTO SQL command to reload subsets of files while an Auto Loader stream is simultaneously running.
NEW QUESTION # 85
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?
- A. Regex
- B. Julia
- C. pyspsark.ml.feature
- D. Scala Datasets
- E. C++
Answer: A
Explanation:
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. References:
* https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions
* https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html
NEW QUESTION # 86
A junior developer complains that the code in their notebook isn't producing the correct results in the development environment. A shared screenshot reveals that while they're using a notebook versioned with Databricks Repos, they're using a personal branch that contains old logic. The desired branch named dev-2.3.9 is not available from the branch selection dropdown.
Which approach will allow this developer to review the current logic for this notebook?
- A. Use Repos to merge the current branch and the dev-2.3.9 branch, then make a pull request to sync with the remote repository
- B. Use Repos to make a pull request use the Databricks REST API to update the current branch to dev-2.3.9
- C. Merge all changes back to the main branch in the remote Git repository and clone the repo again
- D. Use Repos to checkout the dev-2.3.9 branch and auto-resolve conflicts with the current branch
- E. Use Repos to pull changes from the remote Git repository and select the dev-2.3.9 branch.
Answer: E
Explanation:
This is the correct answer because it will allow the developer to update their local repository with the latest changes from the remote repository and switch to the desired branch. Pulling changes will not affect the current branch or create any conflicts, as it will only fetch the changes and not merge them. Selecting the dev-2.3.9 branch from the dropdown will checkout that branch and display its contents in the notebook. Verified Reference: [Databricks Certified Data Engineer Professional], under "Databricks Tooling" section; Databricks Documentation, under "Pull changes from a remote repository" section.
NEW QUESTION # 87
Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM.
Given a job with at least one wide transformation, which of the following cluster configurations will result in maximum performance?
- A. * Total VMs; 1
* 400 GB per Executor
* 160 Cores / Executor - B. * Total VMs: 8
* 50 GB per Executor
* 20 Cores / Executor
Answer: B
Explanation:
C.
* Total VMs: 4
* 100 GB per Executor
* 40 Cores/Executor
D.
* Total VMs:2
* 200 GB per Executor
* 80 Cores / Executor
Explanation:
This is the correct answer because it is the cluster configuration that will result in maximum performance for a job with at least one wide transformation. A wide transformation is a type of transformation that requires shuffling data across partitions, such as join, groupBy, or orderBy. Shuffling can be expensive and time-consuming, especially if there are too many or too few partitions. Therefore, it is important to choose a cluster configuration that can balance the trade-off between parallelism and network overhead. In this case, having 8 VMs with 50 GB per executor and 20 cores per executor will create 8 partitions, each with enough memory and CPU resources to handle the shuffling efficiently. Having fewer VMs with more memory and cores per executor will create fewer partitions, which will reduce parallelism and increase the size of each shuffle block. Having more VMs with less memory and cores per executor will create more partitions, which will increase parallelism but also increase the network overhead and the number of shuffle files. Verified Reference: [Databricks Certified Data Engineer Professional], under "Performance Tuning" section; Databricks Documentation, under "Cluster configurations" section.
NEW QUESTION # 88
A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure.
The silver_device_recordings table will be used downstream to power several production monitoring dashboards and a production model. At present, 45 of the 100 fields are being used in at least one of these applications.
The data engineer is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields.
Which of the following accurately presents information about Delta Lake and Databricks that may impact their decision-making process?
- A. The Tungsten encoding used by Databricks is optimized for storing string data; newly-added native support for querying JSON strings means that string types are always most efficient.
- B. Because Delta Lake uses Parquet for data storage, data types can be easily evolved by just modifying file footer information in place.
- C. Human labor in writing code is the largest cost associated with data engineering workloads; as such, automating table declaration logic should be a priority in all migration workloads.
- D. Schema inference and evolution on .Databricks ensure that inferred types will always accurately match the data types used by downstream systems.
- E. Because Databricks will infer schema using types that allow all observed data to be processed, setting types manually provides greater assurance of data quality enforcement.
Answer: E
Explanation:
This is the correct answer because it accurately presents information about Delta Lake and Databricks that may impact the decision-making process of a junior data engineer who is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields. Delta Lake and Databricks support schema inference and evolution, which means that they can automatically infer the schema of a table from the source data and allow adding new columns or changing column types without affecting existing queries or pipelines. However, schema inference and evolution may not always be desirable or reliable, especially when dealing with complex or nested data structures or when enforcing data quality and consistency across different systems. Therefore, setting types manually can provide greater assurance of data quality enforcement and avoid potential errors or conflicts due to incompatible or unexpected data types. Verified Reference: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Schema inference and partition of streaming DataFrames/Datasets" section.
NEW QUESTION # 89
......
Updated Dec-2024 Pass Databricks-Certified-Professional-Data-Engineer Exam - Real Practice Test Questions: https://www.examcost.com/Databricks-Certified-Professional-Data-Engineer-practice-exam.html
Pass Your Exam With 100% Verified Databricks-Certified-Professional-Data-Engineer Exam Questions: https://drive.google.com/open?id=1Jyfe4eLGMPvnYEMmA0AfowDu5oLS7-C-
