For details about the serverless compute plane that is used for serverless SQL warehouses, see Serverless compute. ![]() The following diagram describes the overall architecture. 5 contributors Feedback In this article Ingest raw data to the bronze layer Validate and deduplicate data in the silver layer Power analytics with the gold layer The medallion architecture describes a series of data layers that denote the quality of data stored in the lakehouse. High-level architectureĪlthough architectures can vary depending on custom configurations (such as when you’ve deployed an Azure Databricks workspace to your own virtual network, also known as VNet injection), the following architecture diagram represents the most common structure and flow of data for Azure Databricks. Note that some metadata about results, such as chart column names, continues to be stored in the control plane. See Configure the storage location for interactive notebook results. If you want interactive notebook results stored only in your Azure subscription, you can configure the storage location for interactive notebook results. For interactive notebook results, storage is in a combination of the control plane (partial results for presentation in the UI) and your Azure storage. Job results reside in storage in your Azure subscription. Your data lake is stored at rest in your Azure subscription and in your own data sources so you maintain control and ownership of your data. To configure the networks for your classic compute plane, see Classic compute plane network connectivity. You can also ingest data from external streaming data sources, such as events data, streaming data, IoT data, and more. Use Azure Databricks connectors to connect clusters to external data sources outside of your Azure subscription to ingest data or for storage. Previously, Azure Databricks referred to the compute plane as the data plane. For additional architecture information, see Serverless compute. For serverless SQL warehouses or Model Serving, the serverless compute resources run in a serverless compute plane in your Azure Databricks account.Azure Databricks uses the classic compute plane for your notebooks, jobs, and for pro and classic Databricks SQL warehouses. This refers to the network in your Azure subscription and its resources. ![]() For most Azure Databricks computation, the compute resources are in your Azure subscription in what is called the classic compute plane.The compute plane is where your data is processed. Notebook commands and many other workspace configurations are stored in the control plane and encrypted at rest. ![]() The control plane includes the backend services that Azure Databricks manages in your Azure Databricks account. Control plane and compute planeĪzure Databricks is structured to enable secure cross-functional team collaboration while keeping a significant amount of backend services managed by Azure Databricks so you can stay focused on your data science, data analytics, and data engineering tasks.Īzure Databricks operates out of a control plane and a compute plane. This article provides a high-level overview of Azure Databricks architecture, including its enterprise architecture, in combination with Azure.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |