workspaces

APRL GUID:  7fb90127-5364-bb4d-86fa-30778ed713fb

Description:

When creating a Databricks cluster, you can set a log delivery location for the Spark driver, worker nodes, and events. Logs are delivered every 5 mins and archived hourly. Upon cluster termination, all generated logs until that point are guaranteed to be delivered.

Potential Benefits:

Improved troubleshooting and audit

Impact:  High Category:  High Availability

Delta Lake is an open source storage format enhancing data lakes' reliability with ACID transactions, schema enforcement, and scalable metadata handling.

Enhances data reliability and processing

Learn More:

ARG Query:

// under-development

Automatically rescue invalid or nonconforming data with Databricks Auto Loader or Delta Live Tables

Invalid or nonconforming data can crash workloads dependent on specific data formats. Best practices recommend filtering such data at ingestion to improve end-to-end resilience, ensuring no data is lost or missed.

Enhanced data resilience and integrity

// under-development

Use Databricks and MLflow for deploying models as Spark UDFs for job scheduling, retries, autoscaling. Model serving offers scalable infrastructure, processes models using MLflow, and serves them via REST API using serverless compute managed in Databricks cloud.

Enhanced reliability and autoscaling

// under-development

Use Databricks and MLflow for deploying models as Apache Spark UDFs, benefiting from job scheduling, retries, autoscaling, etc.

Enhances scalability and reliability

// under-development

Curate data by creating a layered architecture to increase data quality across layers. Start with a raw layer for ingested source data, continue with a curated layer for cleansed and refined data, and finish with a final layer catered to business needs, focusing on security and performance.

Enhances data quality and trust

// under-development

Copying data leads to redundancy, lost integrity, lineage, and access issues, affecting lakehouse data quality. Temporary copies are useful for agility and innovation but can become problematic operational data silos, questioning data's master status and currency.

Enhanced data integrity and quality

// under-development

Uncontrolled schema changes can lead to invalid data and failing jobs. Databricks validates and enforces schema through Delta Lake, which prevents bad records during ingestion, and Auto Loader, which detects new columns and supports schema evolution to maintain data integrity.

Prevents invalid data and job failures

// under-development

Delta tables verify data quality automatically with SQL constraints, triggering an error for violations. Delta Live Tables enhance this by defining expectations for data quality, utilizing Python or SQL, to manage actions for record failures, ensuring data integrity and compliance.

Ensures data quality and integrity

// under-development

To recover from a failure, regular backups are needed. The Databricks Labs project migrate lets admins create backups by exporting workspace assets using the Databricks CLI/API. These backups help in restoring or migrating workspaces.

Ensures data recovery and migration

// under-development

Structured Streaming ensures fault-tolerance and data consistency in streaming queries. With Azure Databricks workflows, you can set up your queries to automatically restart after failure, picking up precisely where they left off.

Fault-tolerance and auto-restart for queries

// under-development

Despite thorough testing, a production job can fail or yield unexpected data. Sometimes, repairs are done by adding jobs post-issue identification and pipeline correction.

Easy rollback and fix for ETL jobs

// under-development

Databricks Workflows enable efficient error recovery in multi-task jobs by offering a matrix view for issue examination. Fixes can be applied to initiate repair runs targeting only failed and dependent tasks, preserving successful outcomes and thereby saving time and money.

Saves time and money with smart recovery

// under-development

Implementing a disaster recovery pattern is vital for Azure Databricks, ensuring data teams' access even during rare regional outages.

It is important to note that the Azure Databricks service is not entirely zone redudant and does support zonal failover.

Ensures service continuity during disasters

// under-development

The Databricks Terraform provider manages Azure Databricks workspaces and cloud infrastructure flexibly and powerfully.

Efficient, reliable automation

// under-development

The Databricks Terraform provider is a flexible, powerful tool for managing Azure Databricks workspaces and cloud infrastructure.

Enhanced reliability and automation

// under-development

Customers often naturally divide workspaces by teams or departments. However, it's crucial to also consider Azure Subscription and Azure Databricks (ADB) Workspace limits when partitioning.

Enhanced limits management, team separation

// under-development

Deploying only one Databricks Workspace per VNet aligns with Azure Databricks' isolation model.

Enhanced security and resource isolation

// under-development

Driven by security and data availability concerns, each Azure Databricks Workspace comes with a default DBFS designed for system-level artifacts like libraries and Init scripts, not for production data.

Enhanced security, data protection

// under-development

Azure Spot VMs are not suitable for critical production workloads needing high availability and reliability. They are meant for fault-tolerant tasks and can be evicted with 30-seconds notice if Azure needs the capacity, with no SLA guarantees.

Ensures high reliability for production

// under-development

Move workspaces to in-region control plane for increased regional isolation. Identify current control plane region using the workspace URL and nslookup. When region from CNAME differs from workspace region and an in-region control is available, consider migration using tools provided below.

Improves resilience and data sovereignty

// cannot-be-validated-with-arg

Azure Databricks planning should include VM SKU swap strategies for capacity issues. VMs are regional, and allocation failures may occur, shown by a "CLOUD PROVIDER" error.

Ensures service availability

// under-development

Where possible Leverage managed services (serverless compute) from the Databricks Data Intelligence Platform, such as:serverless SQL warehouses, model serving, serverless jobs, serverless compute for notebooks.

Improve resiliency and scalability at no additional cost

// cannot-be-validated-with-arg