Can you explain the difference between IaaS, PaaS, and SaaS in the Azure ecosystem, especially in the context of data engineering? IaaS gives you full control over infrastructure—for example, running SQL Server on an Azure VM. PaaS, like Azure SQL Database or Data Factory, abstracts infrastructure and handles scalability, backups, etc. SaaS is a fully managed solution like Power BI, where you just use the service without managing infra or platform. In data projects, I typically use a mix of PaaS and SaaS for speed and scalability.
How do you choose between Azure Synapse Analytics, Azure SQL Database, and Azure Databricks for a specific data processing task? It depends on the use case. If it’s transactional or OLTP, I go for Azure SQL DB. For massive analytical workloads needing MPP, Synapse is a good fit. When I need advanced transformations, big data processing, or machine learning, Databricks with PySpark is my go-to. In real projects, I often use a combination—Databricks for heavy ETL, and Synapse for reporting.
What challenges have you faced while building data pipelines in Azure, and how did you overcome them? Schema drift and unstable source systems are common. I use parameterized, metadata-driven pipelines to handle such changes. For transient failures, I configure retries and alerts. And for governance, I integrate Purview or build lineage tracking to stay compliant.
How would you design a pipeline to copy over 500 tables from an on-prem SQL Server to Azure Data Lake, while accounting for future schema changes? I’d use a metadata-driven pipeline. Table names, source queries, and sink paths go into a control table. Then, I loop through the metadata using a ForEach activity and use a dynamic Copy activity. I enable schema drift and auto-mapping to support schema evolution.
When source schemas are changing, how do you manage schema drift in Azure Data Factory? I enable the “Allow Schema Drift” option in Mapping Data Flows. Additionally, I use derived columns to handle missing or additional fields gracefully. For complex scenarios, I store expected schema in metadata and validate against it during runtime.
Can you walk me through how you’ve implemented CI/CD in ADF using GitHub or Azure DevOps? In ADF, I enable Git integration for source control. For CI/CD, I use Azure DevOps pipelines with ARM templates exported from the ‘Manage hub’. During deployment, I replace parameters using a parameter file, and the pipeline deploys to higher environments using a release pipeline.
How do you manage reusable ADF pipelines that load different tables without duplicating code? I create a generic pipeline that accepts table name, schema, and file path as parameters. The actual source queries and sink destinations are managed in a control table or config file. This avoids code duplication and scales well.
In case a pipeline fails in ADF, how do you ensure retry and proper alerting? I configure retry policies on activities—usually 3 retries with intervals. I also add an If Condition to handle failures and send email or Teams alerts via Logic Apps or Webhook. For enterprise solutions, I integrate Azure Monitor with Log Analytics.
How would you design an ADF pipeline that respects REST API throttling limits during data ingestion? I use pagination and set concurrency to 1 to avoid hitting limits. Additionally, I introduce a wait/sleep mechanism using Until + Wait activities. For dynamic calls, I batch requests using parameter files and handle rate limits using logic in the pipeline.
What steps do you take to optimize Mapping Data Flows in ADF when dealing with large datasets? I use staging transformations, enable partitioning explicitly, and avoid unnecessary derived columns. I also profile data to choose proper partition keys and test performance with debug mode before publishing.
What factors do you consider when choosing between Self-hosted IR and Azure IR? If the data source is on-prem or behind a firewall, I go with Self-hosted IR. For cloud-native sources, Azure IR is preferred. I’ve also used hybrid IR setups when combining on-prem and cloud data sources in a single solution.
How do you implement incremental data loads in ADF using watermark logic? I track the last modified date in a watermark table or use the system column if available. The query in the Copy activity uses this watermark to pull only new or changed records. After successful load, the watermark is updated.
What are your methods for performing data quality and validation checks within ADF pipelines? I use derived columns and conditional splits in Mapping Data Flows to detect nulls, duplicates, or invalid data. Invalid rows are logged to a separate error file. Additionally, I log row counts and perform pre/post load validation in SQL or Python.
What strategies do you use to optimize a slow-running PySpark job in Databricks? First, I check for skewed joins and use broadcast() if applicable. Then, I cache intermediate results, reduce shuffles, and repartition wisely. If data is uneven, I apply salting techniques. Finally, I monitor job execution via Spark UI.
How would you explain the difference between cache(), persist(), and broadcast() in Spark? cache() stores data in memory only. persist() can use memory and disk. broadcast() is for small datasets to send across all nodes to avoid shuffling. I use broadcast() for small lookup tables in joins and persist() for reusing expensive computations.
Have you ever used Z-Ordering or Optimize in Delta tables? Can you explain with a use case? For a retail client, we had frequent queries on Customer_ID. I applied Z-Ordering on Customer_ID after OPTIMIZE to reduce IO. This significantly improved query performance on large Delta tables.
How do you handle skewed joins in Databricks Spark? If one side is much larger or skewed, I use techniques like broadcasting the smaller dataset or salting the key. I also use skewJoin hints and partitioning strategies. Spark UI helps identify skewed stages.
Can you explain how you’ve used Delta Live Tables (DLT) for handling Change Data Capture? I used DLT with expectations and CDC merge logic. The Bronze layer gets raw data; Silver handles deduplication using merge logic on _change_type; and Gold is used for reporting. DLT automatically handles job orchestration and lineage.
How does Unity Catalog help with data governance in Databricks? Unity Catalog centralizes data access control, lineage, and audit logging across workspaces. I’ve used it to manage RBAC down to column level and integrate with Purview for governance. It also simplifies credential passthrough and external location control.
How do you implement data masking and RBAC with Unity Catalog? I define external locations and assign access using Unity Catalog roles. For masking, I use dynamic views with CASE WHEN to hide PII data. RBAC is applied at schema/table/column level using GRANT statements.
Can you explain the difference between Dedicated and Serverless SQL Pools in Synapse, and when you’d use each? Dedicated pools are for high-performance, pre-provisioned workloads; best for predictable, heavy queries. Serverless is pay-per-query and great for ad-hoc or infrequent queries over files in ADLS. I typically use Serverless for exploration, Dedicated for production.
What are PolyBase external tables in Synapse, and how do they compare to COPY INTO? PolyBase lets you query external data in-place via external tables. COPY INTO loads data physically into Synapse tables. For ELT, I use COPY INTO for performance. For lightweight access without ingestion, PolyBase works well.
How do you optimize performance of queries in Dedicated SQL Pools? I create stats manually, use proper distribution types (hash/replicated), partition large tables, and avoid data movement. Monitoring with sys.dm_pdw_exec_requests helps catch slow queries.
Describe a use case where you built a star or snowflake schema in Synapse. In a healthcare project, I modeled patient visits (fact) with dimensions like patient, doctor, facility. Snowflake was used for hierarchical data like facility → region → country. Queries ran faster with proper indexing and partitioning.
How do you implement caching or materialized views in Synapse for performance improvements? I use CREATE MATERIALIZED VIEW to cache frequently joined or aggregated data. These are automatically refreshed. For Serverless, I cache results in Parquet and reuse them across queries.
What are best practices for partitioning and data distribution in Synapse? For large fact tables, I use hash distribution on high-cardinality columns. For smaller tables, replicate them. Partitioning is used on date or region to enable pruning. Monitoring skewness helps validate distribution choices.
How do you monitor query performance in Synapse Analytics? I use the built-in monitoring hub, sys.dm_pdw_exec_requests, and sys.dm_pdw_request_steps. For alerts, I integrate with Log Analytics and build dashboards in Power BI or Grafana.
How do you enforce RBAC in Synapse, ADF, or Databricks? I assign roles via Azure AD groups and control access via IAM roles. In Synapse, I use SQL roles and workspace permissions. In ADF, I restrict access using RBAC roles (Reader, Contributor). In Databricks, Unity Catalog simplifies fine-grained controls.
What is your approach to auditing data access in Azure environments? I enable diagnostic logs in Synapse, ADF, and Databricks. These are sent to Log Analytics or Event Hub. I also use Purview to track access patterns and build reports for audit teams.
How do you implement end-to-end data lineage in Azure Lakehouse architecture? I tag datasets with metadata, use naming standards, and log lineage in a metadata table. For enterprise setups, I integrate with Purview for automated lineage tracking across ADF, Synapse, and Databricks.
How do you secure data at rest and in transit across Azure data services? At rest, I use managed keys (or customer-managed for sensitive data). In transit, HTTPS is default. For highly sensitive solutions, I enable private endpoints and VNET integration.
How does Unity Catalog improve security in multi-tenant environments? It separates governance from workspace-level control, allowing centralized policies across tenants. I can enforce policies like data masking and manage access at the catalog level, not just workspace.
If a pipeline is taking 4 hours, how would you debug and optimize it? First, I check activity execution times in ADF monitor. Then, I review source query performance, check for sequential dependencies, and parallelize where possible. I also profile data volumes and optimize transformations.
How do you manage concurrency and parallel execution in ADF and Databricks pipelines? In ADF, I use ForEach with batch count and manage concurrency via integration runtime limits. In Databricks, I scale clusters and configure parallel jobs using task dependencies and job clusters.
What would you do if your Databricks job faces frequent cluster timeouts? I’d review cluster logs to find root cause—could be idle timeouts or memory errors. I’d optimize the job to run faster or adjust cluster auto-termination settings. If usage is irregular, I may switch to job clusters.
What are your cost optimization strategies in Synapse, ADF, and Databricks? I shut down unused Dedicated pools, switch to Serverless when possible, and use job clusters in Databricks. In ADF, I optimize pipeline runs and avoid unnecessary data movement. Monitoring usage helps fine-tune cost.
What metrics do you track to identify production pipeline issues? Failure rate, data volume variance, execution time, retries, and row counts. I also track SLA adherence and use alerts for anomalies.
Can you walk me through how you implement medallion architecture using ADF and Databricks? ADF ingests raw data to Bronze in ADLS. Databricks then cleans/transforms it to Silver. Gold is for business-ready data. I separate logic per layer, automate dependencies, and use Delta for ACID guarantees.
How do you handle real-time data processing in Azure with Databricks and Event Hubs or Stream Analytics? I connect Event Hubs to a Structured Streaming job in Databricks. For simple scenarios, I use Stream Analytics. Data is transformed and written to Delta tables or pushed to Power BI via Direct Lake or Synapse.
If you had to process millions of small files from ADLS Gen2, what’s your approach? I consolidate small files using Spark to avoid file open overhead. I also use autoloader with Databricks or merge files via ADF. Partitioning helps manage them efficiently.
How do you version control notebooks and pipeline code in Azure projects? I use GitHub or Azure Repos. Notebooks are saved as .ipynb or .dbc. For ADF, I use Git integration in the studio and deploy via ARM templates. Everything goes through PRs and branches.
What’s your strategy for zero-downtime or blue-green deployment in Azure data solutions? I deploy to a staging environment first, validate, then switch over. In Synapse or ADF, I deploy via parameterized templates. For Delta, I use versioning and time travel to revert if needed.
How would you architect a solution integrating Power BI, Synapse, and ADF for a finance dashboard? ADF ingests and prepares data, Synapse stores and serves it via Dedicated SQL Pool, and Power BI connects via DirectQuery. I implement role-level security and optimize queries with materialized views.
Describe a pipeline you built that merges data from SAP (on-prem), Salesforce (cloud), and SQL Server into a Data Lake. I used Self-hosted IR for SAP, REST connector for Salesforce, and SQL for SQL Server. ADF ingested and standardized the data. A metadata table controlled ingestion. Databricks handled further transformation to Delta.
How do you handle data refresh failures that impact downstream Power BI or reporting systems? I log failure details, trigger alerting, and automatically retry. For Power BI, I build a fallback mechanism or show last successful load date. I also use pipeline success flags for dependent refresh triggers.
Can you describe your approach to implementing disaster recovery for Azure Data Factory pipelines across regions? I start by exporting ADF pipelines as ARM templates and storing them in a Git repo. For each region, I parameterize resource names and endpoints. During DR drills, I deploy these templates to the secondary region via Azure DevOps with environment-specific parameters. I also replicate Linked Services’ secrets by syncing Key Vaults. Finally, I validate end-to-end connectivity and run smoke tests to ensure pipelines execute correctly in failover.
How do you manage environment-specific configurations (dev, test, prod) in ADF without hard-coding values? I use Azure Key Vault to store sensitive values like connection strings and credentials. In ADF, Linked Services reference Key Vault secrets. For non-sensitive settings, I maintain a JSON configuration file in a storage account or repo, load it via a Lookup activity at pipeline start, and pass values through parameters. This keeps pipelines environment-agnostic and easy to promote via CI/CD.
What’s your process for implementing data lineage tracking using Azure Purview and ADF? I first register ADF artifacts in Purview by enabling the Purview connector. Then, I tag datasets and Linked Services with custom metadata, which flows into Purview’s lineage graphs. For custom transformations in Databricks, I emit lineage metadata via REST API calls to Purview. This creates an end-to-end lineage—from source systems through ADF and Databricks into Synapse or ADLS—visible in Purview’s UI.
How have you handled GDPR-style data deletion requests in an Azure Data Lake environment? I maintain a catalog of personal data columns via Purview and Unity Catalog. When a deletion request arrives, I run a Databricks notebook that reads the GDPR list, locates affected Delta partitions, and applies a “soft delete” by marking rows with a deletion flag. If hard deletion is necessary, I use Delta’s DELETE WHERE support. I also version control these notebooks and log each action for audit compliance.
How do you create and maintain reusable pipeline templates in ADF for multi-tenant scenarios? I design pipeline modules (e.g., copy, transform, load) with well-defined parameters—source, sink, schema, error path. These modules are stored as ARM template fragments or in a central ADF project folder. For each tenant, I pass tenant-specific configurations via a master orchestration pipeline or DevOps variable groups. This modular design reduces duplication and accelerates onboarding of new tenants.
