This week I accidentally passed a bad update to a Delta Table that totally corrupted my entire table. In most cases, this is a problem that will need to be solved by reloading the table, merging correct data, etc. I thought this would be a great opportunity to showcase one of the benefits of building …
Time Travel Utilizing Delta Lake in Microsoft’s Spark Version Read More »
In the digital landscape of modern manufacturing, the heartbeat of efficiency lies in the Industrial Internet of Things (IIoT). The Industrial Internet of Things refers to a mesh of interconnected sensors, instruments, and machinery that have become an integral part of any Industrial Organization. Imagine a symphony of interconnected devices orchestrating the seamless production of …
Migrating Delta Lake Objects Between Environments with Azure Synapse Introduction I had, what I thought to be, a fairly standard use case come across my desk related to deploying delta lake objects between logical dev/test/prod environments in my recent implementation of delta lake architecture. At the surface, this seems like it would be fairly straightforward …
Migrating Delta Lake Objects Between Environments with Azure Synapse Read More »
Automate Toggling Pipeline Triggers During Office Hours…a Weird Limitation and Solution in Azure Data Factory Preface: This week, an unusual request landed on our desks, prompting me to share a brief highlight on TDAD. I want to give a special shout-out to my colleague Elisabeth Henhapl for devising a solution that, in my opinion, ingeniously …
Case Study: How I Went About Optimizing My On-Prem to Cloud Copy Performance Explaining the Use Case Not too long ago, I built out a solution where the initial component was all centered around data transfer from an On-premises SQL DB and landing that data into ADLS for further ETL processing before landing in my …
Case Study: How I Went About Optimizing My On-Prem to Cloud Copy Performance Read More »
Unraveling Microsoft Fabric: An Introduction to Fabric’s Data Engineering Today we are back at it again with Microsoft Fabric. In this blog post, I thought it would be nice to demonstrate the innerworkings of the Synapse Data Engineering service through the use of practical examples. What I will be attempting to do is load data …
Unraveling Microsoft Fabric: An Introduction to Fabric’s Data Engineering Read More »
Unraveling Microsoft Fabric: An Introduction to Fabric’s Data Factory Hello, everyone! We’re back in the world of Microsoft Fabric this week, ready to delve into one of its more technical components. So far, we’ve provided an overview of Microsoft Fabric and delved into the intricacies of Fabric’s foundation, OneLake. Now, it’s time to explore our …
Unraveling Microsoft Fabric: An Introduction to Fabric’s Data Factory Read More »
PLEASE Replace your Nested Notebook in a ForEach Activity Loop with A Looped Notebook When Leveraging Spark For those of you who are building repeatable processing in Azure Data Factory or Synapse Analytics Workspace Pipelines, you are probably familiar with the concept of the ForEach Activity Loop. This is a powerful tool to create scalable …
Unraveling Microsoft Fabric: An Introduction to OneLake Last week we covered Microsoft Fabric, exploring its core identity, its versatile use cases, and how it marks a significant stride in Microsoft’s ongoing mission to streamline analytics workloads. The goal? To empower organizations to spend less time on integration tasks and more time delivering real value to …
Unraveling Microsoft Fabric: An Introduction to OneLake Read More »
Presenting Microsoft Fabric: A Comprehensive Introduction If you’ve recently accessed your Power BI Portals, you couldn’t have missed the striking transformation in the User Interface. The familiar yellow hues and the iconic Power BI logo have given way to a new greenish-blue emblem, accompanied by a several intriguing tabs leading to uncharted territories within the …
Unraveling Microsoft Fabric: A Comprehensive Introduction Read More »
