Welcome to our “ETL Excellence” series! In this dedicated section, we delve deep into the world of data integration and ETL (Extract, Transform, Load). Explore a series of blog posts that illuminate the path to building scalable, multifunctional ETL streams within your data platforms.
In this section, I will guide you through configuring various connectors, selecting the right integration tools from the Microsoft stack, and crafting design patterns tailored to specific use cases. I recognize that the journey from raw data to actionable insights is a pivotal aspect of cloud data architecture. Therefore, this category serves as your compass in navigating this crucial terrain.
Join me as we unravel the intricacies of data integration, helping you transform raw data into a format primed for actionable insights. Whether you’re a seasoned data architect or just beginning your data journey, the “ETL Excellence” series equips you with the knowledge and tools to conquer this vital aspect of the data puzzle.
One of the newer features discussed this week on the Fabric Community page was the feature of Semantic Model Refresh Logic directly integrated into the Microsoft Fabric Data Factory experience. In this blog, I will talk through how this functionality used to be done in Synapse and Data Factory and demonstrate how easy MSFT has made this type of solution in …
Fabric Data Factory Spotlight: Semantic model refresh activity Read More »
This week I accidentally passed a bad update to a Delta Table that totally corrupted my entire table. In most cases, this is a problem that will need to be solved by reloading the table, merging correct data, etc. I thought this would be a great opportunity to showcase one of the benefits of building …
Time Travel Utilizing Delta Lake in Microsoft’s Spark Version Read More »
Case Study: How I Went About Optimizing My On-Prem to Cloud Copy Performance Explaining the Use Case Not too long ago, I built out a solution where the initial component was all centered around data transfer from an On-premises SQL DB and landing that data into ADLS for further ETL processing before landing in my …
Case Study: How I Went About Optimizing My On-Prem to Cloud Copy Performance Read More »
PLEASE Replace your Nested Notebook in a ForEach Activity Loop with A Looped Notebook When Leveraging Spark For those of you who are building repeatable processing in Azure Data Factory or Synapse Analytics Workspace Pipelines, you are probably familiar with the concept of the ForEach Activity Loop. This is a powerful tool to create scalable …
