In the realm of data engineering and analytics, choosing the right tools can make or break your project. Azure offers two powerful options: Databricks and Azure Data Factory (ADF). While both are incredibly robust, they serve different purposes and are suited to different scenarios. In this blog, I’ll dive deep into when to use Databricks… Continue reading Navigating the Azure Data Ecosystem: When to Use Databricks vs. Azure Data Factory
Problem Statement A growing enterprise needed to streamline its email data management. Email activity data, stored in the organization’s CRM system, was essential for compliance, customer service tracking, and business intelligence. However, they faced major challenges: Email data was only accessible through complex CRM APIs returning paginated JSON. There was no centralized repository for email… Continue reading Email Data Migration from CRM to SQL Server using Azure Data Factory
In today’s dynamic cloud environment, achieving a consistent and reliable IP address for your Azure Data Factory (ADF) is crucial for secure and seamless data integration processes. Utilizing a Self-Hosted Integration Runtime (SHIR) and an Azure Virtual Machine (VM) provides an effective solution to this requirement. This blog details the step-by-step process to achieve a… Continue reading How to Get a Static IP Address for Azure Data Factory Using Self-Hosted Integration Runtime and Azure Virtual Machine
Requirement: While migrating tables from one database to another, there is a possibility of invalid data being inserted into columns that are not supported by their respective column data types. Consequently, errors may arise on the source side when attempting to migrate such data to another database. The objective is to migrate tables from Oracle… Continue reading Dynamically change the data type of Oracle Db and Migrate the data from Oracle to SQL using ADF
In Azure Data Factory (ADF), certain pipelines need to run at a specific time, but standard scheduling mechanisms like triggers or on-completion dependencies may not be suitable due to dependencies on upstream processes. A common challenge is ensuring that a pipeline starts at or after a predefined time while avoiding issues like premature execution, data… Continue reading Delaying Pipeline Execution Until a Specific Time in Fabric Data Pipeline by Implementing a Dynamic Wait Mechanism
Requirements: If the file size of an Excel file exceeds the specified criteria, we need to address the following: The Excel file contains multiple sheets, and these sheets contain inter-related data. We utilize SQL Views to retrieve data from these sheets using the “Get Data” activity in Excel. Although we can split the Excel file… Continue reading Split the Text files using ADF Pipelines
As Azure Data Factory (ADF) does not offer Excel as a sink option, an alternative method is required to copy Excel files using ADF pipelines. Implementation Step 1: Utilize the “Get Metadata” activity to retrieve the list of files stored in the blob storage. Select the “Child Items” option from the field list to include… Continue reading Copying Excel files from source to destination in Excel format using ADF Pipeline
The objective is to copy various types of files, including JSON, CSV, Text, and Excel files, using a single Copy data activity in Azure Data Factory (ADF) pipeline. Implementation: Step 1: Utilize the “Get Metadata” activity to retrieve the list of files stored in the blob storage. Select the “Child Items” option from the field… Continue reading Dynamically copying all types of files, including Excels, JSON, CSV, and TXT, using a single copy data activity in ADF
In one organization, there is one analysis of employee’s performance on different technologies based on the technologies they know and ratings from their manager. If they don’t meet the required criteria, then that employee will get training for those technologies. Input: emp_id MSBI PowerBI ratings from the manager 101 null x 3 101 null null… Continue reading Implementing ADF Branches and Filters for Employee Training
© All Rights Reserved. Inkey IT Solutions Pvt. Ltd. 2026
