Data Sharing in Cloud Platforms is a very big topic, Azure Data Share is a more advanced Azure service for sharing data especially between different organizations which you can read more about here but my blog post is about much more narrow user case where Data Producer (or data asset owner) and Data Consumer are part of the same organization and all the Azure Subscriptions are in the same Azure AD Tenant.

What you will learn?

I have observed this pattern emerge among large enterprises where VNET protected data assets are owned by one team and stored in one Azure Subscription (I will refer…


The purpose of this article is to share some practical guidance (or best practices, usually I refrain from using the term best practices because everything is relatively new) around Azure AD Authorization for users to access data stored in Azure Storage Account when using Azure Storage Explorer.

Objective

Azure Storage Explorer is an easy to use GUI tool for working with Azure Storage data, behind the scenes it uses AzCopy for all data transfer operations. In the enterprise environments with strict Access Control and Audit requirements the default easy to use options of Azure Storage Explorer might require more permissions than…


What is covered in this blog post?

Recently I went through a Cost Optimization exercise of evaluating Azure Synapse Reserved instance pricing with my customer and this blog post documents the learnings from that exercise as I think it would be beneficial for others as well. The blog post covers following main points:

  1. Shares additional perspective with examples on cost optimization for Azure Synapse using Reserved Instance pricing when you plan to run Azure Synapse Instance at variable DWU Levels.
  2. Shares the example Excel spreadsheet which can be used to play around with the usage patterns (variable DWU Levels) and cost estimates (Please refer to Pricing page…

Objective

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. When getting started with Azure Databricks I have observed a little bit of struggle grasping some of the concepts around capability matrix, associated pricing and how they translate to implementation. The objective of my post here is to provide a clear picture in your mind so that you understand the details and make good choices for your workloads.

What you will learn?

You will learn about the following concepts and how they relate to each other, I like to think of them as different…


The purpose of this blog post is to describe how to avoid a common performance pitfall when using Azure Data Factory Mapping Data Flows. The same information exists in the public documentation (Data Flows Performance and Tuning Guide) but can be missed in the ocean of documentation so my goal here is to give the exact instructions in a succinct and easy to consumer manner. Also, I would like to attribute my colleague Roshan Budathoki who shared this nugget of wisdom to help with a customer of mine which subsequently has resulted into this post for benefit of others.

Problem Statement

Azure…


Last week a few questions from a customer gave me an opportunity to delve into Azure Active Directory (Azure AD) Authentication for Azure MySQL Database. I have some interesting learnings to share which I believe would be useful for others as well hence I decided to document them in this article.

Scope and Target Audience

I expect this article to be beneficial to IT Operations who are setting up Azure MySQL, Developers (including DBAs) who will be using Azure MySQL as well IT Security Team members who review the Security aspects. Following are the two things you will learn:

  1. Some intricate details on how…

Polybase and Copy Command are two most prominent methods for performing high throughput loads from Azure Storage to Azure Synapse. In case you are new to these two methods please review at least couple sections from my previous post Practical Tips on Polybase data load to Azure Synapse. In this post I will share detailed instructions for loading data from a VNET protected (VNET Service Endpoint as well as Private Endpoint) Azure Storage account to Azure Synapse using Polybase and Copy Command. The information already exists in the Azure documentation but somewhat spread around and my goal was to consolidate…


Azure Platform has rich RBAC (role based access control) capabilities for very fine grained controls around authorization. There are quite a few built-in roles available but in cases where built-in roles don’t meet the needs of the organization custom roles can also be created. In case of Azure Data Factory (ADF), only built-in role available is Azure Data Factory Contributor which allows users to create and manage data factories as well as any child resources within them. In Production environments, most likely you would want more restricted role where a set of users are only allowed to execute and monitor…


The purpose of this post is to share a reference architecture as well as provisioning scripts for an entire HDInsight Spark environment. There are quite a few samples which show provisioning of individual components for an HDInsight environment but my goal was to showcase how to bring up an entire environment with VNET, Storage Account, external metastore using Azure SQL DB, etc. …


*******************

Important Note Jan 29, 2021: I have been informed that this method of using Managed Identity to interact with network protected storage account for Bulk Insert is no longer supported and can result in errors, the following official documentation calls out explicitly that this is not supported — https://docs.microsoft.com/en-us/azure/azure-sql/database/doc-changes-updates-release-notes?tabs=single-database#bulk-insert-and-backuprestore-statements-cannot-use-managed-identity-to-access-azure-storage

*******************

In this post I will share detailed instructions on how to load data from Firewall Protected Azure Storage to Azure SQL Database using Bulk Insert T-SQL command. …

Inderjit Rana

Senior Cloud Architect @Microsoft. Please feel free to connect with me on LinkedIn: https://www.linkedin.com/in/singhinderjit/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store