Knowledge Mining using Azure Form Recognizer and Power Automate (formerly known as Microsoft Flow)– Part I

Introduction
Azure Form Recognizer is a cognitive service that uses machine learning technology to identify, and extract key-value pairs and table data from form documents. Cognitive Services are a comprehensive family of AI (Artificial Intelligence) services and cognitive APIs that help you build intelligent apps. They bring AI within the reach of every developer – without requiring machine-learning expertise. All it takes is an API call to embed the ability to see, hear, speak, search, understand, and accelerate decision-making into your apps. Azure Form Recognizer is part of the Cognitive Services Family.

Form Recognizer maps out an entire document, extracts text that is possibly a label, along with its corresponding value(s), and puts them together as key-value with the label being the key. The same is repeated across the entire document until all the targeted text is extracted. Text in table is extracted and returned in almost the same fashion, but with text in leading row’s cells as labels, and subsequent rows’ cells as value.

Form recognizer employs the aspect of models, which are services that learn and understand the layout and field data of documents prior to extraction. This learning can be either supervised, with manually labeled data, or unsupervised, without manual data labeling. Models enable Form Recognizer to make intelligent extractions, and those trained with labeled data can perform better and work with more complicated documents.

They comprise of:-

• Custom models – trained with your own data, so they’re tailored to your forms
• Prebuilt receipt model – a prebuilt model meant to extract data based on USA sales receipts
• Layout API – extracts text and table structures, along with their bounding box coordinates, from documents

In this write-up we are going to use Form Recognizer and Microsoft Power Automate to extract data from image or pdf files and update the file’s metadata in its document library in SharePoint.

This is a 3-part blog post series which consist of the following:

1. Part I – Introduction & Form Recognizer Setup
2. Part II – Training your Form Recognizer Model and setting up a Document Library in SharePoint
3. Part III— Power Automate and then testing out the entire set up.

Getting started

Step 1: Form recognizer resource set up

Before setting up this resource, ensure that you have an Azure subscription, and if not, create a free account before you begin. Then go to the Azure Portal, search for Form Recognizer, and create one.

When your Form Recognizer resource finishes deploying, find and select it from the All resources list in the portal. Then select the Keys and Endpoint tab to view your subscription data, as indicated in the image below.

Save the values of Endpoint and Key1 to a temporary location. You will use them in the following steps.

Join me next week for Part II of this blog post series where we will be looking at how you train a Form Recognizer Model and setup the Document Library in SharePoint.