How to extract metadata with AI Builder?
What is AI Builder?
During the last years, especially the last year, there has been a great increase in interest in process automation and artificial intelligence.
Thanks to the Power Platform, Microsoft has given us several tools, such as Power Automate and Power Automate Desktop to help us define automated flows that allow us to eliminate those repetitive processes that make us lose productivity from our day-to-day.
Most of the daily processes are related to the data extraction, from different types of documents, to impute them in different systems.
This is where AI Builder comes into play, a service within the Power Platform that provides us with different artificial intelligence functionalities, not only related to the extraction of data from a document but also with, for example, object recognition functionalities in images, detection of positive or negative opinions about a text or detect the language of a text and automatically translate it.
But in this article, we’ll focus on the functionalities that work on documents.
What predefined models can we find in AI Builder?
For data extraction from documents, there are some predefined models. But what are these predefined models? Which are? We are going to see each of them and try to define them:
Extract Invoice Information
This functionality allows us to send you any invoice and extract the following information:
- Customer address
- Customer Address Recipient
- Client id
- Customer name
- Due date
- Invoice date
- Invoice id
- Subtotal
- Full tax
- Provider address
- Provider Address Recipient
- Provider’s name
- Line elements
Extract all text in photos and PDF documents
This functionality allows us (as the title says) to extract all the text from an image or a PDF document.
You can see it more clearly in the example image:
Extract information from receipts
This functionality allows us to send you any receipt and extract the following information:
- Merchant Name
- Merchant Address
- Merchant phone number
- Transaction date
- Transaction time
- Purchased items
- Subtotal
- Taxes
- Total
Extract information from identity documents
This functionality allows us to pass you an image of any identity document and extract the following information:
- Surname
- First name
- Country
- Birthdate
- Identity document expiration date
- ID number
- Machine-readable area
- Nationality
- Gender
Extract information from business cards
This functionality allows us to pass an image of any business card and extract the following information:
- Full name
- Surnames
- First name
- Position
- Full address
- Company name
- Work phone
- Mobile phone
- Fax
- Address street
- Address city
- Address zip code
- Address country/region
Custom Models
In the previous point, we have seen that there are a series of predefined models that obtain certain information about different types of documents, but there are two situations that usually arise when we want to use these models.
The first one is the poor quality of the responses for some specific documents, there are times when, for example, we realize that some documents with a certain structure do not return the expected data or directly return empty data.
Another usual case is that we want to obtain some data other than what these predefined models return to us, for example, in a taxi receipt, the KMs of the journey.
In these cases, AI Builder gives us the possibility to define our custom model, we are going to take a tour of the creation of these:
How to create Custom Models?
The first thing we will do is define the fields that we will want to obtain from the documents, these fields can be of 3 different types:
- Field: A text
- Box
- Table with their respective columns
On the next page, we can add collections, it is possible that, if we are making an invoice model, we have invoices with different structures, in this case, we can create different collections to train next.
In the image below we can see that a collection has been generated for two different types of invoices:
In the next step we are going to use the documents uploaded to the collections to mark the different data within each of the documents:
Finally, we’ll train the model so that it can be used with the AI Builder connectors.
Once trained we will see the summary of our model with some confidence metrics.
In the “Quick test” button we can do tests, passing a document from our device and seeing the results that it returns.
You can learn more details about how to create custom document processing models here.
How to integrate AI Builder with Power Automate?
Now that we’re familiar with both the predefined and custom models, let’s see how we can use each within a Power Automate flow.
When adding an action in a Power Automate flow we will find the following set of AI Builder actions:
Let’s see, for example, the action “Extract information from invoices” (Predefined model), in it we will have to pass the content of a document and the range of pages that we want to pass from the document. If we leave it empty, by default, it will send all.
Let’s note that we can obtain the content of the file in different ways, if it is a document that we have saved in the cloud, either in a Sharepoint Online library or in Onedrive, we already have different actions that extract the content of a specific file :
In case of knowing the file extension and the Base64 content of a file, we could also generate a variable that the AI Builder accepts with the Compose action, generating a Json with the content type of the file and the base64 content.
Let’s see below the result of executing the AI Builder action:
We see all the data that they take from us and it also gives us a confidence value in the result obtained, thus allowing us to apply a logic of acceptance of the results based on this value.
Now let’s see how to use a custom model.
There is another action called “Extract information from documents”, where we can pass the custom Model that we want to use, the file extension, and the content of the file, in addition, we can indicate which pages we want to process:
As we can see, the results are returned in the same way.
What conclusions can we draw from the use of AI Builder and its integration with Power Automate?
With these new functionalities that AI Builder provides us, we can integrate with Power Automate to automate processes, for example, the receipt of certain documents, extract their data using AI Builder and impute them in any type of data source, thus saving time and performing repetitive tasks to different users, who could use that time in tasks that add greater value to our company.