Report Copilot user prompts and AI responses with PowerShell

Microsoft 365 Copilot is an advanced AI-powered assistant designed to enhance productivity and streamline workflows within the Microsoft 365 suite. By leveraging natural language processing and machine learning, Copilot assists users in tasks such as drafting emails, summarizing documents, and generating creative content. However, organizations often face challenges in gaining visibility over their staff’s interactions with Copilot. This lack of transparency can hinder the ability to monitor usage patterns, ensure compliance, and optimize the deployment of AI tools across the workforce. 

Recently Microsoft has released a new API endpoint for Microsoft Graph which enables developers to gain insights into their organisation’s users on their interactions with the Microsoft 365 Copilot service.

In this article post, we will explore how PowerShell can be utilised to gain insights into these interactions, providing organisations with the necessary tools to manage and oversee their use of Microsoft 365 Copilot effectively. Please note, these API’s are currently in preview and responses may not be completely accurate.

Requirements

To use the Graph API mentioned in this article with PowerShell, you must first register your own application in Microsoft Entra and grant it the AiEnterpriseInteraction.Read.All permission as an Administrator in the application context. This is because this API is currently in preview and the delegated context is not yet supported. 

You must also have the latest version of Microsoft Graph PowerShell installed.

For assistance with both of these steps, I have written articles which will help you install the Microsoft Graph PowerShell SDK and register your own application to connect to Microsoft in the app-only context.

View all Copilot interactions per user

All interactions with the Microsoft 365 Copilot service can be viewed using Microsoft Graph PowerShell on a per-user basis only. To do this, start by modifying the highlighted text below and replace {id} with either the UserPrincipalName or the Object ID of the user for whom you wish to read their Copilot interactions.

/beta/copilot/users/{id}/interactionHistory/getAllEnterpriseInteractions

Then take the example code below and modify the value of the $uri variable with your modified code from above. You can then copy and paste this into your PowerShell console to list all interactions the target user has made with the Microsoft 365 Copilot service.

$uri = "/beta/copilot/users/{id}/interactionHistory/getAllEnterpriseInteractions"

Invoke-MgGraphRequest -Method GET -uri $uri -OutputType PSObject | Select -ExpandProperty Value

The response will contain various information about each item in the output. A detailed list of all the information contained within each item in the response can be found below.

id : #the unique ID of the message
sessionId : #The thread ID or conversation identifier that maps to all Copilot sessions for the user
requestId : #The ID of the request, this will be the same for the matching UserPrompt and aiResponsee
appClass : #The data source for the Copilot data
interactionType : #Indicates whether the interaction is a prompt or a Copilot response
conversationType : #The type of conversation, either appchat or bizchat
etag : #The timestamp of when the interaction was last modified
createdDateTime : #The time when the interaction was created
locale : #Location of the sender
contexts : #The identifer that maps to all contexts associated with an interaction
from : #The user, application, or device that is associated with this interaction
body : # The body of the message, including the text of the body and its body type
attachments : #The collection of documents attached to the interaction, such as cards and images
links : #The collection of links that appear in the interaction
mentions : #The collection of the entities that were mentioned in the interaction, including users, bots, and so on

To make the output more human-readable, I recommend limiting your view to only the necessary properties you may want to read and then displaying them in an interactive table using the built-in PowerShell grid view feature. This approach can be applied to all the further examples in this article.

$uri = "/beta/copilot/users/{id}/interactionHistory/getAllEnterpriseInteractions"

Invoke-MgGraphRequest -Method GET -uri $uri -OutputType PSObject | Select -ExpandProperty Value `
| Select interactionType, requestId, appClass, conversationType, Locale, From, Body, Attachments `
| Sort -Descending createdDateTime `
| Out-GridView

Filter Copilot interactions by application

It can be expected that for most users with a Copilot license, a lot of interaction data may be present in their history. Because of this, you may want to filter your results to a specific application that your target user has interacted with. With the Microsoft Graph API, you can use filters to limit your results to interactions with a certain Microsoft 365 application, such as Microsoft PowerPoint or Microsoft Excel. Below is an example filter which limits results to Copilot interactions with Microsoft PowerPoint.

/beta/copilot/users/{ID}/interactionHistory/getAllEnterpriseInteractions?`$filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.PowerPoint'

Use the full example below to export the interaction data in a more manageable format.

$uri = "/beta/copilot/users/{id}/interactionHistory/getAllEnterpriseInteractions?`$filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.PowerPoint'"
Invoke-MgGraphRequest -Method GET -uri $uri -OutputType PSObject | Select -ExpandProperty Value `
| Select interactionType, requestId, appClass, conversationType, Locale, From, Body, Attachments `
| Sort -Descending createdDateTime `
| Out-GridView

Your output should look like the following. Some columns must be expanded to read the full content of both the user prompt and the AI response.

Copilot response script output
Copilot response script output

A full list of supported applications you filter by with Microsoft Graph PowerShell can be found in the table below.

Application Name Filter
Excel $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Excel'
Loop $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Loop'
Microsoft 365 App $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.M365App'
Microsoft Copilot for Bing (Bizchat) $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.BizChat'
Microsoft Forms $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Forms'
OneNote  $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.OneNote'
Outlook  $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Outlook'
PowerPoint  $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Powerpoint'
Teams AI notes in Chat $filter=appClass eq 'IPM.SkypeTeams.Message.TeamCopilot.AiNotes.Teams'
Teams Channel $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Teams'
Teams Chat $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Teams'
Teams Copilot Chat (Bizchat) $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.BizChat'
Teams Meeting $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Teams'
Teams Microsoft 365 Chat (BF) $filter=appClass eq 'IPM.SkypeTeams.Message'
WebChat $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.WebChat'
Whiteboard $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Whiteboard'
Word  $filter=appClass eq 'IPM.SkypeTeams.Message.Copilot.Word'

Filter Copilot interactions by date range

Currently, when you make a request to this Microsoft Graph API to view all user Copilot interaction data, data will be returned from the point the user was assigned a Microost 365 Copilot license. In most cases, this will return an unmanageable amount of data and it may also take a long time for you to retrieve the results. To avoid these issues, you can filter your results by a specific date range, this will reduce the amount of data retrieved and the amount of time it takes to retrieve the data. The below example, will filter results between the dates 01-12-2024 and 03-12-2024.

/beta/copilot/users/{ID}/interactionHistory/getAllEnterpriseInteractions?`$filter=createdDateTime gt 2024-12-01T00:00:00Z and createdDateTime lt 2024-12-03T00:00:00Z

Us the full example below to view the results in an interactive table.

$uri = "/beta/copilot/users/[email protected]/interactionHistory/getAllEnterpriseInteractions?`$filter=createdDateTime gt 2024-12-01T00:00:00Z and createdDateTime lt 2024-12-03T00:00:00Z"
Invoke-MgGraphRequest -Method GET -uri $uri -OutputType PSObject | Select -ExpandProperty Value `
| Select interactionType, requestId, appClass, conversationType, Locale, From, Body, Attachments `
| Sort -Descending createdDateTime `
| Out-GridView

Will Microsoft charge for using this API?

Based on the loose information in the Microsoft documentation, the query parameters offered as part of the filtering capabilities indicate these this API will eventually become chargeable to use.

Copilot interactions API pricing

Looking further into the ‘licensing and payment models’ documentation, the tables have not yet been updated to include this new API. This may have just been an oversight of the Microsoft team or they could be waiting for further stability of the API before subjecting it to any pricing.

Daniel Bradley

Daniel Bradley is a UK-based technical architect and Microsoft MVP in Security. He specialises in Microsoft Entra, Intune, and Microsoft Graph PowerShell and is the author of Microsoft Graph PowerShell for Administrators. He writes and edits Our Cloud Network using experience gained from working with Microsoft cloud environments.

Leave a Reply