Other

Audit logs in Office 365 navigation guide

Office 365 is a huge ecosystem and audit logs give you insight into your environment. In this blog, find out how to navigate Office 365 audit logs.

Audit logs are your go-to resource if you’re interested in what’s going on in your environment, be that managing Office 365 Groups, or what’s going on with your various resources on Azure, or even who is accessing and moving files around.

In some cases, because of regulations, you might need to keep a record of the various actions that are performed on the environment. Since Office 365 has a fairly rigid data retention period for audit logs, you might even need to handle and store them using a custom solution.

We’ve already talked about audit logs on SharePoint On-premises in the SharePoint Audit Logs: A Key to Better SharePoint Management blog. The reasons why you might want to keep the audit logs at hand hasn’t changed that much in the move from SharePoint On-premises to Office 365.

If we call SharePoint On-premises a complex platform, Office 365 is a huge ecosystem containing multiple interconnected services (Exchange, Teams, Azure, and OneDrive, just to name the more popular ones). Any solution that tracks Office 365 activity for the purposes of auditing must include activity made on the services that make up the platform.

Now, what options do we have when we need to do Office 365 auditing? A few, and I’ll give an overview of SharePoint’s audit log reports and Office 365 audit logs for admins

SharePoint Online audit log reports

If you’ve worked with SharePoint On-premises and have seen its audit logs, you’ll notice a lot has stayed the same in its move to the cloud. These logs can answer most of the questions you might have regarding your SharePoint sites. The SharePoint Online audit log reports will give you information about: 

  • Edited items 
  • Checked-out and checked-in items 
  • Items that have been moved and copied to other locations in the site collection 
  • Deleted and restored items 
  • Changes to content types and columns 
  • Search queries 
  • Changes to user accounts and permissions 
  • Changed audit settings and deleted audit log events 
  • Workflow events 
  • Custom events 

There is no longer an option to record opened, downloaded, and viewed items in the cloud version. I would guess that was changed in order to reduce the resource cost on Microsoft’s side since those events are the most frequently occurring and usually the least critical. But I can’t find any official article on this omission.

How to turn on audit logs in Office 365

To enable Office 365 auditing, SharePoint audit logging needs to be set up for each site collection separately, but you can automate it with a simple PowerShell script and a list of your site collections.  

SharePoint Online’s audit logs have a few constraints. The minor one is that you cannot access the raw audit log data programmatically. Unlike on-premises, there is no endpoint you can connect to if you want to access the collected records to create custom reports.

You can however choose from a selection of Excel reports that should satisfy most of your needs, but you’ll have to generate reports for each site collection separately. This also means that if you want to process the data further, you’re stuck with downloading and parsing Excel documents.

The bigger constraint relates to what is not collected: everything outside of SharePoint is ignored. In an environment such as Office 365, this means many actions, such as any performed in Azure Active Directory or Exchange for instance, will not be visible here. 

Note that these logs have a maximum data retention period of 90 days, which is something Syskit Point can help you with as it can store audit logs for longer periods.

Office 365 audit logs

Office 365 Audit logs capture details about system configuration changes and access events, with details to identify who was responsible for the activity, when and where the activity took place, and what the outcome of the activity was.

Where are Office 365 audit logs stored?

You can find Office 365 audit logs in the Microsoft Purview Compliance Center. While other logs are limited in scope to a particular service, these are collected from multiple Office 365 services and consolidated into a single, searchable log (and they catch page and file views). 

How do I know if audit logs are enabled?

To start using this log, you first have to turn it on by navigating to the Audit log search and then clicking the Turn on auditing button. You’ll need to do this only once per tenant. But, you will have to wait a few hours for the service to prepare itself for use.

Once the preparation on the Office side is complete, audited events will start appearing in the log. Depending on the particular service, it can take up to 30 minutes or even up to 24 hours for an event to be shown in the audit log. Here’s the full list of audited services and actions.

How long are audit logs kept?

Logs are kept for 90 or 365 days, depending on the license. To enable the full year, you’ll need to have an Office 365 E5 subscription or an Office 365 Advanced Compliance add-on license with an E3/Exchange Online Plan 1, and you’ll need to send a request to Microsoft support to enroll in the program. 

Lastly, you can export the results of a search into a csv file to store and analyze further. The format is not pretty and will only be easy to read if you know exactly what you’re looking for, but it’s easy to use if you want to process the data further.

The main issue you’ll come across is that most of the properties are lumped into a simple AuditData JSON, which can contain very different properties depending on the audit event, and a lot of noise to parse through to get to the important stuff. 

There’s also the option of programmatically downloading the audit logs for use in your solutions. This way, you can regularly download the newly available events using the available API’s, using either PowerShell or REST.

Accessing audit logs through Exchange management tools

This method is easier to use since you’ll only need to have PowerShell installed and have either the Audit Logs or View-Only Audit Logs role assigned in Exchange (global admins should have this by default).

To start an Exchange Online PowerShell remote session, this is the script you’ll want to run:

$userCredential = Get-Credential  $connectionUri = 'https://outlook.office365.com/powershell-liveid/' $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri $connectionUri -Authentication Basic -Credential $userCredential -AllowRedirection Import-PSSession $session -DisableNameChecking

This will establish a connection to Exchange Online and allow you to run cmdlets associated with Exchange. After you’re done with the session, it’s best practice to close it using the following:

Remove-PSSession $session

If you’re having trouble with the execution policy on your machine, you will need to first run:

Set-ExecutionPolicy RemoteSigned

The only Exchange cmdlet we’re interested in right now is Search-UnifiedAuditLog. It has a simple basic signature but allows for a lot of customization (find out more here). 

Search-UnifiedAuditLog -StartDate 04/24/2019 -EndDate 04/25/2019

This command will return audit records for the specified date, up to a maximum of 100. You can specify a larger number of records to return, or you can use paging by specifying a SessionId and using the ReturnNextPreviewPage SessionCommand. 

The returned data contains the general data about the event (RecordType, CreationData, UserIds, and Operations) and the AuditData JSON, which contains the event description and whose properties depend on the specific event type.

All the data contained in the Security & Compliance Search can be queried from here, which effectively means they share the same data retention period. 

You can find more info about connecting to Exchange Online by remote PowerShell here

Of the two possible APIs to use to collect Office 365 audit logs, this one is the less recommended one.

Office 365 management API

This API provides access to events from Office 365 audit logs. The data available here is more or less the same as shown in the search log, with very few differences. The major one being you can only get data for the last 7 days, while others make the data available for up to 90 days. 

The audit data is made available using blobs – the service periodically collects events and organizes them into these collections (or blobs), which can then be downloaded. The events contained in the blobs are not in any particular order but are bundled up according to the time when the service collected them.

This means that an older event might appear in a very recent blob (according to Microsoft, in case of a service outage, this might mean a delay of 5 days or more). What this means is that you cannot use this service for alerts since there is no guarantee an event will be made available in a timely manner.

How to start collecting data?

To start collecting this data, you will need to create a new application on your Azure first. Then, using that application, we’ll start subscriptions for the various event content types available. The application requires the ActivityFeed.Read and ServiceHealth.Read application and delegated permissions, and you’ll need to generate a client secret to authenticate your application.

API permissions for access to events from Office 365 audit logs
Office management API client secret

We’ll use the client secret in the next steps, so make sure to store it somewhere. 

The following examples will be written in PowerShell, but they can be easily rewritten in almost any other programming language. 

Using the following script and the values of your clientId, clientSecret and tenantDomain, we’ll get an access token that we can use in subsequent calls to the API.

$clientID = "00000000-0000-0000-0000-000000000000" 
$clientSecret = "]t;{K!.;andsomemoregibberish" 
$tenant = "contoso" 
$tenantdomain = "$tenant.onmicrosoft.com" 
$loginURL = "https://login.microsoftonline.com/" 
$resource = "https://manage.office.com" 
$body = @{grant_type="client_credentials";resource=$resource;client_id=$clientID;client_secret=$clientSecret} 
$oauth = Invoke-RestMethod -Method Post -Uri $loginURL/$tenantdomain/oauth2/token?api-version=1.0 -Body $body 
$headerParams = @{'Authorization'="$($oauth.token_type) $($oauth.access_token)"}

The headerParams attribute now contains the access token. Using this token, we’ll be able to start the subscriptions to each of the audit types (Audit.AzureActiveDirectory, Audit.Exchange, Audit.SharePoint, Audit.General, and DLP.All). To start a subscription, run this command:

Invoke-WebRequest -Method Post -Headers $headerParams -Uri "https://manage.office.com/api/v1.0/$tenant/activity/feed/subscriptions/start?contentType=Audit.AzureActiveDirectory"  

To start a subscription for the other audit types, simply replace the contentType parameter value with the names of the other audit types. 

You can check the activated subscriptions with the following: 

Invoke-WebRequest -Headers $headerParams -Uri "https://manage.office.com/api/v1.0/$tenant/activity/feed/subscriptions/list"

Assuming the scripts ran successfully, you should now be able to query for audit blobs and download their contents like so:

Invoke-WebRequest -Method GET -Headers $headerParams -Uri "https://manage.office.com/api/v1.0/$tenant/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory"

This request will return a JSON object containing content uris for events that have been made available today. You can choose a different time period by specifying a startTime and endTime parameter (both need to be specified and must not span more than 24 hours). 

Invoke-WebRequest -Method GET -Headers $headerParams -Uri "https://manage.office.com/api/v1.0/$tenant/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2019-04-25T00:00&endTime=2019-04-25T11:59"

In any case, the returned JSON object should contain uris we can use to collect events collected in the time period (remember, we don’t have a guarantee the event actually occurred in that time period, only that it was collected then). An example of such a request and uri would look like this:

Invoke-WebRequest -Method GET -Headers $headerParams -Uri 'https://manage.office.com/api/v1.0/00000000-0000-0000-0000-000000000000/activity/feed/audit/20190425071547312049798$20190425071747444061084$audit_azureactivedirectory$Audit_AzureActiveDirectory$emea0035'

This will return a collection of events in a JSON format. The events have schemas depending on the audit event type they’re describing. Microsoft has defined about two dozen different schemas (a couple more or less, depending on how you count them since a few are considered base and aren’t used independently of others). Each event will have CreationTime, RecordType, Operation, and ResultStatus properties.

According to its type, it will also have several other properties used to describe the details of the action that created the record (these extra details can be found by examining a single row in the Audit log search and expanding the More information tab). This can be anything from the user that initiated the action, where the action was made, or what the target was.

How to understand these properties?

Some of these properties are easier to make sense of than others since connecting an email to a particular user is straightforward. But it’s much less so to connect a group id to a group name or a team site id to an Office 365 Group. Microsoft has documented the schemas and their properties, but note that some record types are not officially documented. 

You can find out more about the Office 365 Management API by checking out the official resources

If we compare the PowerShell API to the REST one, we can see that the Search-UnifiedAuditLog cmdlet, with its numerous optional parameters is more suited for searching and filtering logs in search of a specific record, but not so much for collecting new records as they’re coming in since a record’s creation time is not the same time the record was made available. The REST API is much more appropriate for collecting and storing audit records. Since the collection of events in the blobs does not change, you only need to collect new blobs as they become available.

Where Syskit Point comes in

We’ve seen that finding what you’re looking for is rarely easy. Especially if you just want an overview of what’s happening with a given user or site. The default logs are verbose, and because they are generic, they tend not to focus on what is important in a given context. The filters are nice, but you need to know the exact URL of an object beforehand to view the actions associated with it.

The data retention is stingy, although understandably so since Microsoft needs to store a lot of data. And every extra day of data retention they allow means an enormous amount of extra storage space required on their end. For this and other reasons, we’ve made our own custom solution for storing, searching, and exporting Office 365 audit logs. It stores data locally, meaning you can decide how much of it you want to store and for how long.

It’s easily accessible from most reports, allowing you to easily check the audit logs related to a certain SharePoint site and the items in its document libraries, or you can easily check the list of actions made by or on a certain user. We’ve tried to make it as easy as possible to read so you can see what’s important at a glance.

Subscribe to our Newsletter

Related Posts