Detecting credential-driven attacks on Databricks

At Databricks, we know your data is your most valuable asset. Databricks has worked with thousands of customers to securely deploy the Databricks Data + AI Platform with the appropriate features to meet their security, privacy, and regulatory requirements.

The modern cyberattack does not always begin with a shattered perimeter; it begins with a valid login. Once an adversary acquires a leaked session cookie, personal access token (PAT), or service-principal credential, the initial breach is already complete. From that moment on, their entire goal is to blend in and operate under the guise of your compromised user or service.

But having a key doesn't mean you know the building. Imagine an intruder swiping a badge to enter a secure office. They may have cleared the revolving doors, but they still have to navigate the security guard, guess which floors they have access to, and figure out where the vault is. Because an attacker lacks the deep, localized context of your actual employees, their behavior will eventually betray them. Spotting those irregularities is how you catch them.

In this post, we walk through how a credential-driven attack unfolds on the Databricks platform, how to detect the shift in an identity’s behavior using Databricks verbose audit logs and Genie, how to respond once you confirm a compromise, and which controls limit what a stolen credential can do in the first place.

A model for following credential-driven attacks

When establishing access via legitimate credentials, adversaries, like security practitioners, also understand their access may get revoked at any point when they deviate too greatly from the identity’s expected behavior. Adversaries rely on non-linear attack sequencing, such as validating one set of credentials to map your environment, then retrieving another set of credentials to return through a new identity or workspace that doesn’t have reconnaissance activity associated with it. Additionally, adversaries may gather information without establishing persistence or establish their way back into your Databricks platform before wrapping up their objectives.

What this looks like in practice

Consider a data engineer whose personal access token is copied from a misconfigured continuous integration and delivery (CI/CD) log. Within minutes, the token is used from a new IP to list secret scopes, catalogs, and repos across the workspace, something this engineer has never done. A secret holding a service-principal credential is read. An hour later, that service principal, not the engineer, creates a function, adds itself to an admin group, and exports a handful of init scripts. Each of these events could pass as routine platform use, but taken together they show one intrusion moving through the attack chain.

At Databricks, we expect non-linear event sequencing in relation to credential-driven attacks and use this model to connect individual events into one investigation rather than treating them as unrelated events.

The six stages of a credential-driven attack

When adversaries leverage compromised credentials in a Platform-as-a-Service (PaaS) environment, such as Databricks, their activity can be grouped into six stages, although they rarely move through them in a straight line. Each stage below describes what the adversary is trying to do and what to look for in Databricks audit logging tables. Treat this guidance as a starting point and tune it to your environment's normal automation.

Note: Events are written as service_name.action_name, for example secrets.getSecret.

1. The adversary gains access with valid credentials

The login itself rarely looks unusual. Because the credential is real, authentication succeeds as it would for the legitimate owner. However, what the identity does in the minutes after signing in often looks very different from its normal activity, and that change is what responders can detect. In the data engineer example, the stolen token authenticates successfully, and the only difference at login is a source IP the engineer has never used. The activity that follows is what stands out.

Adversary goal: Access the workspace with valid credentials, whether a stolen browser session, a replayed PAT, or a service-principal credential, and blend into activity the controls are designed to allow.

What to look for:

• Use the authentication event as the starting point for your timeline. A login alone is rarely enough to alert on, but its user_identity, source_ip_address, user_agent, sessionId, request_id, and workspace_id let you follow what the credential does next.
• Flag accounts.tokenLogin events that arrive in a short burst from an unfamiliar client and are followed by use of the same token from a different source IP. This pattern can indicate that someone is testing a stolen token, and requestParams.tokenId lets you tie those events to the same token.
• Flag service-principal OAuth tokens that are minted from the expected network with an unexpected user agent, especially when activity from a new source IP follows within a minute.

2. The adversary maps the environment

Next, the adversary needs to learn what the credential can reach before the access is revoked. As a result, a compromised identity can shift from its ordinary activity to listing data, secrets, code, and network controls within seconds. This reconnaissance usually spans several services at once and moves faster than a person navigating the UI. In our scenario, this stage is the burst of calls listing secret scopes, catalogs, and repos within minutes of the token being used, which is activity the engineer has never performed before.

Adversary goal: Build a map of accessible data, credentials, repositories, and guardrails to plan a path deeper into the environment before access is revoked.

What to look for:

• Look for a single identity making rapid list calls across the workspace and account planes, such as secrets.listScopes / listSecrets, unityCatalog.listMetastores / listCatalogs / listSchemas / listExternalLocations / listStorageCredentials, repos.listRepos, and account-level accountsManager.listWorkspaceConfigurations / listNetworkConfigurations / listStorageConfigurations. For example, five or more distinct discovery APIs called within one minute is unlikely to be a person navigating the platform.
• Compare activity to the identity's baseline. The first secret enumeration ever made by an established identity, or a SQL session that shifts from business queries to SHOW TABLES against access-control and configuration schemas, is more meaningful than a high volume of calls. When verbose audit logging is enabled, commandSubmit preserves the query in commandText.
• Include failed requests in your search. Investigate an identity's first IpAccessDenied, especially when it follows accounts.oidcTokenAuthorization or repeats across workspaces. In that case, authorization succeeded but the request came from a blocked network, which is often the earliest record of a credential being replayed from an untrusted location.

3. The adversary harvests more credentials

Harvesting credentials usually takes several steps. The adversary lists secret scopes, lists the secrets in them, and then reads the ones that look useful. The value they retrieve allows them to return as a different, often more privileged, identity. For this reason, every secret read by a suspect identity is worth following up on. In the data engineer example, the adversary reads a secret that holds a service-principal credential. From that point on, the investigation has to follow two identities: the engineer and the service principal.

Adversary goal: Retrieve a token, API key, or service-principal secret that upgrades the initial foothold into environment-wide access under a new identity.

What to look for:

• Alert on the progression of calls. secrets.listScopes on its own is common and generates noise. However, listScopes followed by listSecrets and then getSecret, repeated across multiple scopes in quick succession, indicates harvesting. listSecrets records the scope, and getSecret records both the scope and the key.
• Correlate denied reads with later successful ones. A denied getSecret on a tightly controlled scope, followed by a successful read of the same or a related key in a less restricted scope, suggests the identity is looking for a copy it can access.
• Monitor compute-side access in addition to the secrets API. When verbose audit logging is enabled, notebook submitCommand and Databricks SQL commandSubmit events can show code calling secret utilities or running read_files() against paths that contain credentials. Keep in mind that the submitted text shows what ran, but it does not confirm that a value was returned.
• Connect each secret read to what happens next, such as a new accounts.tokenLogin, first-seen service-principal activity, a changed IP or user agent, or generateTemporaryTableCredential / generateTemporaryVolumeCredential calls. During response, rotate every credential the identity could have reached, and check where each one was used in addition to where it was stored.

4. The adversary finds a workable execution path

Once access is established, adversaries look for a way to run their own code. On Databricks, this typically happens through user-defined functions (UDFs) and jobs. A function can wrap a payload in ordinary-looking code that can be called repeatedly, and a job can run that code on a schedule across different compute. As such, the key question for detection is whether an identity has started performing actions it has not performed before. Returning to the data engineer, an hour after the secret is read, the service principal creates a function. Because the service principal, rather than the engineer, performs this action, a search scoped only to the engineer's activity would miss it.

Adversary goal: Establish a reliable, reusable way to execute payloads, with a fallback when one compute route is blocked.

What to look for:

• Prioritize createFunction events from an identity, IP, or catalog with no history of creating functions, especially when several appear in a short period. Where function definitions are available, flag code that executes processes, evaluates code dynamically, accesses instance metadata, opens outbound network connections, or writes to cloud storage. Also trace references between functions, because a harmless-looking wrapper can call a higher-risk UDF. Note that function_info in system.access.audit is masked except for account admins and databricks_pii_access, so pair audit data with function inventories.
• Track the Jobs path by identity, IP, and job or run ID. A typical sequence is importNotebook, then changeJobAcl, Jobs create, runNow, and getRunOutput. These events rarely preserve the notebook body, so retain notebook versions and job configurations alongside them. Repeated import-and-run activity from one source is a stronger signal than either action alone.
• Look for an identity that switches compute after a failure. For example, a failed cluster creation or submitRun followed shortly by startWarehouse, or by a new job on different compute, may indicate an adversary working around a blocked path. The response codes and the short gap between events are what separate this sequence from routine errors. Serverless compute is a common fallback.

5. The adversary establishes persistence

With a way to run code, adversaries next try to make sure they can return after the original credential is revoked. Common persistence methods include access grants, newly created tokens, group and workspace membership changes, and control-plane edits. The most useful signal is a sensitive change made by an unfamiliar identity or execution context. For example, a permission change made from inside a job deserves more scrutiny than the same change made by an administrator in the console. Next in the scenario, the service principal adds itself to an admin group. Even if the engineer's token is revoked, that group membership keeps the adversary's access in place.

Adversary goal: Create durable, independent access paths that survive revocation of the compromised credential.

What to look for:

• Alert on permission and identity grants from unexpected contexts, such as putAcl on a secret scope, updatePermissionAssignment, addPrincipalToGroup, or setAdmin. Pay particular attention when a service principal grants another identity access, or when multiple assignments occur in a short period.
• Treat attempts to change security controls as high severity, even when they fail. This includes setSetting / deleteSetting, IP access-list edits, and createPrivateAccessSettings. Review the requested change, the response, and the source before concluding whether a control was actually bypassed. Also review SCIM group changes from anything other than the expected IdP.
• Flag generateDbToken calls from an unfamiliar source or during a job run. Each resulting token is a separate credential that will need to be revoked.
• Watch for workloads that continue on their own. Correlate control-plane events with run_by, run_as, and the expected run window. If a job keeps modifying groups, permissions, or settings after the adversary's interactive activity stops, it may be maintaining access by itself. Preserve the job definition and artifacts before terminating it.

6. The adversary collects data and cleans up access

Data collection can be as small as a few targeted exports, and cleanup can involve more than deleting files. Watch for targeted exports, temporary-credential requests, removal of short-lived tooling, and obfuscated commands that make the activity harder to reconstruct. Any one of these events can look routine, so the sequence around them matters more than the individual export or delete. Finally, the service principal from our scenario exports a handful of init scripts. On its own, that is a small export. However, because it follows a secret read and a new function from the same service principal, it should be investigated as part of the same intrusion.

Adversary goal: Take the data that matters and erase the most obvious tooling while leaving quieter access in place.

What to look for:

• Look at the context of each export rather than the number of exports. workspaceExport records the path, the format, and whether it was a direct download. Prioritize exports by unfamiliar identities, to unusual paths or in unusual formats, from previously unseen networks, or that follow secret access or new execution. A small, targeted export of init scripts can leak more than a bulk SOURCE dump.
• Keep records of short-lived objects. A createFunction followed by deleteFunction for the same function in one session, or a repeated create, delete, and recreate cycle, may indicate an attempt to remove evidence. Preserve the definitions and timing before reverting anything. Objects that survive are often given more plausible names than the ones that were deleted.
• Connect temporary-credential and data-read bursts (generateTemporaryTableCredential, generateTemporaryVolumeCredential, Delta Sharing, cloud-storage reads) to earlier reconnaissance. Encoded or scripted commands are too common to alert on by themselves, so correlate them with an unfamiliar user agent, automation-like timing, secret access, or new functions.

Investigating credential-driven attacks with Genie

Using the credential-driven attack chain as a guide, you can direct Genie to work through the attack stages without writing every query from scratch. Every detection in this post runs on telemetry you already have. Databricks writes control-plane activity to the system.access.audit system table.

To get started, install the Detection App, which adds a Genie agent for Databricks’ audit logging. You can work it from Genie One in two ways:

• Ask in your own words, like "Which notebooks did user@example.com export yesterday?" Genie writes and runs the SQL for you.
• Use the built-in detections. The Detection App Genie agent includes more than 30 detections, ranging from token misuse to admin grants. Ask "Who created personal access tokens last week?" and Genie runs the matching detection instead of writing new SQL. Give it a goal, like "Were we compromised in the last 90 days?", and it works through the detections one at a time.

Keep two limits in mind. First, an empty result doesn't mean nothing happened. The event may be outside your time window, or verbose audit logging may be disabled. Second, audit logs record that something changed and who changed it, but often not the old or new values.

From detection to response

Once a hunt surfaces an identity that doesn't fit its normal pattern, the focus shifts to what access the adversary still has. Because credential-driven attacks are non-linear, the first access you find is rarely the only access. In the data engineer example, revoking the engineer's token would leave the service principal's credential, its admin group membership, and the function it created in place. A complete response removes the original credential as well as every path the adversary created afterward, including grants, tokens, functions, and jobs that keep running after the user is disabled.

For a suspected workspace compromise, responders should:

1. Disable the identity. Deactivate the user at the account level, which blocks every workspace. There's no API to end a user's browser sessions, but deactivation blocks them. For a service principal, delete its OAuth secrets and deactivate it with the account SCIM API. OAuth tokens that the service principal already has stay valid for up to an hour.
2. Revoke their personal access tokens. Deactivating a user doesn't delete their tokens, and the tokens work again if the user is reactivated. Use the Token Management API or the token report in the account console.
3. Save what you need, then stop the compute. Export job definitions and notebook versions first. Then cancel job runs, terminate clusters, and stop SQL warehouses. On serverless, cancel the run or statement. Deactivating a user doesn't stop compute that's already running.
4. Rotate every secret the actor read. Change it at the source, then update the stored value. Rotate the credential the adversary used to get in, too.
5. Undo actor-created persistence. Remove secret scope ACLs, workspace assignments, and group memberships. Drop the functions and revoke the grants. Restore your context-based ingress policies (and IP access lists, if you still use them) from your own records.
6. Check code the identity could change outside Databricks, like init scripts and ETL code in cloud storage.
7. Confirm it stopped. Run a search on activity by the actor. If verbose audit logging was off, turn it on. Command text is only logged while it's on.

Preventing credential-driven attacks

To return to the office building: context-based ingress decides who gets through the revolving doors, and least privilege decides which floors their badge can open. Early detection buys response time, but preventive controls limit what a stolen credential can do. The controls below are grouped by what they take away from the adversary, in roughly the order they break their attack chain.

Keep stolen credentials out

The strongest preventive control against credential reuse is context-based ingress control. It evaluates who is making a request, where it comes from, and what it is for, instead of relying on a single network perimeter. This is especially useful as more organizations move away from VPNs. Even without a corporate network to anchor on, you can limit a stolen token to the contexts its owner actually uses. For example, you can restrict a privileged service principal to API requests from trusted networks while applying a different policy to human users.

• Apply policies to both workspace and account requests. Workspace policies don't cover the account console or account-level APIs.
• Start in dry-run mode. Review denials and dry-run violations in system.access.inbound_network, protect privileged service principals first, and enforce after you've accounted for legitimate automation and partner integrations.
• Test serverless and private-connectivity workflows before broad enforcement, because they follow separate policy rules. Document any exceptions and monitor their use.
• Treat IP access lists as an additional layer. Where both apply, a request must pass both. Configure workspace and account-console lists if you still use them.

Make harvested credentials worth less

• Set expiration and rotation requirements for service-principal credentials, integration credentials, and personal access tokens, and retire credentials that are no longer used. Dormant credentials are particularly attractive to adversaries because a compromise can go unnoticed for months.
• Enforce least privilege on secret scopes. Avoid storing copies of powerful credentials in scopes with different ACLs, and review who can list scope contents as well as who can retrieve values.

Limit what a credential can run or change

• Restrict high-impact operations. Limit UDF creation, secret-scope ACL changes, workspace permission changes, job administration, and ingress policy changes to approved identities and controlled automation.
• Separate administrative paths from compute paths. Ordinary job runtimes should not be able to modify account and network controls, and any attempt to do so should generate an alert.
• Protect cloud execution environments. Use hardened metadata-service configurations where applicable and restrict egress from compute so workloads cannot freely reach credential endpoints or unapproved destinations.
• Restrict which identities can manage account identities, assign workspace access, and create jobs.

Keep the evidence

• Preserve investigation-ready telemetry for potential security compromises.
• Retain audit logs, notebook and job version history, UDF definitions and dependencies, permission-change history, cloud-storage write events, and context-based-ingress denial logs long enough to reconstruct a campaign.

Remember, a stolen credential’s blast radius is only as damaging as the privileges assigned to it. Reduce that exposure and be able to hunt for anomalous activity by taking these steps:

• Review the Databricks security best practices to harden your workspaces and account.
• Turn on verbose audit logging and set up context-based ingress control so both planes are covered.
• Deploy the Detection App to start your hunts and ask Genie your first question: "Were we compromised in the last 90 days?"