I recently had to work on some stuff around PAN-OS API keys which made me want to understand how they are built and what they contain.
This is part of the “secrets” embedded in PAN-OS code, but trying to understand what API keys are built from might help you understand some strange behaviours you could eventually match, and perhaps avoid you some service disruptions… 🙂
The observation :
On its documentation, Palo Alto explicitly writes “To change an API key associated with an administrator account change the password associated with the administrator account.“
And… hopefully… that’s right 🙂
However, using a Palo Alto appliance configured to use an external LDAP authentication profile, I noticed that after a password change on the LDAP server for an account previously used to generate an API key, API calls using this key generated before the password change were still working properly, for, let’s say, a “random” amount of time.
The lab :
– Palo Alto firewall appliance (running PAN-OS 10.1.6)
– API user (local to the appliance)
– API user mapped to an LDAP authentication profile
– OpenLDAP server (running as a container) –> https://github.com/osixia/docker-openldap
Test with a local user
Let’s create an API key with the local user localapiuser :
curl -k -X POST 'https://192.168.192.128/api/?type=keygen&user=localapiuser&password=local@piuser'
<response status = 'success'><result><key>LUFRPT0zc0wwS0NVcThKczJiS2lUSHRZaENmM0I5UDg9RHoyUHdiMkpob3BhaHg5bFo4ZEx1UUI4Q0dObVVsNkFRMTlXcVpKSUFEckNERlU1YzVmOFNBcWxrRXBJK01UYw==</key></result></response>Then let’s issue a first API call with this API key to confirm it’s working :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT0zc0wwS0NVcThKczJiS2lUSHRZaENmM0I5UDg9RHoyUHdiMkpob3BhaHg5bFo4ZEx1UUI4Q0dObVVsNkFRMTlXcVpKSUFEckNERlU1YzVmOFNBcWxrRXBJK01UYw=='
<response status="success"><result><system><hostname>PA-VM</hostname><ip-address>192.168.192.128</ip-address><public-ip-address>unknown</public-ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.192.1</default-gateway><is-dhcp>yes</is-dhcp><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::20c:29ff:fe81:69d0/64</ipv6-link-local-address><mac-address>00:0c:29:81:69:d0</mac-address><time>Mon Jan 16 13:11:13 2023
</time>
[...]As we issue this request, we can see that our localapiuser appears in the list of logged in admins :
admin@PA-VM> show admins
Admin From Client Session-start Idle-for Session-expiry
---------------------------------------------------------------------------------------------
* admin 192.168.192.1 CLI 01/16 13:09:52 00:00:00s 02/15 13:09:52
localapiuser 192.168.192.1 Web 01/16 13:11:13 00:02:05s 02/15 13:11:13Now, let’s change the password of the localapiuser and issue an API call again using the same API key.
First interesting thing to note is that as soon as you change the password of a local user, it is immediately logged out :
admin@PA-VM> show admins
Admin From Client Session-start Idle-for Session-expiry
---------------------------------------------------------------------------------------------
* admin 192.168.192.1 CLI 01/16 13:09:52 00:00:00s 02/15 13:09:52Then let’s send our API request :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT0zc0wwS0NVcThKczJiS2lUSHRZaENmM0I5UDg9RHoyUHdiMkpob3BhaHg5bFo4ZEx1UUI4Q0dObVVsNkFRMTlXcVpKSUFEckNERlU1YzVmOFNBcWxrRXBJK01UYw=='
<response status = 'error' code = '403'><result><msg>Invalid Credential</msg></result></response>We immediately get an “Invalid Credential” error.
Other interesting thing is that if we change back the password of localapiuser to its previous value, we can reuse the API key again :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT0zc0wwS0NVcThKczJiS2lUSHRZaENmM0I5UDg9RHoyUHdiMkpob3BhaHg5bFo4ZEx1UUI4Q0dObVVsNkFRMTlXcVpKSUFEckNERlU1YzVmOFNBcWxrRXBJK01UYw=='
<response status="success"><result><system><hostname>PA-VM</hostname><ip-address>192.168.192.128</ip-address><public-ip-address>unknown</public-ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.192.1</default-gateway><is-dhcp>yes</is-dhcp><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::20c:29ff:fe81:69d0/64</ipv6-link-local-address><mac-address>00:0c:29:81:69:d0</mac-address><time>Mon Jan 16 13:18:38 2023
</time>
[...]Test with an LDAP user
Let’s repeat those operations with an external user we authenticate through an LDAP authentication profile.
First, let’s create a new API key for this user :
curl -k -X POST 'https://192.168.192.128/api/?type=keygen&user=ldapapiuser&password=ldapapiuser'
<response status = 'success'><result>
<key>LUFRPT1rZGhvc0RkOGcvNUxNRG1TUlY1QWhqZjBDRUU9ZEhNWkxnbEt3M3FsNGxSazVDSHZZM1UwSVFNSWhLUURLbCtJQVB2WE5XMlQ1TWpURDlKWVpzSnI5eWExV2FwMg==</key></result></response>Then we issue a first API request using this new key :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT1rZGhvc0RkOGcvNUxNRG1TUlY1QWhqZjBDRUU9ZEhNWkxnbEt3M3FsNGxSazVDSHZZM1UwSVFNSWhLUURLbCtJQVB2WE5XMlQ1TWpURDlKWVpzSnI5eWExV2FwMg=='
<response status="success"><result><system><hostname>PA-VM</hostname><ip-address>192.168.192.128</ip-address><public-ip-address>unknown</public-ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.192.1</default-gateway><is-dhcp>yes</is-dhcp><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::20c:29ff:fe81:69d0/64</ipv6-link-local-address><mac-address>00:0c:29:81:69:d0</mac-address><time>Mon Jan 16 13:57:41 2023
</time>
[...]Again, the user appears as a logged in admin as soon as we use the associated API key :
admin@PA-VM> show admins
Admin From Client Session-start Idle-for Session-expiry
---------------------------------------------------------------------------------------------
* admin 192.168.192.1 CLI 01/16 13:09:52 00:00:00s 02/15 13:09:52
ldapapiuser 192.168.192.1 Web 01/16 13:57:41 00:01:33s 02/15 13:57:41Then, let’s change the ldapapiuser password on the LDAP server.
This time, of course, there’s no way for the Palo appliance to be aware of this change, so the ldapapiuser remains appearing as a logged in user.
root@1a25a95e316b:/# ldappasswd -H ldap://localhost -x -D "cn=admin,dc=test,dc=local" -W -S "uid=ldapapiuser,dc=test,dc=local"
New password: ldapapiuser2
Re-enter new password: ldapapiuser2
Enter LDAP Password: <adminpass>admin@PA-VM> show admins
Admin From Client Session-start Idle-for Session-expiry
---------------------------------------------------------------------------------------------
* admin 192.168.192.1 CLI 01/16 13:09:52 00:00:00s 02/15 13:09:52
ldapapiuser 192.168.192.1 Web 01/16 13:57:41 00:03:36s 02/15 13:57:41Now, let’s send again an API request using the API key generated before the password change :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT1rZGhvc0RkOGcvNUxNRG1TUlY1QWhqZjBDRUU9ZEhNWkxnbEt3M3FsNGxSazVDSHZZM1UwSVFNSWhLUURLbCtJQVB2WE5XMlQ1TWpURDlKWVpzSnI5eWExV2FwMg=='
<response status="success"><result><system><hostname>PA-VM</hostname><ip-address>192.168.192.128</ip-address><public-ip-address>unknown</public-ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.192.1</default-gateway><is-dhcp>yes</is-dhcp><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::20c:29ff:fe81:69d0/64</ipv6-link-local-address><mac-address>00:0c:29:81:69:d0</mac-address><time>Mon Jan 16 14:02:48 2023
</time>
[...]Surprise…. still working ?!
Thanks to the hits I added in this article, you are probably able to guess why.
Remember that :
– when you use the API key provided to a user, this user appears as logged into the system
– when you change the password of a local user, it is instantly logged out
– when you change the password of an external (LDAP) user, it still appears as logged in until its session timeout
After some time (around 60 minutes by default), the ldapapiuser admin session will be cleared.
(You can speed up this process by using the delete admin-sessions username ldapapiuser PAN-OS CLI command)
If you use the API key again, this time, you’ll be informed about bad credentials :
curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT1rZGhvc0RkOGcvNUxNRG1TUlY1QWhqZjBDRUU9ZEhNWkxnbEt3M3FsNGxSazVDSHZZM1UwSVFNSWhLUURLbCtJQVB2WE5XMlQ1TWpURDlKWVpzSnI5eWExV2FwMg=='
<response status = 'error' code = '403'><result><msg>Invalid Credential</msg></result></response>Changing the LDAP user password back to its original value will make the API key working again :
root@1a25a95e316b:/# ldappasswd -H ldap://localhost -x -D "cn=admin,dc=test,dc=local" -W -S "uid=ldapapiuser,dc=test,dc=local"
New password: ldapapiuser
Re-enter new password: ldapapiuser
Enter LDAP Password: <adminpass>curl -k -X GET 'https://192.168.192.128//api/?type=op&cmd=<show><system><info></info></system></show>&key=LUFRPT1rZGhvc0RkOGcvNUxNRG1TUlY1QWhqZjBDRUU9ZEhNWkxnbEt3M3FsNGxSazVDSHZZM1UwSVFNSWhLUURLbCtJQVB2WE5XMlQ1TWpURDlKWVpzSnI5eWExV2FwMg=='
<response status="success"><result><system><hostname>PA-VM</hostname><ip-address>192.168.192.128</ip-address><public-ip-address>unknown</public-ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.192.1</default-gateway><is-dhcp>yes</is-dhcp><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::20c:29ff:fe81:69d0/64</ipv6-link-local-address><mac-address>00:0c:29:81:69:d0</mac-address><time>Mon Jan 16 14:14:46 2023
</time>
[...]Conclusions
Thanks to those conducted tests and to some interesting information found on some tech forums, here’s a list of interesting things to know about the PAN-OS API keys :
- they are an encrypted tuple of the user and password used to generate the key itself
- using one of those keys creates an admin session on the PAN-OS appliance
- changing the user password then makes the key invalid, but an existing admin session created with a key remains valid (not kicked out if the password change is not on a local admin account) until the “Idle Timeout” expiration
- PAN-OS >= 9.0 adds a timestamp to the API keys (encrypted with the username and password) so that they have a lifetime.
- If you want to revoke all the keys generated on a PAN-OS system, just click the “Revoke all keys” button on the Device –> Setup –> Management –> Authentication Settings panel (see below). It will add a timestamps on the PAN-OS configuration which will make all keys using an inferior timestamp to be unusable.
- If you want to revoke an API key for a given user, the only way to proceed is to change this user’s password
- You can immediately make an external user password change effective to revoke the associated API keys by using the delete admin-sessions username <username> PAN-OS CLI command right after the password change
- If you made a mistake by changing an external (or local) user password and you want to make the API keys working again… just revert the user’s password of the key to its previous value
