Applications need access to API Keys and other secure values to operate properly. Conventional “best-practice” sees these values injected into the runtime environment. This common practice has a fatal flaw, leaving one open to serious security failures. Fortunately, an elegant workaround is not only available for free but also easy to implement.
TL;DR
Injecting secrets into the application environment risks exposing sensitive data in logs, process dumps, or other artifacts. Storing them in runtime variables offers a more secure management strategy, with the singleton pattern providing a practical implementation.
Understanding the Problem
In the case of environment-managed secure values, the problem is a side effect of a solution to another problem: how to provide access to secure values (passwords, API Keys, etc.) to a distributed system without hard-coding them?
The core issue lies outside the realm of which encryption algorithm, encryption-at-rest policy, or key management provider solution has been chosen. To access any of that, one needs proper authorization, and that authorization has to be stored somewhere.
Back in the cowboy days of software development, these things were stored in plain text, security keys were passed around casually, or sometimes things just … weren’t encrypted.
… breaks from conventional runtime credential management, which in turn breaks from conventional access, testing, and debugging patterns. The example here provides minimal thread-safety accommodations but would fall short in many production settings …
The solution to this historic issue was to inject secure variables at runtime. A modern approach is to use a secure third-party provider to store these secrets, such as AWS Secrets Manager or HashiCorp Vault.
An application is then given a single secure access value via which it can make a remote call to retrieve those values — rather than having to hard-code them anywhere. This works fine, but the conventional approach is to inject these values into the local runtime environment.
This is Ok up to a point — it’s certainly better than dumping them to a file. The problem is logs. Log files often dump the local OS environment for context, which, while helpful, might also expose sensitive values to unintended viewers.
Conventional Solutions
The solution is really just a patch to an existing, mostly practical approach and reflects very little novelty. The crux is to avoid injecting secrets into the environment so they don’t appear in logs when an environment is dumped. When we talk about storing things, the choices are pretty simple:
- CPU Memory
- RAM
- Disk
CPU isn’t practical, and Disk (file, DB, etc.) defeats the purpose. That leaves us with RAM — which is already where the runtime environment is, which is good. The next question is how we can improve the location within the RAM so that secure values won’t appear in logs? Simple — put them somewhere that isn’t conventionally included with logs.
The Secrets Singleton
A singleton class is a design pattern in which a single instance of an object exists throughout the entire application. The general pattern is shown below:
import threading
class SimpleSingleton:
"""
A thread-safe singleton
"""
# the instance accessible to all instantiations
_instance = None
# the lock to assure duplicate instances aren't created
_lock = threading.Lock()
def __new__(cls, *args, **kwargs):
"""
If an existing instance is found, returns that instance. Otherwise,
acquires the lock and create a new instance.
"""
# check for existing instance
if cls._instance is None:
# get_item a lock to create a new one
with cls._lock:
# double-check that a new instance hasn't been created during
# the time it took to acquire a lock
if not cls._instance:
# create the new instance.
cls._instance = super().__new__(cls)
return cls._instance
A GIST of this class is available here.
This isn’t a deep dive on the Singleton pattern, but briefly, this ensures that all instantiations of the class receive the same _instance value when calling the get_instance method. This has a bit of thread-safety built in to ensure that if 50 callers all requested the Singleton simultaneously for the first time, there would be no unexpected overwrites.
The Secrets Singleton
The Singleton allows an application to make a single API call during first instantiation (note the Python quirks below), thereby injecting the secrets into the runtime memory and making them available to all subsequent callers. It still works like OS environments with slightly different syntax. Here’s an example SecretsSingleton class implementation:
import threading
from typing import Optional
class SecretsSingleton:
"""A thread-safe singleton."""
_instance = None
_lock = threading.Lock()
def __new__(cls, *args, **kwargs):
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._initialized = False
return cls._instance
def __init__(self):
if self._initialized: # b/c Python is quirky
return
self._secrets = get_secrets()
self._initialized = True
@classmethod
def get_secret_value(cls, key: str) -> Optional[str]:
"""Returns secret value if exists."""
return cls()._secrets.get(key)
A GIST of this class is available here.
The only real note here is the defensiveness towards self._initialized which is a Python-specific requirement due to how subclasses call the __init__ method during the __new__ call. Here we are defending against successive/unnecessary calls to get_secrets() .
This class can be instantiated at runtime as part of the application initialization or deferred to the first time a secret is needed — dealer’s choice there. Let’s compare the access patterns below:
Before via OS/ENV:
my_secret_value = os.envion.get("MY_API_KEY")
After via SecretsSingleton:
# get an instance of the singleton; makes API call if first time
auth = SecretsSingleton()
# access secrets
my_secret_value = auth.get_secret_value("MY_API_KEY")
Discussion
This approach mitigates the possibility (not eliminating) of application credentials leaking into logs or other unexpected areas. Application state can still be logged, so depending on your application’s design, it might just be moving the problem to a less obvious place.
This approach is also subject to a bootstrap problem: where are the credentials needed to make the remote call to your secrets manager coming from? Those are still required, though they can be isolated to deployment providers and not included in source code (duh), and not distributed to the entire dev team. This is a standard service provided by any PaaS or IaaS providers.
There are other points to consider here; this breaks from conventional runtime credential management, which in turn breaks from conventional access, testing, and debugging patterns. The example here provides minimal thread-safety accommodations but would fall short in many production settings, namely when used with subclasses. Dependency Injection is another strong pattern relevant here, which should be considered for any production implementation of this high-level concept of segregating credentials from the local environment.
