New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Service account proposal. #2297
Conversation
I'm pasting the contents of the proposal in the next comment, for ease of reading in email. But, if you want to comment, go to the files part of the pull request. |
Robot AccountsProcessed in Pods may need to call the Kubernetes API. For example:
Distinguishing between humans and robot accounts (a.k.a. service accounts or roles) is a practice used by a number of cloud providers, such as:
The requirements for humans and VMs/containers are different:
Therefore, Kubernetes should support both human and robot accounts, with different ways of managing credentials and policies for each. ProposalWe currently have basic support for user-strings and tokens via the Secret objectsA new object Kind is added to hold secrets. type Secret struct {
TypeMeta `json:",inline" yaml:",inline"`
ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
secret []bytes
type string // type must be "KubernetesAuth". Other types may be added later.
} The secret is the actual secret value, such as a bearer token or key. Typically, there will be built-in policy which denies all users except kubelets and admins from reading secrets. Robot objectsA new object Kind is added for Robot account names: type Robot struct {
TypeMeta `json:",inline" yaml:",inline"`
ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
secrets []ObjectReference
} The Name and Namespace of the /robots object define a robot account. The robot The The secrets are not inline with the robot object. This way, most or all users can have read permission on robots so they can remind themselves ApiserverValidation will prevent creation of a robot with two secrets of type When a robot and a matching secret exist, then a Making Pods with Robot AccountsPods are extended to have a "robotAccount" field in the desired_state and current_state. When a pod is created, a user can give it a "robotAccount". TBD: how policy limits which users can make pods with which robot accounts. Pod SetupWhen a kubelet sees a pod with a Future Improvements:
Related:Discussion of why robot accounts are needed and why pods do not get the human identity of their creator in issue 2211. The general problem of secret distribution for both kubernetes-api secrets and other-service secrets is discussed in these issues:
This proposal is just focused on the narrower problem of access to the kubernetes API by pods. |
- scheduler | ||
- replication controller | ||
- minion controller | ||
- a map-reduce type framework which has a controller than then tries to make a dynamically determined number of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
*that then
Use case brought up today in the hangout:
|
I read the whole discussion here and I have a lot of ideas even for this narrow issue, but can't sort my thoughts it seems. Therefore I will throw in my recent specification proposal, which includes secret and token distribution and defines a standard way of doing it. moby/moby#9277 |
Can one of the admins verify this patch? |
461e3ff
to
cea1bcc
Compare
REST object is added to the apiserver that represents a secret which an account needs to prove its identity to peers. | ||
This is called `secret`. A `serviceAccount` has access to one or more `secret` objects. The storage of secrets is separate to allow listing `/serviceAccounts` without seeing secrets. | ||
|
||
The `PodSpec` is extended to allow Pods to be started with an associated serviceAccount. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It may be desirable to allow all pods to run with a default set of secrets in a namespace. I'm thinking things like docker registries, git repos, remote services.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That does sound desirable. Mechanism needs thought.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though experiment:
- sooner or later, every process in a cluster has to run in some kind of security context
- the reality of Unix is that uid/gid and to a lesser extent selinux mcs labels are the only mechanisms in a cluster for identifying the difference between security contexts in the kernel, process space, and filesystem
- it follows that every process run by Kubernetes at a node level should be assigned to at least one security context
- security contexts should ideally be isolated across namespaces, but no process should be run outside of a security context
- an auditor on a host has no option in some cases but to attempt to associate a process uid/gid to higher level account, and real world large scale clusters reflect this by avoiding reusing uids/gids across unrelated services
- to me, the secrets available to a process constitute part of a security context, since audit must be able to reason about the capabilities a process has on other systems
If the system has a default security context, all pods would run under that. If a namespace has a default security context, it's pods would run under that. There may be a desire to run individual pods in a namespace within different contexts. However, both of the first two options do not require pods to be aware of a security context, just the kubelet. A service account with secrets might be one mechanism for that security context to be configured, but does not imply that is the only mechanism.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@csrwng and I are going to turn this into a concrete short term proposal to make steps towards this from the kubelet end.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There may also be good reasons for isolating security contexts of pods in the same namespace. If possible, suggest avoid hardcoding a requirement for all pods in namespace to share uid.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think only part of this comment came through
On Jan 13, 2015, at 5:58 PM, Eric Tune notifications@github.com wrote:
In docs/design/robots.md:
- Humans may better tolerate the exceptional conditions created by expiration of a token. Remembering to handle
this in a program is more annoying. So, either long-lasting credentials or automated rotation of credentials is
needed.
- A Human typically keeps credentials on a machine that is not part of the cluster and so not subject to automatic
management. A VM with a role/service-account can have its credentials automatically managed.
+Therefore, Kubernetes should support both human and service accounts, with different ways of managing credentials and
+policies for each.
+
+## Design Overview
+
+A new REST object is added to the apiserver that represents a service account, calledserviceAccount
. Another new
+REST object is added to the apiserver that represents a secret which an account needs to prove its identity to peers.
+This is calledsecret
. AserviceAccount
has access to one or moresecret
objects. The storage of secrets is separate to allow listing/serviceAccounts
without seeing secrets.
+
+ThePodSpec
is extended to allow Pods to be started with an associated serviceAccount.
There may also be good reasons for isolating security contexts of pods in the same namespace. If possible, suggest avoid hardcoding a requirement for all pods in namespace to share uid.If you want to avoid uid reuse
—
Reply to this email directly or view it on GitHub.
@smarterclayton the issue with using a kubernetes specific api for secrets is that it involves a non standard interface. Secret Environment variables or using tmpfs volumes would be much more interchangeable. Even if the actually provided volume is created from a secret api. |
What standard interface for secrets exist? Registering secrets for use with the pod is no different than adding an env var to the pod definition, or registering a volume. The secret source has to be accessible to the Kubernetes infrastructure by definition, and I don't think anything here prevents external secrets as volumes, nor prevents an external secrets api. When you say this is not interchangeable, what are you comparing it to? The point of secrets as separate from pods is to control scope - I offer secrets to a pod, vs embedding them into pods. Secrets in the pod prevents pod templates from being reused
|
@smarterclayton perhaps I misunderstood, but my notion was, that you propose a specific service to be used for secret distribution within kubernetes. So in my opinion a volume interface is more interchangeable than specific tools. Please correct me, if I misunderstood your intentions. |
Container authors should not, within reason, have to change to run in containers or on secure infrastructure. Volume conventions are a good way of providing better tools for those who want to adopt them, but there still has to be a way to adapt them. Also, secrets and service accounts are for consumers of Kubernetes that may lack the underlying infrastructure necessary for secret distribution. Those concepts should where practical be possible to be backed by existing infrastructure. The goals you describe should all be achievable without requiring the use of secrets or service accounts.
|
The proposal in moby/moby#9277 proposes ENV sourcing from volumes, which the actual applications don't need to understand only the container. But I didn't want to start a proposal discussion here, just wanted to get another opinion in to make the service account proposal more future proof from a users point of view. |
A key part of this proposal is about how to generate and distribute limited scope and time authorizations to the pods running in Kubernetes such that they can perform appropriate actions against the api as designated by a caller. That especially includes infrastructure components. So at a minimum, we need a way to ensure that infrastructure pods can receive temporary secrets at execution time, make them easy to be regenerated, and easy to clear. That does not have to be a generic solution, although practically speaking elements of that solution should be reusable for other similar needs. A concern I've spent some time thinking about the past few weeks is what is the optimal way to ensure that these automatic values get applied to pods when they run on the Kubelet. The current pattern is to write them into the bound pod at scheduling time, which is inflexible to extension. Eric has a proposal to change how bound pods are delivered to the nodes by watching and reading pod changes directly. The current pattern would provide no story for updating pods when the token expires (unless the pod also had an expiration date after which it was deleted), while the new pattern the kubelet would need to read this new resource (and watch for updates)? We probably want to sketch out this change, the volume change, and the proposed Kubelet pods change together and see if there are any requirements we're not considering on how we move pod creation requests to bound pods to kubelets. |
Cesar and I are still working on security context and service account, but @liggitt and I had a good discussion about service accounts overlaps with Kubelet:
|
@smarterclayton |
Hi, sorry for the noise, feel free to disregard my comment, I'm just getting my feet wet in kubernetes and I have a huge lack of context. Have you thought about using macaroons for authorization? It sounds like it would be a good fit for the layered approach that you're pursuing here: http://research.google.com/pubs/pub41892.html |
We have - it's something we've been looking at adding to the path for delegation of authority between system components, and it would also let us dynamically scoped tokens which is awesome. I think macaroons would let us give the service account an identity on the server and then craft scoped tokens for use in contexts where the pod needs to talk back to the master. Some problems have to be solved as a part of secrets still, namely how rotation would work.
|
This proposed update to docs/design/security.md includes proposals on how to ensure containers have consistent Linux security behavior across nodes, how containers authenticate and authorize to the master and other components, and how secret data could be distributed to pods to allow that authentication. References concepts from kubernetes#3910, kubernetes#2030, and kubernetes#2297 as well as upstream issues around the Docker vault and Docker secrets.
cea1bcc
to
687b6d7
Compare
I've updated this to reflect the progress made in defining secrets and security contexts. I've remove the discussion about non-volume secret distribution. We may come back to that, but it doesn't appear to be the top priority. I've outlined how a finalizer could be used to handle defaulting without putting business logic in the apiserver, and thought a bit about how to layer service accounts on to of the more primitive secrets/securityContexts. |
@pmorie @smarterclayton @csrwng please take a glance when convenient. |
might have some types that do not do anything on apiserver but just get pushed to the kubelet.) | ||
|
||
### Pods | ||
The `PodSpec` is extended to have a `Pods.Spec.ServiceAccountUsername` field. If this is unset, then a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should spec that an administrator may use an admission control hook or a finalizer to define rules and behavior around the assignment of service accounts to pods - and that we expect that to be mostly configured by the administrator (for instance, to force a default service account, or to limit a user from specifying a service account directly if they are not part of a certain security group).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh wait, you covered it below.
This looks good to me. Everything else seems like iteration and refinement. |
Going to merge tomorrow. |
|
||
### Names | ||
|
||
From the standpoint of the Kubernetes API, a `user` is any pricipal which can authenticate to kubernetes API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pricipal -> principal
Apologies for the late review, I missed the tag. LGTM aside from a couple typos. |
5b70b73
to
606a994
Compare
COMMIT_BLOCKED_ON_GENDOCS
606a994
to
38a85ef
Compare
Addressed @pmorie comments. |
Merging |
Service account proposal.
This proposed update to docs/design/security.md includes proposals on how to ensure containers have consistent Linux security behavior across nodes, how containers authenticate and authorize to the master and other components, and how secret data could be distributed to pods to allow that authentication. References concepts from kubernetes#3910, kubernetes#2030, and kubernetes#2297 as well as upstream issues around the Docker vault and Docker secrets.
In a rough state but putting out for comment.
There has been considerable discussion about the general problem of secret distribution for docker and kubernetes.
While that is a great goal, I think it will take some time.
In the shorter term, I want to be able to just distribute secrets for access to the kubernetes API.
This proposal just focuses on that.
That problem is important to solve because: