Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service account proposal. #2297

Merged
merged 1 commit into from Feb 20, 2015
Merged

Conversation

erictune
Copy link
Member

In a rough state but putting out for comment.

There has been considerable discussion about the general problem of secret distribution for docker and kubernetes.
While that is a great goal, I think it will take some time.
In the shorter term, I want to be able to just distribute secrets for access to the kubernetes API.
This proposal just focuses on that.

That problem is important to solve because:

  • it lets us run scheduler and controller-manager via /pods
  • lets users write their own services, like cron, build controllers, etc, more easily.
  • gets us closer to self-hosting.

@erictune
Copy link
Member Author

I'm pasting the contents of the proposal in the next comment, for ease of reading in email. But, if you want to comment, go to the files part of the pull request.

@erictune
Copy link
Member Author

Robot Accounts

Processed in Pods may need to call the Kubernetes API. For example:

  • scheduler
  • replication controller
  • minion controller
  • a map-reduce type framework which has a controller than then tries to make a dynamically determined number of workers and watch them
  • continuous build and push system
  • monitoring system

Distinguishing between humans and robot accounts (a.k.a. service accounts or roles) is a practice used by a number of cloud providers, such as:

The requirements for humans and VMs/containers are different:

  • Humans need a wide range of capabilities to do their daily activities. Robots often have more narrowly-defined activities.
  • Humans may better tolerate the exceptional conditions created by expiration of a token. Remembering to handle this in a program is more annoying. So, either long-lasting credentials or automated rotation of credentials is needed.
  • A Human typically keeps credentials on a machine that is not part of the cluster and so not subject to automatic management. A VM with a role/service-account can have its credentials automatically managed.

Therefore, Kubernetes should support both human and robot accounts, with different ways of managing credentials and policies for each.

Proposal

We currently have basic support for user-strings and tokens via the --token_auth_file flag of apiserver. It is assumed that this will expand to include bedrock authentication sources and short-lived tokens. However, robot accounts will likely not use those improvements.

Secret objects

A new object Kind is added to hold secrets.

type Secret struct {
    TypeMeta   `json:",inline" yaml:",inline"`
    ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`

    secret  []bytes
    type    string // type must be "KubernetesAuth".  Other types may be added later.
}

The secret is the actual secret value, such as a bearer token or key.
The type tells the kubelet what to do with the value. Initially, the only option is "SecretTypeKubernetesToken".

Typically, there will be built-in policy which denies all users except kubelets and admins from reading secrets.

Robot objects

A new object Kind is added for Robot account names:

type Robot struct {
    TypeMeta   `json:",inline" yaml:",inline"`
    ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`

    secrets []ObjectReference 
}

The Name and Namespace of the /robots object define a robot account. The robot
account may be refered to as $NAMESPACE+$NAME@robots.$CLUSTERID.kubernetes.io
in UIs, audit-logs or other places that expect an email-type string.

The secrets field is a list of references to /secret objects that the robot should have access to.

The secrets are not inline with the robot object. This way, most or all users can have read permission on robots so they can remind themselves
what robots are available for use.

Apiserver

Validation will prevent creation of a robot with two secrets of type SecretTypeKubernetesAuth.

When a robot and a matching secret exist, then a User.Info for the robot and a BearerToken from the secret
are added to the map of tokens used by the authentication process in the apiserver.

Making Pods with Robot Accounts

Pods are extended to have a "robotAccount" field in the desired_state and current_state. When a pod is created, a user can give it a "robotAccount".

TBD: how policy limits which users can make pods with which robot accounts.

Pod Setup

When a kubelet sees a pod with a robotAccount field, it requests that robot object from the apiserver.
When the kubelet gets the robot list, it then requests each referenced secret.
When it gets a secret with type 'SecretTypeKubernetesAuth', it writes .kubernetes_auth file in a known location
with field BearerToken equal to the secret value. The process in the pod uses a kubernetes client library or kubectl and accesses the API.

Future Improvements:

  • authorization checks to ensure a kubelet only requests /secrets that are bound to it.
  • rotation of tokens. (Modify .kubernetes_auth. Make both old and new tokens valid in master for a limited duration. Then expire old one.)
  • local proxy holds tokens. Instead of putting BearerToken in .kubernetes_auth, the kubelet puts the proxy url into the .kubernetes_auth file, and proxy injects auth. See LOASD proposal.

Related:

Discussion of why robot accounts are needed and why pods do not get the human identity of their creator in issue 2211.

The general problem of secret distribution for both kubernetes-api secrets and other-service secrets is discussed in these issues:

This proposal is just focused on the narrower problem of access to the kubernetes API by pods.

- scheduler
- replication controller
- minion controller
- a map-reduce type framework which has a controller than then tries to make a dynamically determined number of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*that then

@smarterclayton
Copy link
Contributor

Use case brought up today in the hangout:

  • Supporting client cert secrets so that a consumer in a small cluster could reuse the cluster etcd instance securely

@stp-ip
Copy link
Member

stp-ip commented Nov 21, 2014

I read the whole discussion here and I have a lot of ideas even for this narrow issue, but can't sort my thoughts it seems. Therefore I will throw in my recent specification proposal, which includes secret and token distribution and defines a standard way of doing it. moby/moby#9277

@kubernetes-bot
Copy link

Can one of the admins verify this patch?

@erictune erictune force-pushed the robot_design branch 2 times, most recently from 461e3ff to cea1bcc Compare December 5, 2014 03:38
@erictune erictune changed the title WIP. Robot account proposal. Service account proposal. Dec 5, 2014
REST object is added to the apiserver that represents a secret which an account needs to prove its identity to peers.
This is called `secret`. A `serviceAccount` has access to one or more `secret` objects. The storage of secrets is separate to allow listing `/serviceAccounts` without seeing secrets.

The `PodSpec` is extended to allow Pods to be started with an associated serviceAccount.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be desirable to allow all pods to run with a default set of secrets in a namespace. I'm thinking things like docker registries, git repos, remote services.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does sound desirable. Mechanism needs thought.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though experiment:

  • sooner or later, every process in a cluster has to run in some kind of security context
  • the reality of Unix is that uid/gid and to a lesser extent selinux mcs labels are the only mechanisms in a cluster for identifying the difference between security contexts in the kernel, process space, and filesystem
  • it follows that every process run by Kubernetes at a node level should be assigned to at least one security context
  • security contexts should ideally be isolated across namespaces, but no process should be run outside of a security context
  • an auditor on a host has no option in some cases but to attempt to associate a process uid/gid to higher level account, and real world large scale clusters reflect this by avoiding reusing uids/gids across unrelated services
  • to me, the secrets available to a process constitute part of a security context, since audit must be able to reason about the capabilities a process has on other systems

If the system has a default security context, all pods would run under that. If a namespace has a default security context, it's pods would run under that. There may be a desire to run individual pods in a namespace within different contexts. However, both of the first two options do not require pods to be aware of a security context, just the kubelet. A service account with secrets might be one mechanism for that security context to be configured, but does not imply that is the only mechanism.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@csrwng and I are going to turn this into a concrete short term proposal to make steps towards this from the kubelet end.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may also be good reasons for isolating security contexts of pods in the same namespace. If possible, suggest avoid hardcoding a requirement for all pods in namespace to share uid.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think only part of this comment came through

On Jan 13, 2015, at 5:58 PM, Eric Tune notifications@github.com wrote:

In docs/design/robots.md:

    • Humans may better tolerate the exceptional conditions created by expiration of a token. Remembering to handle
  •  this in a program is more annoying.  So, either long-lasting credentials or automated rotation of credentials is
    
  •  needed.
    
    • A Human typically keeps credentials on a machine that is not part of the cluster and so not subject to automatic
  •  management.  A VM with a role/service-account can have its credentials automatically managed.
    
    +Therefore, Kubernetes should support both human and service accounts, with different ways of managing credentials and
    +policies for each.
    +
    +## Design Overview
    +
    +A new REST object is added to the apiserver that represents a service account, called serviceAccount. Another new
    +REST object is added to the apiserver that represents a secret which an account needs to prove its identity to peers.
    +This is called secret. A serviceAccount has access to one or more secret objects. The storage of secrets is separate to allow listing /serviceAccounts without seeing secrets.
    +
    +The PodSpec is extended to allow Pods to be started with an associated serviceAccount.
    There may also be good reasons for isolating security contexts of pods in the same namespace. If possible, suggest avoid hardcoding a requirement for all pods in namespace to share uid.

If you want to avoid uid reuse


Reply to this email directly or view it on GitHub.

@stp-ip
Copy link
Member

stp-ip commented Dec 11, 2014

@smarterclayton the issue with using a kubernetes specific api for secrets is that it involves a non standard interface. Secret Environment variables or using tmpfs volumes would be much more interchangeable. Even if the actually provided volume is created from a secret api.

@smarterclayton
Copy link
Contributor

What standard interface for secrets exist?

Registering secrets for use with the pod is no different than adding an env var to the pod definition, or registering a volume. The secret source has to be accessible to the Kubernetes infrastructure by definition, and I don't think anything here prevents external secrets as volumes, nor prevents an external secrets api.

When you say this is not interchangeable, what are you comparing it to? The point of secrets as separate from pods is to control scope - I offer secrets to a pod, vs embedding them into pods. Secrets in the pod prevents pod templates from being reused

On Dec 11, 2014, at 5:01 AM, Michael Grosser notifications@github.com wrote:

@smarterclayton the issue with using a kubernetes specific api for secrets is that it involves a non standard interface. Secret Environment variables or using tmpfs volumes would be much more interchangeable. Even if the actually provided volume is created from a secret api.


Reply to this email directly or view it on GitHub.

@stp-ip
Copy link
Member

stp-ip commented Dec 12, 2014

@smarterclayton perhaps I misunderstood, but my notion was, that you propose a specific service to be used for secret distribution within kubernetes.
This would be useable in other kubernetes clusters, but not in other container based systems. I agree that defining a volume could be the same as defining a secret provider, the difference in my opinion is, that a volume is a broader tooling to join different APIs or secret providers under one hood. One would just define a per pod Side Container providing secret volumes or use a Volume provided from Kubernetes, but one could easily recreate such a volume and it's content via Host Volumes and run it anywhere.

So in my opinion a volume interface is more interchangeable than specific tools.

Please correct me, if I misunderstood your intentions.

@smarterclayton
Copy link
Contributor

On Dec 12, 2014, at 3:14 AM, Michael Grosser notifications@github.com wrote:

@smarterclayton perhaps I misunderstood, but my notion was, that you propose a specific service to be used for secret distribution within kubernetes.
This would be useable in other kubernetes clusters, but not in other container based systems. I agree that defining a volume could be the same as defining a secret provider, the difference in my opinion is, that a volume is a broader tooling to join different APIs or secret providers under one hood. One would just define a per pod Side Container providing secret volumes or use a Volume provided from Kubernetes, but one could easily recreate such a volume and it's content via Host Volumes and run it anywhere.

So in my opinion a volume interface is more interchangeable than specific tools.

Env as Volumes require the container to understand the format. A large portion of the work Kubernetes needs to do is adapt new patterns to work with existing software. Exposing secrets as env via some endpoint is part of adaptation.

Container authors should not, within reason, have to change to run in containers or on secure infrastructure. Volume conventions are a good way of providing better tools for those who want to adopt them, but there still has to be a way to adapt them.

Also, secrets and service accounts are for consumers of Kubernetes that may lack the underlying infrastructure necessary for secret distribution. Those concepts should where practical be possible to be backed by existing infrastructure. The goals you describe should all be achievable without requiring the use of secrets or service accounts.

Please correct me, if I misunderstood your intentions.


Reply to this email directly or view it on GitHub.

@stp-ip
Copy link
Member

stp-ip commented Dec 12, 2014

The proposal in moby/moby#9277 proposes ENV sourcing from volumes, which the actual applications don't need to understand only the container. But I didn't want to start a proposal discussion here, just wanted to get another opinion in to make the service account proposal more future proof from a users point of view.

@smarterclayton
Copy link
Contributor

A key part of this proposal is about how to generate and distribute limited scope and time authorizations to the pods running in Kubernetes such that they can perform appropriate actions against the api as designated by a caller. That especially includes infrastructure components. So at a minimum, we need a way to ensure that infrastructure pods can receive temporary secrets at execution time, make them easy to be regenerated, and easy to clear. That does not have to be a generic solution, although practically speaking elements of that solution should be reusable for other similar needs.

A concern I've spent some time thinking about the past few weeks is what is the optimal way to ensure that these automatic values get applied to pods when they run on the Kubelet. The current pattern is to write them into the bound pod at scheduling time, which is inflexible to extension. Eric has a proposal to change how bound pods are delivered to the nodes by watching and reading pod changes directly. The current pattern would provide no story for updating pods when the token expires (unless the pod also had an expiration date after which it was deleted), while the new pattern the kubelet would need to read this new resource (and watch for updates)?

We probably want to sketch out this change, the volume change, and the proposed Kubelet pods change together and see if there are any requirements we're not considering on how we move pod creation requests to bound pods to kubelets.

@smarterclayton
Copy link
Contributor

Cesar and I are still working on security context and service account, but @liggitt and I had a good discussion about service accounts overlaps with Kubelet:

  • A kubelet has a unique identity kubelet/127.0.0.1, kubelet/127.0.0.2, etc
  • The identity of a kubelet is to a unique actor (i.e., if 127.0.0.1 does something, the action should be attributed to 127.0.0.1), not to a human user or to a generic kubelet account
  • Controller loops will eventually look a lot like kubelets, there will just be one account for each - we should make those loops be running with an identity that can authn/z to the masters just like kubelets
  • If controllers run in pods or from within a master, they're still going to need a unique identity
  • Pods in a namespace are doing things (at a minimum) as the identity associated with that namespace, or as a global identity, or in the future as a distinct identity (such as a service account or user). We should leave the door open to running a pod as a user account in the future, but in the short term it is simpler and better security to have delegation - a namespace identity or a service account.
  • Service accounts could have an identity that corresponds to their namespace - something like "serviceaccount/namespace/default" might be the default, and in the future the identity could be further scoped to "serviceaccount/namespace/other".
  • If service accounts can fit into the global namespace of identity (like the kubelet), then our existing flows simply need to distinguish between the type of account and perform authorization checks based on the identity, with some minor specific details - i.e. "kubelet/foo" is different than "user/bar" which is different than "serviceaccount/baz". Each of those is a distinct actor, and could possibly be modeled as users in the background if necessary.
  • If service accounts have an identity, they can fit into the concepts we discussed elsewhere about policy where you can set a policy rule for a user and grant them access to do X, and then RBAC systems on top of that policy system can bind the service account identity to a role which can then do X.

@erictune
Copy link
Member Author

@smarterclayton
That all sounds good. Having distinct identities for each kubelet, for each controller-loop-process, and for sets of pods is exactly what I intended. Having service accounts be attached to namespaces has some nice properties, and matches what GCE does today with service accounts being attached to projects.

@calavera
Copy link
Contributor

Hi,

sorry for the noise, feel free to disregard my comment, I'm just getting my feet wet in kubernetes and I have a huge lack of context.

Have you thought about using macaroons for authorization? It sounds like it would be a good fit for the layered approach that you're pursuing here:

http://research.google.com/pubs/pub41892.html
https://air.mozilla.org/macaroons-cookies-with-contextual-caveats-for-decentralized-authorization-in-the-cloud/

@smarterclayton
Copy link
Contributor

We have - it's something we've been looking at adding to the path for delegation of authority between system components, and it would also let us dynamically scoped tokens which is awesome. I think macaroons would let us give the service account an identity on the server and then craft scoped tokens for use in contexts where the pod needs to talk back to the master. Some problems have to be solved as a part of secrets still, namely how rotation would work.

On Jan 24, 2015, at 2:53 AM, David Calavera notifications@github.com wrote:

Hi,

sorry for the noise, feel free to disregard my comment, I'm just getting my feet wet in kubernetes and I have a huge lack of context.

Have you thought about using macaroons for authorization? It sounds like it would be a good fit for the layered approach that you're pursuing here:

http://research.google.com/pubs/pub41892.html
https://air.mozilla.org/macaroons-cookies-with-contextual-caveats-for-decentralized-authorization-in-the-cloud/


Reply to this email directly or view it on GitHub.

smarterclayton added a commit to smarterclayton/kubernetes that referenced this pull request Feb 12, 2015
This proposed update to docs/design/security.md includes proposals
on how to ensure containers have consistent Linux security behavior
across nodes, how containers authenticate and authorize to the master
and other components, and how secret data could be distributed to
pods to allow that authentication.

References concepts from kubernetes#3910, kubernetes#2030, and kubernetes#2297 as well as upstream issues
around the Docker vault and Docker secrets.
@erictune
Copy link
Member Author

I've updated this to reflect the progress made in defining secrets and security contexts. I've remove the discussion about non-volume secret distribution. We may come back to that, but it doesn't appear to be the top priority. I've outlined how a finalizer could be used to handle defaulting without putting business logic in the apiserver, and thought a bit about how to layer service accounts on to of the more primitive secrets/securityContexts.

@erictune
Copy link
Member Author

@pmorie @smarterclayton @csrwng please take a glance when convenient.

might have some types that do not do anything on apiserver but just get pushed to the kubelet.)

### Pods
The `PodSpec` is extended to have a `Pods.Spec.ServiceAccountUsername` field. If this is unset, then a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should spec that an administrator may use an admission control hook or a finalizer to define rules and behavior around the assignment of service accounts to pods - and that we expect that to be mostly configured by the administrator (for instance, to force a default service account, or to limit a user from specifying a service account directly if they are not part of a certain security group).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait, you covered it below.

@smarterclayton
Copy link
Contributor

This looks good to me. Everything else seems like iteration and refinement.

@smarterclayton
Copy link
Contributor

Going to merge tomorrow.

@smarterclayton smarterclayton added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 20, 2015

### Names

From the standpoint of the Kubernetes API, a `user` is any pricipal which can authenticate to kubernetes API.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pricipal -> principal

@pmorie
Copy link
Member

pmorie commented Feb 20, 2015

Apologies for the late review, I missed the tag. LGTM aside from a couple typos.

    COMMIT_BLOCKED_ON_GENDOCS
@erictune
Copy link
Member Author

Addressed @pmorie comments.

@smarterclayton
Copy link
Contributor

Merging

smarterclayton added a commit that referenced this pull request Feb 20, 2015
@smarterclayton smarterclayton merged commit 1d82c1f into kubernetes:master Feb 20, 2015
@erictune erictune deleted the robot_design branch September 29, 2015 15:20
xingzhou pushed a commit to xingzhou/kubernetes that referenced this pull request Dec 15, 2016
This proposed update to docs/design/security.md includes proposals
on how to ensure containers have consistent Linux security behavior
across nodes, how containers authenticate and authorize to the master
and other components, and how secret data could be distributed to
pods to allow that authentication.

References concepts from kubernetes#3910, kubernetes#2030, and kubernetes#2297 as well as upstream issues
around the Docker vault and Docker secrets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security lgtm "Looks good to me", indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants