Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet access to API server should be subdivided #40476

Closed
liggitt opened this issue Jan 25, 2017 · 12 comments
Closed

kubelet access to API server should be subdivided #40476

liggitt opened this issue Jan 25, 2017 · 12 comments
Assignees
Labels
area/configmap-api area/kubelet area/secret-api area/security sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/node Categorizes an issue or PR as relevant to SIG Node.
Milestone

Comments

@liggitt
Copy link
Member

liggitt commented Jan 25, 2017

In order to obtain the resources needed to run pods, Kubelets currently have broad access to the API:

  • read any secret
  • read any configmap
  • read any persistentvolumeclaim
  • create/update any pod and pod status
  • create/update any node and node status

To properly secure individual nodes and limit the ability of a particular node to access the cluster, the kubelet should only be allowed to retrieve data from the apiserver for resources associated with it. This essentially means that it is only be able to get pods scheduled to it, along with any related items, such as secrets, configmaps, pvcs, etc. This filtering applies to get/list/watch calls as well.

We will not be able to use the generic policy engine for this, as it’s not possible to express rules for this scenario. Instead, we’ll likely need to create a new authorizer with hard-coded rules. We can make this work by building a ref-counter based summary of referenceable objects based on a watch of a subset of resources.

Authorizer/Admission:

  • identify requests from particular nodes (pattern already in place with node TLS bootstrapper, builds a user name the node name can be extracted from)
  • build the graph of objects referenced by pods bound to the node
  • limit the pods that a node is inherently allowed to create (only static pods, only bound to itself, no secret/pvc/configmap references)

New API access patterns needed by the kubelet (only needed if we want to allow list/watch of secrets): moved to kubernetes/community#443

  • {list,watch,get} {secrets,configmaps,persistentvolumeclaims,...} used by pods bound to the node
@liggitt liggitt added area/configmap-api area/kubelet area/secret-api area/security kind/enhancement sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Jan 25, 2017
@liggitt liggitt self-assigned this Jan 25, 2017
@liggitt
Copy link
Member Author

liggitt commented Jan 25, 2017

cc @kubernetes/sig-auth-feature-requests @kubernetes/sig-node-feature-requests

@ericchiang
Copy link
Contributor

Do we have any existing APIs that filter list and watch results like this?

Would it be reasonable to implement restrictions for gets on {secrets, configmaps, persistentvolumeclaim}, and updates on pods, then defer futher restrictions on {list,watch} operations once we figure out filtering? Or do nodes {list,watch} secrets today?

@deads2k
Copy link
Contributor

deads2k commented Jan 25, 2017

Or do nodes {list,watch} secrets today?

they do not

Do we have any existing APIs that filter list and watch results like this?

Yes, label and field selection filters can be applied and they are available (and restrictable) on an http request, so it would be consistent to enforce it with an authorizer.

@timstclair
Copy link

I'm assuming the motivation here is to prevent a node compromise from being equivalent to a cluster compromise, by limiting the potential damage to only resources scheduled (transitively) to the node. Given this goal, what's to stop an attacker from simply updating a pod spec to reference the resource they want access to? An obvious solution is to prevent the node from posting anything other than status updates, but this would break the implementation of static pods on the node (via mirror pods).

Are there any other motivations to this proposal, or do we need to tackle these issues first?

@liggitt
Copy link
Member Author

liggitt commented Jan 25, 2017

I'm assuming the motivation here is to prevent a node compromise from being equivalent to a cluster compromise, by limiting the potential damage to only resources scheduled (transitively) to the node.

Correct.

Given this goal, what's to stop an attacker from simply updating a pod spec to reference the resource they want access to?

Pod spec is immutable. I would see nodes as being limited to:

  • create mirror pod bound to self (which already cannot reference secrets)
  • update pod status for pod bound to self

An obvious solution is to prevent the node from posting anything other than status updates, but this would break the implementation of static pods on the node (via mirror pods).

Mirror pods are not allowed to reference secrets

@timstclair
Copy link

Cool, good to see this is already in place. Perhaps we should extend the limits to prevent certain volume types as well?

@thockin
Copy link
Member

thockin commented Jan 26, 2017

Would this allow something like "watch all secrets in all namespaces" to server-side filter and return only secrets on this node? Or would it be an error, and we would have to set up N individual watches ourself?

@liggitt
Copy link
Member Author

liggitt commented Jan 26, 2017

Would this allow something like "watch all secrets in all namespaces" to server-side filter and return only secrets on this node? Or would it be an error, and we would have to set up N individual watches ourself?

For lists/watches, the request for ACL-filtered secrets would likely need to be explicit (possibly something explicitly requesting "secrets for pods for node X", which is gorpy to express, or bespoke resource endpoints for nodes, which is gorpy to add for just one type of user). Adding ACL filtering to existing list/watch APIs would be... unexpected.

We'd like to find a way to let nodes watch the secrets they're supposed to have access to instead of making them poll or manage lots of individual watches, but even protecting the existing GET requests based on the node->pod->secret relationship would be an improvement.

@liggitt liggitt changed the title kubelet API access should be subdivided kubelet access to API server should be subdivided Jan 27, 2017
@liggitt liggitt added this to the v1.7 milestone Mar 6, 2017
@wojtek-t
Copy link
Member

wojtek-t commented Mar 9, 2017

We'd like to find a way to let nodes watch the secrets they're supposed to have access to instead of making them poll or manage lots of individual watches, but even protecting the existing GET requests based on the node->pod->secret relationship would be an improvement.

I agree. Though, since we would like to change kubelet to actually "list+watch" secrets, configmaps, etc., I'm going to put together a proposal for this part once I'm done with 1.6 work. I will focus more on that part, but this will be strictly related to this one.
And I agree that the request should be "explicitly asking for my secrets" not implicit filtering. I will try to put some proposal together in a week or two.

@wojtek-t
Copy link
Member

So I created the initial draft of my proposal here: kubernetes/community#443
It still has some TODOs to think about it, and I will work on improving the doc itself more towards end of 1.6, but the high level idea should probably be roughly clear.

@timstclair
Copy link

Can we dedupe this with kubernetes/enhancements#279 ?

@liggitt
Copy link
Member Author

liggitt commented May 14, 2017

closing in favor of kubernetes/enhancements#279

@liggitt liggitt closed this as completed May 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/configmap-api area/kubelet area/secret-api area/security sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

No branches or pull requests

6 participants