Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secrets proposal #4126

Merged
merged 1 commit into from Feb 18, 2015
Merged

Secrets proposal #4126

merged 1 commit into from Feb 18, 2015

Conversation

pmorie
Copy link
Member

@pmorie pmorie commented Feb 4, 2015

A proposal for the distribution of secrets (passwords, keys, etc) to containers inside Kubernetes using a custom volume type.

@roberthbailey
Copy link
Contributor

/cc @erictune

@smarterclayton
Copy link
Contributor

This is subordinate to #4029 - high level discussion there (argument about the linkage between container security context, service account, and secrets). Also see the original service accounts proposal (linked over there).

@markturansky
Copy link
Contributor

How do the secrets get onto the host by the volume plugin? It seems implied that the secrets would be read from a store by the plugin and written to local disk? No actual watching pods or attaching required. The plugin is a pass through to the secret data?

@pmorie
Copy link
Member Author

pmorie commented Feb 5, 2015

@markturansky Your reading of the implication is correct (at least in this iteration of our design); the volume plugin is responsible for fetching the needed secrets and writing them to the local disk.

@erictune
Copy link
Member

erictune commented Feb 5, 2015

Looking forward to giving this a close read on monday.
On Feb 4, 2015 4:49 PM, "Paul Morie" notifications@github.com wrote:

@markturansky https://github.com/markturansky Your reading of the
implication is correct (at least in this iteration of our design); the
volume plugin is responsible for fetching the needed secrets and writing
them to the local disk.


Reply to this email directly or view it on GitHub
#4126 (comment)
.

@markturansky
Copy link
Contributor

@pmorie so, do you really need a file or do you actually want a stream? The volume plugin would need to read the secrets from somewhere, too. Why write to disk at all?

@markturansky
Copy link
Contributor

@pmorie nevermind, i see you want the various tools to read from those config files. And the volume plugin would clean up after itself, like EmptyDir.

@pmorie
Copy link
Member Author

pmorie commented Feb 5, 2015

@markturansky Do you think that should be made more clear in the proposal?

@markturansky
Copy link
Contributor

It's interesting that you want to make it a volume, because i'm looking into that same stuff right now. There seem to be strong parallels with what I'm doing.

We both want to:

  • Allow a user to "Define something that persists that I can mount"
  • Allow me to put my secret data in there ('secrets' and my MySQL database are both secret to our namespaces)
  • Allow that data to live longer than my pod and follow the pod around the cluster.
  • Your data is read-only many

2. As a user, I want to allow containers to consume supplemental information about services such
as username and password which should be kept secret, so that I can share secrets about a
service amongst the containers in my application securely

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are operator secrets global? Meaning, is that one .dockercfg for all pods or do you intend to allow each pod to have their own .dockercfg?

If global, would would you schedule that resource onto all hosts exactly once? If on a pod basis, how would you inject that operator secret volume into the pods?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Feb 6, 2015, at 10:24 AM, Mark Turansky notifications@github.com wrote:

In docs/design/secrets.md:

+## Use Cases
+
+1. As a user, I want to store secret artifacts for my applications and consume them securely in

  • containers, so that I can keep the configuration for my applications separate from the images
  • that use them:
    1. As a cluster operator, I want to allow a pod to access the Kubernetes master using a custom
  •    `.kubeconfig` file, so that I can securely reach the master
    
    1. As a cluster operator, I want to allow a pod to access a Docker registry using credentials
  •    from a `.dockercfg` file, so that containers can push images
    
    1. As a cluster operator, I want to allow a pod to access a git repository using SSH keys,
  •    so that I can push and fetch to and from the repository
    
    +2. As a user, I want to allow containers to consume supplemental information about services such
  • as username and password which should be kept secret, so that I can share secrets about a
  • service amongst the containers in my application securely

Are operator secrets global? Meaning, is that one .dockercfg for all pods or do you intend to allow each pod to have their own .dockercfg?

Both. Pods carry with them authorization info about the ability to pull images.
If global, would would you schedule that resource onto all hosts exactly once? If on a pod basis, how would you inject that operator secret volume into the pods?


Reply to this email directly or view it on GitHub.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this Proposal doesn't yet say how you would use this for a .dockercfg. Kubelet would need to:

  1. pull the secret with the per-pod dockercfg
  2. merge that into the actual system dockercfg
  3. pull the pod's images
  4. somehow make sure that other pods that should not have access to that secret do use it for their images via some sort of race condition.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somehow make sure that other pods that should not have access to that secret do use it for their images via some sort of race condition.

@erictune confirm that you meant "don't" instead of "do" here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i meant don't

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erictune Reading over this again and discussing it with @ncdc, I wonder if there's some stream-crossing in the document. In this case I'm referring to a use-case in OpenShift of wanting to push to a docker registry, rather than authorizing the kubelet to use a .dockercfg contained in a secret to docker pull an image to run in a pod.

I did a quick look for the use case I described above (kubelet uses a secret as credential for docker pull) and couldn't find anything which I felt already captured it, though it's hinted at in:

#2209
#499

Do we need to call this out as a use-case to solve with secrets @erictune @smarterclayton ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Feb 12, 2015, at 8:15 PM, Paul Morie notifications@github.com wrote:

In docs/design/secrets.md:

+## Use Cases
+
+1. As a user, I want to store secret artifacts for my applications and consume them securely in

  • containers, so that I can keep the configuration for my applications separate from the images
  • that use them:
    1. As a cluster operator, I want to allow a pod to access the Kubernetes master using a custom
  •    `.kubeconfig` file, so that I can securely reach the master
    
    1. As a cluster operator, I want to allow a pod to access a Docker registry using credentials
  •    from a `.dockercfg` file, so that containers can push images
    
    1. As a cluster operator, I want to allow a pod to access a git repository using SSH keys,
  •    so that I can push and fetch to and from the repository
    
    +2. As a user, I want to allow containers to consume supplemental information about services such
  • as username and password which should be kept secret, so that I can share secrets about a
  • service amongst the containers in my application securely

@erictune Reading over this again and discussing it with @ncdc, I wonder if there's some stream-crossing in the document. In this case I'm referring to a use-case in OpenShift of wanting to push to a docker registry, rather than authorizing the kubelet to use a .dockercfg contained in a secret to docker pull an image to run in a pod.

I did a quick look for the use case I described above (kubelet uses a secret as credential for docker pull) and couldn't find anything which I felt already captured it, though it's hinted at in:

#2209
#499

Do we need to call this out as a use-case to solve with secrets @erictune @smarterclayton ?

Yes, we discussed this today. It's two different use cases, pull from an integrated registry is an Openshift and CoreOS problem, and it's generally useful to the community.

Reply to this email directly or view it on GitHub.

@markturansky
Copy link
Contributor

I think I know why this overlaps so much. Data are secrets or secrets are data.

The secret volume plugin would implement the actual retrieval and laying down of secrets on the Kubelet's file system. Secrets may be stored in a special secret registry, as Docker volumes, LDAP etc.

Many data stores could be supported by implementing many volume plugins, as you are doing with a secret volume plugin. All secrets are pets, so a plugin could support any means of obtaining that data.

If that is true, then the operator can describe all his secrets as a set of volumes (secret volume or otherwise) in yaml or json.

The user wanting to store a secret has to request a resource (the secret volume), put the secret in the volume, and then reference that volume in the pod.

@smarterclayton
Copy link
Contributor

Also, any data written to a disk is done in a security context (a Linux user uid and mcs labels), and for a disk to be reused it needs to be within the same security context or a permission change needs to occur. All disk sharing must be done within a single security context (the underlying uid when user namespaces are in use).

On Feb 6, 2015, at 9:21 PM, Mark Turansky notifications@github.com wrote:

I think I know why this overlaps so much. Data are secrets or secrets are data.

The secret volume plugin would implement the actual retrieval and laying down of secrets on the Kubelet's file system. Secrets may be stored in a special secret registry, as Docker volumes, LDAP etc.

Many data stores could be supported by implementing many volume plugins, as you are doing with a secret volume plugin. All secrets are pets, so a plugin could support any means of obtaining that data.

If that is true, then the operator can describe all his secrets as a set of volumes (secret volume or otherwise) in yaml or json.

The user wanting to store a secret has to request a resource (the secret volume), put the secret in the volume, and then reference that volume in the pod.


Reply to this email directly or view it on GitHub.

@markturansky
Copy link
Contributor

@smarterclayton wouldn't those security features be similarly useful in persistent volumes?

So long as secrets (from any store) are expressed as volumes, couldn't this entire feature be implemented using persistent volumes w/ the additional security you mention?

The only piece missing is the once-per-host volume, which I'm sure is easily accomplished because there is, i believe, a means of having the cluster run one type of pod per host for administration purposes.

There is so much overlap here that I wonder about the design of this one (or the other one). They seem essentially the same, though, just with different nouns, but both nouns (data and secrets) are both still secrets.

@smarterclayton
Copy link
Contributor

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com wrote:

@smarterclayton wouldn't those security features be similarly useful in persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't this entire feature be implemented using persistent volumes w/ the additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They have lots of similarities, which is good, but I don't think they are used the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is easily accomplished because there is, i believe, a means of having the cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact they are an implementation detail.
There is so much overlap here that I wonder about the design of this one (or the other one). They seem essentially the same, though, just with different nouns, but both nouns (data and secrets) are both still secrets.


Reply to this email directly or view it on GitHub.

@pmorie
Copy link
Member Author

pmorie commented Feb 8, 2015

I agree with @smarterclayton. There are a lot of similarities but there are
also significant differences in user experience. For example, I don't see
much of a use case to store content in a persistent volume via the REST
api, but that is definitely how users will expect to create secrets.
On Sat, Feb 7, 2015 at 6:35 PM Clayton Coleman notifications@github.com
wrote:

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com
wrote:

@smarterclayton wouldn't those security features be similarly useful in
persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't
this entire feature be implemented using persistent volumes w/ the
additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They
have lots of similarities, which is good, but I don't think they are used
the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is
easily accomplished because there is, i believe, a means of having the
cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact
they are an implementation detail.
There is so much overlap here that I wonder about the design of this one
(or the other one). They seem essentially the same, though, just with
different nouns, but both nouns (data and secrets) are both still secrets.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub
#4126 (comment)
.

@smarterclayton
Copy link
Contributor

We should catalog similarities, but not get hung up on perfectly aligning them.

For example, we will eventually want to generate secrets on demand for containers (mint a .kubeconfig that is automatically refreshed on demand, mint private keys that can be shared to all pods in a pool), and persistent volumes may have some use cases for pre generated data, but the problem domains really don't overlap.

On Feb 7, 2015, at 7:20 PM, Paul Morie notifications@github.com wrote:

I agree with @smarterclayton. There are a lot of similarities but there are
also significant differences in user experience. For example, I don't see
much of a use case to store content in a persistent volume via the REST
api, but that is definitely how users will expect to create secrets.
On Sat, Feb 7, 2015 at 6:35 PM Clayton Coleman notifications@github.com
wrote:

On Feb 7, 2015, at 10:11 AM, Mark Turansky notifications@github.com
wrote:

@smarterclayton wouldn't those security features be similarly useful in
persistent volumes?

What I was describing was for persistent volumes.
So long as secrets (from any store) are expressed as volumes, couldn't
this entire feature be implemented using persistent volumes w/ the
additional security you mention?

The use cases for persistent volumes and secrets aren't the same. They
have lots of similarities, which is good, but I don't think they are used
the same way or users experience them the same way.
The only piece missing is the once-per-host volume, which I'm sure is
easily accomplished because there is, i believe, a means of having the
cluster run one type of pod per host for administration purposes.

I don't think once per host volumes are the key part of secrets, in fact
they are an implementation detail.
There is so much overlap here that I wonder about the design of this one
(or the other one). They seem essentially the same, though, just with
different nouns, but both nouns (data and secrets) are both still secrets.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub
#4126 (comment)
.


Reply to this email directly or view it on GitHub.

@markturansky
Copy link
Contributor

Different users manage secrets than persistent volumes. Agreed, secrets should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a volume, we have a common scheduling problem: getting the right volumes to the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling would intrinsically be able to schedule your volumes. All you need to do is put a claim on it.

We might be able to handle this distribution of volumes to hosts (the scheduling problem) the same way.

@smarterclayton
Copy link
Contributor

I don't see how secrets are a scheduling problem?

----- Original Message -----

Different users manage secrets than persistent volumes. Agreed, secrets
should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a
volume, we have a common scheduling problem: getting the right volumes to
the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling
would intrinsically be able to schedule your volumes. All you need to do is
put a claim on it.

We might be able to handle this distribution of volumes to hosts (the
scheduling problem) the same way.


Reply to this email directly or view it on GitHub:
#4126 (comment)

@markturansky
Copy link
Contributor

I'm sure you're right. Secrets, by definition, are pets and so are volumes. I took matching pets to hosts/pods as the common scheduling problem/pattern.

----- Original Message -----
From: "Clayton Coleman" notifications@github.com
To: "GoogleCloudPlatform/kubernetes" kubernetes@noreply.github.com
Cc: "Mark Turansky" mturansk@redhat.com
Sent: Tuesday, February 10, 2015 2:55:55 PM
Subject: Re: [kubernetes] WIP: Secrets proposal / prototype (#4126)

I don't see how secrets are a scheduling problem?

----- Original Message -----

Different users manage secrets than persistent volumes. Agreed, secrets
should not be implemented as volumes.

SecretSource is the common denominator. If you want to adapt the secret to a
volume, we have a common scheduling problem: getting the right volumes to
the right hosts and pods.

Or perhaps put another way, the volume controller I'm making for scheduling
would intrinsically be able to schedule your volumes. All you need to do is
put a claim on it.

We might be able to handle this distribution of volumes to hosts (the
scheduling problem) the same way.


Reply to this email directly or view it on GitHub:
#4126 (comment)


Reply to this email directly or view it on GitHub:
#4126 (comment)

@pmorie pmorie force-pushed the secrets_proposal branch 2 times, most recently from 375d4bf to 3d37204 Compare February 11, 2015 04:59
@pmorie
Copy link
Member Author

pmorie commented Feb 11, 2015

Rebased into a single commit for ease of rebase.

In use cases for CRUD operations, the user experience for secrets should be no different than for
other API resources.

TODO: document possible relation with Service Accounts
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and Security Contexts #3910

#### Example: service account consumes auth token secret

As an example, the service account proposal discusses service accounts consuming secrets which
contain kubernetes auth tokens. When a Kubelet starts a pod associates with a service account
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/starts a pod associates/starts a pod that associates/ ?

@pmorie pmorie force-pushed the secrets_proposal branch 3 times, most recently from 27d8018 to e26050a Compare February 17, 2015 21:40
@pmorie
Copy link
Member Author

pmorie commented Feb 17, 2015

I think it should be a validation error if you declare a VolumeMount with readonly = false on a secret volume, thoughts @EricMountain-1A @erictune @smarterclayton


For now, we will not implement validations around these limits. Cluster operators will decide how
much node storage is allocated to secrets. It will be the operator's responsibility to ensure that
the allocated storage is sufficient for the workload scheduled onto a node.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other option is for kubelet to add up the size of all the secrets files as it writes them to tmpfs, and then to subtract this total from the memory limit that it writes for the container. This makes most or all of the limits mentioned above go away.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, IIRC we only have per-container memory limits and not per-pod memory limits, at present. So, either you have to decide how to apportion the shortfall across the various containers (proportional?), or else charge it to the pod as a whole, which means the scheduler can't see the charge at scheduling time.

@erictune
Copy link
Member

The design doc LGTM. Happy to merge that as is.

The new volume type definition also looks good.

Are you ready to have the volume plugin code reviewed? I'm fine with reviewing what you have and then working on the /tmpfs stuff in a separate PR.

@erictune
Copy link
Member

If you want to break this into 3 PRs:

  1. design doc
  2. secrets type definition
  3. volume plugin and secrets watching impl.

I won't complain. 😄

@pmorie
Copy link
Member Author

pmorie commented Feb 18, 2015

@erictune thanks for the feedback. I will split out the PRs as you
suggested.
On Tue, Feb 17, 2015 at 7:52 PM Eric Tune notifications@github.com wrote:

If you want to break this into 3 PRs:

  1. design doc
  2. secrets type definition
  3. volume plugin and secrets watching impl.

I won't complain. [image: 😄]


Reply to this email directly or view it on GitHub
#4126 (comment)
.

@pmorie pmorie force-pushed the secrets_proposal branch 2 times, most recently from 361b977 to c80ec0a Compare February 18, 2015 01:28
@pmorie pmorie changed the title WIP: Secrets proposal / prototype Secrets proposal Feb 18, 2015
@pmorie
Copy link
Member Author

pmorie commented Feb 18, 2015

@erictune I have turned this into the PR for the proposal, separate PRs coming for type definition (needs a little cleanup and final review) and the volume plugin (where we can work through tmpfs)

This was referenced Feb 18, 2015
erictune added a commit that referenced this pull request Feb 18, 2015
@erictune erictune merged commit e045b19 into kubernetes:master Feb 18, 2015
@thockin
Copy link
Member

thockin commented Feb 20, 2015

I just wanted to say that I finally read this whole thing and (aside from some nits which I will address as a PR or just ignore) I found it to be incredibly well written and thorough. This is a model design doc.

@erictune
Copy link
Member

+1

On Thu, Feb 19, 2015 at 8:56 PM, Tim Hockin notifications@github.com
wrote:

I just wanted to say that I finally read this whole thing and (aside from
some nits which I will address as a PR or just ignore) I found it to be
incredibly well written and thorough. This is a model design doc.


Reply to this email directly or view it on GitHub
#4126 (comment)
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants