10 ms·

One thing that is not super clear from the blog post, and which may be a surpsise to some, is that Cloud Functions will require you to upload your source code.

by st3fan 8y ago

stewart27 8y ago
Indeed. If you want to compile the binary yourself/don't want to upload source code, we have a serverless containers product currently available as an early preview (sign up at g.co/serverlesscontainers). This would allow you to compile your binary locally, write a simple Dockerfile and then build/deploy the resulting container.
- sjroot 8y ago
  This is great. It sounds like what people were hoping would come from Zeit Now, though in their latest major release they moved away from serverless docker deployments. Their community was quite upset at that decision I think. I will have to check this out!
benatkin 8y ago
This is a much better solution, I think. Uploading your source code is often easier than uploading your binaries. If, for example, you have a new contributor you don't yet trust creating a cloud function in Go and you know enough Go to review the code, but it's not convenient for you to compile it yourself, you can just upload the code yourself. You can't easily review the binary.
- vira28 8y ago
  If i understand correctly the binary is created from the source code right. Am i missing anything?
  - benatkin 8y ago
    On AWS Lambda, which has had Go support for almost a year now, you upload the binary, which you create from the source code on your own system, using the go compiler. On Google Cloud Functions, you upload the source code and Google's servers build it.
- st3fan 8y ago
  I don't really understand what you are saying here .. It should not be difficult to run `go build` locally and then tell a tool to upload `foobar` instead of `main.go`. Uploading source code is a dealbreaker for me because I do not trust Google or myself. Google probably has language in their terms that they may do whatever they want with anything you upload to them. I also do not trust myself to properly configure security parameters around this with the risk of leaking source code. Binaries are a great strength of Go. Too bad this platform doesn't use them. I will take a peek at the new Container platform that they are building, but that sounds more heavyweight, which probably comes with a higher price. (I have not seen any actual pricing)
  - nikki93 8y ago
    can't your security params be read out of the binary in any case? i'm not sure how binaries offer more security.
    - weberc2 8y ago
      His post is super unclear but I think he is concerned about someone getting their hands on his source code. In general, a compiled Go binary is not trivial to reverse engineer, so there’s some security through obscurity there.
      - st3fan 8y ago
        Not 'someone'. Google.
        nikki93 8y ago
        Your secrets are not safe by embedding them in a binary. Compilation is not (necessarily, and at least so in the Go compiler's case) security...
      - saagarjha 8y ago
        Not a whole lot, though. A determined reverse engineer will likely be able to grab something close to your source code without much difficulty, assuming you haven't applied any wacky obfuscation.
- omeid2 8y ago
  Your code review and deployment pipeline should be all automated from day zero. Setting up a CI pays off in the first couple of days in terms of time investment and massive returns on longer term mostly for avoiding human error and confusions that ensues.
  - benatkin 8y ago
    I didn't say that you don't need CI. I use CI regularly, and I think it's best for CI to be for testing, and to minimize the build artifacts except those used for test results. A lot of developers have this preference for client side code. Since the early days heroku has been building the client side bundles for rails.
    - weberc2 8y ago
      The parent’s point was that contributors shouldn’t have permissions to upload binaries; they only commit source code and your CI infrastructure compiles, releases, and deploys it.
  - solatic 8y ago
    CI is a continuous integration tool. There are inherent latencies to committing, pushing, deploying to AWS, running tests. Mature development tooling, if it's not REPL-based, will at least allow a local debugger to step through the code compiled/run locally, without these remote latencies. Both tools are necessary for fast engineering, but in different contexts.
- skywhopper 8y ago
  The fact that it's easier to do things this way just shows how immature the toolchain surrounding serverless development is. It shouldn't be difficult to simulate the serverless environment on a workstation, and yet somehow that's a huge gap, even for Lambda.
  - benatkin 8y ago
    Fair enough, but I don't think it's wrong to want a PaaS/FaaS that automagically compiles things. You'll be hard pressed to convince anyone who genuinely likes this approach to switch over to having their IDE, developer tools, or CI do it.
  - ricktdotorg 8y ago
    the Cloud Functions emulator works well, and helps https://cloud.google.com/functions/docs/emulator https://cloud.google.com/functions/docs/emulator
  - rakoo 8y ago
    Then again, what's the point of controlling your own repository of binaries if their only goal in life is to be deployed somewhere else anyway ? It's much easier to just cut the middleman.
stcredzero 8y ago
One thing that is not super clear from the blog post, and which may be a surpsise to some, is that Cloud Functions will require you to upload your source code. The compilation happens not on your end but on Google servers. This is probably not a surprise for folks who have been using Python or Node for cloud functions, but it is something to think about. I've been thinking of using this approach for a golang MMO game SaaS which is really an aggregation of a bunch of FaaS lambdas tied to an account. What upload allows you to do, is to do a syntactic scan of all of the code, particularly the package imports and function calls. All of the above can be whitelisted, which adds security. Also, you can munge the function calls against a generated munged version of the available libraries, which also adds security. (In addition to using containers under the covers.)
- winrid 8y ago
  Like Screeps? Even more niche? :)
  - stcredzero 8y ago
    That looks cool. What I'm describing is for implementing MMOs, though.
skywhopper 8y ago
I was wondering about this, given the examples. AWS's Golang support for Lambda involves including their library and passing your function to a Lambda initializer function. Which I'm sure is all that's happening with this, but it's weird to hand off the Go source to Google. I wonder if they are multiplexing multiple functions in single binaries or using special Go toolchains. Or just don't want to lock into specific backends yet.
- justasitsounds 8y ago
  > AWS's Golang support for Lambda involves including their library and passing your function to a Lambda initializer function AWS announced 'native' Golang support last year - since then there hasn't been a need to use a Python or node 'shim' to bootstrap the golang executable like this. You just upload your compiled binary to S3 and provide the path to AWS lambda
- DSingularity 8y ago
  That would be my guess. Since this is all event driven code, why wouldn’t they bundle them for improved system utilization?
  - sudhirj 8y ago
    Because Go's namespacing / packaging model doesn't have the kinds of isolation required to prevent you from accessing the code and variables of the other packages you've been bundled with, compared to NodeJS, which does (VM feature).
- zaphirplane 8y ago
  Unlikely Golang’s init block will get executed in all the multiplexed serverless functions that are bundled in the one binary. Which adds a side effect
- chrisbroadfoot 8y ago
  No, there's no real magic. Each function runs in its own process (which is inside a gVisor sandbox). The toolchain is completely unmodified. Disclaimer: I work at GCP and worked on this product.
sbr464 8y ago
Curious, wouldn’t you want the source code to compile on the final platform? (If not using docker) I’m not familiar with binary or lower level workings but I figured there would be certain benefits of compiling the same code on Linux vs Mac or Windows for example.
- slimsag 8y ago
  The Go compiler produces the same binary regardless of which host OS it is ran on. i.e., the OS you compile a Go program on doesn't matter (assuming pure Go code, not including C etc).
  - sbr464 8y ago
    Thx for the info, I don’t use Go typically.
  - black-tea 8y ago
    I think he was talking more about the machine architecture than the OS.
    - mjlee 8y ago
      As of Go 1.5 the compiler works for any supported target architecture.

by st3fan 8y ago

stewart27 8y ago

Indeed. If you want to compile the binary yourself/don't want to upload source code, we have a serverless containers product currently available as an early preview (sign up at g.co/serverlesscontainers). This would allow you to compile your binary locally, write a simple Dockerfile and then build/deploy the resulting container.

sjroot 8y ago
This is great. It sounds like what people were hoping would come from Zeit Now, though in their latest major release they moved away from serverless docker deployments. Their community was quite upset at that decision I think. I will have to check this out!

benatkin 8y ago

This is a much better solution, I think. Uploading your source code is often easier than uploading your binaries. If, for example, you have a new contributor you don't yet trust creating a cloud function in Go and you know enough Go to review the code, but it's not convenient for you to compile it yourself, you can just upload the code yourself. You can't easily review the binary.

vira28 8y ago
If i understand correctly the binary is created from the source code right. Am i missing anything?
- benatkin 8y ago
  On AWS Lambda, which has had Go support for almost a year now, you upload the binary, which you create from the source code on your own system, using the go compiler. On Google Cloud Functions, you upload the source code and Google's servers build it.
st3fan 8y ago
I don't really understand what you are saying here .. It should not be difficult to run `go build` locally and then tell a tool to upload `foobar` instead of `main.go`. Uploading source code is a dealbreaker for me because I do not trust Google or myself. Google probably has language in their terms that they may do whatever they want with anything you upload to them. I also do not trust myself to properly configure security parameters around this with the risk of leaking source code. Binaries are a great strength of Go. Too bad this platform doesn't use them. I will take a peek at the new Container platform that they are building, but that sounds more heavyweight, which probably comes with a higher price. (I have not seen any actual pricing)
- nikki93 8y ago
  can't your security params be read out of the binary in any case? i'm not sure how binaries offer more security.
  - weberc2 8y ago
    His post is super unclear but I think he is concerned about someone getting their hands on his source code. In general, a compiled Go binary is not trivial to reverse engineer, so there’s some security through obscurity there.
    - st3fan 8y ago
      Not 'someone'. Google.
      - nikki93 8y ago
        Your secrets are not safe by embedding them in a binary. Compilation is not (necessarily, and at least so in the Go compiler's case) security...
    - saagarjha 8y ago
      Not a whole lot, though. A determined reverse engineer will likely be able to grab something close to your source code without much difficulty, assuming you haven't applied any wacky obfuscation.
omeid2 8y ago
Your code review and deployment pipeline should be all automated from day zero. Setting up a CI pays off in the first couple of days in terms of time investment and massive returns on longer term mostly for avoiding human error and confusions that ensues.
- benatkin 8y ago
  I didn't say that you don't need CI. I use CI regularly, and I think it's best for CI to be for testing, and to minimize the build artifacts except those used for test results. A lot of developers have this preference for client side code. Since the early days heroku has been building the client side bundles for rails.
  - weberc2 8y ago
    The parent’s point was that contributors shouldn’t have permissions to upload binaries; they only commit source code and your CI infrastructure compiles, releases, and deploys it.
- solatic 8y ago
  CI is a continuous integration tool. There are inherent latencies to committing, pushing, deploying to AWS, running tests. Mature development tooling, if it's not REPL-based, will at least allow a local debugger to step through the code compiled/run locally, without these remote latencies. Both tools are necessary for fast engineering, but in different contexts.
skywhopper 8y ago
The fact that it's easier to do things this way just shows how immature the toolchain surrounding serverless development is. It shouldn't be difficult to simulate the serverless environment on a workstation, and yet somehow that's a huge gap, even for Lambda.
- benatkin 8y ago
  Fair enough, but I don't think it's wrong to want a PaaS/FaaS that automagically compiles things. You'll be hard pressed to convince anyone who genuinely likes this approach to switch over to having their IDE, developer tools, or CI do it.
- ricktdotorg 8y ago
  the Cloud Functions emulator works well, and helps https://cloud.google.com/functions/docs/emulator https://cloud.google.com/functions/docs/emulator
- rakoo 8y ago
  Then again, what's the point of controlling your own repository of binaries if their only goal in life is to be deployed somewhere else anyway ? It's much easier to just cut the middleman.

stcredzero 8y ago

One thing that is not super clear from the blog post, and which may be a surpsise to some, is that Cloud Functions will require you to upload your source code. The compilation happens not on your end but on Google servers. This is probably not a surprise for folks who have been using Python or Node for cloud functions, but it is something to think about. I've been thinking of using this approach for a golang MMO game SaaS which is really an aggregation of a bunch of FaaS lambdas tied to an account. What upload allows you to do, is to do a syntactic scan of all of the code, particularly the package imports and function calls. All of the above can be whitelisted, which adds security. Also, you can munge the function calls against a generated munged version of the available libraries, which also adds security. (In addition to using containers under the covers.)

winrid 8y ago
Like Screeps? Even more niche? :)
- stcredzero 8y ago
  That looks cool. What I'm describing is for implementing MMOs, though.

skywhopper 8y ago

I was wondering about this, given the examples. AWS's Golang support for Lambda involves including their library and passing your function to a Lambda initializer function. Which I'm sure is all that's happening with this, but it's weird to hand off the Go source to Google. I wonder if they are multiplexing multiple functions in single binaries or using special Go toolchains. Or just don't want to lock into specific backends yet.

justasitsounds 8y ago
> AWS's Golang support for Lambda involves including their library and passing your function to a Lambda initializer function AWS announced 'native' Golang support last year - since then there hasn't been a need to use a Python or node 'shim' to bootstrap the golang executable like this. You just upload your compiled binary to S3 and provide the path to AWS lambda
DSingularity 8y ago
That would be my guess. Since this is all event driven code, why wouldn’t they bundle them for improved system utilization?
- sudhirj 8y ago
  Because Go's namespacing / packaging model doesn't have the kinds of isolation required to prevent you from accessing the code and variables of the other packages you've been bundled with, compared to NodeJS, which does (VM feature).
zaphirplane 8y ago
Unlikely Golang’s init block will get executed in all the multiplexed serverless functions that are bundled in the one binary. Which adds a side effect
chrisbroadfoot 8y ago
No, there's no real magic. Each function runs in its own process (which is inside a gVisor sandbox). The toolchain is completely unmodified. Disclaimer: I work at GCP and worked on this product.

sbr464 8y ago

Curious, wouldn’t you want the source code to compile on the final platform? (If not using docker) I’m not familiar with binary or lower level workings but I figured there would be certain benefits of compiling the same code on Linux vs Mac or Windows for example.

slimsag 8y ago
The Go compiler produces the same binary regardless of which host OS it is ran on. i.e., the OS you compile a Go program on doesn't matter (assuming pure Go code, not including C etc).
- sbr464 8y ago
  Thx for the info, I don’t use Go typically.
- black-tea 8y ago
  I think he was talking more about the machine architecture than the OS.
  - mjlee 8y ago
    As of Go 1.5 the compiler works for any supported target architecture.