5 ms·

Writing HTTP Middleware in Go

112 points by babawere 13y ago

latch 13y ago
I find the use of httptest.ResponseRecorder as well as the proposed context pattern not very elegant. Just because go's HTTP entry point is an http.HandlerFunc, doesn't mean your middleware need to implement that same method. I'd prefer: type Middleware func(context *Context) Response type Context struct { Req *http.Request Token string } type Response interface { Status() int .... } No global variable, and definitely no contention against that global. (note, I'd make the middlewares nesteable, passing a next middleware to each middleware, but that wasn't covered in the article).
- sjustinas 13y ago
  Author here. Thanks for your input, but I'd like to clarify some points. The blog post's intent was not to show how to implement your own middleware mechanism in a web framework you're writing, but how to create HTTP middleware using the standard Go net/http interface. Your proposed middleware type might feel a bit nicer, but it disregards the standard net/http interface and makes your middleware incompatible with it. Well, you could in theory make it compatible, but you would need some more "glue" and hacks. net/http doesn't really work the way you're proposing (returning a Response object, etc.). Please see my other post on the standard Go HTTP tools: http://justinas.org/embrace-gos-http-tools/ http://justinas.org/embrace-gos-http-tools/ If you don't want full compatibility with existing net/http libraries, then sure – there are nicer ways. Again, my intention was to show how to do it with net/http and I agree that there are certain trade-offs. "(note, I'd make the middlewares nesteable, passing a next middleware to each middleware, but that wasn't covered in the article)." It was covered to some extent. In a model where every middleware is http.Handler itself, you nest your middlewares like: finalHandler := Middleware1(Middleware2(MyApp)) So every middleware does actually get a reference to the next handler that should run.
  - latch 13y ago
    I guess we'll disagree about whether eliminating a global variable [in a multi-threaded language] is nothing more than "might feel a bit nicer". Same with using a class which specifically exists for testing. net/http works fine with a response interface: context := newContext(req) response := chain.yield(context) // the exact same code you had: for k, v := range response.Header() { w.Header()[k] = v } ... This has the added benefit of letting you use different response implementations. A streaming implementation, an in-memory implementation, or, probably the most important if you're doing high traffic, one that relies on a pre-allocated byte-pool. A little nitpick on the side, new(sync.RWMutex) isn't needed. You can just do var cmMutex sync.RWMutex (http://golang.org/src/pkg/net/http/server.go?s=45646:46651#L877 http://golang.org/src/pkg/net/http/server.go?s=45646:46651#L...) EDIT: Sorry, not trying to be negative (so sick of this on here), but I do think global variables are really, really bad and we probably shouldn't be promoting patterns that rely on them.
    - sjustinas 13y ago
      Sure, I didn't say it was impossible to convert your proposed Response type to a proper net/http response when you need it. But, like I said and you just demonstrated, glue code is needed to do that. That makes people write differing middleware application mechanisms that are incompatible with each other. Sure, ResponseRecorder is placed in net/http/httptest and that makes sense. But then again, nothing in the actual implementation is integrated with the testing package or tied to testing in general. IMO there's nothing that makes it unsuitable for other usage. It's a simple buffer to accumulate HTTP, that's it. Even if one were to implement a buffering writer like that on their own, it wouldn't differ too much.
bfrog 13y ago
The request context is I think the most annoying part of go http handling. Oh you want to wrap your handler with something that does db.Begin()/db.Commit()? Well you need to stuff that tx somewhere... so then your options are a global map[*http.Request]interface{} with mutexes or pass it as an arg. Passing it in as an arg means you either lose compile time type checking because you use reflect or lose the ability to add more middlewares that build up request context. The map[]interface{} also loses compile time type checking for the most part unless you do something per type of context, with more mutexes. The second option is the sanest, but means either reflect if you have more than one context arg to build out (sql.Tx + cache + oauth grant + whatever else maybe?) or you have some common context object that may or may not use everything every request and your just adding more context members all the time that may or may not be initialized for that particular request. Sometimes I miss the dynamic typing and/or compile time type templating.
- codegangsta 13y ago
  I find the args approach the most appealing to me. Yeah you lose some compile time safety. Like you said the other option is a global context with mutexes which still somewhat misses some compile time safety. It basically boils down to reflection over type assertions. I think Martini handles this pretty darn well if I do say so myself ;)
  - mmgutz 13y ago
    I think you get the best of both worlds with `gocraft/web`. I feel args when you inject 3 or more (db, session ...) is cumbersome. `gocraft/web` is faster too since it has less reflection.
    - codegangsta 13y ago
      Agreed. I really like how gocraft does contexts. Very cool way to structure your app
- acrispino 13y ago
  You can use closures: func SomeHandler(foo Foo) { return func(w http.ResponseWriter, req *http.Request) { // use foo } } // later, during init http.HandleFunc("/foo", SomeHandler(someFoo)) or method values, introduced in go 1.1: type handler struct { foo Foo } func (h *handler) bar(w http.ResponseWriter, req *http.Request) { // use h.foo } func (h *handler) qux(w http.ResponseWriter, req *http.Request) { // use h.foo } // later h := handler{ foo: someFoo, } http.HandleFunc("/bar", h.bar) http.HandleFunc("/qux", h.qux)
tedchs 13y ago
In the Ruby community, the Rack interface has done a lot to enable Web middleware to work together. It would be great if there were an analogous interface for Go's net/http.
- politician 13y ago
  There is, it's the Handler interface. http://slid.es/andybons/high-performance-web-applications-in-go http://slid.es/andybons/high-performance-web-applications-in...
optymizer 13y ago
> a painless and even fun process This claim might be a little too optimistic. I'd like to challenge you (everyone) to associate a User object with an http.Request. Example code in Node.js/ExpressJS: function auth(req, res, next) { req.User = User("username"); next(); } This is such a simple example of a useful piece of middleware, yet it is so difficult to implement in Go in a 'painless and fun' way. I would prefer to avoid contention on a global mutex. I would prefer to keep the compatibility with http.Handler and http.HandlerFunc. Good luck!
- cypriss 13y ago
  I believe this is solved with gocraft/web: https://github.com/gocraft/web https://github.com/gocraft/web The middleware would be, per the example there: func (c *YourContext) UserRequired(rw web.ResponseWriter, r *web.Request, next web.NextMiddlewareFunc) { user := userFromSession(r) // Pretend like this is defined. It reads a session cookie and returns a *User or nil. if user != nil { c.User = user next(rw, r) } else { rw.Header().Set("Location", "/") rw.WriteHeader(http.StatusMovedPermanently) // do NOT call next() } } Compatibility with Handler/HandlerFunc is kept at the top level, but not at every step along the middleware chain.
- elithrar 13y ago
  To be fair, Express is a web framework. Go's net/http package is a little lower level than that. Still, this is possible if you make up for the lack of a framework by writing some code yourself. I'd use (and I do use) gorilla/context[1] to allow me to pass data in the request context between middleware layers (i.e. CSRF middleware passes the token along). You could write a couple of little helpers using gorilla/context (SetUser/GetUser, or more generically, SetString/GetString) and SetUser(username, r) in your middleware. The helpers (see the context docs!) just wrap a type assertion on the Get side, and set a map value on the Set side. // Compatible with http.HandlerFunc func auth(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { SetUser(r, "dave") h.ServeHTTP(w, r) } } // Routes r.HandleFunc("/admin", use(adminHandler, auth) http.Handle("/", r) // Allows you to chain your middleware rather than wrapping functions func use(h http.HandlerFunc, middleware ...func(http.HandlerFunc) http.HandlerFunc) http.HandlerFunc { for _, m := range middleware { h = m(h) } return h } You can mod this to work with http.Handler if you want: replace HandlerFunc and then write r.Handle("/admin", use(http.HandlerFunc(adminHandler), auth) instead. [1]: http://www.gorillatoolkit.org/pkg/context http://www.gorillatoolkit.org/pkg/context

112 points by babawere 13y ago

latch 13y ago

I find the use of httptest.ResponseRecorder as well as the proposed context pattern not very elegant. Just because go's HTTP entry point is an http.HandlerFunc, doesn't mean your middleware need to implement that same method. I'd prefer: type Middleware func(context *Context) Response type Context struct { Req *http.Request Token string } type Response interface { Status() int .... } No global variable, and definitely no contention against that global. (note, I'd make the middlewares nesteable, passing a next middleware to each middleware, but that wasn't covered in the article).

sjustinas 13y ago
Author here. Thanks for your input, but I'd like to clarify some points. The blog post's intent was not to show how to implement your own middleware mechanism in a web framework you're writing, but how to create HTTP middleware using the standard Go net/http interface. Your proposed middleware type might feel a bit nicer, but it disregards the standard net/http interface and makes your middleware incompatible with it. Well, you could in theory make it compatible, but you would need some more "glue" and hacks. net/http doesn't really work the way you're proposing (returning a Response object, etc.). Please see my other post on the standard Go HTTP tools: http://justinas.org/embrace-gos-http-tools/ http://justinas.org/embrace-gos-http-tools/ If you don't want full compatibility with existing net/http libraries, then sure – there are nicer ways. Again, my intention was to show how to do it with net/http and I agree that there are certain trade-offs. "(note, I'd make the middlewares nesteable, passing a next middleware to each middleware, but that wasn't covered in the article)." It was covered to some extent. In a model where every middleware is http.Handler itself, you nest your middlewares like: finalHandler := Middleware1(Middleware2(MyApp)) So every middleware does actually get a reference to the next handler that should run.
- latch 13y ago
  I guess we'll disagree about whether eliminating a global variable [in a multi-threaded language] is nothing more than "might feel a bit nicer". Same with using a class which specifically exists for testing. net/http works fine with a response interface: context := newContext(req) response := chain.yield(context) // the exact same code you had: for k, v := range response.Header() { w.Header()[k] = v } ... This has the added benefit of letting you use different response implementations. A streaming implementation, an in-memory implementation, or, probably the most important if you're doing high traffic, one that relies on a pre-allocated byte-pool. A little nitpick on the side, new(sync.RWMutex) isn't needed. You can just do var cmMutex sync.RWMutex (http://golang.org/src/pkg/net/http/server.go?s=45646:46651#L877 http://golang.org/src/pkg/net/http/server.go?s=45646:46651#L...) EDIT: Sorry, not trying to be negative (so sick of this on here), but I do think global variables are really, really bad and we probably shouldn't be promoting patterns that rely on them.
  - sjustinas 13y ago
    Sure, I didn't say it was impossible to convert your proposed Response type to a proper net/http response when you need it. But, like I said and you just demonstrated, glue code is needed to do that. That makes people write differing middleware application mechanisms that are incompatible with each other. Sure, ResponseRecorder is placed in net/http/httptest and that makes sense. But then again, nothing in the actual implementation is integrated with the testing package or tied to testing in general. IMO there's nothing that makes it unsuitable for other usage. It's a simple buffer to accumulate HTTP, that's it. Even if one were to implement a buffering writer like that on their own, it wouldn't differ too much.

bfrog 13y ago

The request context is I think the most annoying part of go http handling. Oh you want to wrap your handler with something that does db.Begin()/db.Commit()? Well you need to stuff that tx somewhere... so then your options are a global map[*http.Request]interface{} with mutexes or pass it as an arg. Passing it in as an arg means you either lose compile time type checking because you use reflect or lose the ability to add more middlewares that build up request context. The map[]interface{} also loses compile time type checking for the most part unless you do something per type of context, with more mutexes. The second option is the sanest, but means either reflect if you have more than one context arg to build out (sql.Tx + cache + oauth grant + whatever else maybe?) or you have some common context object that may or may not use everything every request and your just adding more context members all the time that may or may not be initialized for that particular request. Sometimes I miss the dynamic typing and/or compile time type templating.

codegangsta 13y ago
I find the args approach the most appealing to me. Yeah you lose some compile time safety. Like you said the other option is a global context with mutexes which still somewhat misses some compile time safety. It basically boils down to reflection over type assertions. I think Martini handles this pretty darn well if I do say so myself ;)
- mmgutz 13y ago
  I think you get the best of both worlds with `gocraft/web`. I feel args when you inject 3 or more (db, session ...) is cumbersome. `gocraft/web` is faster too since it has less reflection.
  - codegangsta 13y ago
    Agreed. I really like how gocraft does contexts. Very cool way to structure your app
acrispino 13y ago
You can use closures: func SomeHandler(foo Foo) { return func(w http.ResponseWriter, req *http.Request) { // use foo } } // later, during init http.HandleFunc("/foo", SomeHandler(someFoo)) or method values, introduced in go 1.1: type handler struct { foo Foo } func (h *handler) bar(w http.ResponseWriter, req *http.Request) { // use h.foo } func (h *handler) qux(w http.ResponseWriter, req *http.Request) { // use h.foo } // later h := handler{ foo: someFoo, } http.HandleFunc("/bar", h.bar) http.HandleFunc("/qux", h.qux)

tedchs 13y ago

In the Ruby community, the Rack interface has done a lot to enable Web middleware to work together. It would be great if there were an analogous interface for Go's net/http.

politician 13y ago
There is, it's the Handler interface. http://slid.es/andybons/high-performance-web-applications-in-go http://slid.es/andybons/high-performance-web-applications-in...

optymizer 13y ago

> a painless and even fun process This claim might be a little too optimistic. I'd like to challenge you (everyone) to associate a User object with an http.Request. Example code in Node.js/ExpressJS: function auth(req, res, next) { req.User = User("username"); next(); } This is such a simple example of a useful piece of middleware, yet it is so difficult to implement in Go in a 'painless and fun' way. I would prefer to avoid contention on a global mutex. I would prefer to keep the compatibility with http.Handler and http.HandlerFunc. Good luck!

cypriss 13y ago
I believe this is solved with gocraft/web: https://github.com/gocraft/web https://github.com/gocraft/web The middleware would be, per the example there: func (c *YourContext) UserRequired(rw web.ResponseWriter, r *web.Request, next web.NextMiddlewareFunc) { user := userFromSession(r) // Pretend like this is defined. It reads a session cookie and returns a *User or nil. if user != nil { c.User = user next(rw, r) } else { rw.Header().Set("Location", "/") rw.WriteHeader(http.StatusMovedPermanently) // do NOT call next() } } Compatibility with Handler/HandlerFunc is kept at the top level, but not at every step along the middleware chain.
elithrar 13y ago
To be fair, Express is a web framework. Go's net/http package is a little lower level than that. Still, this is possible if you make up for the lack of a framework by writing some code yourself. I'd use (and I do use) gorilla/context[1] to allow me to pass data in the request context between middleware layers (i.e. CSRF middleware passes the token along). You could write a couple of little helpers using gorilla/context (SetUser/GetUser, or more generically, SetString/GetString) and SetUser(username, r) in your middleware. The helpers (see the context docs!) just wrap a type assertion on the Get side, and set a map value on the Set side. // Compatible with http.HandlerFunc func auth(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { SetUser(r, "dave") h.ServeHTTP(w, r) } } // Routes r.HandleFunc("/admin", use(adminHandler, auth) http.Handle("/", r) // Allows you to chain your middleware rather than wrapping functions func use(h http.HandlerFunc, middleware ...func(http.HandlerFunc) http.HandlerFunc) http.HandlerFunc { for _, m := range middleware { h = m(h) } return h } You can mod this to work with http.Handler if you want: replace HandlerFunc and then write r.Handle("/admin", use(http.HandlerFunc(adminHandler), auth) instead. [1]: http://www.gorillatoolkit.org/pkg/context http://www.gorillatoolkit.org/pkg/context