Nothing but net(/http)! Fun with HTTP requests in Go

Go's standard net/http library is really popular in the Go community, and for good reasons. Its Handler interface is straightforward, versatile, and easy to learn from Node.js. There are tons of router libraries that build on net/http, like Gorilla Mux and Goji. But besides making web servers, this built-in library also lets you send your own HTTP requests with its HTTP Client type and get and use data from online.

Nothing beats the hacker feeling of sending an HTTP request straight from the black screen, so in this tutorial we're going to try that out with a Client. If you know Go and you're mostly new to HTTP or you want to learn how to send requests from the net/http package, this tutorial is for you!

Also, for the newcomers to HTTP, this tutorial is more focused on programming and doesn't assume much HTTP familiarity, but I highly recommend also reading about HTTP itself to get a better sense of how the Internet works, so I'll link to those at the end of the tutorial. You can find the code from this blog post on this GitHub repository.

Fetching a webpage with a Client

 

If you're reading this, you've been using HTTP clients the whole time. Your browser is basically a really tricked-out HTTP client. When you type in the URL you want to go to, the browser sends an HTTP request to the site's server, the server handles the request and sends back an HTTP response, and then the browser uses the data in the response to render a webpage, as well as doing other browser stuff like making sure you're not downloading a virus, running the site's JavaScript, and putting the site you're going to in your web history.

In Go, a Client from net/http sends the request and gets back the response, but from there on out, what you do with the response is your call. You can fetch a response full of data and plug it into a do data analysis program, you can download pictures, you can take the response and use its bytes as input for LEDs hooked up to a Raspberry Pi to be the life of a dance party, you name it! So for a first example, let's take a look at the response we get back from example.com. Put this code into a file called example.go:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    // Make a Client and new HTTP request
    client := http.Client{}
    req, err := http.NewRequest("GET", "http://example.com", nil)
    if err != nil {
        fmt.Println("Error creating HTTP request:", err)
        return
    }

    // Send the request and get back the HTTP response
    res, err := client.Do(req)
    if err != nil {
        fmt.Println("Error sending request:", err)
        return
    }
    defer res.Body.Close() // Close the response body to prevent a memory leak

    // Print out the response status code and Content-Type
    fmt.Println("Response status was:", res.Status)
    fmt.Println(res.Header.Get("Content-Type"))
    fmt.Println()

    // Print out the contents of the response body, which is
    // the HTML of example.com
    responseBytes, err := ioutil.ReadAll(res.Body)
    if err != nil {
        fmt.Println("Error getting response body bytes:", err)
    }
    fmt.Println(string(responseBytes))
}

Then in the command line, run it by typing go run example.go and you should get:

What you're seeing is the response, which was the HTML of example.com and a bit of information up top about the HTTP response. Let's break our Go code down and see how we got that response!

First, we create a Client with the line:

client := http.Client{}

Then we can make a request with this code:

req, err := http.NewRequest("GET", "http://example.com", nil)
if err != nil {
    fmt.Println("Error creating HTTP request:", err)
    return
}

This call to NewRequest makes a GET request to http://example.com. Since it's a GET request, we're not sending any data in the request's body, so we give the request a nil request body.

If all goes well, NewRequest gives us a net/http Request. To send it to example.com...

res, err := client.Do(req)
if err != nil {
    fmt.Println("Error sending request:", err)
    return
}
defer res.Body.Close()

We pass our request into our HTTP Client's Do method, which sends the request to example.com. If the request is sent successfully, we will get back the server's Response. We also make sure to close the response body with defer res.Body.Close() to prevent a memory leak.

Now that we have that Response, we can take a closer look at it. These lines tell us what the status of the response was and what type of content is in the response's body:

fmt.Println("Response status was:", res.Status)
fmt.Println("Response content type:", res.Header.Get("Content-Type"))

So those lines are what printed out these messages in the terminal:

Response status was: 200 OK

Response content type: text/html

This tells us that Response has a 200 status code, which basically means the request was successfully sent to the server, and according to its Content-Type header, the data in the response body is HTML. To see the HTML, we use ioutil.ReadAll to turn the response's Body into a byte slice:

responseBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
    fmt.Println("Error getting response body bytes:", err)
}
fmt.Println(string(responseBytes))

Which gives us the HTML of example.com! Here's a diagram of what happens when we send this request from our Client:

As you can see, it's really similar to what happens when we send the request from a browser, except that instead of rendering a webpage from the HTML we just print out the HTML.

Downloading pictures

Webpages aren't the only kind of data you can get from the Internet. There's pictures, videos, JSON data files, code, MP3s, any kind of data you want, you can send and receive over HTTP. And the Go net/http Clients can get responses with these different types of content. For example, let's try downloading a sloth picture. Replace the example.com URL we passed into http.NewRequest with this URL:

"https://upload.wikimedia.org/wikipedia/commons/1/18/Bradypus.jpg"

And when you send the request you should get results that look like this:

That's the encoding of a sloth JPEG but printed out as text. We can't read that, but that's a sloth picture. Indeed, if we print out the Content-Type header, we should get image/jpeg

So we're getting back an image as our response body. Now to turn our program into an image downloader, get rid of the lines printing out the Content-Type header and the Status of the response and run the program again with:

go run sloth-download.go > sloth.jpg

Then open sloth.jpg and you should get:

Now what our program does is it gets the bytes of an image and then prints them to an image file. To trick this out some more, you could also use some image processing code to edit the picture. For example, import the image, image/jpeg, image/color, and os packages and replace this code:

responseBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
    fmt.Println("Error getting response body bytes:", err)
}
fmt.Println(string(responseBytes))

with this:

// Decode the response body into an Image and pass it to purpleJPEG
img, _, err := image.Decode(res.Body)
if err != nil {
    fmt.Println("Error decoding JPEG image: ", err)
    return
}
purpleJPEG(img)

Now instead of simply printing the response body, we make an image.Image from the response's Body and we pass that image into a function called purpleJPEG, which, you guessed it, makes a picture purple:

func purpleJPEG(img image.Image) {
    pixels := image.NewRGBA64(img.Bounds())
    for x := pixels.Bounds().Min.X; x < pixels.Bounds().Max.X; x++ {
        for y := pixels.Bounds().Min.Y; y < pixels.Bounds().Max.Y; y++ {
            r, g, b, a := img.At(x, y).RGBA()
            r = r>>8 + 50
            b = b>>8 + 50
            if r > 256 {
                r = 256
            }
            if b > 256 {
                b = 256
            }
            pixels.Set(x, y, color.RGBA64{
                R: uint16(r >> 8),
                G: uint16(g),
                B: uint16(b >> 8),
                A: uint16(a),})
            }
    }
    jpeg.Encode(os.Stdout, pixels, nil)
}

And now if we run go run purple-sloth-download.go > purple.jpg we will get:

There's actually a bug in purpleJPEG that adds in the green. It's only supposed to give the picture a purple tint, but I decided to keep it in because it looks cooler. If psychedelic rock ever makes a comeback, we better see a band called the Sloth Moths!

So as you can see, running web servers isn't the only powerful functionality of net/http (take note, by the way, that all of the code we used was built with only stuff in the standard library!). The Client type gives you a straightforward interface to send HTTP requests and use the data you get back in your Go code, and we haven't even scratched the surface on what you can do with an HTTP Client.

In addition to receiving data, you can send data in your HTTP requests with POST requests. And we haven't even touched using HTTP requests to communicate with third-party APIs to build apps that integrate with sites like Instagram, Google, Slack, or Twitter, which is what my next tutorial will be about.

HTTP homework

Like I said at the beginning of this tutorial, this blog post was aimed mainly at people who are newer to HTTP. If you're one of them and want to do more with HTTP, like if you want to develop web apps for a startup, it is essential that you build a more solid foundation and de-mystify HTTP so you can work with the protocol both effectively and securely. Below I have a list of links to some resources for getting a better understanding on what HTTP really is:

http://www.tutorialspoint.com/http/ A TutorialsPoint series on HTTP

http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340 A Beginner’s Guide to HTTP and REST

https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol The HTTP Wikipedia article

https://tools.ietf.org/html/rfc7230 - https://tools.ietf.org/html/rfc7235 The official documents on the HTTP/1.1 protocol

https://golang.org/pkg/net/http/ Go's net/http documentation

http://curl.haxx.se/docs/httpscripting.html The tutorial for cURL, an easy to use command-line HTTP client

Until next time, stay slothful!

Image credits

• The Go Gopher in the diagrams was originally drawn by Renee French and is licensed under CC BY 3.0

• The Firefox logo used in the first diagram was made by Mozilla and is licensed under CC BY 3.0

• The original sloth picture in the tutorial is from Stefan Laube and is public domain.

• The picture of the web server used in the diragrms was drawn by lyte on openclipart and is public domain.