Network Applications: HTTP

Created

Application Protocols

Application Layer

In the network protocol stack, the application layer is at the top.

  • It consists of applications: web browsers/servers, email clients/servers, P2P file sharing apps, etc.

Application protocols: define how peer applications communicate with each other.

  • Example: HTTP.

HTTP

Invented by Tim Berners-Lee at CERN in 1989 with the goal of sharing scientific data online.

The important content type is HTML: HyperText Markup Langauge, but is flexible enough for access to any kind of data.

Mentioned before,

HTTP is the application protocol underlying the Word Wide Web. HTTP is a synchronous client/server protocol used by web browsers, web servers, web clients, and web services.

Clients send request to server, and server sends back a response.

  • The request specifies a verb (GET, POST, PUT, etc.) and a name of a resource.

Requests and responses may have a body containing data.

  • The content type specifies what kind of data the body contains.

The best way to learn about HTTP is to examine actual HTTP message exchanges. The curl program is a command-line HTTP client: use the -v option to have it print the first line and headers of the HTTP request and HTTP response.

  • E.g., curl -v http://placekitten.com/1920/1080 -o kitten.jpg

Message Format

All HTTP Messages have the same general form:

  • First Line: describes meaning of message
  • Zero or more headers: metadata about the message
  • Optional body: payload of actual application data (HTML document, image, etc.)

Protocol is text-based, with lines used to delimit important structures.

  • each line terminated by CR (ASCII 13) followed by LF (ASCII 10).
  • Line continuation using backslash (\) allowed for headers.

An HTTP header has the form

Name: Content

Each header provides metadata to help recipient understand the meaning of the message.

  • E.g., Host: placekitten.com specifies which host server is accessed. Content-Type: text/html specify that body is an HTML document.

Request

An HTTP request is a message from a client to a server.

Specifies a method and a resource.

  • method: verb specifying what action the client is requesting the server perform (GET, POST, PUT, etc.)
  • resource: the data resource on the server to which the client is requesting access.

For HTTP 1.1, the first line also specifies the protocol version.

A request can have a body (payload): E.g., submitted form data, file upload data.

Example:

GET /1024/768 HTTP/1.1
Host: placekitten.com
User-Agent: curl/7.58.0
Accept: */*
  • GET is the method
  • /1024/768 is the resource
  • Host header specifies which website is being accessed (a web server can host multiple sites).
  • The Accept header indicates what file types the client is prepared to receive.

Response

An HTTP Response indicates protocol version, status code, and reason phrase.

  • status code specifies how the client shoud interpret the response. E.g., 200 (OK), 403 (Forbidden), 404 (Not Found), and more
  • reason phrase is informational and does not affect the meaning of the response.

Example response

HTTP/1.1 200 OK
Date: Wed, 13 Nov 2019 12:33:20 GMT
Content-Type: image/jpeg
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=de2a22cdd3ed939398e0a56f41ce0e4a31573648400; expires=T
Access-Control-Allow-Origin: *
Cache-Control: public, max-age=86400
Expires: Thu, 31 Dec 2020 20:00:00 GMT
CF-Cache-Status: HIT
Age: 51062
Server: cloudflare
CF-RAY: 5350c608682a957e-IAD

This is followed by a blank line and 40,473 bytes of data.

  • Response code is 200, indicating success.
  • Content-Type indicates resource is an image
  • Transfer-Encoding header indicates that the body is encoding using "chunked" encoding (common for streaming content, but also used for static content)
  • Connection: keep-alive header invites the client to keep the connection open, to be reused for subsequent requests. TCP connections are pretty expensive, so it makes sense to reuse.

The Content-Type header indicates what kind of data the message body contains. It is a MIME type.

  • E.g., text/html HTML document, text/html; charset=utf-8 HTML document with UTF-8 character set, image/jpeg JPEG image.
  • Official registry of MIME types.

Message Body

An HTTP request/response can have a body contianing arbitrary data. Compression can be used. Various encodings are possible:

  • raw binary
  • chunked (chunks consist of byte count followed by specified amount of data)

HTTP Server Implementation

HTTP Server: listens for incoming TCP connections, reads client requests, sends back responses.

The server loop:

int main(int argc, char **argv) {
	if (argc != 3) { fatal("Usage: webserver <port> <webroot>"); }
	
	const char *port = argv[1];
	const char *webroot = argv[2];
	
	int serverfd = open_listenfd((char*) port);
	if (serverfd < 0) { fatal("Couldn't open server socket"); }
	
	while (1) {
		int clientfd = Accept(serverfd, NULL, NULL);
		if (clientfd < 0) { fatal("Error accepting client connection"); }
		server_chat_with_client(clientfd, webroot);
		close(clientfd);
		
	}
}
  • open_listenfd: create server socket
  • Accept: wait for incoming connection

Chat with client: The server_chat_with_client function reads a client request and generates an appropriate response:

void server_chat_with_client(int clientfd, const char *webroot) {
	struct Message *req = NULL;
	rio_t in;
	rio_readinitb(&in, clientfd);
	req = message_read_request(&in);
	printf("got request for resource %s\n", req->resource);
	if (req) {
		server_generate_response(clientfd, req, webroot);
		message_destroy(req);
	}

}

It’s useful to have data types representing protocol messages:

/* data type for message headers */
struct Header {
	char *name;
	char *content;
};

/* Message data type, represents a request from a client */
struct Message {
	int num_headers;          /* number of headers */
	struct Header **headers;  /* array of headers */
	char *method;             /* the method */
	char *resource;           /* the resource requested */
};
  • Note that with additional fields, struct Message could also represent a response

Reading Lines: HTTP uses lines (terminated by CRLF) to structure messages, so a function to read a line of text robustly is very helpful.

ssize_t readline(rio_t *in, char *usrbuf, size_t maxlen) {
	ssize_t len = rio_readlineb(in, usrbuf, maxlen);
	if (len > 0 && usrbuf[len-1] == '\n') {
		/* trim trailing LF (newline) */
		usrbuf[len-1] = '\0';
		len--;
	}
	if (len > 0 && usrbuf[len-1] == '\r') {
		/* trim trailing CR */
		usrbuf[len-1] = '\0';
		len--;
	}
	return len;
}
  • Heavy lifting done by rio_readlineb

Reading a message:

struct Message *message_read_request(rio_t *in) {
	struct Message *result = NULL;
	// read first line (method and resource)
	// read 0 or more headers  
	// read optional body  
	return result;
}
  • This is a fairly complicated function

Disecting this:

Read first line of request: It has essential infomration including the method, resource, and protocol.

len = readline(in, linebuf, MAX_LINE_LEN);
if (len < 0) { goto read_done; }
char *savep, *method, *resource, *proto;
method = strtok_r(linebuf, " ", &savep);
if (!method) { goto read_done; }
resource = strtok_r(NULL, " ", &savep);
if (!resource) { goto read_done; }
proto = strtok_r(NULL, " ", &savep);
if (!proto || strcmp(proto, "HTTP/1.1") != 0) { goto read_done; }
  • strtok_r is used to tokenize the line using a set of deliminator.
  • Error handling simplified using goto: not as awful as it sounds.

Reading a header: body of loop to read 0 or more headers (simplified):

len = readline(in, linebuf, MAX_LINE_LEN);
if (strcmp(linebuf, "") == 0) {
	done_with_headers = 1;  /* read a blank line */
} else {
	
	/* try to read a header */
	char *p = strchr(linebuf, ':');
	if (p) {
	
	separate header into name and content parts  
	struct Header *hdr = create and initialize Header object msg->headers[num_headers] = hdr;  
	num_headers++;

	}
}
  • Building a map from header name to the header body
  • strchr function used to find the ':' separating name and content of header
  • Headers are terminated by a blank line.

Reading the request body: left to your own practice.

Responding to a request:

void server_generate_response(int clientfd, struct Message *req, const char *webroot) {
	if (strcmp(req->method, "GET") != 0) {
		server_generate_text_response(clientfd, "403", "Forbidden", "only GET requests are allowed");
		goto response_done; 
	}
	
	filename = concat(webroot, req->resource);
	struct stat s;
	if (stat(filename, &s) < 0) {
		server_generate_text_response(clientfd, "404", "Not Found", "requested resource does not exist");
		goto response_done; 
	}  
	writestr(clientfd, "HTTP/1.1 200 OK\r\n"); 
	// write Content-Type and Content-Length headers 
	// read data from file and copy to clientfd

	response_done:
		//cleanup
}

To send a text response: Error messages are sent back as text:

void server_generate_text_response(int clientfd, const char *response_code,
  const char *reason, const char *msg) {
  writestr(clientfd, "HTTP/1.1 ");
  writestr(clientfd, response_code);
  writestr(clientfd, " ");
  writestr(clientfd, reason);
  writestr(clientfd, "\r\n");
  /* could generate headers... */
  writestr(clientfd, "\r\n");
  writestr(clientfd, msg);
}

To put everything together:

$ gcc -o webserver main.c webserver.c csapp.c -lpthread
$ mkdir site
$ curl http://placekitten.com/800/600 -o site/kittens.jpg

Security Vulnerabilityies

Handling Untrusted Data Data read from the client is untrusted.

A network application which connects to untrusted peers must assume that they are malicious.

Therefore, never ever:

  • trust that data is properly formatted
  • trust no special characters are presented
  • trust that message size limits are not exceeded.

Example 1

char buf[1024], field1[256], field2[256];
rio_readlineb(in, buf, 1024);
sscanf(buf, "%s %s", field1, field2);

A buffer overflow occurs when malicious peer sends more data than can be receieved into the recipient's buffer.

If the buffer is stack allocated, then the malicious client could overwrite the return address in the current stack frame with an arbitrary value.

When the function returns, it jumps to an address controlled by the malicious peer.

  • i.e., the client can cause the program to call the system function, which executes an arbitrary program as a subprocess.

HTTP Client Implementation

To develop a simple HTTP Client, it needs to do the following things:

  • Parse URL
  • Connect to server
  • Request resource
  • Read headers
  • Save body of resource to file.

Pseudocode of main function:

int main(int argc, char **argv) {
  // check command line arguments
  // parse URL
  // connect to server
  // send request
  // read response headers
  // read response body
}

Parse URL

A URL (Uniform Resource Locator) encodes the protocol, name, and location of a data resource.

Example: http://placekitten.com/1024/768

  • protocol is http
  • host is placekitten.com
  • resource name is /1024/768e

TCP port for (unencrypted) HTTP is 80

  • Can be overriden in URL (e.g., http://localhost:30000)

The code to parse URL:

/* parse URL */
if (!strncmp(url, "http://", 7) != 0)
	{ fatal("only http URLs are supported"); }
url += 7;
char *host, *resource;
host = url;
resource = strchr(host, '/');
if (!resource) { fatal("Invalid URL"); }
*resource = '\0';
resource++;

/* see if a port was specified */
const char *port = "80"; /* default HTTP port */
char *p = strchr(host, ':');
if (p) {
	*p = '\0';
	port = p+1; 
}

Connect, request resource

Once host and port have been determined, connecting and sending request is fairly straightforward:

/* Connect to server */
int connfd = Open_clientfd((char *)host, (char *)port);

/* Send request */
writestr(connfd, "GET /");
writestr(connfd, resource);
writestr(connfd, " HTTP/1.1\r\n");
writestr(connfd, "Host: ");
writestr(connfd, host);
writestr(connfd, "\r\nUser-Agent: jhucsf/0.1\r\n");
writestr(connfd, "Accept: */*\r\n\r\n");

Reading response

The first line = status code:

/* Read response */
char linebuf[LINEBUF_SIZE];
Rio_readlineb(&in, linebuf, LINEBUF_SIZE);
printf("First line: %s\n", linebuf);

/* make sure response code was 200 */
char *p2 = strchr(linebuf, ' ');
if (!p2) { fatal("bad HTTP response?"); }
p2++;
if (strncmp(p2, "200", 3) != 0) { fatal("HTTP response not 200"); }

Client reads headers, looking for Transfer-Encoding and Content-Length

Client needs to know encoding of message body in order to know how to read and decode it

Our client will only support chunked encoding

Headers are terminated by a blank line, message body follows

Reading Message Body

The chunked encoding consists of chunks of binary data, each preceded with a size in bytes (encoded as a hexadecimal integer):

while (1) {
	/* determine size of next chunk */
	readline(&in, linebuf, LINEBUF_SIZE);
	unsigned chunk_len;
	sscanf(linebuf, "%x", &chunk_len);
	/* if chunk size is 0, body has ended */
	if (chunk_len == 0) { break; }
	/* read chunk, write to output file */
	char *buf = xmalloc(chunk_len);
	rio_readnb(&in, buf, chunk_len);
	rio_writen(outfd, buf, chunk_len);
	free(buf);
	/* need to read another \r\n following the chunk */
	Rio_readnb(&in, linebuf, 2);
}

Running the program

$ ./webclient http://placekitten.com/1152/864 kitten.jpg
host: placekitten.com
port: 80
resource: /1152/864

First line: HTTP/1.1 200 OK

Header: Date: Sat, 16 Nov 2019 01:15:05 GMT
Header: Content-Type: image/jpeg
Header: Transfer-Encoding: chunked
Chunked encoding

Header: Connection: keep-alive
Header: Set-Cookie: __cfduid=dd8f10ca64c91fce89a52b57aa4ccede71573866904; expires=Sun, 15-Nov-
Header: Access-Control-Allow-Origin: *
Header: Cache-Control: public, max-age=86400
Header: Expires: Thu, 31 Dec 2020 20:00:00 GMT
Header: CF-Cache-Status: MISS
Header: Server: cloudflare
Header: CF-RAY: 53659c9958fbea34-IAD
Download successful?

Linked from Computer System Fundamentals