What Is cURL?

The world wide web has – to say the least – become somewhat ubiquitous, to the point that it’s hard to do much on a computer without the need to communicate. This is equally true when working at the command line on a server as when working on a desktop. There are a number of tools to help you communicate with files on the web, and today we are be going to take a look at cURL.

What is cURL?

cURL comes in two parts:

• First is libcurl, a library that does all the work and provides an interface that other programs can use to make use of cURL’s functionality.
• Second is cURL which acts as a command line interface.

The core functionality of cURL is to perform requests to URLs (Uniform Resource Locators, colloquially called web addresses), which normally results in the fetching of a file from a web server. The default action for cURL is to display the contents of any returned file from the server to the standard output (normally the terminal screen). Here’s an example:
curl www.example.com
In general, displaying the content of the file you are opening on the screen isn’t very helpful, so the -o flag can be used to specify an output file to save the contents to.
curl -o example.com.html www.example.com
Now when you look at the contents of the example.com.html file, you’ll see the site content that you saw in the earlier example. You can alternately use -O to save to a file with the same filename as the one being remotely accessed:
curl -O www.example.com/index.html
Now you will have a file called index.html in the directory where you ran cURL containing the content of the site.

Connecting with HTTPS

cURL also supports connecting using HTTPS, which can be done by including the http:// at the start of the URL:
curl http://www.example.com
Now imagine that you want to obtain a file that is behind the protection of HTTP basic authentication. This can be done by submitting in the form of:
curl -u <username>:<password> www.example.com
In this case, replace <username> with the username you need to access the resource you are requesting, and <password> with the password for the username you gave.
At times you’ll connect to a URL that provides a redirect to another URL. cURL will generally not follow these redirects, and will inform you of the redirection. If you want cURL to follow the redirection and fetch the resource it was directed to, then you need to use the -L flag.
curl -L www.example.com
So far so simple. cURL also has features to help you communicate with sites that require web forms and cookies. The standard request type that cURL makes is a GET request, the arguments for which can be appended to the URL much like in your browser address bar:
curl www.example.com/index.html?key=value&key2=value2
In this example we’ve requested the page with the GET variables of “key” being set to “value” and “key2” being set to “value2”.
To send a POST request you need to specify the request type with the -X flag, and the individual variables need to be set with the -F flag:
curl -X POST -F ‘key=value’ -F ‘key2=value2’ www.example.com/index.html
In this example we’ve sent the same variables as before to the same URL, but using the POST request rather than a GET one.
With those basics out of the way, we can move on to a slightly more advanced real world scenario. Imagine that you need to get the contents of a file, but they’re on a website that requires a login form to be submitted to access. On logging in, a cookie is returned that is then used to allow access to other pages.
curl -X POST -F ‘username=bob’ -F ‘password=pa55w0rd’ -c ~/cookie.txt http://www.example.com/login.php
curl -b ~/cookie.txt -o grabbed.file http://www.example.com/grab-file.php
Now let’s have a quick scan over how this works. With the first command we use a POST request to send username and password variables to the login.php file on the www.example.com server. The -c flag tells cURL to save any returned cookie information to the given file.
The second command uses the -b flag to tell cURL to send the contents of the ~/cookie.txt file as cookie information along with the request. This means that the website will know you have previously logged in from the cookie information, then cURL will save the output to the file named grabbed.file.
We’ve now covered most of what you are likely to need when interacting with websites using cURL at the command line. There are a number of other additional features that it supports, and as usual the man pages are a great start to getting information on those