Linux Commands – curl

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed crontab command in Linux which is used to schedule cron jobs in Linux.

https://cloudaffaire.com/linux-commands-crontab/

In this blog post, we will discuss curl command in Linux. curl command can be used to transfer data from or to a server. You can send request to server using curl command and get the response from the server. The request can range from get data, put data, post data, delete data etc. and curl will send your request to the server and provide you the server response to your request (whether successful or unsuccessful).

curl supports almost all popular protocols like http, ftp, smtp, ldap, pop3 etc. You provide the url of the server along with different options in curl command. curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. By default, without any other options and without any protocol prefixed in url, curl will do a http get request to the given url.

Linux Commands – curl:

You can use curl command to send a get request to a server and get the server response to your request along with data.

You can use curl -o <file> or –output <file> options to write the output of curl command into a file instead of the standard output (default).

You can use curl –create-dirs option in conjunction with the -o option to create the necessary local directory hierarchy as needed. This option creates the dirs mentioned with the -o option, nothing else. If the -o file name uses no dir or if the dirs it mentions already exist, no dir will be created.

You can use curl -O or –remote-name options to write output to a local file named like the remote file we get. With -O the remote file name will be used for saving the file.

You can use curl -# or –progress-bar options to display progress as a simple progress bar instead of the standard, more informational, meter (default).

You can use curl -v or –verbose options print the actions curl command is performing. Mostly useful for debugging. A line starting with ‘>’ means “header data” sent by curl, ‘<‘ means “header data” received by curl that is hidden in normal cases, and a line starting with ‘*’ means additional info provided by curl.

You can use curl –trace <file> option to enable a full trace dump of all incoming and outgoing data, including descriptive information, to the given output file. You can also use curl –trace-ascii <file> option to enable a full trace dump of all incoming and outgoing data, including descriptive information, to the given output file. This is very similar to –trace, but leaves out the hex part and only shows the ASCII part of the dump (human friendly). You can also curl –trace-time option to prepend a time stamp to each trace or verbose line that curl displays. To output in the standard output instead of file use ‘-‘ in place for the file path.

You can use curl -s or –silent options to run curl in silent or quiet mode (will not show a progress bar or error messages). You can use curl -S or –show-error options with -s to show an error message if it fails.

You can use curl -w or –write-out <format> options to define what to display on stdout after a completed and successful operation. The format is a string that may contain plain text mixed with any number of variables. The string can be specified as “string”, to get read from a particular file you specify it “@filename” and to tell curl to read the format from stdin you write “@-“. The variables present in the output format will be substituted by the value or text that curl thinks fit, as described below.

The variables available are:

• content_type: The Content-Type of the requested document.
• filename_effective: The ultimate filename that curl writes out to (only valid with -o or -O).
• ftp_entry_path: The initial path curl ended up in when logging on to the remote FTP server.
• http_code: The numerical response code that was found in the last retrieved HTTP(S) or FTP(s) transfer.
• http_connect: The numerical code that was found in the last response (from a proxy) to a curl CONNECT request.
• local_ip: The IP address of the local end of the most recently done connection.
• local_port: The local port number of the most recently done connection.
• num_connects: Number of new connects made in the recent transfer.
• num_redirects: Number of redirects that were followed in the request.
• redirect_url: When an HTTP request was made without -L to follow redirects, this variable will show the actual URL a redirect would take you to
• remote_ip: The remote IP address of the most recently done connection.
• remote_port: The remote port number of the most recently done connection.
• size_download: The total amount of bytes that were downloaded.
• size_header: The total amount of bytes of the downloaded headers.
• size_request: The total amount of bytes that were sent in the HTTP request.
• size_upload: The total amount of bytes that were uploaded.
• speed_download: The average download speed that curl measured for the complete download. Bytes per second.
• speed_upload: The average upload speed that curl measured for the complete upload. Bytes per second.
• ssl_verify_result: The result of the SSL peer certificate verification that was requested. 0 means the verification was successful.
• time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed.
• time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
• time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.
• time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin.
• time_redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started.
• time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred.
• time_total: The total time, in seconds, that the full operation lasted.
• url_effective: The URL that was fetched last. This is most meaningful if you’ve told curl to follow location: headers.

You can use curl -i or –include options to Include the HTTP-header in the output. The HTTP-header includes things like server-name, date of the document, HTTP-version, and more.

You can use curl -I or –head options to fetch the HTTP-header only. HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE file, curl displays the file size and last modification time only.

You can use curl -D or –dump-header <file> options to write the protocol headers to the specified file. This option is handy to use when you want to store the headers that an HTTP site sends to you.

You can use curl -f or –fail option to fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22. This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).

You can use curl -X or –request <command> options to specify a custom request method to use when communicating with the HTTP server. The specified request will be used instead of the method otherwise used (which defaults to GET). More on this on API section.

You can use curl -d or –data <data> to send the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button. You can use “@FILE_NAME” to pass the data stored in a file or “-” to pass the data reading from standard input.

You can use curl -H or –header <header> options to pass some extra headers when getting a web page. You may specify any number of extra headers. Note that if you should add a custom header that has the same name as one of the internal one’s curl would use, your externally set header will be used instead of the internal one. This allows you to make even trickier stuff than curl would normally do. You should not replace internally set headers without knowing perfectly well what you’re doing.

API call with curl command:

You can use curl command to make API call to your API server. Curl supports all available http methods and you can interact with API using below http methods. You define the http method to use with curl command with -X or –request options.

• GET: The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
• HEAD: The HEAD method asks for a response identical to that of a GET request, but without the response body.
• POST: The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server.
• PUT: The PUT method replaces all current representations of the target resource with the request payload.
• DELETE: The DELETE method deletes the specified resource.
• CONNECT: The CONNECT method establishes a tunnel to the server identified by the target resource.
• OPTIONS: The OPTIONS method is used to describe the communication options for the target resource.
• TRACE: The TRACE method performs a message loop-back test along the path to the target resource.
• PATCH: The PATCH method is used to apply partial modifications to a resource.

Hope you have enjoyed this article. Curl supports a plethora of options and it’s not possible to cover all options in a single blog post. Maybe in future we will have a dedicated series on curl command alone. In the next blog post, we will discuss find command in Linux.