Python Requests Example

P

Quickstart — Requests 2.26.0 documentation

Quickstart — Requests 2.26.0 documentation

Eager to get started? This page gives a good introduction in how to get started
with Requests.
First, make sure that:
Requests is installed
Requests is up-to-date
Let’s get started with some simple examples.
Make a Request¶
Making a request with Requests is very simple.
Begin by importing the Requests module:
Now, let’s try to get a webpage. For this example, let’s get GitHub’s public
timeline:
>>> r = (”)
Now, we have a Response object called r. We can
get all the information we need from this object.
Requests’ simple API means that all forms of HTTP request are as obvious. For
example, this is how you make an HTTP POST request:
>>> r = (”, data = {‘key’:’value’})
Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD and
OPTIONS? These are all just as simple:
>>> r = requests. options(”)
That’s all well and good, but it’s also only the start of what Requests can
do.
Passing Parameters In URLs¶
You often want to send some sort of data in the URL’s query string. If
you were constructing the URL by hand, this data would be given as key/value
pairs in the URL after a question mark, e. g.
Requests allows you to provide these arguments as a dictionary of strings,
using the params keyword argument. As an example, if you wanted to pass
key1=value1 and key2=value2 to, you would use the
following code:
>>> payload = {‘key1’: ‘value1’, ‘key2’: ‘value2’}
>>> r = (”, params=payload)
You can see that the URL has been correctly encoded by printing the URL:
>>> print()
Note that any dictionary key whose value is None will not be added to the
URL’s query string.
You can also pass a list of items as a value:
>>> payload = {‘key1’: ‘value1’, ‘key2’: [‘value2’, ‘value3’]}
Response Content¶
We can read the content of the server’s response. Consider the GitHub timeline
again:
>>> import requests
>>>
‘[{“repository”:{“open_issues”:0, “url”:”
Requests will automatically decode content from the server. Most unicode
charsets are seamlessly decoded.
When you make a request, Requests makes educated guesses about the encoding of
the response based on the HTTP headers. The text encoding guessed by Requests
is used when you access You can find out what encoding Requests is
using, and change it, using the r. encoding property:
>>> r. encoding
‘utf-8’
>>> r. encoding = ‘ISO-8859-1’
If you change the encoding, Requests will use the new value of r. encoding
whenever you call You might want to do this in any situation where
you can apply special logic to work out what the encoding of the content will
be. For example, HTML and XML have the ability to specify their encoding in
their body. In situations like this, you should use ntent to find the
encoding, and then set r. encoding. This will let you use with
the correct encoding.
Requests will also use custom encodings in the event that you need them. If
you have created your own encoding and registered it with the codecs
module, you can simply use the codec name as the value of r. encoding and
Requests will handle the decoding for you.
Binary Response Content¶
You can also access the response body as bytes, for non-text requests:
>>> ntent
b'[{“repository”:{“open_issues”:0, “url”:”
The gzip and deflate transfer-encodings are automatically decoded for you.
The br transfer-encoding is automatically decoded for you if a Brotli library
like brotli or brotlicffi is installed.
For example, to create an image from binary data returned by a request, you can
use the following code:
>>> from PIL import Image
>>> from io import BytesIO
>>> i = (BytesIO(ntent))
JSON Response Content¶
There’s also a builtin JSON decoder, in case you’re dealing with JSON data:
>>> ()
[{‘repository’: {‘open_issues’: 0, ‘url’: ‘
In case the JSON decoding fails, () raises an exception. For example, if
the response gets a 204 (No Content), or if the response contains invalid JSON,
attempting () raises requests. exceptions. JSONDecodeError. This wrapper exception
provides interoperability for multiple exceptions that may be thrown by different
python versions and json serialization libraries.
It should be noted that the success of the call to () does not
indicate the success of the response. Some servers may return a JSON object in a
failed response (e. g. error details with HTTP 500). Such JSON will be decoded
and returned. To check that a request is successful, use
r. raise_for_status() or check atus_code is what you expect.
Raw Response Content¶
In the rare case that you’d like to get the raw socket response from the
server, you can access If you want to do this, make sure you set
stream=True in your initial request. Once you do, you can do this:
>>> r = (”, stream=True)

>>> (10)
‘\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03’
In general, however, you should use a pattern like this to save what is being
streamed to a file:
with open(filename, ‘wb’) as fd:
for chunk in er_content(chunk_size=128):
(chunk)
Using er_content will handle a lot of what you would otherwise
have to handle when using directly. When streaming a
download, the above is the preferred and recommended way to retrieve the
content. Note that chunk_size can be freely adjusted to a number that
may better fit your use cases.
Note
An important note about using er_content versus
er_content will automatically decode the gzip and deflate
transfer-encodings. is a raw stream of bytes – it does not
transform the response content. If you really need access to the bytes as they
were returned, use
More complicated POST requests¶
Typically, you want to send some form-encoded data — much like an HTML form.
To do this, simply pass a dictionary to the data argument. Your
dictionary of data will automatically be form-encoded when the request is made:
>>> r = (“, data=payload)
{…
“form”: {
“key2”: “value2”,
“key1”: “value1″},… }
The data argument can also have multiple values for each key. This can be
done by making data either a list of tuples or a dictionary with lists
as values. This is particularly useful when the form has multiple elements that
use the same key:
>>> payload_tuples = [(‘key1’, ‘value1’), (‘key1’, ‘value2’)]
>>> r1 = (”, data=payload_tuples)
>>> payload_dict = {‘key1’: [‘value1’, ‘value2’]}
>>> r2 = (”, data=payload_dict)
“key1”: [
“value1”,
“value2″]},… }
>>> ==
True
There are times that you may want to send data that is not form-encoded. If
you pass in a string instead of a dict, that data will be posted directly.
For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data:
>>> import json
>>> url = ”
>>> payload = {‘some’: ‘data’}
>>> r = (url, (payload))
Instead of encoding the dict yourself, you can also pass it directly using
the json parameter (added in version 2. 4. 2) and it will be encoded automatically:
>>> r = (url, json=payload)
Note, the json parameter is ignored if either data or files is passed.
Using the json parameter in the request will change the Content-Type in the header to application/json.
POST a Multipart-Encoded File¶
Requests makes it simple to upload Multipart-encoded files:
>>> files = {‘file’: open(”, ‘rb’)}
>>> r = (url, files=files)
“files”: {
“file”: “<>“},… }
You can set the filename, content_type and headers explicitly:
>>> files = {‘file’: (”, open(”, ‘rb’), ‘application/’, {‘Expires’: ‘0’})}
If you want, you can send strings to be received as files:
>>> files = {‘file’: (”, ‘some, data, to, send\nanother, row, to, send\n’)}
“file”: “some, data, to, send\\nanother, row, to, send\\n”},… }
In the event you are posting a very large file as a multipart/form-data
request, you may want to stream the request. By default, requests does not
support this, but there is a separate package which does –
requests-toolbelt. You should read the toolbelt’s documentation for more details about how to use it.
For sending multiple files in one request refer to the advanced
section.
Warning
It is strongly recommended that you open files in binary
mode. This is because Requests may attempt to provide
the Content-Length header for you, and if it does this value
will be set to the number of bytes in the file. Errors may occur
if you open the file in text mode.
Response Status Codes¶
We can check the response status code:
>>> atus_code
200
Requests also comes with a built-in status code lookup object for easy
reference:
>>> atus_code ==
If we made a bad request (a 4XX client error or 5XX server error response), we
can raise it with
Response. raise_for_status():
>>> bad_r = (”)
404
>>> bad_r. raise_for_status()
Traceback (most recent call last):
File “requests/”, line 832, in raise_for_status
raise _error
TPError: 404 Client Error
But, since our status_code for r was 200, when we call
raise_for_status() we get:
>>> r. raise_for_status()
None
All is well.
Cookies¶
If a response contains some Cookies, you can quickly access them:
>>> r = (url)
>>> okies[‘example_cookie_name’]
‘example_cookie_value’
To send your own cookies to the server, you can use the cookies
parameter:
>>> cookies = dict(cookies_are=’working’)
>>> r = (url, cookies=cookies)
‘{“cookies”: {“cookies_are”: “working”}}’
Cookies are returned in a RequestsCookieJar,
which acts like a dict but also offers a more complete interface,
suitable for use over multiple domains or paths. Cookie jars can
also be passed in to requests:
>>> jar = questsCookieJar()
>>> (‘tasty_cookie’, ‘yum’, domain=”, path=’/cookies’)
>>> (‘gross_cookie’, ‘blech’, domain=”, path=’/elsewhere’)
>>> r = (url, cookies=jar)
‘{“cookies”: {“tasty_cookie”: “yum”}}’
Redirection and History¶
By default Requests will perform location redirection for all verbs except
HEAD.
We can use the history property of the Response object to track redirection.
The Response. history list contains the
Response objects that were created in order to
complete the request. The list is sorted from the oldest to the most recent
response.
For example, GitHub redirects all HTTP requests to HTTPS:

>>> r. history
[]
If you’re using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disable
redirection handling with the allow_redirects parameter:
>>> r = (”, allow_redirects=False)
301
[]
If you’re using HEAD, you can enable redirection as well:
>>> r = (”, allow_redirects=True)
Timeouts¶
You can tell Requests to stop waiting for a response after a given number of
seconds with the timeout parameter. Nearly all production code should use
this parameter in nearly all requests. Failure to do so can cause your program
to hang indefinitely:
>>> (”, timeout=0. 001)
File ““, line 1, in
requests. Timeout: HTTPConnectionPool(host=”, port=80): Request timed out. (timeout=0. 001)
timeout is not a time limit on the entire response download;
rather, an exception is raised if the server has not issued a
response for timeout seconds (more precisely, if no bytes have been
received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do
not time out.
Errors and Exceptions¶
In the event of a network problem (e. DNS failure, refused connection, etc),
Requests will raise a ConnectionError exception.
Response. raise_for_status() will
raise an HTTPError if the HTTP request
returned an unsuccessful status code.
If a request times out, a Timeout exception is
raised.
If a request exceeds the configured number of maximum redirections, a
TooManyRedirects exception is raised.
All exceptions that Requests explicitly raises inherit from
questException.
Ready for more? Check out the advanced section.
If you’re on the job market, consider taking this programming quiz. A substantial donation will be made to this project, if you find a job through this platform.
Python's Requests Library (Guide)

Python’s Requests Library (Guide)

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.
Throughout this article, you’ll see some of the most useful features that requests has to offer as well as how to customize and optimize those features for different situations you may come across. You’ll also learn how to use requests in an efficient way as well as how to prevent requests to external services from slowing down your application.
In this tutorial, you’ll learn how to:
Make requests using the most common HTTP methods
Customize your requests’ headers and data, using the query string and message body
Inspect data from your requests and responses
Make authenticated requests
Configure your requests to help prevent your application from backing up or slowing down
Though I’ve tried to include as much information as you need to understand the features and examples included in this article, I do assume a very basic general knowledge of HTTP. That said, you still may be able to follow along fine anyway.
Now that that is out of the way, let’s dive in and see how you can use requests in your application!
Getting Started With requests
Let’s begin by installing the requests library. To do so, run the following command:
If you prefer to use Pipenv for managing Python packages, you can run the following:
$ pipenv install requests
Once requests is installed, you can use it in your application. Importing requests looks like this:
Now that you’re all set up, it’s time to begin your journey through requests. Your first goal will be learning how to make a GET request.
The GET Request
HTTP methods such as GET and POST, determine which action you’re trying to perform when making an HTTP request. Besides GET and POST, there are several other common methods that you’ll use later in this tutorial.
One of the most common HTTP methods is GET. The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request, invoke ().
To test this out, you can make a GET request to GitHub’s Root REST API by calling get() with the following URL:
>>>>>> (”)

Congratulations! You’ve made your first request. Let’s dive a little deeper into the response of that request.
The Response
A Response is a powerful object for inspecting the results of the request. Let’s make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:
>>>>>> response = (”)
In this example, you’ve captured the return value of get(), which is an instance of Response, and stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.
Status Codes
The first bit of information that you can gather from Response is the status code. A status code informs you of the status of the request.
For example, a 200 OK status means that your request was successful, whereas a 404 NOT FOUND status means that the resource you were looking for was not found. There are many other possible status codes as well to give you specific insights into what happened with your request.
By accessing. status_code, you can see the status code that the server returned:
>>>>>> atus_code
200. status_code returned a 200, which means your request was successful and the server responded with the data you were requesting.
Sometimes, you might want to use this information to make decisions in your code:
if atus_code == 200:
print(‘Success! ‘)
elif atus_code == 404:
print(‘Not Found. ‘)
With this logic, if the server returns a 200 status code, your program will print Success!. If the result is a 404, your program will print Not Found.
requests goes one step further in simplifying this process for you. If you use a Response instance in a conditional expression, it will evaluate to True if the status code was between 200 and 400, and False otherwise.
Therefore, you can simplify the last example by rewriting the if statement:
if response:
else:
print(‘An error has occurred. ‘)
Keep in mind that this method is not verifying that the status code is equal to 200. The reason for this is that other status codes within the 200 to 400 range, such as 204 NO CONTENT and 304 NOT MODIFIED, are also considered successful in the sense that they provide some workable response.
For example, the 204 tells you that the response was successful, but there’s no content to return in the message body.
So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code.
Let’s say you don’t want to check the response’s status code in an if statement. Instead, you want to raise an exception if the request was unsuccessful. You can do this using. raise_for_status():
import requests
from requests. exceptions import HTTPError
for url in [”, ”]:
try:
response = (url)
# If the response was successful, no Exception will be raised
response. raise_for_status()
except HTTPError as _err:
print(f’HTTP error occurred: {_err}’) # Python 3. 6
except Exception as err:
print(f’Other error occurred: {err}’) # Python 3. 6
If you invoke. raise_for_status(), an HTTPError will be raised for certain status codes. If the status code indicates a successful request, the program will proceed without that exception being raised.
Now, you know a lot about how to deal with the status code of the response you got back from the server. However, when you make a GET request, you rarely only care about the status code of the response. Usually, you want to see more. Next, you’ll see how to view the actual data that the server sent back in the body of the response.
Content
The response of a GET request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response, you can view the payload in a variety of different formats.
To see the response’s content in bytes, you use. content:
>>> ntent
b'{“current_user_url”:”, “current_user_authorizations_html_url”:”/client_id}”, “authorizations_url”:”, “code_search_url”:”query}{&page, per_page, sort, order}”, “commit_search_url”:”query}{&page, per_page, sort, order}”, “emails_url”:”, “emojis_url”:”, “events_url”:”, “feeds_url”:”, “followers_url”:”, “following_url”:”/target}”, “gists_url”:”/gist_id}”, “hub_url”:”, “issue_search_url”:”query}{&page, per_page, sort, order}”, “issues_url”:”, “keys_url”:”, “notifications_url”:”, “organization_repositories_url”:”org}/repos{? type, page, per_page, sort}”, “organization_url”:”org}”, “public_gists_url”:”, “rate_limit_url”:”, “repository_url”:”owner}/{repo}”, “repository_search_url”:”query}{&page, per_page, sort, order}”, “current_user_repositories_url”:”? type, page, per_page, sort}”, “starred_url”:”/owner}{/repo}”, “starred_gists_url”:”, “team_url”:”, “user_url”:”user}”, “user_organizations_url”:”, “user_repositories_url”:”user}/repos{? type, page, per_page, sort}”, “user_search_url”:”query}{&page, per_page, sort, order}”}’
While. content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. response will do that for you when you access
>>>>>>
‘{“current_user_url”:”, “current_user_authorizations_html_url”:”/client_id}”, “authorizations_url”:”, “code_search_url”:”query}{&page, per_page, sort, order}”, “commit_search_url”:”query}{&page, per_page, sort, order}”, “emails_url”:”, “emojis_url”:”, “events_url”:”, “feeds_url”:”, “followers_url”:”, “following_url”:”/target}”, “gists_url”:”/gist_id}”, “hub_url”:”, “issue_search_url”:”query}{&page, per_page, sort, order}”, “issues_url”:”, “keys_url”:”, “notifications_url”:”, “organization_repositories_url”:”org}/repos{? type, page, per_page, sort}”, “organization_url”:”org}”, “public_gists_url”:”, “rate_limit_url”:”, “repository_url”:”owner}/{repo}”, “repository_search_url”:”query}{&page, per_page, sort, order}”, “current_user_repositories_url”:”? type, page, per_page, sort}”, “starred_url”:”/owner}{/repo}”, “starred_gists_url”:”, “team_url”:”, “user_url”:”user}”, “user_organizations_url”:”, “user_repositories_url”:”user}/repos{? type, page, per_page, sort}”, “user_search_url”:”query}{&page, per_page, sort, order}”}’
Because the decoding of bytes to a str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting. encoding before accessing
>>>>>> response. encoding = ‘utf-8’ # Optional: requests infers this internally
>>>
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from and deserialize it using (). However, a simpler way to accomplish this task is to use ():
>>>>>> ()
{‘current_user_url’: ”, ‘current_user_authorizations_html_url’: ‘/client_id}’, ‘authorizations_url’: ”, ‘code_search_url’: ‘query}{&page, per_page, sort, order}’, ‘commit_search_url’: ‘query}{&page, per_page, sort, order}’, ’emails_url’: ”, ’emojis_url’: ”, ‘events_url’: ”, ‘feeds_url’: ”, ‘followers_url’: ”, ‘following_url’: ‘/target}’, ‘gists_url’: ‘/gist_id}’, ‘hub_url’: ”, ‘issue_search_url’: ‘query}{&page, per_page, sort, order}’, ‘issues_url’: ”, ‘keys_url’: ”, ‘notifications_url’: ”, ‘organization_repositories_url’: ‘org}/repos{? type, page, per_page, sort}’, ‘organization_url’: ‘org}’, ‘public_gists_url’: ”, ‘rate_limit_url’: ”, ‘repository_url’: ‘owner}/{repo}’, ‘repository_search_url’: ‘query}{&page, per_page, sort, order}’, ‘current_user_repositories_url’: ‘? type, page, per_page, sort}’, ‘starred_url’: ‘/owner}{/repo}’, ‘starred_gists_url’: ”, ‘team_url’: ”, ‘user_url’: ‘user}’, ‘user_organizations_url’: ”, ‘user_repositories_url’: ‘user}/repos{? type, page, per_page, sort}’, ‘user_search_url’: ‘query}{&page, per_page, sort, order}’}
The type of the return value of () is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
Query String Parameters
One common way to customize a GET request is to pass values through query string parameters in the URL. To do this using get(), you pass data to params. For example, you can use GitHub’s Search API to look for the requests library:
# Search GitHub’s repositories for requests
response = (
”,
params={‘q’: ‘requests+language:python’}, )
# Inspect some attributes of the `requests` repository
json_response = ()
repository = json_response[‘items’][0]
print(f’Repository name: {repository[“name”]}’) # Python 3. 6+
print(f’Repository description: {repository[“description”]}’) # Python 3. 6+
By passing the dictionary {‘q’: ‘requests+language:python’} to the params parameter of (), you are able to modify the results that come back from the Search API.
You can pass params to get() in the form of a dictionary, as you have just done, or as a list of tuples:
>>>>>> (… ”,… params=[(‘q’, ‘requests+language:python’)],… )
You can even pass the values as bytes:
>>>>>> (… params=b’q=requests+language:python’,… )
Query strings are useful for parameterizing GET requests. You can also customize your requests by adding or modifying the headers you send.
Other HTTP Methods
Aside from GET, other popular HTTP methods include POST, PUT, DELETE, HEAD, PATCH, and OPTIONS. requests provides a method, with a similar signature to get(), for each of these HTTP methods:
>>>>>> (”, data={‘key’:’value’})
>>> (”, data={‘key’:’value’})
>>> (”)
>>> requests. options(”)
Each function call makes a request to the bin service using the corresponding HTTP method. For each method, you can inspect their responses in the same way you did before:
>>> response. headers[‘Content-Type’]
‘application/json’
>>> response = (”)
>>> json_response = ()
>>> json_response[‘args’]
{}
Headers, response bodies, status codes, and more are returned in the Response for each method. Next you’ll take a closer look at the POST, PUT, and PATCH methods and learn how they differ from the other request types.
The Message Body
According to the HTTP specification, POST, PUT, and the less common PATCH requests pass their data through the message body rather than through parameters in the query string. Using requests, you’ll pass the payload to the corresponding function’s data parameter.
data takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with.
For example, if your request’s content type is application/x-www-form-urlencoded, you can send the form data as a dictionary:
You can also send that same data as a list of tuples:
>>>>>> (”, data=[(‘key’, ‘value’)])
If, however, you need to send JSON data, you can use the json parameter. When you pass JSON data via json, requests will serialize your data and add the correct Content-Type header for you.
is a great resource created by the author of requests, Kenneth Reitz. It’s a service that accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic POST request:
>>>>>> response = (”, json={‘key’:’value’})
>>> json_response[‘data’]
‘{“key”: “value”}’
>>> json_response[‘headers’][‘Content-Type’]
You can see from the response that the server received your request data and headers as you sent them. requests also provides this information to you in the form of a PreparedRequest.
Inspecting Your Request
When you make a request, the requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.
You can view the PreparedRequest by accessing. request:
>>> quest. headers[‘Content-Type’]

b'{“key”: “value”}’
Inspecting the PreparedRequest gives you access to all kinds of information about the request being made such as payload, URL, headers, authentication, and more.
So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they’re unauthenticated requests to public APIs. Many services you may come across will want you to authenticate in some way.
Authentication
Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called auth, which allows you to pass your credentials.
One example of an API that requires authentication is GitHub’s Authenticated User API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to get():
>>>>>> from getpass import getpass
>>> (”, auth=(‘username’, getpass()))
The request succeeded if the credentials you passed in the tuple to auth are valid. If you try to make this request with no credentials, you’ll see that the status code is 401 Unauthorized:

When you pass your username and password in a tuple to the auth parameter, requests is applying the credentials using HTTP’s Basic access authentication scheme under the hood.
Therefore, you could make the same request by passing explicit Basic authentication credentials using HTTPBasicAuth:
>>>>>> from import HTTPBasicAuth
>>> from getpass import getpass
>>> (… auth=HTTPBasicAuth(‘username’, getpass())… )
Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. requests provides other methods of authentication out of the box such as HTTPDigestAuth and HTTPProxyAuth.
You can even supply your own authentication mechanism. To do so, you must first create a subclass of AuthBase. Then, you implement __call__():
from import AuthBase
class TokenAuth(AuthBase):
“””Implements a custom authentication scheme. “””
def __init__(self, token):
= token
def __call__(self, r):
“””Attach an API token to a custom auth header. “””
r. headers[‘X-TokenAuth’] = f'{}’ # Python 3. 6+
return r
(”, auth=TokenAuth(‘12345abcde-token’))
Here, your custom TokenAuth mechanism receives a token, then includes that token in the X-TokenAuth header of your request.
Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth.
While you’re thinking about security, let’s consider dealing with SSL Certificates using requests.
SSL Certificate Verification
Any time the data you are trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL Certificate is critical.
The good news is that requests does this for you by default. However, there are some cases where you might want to change this behavior.
If you want to disable SSL Certificate verification, you pass False to the verify parameter of the request function:
>>>>>> (”, verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: InsecureRequestWarning)
requests even warns you when you’re making an insecure request to help you keep your data safe!
Performance
When using requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.
Timeouts
When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang.
By default, requests will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the timeout parameter. timeout can be an integer or float representing the number of seconds to wait on a response before timing out:
>>>>>> (”, timeout=1)
>>> (”, timeout=3. 05)
In the first request, the request will timeout after 1 second. In the second request, the request will timeout after 3. 05 seconds.
You can also pass a tuple to timeout with the first element being a connect timeout (the time it allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection):
>>>>>> (”, timeout=(2, 5))
If the request establishes a connection within 2 seconds and receives data within 5 seconds of the connection being established, then the response will be returned as it was before. If the request times out, then the function will raise a Timeout exception:
from requests. exceptions import Timeout
response = (”, timeout=1)
except Timeout:
print(‘The request timed out’)
print(‘The request did not time out’)
Your program can catch the Timeout exception and respond accordingly.
The Session Object
Until now, you’ve been dealing with high level requests APIs such as get() and post(). These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.
Underneath those abstractions is a class called Session. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session instance directly.
Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session:
from getpass import getpass
# By using a context manager, you can ensure the resources used by
# the session will be released after use
with ssion() as session:
= (‘username’, getpass())
# Instead of (), you’ll use ()
response = (”)
# You can inspect the response just like you did before
print(response. headers)
print(())
Each time you make a request with session, once it has been initialized with authentication credentials, the credentials will be persisted.
The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a Session, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one.
Max Retries
When a request fails, you may want your application to retry the same request. However, requests will not do this for you by default. To apply this functionality, you need to implement a custom Transport Adapter.
Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to to retry three times before finally raising a ConnectionError. You would build a Transport Adapter, set its max_retries parameter, and mount it to an existing Session:
from apters import HTTPAdapter
from requests. exceptions import ConnectionError
github_adapter = HTTPAdapter(max_retries=3)
session = ssion()
# Use `github_adapter` for all requests to endpoints that start with this URL
(”, github_adapter)
(”)
except ConnectionError as ce:
print(ce)
When you mount the HTTPAdapter, github_adapter, to session, session will adhere to its configuration for each request to Timeouts, Transport Adapters, and sessions are for keeping your code efficient and your application resilient.
Conclusion
You’ve come a long way in learning about Python’s powerful requests library.
You’re now able to:
Make requests using a variety of different HTTP methods such as GET, POST, and PUT
Customize your requests by modifying headers, authentication, query strings, and message bodies
Inspect the data you send to the server and the data the server sends back to you
Work with SSL Certificate verification
Use requests effectively using max_retries, timeout, Sessions, and Transport Adapters
Because you learned how to use requests, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
How to Use the Python Requests Module With REST APIs

How to Use the Python Requests Module With REST APIs

Learn how to use the Python Requests module to interact with any REST API in the world.
Ben Lloyd Pearson | June 11, 2020
The clear, simple syntax of Python makes it an ideal language to interact with REST APIs, and in typical Python fashion, there’s a library made specifically to provide that functionality:. Python Requests is a powerful tool that provides the simple elegance of Python to make HTTP requests to any API in the world. At Nylas, we built our REST APIs for email, calendar, and contacts on Python, and we process over 500 million API requests a day, so naturally, we depend a ton on the Python Requests library.
In this guide, we’ll take a comprehensive look at making HTTP requests with Python Requests and learn how to use this functionality to integrate with REST a PDF of this article? Share it with a friend or save it for later reading.
Contents:
The roles of HTTP, APIs, and REST
How to use Python Requests with REST APIs
How to authenticate to a REST API
How to handle HTTP errors with Python Requests
How to make robust API Requests
The Roles of HTTP, APIs, and REST
An Application Programming Interface (API) is a web service that grants access to specific data and methods that other applications can access – and sometimes edit – via standard HTTP protocols, just like a website. This simplicity makes it easy to quickly integrate APIs into a wide variety of applications. REpresentational State Transfer (REST), is probably the most popular architectural style of APIs for web services. It consists of a set of guidelines designed to simplify client / server communication. REST APIs make data access much more straightforward and logical.
The Request
When you want to interact with data via a REST API, this is called a request. A request is made up of the following components:
Endpoint – The URL that delineates what data you are interacting with. Similar to how a web page URL is tied to a specific page, an endpoint URL is tied to a specific resource within an API.
Method – Specifies how you’re interacting with the resource located at the provided endpoint. REST APIs can provide methods to enable full Create, Read, Update, and Delete (CRUD) functionality. Here are common methods most REST APIs provide:
GET – Retrieve data
PUT – Replace data
POST – Create data
DELETE – Delete data
Data – If you’re using a method that involves changing data in a REST API, you’ll need to include a data payload with the request that includes all data that will be created or modified.
Headers – Contain any metadata that needs to be included with the request, such as authentication tokens, the content type that should be returned, and any caching policies.
The Response
When you perform a request, you’ll get a response from the API. Just like in the request, it’ll have a response header and response data, if applicable. The response header consists of useful metadata about the response, while the response data returns what you actually requested. This can be any sort of data, as it’s really dependent on the API. The text is usually returned as JSON, but other markdown languages like XML are also possible.
Let’s look at a simple example of a request and a response. In the terminal, we’ll use curl to make a GET request to the Open Notify API. This is a simple, yet nifty API that has information about astronauts that are currently in space:
curl -X GET ”
You should see a response in JSON format that lists data about these astronauts, at the time of this article there are three people on a historic trip to the International Space Station:
{
“number”: 3,
“message”: “success”,
“people”: [
“craft”: “ISS”,
“name”: “Chris Cassidy”},
“name”: “Anatoly Ivanishin”},
“name”: “Ivan Vagner”}]}
How to Use Python Requests with REST APIs
Now, let’s take a look at what it takes to integrate with a REST API using Python Requests. First, you’ll need to have the necessary software; make sure you have Python and pip installed on your machine. Then, head over to the command line and install the python requests module with pip:
pip install requests
Now you’re ready to start using Python Requests to interact with a REST API, make sure you import the Requests library into any scripts you want to use it in:
import requests
How Request Data With GET
The GET method is used to access data for a specific resource from a REST API; Python Requests includes a function to do exactly this.
response = (“)
print(response)
>>>> Response<200>
The response object contains all the data sent from the server in response to your GET request, including headers and the data payload. When this code example prints the response object to the console it simply returns the name of the object’s class and the status code the request returned (more on status codes later).
While this information might be useful, you’re most likely interested in the content of the request itself, which can be accessed in a few ways:
ntent() # Return the raw bytes of the data payload
() # Return a string representation of the data payload
() # This method is convenient when the API returns JSON
How to Use Query Parameters
Queries can be used to filter the data that an API returns, and these are added as query parameters that are appended to the endpoint URL. With Python Requests, this is handled via the params argument, which accepts a dictionary object; let’s see what that looks like when we use the Open Notify API to GET an estimate for when the ISS will fly over a specified point:
query = {‘lat’:’45’, ‘lon’:’180′}
response = (”, params=query)
print(())
The print command would return something that looks like this:
‘message’: ‘success’,
‘request’: {
‘altitude’: 100,
‘datetime’: 1590607799,
‘latitude’: 45. 0,
‘longitude’: 180. 0,
‘passes’: 5},
‘response’: [
{‘duration’: 307, ‘risetime’: 1590632341},
{‘duration’: 627, ‘risetime’: 1590637934},
{‘duration’: 649, ‘risetime’: 1590643725},
{‘duration’: 624, ‘risetime’: 1590649575},
{‘duration’: 643, ‘risetime’: 1590655408}]}
How to Create and Modify Data With POST and PUT
In a similar manner as the query parameters, you can use the data argument to add the associated data for PUT and POST method requests.
# Create a new resource
response = (”, data = {‘key’:’value’})
# Update an existing resource
(”, data = {‘key’:’value’})
How to Access REST Headers
You can also retrieve metadata from the response via headers. For example, to view the date of the response, just specify that with the `headers` property:
print(response. headers[“date”])
>>>> ‘Wed, 11 June 2020 19:32:24 GMT’
For open APIs, that covers the basics. However, many APIs can’t be used by just anyone. For those, let’s go over how to authenticate to REST APIs.
How to Authenticate to a REST API
So far you’ve seen how to interact with open REST APIs that don’t require any authorization. However, many REST APIs require you to authenticate to them before you can access specific endpoints, particularly if they deal with sensitive data.
There are a few common authentication methods for REST APIs that can be handled with Python Requests. The simplest way is to pass your username and password to the appropriate endpoint as HTTP Basic Auth; this is equivalent to typing your username and password into a website.
(
”,
auth=HTTPBasicAuth(‘username’, ‘password’))
A more secure method is to get an access token that acts as an equivalent to a username/password combination; the method to get an access token varies widely from API to API, but the most common framework for API authentication is OAuth. Here at Nylas, we use three-legged OAuth to grant an access token for user accounts that is restricted to scopes that define the specific data and functionality that can be accessed. This process is demonstrated in the Nylas Hosted Auth service.
Once you have an access token, you can provide it as a bearer token in the request header: this is the most secure way to authenticate to a REST API with an access token:
my_headers = {‘Authorization’: ‘Bearer {access_token}’}
response = (”, headers=my_headers)
There are quite a few other methods to authenticate to a REST API, including digest, Kerberos, NTLM, and AuthBase. The use of these depends on the architecture decisions of the REST API producer.
Use Sessions to Manage Access Tokens
Session objects come in handy when working with Python Requests as a tool to persist parameters that are needed for making multiple requests within a single session, like access tokens. Also, managing session cookies can provide a nice performance increase because you don’t need to open a new connection for every request.
session = ssion()
({‘Authorization’: ‘Bearer {access_token}’})
response = (”)
How to Handle HTTP Errors With Python Requests
API calls don’t always go as planned, and there’s a multitude of reasons why API requests might fail that could be the fault of either the server or the client. If you’re going to use a REST API, you need to understand how to handle the errors they output when things go wrong to make your code more robust. This section covers everything you need to know about handling HTTP errors with Python Requests.
The Basics of HTTP Status Codes
Before we dive into the specifics of Python Requests, we first need to take a step back and understand what HTTP status codes are and how they relate to errors you might encounter.
All status codes fall into one of five categories.
1xx Informational – Indicates that a request has been received and that the client should continue to make the requests for the data payload. You likely won’t need to worry about these status codes while working with Python Requests.
2xx Successful – Indicates that a requested action has been received, understood, and accepted. You can use these codes to verify the existence of data before attempting to act on it.
3xx Redirection – Indicates that the client must make an additional action to complete the request like accessing the resource via a proxy or a different endpoint. You may need to make additional requests, or modify your requests to deal with these codes.
4xx Client Error – Indicates problems with the client, such as a lack of authorization, forbidden access, disallowed methods, or attempts to access nonexistent resources. This usually indicates configuration errors on the client application.
5xx Server Error – Indicates problems with the server that provides the API. There are a large variety of server errors and they often require the API provider to resolve.
How to Check for HTTP Errors With Python Requests
The response objects has a status_code attribute that can be used to check for any errors the API might have reported. The next example shows how to use this attribute to check for successful and 404 not found HTTP status codes, and you can use this same format for all HTTP status codes.
if (atus_code == 200):
print(“The request was a success! “)
# Code here will only run if the request is successful
elif (atus_code == 404:
print(“Result not found! “)
# Code here will react to failed requests
To see this in action, try removing the last letter from the URL endpoint, the API should return a 404 status code.
If you want requests to raise an exception for all error codes (4xx and 5xx), you can use the raise_for_status() function and catch specific errors using Requests built-in exceptions. This next example accomplishes the same thing as the previous code example.
try:
response. raise_for_status()
# Additional code will only run if the request is successful
except TPError as error:
print(error)
# This code will run if there is a 404 error.
TooManyRedirects
Something that is often indicated by 3xx HTTP status codes is the requirement to redirect to a different location for the resource you’re requesting. This can sometimes result in a situation where you end up with an infinite redirect loop. The Python Requests module has the TooManyRedirects error that you can use to handle this problem. To resolve this problem, it’s likely the URL you’re using to access the resource is wrong and needs to be changed.
except oManyRedirects as error:
You can optionally use the request options to set the maximum number of redirects:
response = (”, max_redirects=2)
Or disable redirecting completely within your request options:
response = (”, allow_redirects=False)
ConnectionError
So far, we’ve only looked at errors that come from an active server. What happens if you don’t receive a response from the server at all? Connection errors can occur for many different reasons, including a DNS failure, refused connection, internet connectivity issues or latency somewhere in the network. Python Requests offers the ConnectionError exception that indicates when your client is unable to connect to the server.
except nnectionError as error:
This type of error might be temporary, or permanent. In the former scenario, you should retry the request again to see if there is a different result. In the latter scenario, you should make sure you’re able to deal with a prolonged inability to access data from the API, and it might require you to investigate your own connectivity issues.
Timeout
Timeout errors occur when you’re able to connect to the API server, but it doesn’t complete the request within the allotted amount of time. Similar to the other errors we’ve looked at, Python Requests can handle this error with a Timeout exception:
response = (”, timeout=0. 00001)
except requests. Timeout as error:
In this example, the timeout was set as a fraction of a second via the request options. Most APIs are unable to respond this quickly, so the code will produce a timeout exception. You can avoid this error by setting longer timeouts for your script, optimizing your requests to be smaller, or setting up a retry loop for the request. This can also sometimes indicate a problem with the API provider. One final solution is to incorporate asynchronous API calls to prevent your code from stopping while it waits for larger responses.
How to Make Robust API Requests
As we’ve seen, the Requests module elegantly handles common API request errors by utilizing exception handling in Python. If we put all of the errors we’ve talked about together, we have a rather seamless way to handle any HTTP request error that comes our way:
response = (”, timeout=5)
except TPError as errh:
print(errh)
except nnectionError as errc:
print(errc)
except requests. exceptions. Timeout as errt:
print(errt)
except questException as err:
print(err)
If you’ve made it this far, congrats! You’re well on your way to becoming a Python Requests wizard for whom no REST API is too great a match. Want to keep learning? We have tons of knowledgable Python experts here at Nylas, and we have in-depth content on our blog about packaging and deploying Python code to production, and using environment variables to make your Python code more secure.
Ben Lloyd Pearson
Ben is the Developer Advocate for Nylas. He is a triathlete, musician, avid gamer, and loves to seek out the best breakfast tacos in Austin, Texas.

Frequently Asked Questions about python requests example

How do you make a request in Python?

PUT method – Python requestsRequests library is one of the important aspects of Python for making HTTP requests to a specified URL. … Python’s requests module provides in-built method called put() for making a PUT request to a specified URI. … Example – … Output –More items…•Sep 22, 2021

How does Python request work?

Python requests module has several built-in methods to make Http requests to specified URI using GET, POST, PUT, PATCH or HEAD requests. A Http request is meant to either retrieve data from a specified URI or to push data to a server. It works as a request-response protocol between a client and a server.Mar 12, 2020

What does requests Request do in Python?

Definition and Usage The requests module allows you to send HTTP requests using Python. The HTTP request returns a Response Object with all the response data (content, encoding, status, etc).

About the author

proxyreview

If you 're a SEO / IM geek like us then you'll love our updates and our website. Follow us for the latest news in the world of web automation tools & proxy servers!

By proxyreview

Recent Posts

Useful Tools