In this tutorial, we will see how to connect to the Tracking Stream API for a limited/unlimited amount of time, and generate a CSV file from the obtained results in 30 minutes time buckets.
You can find and download the source code for this example here.
Let's start by installing the request package that will be necessary for this example to work.
We can create our
main.py file, with the
listen_to_stream function in it:
Let's breakdown what was done here:
- We have added a retry strategy, in case the connection fails the first few times. We will use that later to ensure that we stay connected to the stream at all times.
- We have set up a
timeoutparameter, which will allow us to listen to the Tracking Stream for a specific duration if we want to
- We have called the Tracking Stream endpoint using
http.getafter creating a
- We are looping through the response lines and parsing the JSON object in each of them, adding it to a global array
target_updatedeclared at the top.
If we want our Tracking Stream connection to stay open and generate CSV files on the go, we need to have some sort of scheduling process that will generate the CSV files.
For that, we will use the package
apscheduler and its
We can now create our scheduler and its associated callback:
Let's go through what we have added here:
- We have created a
reset_bucket()function that will empty our data buckets from which we are generating every CSV file when the Background Scheduler triggers a new job.
- We have created the job function
export_to_csv_job(), which takes our bucket data, processes it, and writes it down in a CSV.
- We are launching our scheduled job, set up to every 30 minutes:
- We are handling error cases by removing our job when errors happen, or when the timeout is reached.
We can now add a connection manager, which will handle possible disconnections from the endpoint, and try reconnecting after a while.
The connection manager will look like that:
The connection manager will call our
listen_to_stream function, awaiting any possible exception to try the following:
- If the exception is
MaxRetries, the endpoint was called multiple time to no avail, which will trigger the connection manager to wait for 30 minutes before retrying to connect.
- The exception is
ConnectionLost, the connection manager will retry to connect right away.
Calling our function
We can now simply call our
connection_manager() function anywhere and start listening and exporting CSV files from the Tracking Stream endpoint!