Uncovering Earnings Data

Despite living in a time where content creation is more accessible than ever, creators are also navigating a new era defined by analytics dashboards.

While many writers claim to ignore feedback or statistics, the reality is that metrics are essential to gauge the effectiveness of their work—whatever that may mean for individual goals.

The significance of dashboard functionality is evident in the latest platform updates. Alongside a renewed focus on authentic writing, a revamped dashboard has been introduced to help track user engagement and earnings.

For those who are data-savvy and wish to delve deeper, API integrations are available to extract this information for customized reporting.

However, even third-party APIs often miss a critical piece of data: the earnings figure, particularly the total monthly sum displayed in the Stats dashboard.

As I developed a personalized dashboard capturing nearly every engagement metric, I encountered a challenge when attempting to incorporate earnings data into my reports.

The issue is straightforward yet frustrating: this crucial monthly payout information isn't housed within Medium itself.

To access it, you'll need to set up an account with Stripe, which may not be on your radar.

Locating Your Data and Earnings

A common misconception in data engineering is that constructing a data pipeline is the most challenging aspect. In reality, one of the trickiest tasks is simply identifying where your data resides.

In this instance, while navigating the comprehensive Stripe API documentation for another project, I stumbled upon the aptly named "payouts" endpoint.

If you are part of the Medium Partner Program and your stories generate income, funds move from Medium to Stripe, and then from Stripe to your bank account. Since payouts occur monthly, this endpoint typically contains one record per month.

This data is readily accessible in your dashboard, where you can adjust the date range and other parameters to suit your analytical needs.

To access the backend of this dashboard, you will need an API key.

Authentication Details

Stripe’s reporting APIs function in a conventional manner; your API key acts as a bearer token.

If you see a key that starts with "Bearer" followed by a mix of letters and numbers, you're on the right track.

However, exercise caution: Stripe's developer portal provides both test and production keys. If you use a test key without corresponding test data, you will not receive any results.

The payouts endpoint (v1) is particularly user-friendly, as it does not require any additional parameters. Simply make a call to this URL: https://api.stripe.com/v1/payouts.

Let's validate the connection to ensure the configuration is correct.

def make_request(url, token):

logging.info("Making request to Stripe 'payout' endpoint...")

req = requests.get(url, headers=token)

logging.info(f"Response: {req.status_code}")

make_request(cfg.base_url, token)

For more insights on Python, SQL, and cloud computing, follow **Pipeline: Your Data Engineering Resource*.*

You can also **follow me* for my latest writings!*

Retrieving Earnings Data

By making an actual request, we should receive several months' worth of payment data formatted in JSON.

def make_request(url, token):

logging.info("Making request to Stripe 'payout' endpoint...")

req = requests.get(url, headers=token)

logging.info(f"Response: {req.status_code}")

if req.status_code == 200:

data = req.json()['data']

return data

else:

logging.info(f"Response is {req.status_code}.")

return 0

make_request(cfg.base_url, token)

To verify that we're receiving production data, we can check the "livemode" field.

If livemode is set to true, it indicates that we are receiving live production data.

The good news is that the returned JSON is not nested, meaning no further parsing is required.

We can directly access the "data" key and use it with Pandas’ pd.DataFrame() to create a quick data frame.

def make_request(url, token):

logging.info("Making request to Stripe 'payout' endpoint...")

req = requests.get(url, headers=token)

logging.info(f"Response: {req.status_code}")

if req.status_code == 200:

data = req.json()['data']

return data

else:

logging.info(f"Response is {req.status_code}.")

return 0

def format_df(data):

df = pd.DataFrame(data)

return df

def main():

data = make_request(cfg.base_url, token)

df = pd.DataFrame(data)

return df

main()

Now we can view a clearer, nearly finalized output of the payment data we need.

However, to utilize this data in a dashboard, you will likely need to refine specific fields through data frame manipulation.

Cleaning Up Your Earnings Data Frame

In my experience, the most problematic fields are the amount and arrival_date.

To clarify, arrival_date represents the date funds were received in Stripe from the payment source. The amount refers to the total income.

For privacy reasons, I will omit outputs that display the amount.

However, your amount field may appear inflated if it isn’t converted to a float correctly.

df['amount'] = df['amount'].astype(float) / 100

Similarly, the arrival_date field may be captured as a messy Unix timestamp that also needs conversion.

df['arrival_date'] = pd.to_datetime(df['arrival_date'], unit='s')

df['arrival_date'] = df['arrival_date'].dt.strftime("%Y-%m-%d")

After implementing these changes, the data frame will look much cleaner.

If you have had funds deposited into your Stripe account for an extended period (e.g., over a year), you may notice that the API does not return all values.

Stripe APIs utilize pagination, meaning you will only retrieve the first page of results.

If I were constructing a pipeline that requires historical data on a regular basis, this would be significant. However, since I only need to retrieve last month's earnings moving forward, I can take a shortcut for the remaining values.

By checking the Stripe dashboard, you can adjust the parameter for a custom date range.

Adjusting this will allow you to gather data for your account's entire history.

Unless you've maintained a Stripe integration for years, manually entering these values should not be overly burdensome. Remember, since this is a backfill, it only needs to be done once.

As a final step, I highly recommend summing your amount column and comparing it to the values displayed in your Stripe dashboard.

If the totals match, congratulations—you've successfully completed this process!