Lessons Learned from Building a Silly Twitter Bot

Why Have Side Projects?

Whenever I am asked the best ways to learn data science, I always encourage folks to find a problem to solve that is interesting and unrelated to work. In other words, a side project.

The rationale for this advice is simple: learning new tech skills is difficult. Therefore, you should tackle problems that bring as much joy as possible while also avoiding email entanglements or late-night Slack boondoggles.

Building a Twitter Bot

One my ongoing side projects (what I refer to as “dessert”) is building and maintaining a Twitter bot that behaves like the famed “W” Flag flown above the Wrigley Field scoreboard each time the Cubs win.

Although this seems like a fairly lightweight data engineering task (and probably is for people who already know how), I wanted to gain some low-risk, low-cost experience putting a data pipeline into production.

Three years later, I am still learning and optimizing. And it’s still a fun, silly project that gives me just a little something else to root for beyond the team on the field.

Here are some lessons I’ve learned along the way.

Lesson #1: Putting something into production is totally different than doing it locally

For several years in a row, I produced Twitter Analytics during the Annual Meeting for the National Association for College Admissions Counseling (NACAC). ¹

These visualization involved the same basic pipeline (API call, data pull, aggregation and posting), but required my sitting in front of my computer every night to manually run the script. Given the anticipation for these results ² it was on my list until it wasn’t.

Because I wasn’t committed to watching every game all season, and had no plans to manually run the script each time the final out was recorded, I created a production pipeline using Amazon Web Services (AWS).

Initially, the stack was

An R script that:

✅ Pulls raw data from the MLB in-game API
✅ Determines the current state and score of the Cubs game
✅ Creates a data frame with the fields of interest (home score, away score, winning pitcher, losing pitcher, date, time, venue)
✅ Utilizes the rtweet package to post a Tweet IF the game is complete AND the Cubs won
✅ Logs a copy of the data frame along with a time stamp

Running inside of:

✅ AWS EC2 t2.micro instance

EC2 stands for “elastic compute”. Think of this as a computer in the cloud of varying sizes and configurations that you can build as you want and use when you want. Critically, you only pay for what you use. The micro is one of the smallest with a single core and 1GB of ram. If you had a much larger job, you could buy time on a much larger machine and subsequently pay a much larger cost.

✅ RStudio AMI (Ubuntu 18.04-LTS-64bit, RStudio 1.2.1335, R 3.6.0).

AMI stands for Amazon Machine Image. Rather than building a full remote computer and installing the necessary software, people have done the work for you already. In my case, it comes pre-installed with R and RStudio
running on a Linux machine.

✅ 20GB attached EBS volume - the minimum requirement for the AMI.

EBS is Elastic Block Store. This is a flexible mounted hard drive that is attached to the machine above.

Running on a cronjob that:

✅ Re-runs the script every 2 minutes
✅ Between April and October (the span of the baseball season)

Lesson #2: Optimize everything, including costs

After getting the basic setup deployed, the next challenge involved minimizing costs.

To begin, the entire setup is relatively inexpensive.

Given the very small storage requirements, the EBS pay-for-what-you-use model runs about seven cents per day. Assuming no giant jumps in data storage, this cost is essentially fixed.

The real variability comes from the EC2 compute time, which, on this t2.micro machine, runs at $0.0116 cents per hour. Back-of-the-napkin, the whole shebang would cost around $0.35 cents per day, or $70 for the full season ³ if it ran continuously.

For context, a full MLB.tv package costs $129. That’s to watch every single game from almost every team, all season long. Conversely, it’s possible that a fully headless/serverless script could run for free.

Ultimately, the rationale for cost optimization was to replicate choices existing in the actual business world. Even though I might personally value this bot to the tune of $70 annually, a future CEO or CTO might have a different take when compute and storage costs put real strains on budgets or organizational ambitions. If this fictional CTO were to ask me to shave 10% of our AWS budget, I wanted to experience this scenario at the penny level rather than the “burn rate” level.

To minimize costs, I sought to programmatically narrow the window of time when the EC2 instance was running.

1. Limit the server run time to a reasonable window when Cubs games would end
2. Turn off instance on days when the Cubs didn’t play
3. Turn off instance when Cubs had already won for the day
4. Turn off the instance when the Cubs had already lost for the day

Step #1 isn’t as easy it might sound. During the 2021 season, Cubs games will start at 28 different times across four American time zones. Furthermore, the un-timed nature of baseball leads to highly variant durations.

In 2019, the shortest Cubs game wrapped up in a tidy 145 minutes while the longest ran 336. The average duration was 191 minutes with a whopping standard deviation of 27.4 minutes! Think about that the next time you sit down to watch baseball. Random variation will determine how you spend nearly a half hour of your life! 162 times per year.

To avoid all that pesky variation, one solution could be standardization. Spinning up the server every morning at, say, 2am when it is highly likely that the preceding day’s game has concluded would be easy and very cheap.

This approach, while pragmatic, dodges the product requirement that the bot represents a close to real-time version of the actual flag, which is already flying by the time Wrigleyville denizens crack open a post-game Old Style ⁴.

This required creating:

✅ Lambda function to start the server
✅ Lambda function to stop the server
✅ Cloudwatch event triggers to fire the start function at 5pm CST every day, and one to fire the stop function at 3am CST every day.

This range reduces server run time from 24 hours to a maximum of 10. It is possible that this range will delay Tweets from quickly played, early starting games, but is unlikely to shut down before the completion a late night, extra inning barnburner.

✅ Logic in the R script that overrides the Cloudwatch end time if the Cubs win or lose before the Cloudwatch stop trigger.

The real server wastage occurs when the instance runs past the completion of a game or on days when the Cubs don’t play at all. Having an outside boundary is acceptable to ensure that the bot is awake when any random game ends, but it is sub-optimal to assume ten hours of run time when the average game ends after three. To alleviate this, I created some logic to shut down the server once a game has wrapped up.

Lesson #3: Eventually, you should learn some Python

As a longtime R enthusiast I have found that every tool required for my professional and personal work so far, has been totally do-able within the R ecosystem. I have gotten deeper into Python over the last few years, mostly as a way to be a better-than-average manager of multi-lingual data science teams ⁵.

But, when it comes to data engineering tasks in AWS, R is simply not supported. Much of the AWS ecosystem has a Pythonic version including some key features in Lambda (serverless functions) and Glue (scaled ETL).

Additionally, basic connections to the AWS ecosystem with Python’s boto3 are well supported and documented, while R implementations of similar tools are less so. In fact, several of the walk throughs and blogs about boto3 in R utilize the super-sweet reticulate package which effectively enables Python libraries and methods from within R.

While I am a mega-supporter of the R ecosystem and believe strongly in the tools and the community, there is no doubt that the world’s most popular programming language is widely used within and beyond the data science and analytics communities. If you think that your analytic work will eventually get handed off to data engineers working in the cloud, I recommend getting to a point where you can comfortably ski greens in Python.

Lesson #4: Mistakes are made. Things will break.

Every project, whether it’s data related or not, has a limitless number of ways where it could go sideways. Some of these start as unknown unknowns, but some are legitimate oopsies that exist between the keyboard and the chair.

In the spirit, of Data Mishaps Night and to help aspiring data folks alleviate imposter syndrome, I want to candidly and transparently expose a few blunders.

Lesson #5: Have a Punch List

There will always be bugs and new features and grand ambitions. To keep it all straight, be sure to keep a punch list. As you may recall, the first two items on mine are:

🏆 Create a version-controlled script in GitHub
🏆 Continuously deploy the newest version of the “software” by pulling directly from GitHub

But there are others

❗ Write logic to account for double headers
❗ Create logic to shut down system if the game is suspended for weather
❗ Create a variety of unique Tweets that can be randomly selected
❗ Add an inning score line
❗ Create modular functions allowing for similar processes with other teams
❗ Create an alert if there is a reasonable chance that a no-hitter is being thrown
❗ Re-build the entire setup on a Raspberry Pi that I can leave on until it catches on fire
❗ Re-build the entire setup using Python. For fun.
❗ Reduce storage by automatically deleting logs
❗ Prettify code. Delete unhelpful comments
❗ Convert all code to functions and into a documented R package
❗ Try building a complete AMI so I can tear down the full thing in the off-season

It’s fun and interesting to constantly re-imagine your own project. But it’s also important to remember that once you build something, you own and maintain it until it is sunsetted or removed from the product line.

Again, this bot is by me, for me, and totally irrelevant to the great wide world. But the same may not be true for your production code. Always be on the lookout for improvements, even if nobody is requesting them.

Which…brings me to the final lesson…

Lesson #6: Listen to Feedback

Sometimes people are requesting improvements. These can come in the form of GitHub issues, failure fire drills,⁸ feature requests, or just plain old product feedback.

Since FTWB goes out publicly on my personal Twitter feed people who may not be interested in the Cubs have made suggestions and also pointed out when things appear to be going awry.

It’s important to listen and make sure that your products (even silly ones like this) are meeting your end-user’s needs.

A few pieces of feedback I have gotten:

1. Make the Tweet more informative. At first it was just a “W” with a date which lacked information richness relative to 280 character limit. Over time, I added scores, the venue, and the winning and losing pitchers with more to come on the punch list.

2. Initially, I saw this as a project that would be of interest to the R community and therefore FTWB went out with the hashtag #rstats.

As it turned out, there are other bots that automatically Retweet anything containing #rstats. Each time the Cubs won, everyone following a totally different bot was hit up with the “W”.

It was recursive, and annoying. Someone reached out and politely applauded the project but indicated it wasn’t helpful to have an army of self-perpetuating bots Retweeting one another. This was great feedback which prompted me to remove the #rstats hashtag for good.

It also gave me the idea to give this bot its own hashtag #FlyTheWBot which makes it discoverable for people who care, yet mute-able for those who want to hear from @brad_weiner but not from #FlyTheWBot.

3. On opening day 2020, when everything went awry, it wasn’t long before someone pointed that out to me. Again, this is a low-stakes, silly Twitter bot, but in the real world, this would be the “Your-system-is-broken-cancel-your-weekend” kind of notification. This feedback is the kind that can’t go ignored. It is also reminder that end users or clients are often the first to notice when something is down. Have a process in place to quickly discover outages and a rapid response plan to correct them.

Lesson #7: Know When to Fold ’Em

Sometimes the juice just isn’t worth the squeeze and it’s worth knowing when it’s time to pack up. After Elon Musk took over Twitter in October, 2022, there were immediate and frustrating changes to the way that Twitter behaved. Most critcally, the API endpoints changed, along with the rules for running automated accounts. These changes, caused breakage in the underlying code, especially since the API wrappter R package {rtweet} was not being actively maintained. All of this led me to realize that as much fun as this project has been, it is no longer worth maintaining.

So, it is time to shut it down. The Twitter @flythewbot Twitter (now, X, which is so stupid) has Tweeted it’s last Cubs win.

Thanks for hanging out for the journey. Now, I need to find some other project to take on…

TL;DR

• Side projects are an awesome way to learn new tech skills
• Fun side projects make it easy to learn these skills and not have them feel like “work”
• Putting a data pipeline into production is complicated, but well worth the effort.
• Every data scientist should test out AWS or other cloud environments on a low-stakes, low-cost project. This project should simulate what a real-world setup that could be higher stakes.
• Mistakes will happen. Enjoy them and learn from them.
• Projects eventually run their course. Know when it is time to cut bait, when when it is time to keep fishing.
• Give thanks to everyone who helped along the way 👇

Acknowledgements

Others have made the case for silly or whimsical side projects and I want to point to the excellent rstudioconf::2020 talk and slides by Ryan Timpe on learning R with humorous side projects.

I am convinced that no Blogdown post actually gets out into the world without substantial Googling and assistance from the tremendous team at RStudio. Alison Hill deserves special praise for her concise code tutorials and seemingly endless generosity responding to issues and on various boards. I am currently running Hugo 0.74.3 and Blogdown version 0.19, so something tells me I will be asking my own questions in the very near future.

Thanks to everyone who follows me on Twitter. Very few of you probably like or care about the Chicago Cubs, but you deal with my nonsense anyway.

And, of course, thanks to all the Cubs fans out there. May there be many W flags in your future.