Don’t Be Afraid to Cheat

It’s very easy to get caught up in the technology that surrounds your professional life.  Problems will be brought to you as technology issues with a simply statable problem that needs to be solved.  By taking a step back and examining the actual end goals, we often find the opportunities to “cheat” and solve the business problems without needing to solve the technical ones - and at the end of the day, the business problems are the only ones that matter (or indeed should be externally visible - if an outside observer can diagnose your technology issues then you’re doing something remarkably wrong).

Giving Days

About a decade ago now, we started talking to a prospect that ran high volume giving days.  They’d raise north of $10 Million in 24 hours for a slate of 500 or so non-profits.  What was even more remarkable is that they managed to do this despite dealing with what they saw as inevitable system issues - people would try to donate and get no website, unresponsive searches, failed transactions, all sorts of issues.

The year before we took the project on they’d worked with their solution provider who did all of the “normal” things - bigger databases, more servers, et cetera.  While they’d had a better year than the one before they were still running into serious issues.

Having seen that, I knew that the last thing I wanted to do was to keep playing the same game, so we decided to cheat.

The overall site was set up so that there was a landing page that displayed fundraising progress and a lot of static content, and also had a search box.  This allowed people to find a cause to support by doing a full text search on the back end that was often a source of difficulty for the provider.  Each NPO had a dedicated giving page with text, image and video information about their cause and a donation form.  Additionally there were overall leaderboard pages that would reflect everyone’s current fundraising totals - when they worked.

A Quick Analysis

We realized that almost all of the traffic that took down the sites was people not actively making donations.  Actual donation activity peaked at around 2-3 gifts per second which was well within the capabilities of our existing backend.  Building the leaderboards and displaying the fundraising totals were big drains, and while I still feel that I could have optimized the database and introduced other backend caches to correct those issues that wasn’t really cheaty enough.  We also realized that searching was killing the system, especially since it was the first thing that everybody wanted to do when they opened the site.

So how did we cheat?

Keep in mind that was considerably before the days where static websites communicating with RESTful backends had been accepted.  Our platform was one of the earliest commercial users of the JSONP pattern before it had been named as one.  So it was time to innovate.

First, we did a little basic templating.  We took all of the data for the various NPOs out of the database and into CSVs, then wrote a quick and dirty shell script to generate a completely static website with a separate HTML page for each NPO (along with the indexes, etc) that would be uploaded to Amazon S3.  Cloudfront and other similar services all took far too long to expire content and we didn’t want to risk that, but a public-readable S3 bucket cost pennies and worked amazingly well.  No data was ever read from anywhere other than the S3 bucket.

We took the same CSV of data and wrote a script to build out a JS array of search data, then wrote a script that would take a text string and perform a client-side text search against it.  The whole mess was around 2MB uncompressed, but only ~256K as it was gzipped for transfer.  We dropped it into a static search.html page that would check its URL for a ?s= parameter and give the appearance of a server-side search.  It took almost no time at all to run, and completely removed that load from our servers.

Finally the leaderboard data and overall fundraising progress were included as IFRAMEs (again, this was some time ago), also loaded from S3 URLs.  Those 3 or 4 files were regenerated every 15 seconds or so by a serverside cron job and uploaded - if anything bad happened (which it never did), people would see stale data rather than no data.

The only thing that our actual expensive platform had to do was process donations.  Not only was that traffic was sparse enough to handle without sweating, if it ever became problematic we’d have been taking in so much money that scaling the stateless backends or DBMS would have been a no-brainer.

Conclusion

I like to think that solving the technology problem that the original vendor had would have been both fairly simple and rewarding.  I also recognize that there was probably at least one smart person on their end who had the same attitude and was proven wrong.  Part of me still wants to roll up my sleeves and try to beat it the traditional way, but solving it the way that we did - while totally cheating - only took us about three man-weeks of development and scaled almost infinitely since the only stressed component was Amazon S3.

Solving technology problems is fun, rewarding, and ultimately missing the point. Applying the right about of technology to solve business problems as simply as possible gets the job done.