We migrated our stack from Heroku to AWS and cut 60% of operation cost
It took me 2 years to write this post for many reasons. Well I have no one to blame but me and start doing the actual writing 😇 Back then when Pretty.Tips was in its early phase, our technology stack was super simple. Everything ran on Heroku, from database, to frontend to backend to back-office. Before each campaign we estimated incoming traffic and provisioned instances in advance, a bit manual control but we were happy with Heroku services; the average technical operation cost was roughly 800 to 900 USD per month. Then Pretty.Tips made a move to scale up its business, we realized that scaled-along cost would be too high for a growing start-up. It was lucky that Pretty.Tips got some AWS credits from its accelerator program to do something amazing. The migration experience was both sweet and bitter.
It was sweet because the whole stack was migrated swiftly with no down time, much smoother than we had anticipated. We really enjoyed the stability and excellent customer supports from AWS. In the end we cut down our operation cost by 60%, which was one of the biggest achievements. It was bitter because the whole process was as painful as driving on bumpy road. I must say the tech team really strugged to get there; sleepless nights, gotchas, frustration, bugs, back-and-forth deployments, etc. you name it. Here are a few things we did on the road to get where to the happy state. Hope some can learn from it and save a lot of time/effort.
✅ We started with building a staging backend: by the time we thought of migrating to AWS, there were already 80k+ live customers to serve so we had to be careful with touching production. The initial staging environment contained only backend running on one Elastic Beanstalk (ELB) app. Once we felt good with the backend part, it was just natural to have more ELB applications for frontend and back-office. One may ask why ELB for frontend; SEO plays a crucial role for an e-commerce platform therefore server-side rendering is needed, so is ELB for frontend.
❌ Spending a good chunk of time on Cloud Formation wasn't good actually. Cloud Formation offers great flexibility but my advise is staying away from it if your needs are as simple as having staging/prod environments with simple web server and backend APIs. It may take an hour on AWS console to set that up, compared with days (or weeks) spending on Cloud Formation templates and auto deployment quirks.
✅ Migrate production database last: Once we were confident enough with all the setup on AWS, we made the DNS switch to point to ELB backend as picture below. The ELB still pointed to the database running on Heroku in case there was disruption we could quickly revert. After a few days observing, we migrated database instance from Heroku to AWS RDS to finish the migration. The celebration 🎉🚀 happened a week later after the whole Heroku environment was finally retired.
✅ Invest in Serverless: We just didn't stop there after everything was moved to AWS. A bunch of EC2 instances stayed idle most of the time or had very low resource utilisation, making us think about Serverless architecture. Basically serverless architecture assumes breaking your backend into small, short-lived, stateless API invocations; there is no dedicated host but dynamically allocated resources to run on. We had to fight timeout invocation with unclear logs at that time, a few days wasted on trying to run serverless lambda functions on VPC which we gladly ignored in the end. Amazon AWS actually admitted the issue and fixed a few months later. After having our backend and back-office run on serverless functions, the operation cost dropped significantly from $900 to less than $350/month. Tech team didn't have to think about dynamic but slow scalling of ELB while business team could happily spend more money on business operations. A great optimization I would call.
Fast forward to today, it is much easier to control your technology stack. Amazon, Google, Microsoft offer great developer experiences to run the entire business solution with almost instant feedbacks. I would still go to Heroku though for some quick proof of concept works, it's by far the most friendly environment to me. For medium to large projects I would advise planning/controlling your technology stack carefully because that would be one of the best investments you may do, sometimes it becomes a do-or-die challenge for small startups. Happy shipping, everyone.