Last Monday, I sent the usual newsletter to the users of CloudNews. But instead of receiving one unique email, they received dozens of them, some received more than a hundred in a few seconds.
As I don't want this to happen again. It's time for a bit of thinking.
Why did it happen?
Simply put, I didn't have any automated tests. I launched CloudNews in November after building it in 3 weeks. I preferred development speed and actual value for users over reliability and code quality.
Few years ago, I built a product that didn't get any traction. It had a staging environment, automated tests, and a complicated CI. It checked all the boxes of good practices. But it had no active users.
As an engineer, I was delighted with the result, it was working perfectly and it probably could have scaled. If only people were using it.
I now have a full time job, I can't spend months on a project only to see it failed miserably. My time is as limited as ever. Which is why I chose to skip the good practices this time.
What caused the issue?
I edited a critical part of my app, which I don't update often.
The module that sends emails. I updated a SQL request that fetches the users based on some attributes and a misconceived JOIN duplicated all email addresses.
I didn't realise it, because the local database was only populated with my own email address.
What could have been worse?
I could have been on the free tier of my email provider. I could have reached the daily limit on email sent in seconds. It then could have sent the remaining emails placed in a waiting queue when came the next day.
Well it happened too.
What action did I take to solve this?
A few things could have prevented it. I now have implemented most of them.
- My local database has the same data as the production database.
- I added automated tests on the critical parts of the app.
- I added safeguards triggered on abnormal situations.
For example, if there are more emails to be sent than total users. This triggers an exception. - I used to code on master (I know I know), now I code on a dev branch and review each pull requests.
- Finally I added GitHub Actions.
Conclusion
There is a fine line between being too scrappy and being too perfectionist. I now know the trade offs better and it taught me a valuable lesson.
I'll end this post with a borrowed conclusion from Francesco Di Lorenzo:
For me, the biggest learning in shipping and iterating on many projects has been the ability to recognize lagom — a Swedish word that means "not too much, not too little, just right.