 |
| $17,000 AWS bill in the making. |
Part 2 Here
The problem with the Heartbleed bug is you never know when and where you'll get hit. Actually, this is true for all security breaches.
Yesterday, I received an e-mail from Amazon asking me to update my credit card info for one of my personal Amazon Web Services (AWS) accounts. I logged in and saw that my running total for April was over $5,300. My typical monthly bill is less than $6.00 which is about 1,000 times less.
At first I thought it was a mistake. I hadn't fired up any EC2 instances this month and my account had no EC2 instances running in my region. I filled out a billing inquiry request form and selected the "call me now" option. Within a minute my phone rang and I was speaking to an AWS customer service rep.
I explained that I've been an AWS customer since 2007 and I've never seen a billing issue like this.
He said, "We've been seeing more and more of this. Check your spot EC2 instances in other regions and you'll see high end instances running."
Sure enough, he was right. In Tokyo, São Paulo, Sydney, and Singapore I had expensive server instances running.
"Your AWS credentials have been compromised," he said.
How did they get compromised? When did it happen? Was my development machine hacked? It couldn't be my Time Capsule since that's encrypted. Were one of my physical servers hacked? Did I have a backup, sitting on a server, somewhere, that was hacked? Am I about to get stuck with a $5,360 bill?
"'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.
"When did this happen?" I asked.
"Let me look," he replied.
My mind was still racing as I tried to figure out the source of the breach.
"These instances were spun up on April 2," he said.
Very smart; launch the attack early in the month so the victim won't know anything's wrong until they get next month's bill.
"Is this related to Heartbleed?" I asked. It had to be.
"No, it's just a case of your AWS credentials getting compromised," he answered.
He walked me through the steps to secure my account.
"Can you see what 'they' were doing with these spot instances?" I asked.
"No, we can't see inside the instances. But, they're usually mining for Bitcoin," he answered.
Ahh, now that makes sense. Spend $5,000 of someone else's money to mine, say, $1,000 of Bitcoin for yourself. Can't follow that money trail.
"I'm going to send you a questionnaire. Please fill it out describing what happened and, due to the large amount involved, I'll need a manager to review it. But you won't have to pay for what you're not responsible for," he said.
I let out a sigh of relief.
I was still dripping with sweat since I'd just returned from a run when I saw the initial e-mail from Amazon. While I was in the shower it hit me. I know how my AWS credentials were compromised. But I'll need to do a little more research first.
4/15/2014 Update: How did this happened? See Part 2 of this story to find out.
Author: Joe Moreno
This feels like a novel cliffhanger and I'm waiting for the next book to find out how your credentials were compromised.
ReplyDeleteHow?! How long is this shower anyway?
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteAny chance you have a Github account, from time to time people forget to clear out AWS credentials from code uploaded to GitHub. These are publicly accessible and can be scraped by people with nefarious intentions.
ReplyDeleteThis is good to know. Any reccomendations for securing the instances (say making it air tight) and secondly can logging and monitoring for account level activity made very tight ?
ReplyDeleteThis is good to know. Any reccomendations for securing the instances (say making it air tight) and secondly can logging and monitoring for account level activity made very tight ?
ReplyDeleteCloudTrail can help identify where credentials are being used. Currently available in US-East-1 and US-West-2, it captures API calls for EC2 and other services. I use SumoLogic to analyze mine and wrote a blog post to help others get started: http://blog.joehack3r.com/cloudtrail-and-sumologic-getting-started/
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteHere's part 2 of the story about how my AWS credentials were compromised. Long story short, they were probably checked into my GitHub account.
ReplyDeletehttp://blog.joemoreno.com/2014/04/5000-security-breach-part-2.html
So, not to be a spammer but I helped build a company around mitigating exactly this sort of problem -- Cloudability.com. You can set up budget alerts, see reports and trends about your spending over time, etc.
ReplyDeleteFor Amazon you just tell it to put your billing and usage data in a bucket, and set up an IAM credential that just has access to read that bucket and nothing else and they gather the data every couple hours or so.