I can’t believe what I did on Friday, probably the worst thing that I ever did in my working experience.
I was testing my local development environment, in particular I was testing the SQL scripts that I have for re-creating tables. On my SQL Manager, I have two tabs open, one for development database and the other one is for production database (as I was doing another task on production database).
Now, I wanted to drop all the tables on my local database, so I opened up a new query tab on SQL Manager and run my query there. Seconds later after hitting F5 to run the query, I realized that the query tab opened was connected to the production database not on my local database.
I was frozen, didn’t know what to do, was hoping that this is just a bad dream. A minute later, some people saying that they saw hard errors on our website, by this time I know it’s very real and I have to do something about it.
I quickly told DBA, Team leader and Production Manager, they all were very shocked needless to say. To cut the story short, for the next 2-3 hours or so the whole development and customer relationship team worked together to get the site up and running again (some of the data had to be re-entered because they didn’t exist on the backup).
As well as keeping the affected clients informed with the restoration progress. The people were understanding, of which I was very grateful. I was also quite amazed how the Product Manager calmly handled the situation, he lead the development team so well through the crisis. I guess the similar thing also happened on the customer team.
I was impressed as well how everyone able to work together to pull the company from the messy situation that I single handedly have created. I think I shouldn’t be trusted with administration privilege on production servers every again..
I always consider myself to be very cautious person, but I was proven wrong that day, a timely reminder for me.