Respect your production data

I read yesterday this blog post: I Accidentally Deleted All Our Data by Taylor Fausak. Probably you’ll end with the same expression in your face that I did. An a palm covering it.

Something in advance. It takes GREAT courage and openness to tell in your blog this story. I think is really a great attitude about it.

Saying this, I must say that the whole story a recipe for disaster. Lots of steps make my spider-sense to tingle. Strongly.

Doing a script on the python interactive shell to update your production data, while in a convention, between presentations… Well, it’s not the right moment to do ANYTHING that could change your data. A quick look a monitoring tool, that’s grand. But anything more complex that that is highly risky. And specially using the interactive shell.

You have to RESPECT your production environment and data. Ideally, every change in production should be automated and tested before in advance. That means everything but the most extreme cases, like bugs that are blocking the whole application. Sometimes, in extraordinary cases, could necessary to take extraordinary measures. But it should always be treated with the proper caution.

You have to set all your attention each time you have to change anything on production and have a clear view in advance of what are you trying to do. Think really carefully what are you going to do. And double check everything you type. Every step that has not been previously tested on a staging environment is a possible disaster for your application.

Anyway, stories like that only make me remember how much attention should I put into changing production data and keep a healthy fear of what could happen. Treat your production environment with proper R-E-S-P-E-C-T or it can bite. Hard.

Bonus: Really, really, REALLY the best way of testing that something works is SAVING everything AGAIN????????

This code give me nightmares…

from mongoengine import connect
from models import Family
for family in Family.objects:

Database madness with mongoengine and SQLAchemy

Yesterday I gave a presentation in the Python Ireland October meeting about some work we are doing with mongoengine and SQLAchemy and how we are managing three databases (MS SQL server, MySQL and MongoDB) on an online football management game we are working on.

So, here are the slides, so feel free to make comments, ask questions and even criticize them!

You can also download the presentation on PDF here.

PD: When I talk about football game, I’m referring to soccer.

ORMs and threads

Do you remember the post from Joel Spolsky about leaking abstractions? It’s the kind of idea that, the first time I read, about it, was intrigued, but after some time, I began to see it on every place. There are from time to time some problems on my Python code (as well as in other high-level languages) that I am really glad to be able to have an idea of the underlaying low level C, or I will be struggling with some very weird, confusing problems. I have enough confusing and weird problems of my own to add more…

One of my recent leaking abstractions has come using a ORM, in particular mongoengine, but I think it will happen probably on every ORM. On a web application I am developing at the moment, we need to launch a thread to perform some operations, in a timed manner. A request comes to the server, launches a thread, and then that thread stores its status on the database. Then the user can check the status from the database (and do more operations, like pause, etc, but that I will leave that). While performing some tests on the application, I made the following code:

def testing():
    user = User.objects.get(TEST_USER)
    assert user.status == END

Inside the thread, the code looks similar to this

def thread(user_id):
    thread_user = User.objects.get(user_id)
    # Do things that take a while, but less than TIME
    thread_user.status = END

Ok, so we’re getting an object from the database, the object launches a thread that changes its state to END and saves it after a while. Well, not really… Obviously it’s not working (or I wouldn’t be writing this). But we all already know that threads are the root of all evil, and always have nasty hidden surprises.

The error I was making was assuming (and that’s the abstraction in my mind) that the ORM maps the database into memory, and that the copy is unique. After all, that’s why you have a database. But it’s not true. What it’s happening here it’s that we are creating two different objects in memory. I have (now) used two different names, user and thread_user. In my code I used the same name (user) which probably adds to the confusion. Each one reflects the status of the database when you read the database, but after that, you are not updating the object with the real information on the DB. So, the user object has still the starting status, the first one, as we haven’t refreshed it with the new and changed information that another, rogue thread, has changed while we naively thought that was under our control.

Usually, on a web application (at least the ones developed with high-level tools) the usual situation is having a request, read the data from the DB using a ORM, change something, and then save. We don’t have rogue threads interrupting that operations and requests can be processed fast enough. And even user data is different so two users probably don’t need to write any related information. BUT definitively another request (faster one) could interrupt the process and make the data to not be coherent. It’s going to be (extremely) rare in most applications, but in case of long, threaded operations, could be important to be aware of this and try not to relay on the ORM as a virtual copy of the DB, but to read and write in short operations. Or lock the database.

Just one more thing. It’s possible to use only one object in memory, and pass it to the thread, and avoiding this problems. But that could generate others, like not storing (and loading) any intermediate steps of the process. So, in case the thread is stopped (for example, a server restart), the process is totally lost. Any operation that takes time to execute will ideally have some “resume” process, so that will include storing the partial state, as well as a resume, which will need to read from the DB. Also, in this particular case, there are more than one thread working the same process, communicating through the DB.

But wait! There is still a little more unexpected and funny behavior!

To reload the user object, my first idea was to generate a refresh method, this way:

class User(mongoengine.Document):

     def refresh(self):
          ''' Refresh the object '''
          self = User.objects.get(

And again, it’s not working… 😦
Again, the problem is an abstraction. self it’s not always the object, not outside the method. It’s just a label (or pointer, if you know C) to the object. Yes, we have created a new object called selfwhich has the new (and correct) object. BUT the label user is still pointing to the not-updated object we have since the beginning.

So… no shortcuts, we will have to reload the object after the sleep to check that the object on the DB it’s behaving properly

def testing():
    user = User.objects.get(TEST_USER)
    user = User.objects.get(TEST_USER)
    assert user.status == END