Gunicorn, Django, Gevent: Spawned threads are blocking

Vote count:

0

we recently switched to Gunicorn using the gevent worker.

On our website, we have a few tasks that take a while to do. Longer than 30 seconds.

Preamble

We did the whole celery thing already, but these tasks are run so rarely that its just not feasible to keep celery and redis running all the time. We just do not want that. We also do not want to start celery and redis on demand. We want to get rid of it. (I'm sorry for this, but I want to prevent answers that go like: "Why dont you use celery, it's great!")

The tasks we want to run asynchronously

I'm talking about tasks that perform 3000 SQL queries (inserts) that have to be performed one after each other. This is not done all too often. We limited to running only 2 of these tasks at once as well. They should take like 2-3 minutes.

The approach

Now, what we are doing now is taking advantage of the gevent worker and gevent.spawn the task and return the response.

The problem

I found that the spawned threads are actually blocking. As soon as the response is returned, the task starts running and no other requests get processed until the task stops running. The task will be killed after 30s, the gunicorn timeout. In order to prevent that, I use time.sleep() after every other SQL query, so the server gets a chance to respond to requests, but I dont feel like this is the point.

The setup

We run gunicorn, django and use gevent. The behaviour described occurs in my dev environment and using 1 gevent worker. In production, we will also run only 1 worker (for now). Also, running 2 workers did not seem to help in serving more requests while a task was blocking.

TLDR

• We consider it feasible to use a gevent thread for our 2 minute task (over celery)


• We use gunicorn with gevent and wonder why a thread spawned with gevent.spawn is blocking


• Is the blocking intended or is our setup wrong?

Thank you!

asked 44 secs ago

user2033511

376