Queue service with priorities and equal distribution

Vote count:

0

We are currently using a single Amazon SQS for job queueing. We are looking for a solution that has more features:

1) distribute workers evenly among users (who currently have open jobs)

If user A runs 100,000 jobs and right after that user B runs 10 jobs, the first 10 of both users should approximately be run in the same time. - Currently user B would have to wait for a long, long time.

2) nice to have: throttle workers to x jobs per second per category

Our jobs involve accessing third party systems. We would like to make sure that we don't send too many requests to them per time. But if we have lots of jobs that involve service S1, we don't want to throttle jobs involving service S2.

3) nice to have: job priorities

Two priorities would be okay, if arbitrary levels are too difficult.

Ideally, we would like a system that is well-known and that has hosted solutions (or is easy to maintain).

We need a throughput of millions per hour, with less than 100 bytes each message.

We are developing in ruby if that makes any difference.

asked 39 secs ago

Jan

852

Queue service with priorities and equal distribution