HireFire¶
This is a Python package for HireFire – The Heroku Process Manager:
HireFire has the ability to automatically scale your web and worker dynos up and down when necessary. When new jobs are queued in to your application’s worker queue [..], HireFire will spin up new worker dynos to process these jobs. When the queue is empty, HireFire will shut down the worker dynos again so you’re not paying for idle workers.
HireFire also has the ability to scale your web dynos. When your web application experiences heavy traffic during certain times of the day, or if you’ve been featured somewhere, chances are your application’s backlog might grow to a point that your web application will run dramatically slow, or even worse, it might result in a timeout. In order to prevent this, HireFire will automatically scale your web dynos up when traffic increases to ensure that your application runs fast at all times. When traffic decreases, HireFire will spin down your web dynos again.
—from the HireFire frontpage
It supports the following Python queuing systems as backends:
Feel free to contribute other backends if you’re using a different queuing system.
Installation¶
Install the HireFire package with your favorite installer, e.g.:
pip install HireFire
Sign up for HireFire and set the HIREFIRE_TOKEN
environment variable
with the Heroku CLI as provided on the specific HireFire application page,
e.g.:
heroku config:set HIREFIRE_TOKEN=f69f0c0ddebe041248daf187caa6abb3e5d943ca
Now follow the quickstart guide below and don’t forget to tweak the options in the HireFire management system.
For more help see the Hirefire documentation.
Configuration¶
The hirefire
Python package currently supports two frameworks:
Django and Tornado. Implementations for other frameworks are planned but
haven’t been worked on: Flask, Pyramid (PasteDeploy), WSGI middleware, ..
Feel free to contribute one if you can’t wait.
The following guides imply you have defined at least one
hirefire.procs.Proc
subclass defined matching one of the processes in your
Procfile. For each process you want to monitor you have to have one subclass.
For example here is a Procfile
which uses RQ for the “worker” proccess:
web: python manage.py runserver
worker: DJANGO_SETTINGS_MODULE=mysite.settings rqworker high default low
Define a RQProc
subclass somewhere in your project, e.g.
mysite/procs.py
, with the appropriate attributes (name
and
queues
)
from hirefire.procs.rq import RQProc
class WorkerProc(RQProc):
name = 'worker'
queues = ['high', 'default', 'low']
See the procs API documentation if you’re using another backend. Now follow the framework specific guidelines below.
Django¶
Setting up HireFire support for Django is easy:
Add
'hirefire.contrib.django.middleware.HireFireMiddleware'
to yourMIDDLEWARE
setting# Use ``MIDDLEWARE_CLASSES`` prior to Django 1.10 MIDDLEWARE = [ 'hirefire.contrib.django.middleware.HireFireMiddleware', # ... ]
Make sure it’s the first item in the list/tuple.
Set the
HIREFIRE_PROCS
setting to a list of dotted paths to your procs. For the above example procHIREFIRE_PROCS = ['mysite.procs.WorkerProc']
Set the
HIREFIRE_TOKEN
setting to the token that HireFire shows on the specific application page (optional)HIREFIRE_TOKEN = 'f69f0c0ddebe041248daf187caa6abb3e5d943ca'
This is only needed if you haven’t set the
HIREFIRE_TOKEN
environment variable already (see the installation section how to do that on Heroku).Check that the middleware has been correctly setup by opening the following URL in a browser:
http://localhost:8000/hirefire/test
You should see an empty page with ‘HireFire Middleware Found!’.
You can also have a look at the page that HireFire checks to get the number of current tasks:
http://localhost:8000/hirefire/<HIREFIRE_TOKEN>/info
where
<HIREFIRE_TOKEN>
needs to be replaced with your token or – in case you haven’t set the token in your settings or environment – just usedevelopment
.
Tornado¶
Setting up HireFire support for Tornado is also easy:
Use
hirefire.contrib.tornado.handlers.hirefire_handlers
when defining yourtornado.web.Application
instanceimport os from hirefire.contrib.tornado.handlers import hirefire_handlers application = tornado.web.Application([ # .. some patterns and handlers ] + hirefire_handlers(os.environ['HIREFIRE_TOKEN'], ['mysite.procs.WorkerProc']))
Make sure to pass a list of dotted paths to the
hirefire_handlers
function.Set the
HIREFIRE_TOKEN
environment variable to the token that HireFire shows on the specific application page (optional)export HIREFIRE_TOKEN='f69f0c0ddebe041248daf187caa6abb3e5d943ca'
See the installation section above for how to do that on Heroku.
Check that the handlers have been correctly setup by opening the following URL in a browser:
http://localhost:8888/hirefire/test
You should see an empty page with ‘HireFire Middleware Found!’.
You can also have a look at the page that HireFire checks to get the number of current tasks:
http://localhost:8888/hirefire/<HIREFIRE_TOKEN>/info
where
<HIREFIRE_TOKEN>
needs to be replaced with your token or – in case you haven’t set the token as an environment variable – just usedevelopment
.
Flask¶
Setting up HireFire support for Flask is (again!) also easy:
The module
hirefire.contrib.flask.blueprint
provides abuild_hirefire_blueprint
factory function that should be called with HireFire token and procs as arguments. The result is a blueprint providing the hirefire routes and which should be registered inside your appimport os from flask import Flask from hirefire.contrib.flask.blueprint import build_hirefire_blueprint app = Flask(__name__) bp = build_hirefire_blueprint(os.environ['HIREFIRE_TOKEN'], ['mysite.procs.WorkerProc']) app.register_blueprint(bp)
Make sure to pass a list of dotted paths to the
build_hirefire_blueprint
function.Set the
HIREFIRE_TOKEN
environment variable to the token that HireFire shows on the specific application page (optional)export HIREFIRE_TOKEN='f69f0c0ddebe041248daf187caa6abb3e5d943ca'
See the installation section above for how to do that on Heroku.
Check that the handlers have been correctly setup by opening the following URL in a browser:
http://localhost:8080/hirefire/test
You should see an empty page with ‘HireFire Middleware Found!’.
You can also have a look at the page that HireFire checks to get the number of current tasks:
http://localhost:8080/hirefire/<HIREFIRE_TOKEN>/info
where
<HIREFIRE_TOKEN>
needs to be replaced with your token or – in case you haven’t set the token as an environment variable – just usedevelopment
.
Proc backends¶
Two base classes are includes that you can use to implement custom backends. All the other contributed backends use those base classes, too.
hirefire.procs.Proc
¶
-
class
hirefire.procs.
Proc
(name=None, queues=None)[source]¶ The base proc class. Use this to implement custom queues or other behaviours, e.g.:
import mysite.sekrit from hirefire import procs class MyCustomProc(procs.Proc): name = 'worker' queues = ['default'] def quantity(self): return sum([mysite.sekrit.count(queue) for queue in self.queues])
Parameters: -
name
= None¶ The name of the proc
-
quantity
(**kwargs)[source]¶ Returns the aggregated number of tasks of the proc queues.
Needs to be implemented in a subclass.
kwargs
must be captured even when not used, to allow for future extensions.The only kwarg currently implemented is
cache
, which is a dictionary made available for cross-proc caching. It is empty when the first proc is processed.
-
queues
= []¶ The list of queues to check
-
hirefire.procs.ClientProc
¶
-
class
hirefire.procs.
ClientProc
(*args, **kwargs)[source]¶ A special subclass of the
Proc
class that instantiates a list of clients for each given queue, e.g.:import mysite.sekrit from hirefire import procs class MyCustomProc(procs.ClientProc): name = 'worker' queues = ['default'] def client(self, queue): return mysite.sekrit.Client(queue) def quantity(self): return sum([client.count(queue) for client in self.clients])
See the implementation of the
RQProc
class for an example.-
client
(queue, *args, **kwargs)[source]¶ Returns a client instance for the given queue to be used in the
quantity
method.Needs to be implemented in a subclass.
-
quantity
(**kwargs)¶ Returns the aggregated number of tasks of the proc queues.
Needs to be implemented in a subclass.
kwargs
must be captured even when not used, to allow for future extensions.The only kwarg currently implemented is
cache
, which is a dictionary made available for cross-proc caching. It is empty when the first proc is processed.
-
Issues & Feedback¶
For bug reports, feature requests and general feedback, please use the Github issue tracker.
Thanks¶
Many thanks to the folks at Hirefire for building a great tool for the Heroku ecosystem.
Authors¶
- Emmanuel Leblond
- Jannis Leidel
- Marc Tamlyn
- Ryan Hiebert
- Ryan West
- Shravan Reddy
Changes¶
0.7 (2018-12-10)¶
- Make ProcSerializer inherit from object (#47)
0.6 (2018-10-10)¶
- Use concurrent futures to reduce blocking IO (#37)
- Add a test suite (#41)
- Fix RabbitMQ connection TimeoutError (#42). Acquire and dispose broker connection per request
0.5 (2017-01-20)¶
- Add
simple_queues
feature to Celery Proc, to enable optionally skipping one inspect call. (#23) - Make the default quantity 0 by the recommendation of the HireFire team.
0.4 (2016-06-04)¶
- Consider all Celery tasks including the ones in the active, reserved and scheduled queues. This fixes a long standing issue where tasks in those queues could have been dropped if HireFire were to scale down the workers. Many thanks to Ryan Hiebert for working on this.
- Removed django-pq backend since the library is unmaintained.
0.3 (2015-05-05)¶
- Added Flask blueprint.
- Fixed Celery queue length measurement for AMQP backends.
0.2.2 (2014-11-27)¶
- Fixed a regression in 0.2.1 fix. Thanks to Ryan West.
0.2.1 (2014-05-27)¶
- Fix the RQ Proc implementation to take the number of task into account that are currently being processed by the workers to prevent accidental shutdown mid-processing. Thanks to Jason Lantz for the report and initial patch.
0.2 (2014-04-20)¶
- Got rid of d2to1 dependency.
- Added django-pq backend.
- Ported to Python 3.
- Added Tornado contrib handlers.