FreakU
/
takahe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add Stator tuning notes

This commit is contained in:
Andrew Godwin 2023-11-13 10:52:22 -07:00
parent 308dd033e1
commit e49bfc4775
2 changed files with 26 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
docs/releases/0.10.rst
View File

@ -37,10 +37,10 @@ Minor changes also include:
* Python 3.11 is now the minimum version required; this will not affect you at
all if you run Takahē via our docker image, as is recommended.
A remote post pruning system, to shrink the database of old data that was no
longer needed, was in the development version but has been removed in this
release due to the extra database load it caused. An alternative approach to
this will hopefully land in a future release.
An automatic remote post pruning system, to shrink the database of old data
that was no longer needed, was in the development version but has been switched
to a set of manual commands as of 0.10.1 - you can read more below or in
:doc:`/tuning`.
If you'd like to help with code, design, or other areas, see
:doc:`/contributing` to see how to get in touch.
@ -62,6 +62,11 @@ This is a bugfix and small feature addition release:
pruning (deletion of old content) of Posts and Identities respectively.
You can read more about them in :doc:`/tuning`.
* Stator's default concurrency levels have been significantly reduced as it's
now way more efficient at using individual database connections, but as a
result it places way more load on them. You can read more about tuning this
in :doc:`/tuning`.
Upgrade Notes
-------------
@ -79,3 +84,13 @@ We recommend:
* Temporarily stopping all instances of the webserver and Stator
* Applying the migration (should be less than a few minutes on most installs)
* Restarting the instances of webserver and Stator
Stator
~~~~~~
Stator's new internal architecture allocates a worker thread and a database
connection up to its concurrency value; this means it is a _lot_ more efficient
for a given "concurrency" number than the old system and also uses a lot more
database connections. We recommend you reduce your configuration values for
these by 5-10x; if you didn't set them manually, then don't worry, we've
reduced the default values by a similar amount.

11
docs/tuning.rst
View File

@ -56,10 +56,13 @@ Stator (Task Processing)
Takahē's background task processing system is called Stator, and it uses
asynchronous Python to pack loads of tasks at once time into a single process.
By default, it will try to run up to 100 tasks at once, with a maximum of 40
from any single model (FanOut will usually be the one it's doing most of).
You can tweak these with the ``TAKAHE_STATOR_CONCURRENCY`` and
``TAKAHE_STATOR_CONCURRENCY_PER_MODEL`` environment variables.
By default, it will try to run up to 20 tasks at once, with a maximum of 4 from
any single model (FanOut will usually be the one it's doing most of). You can
tweak these with the ``TAKAHE_STATOR_CONCURRENCY`` and
``TAKAHE_STATOR_CONCURRENCY_PER_MODEL`` environment variables; for every extra
element of concurrency you add, however, it will use an additional database
connection in a new worker thread. Be wary of hitting your database's
connection limits.
The only real limits Stator can hit are CPU and memory usage; if you see your
Stator (worker) containers not using anywhere near all of their CPU or memory,