setting cpu affinity doesn't work on py3 #3973
Comments
Probably the first two questions to ask are "did you install this on pip3 rather than pip, which is usually the py2 version" and "how did you install synapse?" |
synapse is installed in the virtualenv with |
This is something that should work on Python 3, so I'll take a look. Thanks for the report, @Valodim . |
FWIW, it looks like the
|
That's rather unfortunate. However, looking at what it actually wraps, I believe a CFFI-using replacement which covers our purposes (Linux only) would be almost trivial to write, so I'll take a stab at that next week. |
Do we even need this on Py3? The reason it was added in the first place was due to https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/, which i thought might have been addressed in Python 3.5+? |
No -- it's up to the operating system to decide where to schedule
something, not the application. Linux schedulers sometimes try and avoid
breaking caches and keep things on the same CPU, but that isn't always the
case.
Doing a bit of research, it appears psutil has CPU affinity setting, and
since I believe we already depend on that library, we can use that instead.
https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity
…On Fri, 5 Oct 2018, 08:32 Matthew Hodgson, ***@***.***> wrote:
Do we even need this on Py3? The reason it was added in the first place
was due to
https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/,
which i thought might have been addressed in Python 3.5+?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#3973 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADJ2XJbzxWxCSWe5JPaHhOApiGEKJVN6ks5uhoyLgaJpZM4W9dwR>
.
|
The problem is that it interacts badly with the implementation of the GIL, since the OS may schedule the threads onto multiple cores, but then the python threads fight/thrash over the GIL locks. It looks like the GIL may have changed to make it thrash less in python 3.2 http://www.dabeaz.com/python/UnderstandingGIL.pdf ? If psutil has an implementation, lets just use that. Once we get matrix.org on py3 we should definitely try removing the CPU affinity options to see if the thrashing has indeed been fixed or not, since pinning to a single core does have some performance penalties. |
Fixed in #4027 |
the affinity package is installed, I ran pip in that virtualenv to make sure, and checked it's there in site-packages. not sure why synapse can't find the package.
perhaps the setting no longer makes sense with py3? anyways this prevents synapse from starting for me. commenting out the setting works fine, however.
This is on 0.33.5.1
The text was updated successfully, but these errors were encountered: