Python tooling 2024-11-11 Python tooling currently is a mess. Other modern programming languages, like Go or Rust, have a unified standard tooling to cover all bases. Meanwhile Python suffers from tool overload. For every aspect of working with a project there's at least a few different tools you can choose from. While development tools are greatly improving, it seems like project management has gotten a little more complicated. Good overview is presented here: https://alpopkes.com/posts/python/packaging_tools/ Another interesting insight is presented here: https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/ Here's a "very short" list of development tools for Python: - pip - managing python packages - venv - isolated virtual environments - pyenv - managing python versions - setuptools - building packages - twine - publishing packages - poetry - complete project management - hatch - project management, again - rye - different project management, again - maturin - project management for Rust packages - pyinstaller - freezing into binaries - pytest - testing framework - black - simple linter - isort - import linter - flake8 - extensible linter - ruff - linter and code formatter (+LSP, very cool!) - autopep8, pyflakes, pylint, ... - more linters - mypy - type checker - jedi - autocompletion, static analysis - pyright - static type checker (+LSP) For some tools there might be more alternatives available too... In short - tool overload. Let's first cover dependency management, then project management, and lastly development tools like linters and static checkers. I'll mention my personal choices for each. The most basic tools are pip (a package manager) and venv (virtual environment to contain and separate packages). They are very useful and probably enough for simple development purposes, but not quite enough for building and distributing packages. You can use them along with 'requirements.txt' file for project dependency management. Now for some people, getting the Python version right is also an issue. I don't see why you wouldn't just use the lastest version offered by your distro, but I guess some poeople are still stuck with inferior operating systems or are constrained by the deployment environment. This is where pyenv or conda comes in. I haven't used either much myself, but I mention them here for completeness. There used to be a common and simple tool for package building - setuptools. You would set your metadata and configure the build in a 'setup.py' file, then you could build the project by executing that file. This offers a lot of flexibility, since a regular python file realistically can contain any code you would need. The downside is that it's difficult to parse for external tools. Then 'pyproject.toml' configuration got introduced with PEP-518: https://peps.python.org/pep-0518/ Many build tools make use of the 'pyproject.toml', including setuptools, which is not as dominant anymore. Recently my favorite is poetry. It's a complete project management tool, which by now plays very well with 'pyproject.toml', and takes care of pretty much everything: dependency management, virtual environments, package builds, package distribution, etc. I find it very easy and pleasant to use. It doesn't fuck up my project structure and nicely integrates with other tools, for example pytest. Another project management tool, rye, embraces a slightly different vision. It also covers most bases, but it uses a specific set of specialised utilities to accomplish that. Their choice of tooling is opinionated, but very solid and also quite interesting. For example: they use uv instead of pip. I believe rye, uv, ruff were all written in Rust by the same original author. On their "philosophy" page you can read how exactly rye is different from other project management tools and why: https://rye.astral.sh/philosophy/ PyInstaller is useful when you want to distribute your program for a platform that might not have the appropriate python version already available in the PATH. You freeze your project along with dependencies into a single binary file and run it on your target. It's pretty cool for some use cases. When it comes to development, Python doesn't have a strict style. However, especially when you're working with others, you would probably like to automatically enforce a uniform and clean style that doesn't make your eyes bleed. That's exactly what a linter does. There's more than a couple of those you can choose from for Python. Black is a pretty simple yet solid choice. Just for the purpose of sorting your imports correctly there is isort. Extensible and powerful flake8 is a popular linter too. It's very good, but not very fast. That's where ruff comes in, offering parity with all the three at faster execution time. Speaking of Python style, the classic style guide is PEP-8: https://peps.python.org/pep-0008/ Python is a dynamic language, which means that variables don't have strictly assigned types. Nonetheless, we can enforce type consistency through static analysis with a type checker such as mypy. Static typing makes it easier to catch some errors before reaching the faulty program state. Static analysis can extend beyond that to catch all sorts of mistakes and provide helpful hints about the correctess of the program. Good tool examples are pyright and jedi, which can also provide completions and help effectively browse the code through LSP integrations. You don't even have to provide any type hints yourself to make good use of a static analysis tool such as pyright. In case you want to get a refresher on Python type hints anyway, they were described in PEP-484: https://peps.python.org/pep-0484/ LSP is a protocol that allows for integrating tools with IDEs or editors such as Neovim. LSP tools help out a whole bunch during source code editing. I myself use aforementioned ruff and pyright that way. In the end you have to reach for multiple tools to get what cargo gives you by default for Rust. For now poetry+ruff+pyright feels comfy enough to me, but it was quite a journey to get there. I'm sure Python tooling is going to evolve further, hopefully for the better.