Spy (SteelPython)

SteelPython ('spy')

A fast native Python 3.12 for Divooka.

SteelPython is a custom Python 3.12 implementation by Methodox™ Technologies, Inc. It is built on top of C# and integrates natively with Divooka.

SteelPython enables smooth frictionless C#/Python interoperation in the Divooka ecosystem. This page provides standalone download of SteelPython that doesn't depend on a Divooka Compute distribution.

Overview

This implementation is as fast as CPython on most tasks and slightly slower for others. For more on this, see Difference from CPython. An official benchmark is available in Divooka Mandelbrot Benchmark. Another major limitation is SteelPython currently consumes more memory than CPython on the same task, according to Mandelbrot Benchmark.

The goal of SteelPython are twofold:

1. Smooth, native transition to Divooka for existing Python users
2. Performance and feature parity with CPython, ensuring frictionless transition

If possible, we intend to beat CPython in runtime performance as well.

Motivation

Native integration with Python hits major limitations on hosting model with PythonNet, which interops with CPython - further complicated by the existence of GIL (Global Interpreter Lock). Distribution complexity is also a major concern even with embedded python. A smooth, native experience demands for more integrated Python embedding, as required by Foreign Language Interface (FLI) for Divooka.

Also importantly, SteelPython enables seamless invokation of Divooka and Divooka ecosystem functions, allowing users to make use of the vast DSLs offered to Divooka.

Benchmarks

Steady state (CPU s)	CPython 3.12	SteelPython	Op. Tier	vs CPython
richards	0.55	0.69	0.42	1.30x faster
deltablue	0.41	0.50	0.31	1.30x faster
raytrace	0.33	0.38	0.22	1.50x faster
calls	0.52	0.45	0.36	1.44x faster
nbody	1.28	1.06	0.61	2.1x faster

Utilities

Besides spy CLI utility which features most of python CLI command line arguments, we ship Misp and Spy Notebook, two standalone GUI applications. Details of which are available in their respective documentation:

• Misp: A lightweight GUI IDE for SteelPython.
• Spy Notebook: A notebook style interactive scripting environement for SteelPython, compatibile with .ipynb file format

Extension Development

Besides regular C Extensions, SteelPython naturally supports C# extensions. See Extension Development page for more details.

Changelog

Changelog can also be found in the CHANGELOG.md file in the installation folder.

v0.3.1

1. sqlite3: a statement that writes and answers with rows no longer stays open when its rows are not read. Closing a connection also releases the file whatever its cursors were doing.
2. Text files left open keep everything written to them when the program exits.

v0.3.0

• Performance optimization with optimising tiers.
• Imports search folders through cached listings and read again when a folder changes.
• Fix some issues.
• Reduce venv footprint.
• Reduce memory consumption.
• Change hosting behaviors: The NuGet package no longer changes the host application's garbage-collector settings by default
• Fix GUI issues with Misp.
• Pin native libraries for Windows builds: OpenSSL 3.5.9 and libffi 3.4.4 as CPython's and SQLite 3.54.1 from the SQLite project.
• Improve multithreading.

v0.2.0

Since 0.1: buffers over 2 GiB (pip's cache of PyTorch's CUDA wheels), module names matched case-sensitively on Windows and macOS, C# extensions through `clr.AddReference`, Methodox Technologies, Inc. as the copyright holder, Misp and Spy Notebook distributed next to `spy` in the .NET distributions, Misp's highlighting of files opened from disk.

• .NET without `import clr`: namespaces import directly (`from System.Collections.Generic import List`), the .NET runtime's own assemblies are found by the namespaces they declare and loaded when first used, and a `.dll` on `sys.path` imports like an extension module (`import Acme.Pricing`, or a C# extension's `[PythonExport(Module = ...)]` name). 
• `clr` is still available for `AddReference`, its helpers and pythonnet code.
• The per-user site directory, in a location of SteelPython's own (`%APPDATA%\SteelPython\Python312\site-packages`, `~/.local/steelpython/...`): `pip install` on an all-users installation falls back to a user installation as CPython's does, and `--user`, `-s` and `PYTHONNOUSERSITE` work.
• Fix `ctypes` looking up for Windows DLL exports by ordinal; pyglet's XInput backend, and so `import arcade` works.
• `ctypes.cast(id(obj), ctypes.py_object).value` returns `obj` instead of crashing `spy`.
• Windows: make threads that run Python code start without the COM apartment as CPython's threads, so `CoInitializeEx(COINIT_APARTMENTTHREADED)` succeeds (pyglet no longer warns `COM mode set by another library in a different mode`); WinForms' clipboard, drag and drop and dialogs work from Python; an embedding host's threads keep their apartment.

v0.1.0-preview.1

First preview of SteelPython, a Python 3.12 implementation in C# for .NET 10.

• 96.3% of the CPython 3.12 test-suite passes on SteelPython.
• 111 of 111 popular libraries give the same output as on CPython, from unmodified PyPI wheels: numpy, pandas, scipy, matplotlib, torch, numba, polars, duckdb, opencv, pillow, lxml, pydantic, cryptography, orjson, pyarrow, scikit-learn, sqlalchemy, aiohttp, gevent, mypy, black, coverage, sphinx, ipykernel and more; Flask, FastAPI/uvicorn and Django applications run unchanged. Extension modules load through a CPython C-API emulation layer.
• Code objects describe CPython 3.12's bytecode for their source (`dis`, `co_code`, `co_positions()`, `f_lasti`/`tb_lasti`), built on first use and never executed; cloudpickle rebuilds functions from it.
• `pip`, `venv`, `ensurepip`, `ssl` (system OpenSSL 3), `sqlite3` (system SQLite), `ctypes` (system libffi), `asyncio`, `subprocess`, `multiprocessing` (spawn), signals, `sys.settrace`/`sys.monitoring`/`pdb`, greenlet.
• An interactive `spy` with a C# `readline` (editing, history, completion) whose sessions match CPython 3.12.
• Embedding and .NET interop: `SteelPython.Hosting`, the `[PythonExport]`/`[PythonInterface]`/`[PythonSubclassable]` source generator and the `clr` module; asyncio ⇄ `Task` bridge, Python classes deriving from .NET types, exact decimal/datetime/uuid/path/enum/buffer conversions, a NuGet package with the standard library.
• Misp, the IDE (editor, debugger, venvs, pip), and Spy Notebook (Pure Notebook's format with a SteelPython kernel), Native AOT GUI applications.

Resources

• Documentation: Methodox Wiki