# Call a Python program that keeps running with PYPELINE()

> Start a Python script from the application, send it JSON messages over a socket and receive its answers, with the script restarted if it dies.

Framework: total5
Product: Total.js 5 (total5@0.0.19)
Language: en
Content-SHA256: c346fc22c80a08961b4ed4e5da21cc650cb0f3fac63fddec6f03504e6f5bcaf7
Snapshot: c0700e321579725c5cf4f5ace1ad7d2866d1b4f93c24a63c0b41b6b7998e976c
Sources: total5-installed@total5@0.0.19
Review: not-recorded

`PYPELINE(name, [options])` starts `pypelines/<name>.py` with `python3` and keeps a connection to it, so the application can use Python libraries (data, models, image processing) without starting a new process for every request. It returns an object with `send(data)` to send a message, `on('data', fn)` to receive the answers and `close()` to stop the script. A name that starts with `~` is the path of a file instead (`~/opt/job.py`), and a text that is code (it has line breaks or looks like Python) is run as an inline program with `python3 -c`.

- **The protocol.** The script is started as `python3 -u script.py <socket>`: the **last argument is the path of a Unix socket** that the application listens on, and the script connects to it. Every message, in both directions, is a frame: 4 bytes with the length (big endian) followed by the body, which is JSON by default.
- **Options:** `type` (`'json'` default, `'text'`, or anything else for raw `Buffer`s), `autorestart` (`false` to not restart the script when it ends; by default it is started again), `threads` and `socket: false` (no socket, only start the program).
- Messages sent before the script has connected wait and are delivered when it does.
- Events: `open` (the script connected), `data` (a message), `restart` and `close`.
- **The path of the socket has a limit.** The socket is a file in the `tmp` folder of the application and the system limits the whole path (about 104 characters on macOS and 108 on Linux). In a project that is deep in the disk the server fails with `listen EINVAL`, the script never connects and the messages wait forever. The path is the property `pypeline.socket`: set it right after `PYPELINE()` (the file is checked and the server started afterwards) to a short one, as the example does.
- The Python program is a **normal program that runs with the rights of the application**: it is only as safe as the code of the file, and Python 3 has to be installed (`python3` in the `PATH`).

```python pypelines/shout.py
import json
import socket
import struct
import sys

# The last argument is the socket that the application listens on
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(sys.argv[-1])

def read(count):
    data = b''
    while len(data) < count:
        chunk = sock.recv(count - len(data))
        if not chunk:
            raise SystemExit(0)
        data += chunk
    return data

while True:
    size = struct.unpack('>I', read(4))[0]
    message = json.loads(read(size).decode('utf-8'))
    answer = json.dumps({ 'id': message['id'], 'upper': message['text'].upper(), 'words': len(message['text'].split()) }).encode('utf-8')
    sock.sendall(struct.pack('>I', len(answer)) + answer)
```

```js controllers/py.js
exports.install = function() {
	ROUTE('POST /shout/ @json', shout);
};

const pypeline = PYPELINE('shout');

// A short path for the socket: the folder of a project can be too deep for a Unix socket
pypeline.socket = require('os').tmpdir() + '/forge-shout-' + process.pid + '.socket';
const pending = {};
let counter = 0;

// Every answer says which message it is for
pypeline.on('data', function(answer) {
	const callback = pending[answer.id];
	if (callback) {
		delete pending[answer.id];
		callback(answer);
	}
});

function shout($) {
	const id = ++counter;
	const timer = setTimeout(() => $.invalid('The script did not answer'), 5000);
	pending[id] = function(answer) {
		clearTimeout(timer);
		$.json(answer);
	};
	pypeline.send({ id: id, text: $.body.text });
}
```