Every engine is a list of debts. Here is ours.
Every few years, a new planning engine announces that it has escaped the old trade-offs. It never has. The debts were only moved out of sight, usually onto the user, usually denominated in waiting. So here is SFMT, the engine beneath Novi, the way one engineer would explain it to another. Including the bills.
I've been around this industry long enough to have heard the announcement several times: a new engine, a new paradigm, the old trade-offs finally transcended. The trade-offs are all still here. They always are.
Engineering honesty
An architecture is not a triumph. It is a list of debts. Memory or latency. Elasticity or residency. Throughput or interactivity. Every engine ever built took on some of these debts to be free of others, and the only real difference between engines is which creditors they chose. Mature engineering names its creditors out loud. Marketing, when it gets to the architecture before the engineers do, tends to claim the debts were abolished.
They never are. They are only moved to where you can't see them, usually onto the user, usually denominated in waiting.
So here is SFMT (Simple Financial Modelling Tool, the deliberately modest name for the engine we have built) stated plainly: what it is, what it refuses to pay, and what it pays instead.
One paradigm: latency
Every architectural decision in SFMT is subordinated to a single budget: the answer is there when you look. Not "fast." Not "scalable on demand." Already there. A person changing a driver in a plan has a patience budget of roughly a tenth of a second before thinking turns into waiting. Everything below this paragraph exists to stay inside that budget.
The principle that follows is zero distance: no gap between the question and the data. No fetch, no queue, no network hop standing between a keystroke and its consequence.
The topology
- SFMT is a computation grid. The model is partitioned across coordinated nodes, and calculation happens where the data lives, never the other way around. Compute travels to data; data never commutes.
- It is a data grid for multidimensional data. Classic data grids keep compute and data together but hold flat entries. SFMT's grid holds dimensional structures natively. Cubes and hierarchies are the unit of distribution, not an afterthought serialized on top.
- The analytical machinery is in-process. There is no database server on the other side of a connection. The engine computes inside the same process where the data sits: no round trip, no serialization boundary, no distance at all.
The machinery
- Compiled, not interpreted. Formulas are compiled into executable form, the way real software is built, not read line by line at the moment you can least afford it.
- Two purpose-built machines. A calculation machine, the C-class, evaluates cells the instant their inputs change. An aggregation machine, the A-class, rolls values up through every hierarchy so totals are always current, never stitched together on request.
- Vectorized, columnar execution. Data is laid out in columns and processed thousands of values per CPU instruction. Aggregation runs at hardware speed because the layout was designed for the silicon, not the other way around.
- Append-only recalculation. Nothing is ever updated in place. Recalculation derives new results and appends them, which turns the classic weakness of columnar processing (point updates) into its greatest strength (bulk sequential writes). It also means history is a property of the physics: versions, scenario snapshots and audit trails are what naturally remains when nothing is overwritten. For finance, an engine that cannot help but keep an audit trail.
- RAM as cache, not as warehouse. The hot working set lives resident in memory; the long tail lives in memory-mapped columnar storage behind the operating system's page cache. The latency budget holds; the memory bill doesn't balloon, which matters in a year when AI demand has repriced server RAM by multiples.
One model, two shapes of data
Planning data comes in two shapes: multidimensional (cubes, drivers, hierarchies) and transactional (lists, ledgers, line items). Most architectures pick one and bolt the other on. In SFMT both are first-class citizens, and links physically join them inside the engine, not an import, not a sync job, not a connector between modules. One connected model, both shapes, no seams. This is not a brochure sentence; it is a structural property of the engine.
The debts we refused
You can read an engine's priorities in the debts it declined. SFMT refuses four:
- The fetch. Architectures that separate storage from compute pay a distance tax on every question. The data must travel before the answer can exist. We keep them together, so there is nothing to wait for.
- The queue. Where calculation is expensive and contended, engines grow schedulers: prioritizing whose recalculation matters more, postponing work nobody is watching. A priority queue is an architecture's confession that calculations wait. SFMT has nothing to prioritize, because nothing contends.
- The ramp. Elastic compute arrives in chunks, seconds to minutes after the demand spike that summoned it, magnificent for batch jobs, useless for a keystroke. Capacity that arrives after the question helps the next batch, not you. We hold capacity resident instead.
- The ceiling. An engine built around waiting eventually needs a timeout, a designed-for failure mode where the calculation simply gives up. SFMT's interaction model has no such ceiling, because no calculation is ever far enough away to need one.
Elasticity is how you manage queues. Residency is how you eliminate them.
Neither choice is wrong. They are different creditors. But you can read an architecture's true priorities in which debt it took on, and you can read its marketing's honesty in whether it admits to having any.
The receipts
Claims about engines should come with conditions attached, so here are ours. Our standard stress test is a fully dense cube, every cell populated, no sparsity to hide behind:
- Test cube
- 100,000,000 cells
- Fill rate
- 100%, fully dense, no sparsity to hide behind
- Composition
- ~1/3 input cells · ~1/3 C-class (calculation) · ~1/3 A-class (aggregation)
- Machine
- single node · 32 GB RAM · Linux
- Full recalculation
- single-digit seconds, no incrementality, everything recomputed
That is the worst case, a complete recomputation of everything, with zero shortcuts. The interactive case, where a driver changes and only its consequences recompute, stays inside the tenth-of-a-second budget that defines the whole design.
Our list
And here is what we pay, plainly. We pay for memory, and we re-engineered the engine this year precisely because the market repriced that debt. We size deliberately: every client runs on their own machine and their own grid, which costs us convenience and buys them isolation by physics rather than by permission model. We run machinery at full effort so that nothing on the surface ever appears to move. And for workloads where economics outweigh the last millisecond of tail latency, we offer a hybrid path with its own, smaller debts, stated just as plainly to the customers who choose it.
We chose to pay in hardware and engineering so that the people doing the planning never pay in waiting.
When someone tells you their engine escaped the trade-offs, ask to see the list.
That's the whole philosophy, and it's why I distrust any architecture story without a debts section. Every engine has one. The only question is who's carrying it, and too often, it's the person in front of the model, paying it down in minutes of their life.
Feel the swan effect
The fastest way to understand SFMT is to watch Novi move: a number that updates the instant you change a driver, a model that opens without a spinner. The machinery is ours to build and keep. The stillness is yours to use.