Case study
What was at stake
Credit risk at MercadoLibre is decided with models, batch and online, and the output of those models is not a dashboard. It is a set of persistence tables that other teams query so the business can make decisions about actual customers: who gets credit, and on what terms.
I worked in the Credits area, on the MLOps team. Our job was not to build the models. It was to build the internal tooling that let data scientists deploy and operate them reliably, repeatedly, and without needing us in the loop for every run.
That makes the data scientists our users, not our requesters. The distinction matters. A platform team that takes tickets ends up automating whatever was asked for, while the actual friction stays where it was. Most of the design work here started from watching how models were being deployed and operated in practice, and from the parts of that process people had quietly accepted as unavoidable.
The constraint
Two things shaped the work.
The first is that the models were driven by cron. A scheduled job runs at the time written in a crontab, which has nothing to do with the moment its input data actually lands. Schedule it too early and it runs against yesterday’s data, or fails. Schedule it late enough to be safe and you burn the gap between when the data was ready and when anyone acted on it. Multiply that across a large portfolio and the waste is structural, not incidental.
The second is scale. This was not a handful of models to babysit. The portfolio grew past 200, each model with its own parameterisation, its own inputs and its own downstream consumers. Anything that needed per-model manual handling was not going to hold.
What I built
Three pieces, in Go and Python, on top of the existing internal stack.
A model store
A single place where models and their parameterisations live, covering the full portfolio, which grew past 200 models. Before you can automate running a model you need an unambiguous answer to which model, at which version, with which parameters. The store is what makes everything after it addressable.
An executor
The replacement for cron. Instead of firing on a clock, the executor runs a model on demand, or as soon as its datasets are ready, which it determines with a simple query against the model’s respective buckets in GCP.
The shift is small to describe and large in effect. The trigger moves from the time it is to the state the data is in, so work happens when there is work to do. Nothing burns compute waiting for data that has not arrived, or sitting idle after it has. Against the cron jobs it replaced, the executor cut cost by 75%.
A persistor
The executor produces scores, and something has to put them where they are read. The persistor works on the same principle. It writes model outputs into the persistence tables on demand, or as soon as a scored dataset exists, rather than on a schedule of its own.
These are Credit Risk models reaching 50M+ users, so the delay between a score existing and a score being queryable is not an abstract number. It is the window in which the Credit Offers ecosystem is working from stale data. Taking persistence off its own schedule cut those times by 70%.
Those tables are the handoff point. They are what other teams consume, and what the business ultimately decides on.
What came of it
Deploying a model stopped meaning “find a slot in the schedule and hope the inputs are there”. The portfolio runs when its data is ready, the scores land without a second scheduled hop, and the whole portfolio has one place to be described instead of a spread of bespoke arrangements.
The lesson I took from it is that a large share of infrastructure spend is not a performance problem at all. It is a coordination problem that shows up on the bill. Nothing about those models got faster. We only stopped guessing, in a crontab, about when their data would show up.