Banking integration has a shape that surprises people coming from product engineering. The data does not arrive as a tidy stream of JSON over HTTP. It arrives as fixed-width files on an SFTP drop at 02:00, as a message on a queue whose schema was agreed in 2011, as a correction file that supersedes yesterday's, and occasionally as a spreadsheet from an operations team. Then it has to be processed with correctness guarantees appropriate to money.
At SunTec I built Kafka, Apache NiFi and Ignite Cache streaming pipelines for banking billing and payments on the Xelerate platform, along with the billing functionality on top — payment follow-up and bill generation. The architectural lesson I took from it is that those three technologies are frequently presented as alternatives and are in fact answers to three unrelated questions.
NiFi is an ingest problem, not a streaming engine
The instinct when you already run Kafka is to write producers and skip the extra moving part. That works right up until the fourth source system, and then you notice you have written four bespoke jobs that each independently handle character encoding, a header/trailer record count that must reconcile, a file that arrived twice, a file that arrived empty, and a file whose schema changed without notice.
NiFi exists for precisely that layer — flow-based ingest with backpressure, per-flowfile attributes and, crucially for this domain, data provenance. Provenance is the feature that justified it in banking: for any record, you can answer which file it arrived in, when, what transformations it passed through, and where it went. Reconstructing that from application logs across four bespoke producers is a project. Here it is a property of the tool.
The judgement call is where to stop. NiFi will happily do enrichment and business logic, and every processor you add there is logic that lives in a flow definition instead of in code with tests and code review. We drew the line at: decode, validate structurally, deduplicate, route. Anything that needs a business rule happens downstream, in a service.
Kafka is the record of what happened
Once data is decoded and validated it goes onto Kafka, and from there on the pipeline is event-driven. Two properties do the work.
Ordering where it matters, parallelism everywhere else. Keying by account gives strict ordering within an account — a payment reversal can never be processed before the payment it reverses — while throughput scales with partition count, since Kafka guarantees order within a partition and not across a topic. Every ordering bug I have seen in this kind of pipeline traces back to a key chosen for even distribution rather than for the ordering the domain actually requires.
Redelivery is normal, so consumers must be idempotent. The delivery-semantics documentation is worth reading in full before designing anything financial on top of it: at-least-once is the pragmatic default, and a consumer rebalance can hand you a batch you have already processed. In billing that is not a rounding error, it is a duplicate invoice. Bill generation has to be idempotent on a natural key — account plus cycle plus rule version — and not on “have I seen this offset”.
One Kafka feature is underused in this domain: log compaction. Reference data — rate cards, client hierarchies, product catalogues — is naturally a keyed changelog rather than an event stream, and a compacted topic retains the latest value per key indefinitely. That means a new consumer can rebuild its full view of reference state from the topic alone, with no back-fill job and no call to the system of record.
Ignite is about the per-event lookup, not about being fast
The distributed cache is the piece most often adopted for the wrong reason. “We added a cache to make it faster” usually means someone measured a slow query and put a cache in front of it, which works and also creates a second source of truth nobody has thought about invalidating.
The real driver is structural. Every billing event needs the same handful of reference lookups: which rate applies, where this account sits in the client hierarchy, which product bundle is in effect. Done against the system of record, that is several round trips per event, and the event rate is set by the bank's volume rather than by your capacity planning. Held in an Ignite cluster, it is an in-memory lookup that scales by adding nodes, and a near cache keeps the hottest subset local to the process doing the work.
Which brings the two halves together: the reference data in the cache is populated from the compacted Kafka topic, so cache invalidation is not a separate mechanism that can drift. A rate change is published once; every cache node converges on it by consuming the same changelog every other consumer reads.
What none of this fixes
The pipeline is the easy half. The hard half is that the upstream file will one day contain a trailer count that does not match its body, and you have to decide — before it happens, in writing, with the client — whether that file is rejected wholesale or processed partially. Streaming architecture gives you somewhere clean to implement that decision. It does not make it for you.
The other thing worth saying: none of it is safely changeable without a test cycle you can actually afford to run. On the same platform, rebuilding the automated test framework took the regression cycle from 5 days to 8 hours, and that number is what made the pipeline work above shippable at any reasonable pace. A five-day feedback loop does not just slow delivery down, it changes what people are willing to attempt.