scaling posts

As I was reading through the Petar Maymounkov and David Mazières paper "Kademlia: A Peer-to-peer Information System Based on the XOR Metric", I was interested in the application of this XOR metric on work I had previously done on clustering embeddings. When implementing the Infer API for Clarity Hub, we focused on using the distance between vectors as a way to cluster utterances together into a topic. We did this via Euclidean distance between vectors, which we had to compare against all other vectors within the space.

Every system has its limits. When designing and architecting systems, sometimes we as engineers like to assume things can scale infinitely. Maybe we can use larger machines, or maybe we can deploy more instances of a service. But even trying to scale systems isn't magically infinite. It's a smart move to understand the limitations of what we are building upfront so we can make better technology choices, and squeeze the performance out of our existing systems.

Back when Meteor was in vogue, they popularized the concept of “Data on the wire”. This was the idea that the UI would be updated with changes from the dataset it is subscribed to.