Engineering Blog

Engineering

Learn from our challenges and triumphs as our talented engineering team offers insights for discussion and sharing.

Utilizing MapReduce Combiners and HyperLogLog++ to process millions of queries over datasets with billions of records

The fundamental challenge being solved is aggregating and counting the unique values in a large multiset. Here is a matrix visualization of one dataset:  The goal is to be able to retrieve counts of unique IDs with different constraints. For example (using SQL-like syntax to illustrate), computing COUNT (DISTINCT ID) WHERE (Field 1 = 0) ...

Friday Thoughts: strive for phoenixes, not snowflakes

tl;dr Avoid surprises, restart your workloads often to ensure your app starts as expected. Server outages can produce some interesting surprises. As we continue to strive for dependable interfaces and services between our squads, I am reminded of the snowflakes and phoenix analogy, a re-branding of pets vs. cattle. Snowflake: unique, fragile, unpredictable Phoenix: rise from nothing at any time, ...

Hackweek XLIV

For many LiveRampers, hackweek is one of their favorite parts of working here. It’s a time when you can work on whatever you think will bring the most value to the company, whether it is fixing a process that’s been making you miserable, cleaning up some old code that’s dragging down the system, or ...