Engineering Blog

Engineering

Learn from our challenges and triumphs as our talented engineering team offers insights for discussion and sharing.

[Opinion] Report Guidelines

Reports are our chief vehicle for influencing organizations, especially as they grow to a larger number of people. Further, they are also how we assemble our collective knowledge about the problems we work on and the solutions to these problems. Because reports are so important, we’ve collected some guidelines about how to create amazing ...

Introducing MockRDD for testing PySpark code

Summary

The LiveRamp Identity Data Science team is excited to share some of our PySpark testing infrastructure in the new open source library mockrdd. This contains the class MockRDD, which mirrors the behavior of PySpark RDD with several additions: Extensive sanity checks to identify invalid inputs More meaningful error messages for debugging issues ...

[Opinion] Guidelines for Effective Meetings

Meetings can be an effective way to discuss projects and reach consensus on decisions. But I believe when ran improperly, they can also be a huge waste of our valuable time. Hence, we should have some guidelines on how to create and run effective meetings.

Meetings are about discussions

The whole reason ...

Utilizing MapReduce Combiners and HyperLogLog++ to process millions of queries over datasets with billions of records

The fundamental challenge being solved is aggregating and counting the unique values in a large multiset. Here is a matrix visualization of one dataset:  The goal is to be able to retrieve counts of unique IDs with different constraints. For example (using SQL-like syntax to illustrate), computing COUNT (DISTINCT ID) WHERE (Field 1 = 0) ...