Generative Benchmark Creation for Table Union Search

by   Koyena Pal, et al.

Data management has traditionally relied on synthetic data generators to generate structured benchmarks, like the TPC suite, where we can control important parameters like data size and its distribution precisely. These benchmarks were central to the success and adoption of database management systems. But more and more, data management problems are of a semantic nature. An important example is finding tables that can be unioned. While any two tables with the same cardinality can be unioned, table union search is the problem of finding tables whose union is semantically coherent. Semantic problems cannot be benchmarked using synthetic data. Our current methods for creating benchmarks involve the manual curation and labeling of real data. These methods are not robust or scalable and perhaps more importantly, it is not clear how robust the created benchmarks are. We propose to use generative AI models to create structured data benchmarks for table union search. We present a novel method for using generative models to create tables with specified properties. Using this method, we create a new benchmark containing pairs of tables that are both unionable and non-unionable but related. We thoroughly evaluate recent existing table union search methods over existing benchmarks and our new benchmark. We also present and evaluate a new table search methods based on recent large language models over all benchmarks. We show that the new benchmark is more challenging for all methods than hand-curated benchmarks, specifically, the top-performing method achieves a Mean Average Precision of around 60 existing manually created benchmarks. We examine why this is the case and show that the new benchmark permits more detailed analysis of methods, including a study of both false positives and false negatives that were not possible with existing benchmarks.


page 5

page 10

page 11


SANTOS: Relationship-based Semantic Table Union Search

Existing techniques for unionable table search define unionability using...

LST-Bench: Benchmarking Log-Structured Tables in the Cloud

Log-Structured Tables (LSTs), also commonly referred to as table formats...

Pylon: Semantic Table Union Search in Data Lakes

The large size and fast growth of data repositories, such as data lakes,...

TabSim: A Siamese Neural Network for Accurate Estimation of Table Similarity

Tables are a popular and efficient means of presenting structured inform...

Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning

Dataset discovery from data lakes is essential in many real application ...

Learning to Detect Touches on Cluttered Tables

We present a novel self-contained camera-projector tabletop system with ...

rtables – A Framework For Creating Complex Structured Reporting Tables Via Multi-Level Faceted Computations

Tables form a central component in both exploratory data analysis and fo...

Please sign up or login with your details

Forgot password? Click here to reset