Hashing Extermly Large Datasets: Coffee Chat with Frederick Kautz
Meet Frederick Kautz and learn about Terrapin
Terrapin is an open-source hashing algorithm developed by Frederick Kautz, designed for efficiently and securely hashing large datasets, especially in the petabyte range. It uses parallel processing and recursive hashing to overcome the limitations of traditional methods.
Key Features:
Parallel Processing: Divides datasets into 2 MB chunks for simultaneous processing, speeding up the hashing of vast data.
Recursive Hashing: Builds a hierarchical structure of hashes, allowing efficient validation of specific data portions while ensuring overall data integrity.
Efficient Validation: Enables quick validation of data slices without rehashing the entire dataset, reducing computational overhead.
Security: Uses SHA-256 GitOID to enhance data integrity and detect tampering, with changes in data affecting the final hash.
Scalability: Optimized for distributed computing, scaling performance with available processors or machines.
Find more about Terrapin Here: https://hackmd.io/rXTXMhjwTgGY3Yn4qEX96A#Terrapin-Specification
Frederick Kautz is a technical leader with deep expertise in infrastructure, Zero Trust, networking, and storage. He excels in guiding organizations through Cloud Native and Zero Trust transformations, ensuring compliance with standards like HIPAA and HITRUST. Known for his hands-on leadership, Frederick has helped service providers build robust infrastructure on Kubernetes, addressing gaps with solutions like Network Service Mesh.
Key accomplishments include:
Defining Cloud-Native Network Functions (CNF): Frederick created the first CNF definition, leading to the CNCF Telecom User Group and CNF Working Group.
Co-Founding Network Service Mesh: He developed this L2/L3 service mesh to simplify network configurations based on cloud-native principles.
Early Contributions to Docker and Red Hat: Frederick had early commit access to Docker and developed the Red Hat Container Storage Engine.
Industry Thought Leadership: He has delivered over 30 presentations on networking, cloud-native technologies, AI, and security at major industry conferences.
Frederick's work has significantly influenced the evolution of cloud-native networking, making him a key figure in the tech industry.
This event series is co-hosted by OpenFuture Forum and iAndMy.ai.
OpenFuture Forum: A community of leaders, investors, founders, and engineers based in the Bay Area, CA, dedicated to advancing innovation and collaboration in the tech industry.
iAndMy.ai: Community of Deep-tech Researchers, Builders,
VCs, and Industry experts