Suggestions
Diptanu Gon Choudhury
Founder and CEO @ Tensorlake
Diptanu Gon Choudhury is a prominent figure in the field of artificial intelligence and distributed systems, currently serving as the founder of Tensorlake, a company focused on developing innovative solutions for handling unstructured data. He has a rich background in technology, having held significant roles at major companies such as Facebook, Netflix, and HashiCorp.
Professional Background
- Founder of Tensorlake: Choudhury leads Tensorlake's efforts to create Indexify, an open-source, scalable structured extraction engine designed to process unstructured data for AI applications. This project aims to facilitate near-real-time knowledge bases for AI-driven workflows and query engines.12
- Previous Experience:
- Facebook: He was the tech lead for the FB Learner machine learning platform and developed a real-time speech inference engine.23
- Netflix: Choudhury invented the Titan/Titus cluster scheduler, which is pivotal for managing cloud services.2
- HashiCorp: He created the Nomad cluster scheduler, further showcasing his expertise in distributed systems.24
Contributions and Philosophy
Choudhury emphasizes the importance of embracing challenges and failures as integral parts of technological innovation. He believes that understanding the complexities of unstructured data is crucial for leveraging its potential in analytics and AI applications. His work aims to democratize access to advanced data processing tools by promoting open-source solutions, fostering collaboration within the tech community.14
Vision for AI
In his discussions, Choudhury highlights the need for tailored solutions in AI deployment, recognizing that a one-size-fits-all approach is insufficient. He advocates for a deeper understanding of business use cases to effectively integrate AI technologies into various industries. His insights reflect a commitment to advancing the field of AI while ensuring that developers have the necessary tools and resources to innovate.45
Choudhury's LinkedIn profile can be found under the username diptanu, where he shares updates about his work and insights into the evolving landscape of AI technology.5
Highlights
Apple does very little these days to justify upgrading to new iPhone hardware.
I wrote an executor by wrapping ProcessPoolExecutor, which loads some serialized code(from users), runs the functions and returns the objects back. Pickle refuses to work since the workers which produce the objects are on different processes. The only way to get around it is using cloud pickle or using a wrapper like Loky.
But my instinct tells me they are fragile as well, so instead I am serializing the output in a binary encoding(CBOR) format and moving them across process boundaries. This allows us to not pickle complex objects, and second allows us to deserialize them before invoking functions in various runtimes.
