Pathway

What is Pathway?

Pathway is a software framework designed to tackle a major industry problem: the lack of infrastructure that can automatically reason over data streams in real-time. Essentially, it lets software engineers build real-time data products without getting bogged down by constantly changing data inputs. Because Pathway is built with a reactive design, it handles data updates smoothly. This means the data products you create are not only easier to develop and maintain but also run really efficiently on its engine. It really simplifies the whole ML/AI project lifecycle, taking you from a simple prototype all the way to production. Plus, it offers flexible deployment options and can process updates at high speed from live data sources like cloud folders, databases, and APIs. Some of its key features include making cloud-native development much simpler, processing data quickly and in sync, offering flexible ways to deploy your applications, providing customizable code templates, and allowing you to easily interact with external APIs and models, including LLMs.

Who created Pathway?

Pathway was built by a team with serious AI chops, including folks who previously worked at top AI labs like Google Brain. We’ve got ex-Googlers, some of the best competitive programmers out there, and experts in areas like Distributed Systems, Machine Learning, and Complexity Science. Our founder, Zuzanna, actually finished her PhD focusing on forecasting maritime trade. It was during that research that she noticed a real need for applying Machine Learning to data streams in logistics. She teamed up with Jan, who brought his experience from Google Brain, and together with Adrian and Claire, they developed Pathway to solve the industry’s gap in software infrastructure for real-time data processing.

Who is Pathway for?

• Software engineers
• Python Developers

How to use Pathway?

To get the most out of Pathway, here’s a straightforward way to approach it:

1. Setting Up Data Ingestion: You can easily connect and sync data from over 300 different sources. This is great for vector search, creating real-time features, and setting up anomaly alerts. Pathway automatically keeps your data in sync, ensuring your AI insights are accurate, whether they come from connected documents or tables.
2. App Templates Setup Guide: Pathway provides helpful app templates and a setup guide. These make it simple to create applications that fetch real-time data, so you don’t need to worry about setting up complex databases or entire tech stacks.
3. Streamlined Development: Built using Rust but designed for Python developers, Pathway really smooths out the ML/AI project lifecycle, from the initial prototype all the way to production. You can deploy your applications in various ways, whether it’s locally, within a notebook, or scaled up in containers.
4. Connect Live Data Sources: You can access live data from a wide range of sources, including SharePoint, Google Drive, S3, Delta Tables, Kafka, various databases, and over 300 APIs like Salesforce and Hubspot. Pathway’s high-speed engine is built for real-time updates, scalability, and cost-effective computing.
5. Deployment Flexibility: You have full ownership of your deployment. Use Docker or Kubernetes, whether you’re running on-premises or in the cloud. This means you can avoid juggling multiple databases and compute engines, simplifying your setup and giving you complete control over your data pathways.
6. Customization and Integration: We offer ready-to-use Pathway templates that are tailored for different industries and data types. You can also customize these using Pathway’s library, connectors, and integrations, which are designed for safe interaction with LLMs and external APIs.
7. Reactive Design: Pathway makes it easy for Python developers to build real-time data products at scale. You can focus on your code logic without worrying about how data updates are managed. The framework handles data processing seamlessly, which is perfect for keeping BI dashboards up-to-date.

By following these steps, you can really make the most of Pathway’s capabilities when you’re building real-time data applications for all sorts of different needs.