Enigma Data Infrastructure
Enigma Concourse
Acquire data at scale
Build resilient and reliable data pipelines
Our data operations platform and runtime are designed to build data pipelines that are easy to maintain. Enigma Concourse provides greater visibility into ingestion, improves error resilience with minimal human intervention, and equips developers with the tooling they need for consistent data ingestion.
Manage unpredictable data sources
Minimize downtime with granular error reporting
Precise error alerting and identification capabilities inform you the minute a pipeline breaks and reveal exactly where the error took place. Get detailed guidance to get your pipeline back up and running.
Get visibility into your live data flow
Depend on a tool that goes beyond job scheduling. The Concourse platform provides a layer of intelligence on top of data ingestion and offers complete transparency into the datasets flowing through your pipelines.
Concourse is ready for complexity
A Fortune 500 manufacturer integrates data from factories around the globe with Concourse.
Solving the California drought crisis with Enigma Concourse
Concourse provides critical infrastructure to integrate water use data from 411 different databases across California.
A proven integration framework for scale
Assemble pipelines with minimal custom code
Avoid the redundant work and challenging maintenance that comes with custom code. Building pipelines with our expansive library of reusable “steps” allows you to standardize data ingestion and onboard new team members efficiently.
Tackle all formats from all sources to all destinations
Rely on a framework with proven experience in handling public data’s most difficult formats and scenarios. Both source-and destination-agnostic, the Concourse runtime is fully extensible with Python and designed to support any data format or protocol.
Automate and autoscale your data delivery
Schedule pipelines for specific times or set dynamic rules to run recurring jobs that keep your data current. The Concourse platform dynamically provisions and optimizes server-resource requests for each run and delivers detailed performance metrics for every execution cycle.
Learn what Concourse can do for your data operations
Work with the tools you know and trust
Keep your current developer workflow
Construct and test pipelines on your local machine, using familiar tools and conventions – if a pipeline works on your local machine it also works on the platform. Simplify pipeline maintenance by relying on the Concourse runtime’s interactive debugging features.
Extend Concourse with Python
Extend the Concourse runtime with custom steps written in Python, with support from any of the 100,000 open-source packages actively maintained by the community. Host those new steps on an internal PyPI server to instantly share them with anyone in your organization.
Trusted and currently used by Fortune 500 IT teams and developers
Concourse features
- Define your pipelines with YAML
- Interactive debugging with breakpoints, pause on exception and more
- Dozens of prebuilt steps to support popular sources, formats and destinations
- Powerful CLI
- Private Pypi-like server to host custom Concourse steps and packages
- REST API to control and automate every aspect of the Platform – Fine-grained scheduling with support for time zones
- One-click execution of any pipeline from the UI
- Instant alerts and notifications for quick error recovery
- Webhooks for real-time integrations
- Full parser execution logs and metrics
- Full encryption of data in transit and at rest
- LDAP authentication
- Support for Kerberos
- Complete audit trail of data and users
- Detailed documentation
Interested?
Discover what Enigma Concourse can do for your organization.