Palantir Pipeline Builder and Foundry Data Integration: Data Connection, Code Repositories, and What Actually Breaks

Last updated: October 9, 2026

By Omar J, Director, PAL4C Labs

TL;DR Palantir Foundry data integration runs in three steps before the Ontology. Data Connection brings data in through agent workers or Foundry workers, Pipeline Builder or Code Repositories transforms it, and lineage, health checks, and expectations keep it trustworthy before it feeds the Ontology. Palantir recommends designing pipelines in Pipeline Builder first. Pick batch or incremental unless you truly need sub-10-minute latency. Our Palantir Foundry and AIP implementation team spends most of a first project right here.

Nobody buys Palantir for the pipelines. They buy it for the Ontology, the AIP agents, the operational apps. Then they spend most of the first three months on pipelines anyway.

That’s not a complaint about Foundry. It’s the nature of the work. MuleSoft’s 2025 Connectivity Benchmark found the average organization runs 897 applications with only 29% of them integrated, and 80% of IT leaders named data integration as their biggest obstacle to AI. Palantir doesn’t make that integration gap go away. It gives you better tools for dealing with it.

This guide walks the full path inside Foundry, from connecting a source to handing clean data to the Ontology. It covers Data Connection and its worker model, Pipeline Builder versus Code Repositories, batch versus incremental versus streaming, LLMs inside pipelines, the quality controls that keep pipelines honest, SAP, and the gotchas we see on real projects. It’s written for data engineers and data leads scoping or inheriting a Foundry build. Our data integration and pipeline engineering practice does this work daily.

How Does Data Get Into Palantir Foundry?

Three steps, then a handoff. Data Connection syncs data from source systems, Pipeline Builder or Code Repositories transforms it, Data Health and expectations check it, and the clean outputs back Ontology objects that apps and agents use.

Every stage is versioned and traceable, which is the part that actually matters in a regulated or audited environment. When a planner asks why a number changed, Data Lineage gives you an interactive view of how data flowed through the platform to get there.

How Does Data Connection Work?

Data Connection defines each external system as a source and syncs data through one of two worker types. Agent workers run on a host you provide inside your network. Foundry workers run inside Foundry for internet-reachable systems. Virtual tables register external data without copying it.

Palantir’s Data Connection documentation defines a source as a single connection to an external system. The worker type decides where the sync compute runs.

If you’ve read older guides, you’ve seen the term direct connection. Palantir’s docs now say direct connections have been incorporated into the Foundry worker concept, and that Foundry worker sources avoid the maintenance overhead of agents. Same idea, new name.

Which should you pick? The Palantir Community advice is short and right. In a thread on agents versus direct connections, the recommendation was to use a direct connection when you can, because there’s less to set up and maintain. Agents are for when the network leaves you no choice. And they need care, since the same thread notes it’s possible to fill up agent queues.

The slow part is rarely technical. Firewall rules, service accounts, and security reviews for an agent host run on someone else’s calendar. Start those requests in week one.

What Is Palantir Pipeline Builder?

Pipeline Builder is Foundry’s primary application for data integration. It’s a visual, form-based tool for building pipelines that turn raw sources into clean outputs, running on Spark and Flink, with Git-style branches and full version control.

That definition comes straight from Palantir’s Pipeline Builder overview, which calls it “Foundry’s primary application for data integration.” The core concepts page explains that a pipeline branch is a copy you can iterate on without touching the main pipeline, “similar to a code branch in Git,” and that outputs can be datasets, files, virtual tables, or Ontology components such as object types, link types, or time series.

Read that output list again. Pipeline Builder can write straight into Ontology object types. That cuts out a whole handoff step between data engineering and the people building apps.

Who uses it? Not only engineers. Analysts who know the business logic but don’t write PySpark can build and review transforms, and engineers can still read every step. That shared view is worth more than it sounds. In our experience, a lot of pipeline bugs turn out to be misunderstood business rules, not bad code.

Pipeline Builder vs Code Repositories, Which Should You Use?

Start in Pipeline Builder. Palantir recommends it for pipeline design. Move to Code Repositories when you need API calls, filesystem access, custom libraries, advanced streaming, or complex logic your team would rather test as code.

Palantir publishes its own comparison of Pipeline Builder and Code Repositories, and it opens with a clear recommendation to design pipelines in Pipeline Builder. Here’s the practical version.

Code Repositories is a web-based IDE on Git for production code, and Palantir calls Python “the most full-featured language for authoring data transformations in Foundry.” Most teams that pick Code Repositories end up writing PySpark.

Our rule of thumb? If a business analyst should be able to review the logic, Pipeline Builder. If the step calls an outside service or needs a unit test suite, Code Repositories. Plenty of projects use both, with the gnarly transforms in code and everything else visual.

What we’d avoid is the religious version of this debate. A team of Python engineers forcing everything into code makes pipelines nobody else can read. A team forcing everything into the visual tool ends up building awkward workarounds for things code does in four lines.

Should Your Pipeline Be Batch, Incremental, or Streaming?

Batch or incremental for anything that can tolerate more than about 10 minutes of latency. Streaming only when the decision truly needs fresher data, because streaming pipelines usually cost more to run.

Palantir’s own guidance on streaming versus batch is refreshingly direct. For use cases that can tolerate more than 10 minutes of latency, incremental or standard batch datasets may be suitable, and the average cost of a streaming pipeline is often higher than the average cost of a batch pipeline.

Incremental is the underrated middle option. Incremental transforms use the build history to process only new or changed data instead of recomputing the whole output every run. For a table that grows by 50,000 rows a day on top of 200 million, that can be the difference between a build measured in minutes and one measured in hours.

So ask the business, not the engineers. Does a dispatcher need this within seconds, or is every 15 minutes fine? We’ve watched teams build streaming pipelines for dashboards people check twice a day. Expensive habit.

Can You Use LLMs Inside Foundry Pipelines?

Yes. Pipeline Builder’s Use LLM node runs language models on your data at scale, with templates for classification, summarization, translation, sentiment analysis, and entity extraction. LLM calls inside streaming pipelines have more limits, so plan those carefully.

Palantir’s Use LLM documentation describes the node as a convenient way to execute LLMs on data at scale. Practical uses we see are classifying free-text maintenance notes, extracting supplier names from PDFs, and tagging customer complaints by topic before they become Ontology properties.

Streaming is the catch. In a Palantir Community thread on LLMs in streaming, a moderator noted that using SDKs in functions wasn’t yet supported in streaming and pointed users to function-backed models instead. Check the current docs before you design around it.

Cost counts too. Every row through an LLM node uses tokens, and AIP converts tokens into compute. Run the node on a sample, check output quality, then scale. Our breakdown of AIP agents and evals covers how to test LLM outputs before trusting them.

How Do You Keep Foundry Pipelines Trustworthy?

With three controls. Data Health checks monitor datasets for status, freshness, size, content, and schema problems. Data expectations fail a build when data breaks a rule. Lineage shows where every field came from when someone asks why a number changed.

Data Health provides monitoring and alerting across five check categories, and Monitoring Views extend that across whole projects at once. Expectations go further, since they stop a build rather than just alerting, which is what you want for rules like no duplicate order IDs or no shipments with a missing destination.

Our minimum set for any production pipeline.

• A freshness check on every dataset an app or agent depends on
• Schema checks on raw inputs, because source systems change columns without telling anyone
• Expectations for primary keys being unique and not null
• Row count bounds on key outputs. A 90% drop overnight is a bug, not a quiet day
• An owner for every alert. Unowned alerts become wallpaper within a week

Gartner research has estimated that poor data quality costs organizations at least $12.9 million a year on average. That figure is from 2020, and your number will differ. The direction won’t.

How Do You Connect SAP to Palantir Foundry?

Through Palantir’s SAP connector, an SAP-certified add-on installed in your SAP system, paired with a Data Connection agent. It supports SAP ECC, S/4HANA, BW, and SLT, and setup involves both SAP Basis and Foundry teams.

Palantir’s SAP documentation describes the Palantir Foundry Connector 2.0 for SAP Applications as an SAP Certified Add-On developed with Diskover Limited, covering ECC, S/4HANA on-premises and Cloud Private Edition, BW, and SLT. Setup runs through a defined sequence that includes installing an agent.

The technical install is the easy half. The harder half is deciding which tables you actually need. SAP has a lot of them. The useful data for a procurement use case might sit across a dozen, and pulling everything just in case makes pipelines slow and expensive. Get someone who knows your SAP configuration in the room for scoping. Every time.

What Breaks in Real Foundry Pipeline Projects?

Access delays, schema drift, oversized full rebuilds, unowned alerts, and preview behavior that hides incremental bugs. Most problems are organizational or operational, not limits of the tools.

Here’s what we see most, roughly in order of how much time it costs.

Access comes first. Source owners, service accounts, firewall rules, and security reviews. It’s the longest pole in nearly every project we’ve run. In our engagement with a global logistics operator, connecting 18 disconnected warehouse and inventory systems was the foundation of a build that ran 5 months from kickoff to deploy, and it’s what made the 34% cut in procurement delays possible, per our case studies.

Then schema drift. A source system adds a column or renames one, and a pipeline downstream fails at 3 a.m. Schema checks and clear ownership fix this. Hope doesn’t.

The incremental preview trap catches experienced engineers too. Palantir’s incremental docs note that the Code Repositories preview always runs transforms in non-incremental mode, even if you require incremental. So a transform can look perfect in preview and behave differently in a real build. Test incremental logic with real builds.

Full rebuilds by default are a cost leak. Pipelines that reprocess years of history every night because nobody made them incremental. They work. They’re just expensive.

And alert fatigue. Fifty health checks with no owners. Nobody reads them by the end of the month.

Most of this gets caught, or doesn’t, in operations. That’s why pipeline monitoring and cost hygiene are core to our platform operations and MLOps work.

How Do Pipelines Feed the Ontology?

Pipeline outputs become the backing datasets for Ontology object types. Clean, deduplicated tables with stable primary keys map to objects like Shipment or Supplier, and links between those tables become relationships agents and apps can follow.

That’s the payoff for all the work above. A Supplier object is only trustworthy if the pipeline behind it has already merged the three duplicate vendor records from your two ERPs. Get the keys right in the pipeline and the Ontology is straightforward. Get them wrong and every app built on top inherits the mess. Our Palantir Ontology explainer covers objects, links, and actions in depth.

Plan the Integration Before You Plan the Agents

Three things to remember. Connect with Foundry workers where you can and agents where you must. Design in Pipeline Builder, move to code when the logic demands it, and choose batch or incremental before streaming. Put health checks and expectations on every output an app depends on.

PAL4C Labs is a Palantir-exclusive consultancy that builds Foundry pipelines, Ontologies, and AIP workflows for operations, supply chain, healthcare, and regulatory teams. Integration is where we start every project, because it’s where most projects are won or lost.

Send us your list of source systems and the decision you want to support. We’ll sketch the integration plan with you, including which worker type each source needs, which pipelines should be incremental, and where the access requests should start this week.

Data Engineering Questions We Get

So What Exactly Is Palantir Pipeline Builder?

Pipeline Builder vs Code Repositories, Does the Choice Actually Matter?

Does Palantir Foundry Support Streaming Pipelines?

How Do You Connect SAP to Foundry?

Agent Worker vs Foundry Worker, What’s the Difference?

What Are Incremental Pipelines in Foundry?

Can Pipeline Builder Use LLMs?