API integration services connect two systems so data moves between them reliably. The work that consumes budget is not the first successful call. It is rate limiting, pagination at scale, partial failure in batches, idempotency on retries, token rotation, schema drift, sandbox mismatch and webhook redelivery. Each has a known fix, and each costs more after launch.
What this covers
- The failure catalogue, with the engineering cost of fixing each one before and after launch
- Documented rate limits from Stripe, Shopify, Xero, NetSuite and Salesforce, and why they surface in production
- Pagination that changes shape at scale, and what breaks when records are inserted mid-run
- Partial failure semantics inside a batch of fifty records
- Idempotency keys, and the most expensive bug in this category
- OAuth 2.0 token rotation and the three in the morning expiry
- Schema drift, deprecation notice periods and what vendors actually give you
- Sandbox environments that do not behave like production
- Webhook delivery guarantees, ordering and replay
- The receiving side: signature verification on raw bytes, replay windows and the idempotent consumer pattern
- Contract tests against a recorded fixture, and why a sandbox that never drifts proves nothing
- What to log on every call, correlation IDs across systems, and the alert that catches partial failure first
- Why the vendor's hourly rate is almost irrelevant to the total cost of API integration services
The failure catalogue, priced
Every page selling api integration services describes capability. Almost none describe what goes wrong. That is the gap this piece fills, because the eight failures below account for most of the money spent on integration work after the first release, and all eight are predictable before a line of code exists.
The table uses engineering days as the unit. Those day ranges are our own estimate drawn from our build and remediation work, not a published benchmark, and they assume a mid-level engineer already familiar with the codebase. Treat them as relative weights rather than a quote.
| Failure mode | How it presents | When you normally find it | Cost to build in before launch | Cost to retrofit after launch |
|---|---|---|---|---|
| Rate limiting | Intermittent 429s, then a backlog that never drains | First real data migration or first busy day | 1 to 2 days | 5 to 15 days plus a backfill |
| Pagination at scale | Missing or duplicated records, silently | Weeks later, during a reconciliation | 1 day | 8 to 20 days plus data repair |
| Partial failure in batches | A batch reported as successful, three records absent | When a customer notices the gap | 2 to 3 days | 10 to 25 days plus manual correction |
| Missing idempotency key | Duplicate charges, duplicate orders, duplicate invoices | Immediately, from an angry finance team | 1 to 2 days | 5 to 10 days plus refunds and goodwill |
| Token rotation | The integration stops overnight, no error surfaced to users | The following morning | 1 to 2 days | 3 to 8 days plus the outage |
| Schema drift | Parse errors, or worse, silent type coercion | On the vendor's release day | 2 to 4 days | 5 to 15 days, under time pressure |
| Sandbox mismatch | Passes in test, fails in production | Launch week | 2 to 5 days | 5 to 12 days plus a rollback |
| Webhook replay and ordering | State machine ends up in an impossible state | Three to six months in | 2 to 4 days | 10 to 30 days plus a state audit |
Read the last two columns together. The ratio is roughly five to one, and in the pagination and webhook rows it is closer to ten to one, because by the time you find those failures you are not only fixing code, you are also working out which historical records are wrong and repairing them while the system stays live. That asymmetry is the whole argument of this article.
The scale of the exposure is documented. Fivetran's Enterprise Data Infrastructure Benchmark Report, published 26 March 2026 from a survey of 500 senior data and technology leaders conducted in Q4 2025, found enterprises spending an average of $29.3 million a year on data programmes, of which $2.2 million goes purely to keeping pipelines running, with 53% of engineering time absorbed by pipeline maintenance and an average of 4.7 pipeline breaks per month. The full Fivetran benchmark report also puts average monthly downtime at 60.4 hours. Salesforce's 2026 Connectivity Benchmark Report, published 5 February 2026 from a survey of 1,050 IT leaders across nine countries taken in October and November 2025, reported the average organisation running 957 applications, up from 897 the previous year, with only 27% of them integrated.
Those two numbers sit oddly together. Organisations are adding applications faster than they are connecting them, and the connections they do build consume half of an engineering team's time to keep alive.
Rate limits you will not find until production
Rate limiting is the failure everyone has heard of and almost nobody budgets for, because a development integration moving forty records never touches a limit. The first migration moving four hundred thousand does.
The published limits vary enormously in shape, and that shape determines your architecture. Stripe's documented global limit is 100 requests per second per account in live mode and 25 per second in a sandbox, with individual endpoints capped at 25 per second unless stated otherwise, and specific resources carrying their own ceilings: 1,000 update requests per PaymentIntent per hour, 20 read and 20 write requests per second on the Files API, 15 payout creations per second. Exceeding any of them returns a 429 with a Stripe-Rate-Limited-Reason header that distinguishes a global rate breach from an endpoint breach, a concurrency breach or a resource-specific one. Stripe documents a separate 429 with code lock_timeout, which is not a rate limit at all but object-level lock contention, and the fix for that one is serialising writes to the same object rather than backing off globally.
Shopify does something different. Its GraphQL Admin API prices each query in points against a leaky bucket, with 100 points per second on Standard, 200 on Advanced, 1,000 on Shopify Plus and 2,000 on Commerce Components, and a hard ceiling of 1,000 points for any single query regardless of plan. Object fields cost one point, scalars cost nothing, mutations cost ten, and connection fields are priced by the first or last argument you pass. Shopify reserves the requested cost before execution and refunds the difference afterwards. That means a badly shaped GraphQL query can be rejected for a cost it never actually incurred, and the fix is query design rather than throttling.
Xero publishes per-tenant limits of five concurrent calls, 60 calls per minute and 5,000 calls per day, with a separate app-wide ceiling of 10,000 calls per minute across every connected organisation. Every response carries X-DayLimit-Remaining, X-MinLimit-Remaining and X-AppMinLimit-Remaining, and a breach returns 429 with a Retry-After header. Oracle's NetSuite governs concurrency rather than request rate: the account base limit is five concurrent requests on the Standard service tier, 15 on Premium and 20 on Enterprise and Ultimate, increased by ten for each SuiteCloud Plus licence, so a Premium account with three licences gets 45. Development and partner accounts are fixed at five and do not scale. Salesforce caps concurrent inbound requests running 20 seconds or longer at 25 in a production org and returns REQUEST_LIMIT_EXCEEDED beyond that, while placing no limit on the number of shorter concurrent calls.
Five vendors, five completely different mental models. A rate limiter written for Stripe's requests-per-second model will be wrong for Shopify's points model, wrong for NetSuite's concurrency model, and wrong for Xero's daily cap, which cannot be smoothed by backoff at all. If you have exhausted 5,000 Xero calls by two in the afternoon, exponential backoff does nothing. You need a different sync strategy.
Why do rate limits only show up after launch?
Because development data sets are small and production data sets are not, and because retries multiply load exactly when the system is already under strain. A naive retry policy turns one rate-limited request into three, which pushes the account further over the limit, which rate-limits more requests. Stripe's own guidance is to retry on an exponential backoff schedule with randomness added to avoid a thundering herd, and to consider a client-side token bucket that throttles traffic globally rather than per call site.
The correct design is a shared budget. One process, or one queue, holds the account's rate allowance and everything else asks it for permission. Retry with exponential backoff and jitter, cap the attempt count, and put anything that exhausts its retries into a dead letter queue rather than dropping it. Add a circuit breaker so that a vendor outage stops your workers hammering a dead endpoint. None of this is difficult. It is one to two days of work at the start of a project and a fortnight in the middle of an incident.
Pagination that changes shape at scale
Offset pagination is the default in most api integration tools and most hand-rolled clients, because it is easy to reason about. Ask for records 0 to 99, then 100 to 199. It works perfectly until records are inserted while you are paginating.
Consider a list sorted newest first, with offset paging. You read page one, one hundred records. While you are processing it, twelve new records arrive. You request page two at offset 100. Those twelve new records have pushed everything down by twelve positions, so the first twelve records of your page two are records you already read on page one, and twelve other records have moved past your window and will never be read at all. You get duplicates and silent gaps in the same run, and nothing errors.
Cursor-based pagination solves it by anchoring to a record rather than a position. Stripe's list endpoints use exactly this scheme: starting_after and ending_before both take an existing object ID, results come back in reverse chronological order, limit accepts 1 to 100 and defaults to 10, and a has_more boolean tells you whether to continue. Because the cursor is an object ID rather than a count, an insertion during your run cannot shift your position.
| Scheme | Behaviour when records are inserted mid-run | Can you jump to page N | Where it is normally found |
|---|---|---|---|
| Offset and limit | Duplicates and skipped records, silently | Yes | Older REST APIs, most internal admin APIs |
| Cursor on an opaque ID | Stable, no duplicates or skips | No | Stripe, most modern REST APIs |
| Keyset on a sorted column | Stable if the sort column is immutable and unique | No | Databases, high-volume export endpoints |
| Page token with server-side snapshot | Stable, but the token expires | No | Bulk and export APIs, GraphQL connections |
What breaks when records are inserted mid-pagination?
Three things, in order of how expensive they are to discover. First, duplicates, which you will catch if your write path has a uniqueness constraint and will not if it does not. Second, skipped records, which produce no error anywhere and surface months later when somebody reconciles totals. Third, and most awkward, a sync watermark that has advanced past records it never read, so a rerun will not recover them either.
The defence is not clever. Use cursor pagination where the vendor offers it. Where only offset paging exists, sort on an immutable field such as a creation timestamp plus an ID tiebreak, and page forward on that value rather than on a position. Then write a reconciliation job that counts records on both sides on a schedule and alerts on drift. That job is the single highest-value thing you can build into a custom api integration, and it is usually the first thing cut for time.
Partial failure: when three of fifty records fail
Batch endpoints are where integrations quietly become incorrect. You send fifty records. The response is a 200. Forty-seven were written.
Most vendor batch endpoints are not transactional. They process records independently and return a per-record result array, and if your client checks only the HTTP status it will record all fifty as delivered. We have been called in to fix this specific bug more often than any other single defect in api development and integration services work, and it is always found the same way: somebody reconciles two systems and the counts differ.
Three semantics exist, and you must know which one you are dealing with before you write the handler.
All-or-nothing batches roll back the whole set if any record fails. These are the easiest to handle and the rarest to find. Per-record batches return a status for each item, which is the common case, and require you to parse the result array, split successes from failures, and route the failures somewhere durable. Best-effort batches accept the payload, return an acknowledgement, and process asynchronously, which means your only evidence of failure arrives later through a webhook or a status endpoint. NetSuite api integration work runs into the third shape constantly, because bulk operations in ERP systems are queued rather than synchronous.
Should a batch of fifty fail as one unit or fifty units?
Fifty units, almost always, with one exception. If the fifty records form a single business fact, such as the lines of one invoice or the legs of one journal entry, they must succeed or fail together, and if the vendor will not give you that guarantee you should not be using the batch endpoint for them. For everything else, partial success with per-record handling moves more data with less operator intervention.
What partial failure handling actually requires is a dead letter queue, a retry policy that distinguishes transient failures from permanent ones, and a human-readable reason on every failed record. A 429 is transient and should be retried. A 422 for a missing required field is permanent and retrying it forever just burns rate limit budget that successful records need. Getting that classification right is maybe half a day of thought and it is the difference between an integration that heals itself and one that needs a person every Monday.
Idempotency, and the most expensive bug in this category
If you retry a request that creates something, and the original request actually succeeded but the response was lost, you create the thing twice. On a payment api integration that means charging a customer twice. On an order integration it means shipping twice. This is the most expensive bug in the whole category, and the fix is a header.
An idempotency key is a unique value the client generates and sends with a write request. The server stores the result against that key. If the same key arrives again, the server returns the stored result rather than performing the operation a second time. Stripe's documentation on idempotent requests spells out the mechanics precisely: keys can be up to 255 characters, Stripe recommends V4 UUIDs or another random string with enough entropy to avoid collisions, the status code and body of the first request are saved regardless of whether it succeeded or failed, and subsequent requests with the same key return that same stored result including 500 errors. Keys may be removed automatically once they are at least 24 hours old, at which point a reuse generates a new request. Stripe also compares the incoming parameters against the original and errors if they differ, which stops a key being accidentally reused for a different operation. All POST requests accept an idempotency key; GET and DELETE do not need one.
Two details in that paragraph cause real bugs. The 24-hour pruning window means an idempotency key is not a permanent deduplication record, so a retry from a queue that has been stuck for two days will create a second object. And the stored-result-includes-errors behaviour means that if your first attempt returned a 500 and you retry with the same key, you get the same 500 back forever, which is correct behaviour and looks like a broken integration to anyone who has not read the docs.
The rules we apply on every build are short. Generate the key on the client, deterministically, from the business fact rather than randomly per attempt, so that a retry from a cold-started worker produces the same key. Persist the key alongside the record before sending, not after. Never reuse a key across different payloads. And where the vendor offers no idempotency mechanism at all, which is common in older crm api integration work, build a deduplication table keyed on your own natural key and check it before every write.
What happens if you retry a payment without an idempotency key?
You take the money twice, and you find out from the customer. The refund is the cheap part. The expensive parts are the finance reconciliation, the support load, the chargeback risk if the customer disputes rather than contacts you, and the fact that a duplicate charge is the single fastest way to lose a B2B account. This is why we treat idempotency as a first-commit concern rather than a hardening task, and why it is the first thing we ask about when reviewing somebody else's integration.
Token rotation and the three in the morning expiry
OAuth 2.0 access tokens expire. Refresh tokens are exchanged for new access tokens, and in most modern implementations the refresh token itself is replaced at the same time. That replacement is where integrations break.
RFC 9700, the OAuth 2.0 Security Best Current Practice published by the IETF in January 2025, states that refresh tokens for public clients must be sender-constrained or use refresh token rotation, and recommends sender-constraining access tokens through mutual TLS or DPoP. Rotation means the old refresh token is invalidated the moment a new one is issued. If two workers refresh at the same moment, one of them wins and stores a valid token, and the other stores a token that was invalidated a millisecond later. The integration then fails at whatever hour the current access token expires, which is disproportionately overnight, because that is when batch jobs run and nobody is watching.
The fixes are all boring and all necessary. Refresh through a single flight lock so that only one process can exchange a refresh token at a time and the rest wait for the result. Persist the new refresh token in the same transaction that consumes the old one, so a crash between the two cannot lose it. Refresh proactively at a fraction of the token lifetime rather than reactively on a 401, because a reactive refresh under load produces exactly the concurrent-refresh race you are trying to avoid. Allow for clock skew. And alert on refresh failure specifically, separately from general error rates, because a failed refresh produces no user-visible error until the next scheduled sync does not run.
Why do integrations fail at three in the morning specifically?
Because that is when scheduled jobs run, when token lifetimes that started during business hours expire, and when nobody is looking at a dashboard. An integration that fails during the working day gets noticed in minutes. One that fails at three in the morning is discovered at nine, by which time eight hours of records are missing and the backfill has to contend with rate limits. Observability on integrations means alerting on the absence of expected activity, not only on errors. A sync that should run hourly and has not run for three hours should page someone, even though nothing has technically thrown.
Schema drift: the field that appeared on Tuesday
Vendors change their APIs. They add fields, change types, rename enumerated values and deprecate endpoints, and the notice you get varies from generous to none.
Shopify publishes the generous end of that range. Shopify's API versioning policy commits to a new stable version every three months, released at 5pm UTC on the first day of the quarter and named by date such as 2026-04, with each stable version supported for a minimum of 12 months and at least nine months of overlap between consecutive versions. Under that cadence the next stable version lands on 1 October 2026. A stable version is guaranteed not to change for its supported lifetime, which is the property that actually matters: it means a breaking change arrives on a date you can put in a calendar rather than on a Tuesday afternoon.
The other end of the range is harder. OpenAI notified developers of the Assistants API deprecation on 26 August 2025 and removed the endpoints exactly one year later on 26 August 2026, directing integrations to the Responses API and the Conversations API. That is twelve months of notice for a total removal, and teams still missed it. Cloudflare's published deprecation list carries an end of life of 27 September 2026 for the legacy Registrar domain management endpoints and 15 October 2026 for the legacy Workers KV routes. Salesforce retired platform API versions 21.0 through 30.0 in Summer '25 after deprecating them in Summer '22, and calls against a retired version now return 410 Gone on REST, a 500 with UNSUPPORTED_API_VERSION on SOAP and a 400 with InvalidVersion on Bulk.
How much notice does a vendor actually give before a breaking change?
Between nothing and twelve months, and the variance is the problem rather than the average. Additive changes usually get no notice at all, because most vendors do not class adding a field as breaking. That is fine if your parser ignores unknown fields and catastrophic if it validates strictly and rejects the payload.
Our rule is tolerant reading and strict writing. Parse defensively, ignore fields you do not recognise, and never assume an enumerated value list is closed. Then snapshot the response schema in a contract test that runs on a schedule against the vendor's sandbox, and fail the build when the shape changes. That test is the difference between finding out about schema drift from your CI pipeline and finding out from a customer. Pin the API version explicitly in every request where the vendor supports it, because a client that defaults to "latest" is opting into surprise. And subscribe a real, monitored inbox to the vendor's developer changelog, because deprecation notices are sent to whichever address was on the account when the app was registered, which in our experience is frequently a developer who left two years ago.
Sandbox environments that do not behave like production
A sandbox is a model of the vendor's system, and models are wrong in specific ways. The most common surprises are rate limits, latency and data shape.
Stripe documents its own case plainly: sandbox limits are lower than live mode, 25 requests per second against 100, and Stripe explicitly discourages load testing against a sandbox because the test will hit limits it would never hit in production. It also notes that creating a charge in live mode sends a real request to a payment gateway while the sandbox mocks it, producing significantly different latency profiles. Stripe's recommendation is to build a configurable mocking layer inside your own integration and simulate latency sampled from real live-mode call durations.
Data shape is the other trap. Sandbox accounts contain small, tidy, recently created records. Production accounts contain a decade of history, records created by five generations of internal tooling, null fields that the current documentation says are required, and text fields containing characters nobody anticipated. An integration tested only against sandbox data has never met a customer name with an apostrophe in it, a product with a 900-character description, or a record whose currency field is empty because it predates multi-currency support.
What we do about it is unglamorous. Take an anonymised extract of real production data early and test against that. Where the vendor permits it, run read-only calls against the live account during development. And treat launch week as a ramp rather than a switch: run the integration in shadow mode against production for a few days, writing nowhere, and compare what it would have done against what the existing process did.
Webhook delivery, ordering and replay
Webhooks look simple and carry the least intuitive semantics of anything in this article. The three properties that matter are delivery guarantee, ordering and duplication, and the honest answer for most vendors is at-least-once, unordered, and yes.
Stripe documents this directly. Events are not guaranteed to arrive in the order they were generated, and Stripe's own example is a subscription creation producing customer.subscription.created, invoice.created, invoice.paid and charge.created in an order you cannot rely on. Stripe advises against using the created timestamp to determine order or to detect reprocessing, because snapshot events record created in whole seconds and distinct events can share one. Track event IDs instead. Delivery is retried with exponential backoff for up to three days in live mode, three times over a few hours in a sandbox, and events can be resent manually for up to 15 days from the Dashboard or 30 days through the CLI. Redirect responses are treated as failures, so a 302 from your load balancer silently kills the endpoint. The libraries apply a five-minute tolerance between the signature timestamp and the current time, which means a server with a drifting clock will reject valid events.
Shopify's numbers are tighter. It retries eight times over four hours, applies a one-second connection timeout and a five-second timeout for the whole request, and after eight consecutive failures deletes the webhook subscription entirely if it was created through the Admin API, with a warning email to the app's emergency developer address. That last behaviour catches people out. An endpoint that is down for a morning does not merely miss events, it loses the subscription, and nobody notices until somebody asks why orders stopped syncing.
Do webhooks arrive in order, and does it matter?
They do not, and it matters whenever your handler transitions a state machine. If order.updated arrives before order.created, a handler that assumes creation precedes update will either error or create a phantom record. If payment.refunded arrives before payment.succeeded, a naive balance calculation ends up negative.
The pattern that works is to treat every webhook as a notification rather than as data. Record the event ID for deduplication, acknowledge with a 2xx immediately, and put the work on a queue. Then have the worker fetch the current state of the object from the API rather than trusting the payload, so ordering stops mattering: whichever event arrives last, you read the truth. Where you must act on the payload itself, carry a version or sequence number on your own record and discard any event whose version is older than what you have already applied. Both approaches are two to four days of work at the start and a state audit later.
Stripe also limits you to 16 registered webhook endpoints per account, which sounds generous until a company with several product teams starts allocating them.
Webhook consumers: the half of the contract you own
A webhook has two sides, and the vendor's retry policy is only the first one. The receiving endpoint carries its own guarantees, and that is where the expensive defects live.
Two delivery models exist. At-most-once means the vendor fires the event and forgets it, so a thirty-second deployment window costs you every event sent during it, permanently. At-least-once means the vendor retries until it gets a 2xx, so duplicates are not a risk to mitigate, they are a certainty to design around. Almost every commercial webhook is at-least-once. Exactly-once delivery across a network boundary is not on offer from anyone, and a vendor implying otherwise is describing at-least-once delivery with a deduplication layer in front of it. You are the deduplication layer.
What does an idempotent webhook consumer actually look like?
The first statement in the handler inserts the vendor's event ID into a table with a unique constraint. On conflict, return 200 and do nothing else. On success, apply the effect inside the same database transaction as that insert, so a crash between the two rolls both back. One constraint turns at-least-once delivery into once-only processing, which is the closest thing to exactly-once that this problem permits.
It breaks in two places. Where the effect lands somewhere that cannot join your transaction, a payment capture or an outbound email, the insert and the effect can diverge, and the repair is a claim row carrying an explicit state of received, in flight or applied, plus an idempotency key on the outbound call derived from the event ID. The second place is a deduplication table pruned faster than the vendor can replay. Stripe's manual resend window runs to 15 days from the Dashboard and 30 days through the CLI, so a dedupe table holding event IDs for 24 hours treats a day-twelve replay as a brand new event and applies it a second time. The usual trigger is an engineer replaying a week of events to close a gap, which means the duplicate lands during a recovery, on top of an incident already in progress. Retain processed event IDs past the vendor's longest replay window, with margin.
Signature verification has its own failure shape. Verify the HMAC against the raw request bytes before anything parses them. The common defect is a framework that decodes the JSON and hands the handler a re-serialised copy, with different key order and whitespace, so the signature stops matching on every event. Under launch pressure somebody disables verification to get the integration working, and the endpoint becomes an unauthenticated write path into your database for anyone who can guess the URL. Nothing in your monitoring will ever mention it again. Secret rotation has the same shape: accept the old and the new secret through the overlap period, because a straight swap rejects every event already in flight.
Then there is the handler that does the work before it answers. Shopify allows one second to connect and five seconds for the entire request. A handler taking eight seconds has already been recorded as a failed delivery and queued for redelivery while its original work completes normally, which produces duplicates that look like vendor misbehaviour and are not. Acknowledge with a 2xx first, queue the work, and let the worker take as long as it needs. Where that worker starts an action rather than writing a row, a duplicate stops being a duplicate record and becomes a second purchase order, and the retry semantics tighten accordingly. The architectural difference between an agent that acts and a pipeline that records decides how much that distinction costs you.
The integration test harness nobody builds
Most integration test suites assert against mocks a developer hand-wrote from the vendor's documentation on the day the work started. Those mocks encode what the team believed the API returned, and they keep passing regardless of what it returns now. A vendor adds a value to an enumerated field, or changes an amount from a string to a number, and the suite stays green through the entire release. Production finds it, then a customer does.
A harness that catches this has three layers and is not expensive. Record real responses from live calls, redact the personal data, and commit them as fixtures, so unit tests run offline and fast against payloads that genuinely existed. Then run a scheduled contract test against the vendor that diffs today's response shape against the committed fixture: a new field is logged, a removed field or a changed type fails the build, and an enumerated value outside the known set raises a warning. A diff that fails in CI on a Tuesday morning costs an afternoon. The same drift found by a batch job at three the following morning costs the backfill as well.
The trap is assuming the sandbox is the vendor. Some sandboxes are pinned to an older API version, or updated on a schedule that differs from production, so a contract test watching a frozen copy reports a stability that does not exist anywhere. Check whether the vendor version-pins its sandbox, and where it does, point the contract test at a read-only call against a real account instead. A sandbox that never drifts is not evidence of a stable API. It is evidence of a stale environment, and it produces a test suite whose green run means nothing.
What the harness should exercise is the failure path, because that is the code that has never run. Simulate a 429 carrying a Retry-After, a 500 halfway through a batch, a connection reset after the request is sent but before the response arrives, a duplicate webhook, an out-of-order webhook, and an expired refresh token. The connection reset matters most: it is the exact scenario idempotency keys exist for, and in most codebases we are asked to review, the idempotent retry path has never executed once outside a real incident. Code that first runs during an outage is not a mitigation. On XOVO's own estimating basis, a fixture-backed harness covering those six cases is two to three engineer-days on a typical two-system build.
Observability for an integration, and the one alert that matters
An integration produces almost no user-visible signal when it degrades, so the logging decisions made in week one determine how long an investigation takes in month nine.
Log one structured line per attempt rather than per call: correlation ID, vendor, endpoint, method, status code, attempt number, duration, record count, the idempotency key sent, and the vendor's own request identifier from the response. Stripe returns a request ID on every response and it is the first thing their support will ask for, and Shopify returns one on Admin API responses. A team that discards those identifiers has no way to ask the vendor about a specific call, which turns a ten-minute support exchange into an argument about timestamps. Emit the rate limit headers as gauges rather than as log text, so Xero's remaining daily allowance becomes a line on a graph you can watch approach zero at two in the afternoon rather than a 429 you discover afterwards. Never log full payloads or tokens. Log the record key and a hash of the body.
Correlation IDs have to be minted per business fact rather than per HTTP call. One ID for the order, carried across the webhook that announced it, the calls that fetched and enriched it, the retry that duplicated it, and the reconciliation that caught the duplicate. Without that, answering "what happened to invoice 4471" means correlating three log stores by timestamp while the finance team waits. With it, the answer is one query and the same day.
Which alert catches a partial failure first?
Not the error rate. A threshold such as "page when more than five per cent of requests fail" is the wrong instrument here, because two per cent of records failing quietly and permanently is a data divergence that never breaches it. The unit is the record, not the request. The alert that earns its keep compares records accepted by the destination against records sent, per sync run, and fires on any non-zero gap, paired with a dead letter queue depth alarm whose threshold is one rather than a percentage. Put the absence check beside it, because a sync that should have run hourly and has not run for three hours fails without throwing anything at all. Caught by those two, a partial failure is a replay from the dead letter queue. Caught by the finance team at month end, it is a reconciliation across a month of records while the system carries on writing new ones.
Why hourly rate is the wrong way to compare API integration services
Here is the position, stated plainly. When you are choosing an api integration company, the hourly rate is one of the least useful inputs available to you, and the difference between a good and a bad choice is almost never priced into it.
Take a mid-sized integration quoted at, say, 30 days of work. A vendor at $50 an hour and a vendor at $90 an hour are separated by roughly $9,600 on that build. Now look at the retrofit column in the failure table at the top of this article. A missing idempotency key discovered after launch is 5 to 10 days of remediation. Webhook ordering discovered six months in is 10 to 30 days, plus the audit. Two of those, and the cheaper vendor has cost more than the difference several times over, before counting the incident, the customer trust, and the finance team's weekend.
This is why the question to ask a prospective api integration service is not what they charge. It is how they handle partial failure, what their retry policy looks like, whether they use idempotency keys on writes, where failed records go, and how they will know the integration has stopped. Ask to see the code of a previous integration's error handling path. A team that builds for partial failure from the first commit produces an integration you can forget about. A team that builds the happy path and adds error handling at the end produces one that needs a person.
Most vendors do not build for partial failure first, and the reason is structural rather than dishonest. Error handling does not demo. A client watching a progress review sees records appearing in the destination system and concludes the work is going well, and nothing in that review reveals whether the retry policy exists. The cost surfaces six months later, in a quarter nobody budgeted for.
If you are commissioning this work, our full-stack development team will tell you what your failure surface actually looks like before you commit to a number. That review takes a few days and it is the cheapest thing in this article.
What goes into the first commit
For any custom api integration services engagement we run, the following exists before the first feature works end to end. This is our own practice, not an industry standard, and it is the part of the process that makes the day ranges in the table above achievable.
A shared rate limit budget, sized from the vendor's published limits rather than guessed, with exponential backoff, jitter and a capped attempt count. A dead letter queue with a human-readable failure reason on every entry, plus a way to replay from it. An idempotency key on every write, derived from the business fact. A correlation ID threaded through every request and log line, so a single record's journey can be reconstructed from one search. Explicit API versioning in every request. A contract test on a schedule, pointed at whichever environment actually tracks the vendor's current version, failing loudly when the response shape changes. A reconciliation job comparing counts and checksums across the boundary. And an alert on the absence of expected activity, not only on error rate.
That list is roughly a week of work on a typical build. It is also the entire difference between an integration that runs for three years without attention and one that generates a support ticket every fortnight. Teams building this alongside a new product should read our 90-day MVP scoping guide, which covers where integration work sits in a compressed first release, and the SaaS product development practice page for how we sequence it against tenancy and billing.
Custom build, integration platform or middleware
Not everything should be hand-built, and an api integration platform earns its licence fee in specific conditions. The decision turns on volume, transformation complexity and how much the two schemas disagree.
| Situation | Custom api integration | iPaaS or integration platform | Why |
|---|---|---|---|
| Two systems, stable schemas, low volume | Reasonable | Reasonable | Either works, pick on team skills |
| Many systems, similar patterns, non-technical owners | Rarely | Yes | The connector library and the visual editor are the product |
| Heavy transformation, business logic in the mapping | Yes | Usually not | Logic in a visual mapper is logic you cannot test properly |
| Real-time, sub-second, high throughput | Yes | Usually not | Platform round trips and per-operation pricing dominate |
| ETL into a warehouse on a schedule | No | Yes | This is a solved problem, buy it |
| Regulated data with residency constraints | Yes | Only if certified | Where the data physically sits is contractual |
| Vendor with no published API, screen scraping required | Yes | No | The fragility needs to be yours to manage |
The honest middle ground is that most companies end up with both. A managed platform handles the ETL and the long tail of low-value connections, and hand-built middleware behind an API gateway handles the two or three integrations that carry actual business logic. Trying to force the second category into a visual mapper is how organisations end up with business rules nobody can find and nobody can test. Common api integration examples that belong in custom code include anything touching money, anything where a partial failure has a legal consequence, and anything where the transformation encodes a pricing rule.
Where the integration feeds an automated decision rather than a database, the design changes again. Our writing on enterprise workflow automation agents covers what changes when a system acts on the data it receives, and the AI procurement agent product page shows a working example of an integration whose output triggers an action rather than a row. For agent-driven work specifically, the agentic AI development page sets out how we handle tool-call retries, which are the same problem as API retries with a less forgiving failure mode.
What we would not do
We would not quote an integration without reading the vendor's rate limit page, pagination scheme and webhook retry policy first. Those three documents change the estimate by a factor of two or more, and any api integration developer quoting without them is guessing.
We would not accept "the sandbox works" as evidence that anything works.
We would not build a write path without idempotency, even when the vendor does not support it, because the deduplication table is half a day and the duplicate charge is a lost account.
We would not tell you that api integration tools have made this category easy. The tools have made the first call easy, which is the part that was never expensive. Everything in this article is still yours to handle, whether you build it or buy it, and the platform that abstracts the failure away is usually the platform that hides it until it is expensive.
And we would not put business logic inside a visual mapping tool, however quickly it demos.
If you are scoping api development and integration services now, bring us the vendor documentation and the record volumes and we will give you a failure surface before a price. Book a technical scoping session and we will go through the eight failures in this article against your specific systems. For teams evaluating a broader build, the B2B website development playbook and the 2026 guide to custom website development cover the surrounding decisions.
