It has never been easier to connect systems together. With enterprise applications increasingly being web-based and being developed with out-of-the-box application programming interfaces (APIs), the ease of which you can push or pull data from different systems is a far cry from the days of file transfers, data streams, or even manual data entry.
Time to REST
Modern architectural styles such as RESTful APIs built on top of HTTP, provide a common structure and integrity for interoperability on these systems. When a request is made to a RESTful API, a HTTP method will be specified (commonly GET, POST, PUT, DELETE) and the service will return a payload (commonly in the format of JSON or XML). By using a stateless protocol and standard operations, RESTful systems aim for fast performance, reliability and the ability to grow by reusing components that can be managed and updated.
There are some downsides with doing direct, point-to-point integrations, however, even if RESTful APIs are used. Let’s say system A wants to send a request to system B and retrieve some data. system A might only accept JSON responses, but system B only responds with XML. Perhaps, system B requires multiple web service requests to perform a simple task for system A. Maybe a change to a web service in system B results in a change needed to system A. When system C and system D come along, common services such as authentication, error handling, logging, monitoring will all need to be configured on a system-by-system basis.
De-coupling with iPaaS
This is where integration platform-as-a-service (iPaaS) comes along. iPaaS solutions such as MuleSoft, Dell Boomi, Oracle Fusion and others act as a middleman between different systems, providing a centralised platform for integrations. Instead of system A sending a request to system B, system A sends a request to iPaaS, which then sends a request to system B. This de-coupling means that iPaaS can take care of much of the heavy lifting on a centralised, re-usable platform.
If system A only accepts JSON responses and system B only responds in XML, iPaaS can convert the XML to JSON. If system B requires multiple web service requests to perform a simple task for system A, iPaaS can perform all these requests and pass back a single response, with system A only needing to make a single request. If a change is made to system B, iPaaS can ensure that no change is needed to system A. When system C and system D come along, iPaaS provides a centralised platform for common services such as authentication, error handling, logging, monitoring.
Re-using APIs
The design of the APIs that you build on your iPaaS, also has an impact on the success of your integrations, because, if done well, you can ensure re-usability of your APIs (which means the time and cost of implementing new integrations becomes minimal). MuleSoft, for example, promotes the use of what they call API-Led Connectivity, which splits APIs into three tiers: System, Process and Experience. System A PIs unlock data from back-end systems Process APIs do any transformation and orchestration required for the data pulled from system A PIs.
Experience APIs present the data from process APIs in a format which the API consumer requires. This approach ensures that you design your APIs in a modular fashion, which is key to enabling re-use, as your APIs are then not unique to every specific integration, but rather designed around the general data and processes that your business uses.
Coarse or fine-grained design?
Within this API-led connectivity methodology, there are further architectural decisions to make. One key decision is choosing between a fine-grained or a coarse-grained API design – and there is no silver bullet solution to this, because going too far in either direction can pose difficulties.
Choosing a fine-grained design means you end up with lots of small APIs which can be used independently of each other. It is great for creating modular, re-usable APIs, however it comes with some challenges. Having lots of small APIs means that discovering and understanding each API and how they work together to provide the data you need, requires more time and becomes a bit more challenging, compared to using a coarse-grained design.
Choosing a coarse-grained design means that you end up with fewer but larger APIs, which means discovering and understanding what each API does potentially becomes easier; this can lead to more re-use compared to fine-grained APIs – even if fine-grained is architecturally more optimal for re-use. The downside to coarse-grained design is that it is potentially less modular and fit for re-use and there are operational challenges to managing changes, as there is more chance that multiple people are going to be developing on the same API.
The optimal design is probably somewhere in the middle of fine-grained and coarse-grained for most businesses. You want APIs to be small enough that they are re-usable for multiple integrations, but not so small that it becomes difficult to discover and understand how each API fits in the bigger picture. Your expected response time requirement (usually measured in milliseconds) is also a consideration because a fine-grained design will involve more API calls, which will increase the end-to-end response time.
Centres for enablement
Some organisational changes are often required when embracing new platforms in IT. Often these are in the form of a centre for enablement (C4E) which consists of an enduring team who ensure that the platform is set up and used according to best practices, with the right amount of support and expertise for delivery teams looking to use the platform.
The core roles in the C4E include: C4E lead, platform lead, platform architect, delivery service lead. As the scale of the demand for the platform increases, more specialised roles can be created under these roles.
Initially, you may want to start with all development and associated activities being done under this C4E to ensure ultimate control while you get used to the platform. As you grow and mature, you may further benefit from a de-centralised model with development left to individual project teams and the C4E largely providing quality assurance.
The size and scope of the C4E largely depends on the size of the demand for the platform, the maturity of the teams using the platform and the resourcing model of the organisation.
Conclusion: embrace the new
At National Grid, we have recently started to embrace these new platforms, moving away from traditional file transfers and onto modern API integrations. Personally, I find API integrations very interesting. It is based on a core set of simple, well understood web-standards, which means you are not constantly re-inventing the wheel to achieve something and everyone that you are working with, is singing from the same hymn sheet. This means that you can dedicate your time to designing well-architected solutions to problems, which can be very intellectually-stimulating.
Looking more broadly at the IT industry outside of traditional enterprise applications, you can see that many IT industry trends in the near future, involve connecting systems together and therefore will rely on APIs for their success. The Internet of Things (IoT) where everyday devices such as cars, home automation systems, or kitchen appliances are connected to the internet in order to retrieve data, exchange data between devices, or control devices remotely being a prime example. The upcoming rollout of 5G might also accelerate this.
Another example is big data: by building APIs on top of large datasets, it becomes easier to retrieve data and do something useful with it, such as analytics and reporting. Artificial intelligence is another example: by using APIs to pull data from multiple sources, analysing that data and making decisions on what to do next.
We’ve already come a long way, and it looks like the demand for connecting systems together is only going to increase.
