3PL Integration Pipeline
This section provides an overview of the 3PL Integration Pipeline, focusing on the extraction, transformation, and loading (ETL) process of data from Linnworks into a relational database. We utilize the Linnworks API to retrieve data, which is then processed and organized into various tables within our database. Given the substantial volume of data involved, we employ asynchronous extraction methods. Therefore, a basic understanding of asynchronous programming in Python is necessary to comprehend the pipeline’s operation. The relational database we use for this pipeline is PostgreSQL. The primary types of data managed by this pipeline include inventory data and levels data.
- Clone the repository and install the dependencies
- Create a .env file and add the get the keys from Linnworks and remote database credentials.
- Running the script
The Data Pipeline can be run through the main.py where it will Fetch and load the Inventory and Levels Data.
Data is fetched using various endpoints within the LinnsApi. These endpoints include:
- https://api.linnworks.net/api/Auth/AuthorizeByApplication - Used to obtain an access token for interacting with the Linnworks API.
- https://eu-ext.linnworks.net/api/Stock/GetStockItems - Retrieves basic stock item data, including stock levels for the provided location.
- https://eu-ext.linnworks.net/api/Stock/GetStockLevel_Batch - Retrieves all stock level data.
- https://eu-ext.linnworks.net/api/Stock/GetStockItemsFullByIds - Retrieves comprehensive stock item data, including extended properties, based on the provided Stock Item IDs.
For more reference on the endpoints ot the entire Linnworks API documentation, visit here
The documetation of this pipeline has been subdivided into several parts:
I recommend reading the documentation in this format, as it provides a comprehensive understanding of the entire process.
