Pipelines
3PL Integration Pipeline

7min
Introduction
This section provides an overview of the 3PL Integration Pipeline, focusing on the extraction, transformation, and loading (ETL) process of data from Linnworks into a relational database.
We utilize the Linnworks API to retrieve data, which is then processed and organized into various tables within our database. Given the substantial volume of data involved, we employ asynchronous extraction methods. Therefore, a basic understanding of asynchronous programming in Python is necessary to comprehend the pipeline’s operation.
The relational database we use for this pipeline is PostgreSQL. The primary types of data managed by this pipeline include inventory data and levels data.
Installation
Clone the repository and install the dependencies
Shell
git https://github.com/E-MPS-CO-UK/LinnsApi
cd LinnsApi
python3 -m venv linnsvenv
source linnsvenv/bin/activate
pip install -r requirements.txt
git https://github.com/E-MPS-CO-UK/LinnsApi
cd LinnsApi
python3 -m venv linnsvenv
source linnsvenv/bin/activate
pip install -r requirements.txt
﻿
Create a .env file and add the get the keys from Linnworks and remote database credentials.
Shell
# database
DB =
USERNAMEdb =
PASSWORD =
server_ip =
host =


token =
ApplicationId = 
ApplicationSecret =
tokenurl = 
apiurl = 

# database
DB =
USERNAMEdb =
PASSWORD =
server_ip =
host =


token =
ApplicationId = 
ApplicationSecret =
tokenurl = 
apiurl = 
﻿
Running the script
The Data Pipeline can be run through the main.py where it will Fetch and load the Inventory and Levels Data.
Code Structure
Shell
.
├── LinnsApi
│   ├── LinnworksData
│   │   ├── __init__.py
│   │   ├── inventory.py
│   │   ├── inventoryitems.py
│   │   ├── levels.py
│   │   ├── linnworksapi.py
│   │   ├── locations.py
│   │   ├── scripts
│   │   │   ├── __init__.py
│   │   │   └── sql_scripts.py
│   │   └── threepl_inventory.py
│   ├── __init__.py
│   ├── alldata.py
│   ├── config.py
│   ├── get_stocks.py
│   ├── log.py
│   ├── models
│   │   ├── __init__.py
│   │   └── models.py
│   └── upload_logger.py
├── README.md
├── alembic.ini
├── archbee.yaml
├── config.ini
├── main.py
├── pyproject.toml
├── requirements.txt
├── tests
│   ├── test_api.py
│   ├── test_inventory.py
│   └── test_inventoryitems.py
└── tox.ini
.
├── LinnsApi
│   ├── LinnworksData
│   │   ├── __init__.py
│   │   ├── inventory.py
│   │   ├── inventoryitems.py
│   │   ├── levels.py
│   │   ├── linnworksapi.py
│   │   ├── locations.py
│   │   ├── scripts
│   │   │   ├── __init__.py
│   │   │   └── sql_scripts.py
│   │   └── threepl_inventory.py
│   ├── __init__.py
│   ├── alldata.py
│   ├── config.py
│   ├── get_stocks.py
│   ├── log.py
│   ├── models
│   │   ├── __init__.py
│   │   └── models.py
│   └── upload_logger.py
├── README.md
├── alembic.ini
├── archbee.yaml
├── config.ini
├── main.py
├── pyproject.toml
├── requirements.txt
├── tests
│   ├── test_api.py
│   ├── test_inventory.py
│   └── test_inventoryitems.py
└── tox.ini
﻿
Endpoints
Data is fetched using various endpoints within the LinnsApi. These endpoints include:
https://api.linnworks.net/api/Auth/AuthorizeByApplication - Used to obtain an access token for interacting with the Linnworks API.
https://eu-ext.linnworks.net/api/Stock/GetStockItems - Retrieves basic stock item data, including stock levels for the provided location.
https://eu-ext.linnworks.net/api/Stock/GetStockLevel_Batch - Retrieves all stock level data.
https://eu-ext.linnworks.net/api/Stock/GetStockItemsFullByIds - Retrieves comprehensive stock item data, including extended properties, based on the provided Stock Item IDs.
For more reference on the endpoints ot the entire Linnworks API documentation, visit here﻿
The documetation of this pipeline has been subdivided into several parts:
﻿Database Structure and Installation﻿﻿
﻿Models﻿﻿
﻿Linnworks API﻿﻿
﻿Main Inventory Data﻿﻿
﻿Levels Data﻿
I recommend reading the documentation in this format, as it provides a comprehensive understanding of the entire process.
Titles Generation Script Documentation
Database Setup