3PL Integration Pipeline

introduction this section provides an overview of the 3pl integration pipeline, focusing on the extraction, transformation, and loading (etl) process of data from linnworks into a relational database we utilize the linnworks api to retrieve data, which is then processed and organized into various tables within our database given the substantial volume of data involved, we employ asynchronous extraction methods therefore, a basic understanding of asynchronous programming in python is necessary to comprehend the pipeline’s operation the relational database we use for this pipeline is postgresql the primary types of data managed by this pipeline include inventory data and levels data installation clone the repository and install the dependencies git https //github com/e mps co uk/linnsapi cd linnsapi python3 m venv linnsvenv source linnsvenv/bin/activate pip install r requirements txt create a env file and add the get the keys from linnworks and remote database credentials \# database db = usernamedb = password = server ip = host = token = applicationid = applicationsecret = tokenurl = apiurl = running the script the data pipeline can be run through the main py where it will fetch and load the inventory and levels data code structure ├── linnsapi │ ├── linnworksdata │ │ ├── init py │ │ ├── inventory py │ │ ├── inventoryitems py │ │ ├── levels py │ │ ├── linnworksapi py │ │ ├── locations py │ │ ├── scripts │ │ │ ├── init py │ │ │ └── sql scripts py │ │ └── threepl inventory py │ ├── init py │ ├── alldata py │ ├── config py │ ├── get stocks py │ ├── log py │ ├── models │ │ ├── init py │ │ └── models py │ └── upload logger py ├── readme md ├── alembic ini ├── archbee yaml ├── config ini ├── main py ├── pyproject toml ├── requirements txt ├── tests │ ├── test api py │ ├── test inventory py │ └── test inventoryitems py └── tox ini endpoints data is fetched using various endpoints within the linnsapi these endpoints include https //api linnworks net/api/auth/authorizebyapplication used to obtain an access token for interacting with the linnworks api https //eu ext linnworks net/api/stock/getstockitems retrieves basic stock item data, including stock levels for the provided location https //eu ext linnworks net/api/stock/getstocklevel batch retrieves all stock level data https //eu ext linnworks net/api/stock/getstockitemsfullbyids retrieves comprehensive stock item data, including extended properties, based on the provided stock item ids for more reference on the endpoints ot the entire linnworks api documentation, visit here https //apidocs linnworks net/reference/ the documetation of this pipeline has been subdivided into several parts database setup docid\ ivdzowj gbshokwjvhsw3product item models docid\ n5q1lsjsxc8rpqzvanb5xconnecting and interacting with the linnworks api docid\ koflji62undpuorvrgtybinventory items retrieval module docid\ ohbcfncojomy2k12ul zhlevels data levels md i recommend reading the documentation in this format, as it provides a comprehensive understanding of the entire process