Handling Marketplaces Data and FTP Operations

this document provides an overview of key components and processes for managing marketplace data and performing ftp operations in our script code overview main scripts in this section, we will delve into the two primary modules that play a central role in our script get data module the most pivotal module in our script is get data, which is used globally it contains a function, get listing data, designed to extract, transform, and load (etl) data into a csv file the function connects to a database, executes sql scripts, fetches data, processes it, and saves it to a csv file with connect(host=config server, database=config database, user=config user, password=config password, ) as db conn with db conn cursor(cursor factory=realdictcursor) as cur cur execute(script) rows = cur fetchall() regular dicts = \[dict(row) for row in rows] products df = pd dataframe(regular dicts) if 'ean' in products df columns pattern = re compile(r'^\[0 9]{12,13}$') products df = products df\[products df\['ean'] str match( pattern, na=false)] if marketplace lower() == 'kaufland' products df = products df dropna(subset=\['ean']) products df to csv(filename, header=true, index=false, sep=';', encoding='utf 8') else products df to csv(filename, header=true, index=false, encoding='utf 8') the code snippet provides a brief overview of what happens in the function ftp config module the ftp config module houses the upload files function, which is responsible for uploading files to an ftp server this function ensures proper ftp host, user, and password values are provided, logs into the ftp server, and uploads files to a specified path with ftp(config ftp host) as ftp ftp login(user=config ftp user, passwd=config ftp password) print(ftp getwelcome()) with open(file name, 'rb') as f ftp storbinary( 'stor ' + upload path, f) this snippet provided an overview of how the function works now lets look at how the ftp marketplaces function ftp marketplaces this module contains 3 functions, execute sql , upload files to ftp and main before this 2 list that contain tuples are defined 1\ sql scripts and filenames this list contains typles that reference maretplace name, sql script, and filenames 2\ file and upload paths this list contains tuples that contains the upload paths and the fienames of the files execute sql function the execute sql function executes a list of sql scripts and saves the results to corresponding files it iterates through the list of tuples, calling the get listing data function for each tuple and logging progress for marketplace, script, filename in scripts and filenames try get listing data(marketplace, script, filename) main log info(f"completed getting data for {filename}") time sleep(30) except exception as e main log error(f"error executing sql script {script} {e}") upload files to ftp this function uploads a list of files to an ftp server it iterates through the list of tuples, calls the upload files function for each, and logs the upload process for upload path, file in file paths try upload files(upload path, file) main log info(f"completed running upload files for {upload path}/{file}") time sleep(10) except filenotfounderror as file error main log error(f"file not found {file error}") main function the main function combines the above functions for seamless execution and scheduling it orchestrates the data retrieval and ftp upload processes conclusion this document has provided an insight into how marketplace data is fetched and uploaded to the ftp server for listing for further details, refer to the code and comments within the script