Marketplaces Scripts
Handling Marketplaces Data and FTP Operations

7min
This document provides an overview of key components and processes for managing marketplace data and performing FTP operations in our script.
Code Overview
Main Scripts
In this section, we will delve into the two primary modules that play a central role in our script.
get_data module - The most pivotal module in our script is get_data, which is used globally. It contains a function, get_listing_data, designed to Extract, Transform, and Load (ETL) data into a CSV file. The function connects to a database, executes SQL scripts, fetches data, processes it, and saves it to a CSV file.
Python
with connect(host=config.server, database=config.database,
                     user=config.user, password=config.password,
                     ) as db_conn:
    with db_conn.cursor(cursor_factory=RealDictCursor) as cur:
        cur.execute(script)
        rows = cur.fetchall()
        regular_dicts = [dict(row) for row in rows]
        products_df = pd.DataFrame(regular_dicts)
        if 'EAN' in products_df.columns:
            pattern = re.compile(r'^[0-9]{12,13}$')
            products_df = products_df[products_df['EAN'].str.match(
                pattern, na=False)]
        if marketplace.lower() == 'kaufland':
            products_df = products_df.dropna(subset=['ean'])
            products_df.to_csv(filename, header=True, index=False,
                               sep=';', encoding='utf-8')
        else:
            products_df.to_csv(filename, header=True, index=False,
                               encoding='utf-8')
with connect(host=config.server, database=config.database,
                     user=config.user, password=config.password,
                     ) as db_conn:
    with db_conn.cursor(cursor_factory=RealDictCursor) as cur:
        cur.execute(script)
        rows = cur.fetchall()
        regular_dicts = [dict(row) for row in rows]
        products_df = pd.DataFrame(regular_dicts)
        if 'EAN' in products_df.columns:
            pattern = re.compile(r'^[0-9]{12,13}$')
            products_df = products_df[products_df['EAN'].str.match(
                pattern, na=False)]
        if marketplace.lower() == 'kaufland':
            products_df = products_df.dropna(subset=['ean'])
            products_df.to_csv(filename, header=True, index=False,
                               sep=';', encoding='utf-8')
        else:
            products_df.to_csv(filename, header=True, index=False,
                               encoding='utf-8')
﻿
The code snippet provides a brief overview of what happens in the function.
ftp_config module - The ftp_config module houses the upload_files function, which is responsible for uploading files to an FTP server. This function ensures proper FTP host, user, and password values are provided, logs into the FTP server, and uploads files to a specified path.
Python
with FTP(config.ftp_host) as ftp:
    ftp.login(user=config.ftp_user, passwd=config.ftp_password)
    print(ftp.getwelcome())
    with open(file_name, 'rb') as f:
        ftp.storbinary(
            'STOR ' + upload_path, f)
with FTP(config.ftp_host) as ftp:
    ftp.login(user=config.ftp_user, passwd=config.ftp_password)
    print(ftp.getwelcome())
    with open(file_name, 'rb') as f:
        ftp.storbinary(
            'STOR ' + upload_path, f)
﻿
This snippet provided an overview of how the function works
Now lets Look at how the FTP Marketplaces Function.
FTP Marketplaces
This module contains 3 functions, execute_sql, upload_files_to_ftp and main.
Before this 2 list that contain tuples are defined.
1. sql_scripts_and_filenames - This list contains typles that reference maretplace name, sql script, and filenames.
2. file_and_upload_paths - This List contains tuples that contains the upload paths and the fienames of the files.
execute_sql function - The execute_sql function executes a list of SQL scripts and saves the results to corresponding files. It iterates through the list of tuples, calling the get_listing_data function for each tuple and logging progress.
Python
for marketplace, script, filename in scripts_and_filenames:
    try:
        get_listing_data(marketplace, script, filename)
        main_log.info(f"Completed getting data for {filename}")
        time.sleep(30)
    except Exception as e:
        main_log.error(f"Error executing SQL script {script}: {e}")
for marketplace, script, filename in scripts_and_filenames:
    try:
        get_listing_data(marketplace, script, filename)
        main_log.info(f"Completed getting data for {filename}")
        time.sleep(30)
    except Exception as e:
        main_log.error(f"Error executing SQL script {script}: {e}")
﻿
upload_files_to_ftp - This function uploads a list of files to an FTP server. It iterates through the list of tuples, calls the upload_files function for each, and logs the upload process.
Python
for upload_path, file in file_paths:
    try:
        upload_files(upload_path, file)
        main_log.info(f"Completed running upload files for {upload_path}/{file}")
        time.sleep(10)
    except FileNotFoundError as file_error:
        main_log.error(f"File not found: {file_error}")
for upload_path, file in file_paths:
    try:
        upload_files(upload_path, file)
        main_log.info(f"Completed running upload files for {upload_path}/{file}")
        time.sleep(10)
    except FileNotFoundError as file_error:
        main_log.error(f"File not found: {file_error}")
﻿
main function - The main function combines the above functions for seamless execution and scheduling. It orchestrates the data retrieval and FTP upload processes.
Conclusion
This document has provided an insight into how marketplace data is fetched and uploaded to the FTP server for listing. For further details, refer to the code and comments within the script.
Updated 25 Mar 2024
Did this page help you?
Marketplaces Documentation
Spartoo Web Service Integration