Hi, I have a somewhat peculiar question. I have some files on a windows server that I would like to automatically upload into erpnext for accounting people to view. What is the best choice? I have tried uploading them via ftp to the private/files folder directly in the OS but these I don’t see in the file manager. Can anyone help me?
And, while importing, make sure you are importing to a folder, such as Attachments, Home, or any other folder, so that you can unzip the file.
thank you for your reply, but I need an automatic system. I try to explain myself better:
i have files in a folder on disk > start a script every night > send the files in the folder to erpnext.
Currently, also thanks to chatgpt, I have tried various api rest based methods with powershell but to no avail.
You can try two things:
Option 1:
- Copy the files to a path under “private/files” folder of the site using SFTP or any other file protocol.
- Create a “File” Doctype record with necessary attributes through a scheduled job in Frappe or directly from Windows using curl.
Option 2:
Option 1 or 2 will be based on the number of files that you need to process. If there are 100s of files, my preference will be to use Option 1.
1 Like
Hi, thank you for your reply. I set up an FTP server pointing to the erpnext private/files folder and via a powershell script I upload the files from the windows server to the ftp located on the ubuntu server (where erpnext is installed).
Through various answers online I came to the conclusion that I should rely on your option 1 but could you be more specific? I don’t really understand how it works. If I create a scheduled process in frappe that logs the files in the folder at regular intervals this means that at each startup the files already present are duplicated in the file manager, is this correct? It happened to me using various powershell scripts that took advantage of curl.
Thanks for the clarification
Take a look at this diagram below. You can manage duplication by querying the ‘File’ doctype to see if the file with same name and size exists. But general ETL best practices will help you here.
1 Like
ok then, I followed model 1 as it seems to be the most suitable for my case: the source data comes from multiple windows servers and not one source, so it is easier to manage it directly from erpnext.
I created a script server scheduled every minute (for debugging) with a function that, theoretically, should check the staging folder and save them in the erpnext database by moving them to private/files. I have helped myself with chatgpt but it is still a dead end. I am attaching the script so you can see if I am proceeding in the right direction.
def move_files_to_private_directory():
staging_dir = "url/private/staging"
private_files_dir = "url/private/files"
frappe.log("Inizio spostamento file dalla directory di staging.")
# Verifica se la directory di staging esiste
if not frappe.utils.file_manager.file_exists(staging_dir):
frappe.errprint(f"La directory di staging {staging_dir} non esiste.")
return
files = frappe.get_all('File', filters={'file_url': ['like', f'%{staging_dir}%']})
for file in files:
file_doc = frappe.get_doc('File', file.name)
destination_url = private_files_dir + "/" + file_doc.file_name
# Verifica se il file esiste già e se la dimensione è diversa
if not frappe.utils.file_manager.file_exists(destination_url) or file_doc.file_size != file_doc.file_size:
frappe.utils.file_manager.move_file(file_doc.name, destination_url)
frappe.log(f"Spostato file: {file_doc.file_name} in {destination_url}")
else:
frappe.log(f"Il file {file_doc.file_name} esiste già e ha la stessa dimensione.")
move_files_to_private_directory()
1 Like
Hi @Giastru:
This simple Powershell script will work.
Here we upload all the files in a folder and, after succesfully upload, renames the file with “_uploaded” suffix … Use a scheduled task on your Windows Server to run it periodically. You can even link the file to a document …
$api_key = "yourapikey"
$api_secret = "yourapisecret"
$folder_path = "C:\yourfolder"
$frappe_url = "https://yoursite.com/api/method/upload_file"
$headers = @{
"Authorization" = "token $($api_key):$($api_secret)"
}
$files = Get-ChildItem -Path $folder_path
foreach ($file in $files) {
$file_path = $file.FullName
$file_name = $file.Name
$file_stream = [System.IO.File]::OpenRead($file_path)
$file_content = New-Object System.Net.Http.StreamContent($file_stream)
$multipartContent = New-Object System.Net.Http.MultipartFormDataContent
$multipartContent.Add($file_content, "file", $file_name)
# Want to add to a document?
$multipartContent.Add((New-Object System.Net.Http.StringContent("1")), "is_private") # 1 for private files
# $multipartContent.Add((New-Object System.Net.Http.StringContent("Sales Invoice")), "doctype")
# $multipartContent.Add((New-Object System.Net.Http.StringContent("SINV-0001")), "docname")
$httpClient = New-Object System.Net.Http.HttpClient
$httpClient.DefaultRequestHeaders.Add("Authorization", "token $($api_key):$($api_secret)")
try {
$response = $httpClient.PostAsync($frappe_url, $multipartContent).Result
$responseContent = $response.Content.ReadAsStringAsync().Result
if ($response.IsSuccessStatusCode) {
Write-Host "Successfully uploaded: $file_name"
$new_file_name = [System.IO.Path]::GetFileNameWithoutExtension($file_path) + "_uploaded" + [System.IO.Path]::GetExtension($file_path)
$new_file_path = [System.IO.Path]::Combine($file.DirectoryName, $new_file_name)
Rename-Item -Path $file_path -NewName $new_file_path
Write-Host "Renamed file to: $new_file_name"
} else {
Write-Error "Failed to upload file: $file_name. Server response: $responseContent"
}
} catch {
Write-Error "Error uploading file: $file_name. Error: $_"
} finally {
$file_stream.Dispose()
}
}
1 Like
Lines 8 to 12 needs is not correct. You have to use python’s os utils to get the file list and then create the doctype like this code. or use this function.
Please keep the staging directory outside the site directory
This will work in most cases but may not be suitable for saving lot of files /large files due to HTTP limitation.
I have ruled out using a powershell script located on the windows server because there are multiple points from which the files transferred to erpnext start (we are talking about 20/30 different windows servers) and maintaining the script could be risky, I would rather have the control done from the ubuntu server where erpnext is hosted but I will give it a try anyway. Thank you very much!!!
I realized that the script servers created within erpnext are unable to see files present on the file system, so I created a custom app to which I associated the code I made earlier with the changes you pointed out and this is the result:
import os
import frappe
from frappe.utils.file_manager import save_file, remove_file
def register_files_in_directory():
# Percorso della directory di staging (esiste sul filesystem)
staging_dir = "/home/erpnext/frappe-bench/sites/url/private/staging/"
private_files_dir = "/home/erpnext/frappe-bench/sites/url/private/files/"
frappe.log(f"Registrazione file dalla directory: {staging_dir}")
# Controlla se la directory di staging esiste
if not os.path.exists(staging_dir):
frappe.throw(f"La directory di staging {staging_dir} non esiste.")
# Scansiona i file nella directory di staging
for file_name in os.listdir(staging_dir):
file_path = os.path.join(staging_dir, file_name)
private_file_path = os.path.join(private_files_dir, file_name)
# Verifica se il file esiste già nel filesystem della cartella private
if os.path.exists(private_file_path):
frappe.log(f"Il file {file_name} esiste gia nel filesystem: {private_file_path}. Non sara copiato di nuovo.")
continue # Salta la registrazione se il file esiste già nel filesystem
# Calcola il file_url atteso per il file nel File Manager
file_url = f'/private/files/{file_name}'
# Verifica se il file esiste già nel database (controllo su file_name e file_url)
existing_files = frappe.db.get_all("File", filters={"file_name": file_name, "file_url": file_url})
if existing_files:
# Il file esiste già nel database, non duplicarlo
frappe.log(f"Il file {file_name} esiste gia nel database con URL: {file_url}. Non sara duplicato.")
continue # Salta la registrazione se il file esiste già nel database
# Registra il file solo se non esiste già nel filesystem e nel database
frappe.log(f"Registrazione del file: {file_name}")
try:
with open(file_path, "rb") as file_content:
# Salva il file nel sistema ERPNext e nel database
saved_file = save_file(file_name, file_content.read(), None, None, is_private=1)
frappe.log(f"File {file_name} registrato con successo con URL: {saved_file.file_url}")
# Conferma la transazione nel database
frappe.db.commit()
# Dopo la registrazione, elimina il file originale dalla cartella di staging
os.remove(file_path)
frappe.log(f"File originale {file_name} rimosso dalla directory di staging.")
except Exception as e:
frappe.errprint(f"Errore durante la registrazione del file {file_name}: {str(e)}")
Sorry for the comments but I am Italian. This script, on an hourly schedule is able to transfer files from staging to files and register them within erpnext but unfortunately still not able to transfer folders and creates strange duplicates for me that remain in the /files folder even after deletion from the erpnext file manager.
If there are lot of servers , you can try using ftp letting your erpnext job to pull data periodically. You have just maintain the configuration.
If you want to handle sub folders , I think you have update your code accordingly. If the file is not getting deleted, it can be a file permission issu as well
Thank you very much.
For the subfolders I’m still working on it, I’ve been idle this weekend. As soon as I have a solution, I will share the process here so I can solve someone’s future problem
1 Like
When you deal with sub-folders, pay attention to the below code. Also see if you can use enqueue methods for async processing of folders to speed up the process.