Document and Archiving System

Frappe Framework App Development
7

I’m very interested also! File Manager has its strengths for to capture and provide access to files within a file system, but like @MiM I am equally interested in process that will identify the source of a document, updates made over time, and if the document has been e.g. officially approved for use, or a draft for review. Simply identifying that a particular file (Word doc, PDF, mechanical CAD drawing, etc.) has been updated with a new file is sufficient for my needs, I don’t need to provide visibility of changes within a document (not that I wouldn’t like visibility within a document, but I’m willing to do without to simplify the solution).

My motivation is primarily engineering design support (e.g. PDM, PLM, ECO, Change Management, etc.) but also supply chain control (purchase specifications), project management (e.g. project charters and support docs), as well as issue management and general business process (e.g. ISO-9001).

I’d be happy to join my efforts with others on this, a sort of PDM/PLM working committee.

Cheers,
Dale

6 Likes
8

@Dale_Scott have you heard from anyone regarding PLM? We’re also immensely interested in a solution that will enable us to manage PLM data along side ERPNext. I believe that one will need to have an independent database, and the ability to fill it with CAD/CAM/pdf/Image files of varying sizes. We use multiple CAD and CAM systems therefore for us its difficult to align with one PLM provider. Would you like to fund one such activity together with those of us who would benefit from PLM to make erpnext/frappe folks take this up?

Thanks,
Naresh

9

Hi @nash, there are many commercial PLM enterprise solutions. However, being all things to all users for all CAD file formats is not simple and they are expensive, which does not align well with the organizations I am representing ERPNext to.

I have found that either a disciplined approach to storing files by project in a network file system, or using a general purpose file storage and sharing system (e.g. Nextcloud), is the most pragmatic solution for me.

Cheers,

Dale

10

Hi @Dale_Scott, we use a multitude of PLM solutions but nothing, really nothing, cuts it. Teamcenter is very Siemens/UG focused. The only valid “claimed to be OSS” Aras PLM is actually a click-bait system.

I hear your “disciplined” approach idea, but it will work only for those few “disciplined” folks and even for those, it may fall flat in short-order due to the complexity of the process.

Our work is fairly complex in which we have multitude of projects, sub projects etc… going 2-5 levels deep, and 4 different CAD systems, 3 different CAM systems and nothing really works together. I’m of the opinion that in the short term we’ll setup Alfresco, for CAD/CAM files, using meta data from those system which support it such as Solid Edge, Feamap, NX etc. and some which do not will be BLOBs. This will allow all the tools to work together in a project based manner. The alfresco connection with ERPNext for a bidirectional mapping will be crucial.

I am also not too happy to store files in a single place as is currently done in ERPNext. We need to compartmentalize the store and have it accessible only through the relevant DocTypes. This store should be at least automated.

So my idea was to fund some bright young lad to accomplish this connectivity and add it in the OSS repository as our contribution to the ERPNext.

Just read about NexCloud, well that is what Alfreso internally hosted system will do, in addition to process management such as Engineering Change Order requests. We could do that with NexCloud as well.

Cheers,
Naresh

1 Like
11

My thinking lately has tended towards “simpler is better”. A lot of time and effort can be spent on universal technically-correct solutions, but if the designers continue to keep their own project directories on their own local drives, the effort will be squandered. Of course, this all depends on the authority structure of the organization and the design team, the maturity of the project team and whether the leaders believe in process, how much effort can be justified for non-revenue-supporting “doc-control” work, where decision making authority is rooted (Mechanical engineering? Electronics engineering? Manufacturing and the ECO office? Product management? Accountants?), etc.

If “nothing works together”, the most pragmatic solution might be to not try. :wink:

I agree ERPNext is not the place to store the artifacts for any significantly complex design. For purchased items, dragging a datasheet or catalog page for a COTS part onto the Item master may be more than adequate. Handing relatively simple custom parts in the same way may be practical. However, at some level of complexity clarity will be lost and day-to-day use will become problematic.

For me, it’s usually been enough to capture the design structure and supply chain details in the ERP, and reference the design artifacts stored in a separate PLM. The technology behind the PLM (e.g. Oracle PLM, Nextcloud, or a plain old file system) isn’t as important as preserving truth and a well-defined process for change management. Cross-referencing between the two can be as simple as using the same part number format in both systems.

It would certainly be convenient though if there was a link (button) in an Item Master in ERPNext to access the design artifacts associated with the Item in the PLM. Presumably any web-based PLM could be used if the link was a URL (ERPNext may even already provide such capability). It should however be difficult to access newer or older revision level artifacts by accident (but still possible if needed).

I like that Nextcloud integrates into the local file system, since in my experience that’s typically how designers work. The Nextcloud agent can be used to map the local project directory into a larger project space or master PLM space on the Nextcloud server. Like you, I expect Nextcloud (or a Plugin) will provide an approval workflow. The Nextcloud agent doesn’t have to be used though, and designers could instead drag design artifacts into an appropriate folder using the Nextcloud web client.

Bi-directional synchronization will come with edge cases, and could easily be a black hole of time for your bright young lad. The problem can be simplified enormously if a one-way workflow can be accepted (e.g. creating a part in ERPNext could cause a part number folder to be created in the PLM ready for uploading artifacts). Areas of friction can be addressed when they come to light.

Creating a standardized process for design re-use using different CAE tools will also come with edge cases. It may be more pragmatic to always save a couple standard format artifacts that others will likely find convenient (e.g. a JPEG/PNG/PDF rendering, perhaps a STEP export for custom mechanical parts, a zip archive with Gerbers for a PCB), and deal with anything more complex such as migrating the design to a different CAE tool, within a project when the need arises (rather than having the PLM carry the burden of design re-use if re-use is really more exception than practice).

Of course, IMHO and YMMV… :wink:

Cheers,
Dale

1 Like
12

@Dale_Scott check this out GitHub - doloresjuliana/pibiapp: Application on the Frappe framework. Connect Frappe and Nextcloud, store the attachments on the Nextcloud server. Integrates external data from Excel, CSV, JSON or XML files. Integrates the view of the Redash dashboards in Frappe.. I admit i have not ready the entire post since it long and dont have the time but will come back to it

2 Likes
13

Thanks @woakes070048, I will post more comments later after more review, but at least initially the synchronization seems reasonable. I want to explore the behavior from a user’s perspective, in particular when executing an approval workflow in Nextcloud (e.g. is the existence of a doc visible or can the doc be accessed before approval in ERPNext?)

Have you experimented at all with pibiapp? Do you have any reason to recommend it?

14

We have not but are planning on using it shortly for a project. If it does what it says i fairly certain it will be rolled out to several of our clients

15

@woakes070048 will you have an approval workflow for documents?

E.g. new part number is created in ERPNext. Engineer submits design documents for release approval (using Nextcloud? ERPNext?), approval committee reviews and approves (or rejects), and finally stakeholders are notified of the update, and purchasing and manufacturing leads then allowed access to the new documents (via direct search in Nextcloud? via a manufacturing order in ERPNExt?).

16

p.s. I cross-posted to both ERPNext and Nextcloud support forums re nextcloud integration and pibiapp. I’m hoping someone will share personal experiences.

17

@Dale_Scott

I’ll love to dig into your entire email but am in a time crunch. Thus, let me point out some, IMHO, ideas that a lot of folks have, that may be controversial.

“simpler is better” is an opinion from the point of view of the observer. It applies to the user, not the creator of a product. If you look at the simplest of the things, there is a lot of thought process that goes into giving the simplest “appearing” solutions to a user, but under the hood the reality is extensively complex. That is simply because the real world is not simple.

There are a number of things that will not work with ncloud. ncloud to me is a just a fancy UI on top of rsync, and a few other tools, making things simple for the user… but it is limited at best in its capability to host PLM data, most of such data requires versioning, diff-change-visibility such as using images or pennable images for ECR/ECM, for CAE data, not to speak about meta-data extraction as soon as something is checked into the repository and saving that meta-data into ERPNext. CAE data could be CAD from several CAD systems such as what we have, several CAM systems such as what we have, a multitude of FEA tools, pre-post processors, CFD tools, optimization tools, internally written software, PDFs from customers, suppliers etc. etc. And we are a tiny engineering company, most small and mid sized companies are many orders of magnitude complex than us.

“Bi-directional synchronization” would be great, but that is not what I intended, nor it is what we need, because it will come with even more complexity. And there is no intention to lead a bright young lad to a black hole. The idea is to guide that developer enough such that he/she can implement something that would work.

One cannot create a “standardized” process for PLM, the availability of DocType is the reason I’ve gravitated to ERPNext, since it should be possible to give the basic framework of PLM including ECM and let the users configure it to their internal processes. We’re into ISO AS/EN9100 compliance and a few other standards that we follow, all of those require such a feature set which IMHO, DocType would be suited to provide.

My experience is with PLM from Siemens/EDS called Teamcenter, and many research oriented solutions over the past 30 years or so. I was not even aware that Oracle has a PLM, can you explain a bit about how the Oracle PLM works?

Just my $0.02.

Warm greetings,
Naresh

18

Hi Naresh,

“simpler is better” is an opinion from the point of view of the observer. It applies to the user, not the creator of a product. If you look at the simplest of the things, there is a lot of thought process that goes into giving the simplest “appearing” solutions to a user, but under the hood the reality is extensively complex. That is simply because the real world is not simple.

I completely agree with you. “Simpler is better” is not the same as “simpler is easier” or “simpler is cheaper”, which are often what is being intentionally implied, and not the more often true “simple is hard”.

However, what I had meant in this case was that a document system will be more likely to be used, and more likely to be an effective solution, if it is simpler to use (users being e.g. a doc control admin managing a product change, a design engineer submitting new design files or researching an existing product’s change history, a manufacturing tech accessing the latest test procedure, etc.). From this perspective, it is essentially irrelevant how complex the system is under the hood, and continuing the analogy it is only the view from the driver’s seat that is important.

Fwiw, IMHO every additional degree of complexity under the hood will require 10x more effort to bring the product to market - more requirements to understand and communicate, more code to write, more testing to do, more team meetings, more bugs to fix, more beta testing, more, more, more…

There are a number of things that will not work with ncloud. ncloud to me is a just a fancy UI on top of rsync, and a few other tools, making things simple for the user… but it is limited at best in its capability to host PLM data, most of such data requires versioning, diff-change-visibility such as using images or pennable images for ECR/ECM, for CAE data, not to speak about meta-data extraction as soon as something is checked into the repository and saving that meta-data into ERPNext. CAE data could be CAD from several CAD systems such as what we have, several CAM systems such as what we have, a multitude of FEA tools, pre-post processors, CFD tools, optimization tools, internally written software, PDFs from customers, suppliers etc. etc. And we are a tiny engineering company, most small and mid sized companies are many orders of magnitude complex than us.

I’m not trying to minimize the product development process, which can be very complicated. I’m just saying complex software may not be a solution. Reported failure rates for ERP implementations are in the 30% range[1], and (IMHO) enterprise PLM is just as complex to develop as ERP. Of course, statistics depend on the perspective and goal of the survey maker, but this is in line with other reports (and lower than 10-15 years ago, when ERP was less well understood, implementation projects were less methodical and planned, and project failure rates of 70% were being reported).

“Bi-directional synchronization” would be great, but that is not what I intended, nor it is what we need, because it will come with even more complexity. And there is no intention to lead a bright young lad to a black hole. The idea is to guide that developer enough such that he/she can implement something that would work. One cannot create a “standardized” process for PLM, the availability of DocType is the reason I’ve gravitated to ERPNext, since it should be possible to give the basic framework of PLM including ECM and let the users configure it to their internal processes. We’re into ISO AS/EN9100 compliance and a few other standards that we follow, all of those require such a feature set which IMHO, DocType would be suited to provide.

Still in full agreement. Have you identified a suitable off-the-shelf plug-in for Nextcloud that provides a suitable approval workflow?

My experience is with PLM from Siemens/EDS called Teamcenter, and many research oriented solutions over the past 30 years or so. I was not even aware that Oracle has a PLM, can you explain a bit about how the Oracle PLM works?

See Product Lifecycle Management (PLM) Software | Oracle

All the major CAD/CAE vendors also have a PLM solution, they bundle or embed basic PLM with their design tools (e.g. Desault/SolidWorks, Autodesk Inventor) with separate “enterprise” PLM offerings. Personally, I wouldn’t say Teamcenter is particularly “research oriented”, but I’m also not surprised to hear a research organization is using Teamcenter. IMHO Oracle approaches PLM from a more general business process management perspective than product development, although “Agile PLM” was developed independently and was successful as engineering software before being acquired by Oracle.

It seems to be another truth that ERP and PLM enterprise software costs in the range of $100 per user per month, but this doesn’t include a significant implementation project (and the big ERP/PLM players are only seen in “large” enterprises with at least tens of users and have implementation projects measured in months and number of consultants).

Just my $0.02.
Warm greetings,
Naresh

Mine too, and warm greetings to you too (although it has been -30 C here now for over a week!),

Cheers,
Dale

[1] E.g. https://ultraconsultants.com/erp-software-blog/erp-implementation-survey/

19

I went through the apps categories and did not find a basic document approval plugin. :frowning:

However, I did find Mathias who integrated Nextcloud with Camunda, which presumably could be used to create a [submit > review > approve] workflow to capture a snapshot of a project directory. I would want to develop a proof-of-concept to explore in depth before being confident though. Camunda do not publish pricing information for commercial support, but seem to believe in open source and community. www.camunda.org

One solution could be to use Nextcloud to snapshot project directories for “engineering release” using a release process modeled in Camunda. Then in ERPNext, there might might be an enterprise “ECO” or “Management of Change Process” release with workflow designed in ERPNext. One could run such a process without further integration simply by developing the two workflows (one in Nextcloud, one in ERPNext) and manually cross-linking. In phase two, one could support LDAP in ERPNext and Nextcloud to share login credentials (IIUC should be built-in) and to synchronize group membership for enterprise use (start of support revenue!), and finally in phase three add some degree of automatic synchronization.

My $0.02 .-)

20

Hi @nash , I retract my proposal after actually watching the video. Having two systems would be a bad idea at this time. Perhaps at some number of users it will be likely an enterprise may have separate systems, but for now simpler is better. :wink:

I may have come full circle :wink: I now propose designing a workflow using ERPNext’s built-in workflow capability, that makes API calls into Nextcloud, where there are folders one per Item, for the files associated with an Item, whether a datasheet, a quote, a CAE project zip file, a CAE project directory with all the files, etc. The details are likely best left non-specific. One user might be using git for software management and mechanical engineering needs help, another user may have no software version control, but is using SolidWorks’s built-in PLM.

I’m not sure about authentication in Nextcloud. Initially I thought there would need to be duplication and synchronization of all users between ERPNext and Nextcloud, but now I wonder if a single “erpnext” master user could be used by ERPNext to access documents in Nextcloud and avoid more work.

1 Like
21

Hi @Dale_Scott
Let me mull over this and your last few messages and get back to you during the weekend? I’m currently wading through the learner waters of erpnext with totally banal things like the Italian tax regime and how to make my first invoices on it.

As I told you my experience is with extremely complex and powerful research PLM systems of the mid 1990’s. These have nothing to do with commercial systems. That time commercial PLM systems such as teamcenter were not even on the horizon of mid scale companies.

Anyhow, among the leaders in PLM are actually three systems, one from Dassault, one from PTC and one from Siemens/EDS/UG. I’ve seen all of them and none work well in a mix CAD/CAM/CAE environment. Besides in our business we need ITAR and other security guarantees.

TTYL.
Cheers,
Naresh

22

Hi @Dale_Scott

Understanding, at least superficially, frappe and erpnext took some time. Now I’ve run through the frappe tutorial, made the library app and understand some preliminary capabilities of frappe. Frappe appears to be a platform on which we could realize most of the features that we need. I’ve also installed and run your ncloud in a VM and stress tested it. ncloud, does really well what it claims but extending it to do what we’d like to do may be difficult. Allow me to outline how we are forced to work, under the ISO9100 and aerospace requirements we must meet. Currently we do this using a paper trail. I’d like to move all of this in either 1. ERP Next or 2. Erpnext + Frappe+custom apps or 3. number 2 + a PLM and document tracking system.

Within ERPNext, I may be wrong, but it stores all external files (pdf’s or else) in a common repository in the file system and stores only the file identifiers in the DBMS. This may work for some, but it will not work for us. Since we need document level partitioning between our work, and work of some of our clients. The documents themselves should be stored as blobs in the DBMS which should run on an encrypted file system. Lets take a simple example of material acceptance from a supplier. Let us say we get an epoxy-resin-system in the goods-in of the company. Before it is stored for usage, we need to sample each drum of material, measure its viscosity, its amine content, its strength by making coupons and testing them. All that data needs to be stored with the material as images + pdf’s. Now say one operator is supposed to do a test, what we’d like to do is give him/her an application that shows an image of the workstation, to start the test procedure, it should give them all the instructions in their language if they click on a button or other bunch of images that show the sequence of instructions, each time asking for a response, job done or problem or something else. In the end the application should allow the user to click on the operation finished, and perhaps (s)he make a photo of the job done in some case upload the results file from the machine, or does something else that proves that the job was done. That should pass on the results to the next in route which is usually a Q&A person and the job is checked and approved, and the operator should get the next scheduled job.

Another thing is a rigid structure of naming and numbering everything based on where it is used based on FAA and EASA requirements. So say if the part being worked is going to be used in the Wing_lower_surface all operations and their numbering will be for that. If they’re used on the fuselage_nose they have a different naming. This applies to workstations, operations, operators, machines, tooling, Q&A and all the rest.

As far as CAE assets are concerned, one should have a way for:

  1. revisioning them (through CAD interfaces).

  2. Vaulting them in a safe encrypted and partitioned manner with several levels of safety.

  3. storing their references in the database. The better thing would be to store the entire file as a blob in the dbms in their own column, but that could make a huge table. The attributes such as meta-data on the file could be easily extracted out of the original file before saving.

This will be at the minimum and would be doable. In the erpnext or the PLM application, one could link the file sources for processes.

Currently, I don’t know erpnext enough to make an implementation of whats in my mind and would like to spend a day with someone who’s done a fairly complex implementation to understand whether we should even go this way. There’s a german gentleman, whose company makes electronic parts, who has a few videos on erpnext and I think that their implementation has the same order of complexity as what we’d end up with. I don’t know if he’d even consider a visit from me.

So that is all that I’m working on now.

Have a great day,
Naresh

23

Hi @nash, no real solutions here, only critique from the sidelines…

Yes, but this is common among PLM in general because it is usually not efficient to store large binary blobs in a DBMS and can also reduce performance of the DBMS in general.

Why? Can you show compliance by demonstrating adequate control over documents, regardless of the storage mechanism? Are you required to comply with a specific system architecture? (some standards do, I understand PCI financial payment processing is one, but often ISO and management of change standards only require that you have processes which meet some more general requirements).

Storing document blobs in a database is only as good as the dbms user password. If the password is exposed it really doesn’t matter if the file system is encrypted (unless physical theft of the drive is your greatest concern). Vice versa, if you have good authentication processes, a “belt-and-braces” strategy may only make the system more complicated, not safer (and more complicated is often less safe).

IMHO the key is to have well defined traceable control over documentation related to Items (part numbers) and stock (physical substantiation of a part number). There must be controlled verified entry into the system, controlled management within the system, and no access from outside the system.

These steps can be configured as workflows within ERPNext, with flow moving from one step to the next based on user permissions and assigned roles.

A formal naming and numbering system customized to your unique needs is undoubtedly code you will need to write. I would suggest you keep naming and numbering under centralized manual control (e.g. a “doc control” office) and look to automate later. This can certainly be automated given an appropriate set of rules, but I would not recommend mixing the two.

IMHO you are setting a high minimum bar. I would look for parts of your process that do not need to change (i.e. your manual processes work as-is, although not as efficient as you would like) and initially implement ERPNext for what must be changed (either to correct an aspect of your current manual processes, or if required as part of a minimal ERPNext installation).

If possible, I would try to move automated naming and numbering to a parallel project, Even though it is desirable to complete both projects before going live, any delays in the naming and numbering should not impact the basic ERPNext implementation project. You may argue this is just “tomato tomahto” but perception is everything, and you do not want momentum in the erpnext implementation to be upset if it is difficult to achieve consensus on a definitive naming and number rule, or in its implementation (or just struggling to get consensus on the user interface).

How are you storing CAD documents now? Do you have the concept of a PLM now, or do you have the classic system where design documents are stored by domain (e.g. mechanical, electronics, PCB and especially firmware all have their their own way).

An ERP can be an opportunity to clean house, but only so much before the project loses clarity and focus (and you know where that leads… :wink: ).

What is the primary source of friction in your current system? Is it more related to the transaction processes in an ERP, or the change management of PLM? Can I also assume your “paper trail” process is well documented, and your new electronic system will require certification? I might consider how I might adapt the current certified process to the new process, and if one approach might be preferred over another based on ease of re-certification.

Cheers,
Dale

24

Hi @Dale_Scott

Thanks for your, “Critique from the sidelines”! Much appreciate that, and I mean it.

So here is my plan: I’m going to slowly get something running that represents how we need to do our business. Once it is done, I intend to give it to all for free… including the meta-data tool for the CAE products we use. You can use it or improve it and convert your critique to a viable solution if you dislike it in its native form.

Some of the basics… the reason we cannot do the things as they currently stand. We work very often with customer data that is ITAR and one of the requirements for the data we use is complete transparency in how it is stored in case we’re audited. If that is the case, the data store should not be accessible to anyone else. Now if all the files are stored in one area for all customers, that kind of thing will not work. A local directory separated by customer names + projects may work. A cloud store will not, this is something that is specific to some data we work with. I may be wrong about it, but I believe that in earlier versions of ERPNext, it was possible to create separate directories for customers, and in V12 and in future that may not be the case. Is that true?

Just got back from a longish trip, am going to now huddle up into some preliminary coding.

Cheers,
Naresh

1 Like
25

@nash, I wish you the best of luck and look forward to watching the progress. I was at a very small hardware startup (7 staff) working on their first product, and they will probably start with ERPNext simply attaching related documents to an Item or BOM. For the number of people involved, the risks, and the duration for something better, this will perfectly adequate (but they also working in an industry that, at least in north america, is generally exempt of regulatory compliance requirements).

Regarding specifics of physical file storage, it seems at least by default there is only separation by public or private (although I do not fully understand the reason for, now why one would make one document public but another private). In the v11 ERPNext VM, all uploaded files are in:

/home/frappe/frappe-bench/sites/site1.local/private/files
/home/frappe/frappe-bench/sites/site1.local/public/files

File names of uploaded files are initially preserved, and an upload timestamp is appended to files being uploaded if a file already exists with the same name. For example, if I uploaded file “20000001_WI-01.pdf” for the first time, it will be stored as …/sites/site1.local/private/files/“20000001_WI-01.pdf”, but if I realized the PDF hadn’t been updated yet, fixed the PDF and then uploaded again, the second file will be named (e.g.) “20000001_WI-01 2020-01-25 15:45:02.pdf”.

Good luck and I hope you will post your progress. If you want a beta tester, you know where you can find me… :wink:

1 Like
26

@Dale_Scott

Thanks, will definitely keep you in the loop.

Best,
N