, ,

Mastering Data Import in the Muyan Low-Code Platform: A Comprehensive Guide

Lawrence Liu Avatar
Lawrence Liu

·

, ,

·

Introduction

In the ever-evolving landscape of low-code platforms, efficient data management is crucial for success. The Muyan low-code platform stands out with its innovative “everything is data” approach, offering unparalleled flexibility and power in data import capabilities. This comprehensive guide will walk you through the best practices for data import in Muyan, highlighting its unique features and helping you avoid common pitfalls.

  1. The Muyan Difference: “Everything is Data”

Muyan’s low-code platform is built on a revolutionary principle: everything, including customized code, is treated as data. This approach sets Muyan apart from other low-code platforms in several key ways:

a) Comprehensive Data Relationships:

  • Muyan offers sophisticated relationships between different CSV files and object types.
  • This allows for complex data structures to be imported and maintained with ease.

b) System-Wide Consistency:

  • The entire business system is constructed from seed data, ensuring consistency across all aspects of your application.

c) Unparalleled Flexibility:

  • Customizations, business logic, and even system configurations can be imported as data.
  • This allows for rapid deployment and easy replication of complex systems.

Case Study: Muyan’s Self-Initialization

One of the most compelling demonstrations of Muyan’s data import capabilities is the platform itself. The entire Muyan system initializes and configures itself using a set of predefined data. All customizations, logic, and system configurations are loaded into the system using CSV files. This showcases the power and flexibility of Muyan’s data import features, allowing complex systems to be deployed and replicated with ease.

Data Import Methods in Muyan

Muyan offers multiple methods for importing data, each suited to different scenarios:

a) Automatic Import on System Startup:

  • Muyan can automatically import CSV files from a configured server directory upon system startup.
  • This method is ideal for initializing the system with essential data.

b) Manual Import from Domain List Page:

  • For each domain model, users can manually import CSV files through Muyan’s intuitive interface.
  • This method provides flexibility for updating or adding data on-demand.

c) Reloading Seed Data:

  • On the domain model list page, users can use the “Reload Seed Data” action to refresh data from a configured seed data folder.
  • This is particularly useful for resetting the system to a known state or updating reference data.

d) Importing Seed Data Package:

  • Muyan allows users to upload a packaged zip file containing CSV and other referenced files.
  • This method is ideal for transferring a complete set of data between environments or systems.

Preparing Your CSV Files for Muyan

The structure and content of your CSV files are crucial for successful data import in Muyan. Here are some best practices:

File Naming

  • Name your CSV files starting with the class name of the object, without the package definition.
  • Example: For a tech.muyan.Contract object, use Contract.csv or Contract_2023.csv.

Header Row

  • The first row of your CSV file should always be the header row.
  • It describes the structure of the data in the file.
  • Example: name(*),title,effectiveDate,ownerOrganization.name,Tags(#),Purpose(#)

Query Fields

  • In Muyan, columns with the suffix (*) are query fields used to find existing records.
  • Example: name(*) indicates that the ‘name’ field will be used to query existing records.

Associated Object Queries

  • Muyan uses dot notation for associated object fields.
  • Example: ownerOrganization.name allows querying associated Organization objects.

Dynamic Fields

  • In Muyan, columns with the suffix (#) represent dynamic fields.
  • Example: Tags(#) indicates a dynamic field for tags.

Special Columns

  • DELETE_FLAG: Muyan uses this column to mark rows for deletion (value ‘Y’ for delete).

Comment Lines

  • Muyan supports comment lines in CSV files, starting with a semicolon (;).
  • These lines are ignored during import, allowing for better documentation.

Technical Details of Data Import

Here’s a revised structure for the blog post that incorporates these elements:

  1. Introduction (keep as is)
  2. The Muyan Difference: “Everything is Data” (keep as is)
  3. Data Import Methods in Muyan (keep as is, including videos)
  4. Preparing Your CSV Files for Muyan
    • File Naming
    • Header Row
    • Query Fields
    • Associated Object Queries
    • Dynamic Fields
    • Special Columns
    • Comment Lines
  5. Technical Details of Data Import
    • File Update Determination
    • Import Order and Dependencies
    • Data Type Mapping
    • Handling Special Data Types (Boolean, Date and Time, Enumeration Types, Attachments)
    • Root Directory Fields
  6. A Complete Example (include the example from the original document)
  7. Best Practices and Avoiding Common Pitfalls
  8. Advanced Techniques: Importing System Configurations and Logic
  9. Troubleshooting and Support
  10. Conclusion

I’ll now provide the revised sections that include the technical details and the example:

  1. Technical Details of Data Import

Understanding the technical aspects of Muyan’s data import system is crucial for effective use:

a) File Update Determination:

  • Muyan compares the MD5 code of the CSV file to be imported with the previously imported file.
  • If the MD5 codes match and the previous import was successful, the system skips the import.

b) Import Order and Dependencies:

  • Muyan automatically identifies dependencies between data based on model definitions.
  • You can define special dependencies using loadAfter in the extInfo of the domain definition.

c) Data Type Mapping: Muyan supports various data types for import. Here are some key mappings:

  • Boolean: Supports values like ‘Y’, ‘Yes’, ‘TRUE’, ‘1’ for true, and ‘N’, ‘No’, ‘FALSE’, ‘0’ for false.
  • Date and Time: Supports multiple formats, including “yyyy-MM-dd HH:mm:ss”.
  • Enumeration Types: Can use display values if a reverse lookup method is implemented.
  • Attachments: Use relative file paths from the configured seedData.folder directory.

d) Handling Special Data Types:

  • Root Directory Fields: Use $ROOT_ORG$ placeholder for the current tenant’s root organization.
  • StorageFieldValue: For attachment type columns, use relative file paths.

e) Support for New Data Type Mappings:

  • Developers can register new data mapping implementations using ConverterFactory.registerConverter.

A Complete Example

Let’s walk through a complete example of a CSV file that can be imported into the Muyan system:

File name: Contract.csv

CSV content:


Copied!

name(*),title,effectiveDate,total,ownerOrganization.name(*),contractStatus,Tags(#),Purpose(#),Total Amount(#),Seal Type(#),Contract Quantity(#),Is Important(#),language,lines.name[:],DELETE_FLAG
MuYan Technology 2020 Laptop Purchase Contract,Laptop Purchase Contract,2020-01-01 00:00:00,45000,MuYan Technology,Active,"[\"One-time Contract\",\"Low Risk\"]","Purchase laptops for development use",35000,Official Seal,3,Yes,contracts\language_2020.docx,"[line1:line2]",N
MuYan Technology 2019 Laptop Purchase Contract,Laptop Purchase Contract,2020-01-01 00:00:00,45000,MuYan Technology,Active,"[\"One-time Contract\",\"Low Risk\"]","Purchase laptops for development use",35000,Official Seal,3,Yes,contracts\language_2019.docx,"[line3:line4]",Y

Key points about this example:

  • The file name Contract.csv maps to the Contract domain model.
  • Query fields (name(*) and ownerOrganization.name(*)) are used to look up existing records.
  • Associated object query (ownerOrganization.name) demonstrates relationship handling.
  • Dynamic fields (ending with (#)) show how to import custom attributes.
  • The effectiveDate field demonstrates date formatting.
  • The contractStatus field shows how enumeration types are handled.
  • The Tags field demonstrates handling of multiple-choice fields and escaping.
  • The language field shows how to import attachments.
  • The lines.name[:] field demonstrates importing list fields.
  • The DELETE_FLAG shows how to mark records for deletion.

This example illustrates how Muyan’s import system can handle complex data structures, relationships, and operations in a single CSV file.

Best Practices and Avoiding Common Pitfalls

While Muyan’s data import capabilities are powerful, there are some key best practices to follow and pitfalls to avoid:

a) Setting Query Fields Correctly:

  • Always remember to set query fields (marked with *) in your CSV files.
  • Forgetting to set query fields will cause the import engine to treat every line as a new entry, potentially creating duplicates.

b) Ensuring Unique Query Results:

  • Make sure your query fields are specific enough to return a single record for each line.
  • If multiple records are queried for a single line, the system will fail to import that line.

c) Testing Imports:

  • Always test your imports in a non-production environment first.
  • Use a small subset of data to verify your CSV structure and content before attempting large imports.

d) Maintaining Data Relationships:

  • Take advantage of Muyan’s ability to maintain relationships between different data types.
  • Ensure that related data is imported in the correct order to maintain data integrity.

Advanced Techniques: Importing System Configurations and Logic

Muyan’s “everything is data” approach extends beyond basic data import, allowing you to configure complex system behaviors through CSV imports. Here are key areas where this can be leveraged:

a) Custom Logic with DynamicLogic:

  • Import business rules and workflows as DynamicLogic through CSV files.
  • Associate DynamicLogic with system components like DynamicFieldHook, DynamicObjectHook, DynamicAction, DynamicScheduleTask, and DynamicIntegration.

b) System Configurations:

  • Import user roles, permissions, and system-wide settings via CSV.
  • Configure UI elements like menus, themes, and forms through data imports.

d) Reporting and Dashboards:

  • Set up custom reports and dashboards by importing DynamicReport configurations.

This approach allows for rapid deployment of complex systems, easy replication of configurations across environments, and version control of your entire system setup. By mastering these techniques, you can fully utilize the flexibility and power of the Muyan platform, implementing sophisticated business logic and system behaviors entirely through data-driven configurations.

Real-World Application: MES System Configuration

One powerful application of Muyan’s ‘everything is data’ approach is in configuring a Manufacturing Execution System (MES). Using CSV imports, users can model an entire production line, including:

  • Equipment configurations
  • Product specifications
  • Material definitions
  • Manufacturing recipes

This approach allows for rapid setup and easy modification of complex manufacturing processes, all through data imports rather than traditional coding. It demonstrates how Muyan can quickly adapt to specific industry needs while maintaining flexibility for future changes.

Monitoring and Analyzing Import Records

An essential feature of Muyan’s data import system is the ability to view and analyze import records. This functionality is crucial for maintaining data integrity, troubleshooting issues, and optimizing your import processes. Let’s explore the ImportRecord page and its key features:

a) Import Record Overview: The ImportRecord page provides a comprehensive list of all import attempts, successful or otherwise. Each record contains vital information about the import process, including:

  • ID: A unique identifier for each import attempt
  • Domain class: The type of data being imported (e.g., DynamicMenu, DynamicReport, DynamicConfig)
  • Status: The outcome of the import (e.g., Failed, Successful)
  • Start date and Finish date: Timestamps for the import process
  • Updated ids: Number of records updated during the import
  • Failed lines: Number of lines that failed to import
  • Skipped lines: Number of lines skipped during import
  • Conflict lines: Number of lines that had conflicts during import

Additionally, there’s a crucial Log column that provides comprehensive details about the import process. This log is particularly valuable for troubleshooting and understanding the specifics of any conflicts or issues. For example:

Conflict detected for line 2, 
values: [organization:tech.muyan.Organization : 1, parent:tech.muyan.dynamic.form.DynamicMenu : 3, label:DynamicReport, icon:PrinterOutlined, link:null, type:FORM, displaySequence:94, enableRoles:null, name:DynamicReport], 
existing: tech.muyan.dynamic.form.DynamicMenu : 132, 
last imported: tech.muyan.importexport.ImportLineSuccessRecord(csvFileHeader:organization.name(*),parent.label,label(*),icon,link,type,displaySequence,enableRoles, lineContent:$ROOT_ORG$,Development,DynamicReport,PrinterOutlined,,FORM,94,, importedAt:2023-09-15T06:48:29.496626Z[Etc/UTC], importRecordId:317)

This log entry provides:

  • The specific line number where a conflict was detected
  • The values being imported
  • The existing record that conflicts with the import
  • Details of the last successful import of this record, including the CSV file header, line content, and timestamp

This level of detail is invaluable for diagnosing import issues, understanding data conflicts, and maintaining data integrity across imports.

b) Status Monitoring: The ‘Status’ column allows you to quickly identify successful and failed imports. In the example image, we can see several failed imports, indicated by the red “Failed” status. This immediate visual feedback helps in quickly identifying issues that need attention.

c) Detailed Import Statistics: The page shows columns for Updated ids, Failed lines, Skipped lines, and Conflict lines. These provide a quick overview of the import process outcomes. For instance, record #4392 shows 5 updated ids, 25 failed lines, and 10 skipped lines, giving a clear picture of the import results.

d) Object Type Diversity: The import system handles a wide variety of object types, as evidenced by the different entries in the “Domain class” column. This includes system configurations like DynamicMenu, DynamicReport, and DynamicConfig, showcasing Muyan’s “everything is data” approach.

e) Filtering and Navigation: The page includes filtering options and pagination, allowing users to easily navigate through large numbers of import records and find specific imports they’re interested in.

Best Practices for Using Import Records

  1. Regular Monitoring: Make it a habit to check the ImportRecord page after performing imports, especially for critical data or system configurations.
  2. Troubleshooting: When encountering failed imports, investigate the Failed lines, Skipped lines, and Conflict lines to identify and rectify issues in your CSV files or import process.
  3. Audit Trail: Use this page as an audit trail to track changes to your system’s data and configurations over time.
  4. Performance Optimization: Analyze the start and finish dates to identify any imports that are taking longer than expected, which might indicate areas for optimization.
  5. Data Integrity Checks: Regularly review the import records to ensure that all expected imports are occurring successfully, maintaining the integrity of your system’s data.
  6. Analyze Log Details: When troubleshooting failed imports or conflicts, dive deep into the Log column. It provides specific information about which lines caused issues and why, allowing for targeted fixes in your import data or processes.
  7. Version Tracking: Use the detailed log information to track changes in your data structure or content over time, which can be crucial for understanding system evolution and debugging complex issues.

By effectively utilizing the ImportRecord feature, including its detailed logs, you can ensure smoother data management, quicker issue resolution, and maintain a clear history of changes to your Muyan-based application.

This tool is invaluable for both system administrators and developers in maintaining a robust and reliable low-code platform environment, providing the depth of information needed to manage complex data imports effectively.

CSV Templates for Comprehensive System Configuration

Muyan provides an extensive set of CSV templates that cover virtually every aspect of system configuration and data management. These templates are crucial for efficiently importing data and setting up your Muyan-based application. These templates are provided for reference and are optional. Users can select and use the templates relevant to their specific use case or system configuration needs.

Here’s an overview of the available templates:

  1. User and Organization Management:
    • Organization.csv: Define organizational structure
    • Role.csv: Set up user roles
    • Group.csv: Create user groups
    • GroupRole.csv: Assign roles to groups
    • User.csv: Import user data
    • UserGroup.csv: Associate users with groups
  2. Security and Access Control:
    • RequestMap.csv: Configure URL-based security mappings
  3. Domain Model Configuration:
    • DomainClass.csv: Define custom domain objects
    • DomainClassField.csv: Specify fields for domain objects
  4. Business Logic and Automation:
    • DynamicLogicEngine.csv: Set up logic execution engines
    • DynamicLogicType.csv: Define types of dynamic logic
    • DynamicPrompt.csv: Configure system prompts
    • DynamicLogic.csv: Import custom business logic
    • DynamicFieldHook.csv: Set up field-level hooks
    • DynamicObjectHook.csv: Configure object-level hooks
  5. User Interface and Experience:
    • DynamicMenu.csv: Customize navigation menus
    • DynamicForm.csv: Create custom forms, wizard definitions, and dashboard layouts
    • DynamicFormGroup.csv: Define groups of form fields within a single form
    • DynamicFormField.csv: Define form fields
    • DynamicFormWizardStep.csv: Set up form wizards
    • DynamicDashboardWidget.csv: Configure dashboard widgets
    • DynamicTheme.csv: Customize system appearance
  6. Actions and Workflows:
    • DynamicAction.csv: Define custom actions
    • DynamicActionGroup.csv: Organize actions into groups
    • DynamicActionDomainClass.csv: Associate actions with domain classes
  7. Data Management and Filtering:
    • DynamicFilter.csv: Set up data filters
    • DynamicFieldDefinition.csv: Define custom fields
    • DynamicFieldInstance.csv: Create instances of custom fields
  8. System Configuration and Integration:
    • DynamicConfig.csv / DynamicConfig_withFile.csv: Manage system configurations
    • DynamicPlugin.csv: Configure system plugins
    • DynamicIntegration.csv: Set up system integrations
    • DynamicIntegrationDomainClass.csv: Link integrations with domain classes

How to Use Templates:

  1. Access the templates at https://docs.muyan.io/en/importexport/CSV_FILE_TEMPLATES.html
  2. Download the appropriate template(s) for your configuration needs
  3. Open the template in a spreadsheet application or text editor
  4. Fill in your data, following the structure and data types specified in the header row
  5. Save the file as a CSV, ensuring it follows the naming conventions
  6. Import the file using one of the methods described in the “Data Import Methods” section

Best Practices:

  • Always use the latest version of templates to ensure compatibility with your Muyan version
  • Pay close attention to required fields (often marked with (*) in the header)
  • Use the templates as a guide for data structure, even when preparing large datasets programmatically
  • Consider the relationships between different templates (e.g., Users, Groups, and Roles) when planning your imports
  • Start with core configurations (like Organizations and Users) before moving to more specialized templates
  • When working with forms, use DynamicForm.csv for overall form structure, DynamicFormGroup.csv for organizing fields within forms, and DynamicFormField.csv for individual field definitions
  • Utilize DynamicPrompt.csv to set up templates for AI interactions and system commands, enhancing the platform’s automation capabilities

By leveraging these comprehensive templates, you can configure every aspect of your Muyan-based application efficiently and consistently. Whether you’re setting up a new system, migrating data, or updating configurations, these templates provide a structured approach to managing your low-code platform.

For detailed descriptions of each template and field-specific guidelines, refer to our comprehensive documentation at https://docs.muyan.io/en/importexport/CSV_FILE_TEMPLATES.html

Best Practices for Managing CSV Files

Muyan offers a git template that serves both as a developer environment and a data/code management repository. This approach allows teams to:

  • Version control their CSV files alongside code
  • Track changes to system configurations over time
  • Collaborate effectively on system setups
  • Roll back to previous configurations if needed

By treating configuration data as code, teams can apply software development best practices to their low-code platform setup, ensuring consistency and traceability.

Troubleshooting and Support

Even with best practices, you might encounter issues. Here are some tips for troubleshooting:

a) Check Import Logs:

  • Muyan provides detailed import logs. Always review these for any errors or warnings.

b) Validate CSV Structure:

  • Ensure your CSV files are correctly formatted and adhere to the RFC 4180 standard.

c) Verify Data Types:

  • Double-check that your data matches the expected types for each field.

d) Seek Community Support:

  • Engage with the Muyan user community for advice and shared experiences.

Conclusion

Muyan’s unique “everything is data” approach and powerful import capabilities set it apart in the low-code platform landscape. By treating all aspects of your application as importable data, Muyan offers unparalleled flexibility and power. Whether you’re setting up a new system, replicating configurations, or managing complex data relationships, mastering Muyan’s data import features will allow you to leverage the full potential of this innovative platform.

We encourage you to explore these techniques in your Muyan projects. Start small, test thoroughly, and gradually expand your use of these powerful import features. With practice, you’ll be able to rapidly deploy and manage complex applications with ease.

Additional Resources

To further support your journey with Muyan’s data import capabilities, we’ve compiled some valuable resources:

  1. Comprehensive Documentation: For a deep dive into all aspects of data import and export in Muyan, visit our detailed documentation at https://docs.muyan.io/en/importexport/
  2. CSV File Templates: To help you get started quickly, we’ve prepared CSV file templates for various import scenarios. You can find these templates at https://docs.muyan.io/en/importexport/CSV_FILE_TEMPLATES.html
  3. Community Support: Join our vibrant community of Muyan users and developers at https://community.lcdp.ai. Here, you can ask questions, share experiences, and learn from others who are leveraging Muyan’s powerful features.

Remember, the Muyan team and community are here to support you on your journey. Don’t hesitate to reach out with questions or share your success stories. By leveraging these resources and the power of Muyan’s data import capabilities, you’re well-equipped to create sophisticated, data-driven applications with unprecedented ease and flexibility.

Happy importing, and we look forward to seeing the amazing applications you’ll build with Muyan!

Discover more from LCDP.ai Infinity Code

Subscribe to get the latest posts to your email.

Leave a Reply